Automating HLS-Stream-Creator (Python)

Ben Tasker

2018-04-13 11:33

My HLS Stream Creator script takes input videos and outputs adaptive HLS streams. However, it was written as an experiment and developed over time, so has always been focused more on manual runs.

This snippet gives an example of wrapping automation around the script, and (a slightly different version) is in active use on one of my muxers, fetching muxing jobs from an upstream API (hence the slightly odd input to triggerMux).

It may need a little adaptation for use-cases that differ to mine, for example, the output directory is written into the same path as the input video and all input videos exist under the same base directory (basically, a NFS mount point), but should serve to show the basic concept

Use a variable to fetch an object attribute (Python)

Ben Tasker

2018-04-12 09:22

Sometimes you want to use a variable to contain the name of an attribute that you might later want to fetch from an object. For example, if you're doing hashing operations you might want to have the algorithm name in a variable, and use that in order to trigger behaviour (rather than having a massive if block, duplicating calls).

Generating HMACs with different hashing algorithms (Python)

Ben Tasker

2018-04-12 09:06

Various services require a token to be submitted using a HMAC. Although SHA1-HMAC is common, other hashing algorithms are sometimes required. This snippet allows you to specify the hashing algorithm to use

Check if SSL certificate matches private key (BASH)

Ben Tasker

2018-04-12 08:47

Sometimes you need to double check that a private key matches your certificate (if, for example, you've got multiple badly named keys on your system). You might also occasionally want to check if a certificate matches a Certificate Signing Request (CSR)

Generate a SHA1-HMAC (Python)

Ben Tasker

2018-04-12 08:35

SHA1-HMAC's are often used by API's (and in tokenisation) as a means to prove control over a shared secret. They allow proof of control without sending the secret over the network.

Bulk Reverting Commits (Git)

Ben Tasker

2018-04-11 13:59

Usually, you only want to revert a single commit at once. Just occasionally, though, you want to revert multiple commits all in one fell swoop (making it either a very sad or a very happy day).

Supporting conditional HTTP requests in PHP (PHP)

Ben Tasker

2018-04-05 08:15

RFC 7232 implements support for conditional requests in HTTP. These are commonly used by downstream caches (including those built into browsers) in order to revalidate content - i.e. check whether the copy in cache is stale or whether it should still be considered valid.

The primary mechanisms for this are If-Modified-Since (i.e. has it chanced since date x) and If-None-Match (i.e. does it's Etag match any of those). If the metadata validates, then a HTTP 304 (Not Modified) will be returned, otherwise the requested content will be returned using whatever status code is appropriate for the request (usually a 200, but it could equally be a 206 if the request specifies a range)

Because PHP is used in dynamic pages, conditional requests won't simply work out of the box. To support conditional requests, you need to generate some metadata for whatever asset is being requested and include that in your initial response to the client. You can then validate that if it's later included in a conditional request

The benefit of doing this, though, is that you can save quite a few CPU cycles where a revalidation is possible - particularly if there are any components of your page/asset which are expensive to generate

Returning HTTP status codes in PHP (PHP)

Ben Tasker

2018-04-05 07:46

Most of the time, a PHP application won't need to change the HTTP status code that Apache or FastCGI generates on it's behalf, but occasionally there's a need to do so.

This snippet lists some of the common HTTP Status codes and how to return them

SystemCTL Cheat Sheet (BASH)

Ben Tasker

2018-04-04 09:03

Whether or not you like SystemD it is a reality of the world we currently live in, and anyone managing a modern operating system is likely to have a need to interact with it.

Utilities such as service,ckconfig and update.rc-d have largely been replaced with systemctl and it's syntax isn't always as simple and intuitive - especially when you're used to the other ways of doing things

Reducing latency of SSH tunnelled connections (BASH)

Ben Tasker

2018-04-04 08:37

The ability to tunnel connections over an SSH connection is incredibly useful, essentially creating a poor man's VPN, whether to a specific port (ssh -L 1443:google.com:443 ssh-server.domain) or by standing up a SOCKS proxy (ssh -D 8080 ssh-server.domain).

The most common complaint when doing this, though, is that the latency sucks. This is because of the TCP over TCP Problem. You're essentially running two layers of TCP congestion control, so any loss hurts - a lot.

This can be addressed by installing and using SShuttle (docs here).

It works (on Linux and Mac OS X) by intercepting and termination TCP connections locally, and transmitting payload and packet metadata over a SSH connection as data. The remote end re-assembles the packets and transmits on your behalf. The remote end only needs to have Python installed (as sshuttle remotely runs a python snippet to handle receipt and re-assembly of packet information).

The result is, there's no TCP connection within the tunnel, so you're still only contending with a single layer of congestion control