AWS Adventures, Part 3 – HA Wowza Live HLS

For a number of years, we have streamed HLS video via CloudFront, using a Wowza Streaming Engine server to convert our RTMP streams to HLS on the fly. CloudFront provides almost infinite scalability for the HLS stream, since the static chunk files are highly cacheable.

For high availability purposes, we want to use two independent WSE servers in two AWS availability zones. But this has been problematic. The two servers are never 100% in sync with their HLS chunking of the incoming live stream. This can cause the client to get a bad response to a request, thereby dropping the live stream.

After a lot of experimentation, I have come up with a way to assemble a multi-AZ, high availability cluster of WSE servers that can reliably stream HLS video from an incoming RTMP stream.

Let’s look at what happens if you try to serve HLS from two WSE servers that are both ingesting the same live RTMP stream. Eventually, your client will ask one of the servers for a chunk that is not yet ready on that server. To understand this, you can look at a specific sequence of events that might happen in the middle of a streaming session.

Suppose we have two WSE instances, “Server A” and “Server B”. Both are relaying a live stream from the same liveorigin WSE instance. This session has already been in progress for a while; the client has already made multiple requests for updated chunk lists, pulled multiple chunk files, and played back a number of them.

  1. Client makes request for the updated chunk list
  2. Server A creates chunk 24 (note: Server B has not yet created the chunk 24)
  3. Load balancer routes request to Server A
  4. Server A responds with a chunk list containing chunks 15-24
  5. Client already has chunks 15 through 23, so it requests chunk 24
  6. Load balancer routes request to Server B
  7. Server B does not have chunk 24, so it responds with a 404
  8. Client may or may not handle the 404 gracefully; in the case of VideoJS with the videojs-hls-contrib plugin, it will completely drop the stream

In order to provide a stable stream, we need to make sure that once a client connects to a live stream, it stays with a single server. We do this with a little bit of DNS magic mixed with a little intelligence in the application load balancer.

General approach

Here’s a diagram showing the general architecture. The Wowza ingress servers receive the live stream from the video encoders, and then they repeat out to the Wowza HLS servers. An ALB balances the traffic across the two HLS servers, and CloudFront sits in front of that, delivering playlists, chunklists, and chunk files to your end users.

HA Wowza Live HLS

Autoscaling groups

It’s always a good idea to put your servers into auto-scaling groups. Even if you don’t need to scale up and down, an auto-scaling group can ensure that you always have at least one instance running for a particular function. Define two single-host autoscaling groups (min = 1, max = 1, desired = 1) for your HLS servers. Put each in a private subnet in separate availability zones.

Define a single two-host autoscaling group (min = 2, max = 2, desired = 2) across the two subnets / availability zones for your RTMP ingress servers. Define two Elastic Network Interfaces, one per subnet, and use a userdata script to assign the appropriate ENI to each instance as it starts up in one of the AZs. Here’s an example to get you started:

Each autoscaling group needs a launch configuration that uses the Wowza AMI and has a userdata script that will configure WSE so that they have different hostnames. We’ll get to the Wowza config shortly, so you can come back around to set up the userdata later. For now, use the official Wowza AMI appropriate for your licensing.

Target groups

You will need to define three target groups:

  • wowza-hls: contains both autoscaling groups
  • wowza-hls-az1: contains only the AZ1 autoscaling group
  • wowza-hls-az2: contains only the AZ2 autoscaling group

Load balancer config

Set up an application load balancer. Define a listener for port 80. Set its default to route to target group wowza-hls. Add two rules:

  • if Host == media-hls-az1.example.com then forward to target group wowza-hls-az1
  • if Host == media-hls-az2.example.com then forward to target group wowza-hls-az2

Wowza Streaming Engine config

There are a number of key configurations you need to make to your WSE application. The Wowza documentation will tell you about configuring a server as an HTTP caching origin, and that’s a good start. But their techniques alone won’t be enough to let you run a multi-instance cluster without live stream stability problems.

Configure your RTMP ingress servers to receive an incoming live stream, and configure your HLS servers to point to those origin servers. Wowza provides documentation for this. Note: the HLS server in AZ1 should list the AZ1 ingress server first, and the HLS server in AZ2 should list the AZ2 ingress server first to minimize the amount of inter-AZ traffic, which does have an associated cost in AWS.

Follow Wowza’s guidance to configure the WSE application as an HTTP caching origin by adding the following property to the <HTTPStreamer> / <Properties> container:

Next, set some cache control on your playlist and chunk files. Add the following properties to the <HTTPStreamer> / <Properties> container:

Here’s the part that Wowza doesn’t tell you about (and this is one of the keys to keeping the sessions stuck to a particular server). Configure the application to use absolute paths in its HLS URLs:

The server in AZ1 should have a cupertinoPlaylistHostDomain value of media-hls-az1.example.com and the server in AZ2 should have a value of media-hls-az2.example.com. This makes sure that every URL output by server X during an HLS session is fully qualified and will cause the ALB to route subsequent requests back to it (or to CloudFront cached content that originated with server X.

CloudFront

CloudFront allows your HLS streaming servers to scale more or less infinitely. Very little traffic will come back to your Wowza servers, so the fact that you have just two instances doesn’t matter. CloudFront will do the heavy lifting. As your traffic grows, CloudFront will absorb a higher and higher percentage of the requests, keeping the load essentially constant on your Wowza servers.

Set the origin of the CloudFront distribution to your ALB.

Give the CloudFront distribution three alternate domain names:

  • media-hls.example.com
  • media-hls-az1.example.com
  • media-hls-az2.example.com

This makes sure that the CloudFront distribution can handle requests for any of these three hostnames. We will start users with a media-hls.example.com playlist URL, and if it’s a cache miss, the ALB will route that request to one of the two Wowza servers. That server will respond with a playlist that uses an absolute chunklist URL to make sure the next request comes back to the same server. The chunklist will in turn contain absolute URLs to the chunk files, keeping all of those requests on the same server.

And what if it’s not a cache miss? We’ve got that covered.

Edit the default behavior of the CloudFront distribution. Set “Forward Headers” to “Whitelist”, and add the header “Host” to the whitelist. This does two things:

  • Passes the host header through to the ALB, which allows the ALB to route cache misses for chunklists and chunk files to the appropriate target group.
  • Factors the Host header into the caching key, so that chunklist.m3u8 from media-hls-az1 isn’t “mixed up” with the chunklist.m3u8 from media-hls-az2. Two separate chunklist.m3u8 files will be cached, and depending on the Host header presented by the client, the appropriate file will be returned, keeping the streaming session stable.

DNS config

Finally, we need to make these hostnames actually go somewhere. We use Route 53 to manage our domain, but you could apply these techniques with any DNS provider. Define three hosts, all pointed to your CloudFront distribution:

  • media-hls.example.com
  • media-hls-az1.example.com
  • media-hls.az2.example.com

Test it out

Spin up a live encoder and have it stream to your ingress servers:

Take your favorite HLS player (VLC works great for testing, or use VideoJS HLS), and point it to

You should see your live stream, and if you are able to monitor the network traffic (like with VideoJS HLS), you’ll see that after the initial request for playlist.m3u8, all your requests will either go to wowza-hls-az1 or wowza-hls-az2, resulting in a highly stable stream.

Other approaches

This isn’t the only way to build a high-availability HLS solution on HLS. But I think it’s the only way to do it with an RTMP stream coming out of your live encoder. Some other methods you might consider:

  • If your encoder supports it, you could have it encode to HLS and send the chunklist and chunk files directly to S3, and let CloudFront serve directly from S3. This is a little added complexity on the encoder end, but it pays off with reduced complexity in the serving stack.
  • You might consider trying to use sticky sessions that are cookie-based. In order to do this, you have to use a classic ELB, not an application load balancer. You can add a custom plugin to Wowza Streaming Engine that will generate a cookie suitable for an ALB sticky session. But the big caveat is that you might require special configuration of your HLS player to allow it to pass cookies (many do not pass them by default). Even worse, some players will not pass cookies at all, so if you need broad accessibility of your HLS streams, this solution is not going to be for you.

Leave a Reply

Your email address will not be published. Required fields are marked *