Mixing live television in a browser

Live video mixing is the job a television gallery does: take several incoming camera feeds, cut and mix between them in real time, and send one clean feed out, all while everything is happening live and nothing can be done again. It is usually the domain of dedicated hardware and a room built around it. We were asked whether it could be done in a browser instead.

The honest first answer to that kind of question is “probably, but the risk is not where you think it is.” So the first thing we did was find out.

Prototype the scary part first

There is a temptation on a project like this to start with the parts you know how to build: the interface, the accounts, the plumbing. We do the opposite. The whole value of the work lived in one uncertain question, whether a browser could handle live feeds with the timing this needs, so that is what we built first, before committing to anything else.

That prototype was not pretty and was never meant to ship. It existed to answer one question honestly, and it did. It told us where the real constraints were, and it meant the plan we then made was built on something we had seen work rather than something we hoped would.

The constraint you cannot argue with

Live video has an iron triangle: latency, quality, and cost. You can have a sharp picture, you can have low latency, and you can keep the cost sane, but you cannot have all three at once, and no amount of clever code changes that. Push quality up and either latency or cost follows. Squeeze latency and you pay for it somewhere else.

A lot of the engineering was not about beating that trade-off, because you cannot. It was about placing it deliberately, in the right spot for this use, and being honest with everyone about where we had placed it. The IP networks in the middle do not offer guarantees the way a dedicated video link does, so a sensible chunk of the work went into behaving well when the network did not.

Where the effort actually went

The mixing itself, once we understood it, was the smaller part. The larger part was everything around the edges: keeping feeds in sync, failing gracefully when one misbehaved, and making the tool usable by someone concentrating on a live broadcast rather than on our software. Under that kind of pressure an interface has to be quiet and predictable. There is no room for surprises when it cannot be taken back.

What we took from it

The lesson was not really about video. It was that on any project with one genuinely hard part, the job is to find that part early, prove it or disprove it before the plan hardens, and then be straight about the constraints you cannot engineer away. That is as true of the work we do now as it was here.