DEV Community

Cover image for One Cast button, three kinds of receiver: how a phone decides what to send
Yonto Player
Yonto Player

Posted on

One Cast button, three kinds of receiver: how a phone decides what to send

The Cast button in Yonto Player's phone app looks like one feature. Behind it are three receivers that have almost nothing in common: a Chromecast, an AirPlay screen and a DLNA TV. They speak different protocols, find the phone in different ways, and play different files.

This post is about the part I find most interesting, which is not the protocols. It is the question every cast has to answer before it sends a single byte: what exactly should the receiver be given?

The receiver fetches the video itself

All three work the same way at the top. The phone does not push video frames to the TV. It tells the receiver an address, the receiver fetches the video from that address, and the phone becomes a remote with a play button, a slider and a stop button.

So the phone has to answer two things first. Can the receiver reach the video where it already is? And can the receiver actually play what it finds there?

For the first one, the app serves the video itself when the receiver cannot just fetch it. If the source needs request headers or a sign-in, or the video lives on the phone, the app relays it from a small server on the phone's Wi-Fi address. Otherwise the receiver goes straight to the source. For a Jellyfin, Emby or Plex server, the app also asks the server again for a version the receiver plays, because the receiver's decoders are not the phone's.

Three protocols, one session interface

Each receiver kind is a session class behind one interface, CastSessionClient, with the same small vocabulary: load, play, pause, seek, stop, and a stream of position updates. Google Cast uses Google's sender SDK. Apple publishes no sender SDK for Android, so AirPlay goes through an open-source library that speaks AirPlay 2 video. DLNA is our own code: an SSDP search, the device description, and a dozen SOAP calls such as SetAVTransportURI, Play and Seek.

The player on the phone hands playback over to the session, takes it back when you press Stop casting, and resumes at the same place.

The receivers differ in what they report back. DLNA has no events without running a server on the phone, so that session polls the receiver once a second. A DLNA renderer that stops before it ever starts has met a format it cannot play, and the phone says so instead of showing a black screen.

Decide by what the file is

The interesting decision is what to send. Receivers differ in which containers they open and which codecs they decode, and a title's address says little about any of that. A media server's stream address often has no file extension at all, and a file called .mp4 can be an MKV.

So the cast does not trust the name. Before it loads anything on the receiver, the app reads the first 256 bytes of the stream and recognises the container from its signature. The name is only the fallback when the bytes cannot be read. It is a short list of checks, such as Matroska's header and its document type for MKV and WebM, the ftyp box for MP4 and QuickTime, and the repeating sync byte of an MPEG transport stream.

Each receiver is described by a small capability record: the containers it opens as they are, and the codecs it decodes. The decision is made in two steps, and here is a simplified sketch of them (generic names, not the real code):

// Simplified sketch, not the real classes. Step 1 looks at the container only.
fun needsRepackaging(container: String?, receiver: Receiver): Boolean =
    container != null && container !in receiver.containers

fun load(spec: LoadSpec, receiver: Receiver, session: Session, remuxer: Remuxer) {
    if (!needsRepackaging(spec.container, receiver)) return session.load(spec)  // send as is
    remuxer.start(spec.url, receiver) { stream -> session.load(spec.copy(url = stream.url)) }
}

// Step 2, inside the remuxer, after it has probed the stream.
fun remux(probe: Probe, receiver: Receiver) {
    if (probe.video !in receiver.video) encodePictureAsH264(probe)   // convert
    else copyPicture(probe, reencodeAudio = probe.audio !in receiver.audio)  // repackage
}
Enter fullscreen mode Exit fullscreen mode

The first step is cheap and needs no knowledge of codecs: does this receiver open this container? If it does, the load goes straight through to the Chromecast, AirPlay or DLNA session untouched. If not, a remuxer takes over, and only once it has probed the file does it decide between copying the picture and encoding it.

Three outcomes

Send it as is. For an MP4 this is usually all it takes: the receiver fetches the file and plays it. A Chromecast opens MP4, WebM and MPEG-TS with H.264. An AirPlay screen takes MP4 and HLS with H.264 or H.265. A DLNA renderer is assumed to take MP4 and MPEG-TS with H.264, and its own list of supported types can add more. A TV that lists MKV is sent the MKV as it is.

Repackage the container. Often only the box is the problem: an MKV with H.264 video, say, that a Chromecast will not open. The phone then reads the file with libVLC, copies the video and audio streams into HLS segments without touching the picture, and serves the playlist on the LAN. If the audio is something the receiver does not decode, such as DTS, only the audio is converted, to AAC. Copying the picture involves no decoding or encoding of it.

Convert the picture. Sometimes the picture itself is the problem: an AVI with MJPEG or Xvid video, or H.265 on a Chromecast. libVLC decodes the picture and the phone's own hardware encoder, through MediaCodec, makes H.264 from it, with AAC sound. A picture larger than the receiver takes is scaled down, and HDR10 or HLG is tone-mapped to SDR. If the phone's encoder cannot take the picture's size, the cast says it cannot play it, rather than trying anyway.

The delay

Sending as is needs no converting (the first-bytes read still takes a moment). The other two start a few seconds later, because the phone has to probe the file and write the first segments before the receiver has anything to play. The code waits for three segments before it hands the receiver the playlist, and gives up on a source that has not produced them in 30 seconds.

Converting also keeps the phone's decoder and encoder busy while the film plays, which copying does not, so the app tries the lightest option that works first.

Seeking in a stream that is still being written

A repackaged stream starts where the cast starts, so its clock is behind the title's. The wrapper keeps an offset and translates, so the slider and the position you see are the title's.

A seek inside what is already written goes to the receiver. A seek past it cannot, because the receiver cannot seek into video that does not exist yet. So the phone starts a new remux from there and loads the new playlist, keeping the old stream until the receiver answers, so the receiver never asks for a file that was deleted under it. The same happens when someone scrubs on the receiver's own remote past the written edge.

What each receiver can be told

Chromecast, AirPlay and DLNA all take play, pause, seek and stop, and report a position back. Beyond that they differ, and the phone only sends what a receiver can use.

  • Subtitle files and audio-track choice depend on the receiver: AirPlay takes tracks only inside an HLS stream, and DLNA renderers handle subtitle files each in their own way, so the phone sends none to either.
  • A repackaged video is HLS, so it plays where the TV accepts HLS.
  • An AirPlay screen has to allow anyone on the network, so a Mac shows up in the list once it is set to "Anyone on the same network".

These are properties of the receiver, so the design is to detect them and say so, not to leave a black screen.

What to take away

The thing I would reuse is the shape. Describe a receiver as data, not as a class. Look at the file's bytes, not its name. Pick the cheapest transformation that makes the two agree, and say what the receiver cannot do. The protocol code stays small because the decision lives in one place and the three session classes only carry out the result.

Read more

Top comments (0)