Hacker News Reader

Comment by kortex | Hacker News Reader

Question for the peanut gallery, because it's vaguely topical and I've spent hours researching problem to no avail.

How can I mux arbitrary data streams into a container? Let's say I have a drone and I want to stream audio, video, and some avionics telemetry.

I know FFMPEG can pass through data encoded, and it lists a few binary stream formats:

     ffmpeg -codecs | grep '^ ..D'  # shows all -c=d codecs

    ..D... bin_data             binary data
    ..D... dvd_nav_packet       DVD Nav packet
    ..D... epg                  Electronic Program Guide
    ..D... klv                  SMPTE 336M Key-Length-Value (KLV) metadata
    ..D... mpegts               raw MPEG-TS stream
    ..D... otf                  OpenType font
    ..D... scte_35              SCTE 35 Message Queue
    ..D... timed_id3            timed ID3 metadata
    ..D... ttf                  TrueType font

But I have no idea how to pack some new data stream into the filtergraph.

View on HN

With -map. https://trac.ffmpeg.org/wiki/Map

ffmpeg -i input1.mkv -i input2.mkv -i input3.mkv -map 0:v -map 1:a -map 0:d -c copy output.mkv

...will take the video from input1.mkv, the audio from input2.mkv, and the data stream from input3.mkv. If you telemetry isn't already packetized, you'll need to do that yourself. Converting your drone's telemetry into something that ffmpeg likes is generally non-trivial.

At my day job I maintain a tool that takes a .csv file and a raw video file and outputs a single file. Which can then be viewed, with the data displayed on the map and such. https://pro.arcgis.com/en/pro-app/latest/tool-reference/imag...

Many thanks!

> if you telemetry isn't already packetized, you'll need to do that yourself. Converting your drone's telemetry into something that ffmpeg likes is generally non-trivial.

Yes, this is exactly the non-trivial step I'm struggling with. Let's say I have every time step a msgpack-encoded binary blob with a timestamp. I want to packetize it and pack it into a data stream of some timestamped/sequenced packet/page/atom structures. I might be able to build those packets myself manually if I can find a viable format (mkv and ogg seem the easiest to reason about).

And ultimately I do want to do this with real-time streams, but I'll definitely look at this Video Multiplexer to start with! That at least scratches the itch of archiving this data.

Yup, unfortunately (but fortunately for my employer) there's no off the shelf tool that does this. Ultimately you just have to write a bunch of code.

There does exist a standardized way to encode the data: MISB (Motion Imagery Standards Board) ST 0601: https://nsgreg.nga.mil/doc/view?i=5093

How would you get around a live stream with different encoding that FFMGEG couldn't pick up? Like a m83.h file but I think I'm blanking on the name

Is the data packetized and with timestamps?

The data has temporal component, yes. But how would I packetize it in the first place? I have the high-level concept that I need to take each of my data chunks and put them into some kind of timestamped container/page/atom and feed them into the stream, but I have no idea how to actually emit those packets. Like let's say each time step I emit a binary blob of msgpack-encoded data. How do I wrap that with a timestamp/sequence number container?

Something like https://github.com/buma/GPSOverlay might give you some pointers.

Although it's not clear if you want to display your additional data as an overlay (like this example) or keep it as some kind of metadata stream. I'm not sure if the second option is even a thing.

Thanks, but it actually has to be a logical data stream. It's not actually drone GPS, but it's analogous.

It's definitely a thing that exists, I've seen stackoverflow issues talking about working with containers that have these sorts of streams, but for some reason, I've scoured the net and haven't found almost anything about actually writing those streams.

Reply on news.ycombinator.com