Thumb.co.il is the best web-based video format inspector tool ever created. This is an overview of the reasons for creating thumbcoil and a peek into some of the cool technology contained therein.
2. Who am I?
Nom de Plume:
Email:
HLS:
Transmuxer:
Thumbcoil Lib:
Thumb.co.il UI:
Jon-Carlos Rivera
jrivera@brightcove.com
https://github.com/videojs/videojs-contrib-hls
https://github.com/videojs/mux.js
https://github.com/videojs/thumbcoil
https://github.com/videojs/thumb.co.il
3. What is Thumbcoil?
Thumbcoil is a web-based video inspector for:
- MPEG2-TS (also M2TS or just TS)
- fMP4 (fragmented MP4)
- FLV
4. "If you gaze long into a file format, the file format also gazes into you."
-- Nietzsche, probably
5. In the beginning, there was no Thumbcoil
Thumbcoil eventual grew out of our home-grown web-based video format
transmuxer…
Mux.js!
https://github.com/videojs/mux.js
(Did I mention it's open source?)
6. What is Transmuxing a Container?
Container
Synchronization Synchronization
Bitstream
Info
Audio
Bitstream
Audio
Bitstream
Bitstream
Info
Video
Bitstream
Video
Bitstream
7. What is Transmuxing a Container?
M2TS
PES PES
PMT Audio
Bitstream
Audio
Bitstream
PMT Video
Bitstream
Video
Bitstream
MP4
Sample Sample
TRAK Audio
Bitstream
Audio
Bitstream
TRAK Video
Bitstream
Video
Bitstream
11. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
12. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient
13. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient
Browser MSE implementations with MP4 were NOT
14. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient
Browser MSE implementations with MP4 were NOT
MSE was very strict about EVERYTHING
15. Life was great until...
Pretty quickly we ran into issues with fMP4 & MSE
Flash playback of FLV was very resilient
Browser MSE implementations with MP4 were NOT
MSE was very strict about EVERYTHING
(But it has improved somewhat)
18. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash
19. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash
...And Safari
20. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash
...And Safari
...And Quicktime
21. End result?
Lots of perfectly valid HLS just didn't work!
...But worked in Flash
...And Safari
...And Quicktime
...And VLC
22. The debugging nightmare
How do you inspect a video to do some sanity checking?
Run ffprobe!
- Seems useful at first
- The output is ofttimes just a pack of lies
- ...because ffprobe hates you
23. How does ffprobe hate?
Problem: Detecting M2TS segments that did not begin with an i-frame
Solution: ffprobe --show_frames ?
Now you have two problems!
26. Our struggle...
Solution: Use mux.js to inspect videos!
It was a GREAT IDEA...
...so we didn't do that.
Instead, we kept struggling with ffprobe.
27. Solve all of the easy problems...
...and all that remains are the hard problems.
(ffprobe doesn't help you here!)
We needed to go deeper...
...inside the bitstreams.
28. I'm going to level with you
I was mostly just curious
29. Thumbcoil is born
● I (re)wrote a parser for H.264 bitstreams from scratch
● None of the bitstream parsing code is shared with mux.js
● Implemented using PARSER COMBINATORS...
...for science!
http://beta.thumb.co.il
(Did I mention it's open source?)
32. Parser Combinators
You build a parser by composing high-order functions
All of the combinators in Thumbcoil are designed as "codecs"
- They have a decode function that takes a bitstream and returns an object
representation of the data
- They have an encode function that can take the object representation
and return a bitstream
This is mostly a fun trick that has never been used for anything cool. (Yet!)
37. What is parsing AAC like?
Imagine trying to find a word in a dictionary where pages appear blank unless
you've completely and accurately read the previous page.
It's like that but worse… because the words are also huffman coded.
Using one of any 15 codebooks... per letter.
And each codeword can code for more than one letter...
...and there are no spaces.
...and most of the words are homonyms.
39. The future...
● Improved GOP view
○ More sorting and display options
● Hex-view
● ID3 Parsing
● Bring back MP4 and FLV parsing
● Support MPEG-DASH manifests
● <Your Idea Here> (Did I mention it's open source?)
Talk about FLV being the easiest format in the world to work with.
Why do transmuxing outside of Flash in the first place? Captions and ID3.
STOP
STOP
STOP
Two reasons: To make sure we were parsing correctly in mux.js To craft the parsers such that they looked similar to the specification to reduce the chance of errors
