Hacking in FLV bits

Been a while now since I’ve last opened my Adobe Flex Builder and wrote some AS3…

Working on the  Gendered Advertising Remixer a while ago, made me curious again about whether it will be possible to merge the video & audio channels of given FLVs in run-time on the client-side (in AS3). So I’ve decided to take a few hours this weekend to add a download option to the GAR : )

At Kaltura, we’ve used to do this server side for creating video remixes. The video editors were written in Adobe Flex, targeting Flash 9.0. When-ever a video trimming or merging was needed, the editor would have send a request to the server, then download a whole new video stream. This not only makes poor experience since the user has to wait for videos to download, but it also gets costly as bandwidth consumption grows as well as making it a messy deployment.

A 1.1 version later, Flash 10.1 introduced NetStream.appendBytes() – the ability to push a ByteArray containing FLV bits down the NetStream playback throat. Wonderful! now all I need the server for is to host my videos, all trimming/merging work can now be done on the client computer, all in ActionScript3.

Manipulating bits by the FLV spec

Reading through related posts on the web, I’ve found Thibault Imbert’s FLVSlicer at ByteArray.org. FLVSlicer was a great start – it showed an easy way of slicing through FLV frame-tags and merging slices of the same video.

FLVSlicer was missing 2 pieces important to my desired GAR implementation:

  1. Easily split audio and video channels, and merge audio and video channels to a new video.
  2. Perform all operations on FLVs that are not necessarily the same original FLV (In GAR we merge 2 different videos).

Additionally, I wanted to loose the complex object oriented approach taken in the FLVSlicer. Even though OO might make sense in simplifying complex architectures, I’ve found that in this specific case, it only added unneeded fat to the utility. FLVSlicer seemed to have been built with scale-ability in mind, defining events and separating objects for Slices… maybe for more complex situations like manipulating very long videos, it will be right to go back to this architecture and implement an asynchronous merging/trimming algorithms.

FlvWizard – Utility class for manipulating FLV bits

Re-implementing Thibault’s FLVSlicer, I’ve create FlvWizard. Answering the two needs above and merged into a single class that gets FLV ByteArrays as input and spits FLV ByteArrays as output. No need for more objects, output is ready to be served as is, into the NetStream.appendBytes() function.

var flvwiz:FlvWizard = new FlvWizard ();
videoBytes = flvwiz.extractChannel(boyVideo, FlvWizard.VIDEO_CHANNEL);
soundBytes = flvwiz.extractChannel(girlVideo, FlvWizard.SOUND_CHANNEL);
mergedBytes = flvwiz.mergeChannels(videoBytes, soundBytes);

Currently capable of splitting audio and video channels, and then merging channels into a new video. Followup in the next post when I’ll extend it further to enable easy client side trimming and sequencing of given videos.

Try it out with this little demo

Follow the numbers (1. Click to watch original Boys video, 2. Click to watch original Girls video, 3. Click to merge video from Boys and audio from Girls and watch the merged Flv, 4. Click to save the merged Flv to your disk).

Find the source code at GitHub.

What’s next?

Even though this provides a great solution for the GAR project needs, and even for most smaller remixing use-cases, I find this not satisfying. A better way should be hacked to:

  1. Allow manipulation and streaming of h264/webm and not just FLV. The problem with this at the moment is that appendBytes() only handles FLV.
  2. Better performance so that we can handle long video sequences.
  3. Further manipulation of the actual FLV tags – Allowing trimming in between key-frames.
  4. Encoding of effects and overlays on top of the video. (Lee Felarca’s FLV Encoder might be a good start).

Maybe an alchemy port for an h264/webm/flv codec be a good direction? Or a full ffmpeg port?…