Been a while now since I’ve last opened my Adobe Flex Builder and wrote some AS3…

Working on the  Gendered Advertising Remixer a while ago, made me curious again about whether it will be possible to merge the video & audio channels of given FLVs in run-time on the client-side (in AS3). So I’ve decided to take a few hours this weekend to add a download option to the GAR : )

At Kaltura, we’ve used to do this server side for creating video remixes. The video editors were written in Adobe Flex, targeting Flash 9.0. When-ever a video trimming or merging was needed, the editor would have send a request to the server, then download a whole new video stream. This not only makes poor experience since the user has to wait for videos to download, but it also gets costly as bandwidth consumption grows as well as making it a messy deployment.

A 1.1 version later, Flash 10.1 introduced NetStream.appendBytes() – the ability to push a ByteArray containing FLV bits down the NetStream playback throat. Wonderful! now all I need the server for is to host my videos, all trimming/merging work can now be done on the client computer, all in ActionScript3.

Manipulating bits by the FLV spec

Reading through related posts on the web, I’ve found Thibault Imbert’s FLVSlicer at ByteArray.org. FLVSlicer was a great start – it showed an easy way of slicing through FLV frame-tags and merging slices of the same video.

FLVSlicer was missing 2 pieces important to my desired GAR implementation:

  1. Easily split audio and video channels, and merge audio and video channels to a new video.
  2. Perform all operations on FLVs that are not necessarily the same original FLV (In GAR we merge 2 different videos).

Additionally, I wanted to loose the complex object oriented approach taken in the FLVSlicer. Even though OO might make sense in simplifying complex architectures, I’ve found that in this specific case, it only added unneeded fat to the utility. FLVSlicer seemed to have been built with scale-ability in mind, defining events and separating objects for Slices… maybe for more complex situations like manipulating very long videos, it will be right to go back to this architecture and implement an asynchronous merging/trimming algorithms.

FlvWizard – Utility class for manipulating FLV bits

Re-implementing Thibault’s FLVSlicer, I’ve create FlvWizard. Answering the two needs above and merged into a single class that gets FLV ByteArrays as input and spits FLV ByteArrays as output. No need for more objects, output is ready to be served as is, into the NetStream.appendBytes() function.

Selec All Code:
1
2
3
4
5
var flvwiz:FlvWizard = new FlvWizard ();
videoBytes = flvwiz.extractChannel(boyVideo, FlvWizard.VIDEO_CHANNEL);
soundBytes = flvwiz.extractChannel(girlVideo, FlvWizard.SOUND_CHANNEL);
mergedBytes = flvwiz.mergeChannels(videoBytes, soundBytes);
ns.appendBytes(bytes);

Currently capable of splitting audio and video channels, and then merging channels into a new video. Followup in the next post when I’ll extend it further to enable easy client side trimming and sequencing of given videos.

Try it out with this little demo

Follow the numbers (1. Click to watch original Boys video, 2. Click to watch original Girls video, 3. Click to merge video from Boys and audio from Girls and watch the merged Flv, 4. Click to save the merged Flv to your disk).

Find the source code at GitHub.

What’s next?

Even though this provides a great solution for the GAR project needs, and even for most smaller remixing use-cases, I find this not satisfying. A better way should be hacked to:

  1. Allow manipulation and streaming of h264/webm and not just FLV. The problem with this at the moment is that appendBytes() only handles FLV.
  2. Better performance so that we can handle long video sequences.
  3. Further manipulation of the actual FLV tags – Allowing trimming in between key-frames.
  4. Encoding of effects and overlays on top of the video. (Lee Felarca’s FLV Encoder might be a good start).

Maybe an alchemy port for an h264/webm/flv codec be a good direction? Or a full ffmpeg port?…

  • psyone

    Hy Zohar!

    Sorry for my english. I want to ask you how can I merge, whit your FlvWizard class, two or more different flv file into one.
    Audio and Video also.

    Thanx
    psyone

  • Pic_morakot

    I try vdo => boy.flv and audio => boy.flv when merge vdo and audio, The vdo faster than the audio!!

    Why it’s inconsistent? Or i am doing wrong. ToT

  • Interesting… I haven’t tried merging the same file :*) .
    I’ll take a look at it during the week, thanks for posting!

    Did you extract the audio and video channels before submitting the bytearrays to the merge function?
    Can you post the code you used?

  • Pingback: Hacking in FLV Bits – Part 2 | Articulating ideas()

  • psyone

    Hy Zohar!

    I found some bug in the source. When merge ByteArray’s and write the duration in the header you add this value:
    _merged.writeBytes(writeNumberVariable(FlvWizard.DURATION, (timestampExtended << 8 | time)/1000));
    but this is just the first flv duration

    I thing this is te solution:
    _merged.writeBytes(writeNumberVariable(FlvWizard.DURATION,totalTime/1000));

    thx 
    psyone

  • You are correct! Thanks 🙂
    I’ve fixed it now…

  • Pic Morakot

    From the comments of Mr.psyone.In FlvWizard.as ==> function slice()
    _sliced.writeBytes(writeNumberVariable(FlvWizard.DURATION, (timestampExtended << 8 | time)/1000));
    Should be changed. var totalTime:uint = 0; if(in_point time){ in_point = time; } if((out_point time)){ out_point = time; } totalTime = out_point-in_point; _sliced.writeBytes(writeNumberVariable(FlvWizard.DURATION, (timestampExtended << 8 | totalTime)/1000));For debug about the duration in the header when use slice()
    ^_^

  • Thanks, sounds right. I’ll test it later tonight and update.

  • Kennyl

    Hey Zohar, nice class!  Did you ever figure out why the the Video plays almost double speed when merging the audio and video channels back together?  I’ve had no luck in figuring this out.

  • Zohar Babin

    Hi Kenny,

    Is the above sample behave the same in your computer? 
    If not, can you upload the videos you used?.. 
    What is your Flash version and OS?

  • Kennyl

    Yeah, your boy.flv worked just fine for me.  So, it must be something with the FLV I’m using.  I would send it, but it’s not my content.  When I extract just the Video, it plays fine.  How are your videos compressed?  Mine is 640×360, 15 FPS, Progressive, 22KHz MP3 Mono Audio.

  • Is the Audio and Video encoding specs the same in both files?
    Do you use constant bitrate? (VBR is currently not a good option here).

    To achieve best result, make both videos CBR, use the same gop size (keyframe density) and same Audio bitrate.

  • Kennyl

    I’m actually working with one video only.  I’m separating the one video into video and audio channels and trying to merge them back together. This is where I get the Video running almost double speed and the audio is fine.  I’ll try experimenting with my video settings and use CBR.  Thanks Zohar.

  • Flash tends to do this when audio and video frames are not in sync. (it makes the video run faster to fit the audio rhythm).
    That may very well be the case in this class, if you’re using a VBR video.
    Try with a CBR and let me know if it made a difference.

    Thanks

  • Dave

    Hi Zohar. Thanks for this great class. Do you think it is possible to get the bytes from an MP3 audio channel from an FLV, split from the video using your class, into a Sound object using the new player 11 function Sound.loadCompressedDataFromByteArray() ?

    I tried this with your demo and I got the sound to show a length, but it would not play/decode! Maybe because it needs an MP3 header written to the channel?

  • Hi Dave – most likely you will need construct an MP3 header before sending it to the Sound Object. But I didn’t have much play time with the Sound Object, so can’t be sure…
    Would be a great addition to the library though, let me know if you manage to hack it.

  • Andy Woods

    Hi Zohar, impressive work here!  
    I think the recent updates to flash have mucked things up afraid.  I directly downloaded your code and ran it in flashBuilder. Check out what happens when playing merged: 
    http://www.youtube.com/watch?v=R1reyrjot0Y&feature=youtu.be .  
    My eventual goal is to hack in a way to shift the pitch of the audio (check this out 
    http://krisrok.de/flash/classes/de/kris/sound/SoundPitch.as).  Will keep you updated regarding this (kind of relates to Dave’s above post in that  I’ll have to play with the Sound class as some point).

    Cheers,
    Andy.

  • Pimenov O

     Hi! Its great !
    But how work with netstream.seek(time) ? When this call – video stop (freeze).
    thanks !!

  • Hi Andy, 
    I use the latest version of Flash Player, it seems to work well.
    Which version of Flash are you using?

  • Hi 
    Pimenov – am not sure, I didn’t test seeking as it was out of my scope here…
    But I believe it should work, as the FLV times are being recalculated.

  • Andy Woods

    Hi Zohar,
    sorry for the long delay.  Got distracted by nativeExtensions (ouch).
    I just tried to run it in Flash CS5.5 using Flash 10.2 but got the same result as I did in Flash Builder using Air 4.6.
    I did get an interesting error message I didn’t notice before in FB.  Something about an audioWizard (or like that) class not being found.  Strangely, the code still ran.
    I will continue playing and let you know if I find the bug. 
    Cheers,Andy.

  • satyanarayana

    great job..
    Is it possible merge image bytearray to flv bytearray? or is it possible to make flv bytearray from multiple images? or can we extract frames from flv without playing?

  • @b678d03f172ebad7bb57f3860ae20aae:disqus Check out Lee Felarca’s awesome Alchemy based FLV writer, it is capable of encoding images captured from the webcam to FLV.
    http://www.zeropointnine.com/blog/updated-flv-encoder-alchem/

    Extracting an image from keyframe will probably require something reverse to what Lee’s encoder do. 

  • satyanarayanasv

    Thank you for your reply.
    If I create multiple bitmapdata objects from an animation without audio. can I use FLVEncoder to generate FLV out of these bitmapdata objects? so that I can merge resultant FLV with another FLV.

  • yes. it’s pretty awesome, and also support Audio.

  • satyanarayanasv

    it works..
    With FLVEncoder, I could get flv from multiple bitmaps.
    Thank you

  • satyanarayanasv

    Hi Zohar..
    I have tried with joining two videos with FLVWizard.concatStreams(). 
    With the resultant FLV, I could not find keyframes information in metadata.
    If I update metadata of the same FLV with flvtool2, I can find keyframes with their filepositions. Is there any implementation of updating metadata with keyframes information in as3 like flvtool2 ?

    Thank you

  • @satyanarayanasv:disqus – The current implementation doesn’t add a keyframes array like flvtool. But it should be easy enough to extend the createMetaData function to add such array..

  • satyanarayanasv

    Hi Zohar..
    I have created a blank color(bitmapdata) video using FLVEncoder for example x.flv, if I try to join this video with another flv, for example y.flv, I get the following error
    RangeError: Error #2006: The supplied index is out of bounds. at flash.utils::ByteArray/writeBytes() at com.zoharbabin.bytearray.flv::FlvWizard/extractChannel() at Tests/downloaded2() at flash.events::EventDispatcher/dispatchEventFunction() at flash.events::EventDispatcher/dispatchEvent() at flash.net::URLLoader/onComplete()
    Here x.flv has videocodecid 3, y.flv has videocodecid 2 and x.flv has a less metadata.

    Where could be the problem?

  • satyanarayanasv

    I have tried to join two videos using flvwiz.concatstreams().
    Suppose,
    -> x.flv  is a video of 8 seconds which is generated from a bimapdata  using FLVEncoder 
    -> y.flv is normal video of 10 seconds
    If if join x followed by y (x+y) then they are joined and played well with duration of 18seconds.
    But if I join y followed by x (y+x) then I got duration 18 seconds but only y.flv is being played not x.
    My code

    // WORKS AND DURATION var streams2:Vector. = new Vector.();streams2[0] = x;streams2[1] = y;mergedBytes = flvwiz.concatStreams(streams2);   // DOES NOT WORK, DURATION IS ACCURATE Y VIDEO IS BEING PLAYED BUT NOT X VIDEOvar streams2:Vector. = new Vector.();streams2[0] = y;streams2[1] = x;mergedBytes = flvwiz.concatStreams(streams2);   // Y VIDEO IS BEING PLAYED THEN SKIPPING X VIDEO THEN Y VIDEO IS BEING PLAYED HERE DURATION IS ACCURATEvar streams2:Vector. = new Vector.();streams2[0] = y;streams2[1] = x;streams2[2] = y;mergedBytes = flvwiz.concatStreams(streams2);   
    If both videos are generated using FLVEncoder like x.flv and they are joined and played well in above 3 use cases. But  with one normal video and one one video(  video from bitmap using FLVEncoder) not working.

    Can you please tell me where I have made mistake?

    Thanks 

  • lezlea

    Hi there,

    I’m also trying to join together two bytearrays generated by FLVencoder and am having the same error:
    RangeError: Error #2006: The supplied index is out of bounds. at flash.utils::ByteArray/writeBytes()

    Did you find a solution?

    Thanks,

    Lezlea