Video Coding Format with 13 GB per Hour



  • When I bought my Western Digital MyBook 500 GB external hdd, that the size of pictures that was estimated for illustrative purposes actually matched the size of pictures of my brand new 14.2 Megapixel DSLR Camera. At the maximum resolution of 4592x3056 the average size of a pitcure is about 3.5 MB.

    Although I know that the size of 1 GB may vary, a video using 13 GB per hour is pretty impressive.
    An average Hollywood movie would need 5 single sided single layered 4.7 GB DVDs, a Bollywood movie even 11 DVDs without any extras.

       external hdd package



  • I'm too lazy to do the math, but presumably they're using some large, raw video for the calculation. They might as well use a low figure anyway, so people are pleased when they can surpass it.



  • Raw DV is exactly 13GB an hour.  They are not talking about ripping DVDs here, obviously. 



  •  You have a new Pentax K20D?



  • Nope, Sony Alpha 350. 



  • @MyKey_ said:

    the size of pictures that was estimated for illustrative purposes actually matched the size of pictures of my brand new 14.2 Megapixel DSLR Camera. At the maximum resolution of 4592x3056 the average size of a pitcure is about 3.5 MB.

    That's only true if you shoot in JPG mode. Many people using DSLRs (and even some better point & shoot cameras) shoot RAW or RAW + JPG. On my Canon Rebel, taking a single picture produces a ~10-14 Meg RAW and a 3-4 Meg JPG photo. And of course, if you're editing video it takes up a lot more space than a video produced solely for watching on a DVD.



  • As someone has already said, this is not a WTF.

    If you shoot digital video with a miniDV camcorder, and import that video via FireWire as a DV stream (e.g. using Windows Movie Maker, Premiere, iMovie, Final Cut, or any of the major video editing programs) you will find that a one-hour tape digitizes to a 13 gig DV video file.

    Since external hard drives are frequently used by video editors to archive footage from DV tapes, I think this is perfectly reasonable.

    Sorry to disappoint.

     



  • You forgot to consider Full-HD video content which on average works out at 10-15GB per hour:

    http://www.dbstalk.com/archive/index.php/t-53615.html

     To make sure they don't get sued hard drive manufacterers will take the largest size per pic/hour etc know to be available when demonstrating capacity giving the worst case scenario. For most people they will be able to store far more than specified



  • @Hitsuji said:

    You forgot to consider Full-HD video content which on average works out at 10-15GB per hour:

    http://www.dbstalk.com/archive/index.php/t-53615.html

     To make sure they don't get sued hard drive manufacterers will take the largest size per pic/hour etc know to be available when demonstrating capacity giving the worst case scenario. For most people they will be able to store far more than specified

    Do you people even bother reading the comments before replying?  It is talking about Raw DV.  Fuck. 



  • @morbiuswilters said:

    Do you people even bother reading the comments before replying?  It is talking about Raw DV.  Fuck. 
     Sorry Morbs, I didn't see your earlier post as I'm using a a  WTF-anti-troll addon for Firefox. I don't see post's belonging to any trolls unless I'm quoted by one.



  • @Hitsuji said:

    Sorry Morbs, I didn't see your earlier post as I'm using a a  WTF-anti-troll addon for Firefox. I don't see post's belonging to any trolls unless I'm quoted by one.

    Pwn3d by your own ignorance!  Delicious, I love it! 



  • @Hitsuji said:

    Sorry Morbs, I didn't see your earlier post as I'm using a a  WTF-anti-troll addon for Firefox. I don't see post's belonging to any trolls unless I'm quoted by one.
    Where can I get a copy?



  • Some of the uncompressed video we deal in at the studio I work for hits about 40GB per minute for you average full HD feed, it depends on if we're dealing with 8bit colour or 10bit.  More if you're dealing with 2K and 4K film resolutions.  RedCam footage can fit 5 minutes onto an 8GB memory card, but R3D formats are compressed slightly.

    Since these drives are typically purchased by video professionals to move large chunks of data around (we mail them, express them, even cab them to other studios) it makes a lot of sense to assume that they might be targetting those people with this statement.  Still, most video professionals know exactly how much space their data will consume so giving them "real world figures" is kind of pointless.



  • @MyKey_ said:

    When I bought my Western Digital MyBook 500 GB external hdd, that the size of pictures that was estimated for illustrative purposes actually matched the size of pictures of my brand new 14.2 Megapixel DSLR Camera. At the maximum resolution of 4592x3056 the average size of a pitcure is about 3.5 MB.

    The figures for a raw pic at that resolution would be 42099456 bytes (4592*3056*3 assuming the 3-byte RGB format) = 41112.75 Kb = 40.14 Mb per picture.

    32-bit color: 56132608 bytes = 54817 Kb = 53.53 Mb per picture.

    As it has been already mentioned, 13Gb/hour would be for RAW video.



  • @merreborn said:

    Where can I get a copy?

    There's this awesome invention called Google.  It will lead you to http://www.userscripts.org where you can search for this site.  Now here is the important part: you need to download the script and install it?  Got that?  Don't email it to your mom, don't print it out and try to eat it and for the love of God don't roll around naked on an LCD displaying the script source.



  •  @morbiuswilters said:

    @merreborn said:

    Where can I get a copy?

    *Random Bitching*

     Or in plain nice english: Install greasemonkey firefox add-on. and download one of the sanitiser scripts here: http://userscripts.org/scripts/search?q=tdwtf

    there were more but a few months ago but alas only two survive today.



  • @Hitsuji said:

    there were more but a few months ago but alas only two survive today.
     

    Maybe that is because it is an insanely stupid idea which you have proved will only degrade the quality of our forums.



  • Hi, I'm a video geek in my spare time, so here a couple of cents...

    13GB per Hour at a complete guess sounds like probably standard definition DV format (i.e. tape camcorder standard).

    At least, that matches my own experience for file sizing in that format :)

    This is actually still at roughly 5:1compression (lossless though), so you can image how big raw footage gets.



  • @bikegrinch said:

    Hi, I'm a video geek in my spare time, so here a couple of cents...

    13GB per Hour at a complete guess sounds like probably standard definition DV format (i.e. tape camcorder standard).

    At least, that matches my own experience for file sizing in that format :)

    This is actually still at roughly 5:1compression (lossless though), so you can image how big raw footage gets.

     

    Well I'm a video professional and I can tell you that DV is not lossless!  I am not aware of any mainstream lossless video compression, and infact having worked on developing video compression technologies I would not expect there to be anything that can be lossless with a reliable ratio better than ~ 1.2:1.



  • @GettinSadda said:

    I am not aware of any mainstream lossless video compression

    gzip?



  • @GettinSadda said:

    Well I'm a video professional and I can tell you that DV is not lossless!  I am not aware of any mainstream lossless video compression, and infact having worked on developing video compression technologies I would not expect there to be anything that can be lossless with a reliable ratio better than ~ 1.2:1.
    Depends on what you mean by mainstream. If "used by studios", then you're probably right. If you're looking at home users, Huffyuv and Lagarith are in common use.



  • @ender said:

    @GettinSadda said:
    Well I'm a video professional and I can tell you that DV is not lossless!  I am not aware of any mainstream lossless video compression, and infact having worked on developing video compression technologies I would not expect there to be anything that can be lossless with a reliable ratio better than ~ 1.2:1.
    Depends on what you mean by mainstream. If "used by studios", then you're probably right. If you're looking at home users, Huffyuv and Lagarith are in common use.
     

    Yeah - I tend to find that the sort of people I work with are either happy to work compressed (DV in 25, 50 or 100 variants) or want lossless and so work uncompressed.

    Home video and low-end pro (think wedding videos) the like are quite a different kettle of fish andyou do get some odd stuff used there.

    Also, I would just love to feed Huffuv and Lagarith withsome of the high-entropy test material that I have used at times, but I don't have time to play with this and a quick look at the Lagarith site shows that a) He has proved that there are some sequences where his codec increases the bandwidth and b) He bases most of his comparisons on DVD video which is already heavily compressed!



  • Huffy/Lagarith achieve about 2.3-2.5x compression--on completely uncompressed raw video.  Try it on YUV sequences, such as parkrun, crowdrun, etc.  Running lossless on already-compressed video doesn't even necessarily get better compression than on the original source; in some cases it actually gets worse depending on the kind of artifacts the encoder produced.  I did a test the other day where this was exactly the case and the PNG files for the lossy sources were larger than that for the lossless original.

    FFV1 does even better; around 2.8x.  You can get a bit better if you go for inter compression, but right now I don't know of any good format that allows it.  H.264 lossless sucks for many reasons and is generally considerably worse than FFV1 except in the case of an extremely clean source where inter prediction makes up for the lack of coding efficiency.

    @GettinSadda said:

    Well I'm a video professional ... I am not aware of any mainstream lossless video compression, and infact having worked on developing video compression technologies I would not expect there to be anything that can be lossless with a reliable ratio better than ~ 1.2:1.
    If you are actually serious about this post, I feel sorry for the person who was dumb enough to hire you as a "video professional."



  • @Dark Shikari said:

    @GettinSadda said:
    Well I'm a video professional ... I am not aware of any mainstream lossless video compression, and infact having worked on developing video compression technologies I would not expect there to be anything that can be lossless with a reliable ratio better than ~ 1.2:1.
    If you are actually serious about this post, I feel sorry for the person who was dumb enough to hire you as a "video professional."
     

    Where I work we would not regard these as "high entropy" sources - PM me if you want more details. You and I are talking about very differnt things!



  • @GettinSadda said:

    @Dark Shikari said:

    @GettinSadda said:
    Well I'm a video professional ... I am not aware of any mainstream lossless video compression, and infact having worked on developing video compression technologies I would not expect there to be anything that can be lossless with a reliable ratio better than ~ 1.2:1.
    If you are actually serious about this post, I feel sorry for the person who was dumb enough to hire you as a "video professional."
     

    Where I work we would not regard these as "high entropy" sources - PM me if you want more details. You and I are talking about very differnt things!

    Grainy film camera footage isn't high-entropy enough for you?  Then what are you talking about, /dev/random ?



  • @Dark Shikari said:

    Grainy film camera footage isn't high-entropy enough for you?  Then what are you talking about, /dev/random ?


    When you are designing compression algorithms for professional content creation use, what I was doing a few years back, you find that there are all sorts of situations where you get huge entropy in a sequence.

    One of my faves was always the red carpet shots outside the Oscars - lots of fast panning to get the next celeb while 100 un-correlated flash-guns illuminate a heaving crowd.

    You could also get some great examples from poor quality FM satellite contribution links (think ENG a few years back) and even low-delay digital with dodgy reception.

    But specially manufactured shots were even worse… and those are what we designed to for worst case.



  •  @GettinSadda said:

    When you are designing compression algorithms for professional content creation use
    That isn't much of a qualification; such encoders almost always range from "bad" to "output suggests the encoder was designed by a bunch of monkeys on typewriters."

     Unsurprisingly, the encoder I develop for completely trashes every commercial solution I've put it against.  I'm not sure whether this speaks for our effectiveness or that everyone else just sucks.  Though my own research suggests its because everyone seems to think that PSNR == quality, which is a recipe for visual disaster.

    Back on topic, I would be shocked if you can find a real photograph (i.e. not a contrived /dev/random example) that cannot be compressed losslessly by more than 20% by FFV1 or a similar context-based arithcoding compressor.



  • @Dark Shikari said:

     @GettinSadda said:

    When you are designing compression algorithms for professional content creation use
    That isn't much of a qualification; such encoders almost always range from "bad" to "output suggests the encoder was designed by a bunch of monkeys on typewriters."

     Unsurprisingly, the encoder I develop for completely trashes every commercial solution I've put it against.  I'm not sure whether this speaks for our effectiveness or that everyone else just sucks.  Though my own research suggests its because everyone seems to think that PSNR == quality, which is a recipe for visual disaster.

    Back on topic, I would be shocked if you can find a real photograph (i.e. not a contrived /dev/random example) that cannot be compressed losslessly by more than 20% by FFV1 or a similar context-based arithcoding compressor.

     

    Ah, now I see the dis-joint.

    You are talking about writing code to implement compression systems that others have designed.

    I am talking from the perspective of one of those engineers that designs the compression systems themselves - and we like to thrash the living daylights out of the system to see how it would perform in "worse-than-the-worst-case-you-can-imagine" situations. This is where you get images that are close to /dev/random because shipping a fix for an encoder application that can't cope with some strange new sequence is easy compared to shipping a whole new standard that obsoletes all existing implementations!



  • @GettinSadda said:

    Ah, now I see the dis-joint.

    You are talking about writing code to implement compression systems that others have designed.

    I am talking from the perspective of one of those engineers that designs the compression systems themselves - and we like to thrash the living daylights out of the system to see how it would perform in "worse-than-the-worst-case-you-can-imagine" situations. This is where you get images that are close to /dev/random because shipping a fix for an encoder application that can't cope with some strange new sequence is easy compared to shipping a whole new standard that obsoletes all existing implementations!

    So you work for the JVT?  In that case, I have a very long list of stupid decisions made in the H.264 standard that I would like rectified ;)

    (and if you don't work for the JVT, then you're irrelevent, video standards-wise.  Nobody uses VP7, RV30/40, and even VC-1 is not very popular.  I'm also pretty sure the JVT doesn't "worst-case" test anything; from what I can tell all they do is throw Foreman.cif-like test sequences at things over and over and over until something works.)



  • @Dark Shikari said:

    So you work for the JVT?  In that case, I have a very long list of stupid decisions made in the H.264 standard that I would like rectified ;)

    (and if you don't work for the JVT, then you're irrelevent, video standards-wise.  Nobody uses VP7, RV30/40, and even VC-1 is not very popular.  I'm also pretty sure the JVT doesn't "worst-case" test anything; from what I can tell all they do is throw Foreman.cif-like test sequences at things over and over and over until something works.)

     

    Not JVT, but you are getting close with something else - can't say which if I want to keep my job!



  • @GettinSadda said:

    @Dark Shikari said:

    So you work for the JVT?  In that case, I have a very long list of stupid decisions made in the H.264 standard that I would like rectified ;)

    (and if you don't work for the JVT, then you're irrelevent, video standards-wise.  Nobody uses VP7, RV30/40, and even VC-1 is not very popular.  I'm also pretty sure the JVT doesn't "worst-case" test anything; from what I can tell all they do is throw Foreman.cif-like test sequences at things over and over and over until something works.)

     

    Not JVT, but you are getting close with something else - can't say which if I want to keep my job!

    An NDA that prevents you from saying where you work?  FFS, this is video encoding, not an intelligence agency.


  • @Dark Shikari said:

    An NDA that prevents you from saying where you work?  FFS, this is video encoding, not an intelligence agency.
     

    Funnily enough I have both an NDA stating that I must not mention any of the projects I work on without prior agreement, and a clause in my contract saying that I must not discuss my work in public forums.

    It gets quite interesting when you play with certain litigious Big-Boys!

    Edit: They are fairly OK with me discussing the general field as long as I do not bring them into it by saying who I work for (and best to keep my real name out of it)


  • @Dark Shikari said:

    An NDA that prevents you from saying where you work?  FFS, this is video encoding, not an intelligence agency.
    Well, it's not an uncommon requirement that you not reveal your employer. A little extreme, perhaps, but not crazy. Why, my company goes so far as to remove the all the letters in their name from the keyboard. Of course, now that I've told you this, you can just work backwards and figure it out...



  • @bstorer said:

    @Dark Shikari said:
    An NDA that prevents you from saying where you work?  FFS, this is video encoding, not an intelligence agency.
    Well, it's not an uncommon requirement that you not reveal your employer. A little extreme, perhaps, but not crazy. Why, my company goes so far as to remove the all the letters in their name from the keyboard. Of course, now that I've told you this, you can just work backwards and figure it out...
     

    You work at QZ%`|`  too???



  • @GettinSadda said:

    @Dark Shikari said:

    An NDA that prevents you from saying where you work?  FFS, this is video encoding, not an intelligence agency.
     

    Funnily enough I have both an NDA stating that I must not mention any of the projects I work on without prior agreement, and a clause in my contract saying that I must not discuss my work in public forums.

    It gets quite interesting when you play with certain litigious Big-Boys!

    Edit: They are fairly OK with me discussing the general field as long as I do not bring them into it by saying who I work for (and best to keep my real name out of it)

    I know people who work at or have worked at Ateme, DivX, Mainconcept, Harmonic, and Digital Fountain, and none had anything even close to that--such an NDA is absurd.

    Of course, this is one of the joys of working on open source encoders professionally--my employer is quite limited in how strictly they can NDA me ;)



  • @morbiuswilters said:

    growl growl growl grumble

     

     

    I think it's that time of the month again.



  • @Hitsuji said:

    there were more but a few months ago but alas only two survive today.
     

    i've taken mine off since the others were more advanced...

     



  • @PerdidoPunk said:

    I think it's that time of the month again.

    Time for you to post worthless flamebait?  Yep! 



  • @bikegrinch said:

    Hi, I'm a video geek in my spare time, so here a couple of cents...

    13GB per Hour at a complete guess sounds like probably standard definition DV format (i.e. tape camcorder standard).

    At least, that matches my own experience for file sizing in that format :)

    This is actually still at roughly 5:1compression (lossless though), so you can image how big raw footage gets.

     

    If only DV compression was lossless.  Its 4:2:2 compression, not 4:4:4 so it holds the greens and blues well, but kills reds and other colours that fall into the brown ranges.  From my experiences, if you import your footage from a DV tape, edit it and then export it to either a DV tape or just a file using the DV codec it does get recompressed.

    You're right about the the DV data rates, I was just giving you a heads up incase you were relying on the DV format for your edits.



  • @Soviut said:

    @bikegrinch said:

    Hi, I'm a video geek in my spare time, so here a couple of cents...

    13GB per Hour at a complete guess sounds like probably standard definition DV format (i.e. tape camcorder standard).

    At least, that matches my own experience for file sizing in that format :)

    This is actually still at roughly 5:1compression (lossless though), so you can image how big raw footage gets.

     

    If only DV compression was lossless.  Its 4:2:2 compression, not 4:4:4 so it holds the greens and blues well, but kills reds and other colours that fall into the brown ranges.  From my experiences, if you import your footage from a DV tape, edit it and then export it to either a DV tape or just a file using the DV codec it does get recompressed.

    You're right about the the DV data rates, I was just giving you a heads up incase you were relying on the DV format for your edits.

     

    Um, no - in more ways than I can count!

    I can't be bothered to start this all again, but you can see for yourself.

    Start here:

    [url]http://en.wikipedia.org/wiki/Dv[/url]

    Especially here:

    [url]http://en.wikipedia.org/wiki/Dv#Chroma_subsampling[/url]

    and here:

    [url]http://en.wikipedia.org/wiki/Y%27CbCr[/url]



  • @Soviut said:

    If only DV compression was lossless.  Its 4:2:2 compression, not 4:4:4 so it holds the greens and blues well, but kills reds and other colours that fall into the brown ranges.
    I lol'd hard at this line.



  • @Soviut said:

    From my experiences, if you import your footage from a DV tape, edit it and then export it to either a DV tape or just a file using the DV codec it does get recompressed.

    That will happen only if you use shitty editing software that doesn't copy unchanged data through. Or if you applied any image "enhancement" to the footage, which you should know better not to do.

    With DV, first field in the frame gets compressed by itself (I-frame), and the second field compressed as P-frame with motion compensation. Thus, if you edit on frame basis, you never have to recompress unchanged frames. Again, if your editing software is not written by complete idiots.

     



  • Well I have been using DV, as it seemed the best choice of what I had available (hey, only a beginner really - lots to learn).

     If you'll indulge, I've just started playing with huffyuv and got a query about it's compression ratios.

    To give myself some starting data, I used Super to convert a 35min DV file (7.4GB) to huffyuv, which churned out at 30.6GB file. Not unexpected. What was unexpected, was looking at the disc space actually used on my compressed drive - it showed only 1.1GB!!! I then simply zipped it and it shrank to 267MB! Is this normal? Or did something break?
     
    Now gotta see if Vegas likes it :)


  • scratch that - just confirmed the output was completely broken. 35 mins of black SHOULD compress well lol (my bad)


Log in to reply