Digital Fear-Mongering And Hollywood Accounting
Erick Schonfeld
34 comments »
As movies go digital, they are presenting new archival challenges for the film industry. Film cannisters stored in a salt mine last a lot longer than a hard disk drive or a DVD. The issues of preserving film footage for the ages in a digital form are real and vexing, but a recent report by the Academy of Motion Picture Arts and Sciences titled “The Digital Dilemma” throw out some alarmist numbers that are hard to swallow. According to a story about the assessment in the NYTimes:
To store a digital master record of a movie costs about $12,514 a year, versus the $1,059 it costs to keep a conventional film master.
Much worse, to keep the enormous swarm of data produced when a picture is “born digital” — that is, produced using all-electronic processes, rather than relying wholly or partially on film — pushes the cost of preservation to $208,569 a year, vastly higher than the $486 it costs to toss the equivalent camera negatives, audio recordings, on-set photographs and annotated scripts of an all-film production into the cold-storage vault.
The big numbers help to sex up the “dilemma,” but they appear to be yet another case of Hollywood accounting. The rule of thumb for storing a movie in a high-quality digital archive seems to be about one terabyte per hour. So two terabytes should be enough for most digital masters. Consider that a one-terabyte consumer drive at Amazon costs $250 to $300. The cost per terabyte is much less in an enterprise-class storage network. Of course, that is just the hardware. There are additional costs of facilities, labor, maintenance and licensing costs, but these can be spread across an entire library of movies.
Yes, it costs more to store 10,000 movies than 1,000 movies, but not ten times more. At a certain point, each new movie stored becomes an incremental cost. In other words, if it costs $12,514,000 to set up and run a digital archive facility to store 1,000 movies, it does not cost $125 million to store 10,000 movies.
Now, that is just for the final master copy. The movie studios want to store every outtake and bit of footage captured during the film-making process, and that is where the $208,569 a year comes from. For a single movie, that must amount to many, many terabytes of footage. But again, it is difficult for me to suspend my disbelief here. Storage costs are not constant per terabyte. They scale the more you have to store. The true cost of a well-managed storage program (one where everything is backed up redundantly in more than one location, and where the actual storage technologies are constantly upgraded and swapped out to higher-capacity technologies over time) would surely be much less. (Any storage experts out there willing to venture an educated guess in comments)?
Even if it does turn out to cost $208,569 a year, there is another solution: hire one or two people per film to go through all the extra footage and cull it down to a more manageable amount. Remember, we are just talking about the footage to be stored in a high-quality archival format. You can still store everything else in lower-quality, compressed MPEG-2s or MPEG-4s. The big numbers, though, make it easier to scare up funding for digital archive projects—or to rationalize a slower transition to digital altogether.
(Photo via Catherine Trigg).
Hi
To every problem there is a solution. “Flac” for Audio-Files is a good example of that. Pay 3 persons which really know about realtime - converting without loss of information but with compression and let them get a good solution to that.
Best, jens
I’ve often wondered about how long media can be preserved on devices too like how long can I save photos on a compact flash before it all disappears??
Even at Amazon S3 prices, 1TB of storage only costs $1800 per year. If these rates are close, can they really be storing 115TB of data PER MOVIE?
If they were more open about everything. They could use the idea of putting all the content for a movie on line for public download, or they could have Google archive it for all time. I know that will never happen but in exchange for less control they would actually save bottom line pricing. I also think a workable solution can be hammered out I just know I can not come up with a bullet proof way to satisfy studio exec in a blog comment
It was 1TB per hour in 2003. Now it’s more.
What about copying. And distributing copies. Isn’t that the purpose for having originals? I would imagine over the next decade the cost of archiving, copying and distributing digital bits will fall by several orders of magnitude. The whole archive-copy-distribute chain might cost less today even.
Cost accounting and Hollywood do not mix - this is a non-story. It is so rediculous.
Yes, they are storing 115TB of data per movie, probably more. Assuming they are trying to keep the highest quality master (which they are), they are storing their movies at, at least, 4k res uncompressed. That’s 875.25MB per second (only 8 bit). That’s 3.1TB per hour. Most movies / TV usually shoot somewhere between 10:1 and 100:1, which means that a 90 minute movie has at least 900 minutes of shot footage, and as much as 9,000.
Go up to 10bit (a more likely scenario) it’s 1094MB per/sec, or almost 4TB per hour.
And I’ve personally worked on projects that have shot over 1,000 hours.
I doubt they would store any of this footage compressed, it’s common practice to keep your final project as high as quality as possible. You never know when new technology is going to come out that has much higher-res and you’ll want to re-release (with DVDs the studios went back to their vaults and rescanned a lot of their films to get the higher resolution that DVD offered over VHS).
All of this is assumes 4k, which isn’t even film quality. Once you get up to 8k these numbers get crazy. Somewhere around 4TB per second. And in case you’re wondering, there have been films mastered at 8k already.
So while I can’t vouch for their storage costs, it doesn’t seem unreasonable to me to imagine a 12k per year cost to buy and maintain that much space. If not now, soon.
They make enough, let them burn their money.
Only big corporations are willing to spend that much money on something so ridiculous. They just buy enterprise grade hardware which costs in the $10,000+ range because their tech people aren’t bright enough to code redundancy and failover safety on a software level.
No point in trying to teach them, it’s their money to burn.
These costs are accurate. This post and most of the comments except tom(8) do not understand the industry. There is a LOT of archive footage, effects, etc that go into every project, not just what you see in the “final cut”.
So why not just convert some of these movies back to film for storage? As #8 points out, film is higher resolution then most digital masters so the only quality loss would be during the transfer or through degridation over time. But if you’re bitching about the fact a movie is costing you 12 - 200 times as much to store then it’s probably barely making enough to cover it’s keep.
The other main important bit they seem to be including which wasn’t as much of an option in the film era is attempting to store all the editorial files in their proprietary formats and make sense of them down the road. Sure, you can store your raw footage and end result but there’s also the desire to keep the metadata that gets you from one to the other. Avid/Digi are as wonderful as Microsoft when it comes to controlling the file format and they may be including the cost of having to maintain historical infrastructure that can make sense of all that if needed.
You have a similar issue with simple backups. What version of what app on what platform on what media was the backup made.
While books certainly decay overtime, at least their data format hasn’t historically changed very rapidly, translation tools are generally available and you don’t need to buy a proprietary set of glasses from some recalcitrant company to read them. It would be great to get to a similar point with app development and data types but until then we’re really building ourselves a bit of an information black hole that isn’t really necessary or desirable in the long term.
The question is whether the storage of the “archival footage” is anywhere near as important as that of the final product. Sure, it would be instructive to future film historians to have a copy of every last bit of film shot in the making of “Citizen Kane”, but what are we trying to retain here — a panopticon view of the creative process or the finished product that is what people actually saw in the theaters?
I leave it for others to decide who controls things like George Lucas and Steven Spielberg re-writing their previous work and then saying that the old version should be thrown down the memory hole, or Ridley Scott requiring that all forty-seven different versions of “Blade Runner” be retained at the highest possible fidelity.
“Consider that a one-terabyte consumer drive at Amazon costs $250 to $300. The cost per terabyte is much less in an enterprise-class storage network.”
Actually the exact opposite is true, by a very very large margin. A price of $5 - 15/GB is “normal”. A disk-based archival system to store a hundred TB would run in the very-high six figures just for the upfront capex alone. A secondary unit, a storage network, the software and servers to manage it, and the staff to support it all double that easy. And these are figures based off startup/discount providers. Something from IBM/EMC/HP would easily be twice that again.
Here’s a ton of price details from storage vendors, but its almost impossible to generate your own quote from them unless you already have a vendor quote to compare line items:
http://storagemojo.com/storagemojos-pricing-guide/
Erick, before you go spouting cost-basis figures, realize that consumer and enterprise pricing are completely different. 100$ for 250gb of disk data is standard for consumers. For enterprise-class raid drives it’s several times that.
Add into the mix that the average films I’ve worked onw ere producing nearly a petabyte of data PRE-HD and you can imagine that storage PER MOVIE can run to 5-10pb. And at an upfront cost of 75-100k for that much storage…
While I agree with your premise tat at some point the cost per movie goes down, I’d buy hollywood on a 100k/film outlay as a bare minimum - and hiher than that until they have their own datacenters up and running.
To put this in perspective, one major hollywood studio will produce more data in a year than all of web 2.0 combined.
I think GC above makes a good point. The cost of storage is really not the issue. It is the cost of being able to do something with the files 5, 10 or 50 years in the future.
Film that was stored 50 years ago can still be used today without any equipment at all. You can just look at each frame with your own eye. It could then be scanned in or whatever.
Just try and utilize data that was stored on a 5″ floppy disk from 15 years ago.
@jdblank and GC, that’s a great issue you’ve brought up, one I think most people don’t think much about even with their personal files. My dad (who recently passed) has a whole closet full of slides and some weird 8mm film reels. I have no machine in which to view these. They will become more meaningful to me as time marches by. But still, viewing them? ugh.
This is an interesting part we’re also dealing with at my company. It’s not as much about you uploading your personal jpgs and wmv’s or word docs, but rather determining what file types we *should* support because we’re going to have to use transfer technology as the technology/ formats udate. I doubt mp3 and mpg will be the standard audio and video file formats in 25 years, let alone 50, 100 years. Even if I put everything into xdrive, amazonS3, and even if someone continued to pay the fees after I’m long gone (a big if!!) - will anyone even be able to view the files? Anyway, this is what we’re playing with at work. Makes my job fun at least!
If the storage costs are so high for digital, why not make the companies that are expert in digital storage part of the movie production value chain.
To me, it sounds as if the movie production companies are trying to do all by themselves in this regard.
Looking it this way, adding digital storage companies to the value chain would mean a deal in which the storage costs are tied to the box office and aftermarket revenues, so that the digital storage companies have share of both the risk and reward.
Merry Christmas everyone.
fakesteveballmer.blogspot.com
Hi Antje,
Sorry to hear about your loss; I’d like to help you convert that for you. We can convert those slides and 8mm film for you at a great discount price as Jan is our slow month.
Happy Holidays,
Brad
Bljashinsky at ReimagineMemories.com
Perhaps if the Motion Picture industry could figure out a way to contain their escalating movie production costs, then this would be an interesting side note.
When a simple comedy like “The Heartbreak Kid” from DreamWorks/Paramount reportedly costing over $60 million to produce and distribute, it would appear that they have bigger expense problems to solve ahead of digital storage.
hi Brad, thank you! i’d probably have to consider you a competitor though, OR a potential partner in the future? Let me know …
Happy Holidays everyone!!!!!!!!
antje
That’s nice idea to capture the footage during the film-making process, it will help for marketing purpose and as well for releasing another part of movie.
Two things sound crazy to me:
First, he cost of enterprise TBs is going down all the time, down an amazing spiral fall and we don’t know when will it end. True, at the moment an enterprise TB can easily cost 5$-15$ a gig, not including servers and staff, but these prices were ten times that less than ten years ago.
Second, I’m fron the SAN neighbourhood, not from the movie neighbourhood, but I can’t understand why a movie should be 120TB. Must we keep it completely completely uncompressed? Why? And how high will we go with HD movies and the like? There are only that many pixels the human eye can see and that many signals the human ear will catch.
Actually, I’ll add another thing I thought about: How do you duplicate or migrate to a newer medium 15 Tons of reel buried in a salt mine? How do you search for a particular 15 seconds of them? I think we get a lot more value for digital archive than for analog, and I think the studios - lo and behold - are fear mongering indeedy.
Hi Amy/Antje,
I took a look at your site as well before posting the last comment, and at first glance I thought you were a competitor. However, to me it seems you are competing with more of social networks/website creation tools that can range from Blogger to FreeWebs (Although mostly higher-end I would think). For us we are more focused on the conversion of analog to digital media, and have the digital upload service as more of an add-on for those sharing large high-quality movies, etc. I could almost see us a partner more than anything else as I think you have an audience that would naturally have old physical media, and that is at a skill-level where they want to digitally share it.
As I mentioned before though if you’d like I could convert those slides and film for you at really cheap prices for the first couple weeks of January. On our website we only offer the subscription model, but we also put it onto DVDs and CDs. Currently, Slides to DVD run about 50 cents each and get much cheaper as the quantity gets higher (We scan at 1800 DPI), and 8mm film to DVD starts at $20/50 feet, but gets much cheaper as the quantity increases.
Shoot me an email once you get a chance:
Bljashinsky at ReimagineMemories.com
Thank you and happy holidays,
-Brad
Bljashinsky at ReimagineMemories.com
Erik, great post — very interesting topic. And it sure drew some excellent comments! Good Christmas Day reading, before the festivities start…
I had to immediately pass it on to some of my smart friends in the storage industry, to see if we can get some more insight on the topic.
cheers,
Graeme
This articles completely confuses storage capacity (terabytes vs reels) with storage formats (files vs film). Bit storage is bit storage and only represents a portion of the cost.
The main challenge is that studios and post production houses store all of their film product in proprietary digital containers from NLE and DAW systems by Avid, Digidesign and Apple. In addition, they use esoteric lossless formats for audio and video that are not well documented, usually as a function of their production platform or camera (e.g. did the HD camera use YCBCR 4:2:2 8 bit or RGB 4:4:4 with 10 bit subchroma sampling?) Nobody in the industry uses the consumer or prosumer formats that most people here would be familiar with.
It is these complex collections of files that are large and very expensive to maintain, and this is a major topic of discussion for cinematographers and archivists. All the terabytes of cheap storage in the world won’t help you if you need to resurrect film content that is stored in a proprietary file or video format obsolete decades ago.
The article mentions open formats for archiving (of which there are several). The challenge to a migration to open archive standards is (a) they are relatively primitive compared to the features offered by proprietary NLE and DAW systems and (b) companies like Avid and Apple have kept a lock on the post production industry by leveraging their file formats as competitive trade secrets.
Yaniv,
Are you aware that anytime the studios cook up contracts for a film or media property they generally insist on retaining ‘universal’ rights? Who can say what the visual and auditory capabilities of our extraterrestrial neighbors will be.
But seriously, if restoring past properties has taught the industry anything, it’s try and maintain a forward looking quality for situations you can’t possibly imagine.
Also, regarding the reuse of film, at least you know that you can shine a light through it and expect a useful result. Film to digital transfer is quite well understood at this point and while you have to be concerned with the quality of the celluloid there is no mystery about how to extract the info. With proprietary digital formats, who can say. Without a published spec on the file format and in the case of editorial files, a copy of the version of the software that understand the file(s) you’re a bit lost and have a lot of work cut out for you.
-gc
Hollywood has already contacted me about making a movie from my book. “The Monkey-boy Conspiracy”
fakesteveballmer.blogspot.com
Those numbers are ridiculous.
Good news for Hollywood movie makers, it would be great to achieve all movie starting from the filming.
While a reel of film may last longer than a current generation hard disk drive (HDD) and certainly longer than an older generation HDD, especially one that is not spinning and just sitting on a shelf or in a storage array, there are some other issues to address.
First the article while bringing up some good and interesting topics about the explosive growth in storage needed for production and post production of a movie or other form of entertainment media, the long term preservation of “archive” of the content is a real and growing challenge and not just on the digital side of the table. Silver based film has a finite life and many vaulting or archiving services for the studios or those who own the content are going to great lengths and extremes to protect and slow the aging of the traditional film. One of the approaches is to get a copy of the original film transferred onto digital medium for preservation.
While the new digital image or copy may not be the final resting place, at least the contents are preserved. Given that these contents are preserved for very long periods of time, much longer than traditional IT related archiving, it is safe to say that the final storage medium for these contents/images/films/files to be stored on is yet to be determined and that the data will need to be copied from one medium to another for maintenance and migration purposes every so many years or in the future maybe every so many decades. That is where the real cost of the storage medium comes into play and that is the cost to maintain the medium.
For example while you can buy a 1TB HDD for a few hundreds of dollars, what are you going to do with that 1TB HDD, where is it going to be placed, how will it be stored and protected, how many copies of the data and how often will the data need to be archived? Even if the HDD is powered down, will it power back up, what about the software running on some server to read and access the media, will that server and software also be archived or preserved for future access of the media?
There are many technology improvements including real-time (or on-line) data compression and other data footprint reduction techniques (see report at http://www.storageio.com/xreports.htm). Storage medium improvements include higher density, physically smaller and lower energy consuming HDDs, FLASH based memories, improvements to optical storage mediums and even 1TB (native) magnetic tape among others. Holographic storage is in the wings with companies like Inphase having early versions of their product in initial testing with entertainment and media venues.
There is also management software that can be used to take low cost reliable and energy efficiently storage enclosure shelves that leverage off the shelve servers as storage cluster controller nodes for use as both high performance storage for production and post production as well as for longer term bulk storage. Other improvements include in the area of MAID 2.0 that implements intelligent power management to selectively reduce the energy consumption of HDDs based on service policies to align the applicable level of energy savings to level of performance needed.
Needless to say, demands on storage continue to grow and with continued conversion of non digital data to digital formats and the ever increasing amount of digitally born data being stored, more copies made and retained for longer periods of time, storage technologies including hardware, software and the networks to access and manage all that data continues to evolve as well.
Cheers
GS
Greg Schulz - Author “Resilient Storage Networks - Designing Flexible Scalable Data Infrastructures” (Elsevier)
Founder the StorageIO Group - http://www.storageio.com and http://www.greendatastorage.com
Having spent well over two decade in the tech world supporting digital cinema/Hollywood, both @ vendors (IBM, Sun) and studios (Disney, SPE), and having focused for most of those years on the requirements and technology of digital archiving, a few comments.
Over the past two years while @ Sun Microsystems, at the request of one of the majors, and with the cooperation of several of them, I researched, ran the real numbers, wrote and presented to several professional industry conferences this paper: Archiving Movies in a Digital World. It can be found here:
http://entertainmentstorage.or.....5%20VN.pdf
The costs published in the recent MPAA paper are interesting. What it really would cost, however, assuming fairly standard industry discounting in effect with the major IT manufacturers, are different. These costs can be found in this paper, as well as the supporting assumptions and data.
The file sizes remain under considerable dispute, and debates will continue between those supporting truly massive file sizes and those of the opinion that far smaller sizes - compressed but at 5X the rate of HD-DVD or Blu Ray (the highest bitrate compression at which these files ever will be shown again) - say, 100-120 Mbps.
It would be premature to assume that any model will be an exact answer in the real world, however. At the same time, a serious model seriously arrived at, shows costs well below those of the MPAA paper for archiving a movie digitally.
At the same time, it is incumbent on Hollywood to begin to enter the digital age far more fully than they have to-date. Doing the same old “store everything” model and continuing to look for “digital archive media” (which does not and will never exist, no matter how much time and money is spent looking for it) in order to support an extinct “store and ignore” archive paradigm will not preserve and protect the $100M assets of the studios. It is time to begin the changeover to digital. This will requires, however, far-seeing individuals with the status neceessary to make this happen. These people exist in the studios and industry. They are not, however, in the positions with the power necessary to hep Hollywood effect the necessary changes.
Gladiator Poster will be the one stop, end all, and be all, destination for a collection of the two thousand motion pictures, Gladiator starring Russell Crowe and Directed by Ridley Scott.