• Secure, Searchable Rushes/Dailies Ingest and Protection (Panasonic P2 etc)
• Nearline or parking space for AVID or Final Cut workflows
• Local Library, keeping assets alive to be re-purposed/re-used

The View from OM

NAB is always an experience and shows how niche storage vendors can dominate in a space not yet penetrated by the big tin companies (thought they all claim to be there). Please do get in touch if you would like to have a chat with Mark or myself.

The very same is happening to PPC and early Intel based Xserves. Out with the old, in with the new.

The re-use opportunity lies in that these are still very reliable, durable units that could be re-purposed within organisations for nearline or archive storage rather than ending up as doorstops (albeit very pretty ones) or fodder for Ebay.

One organisation who decided to take that route is AMV BBDO, no less than the UKs largest advertising agency.

The LAB at AMV BBDO is an Apple based post production facility (read more here about The LAB) that has recently upgraded its XSAN from 1.4 to 2.0, whilst also replacing their trusty, loved Xserve RAIDs with the more powerful (though pugly) RAID subsystems from Promise and introducing the lightening fast Nehalem Xserves into the mix. This much needed upgrade would ordinarily leave 6 Xserve RAIDs and 3 PPC Xserves on the shelf but the guys at The LAB wanted to re-use their kit by using it to add storage capacity to their MatrixStore cluster.

“Our Apple XRAIDs had served us well for 5 years, and when we upgraded our SAN it seemed a shame to consign them to the scrap heap. Redeploying them directly into our MatrixStore made commercial sense, and the process was disarmingly straightforward and uncomplicated.” Alex Buchannan, Tech Manager, The LAB.

Using the Mac OS version of the MatrixStore software an archive cluster node can be created using one Xserve connected to one Xserve RAID (see here for brief explanation).

The spare kit at The LAB allowed them to instantly add 3 Xserves and 3 Xserve RAIDs to their existing MatrixStore cluster providing an 20TB raw storage. The remaining Xserve RAIDs will be added at a later date.

The LAB at AMV BBDO have been an Object Matrix customer since 2006. Once the dust has settled on them, they archive projects from their XSAN into the MatrixStore. From the MatrixStore they quickly retrieve assets on demand. The nature of AMV BBDO’s business means that elements of past adverts (lower thirds etc) may be needed for current or future bids or work. They need data elements to hand but do not want them cluttering up the XSAN.

AMV BBDO have a fully automated, integrated Final Cut Server workflow utilising the MatrixStore plugin for Final Cut Server to archive all assets and selected metadata from a production with a single click (see here for more info).

If you are considering rolling out your Xserve RAIDs and would like to implement a MatrixStore cluster then do get in touch. The current free software license offer to manage up to 15TB of data will only be available for a short period of time, after which a 5TB free license will available.

• File based/Tapeless workflows (P2, Sony, JVC)
• Automated Digital Asset Management (Final Cut Server, CatDV etc)
• Digital Preservation
• Disk based archiving
• Pre & post production archiving
• Rugby (Nick)
• Fooseball (Jon)

So pop over and say hello : Video Rescue Stand (6.C28d)

Cheers

N

The archive should be alive; available to use and reuse the data, wherever you are located, and whenever you require it.

Solution: Disk-based archiving, online, strongly fire-walled.

The archive should be offsite; but the data must be completely under the control of Hogarth.

Solution: Co-hosting in an offsite data centre.

Access speed must be fast.

Solution: Be able to reach the target over a Gigabit link.

A scalable solution with discrete data partitions on a per customer basis.

Solution: MatrixStore provided that along with 256 bit access keys and encryption where required.

So all well and good: as you’d expect reading a blog on Object Matrix’s website, our solution fitted perfectly and they purchased.

Then the mud hit the fan – so to speak.

The solution installed perfectly and was up and running in no time, but there was a problem: speed.

Speed was down as low as 25MB/s during some file transfers, faster than many archive solutions (even disk other archive solutions) but still far lower than we’d have expected; so Hogarth turned to us for help, and even considered purchasing some expensive file transfer technologies. However, we quickly realised what the problem was: high network latency.

In most of the environments that we’d previously installed, the networks were either slow, in which case we’d filled bandwidth, or fast and low latency; in which case we’d filled bandwidth. What we hadn’t had to cope with was a network where acknowledges returned for each packet of data sent were delayed by a few nanoseconds. Multiplied by Gigabytes of data, that delay was killing our performance.

A solution was required.

Because the Object Matrix solution runs software at both the client and server “ends” of the solution, we are not dependent on a single transport protocol to use when transferring data. Therefore, without needing to change around the whole solution, we were able to create a protocol more suited to high-latency type connections. The result? Hogarth are now able to see their full (approx. 100MB/s) bandwidth filled, and are able to offer a superb archive facility with instant access to their customers.

At OM we feel it demonstrates the advantage of including an API which runs on the client and the server for secure, reliable transmission of data (saving the customer from having to purchase some other file transfer product/set up VPNs), and are pleased that we were able to release the solution to the customer within a few months of them raising the issue, and at no cost to them.

See also: About MatrixStore

Object Matrix will shortly be helping to host a seminar and panel discussion on digital file based/tapeless workflows in the media and Creatives industries. The event will be hosted in Cardiff, and will be in a similar format to the preceding events held in London and Manchester.

The seminar looks at the full digital production workflow, taking in: Acquisition, Assets, Archiving and going out to Air,  aka .. A4 and is held in conjunction with Skill Set, Panasonic, Apple and Bectu.

Attendance is by invitation only and numbers are strictly limited, but please feel free to contact us if you feel the seminar would be appropriate to your organisation. ]]> http://www.matrixstore.net/2009/04/16/a4-tapeless-workflow-seminar-cardiff/feed/ 0 http://www.matrixstore.net/2008/11/12/towards-100-times-better-energy-efficiency-from-hard-disk-drives/ http://www.matrixstore.net/2008/11/12/towards-100-times-better-energy-efficiency-from-hard-disk-drives/#comments Wed, 12 Nov 2008 14:30:17 +0000 Jon http://www.matrixstore.net/?p=321 Over the past 25 years disk sizes have grown as shown in this diagram:

(From Wikipedia)

The growth rate, as can be seen, is very linear. As can be seen in the diagram, we’ve risen from a 1GB HDD to a 100GB HDD in 10 years, or a 10GB HDD to a 1TB HDD in… 10 years.

Thus, every 10 years storage capacity becomes approximately 100 times more energy efficient to use as an archive.

Meanwhile, the fall in price for a HDD has been huge, but exponential. Back in 1980 you’d be looking at a staggering $233,000 for a GB of storage space, where as today a GB costs less than 20 cents. Placing sample data of HDD prices from 1998 to 2004, I produced the following:

So, growth in disk size is linear, whilst the fall in disk $/GB exhibits exponential decay.

Overall, this makes for very good archiving economics on HDD.

Better Disk Electrical Usage Economies

The amount of electricity used by a disk drive (regardless of storage capacity) has also fallen, though nowhere near as dramatically as the capacity per HDD has risen.

Let’s take a modern HDD: e.g., the Hitachi 1TB Deskstar 7K1000.B gives “43% energy savings when disk is on standby”. That’s 5.2 watts on standy instead of 8.4 watts. Therefore, in archive usage, if a disk is idle for 700 hours in a month, it’ll use 3.64 kWh.

This compares favourably to most typical HDDs 8 years ago; e.g., the 30.7GB Maxtor, which drew 6.8 watts on standby and 12.9 watts when seeking. In archive scenarios, we calculate this as a 30% energy draw saving.

Whilst 30% energy efficiency savings are never to be turned away, the fact that the Hitachi stores 1TB of data (33x the amount of data) is far more significant. Our conclusion? In the long run, more important than paying a premium for the latest “green technology” is buying a solution that allows the modular addition of capacity; a solution that allows the use of the best available technology at the time of that addition.

Taking the trend of increased disk space in a HDD unit and falling energy usage, it can be seen that 100x energy efficiency is being achieved every ~9 years.

New Technologies: 2.5” Drives and SSDs

All general electricity usage improvements on 3.5” drives are good but perhaps more exciting are the new 2.5” drives coming out and SSD technology in general.

A 2.5” drive uses 1.4 watts for Read/Write (e.g. Fujitsu 500GB, Hitachi Travelstar 5K500.B). No word on their standby electrical usage – but even taking the worst case of 1.4 watts that’s a significant improvement over 3.5” drives sucking 5.2 watts on standby.

Plugging the figures for 2.5” drives vs 3.5” drives into our electricity usage formulas results in a slightly higher electricity cost for a 2.5” drives configuration. Why? Because at the moment they store only 50% of the data, and because therefore one ends up using more CPUs, RAID controllers, etc. However, it’s getting close, and it wont be long before 2.5” drives will provide better TCO archive economics. OM therefore predict that within 8 years we’ll have 100 times better energy efficiency from HDDs!

SSDs are a longer term prospect, but a very promising one. Their electricity draw is significantly lower again, and people like (whatever the reality behind it) the concept of non-moving parts. I’d personally be VERY cautious about this technology up front – I believe it’ll take many years before the reliability of SSD will be proven as a suitable long-term storage medium and until then many people will get stung by storage that gets corrupted over time.

Optimum Time Between Disk Recycling Taking Into Account Energy Usage

When should an old unit of storage be rolled out in favour of a (presumably) larger capacity storage unit?

Let’s take this to be the point where a years energy usage to keep the old unit running is greater than the cost to buy a similar amount of storage in a new technology.

Projecting this into the future:

1. If current trends continue then disk sizes will be 4 times current sizes in 3 years time.

2. Calculations show that a 100TB MatrixStore cluster costs in the region of £2,271 per annum in electricity (including cooling, HDDs, CPUs); this is based on an 8p per kWh price for electricity and 24TB nodes. The purchase cost of 100TB (that’s 220TB before Raid and Mirroring) is £120k.

If we then assume that electricity costs rise at 10% per annum, and that rack space is NOT of a premium, we end up with a scenario where the electricity cost for a 100TB of storage “crosses over” the cost for a similar amount of storage after approximately 7 or 8 years.

At that point the new 100TB archive will arrive in 1/25th of the space, and will use 1/25th of the electricity.

Now, since rack space generally IS of a premium, we project rolling out old hardware every 6 years to be a good ballpark time-frame. (NB: this does not take into account the cost of moving the data).

Alternative Solutions: Tape

Changing tact a little here, let’s briefly discuss why tape based solutions might not be as green as you think.

Tapes need to be stored at the correct temperature and often are physically very space consuming. However, the main “non-green” factor of tape archives comes in if they need to be rolled out in their entirety, due to any of the following reasons:

• because of hardware maintenance costs rise for systems that are outdated
• because of higher manpower cost related to maintaining old tape formats / sizes compared to newer solutions
• because new demands on data usage require an updated solution

(to name but a few)

We do not believe it is unfair to suggest that tape solutions often get rolled out every 5 years or so.

When it comes to rolling out a tape solution, it may well be required to move all their data from old tape formats to new tape formats; a labour intensive exercise that is neither “green” nor economically.

In comparing a tape archive to a HDD archive; the cost of transferring data at the end of the tapes lifespan may well exceed the energy cost of keeping that data on spinning disk (and therefore easy to transfer).

OM’s CONCLUSIONS

Based on trends from the first 30 years of HDDs:

• Disk capacity rises by approximately 158.5% per annum
• Electricity usage per GB falls a hundredfold every nine years
• Economically, the overriding most important aspect of a disk-based archive is being able to roll in/roll out new/old technology as appropriate
• Being able to take advantage of SSD/other emerging technologies is advantageous
• With economics typical in our example, hardware should be rolled out every 5-7 years – possibly sooner if rack space is of a premium or electricity prices rises greater than 10% pa
• Alternative solutions often have hidden energy and “green” costs related to manpower required and frequent technology form factor redundancy
• As the costs have fallen, new demands for HDD data storage and usage have come in and will continue to do so

Comparing keeping an archive on disk to the cost of doing the same 10 years ago, that cost has fallen rapidly in terms of hardware, manpower and electrical usage. There is no reason not to believe that trend will continue.

External Links

Wikipedia Disk Capacity Over Time
Indopedia History of the HDD
Dave’s NetApp Blog: Disk Is The New Tape—Flash Is The New Disk
Hitachi 1TB Deskstar 7K1000.B
30.7GB Maxtor

Today the MOD announced yet another very serious breach of security with disks going walkabout that contain confidential information on nearly half the personnel of the British armed forces.  Here in the UK valuable items usually get some sort of escort when in transit, be it gold bullion, jewels or a wad of £50 notes. When 25 million confidential records (worth millions in the wrong hands so we are told) are sent outside an agency in the UK some office junior puts it in his coat pocket with the intention to pop down the post office after his cigarette break. Not of course that the office junior is to blame but he/she will be the one who gets the P45 out of the experience.

<rant on>The way we treat data is just terrible both in a personal and corporate sense. As individuals we give or throw away old computers and mobile phones without giving a second thought to the personal or confidential information held within. We simply have not had enough time or experience with living in a world where our data goes with us 24/7. We are not conditioned to automatically protect the digital information we keep on or about our person in the way that we would protect something tangible like a passport or birth certificate if it were in our possession. Neither do we possess the skills required to adequately remove that digital information from the current device when a new phone or computer takes our eye. Understanding just how big our digital footprint is and subsequently protecting it from miss-use/abuse is an area of daily life that I imagine 80% of the population simply do not understand. <rant off>

Below are links and summaries of just some of the cases of data loss that have occurred in the last two or three years in the UK alone: 

Child Benefit records of 25 million people

Alistair Darling told the House of Commons this afternoon that a police investigation has been launched into how Her Majesty’s Revenue and Customs has lost child benefit records relating to 25 million people.

Lost details of 100,000 servicemen and women.

The Ministry of Defence and contractor EDS are frantically checking the bins this morning for a missing hard drive containing records of 100,000 servicemen and women and their families.

87 USB Sticks containing Classified Information

The UK Ministry of Defence has told parliament that it has lost or had stolen some 87 USB sticks holding “protectively marked” – ie classified – material since 2003.

11,000 Military ID cards

The Ministry of Defence has lost 11,000 military ID cards in the last two years, the government has admitted to parliament.

Details of 600,000 potential Navy recruits

The Ministry of Defence has admitted losing the details of 600,000 people after the theft of a laptop from a Royal Navy officer in Birmingham last week.

Prison Service Courtesy of EDS

An estimated 5,000 prison officer and admin staff after private contractor EDS mislaid a sensitive portable hard drive

Prisoners Data goes Walkies

Names, addresses and expected release dates of all 84,000 prisoners in England and Wales were on the portable USB memory lost by PA Consulting, a Home Office contractor. The lost files also contained data on up to 40,000 repeat offenders, and the initials of people on drug-treatment programmes.

Bank Details found on Motorway

A box of files belonging to UK insurance group Prudential and including the banking details of 200 customers has been found on a motorway slip-road after it apparently fell out of a courier’s van, according to press reports.

Lost Virgin

Virgin Media – the entertainment and communications arm of Richard Branson’s Virgin Group – has lost an unencrypted computer disc containing the bank account details of 3000 UK customers.

3 million Learner Driver Records

It is estimated that up to three million learner drivers details were lost in the incident, which is alleged to have occurred last May.

Insolvency Service Loses Data

Following reports that a laptop containing the details of 122 former directors of insolvent companies has been stolen from its Manchester offices, it has been revealed that this was one of four laptops stolen. Although no bank account details were held on the directors, names, addresses, dates of birth and occupations were. The Insolvency Service also reported that a further 150 people have been directly affected by the loss of the data…

Info sourced from (amongst others):

• www.theregister.com
• ukliberty.wordpress.com/data-loss/

More Data Loss
Scottish Police loss a memory stick containing details of 750 cars that are “of interest” to the police..

Green Storage

Don't Panic

Where is that runner?

Tape Cuts

I will not re-open the whole tape vs disk debate here, there are older, wiser and less biased heads out there in the blogosphere (storagemojo.com, drunkendata.com etc) who have written reams on the pros and cons of both technologies.
Cheers from sunny south Wales and have a nice weekend…

You can also go to the Apple site to get more documentation about MatrixStore and Final Cut Server:

http://www.apple.com/finalcutstudio/solutions/infrastructure.html

See Also:

• Video of archiving from Final Cut Server
• MatrixStore solution brief
• Apple Customer Workflow Case Study

MatrixStore integrates into Final Cut Server adding secure, scalable storage. It can be used as a Nearline store, as a Pre-production archive, and/or as a Long-term storage archive. Best of all, it is integrated right into the Final Cut Server menus so that moving data is as simple as a click. Here’s a picture of the integration in action:

Final Cut Server calling MatrixStore

As can be seen the integration allows assets AND ENTIRE PRODUCTIONS to be copied in and out of the archive with a single click.

Here’s a use-case view of using MatrixStore (pdf):

Final Cut Server can use MatrixStore as an archiving device. This means that MatrixStore can protect both your media files and metadata from within Final Cut Server on disk-based storage. You get the advantages of fast access, really easy management, maintained data authentication and strong data security.

Another interesting aspect of using MatrixStore is that it can be used as an archive not only for individual Final Cut Server assets (along with their associated metadata) but can also archive whole productions; so that the metadata for the production can be archived with the production, and so that the whole production can be retrieved in a single step. Furthermore, MatrixStore can be set to automatically backup Final Cut Server itself and to take backup snapshots of assets whilst they are in production.

Does this mean the end to complex backup applications, processes and backup tapes?

Do you still need to rely on an IT Specialist to find lost or archived media for you?

Of course not in all cases, but in many cases yes. Using MatrixStore allows you to automate your workflow without the need for regular manual operations to be laboriously carried out.

So how does MatrixStore as a device within Final Cut Server shape up against tape backup?

• Metadata is stored on MatrixStore along side its associated media file. Individual files or projects can be searched and retrieved instantly. It’s fast too – because it’s disk.
• One Application. The tool you use throughout your workflow, the one you’re familiar with for your daily work, puts the archiving and retrieval process at your fingertips.
• No additional complex backup applications and processes to deploy and manage.
• Fast Multi-user, instant access to data archives.
• MatrixStore allows you to plug in new storage devices together with older ones, allowing you to roll-in, roll-out technologies as they become relevant/irrelevant. With tape, you normally have to start again all over.
• Arguably more reliable long-term data protection.

In ten years from now, having all your data and metadata stored safely in their original format on a standard file system is of value. And having that data in an easily copyable/reusable digital format is key. Which is the whole point of the process. BTW, anyone got a working DLT drive that can read a 20GB DLTtapeIV from 1998 I could borrow?

Hear one of our customers, EO broadcaster of the Netherlands, talking about how MatrixStore fits their workflow here.