- Preserving and managing rushes and work-in-progress in tapeless environments

- Tracking and exploiting metadata through workflows, including multi-site, multi-vendor workflows

- Cost effective production and media asset management in multi-vendor (Avid, Apple, Adobe, Autodesk to name just the “A”s) environments

- Multi-platform production/project management

Click below to see a time lapse video from the tech room:

The View from OM

This event was the first of many and the feedback was very encouraging in that our community consisting of technologists and creative professionals is ready and willing to collaborate to produce solutions that work.

Thanks to the Mediasmiths (Steve in particular) for running the event and to the other tech companies who took time out to share their thoughts and skills. (Gridiron Software, Vidispine, MXFServer, First Person and the BBC Ingex team)

(Music in video: Chepstow Male Voice Choir)

Adobe/Trams

Digital Workflow Event (15th June) www.trams.co.uk (London)

Who: Nick Pearce & Mark Andrews

Cyfle/S4C/Boomerang/Mwnci

File Based Production Seminar

• Caernarfon (June 22nd) (Who: Mark Andrews)
• Cardiff (June 24th) (Who: Nick Pearce & Mark Andrews)

EBU Technical Seminar – “Networked Content Exchange”

Geneva 22nd/23rd Jun

Who: Nick Pearce

WIG2010

Workflow Innovation Group. Cardiff 29th June.

Who: Jon Morgan, Nick Pearce, Mark Andrews

Feel free to sign up where appropriate and come chat!

The View From OM

June looks busy for both events, sales and technical progress. Exciting times.

• Secure, Searchable Rushes/Dailies Ingest and Protection (Panasonic P2 etc)
• Nearline or parking space for AVID or Final Cut workflows
• Local Library, keeping assets alive to be re-purposed/re-used

The View from OM

NAB is always an experience and shows how niche storage vendors can dominate in a space not yet penetrated by the big tin companies (thought they all claim to be there). Please do get in touch if you would like to have a chat with Mark or myself.

The video (HD version available here) provides a high level view of The MatrixStore in this role:

As shown in the video there are 4 main benefit areas:

• Guaranteed Ingest - MatrixStore generates a hash (digest, digital fingerprint) of the content on the client and also on the two MatrixStore nodes the content lands on. Three different places where the content is checked ensuring the content in MatrixStore is bit for bit the same as came off the P2 or SxS card. Because MatrixStore keeps that digest/fingerprint as metadata with the content we are able to automatically, periodically and systematically (insert line from Grease here) ensure the content on disk matches the digest kept with it. Forever.. and ever.

• Secure Protection - MatrixStore provides the option to keep content in vaults that can have retention periods set upon them. No deletion or modification before the retention date. End of. This helps not only for the long term retention of content but also as a highly secure ingest vault for rushes/dailies coming into the post process. Set a one month retention policy on an ingest vault and that content will be secure from the accidents that do tend to happen.

• Searchable Metadata – “If you can’t find it you don’t have it”. DropSpot harvests metadata contained within the P2 (and other formats) media and creates a searchable rushes (dailies) repository allowing producers or editors to find and retrieve clips in an instant.

• Making use of the content – Now that the data is online and protected at the highest levels you can now access the data. Using either DropSpot or MxFS from any authenticated client content can be copied to and from the archive or can be accessed from within applications like Media Composer or Final Cut Pro. Using MxFS, The MatrixStore is perfect to provide quick access to content to browse or perform a quick edit before working ‘online’.

The View from OM

There is currently no de-facto standard or best practice defined for the safe ingest of tapeless content. The MatrixStore offers the ability to provide guaranteed ingest coupled with an audit trail from ingest through to archive in AVID and Final Cut Workflows.

So that’s what we did. We built security and compliance functionality into MatrixStore from day one. I would like to state now (in the way one of our percieved competitors seems to do quite often .. wow .. geee, aren’t we the bees knees, woot!) that it was our incredible foresight that saw this functionality being 100% applicable to the media and entertainment space… yeehaa. However, the reality is that as a talented team of geekoids we decided to build Rome (compliance, distributed search, seamless scaling, replication .. yada yada yada) and wait to people to come and buy it. If you build it they will come?

Anyway, back to the point of this blog post. How is it that the functionality (included free don’t you know) built for the protection of cheque images and MRI scans is applicable to companies with video workflows or legacy assets? I have covered some of the key points below and at the bottom of this post is a 2 pager on how MatrixStore helps with organisations under the regulatory cosh.

Authenticity

Producers, Facility managers want to know the content that has come off the camera and going into post is authentic. Guaranteed. MatrixStore generates a hash (digest, digital fingerprint) of the content on the client and also on the two MatrixStore nodes the content lands on. Three different places where the content is checked ensuring the content in MatrixStore is bit for bit the same as came off the P2 or SxS card.

Because MatrixStore keeps that digest/fingerprint as metadata with the content we are able to automatically, periodically and systematically (insert line from Grease here) ensure the content on disk matches the digest kept with it. Forever.. and ever.

Retention

Generating video content and adopting creative workflows is no longer restricted to post-production facilities or broadcasters. There is a bucket load of content being generated daily within heavily regulated industries. Hospitals, banks, police forces, utility companies. All producing video and some of them required to adhere to regulations around the retention and authenticity of that video content. Banks for instance may face FSA regulations that require content to be kept protected and available for 1 year from creation. Another factor is that a lot of these organisations are managing the creative process in-house which means they are turning to video workflow solutions over the less appropriate IT based tools. Solutions integrated into those workflows will win.

MatrixStore provides the option to keep content in vaults that can have retention periods set upon them. No deletion or modification before the retention date. End of. This helps not only for the long term retention of content but also as a highly secure ingest vault for rushes/dailies coming into the post process. Set a one month retention policy on an ingest vault and that content will be secure from the accidents that do tend to happen.

Mind you, ask most content owners how long they want to keep their completed works for and the answer will be forever. Good recurring support revenue/business if you can get it.

Access & Availabilty

Not providing an email for a court case has cost some companies a pretty penny. Broadcast or post facilities not being able to serve up content in a timely fashion can be equally costly. Data on demand is key functionality even if components fail.

Find is also key here, the magical eDiscovery. If you can’t find it, you don’t have it (Maxim no. 2). Metadata coupled closely to the content is very important if you are going to stand a hope of getting the proverbial needle out of the 500TB haystack. MatrixStore has that covered.

Equally important is restricting access to content from those who are not authorised to do so. MatrixStore allows content to be kept in vaults that can only be accessed by personnel who have the correct security credentials. Whilst using the same underlying hardware the vaults are completely distinct.

Auditing and Governance

Being able to provide a full audit trail of the actions performed upon a piece of content is crucial in the world of regulations and is also a nice feature for organisations with video workflow (and can actually be very useful where elements of the workflow require repeatable results). MatrixStore provides a built-in full audit trail of all actions, both management and data operations, performed against a MatrixStore cluster. Who archived the content? Who deleted the content? Who TRIED to delete the content? Who retrieved the content and put it on YouTube just before it was officially released? That sort of useful.

The view From OM

Organisations across the globe that are subsiduaries of US companies are pretty much tied to keep in line with US regulations such as Sarbanes Oxley. Even if video data is not a concern of Sarbanes Oxley the internal compliance departments within organisations tend to want all business data treated the same. It has to be protected, authentic and available when required.

We have provided this functionality as part of the solution, at no extra license or cost. As it should be. here is the sheet for your perusal:

One sweet spot for MatrixStore technology is in the creative space. Creative organisations are increasingly being subjected to change driven by external forces such as changes in the way content is captured and our insatiable appetite for HD or 3D content (see The Big Broadcast Survey published in 2009). The industry is one that is used to adapting to whatever is thrown at it but it also relies on solutions that just work.

Below is a quick summary of the challenges they currently face and how MatrixStore is positioned to help. I will elaborate on each point in later posts :

1. File Based/Tapeless Ingest - Some call it pre-production archiving, some call it prechiving, the problem is the same, more and more content is being captured on disk or flash. There is no tape to put on the shelf. Protecting content that enters the building in an audited manner is a key area where MatrixStore is helping. Integration into Final Cut and Avid workflows, digital fingerprints on ingest, locked vaults to protect from deletion and metadata harvesting from P2, SxS etc are some of the benefits.

2. Parking/Nearline – Production disk, the online, the SAN should be kept lean and mean. Some projects can take years to complete so content needs to be moved off and onto the expensive disk repeatedly over a prolonged period. Tape based technologies do not cut the mustard. Organisations are also looking to ‘rightsize’ their expensive disk by offloading older or less-used content that should not sit clogging up the SAN to a more appropriate tier of storage. Seamless one touch capacity expansion, DropSpot (metadata support/search) and performance make MatrixStore a natural choice.

3. Legacy/Local Archive – organisations who own content need to protect their content or they will not be able to re-use or monetise that content. The two maxims come into their own here. Integration with DAM/MAMs, Self-Management,  redundancy of data, retention policies (prevent change or deletion), metadata and search goodness are key here.

4. Content Manipulation/Distribution Repository - If you own the content and have it protected and available on disk then what can you do with that data? Clustered/Grid architecture, high availability, metadata and search again key.

5. Regulations - More and more financial institutions are using video in the way they communicate with staff or shareholders, that video needs to be protected and stored for a pre-determined period in accordance with global regulations. MatrixStore has built in characteristics that satisfy these compliance requirements.

The View From OM

Even medium to large creative organisations will not have dedicated I.T staff to work on the video side of the business so they need solutions that just work and do not take them out of their comfort zone. The MatrixStore is the storage geek so they don’t need one.

The reason is nothing to do with the cloud in itself (though using a cloud vendor as your sole repository for content does, possibly, leave your data held to ransom) it is the ability, or lack of, to move large files to and from the cloud. Companies operating in the creative industries are often crippled by the inability to quickly send/share content with clients or collaborators. The vast majority of this creative community are small to medium businesses 80-90% of who cannot afford fibre connections to their premises and their broadband connections just do not cut the mustard. Its like pushing a 16LB bowling ball through 1/4 inch copper pipes..

The Creation and post-production of video and audio content is a collaborative affair often involving geographically separated teams. Whilst work can and is carried out on proxies there is often the need to share content in its native format. Life would be so much easier if the content could get there that much quicker.

We at Object Matrix would welcome an open forum on calculating TCO with other Vendors but realistically that is not going to happen. Martin also rightly pointed out that defining what should be included in a TCO model will be a challenge so I have decided to bullet point the core elements of a TCO model that I will use and I would ask others to conribute..

TCO Model Traditional Elements

• The Analysis period (should include time to ingest and exit not just time content sat on media @ianhf)
• Initial storage requirements
• Annual growth requirements
• Base hardware and software costs (RRP or Discounted @timothyburton)
• Capacity % utilised of storage media
• Capacity upgrade Unit Size and frequency (@ianhf)
• Annual predicted % change in cost of media
• Media Rotation costs (tape and spindles) @thesharmanator
• Verification of written data costs / proof of ongoing authenticity
• Replication cost (comms link vs manual actions)
• Networking costs/Bandwidth utilisation (LAN/SAN etc). Network hog for minimal gain? (@ianhf)
• Employees required / costs / % annual change
• License costs/transferable? (@ianhf)
• Training/Certification costs of operational/support staff (from @Hugh_Waters)
• Energy usage / energy cost rises / better future energy efficiency % changes
• Servicing costs / employee training
• Failover facilities
• Floorspace (from @storagebod)

Each time I searched for a solution to archive in a tapeless workflow I became disheartened. There are so many products but only a few genres:

1. Magnetic Tape
2. Optical Disc Storage Medium (DVDs, Blu-Rays, CDs…)
3. HDD

And that’s pretty much it! All products derive from the above technologies, even REVs (that are mini-HDDs) and holographic discs. Maybe in the near future SSDs will come into mass storage, but as you know it’s “compatible” with HDDs interfaces.

Of course, that each one has it’s pros and cons is widely discussed around the world (and internet, or it`s the same thing? ), so I’ll not get into this kind of discussion here. But let’s just remember that usually, when someone wants to go tapeless, they also want two things: 

• that those archived assets be safe, and
• to have the ability to access them very quickly.

While I was searching for a solution, I asked myself many times: “Why doesn’t someone come up with something that can use cheap storage (as used daily around the world) and that can be administered in a simple way that doesn’t require a complex structure to accommodate that? And, at the same time, use common components that we know the behaviour of well (including why they fail) so that it is easy to build a redundancy into all of this?

And that’s where MatrixStore comes in.

I was so happy when I found that product that I started to download it without realising that I didn’t have the hardware to use it! And, as a second disappointment, I realised that since I live in Brazil, where Xserves, Xserve Raids and Vtraks are very expensive, and unjustifiable to be used for an archive. However, to my surprise, searching a bit more I found that MatrixStore are “Hardware Independent” and you can set an archive configuration but with relatively little capital investment. You just have to follow a few rules: It has to be Mac OS X or Linux and all discs have to be mounted as one or two volumes on each system.

In fact, an archive can even be installed on 3 Mac Minis (http://www.matrixstore.net/2008/01/25/mac-mini-matrixstore-cluster/ ), although Object Matrix states that’s “only for tests and demos” and is not supported by them.

After pounding my head, I came to a solution that makes sense here in Brazil:

3 – MacPros 8 core

3 – PCIe RAID cards with 8 SATA connectors

15 – 1TB HDD (4 in each MacPro)

The total costs: U$ 16,000 for 12.5TB of a good redundant Archive! About 1/5th of the cost compared to using Xserve and Xserve RAIDs/Vtraks. Another option, even if you do not trust all your data in one geographic location, is to setup two such clusters and use the replication feature in MatrixStore to make a redundant/backup/near-line archive.

And it’s scalable as well. You can add a HDD to each node (8 max in each node), use RAID 5 or RAID 0 or even RAID 6.

So my system looks like this:

MacPro1: Final Cut Server computer

NIC1 and NIC2 with Link-Aggregation

MacPro2: MatrixStore Node 1

NIC1: External (same network as FCSrv)

NIC2: Internal (as mentioned in “prepare Hardware” manual)

MacPro3: MatrixStore Node 2

NIC1: External

NIC2: Internal

MacPro4: MatrixStore Node 3

NIC1: External

NIC2: Internal

External Network connected using Dell PorwerConnect 2716 and Internal using a GS608 – 8-port Gigabit Netgear Switch.

Our system has 3 stations editing over Gigabit Ethernet in a 4 Link-Aggregated Dell Storage. I have recently acquired Final Cut Server to manage not only our workflow but to digitize and catalog as well our tape library. For our luck, Object Matrix also has an FCSrv Plugin! So it’s just a matter of configuring.

You set all this up and running in less than one day. And with GbE, archiving and restoring to and from MatrixStore archive is a snap (just  two mouse clicks in Final Cut Server). 

Since I only have one computer for the Final Cut Server system and we ingest lots of material, I thought that maybe the three nodes from MatrixStore could be a 3 node for Qmaster as well! And it works great! Just DON`T let one of nodes be the Qmaster Controller because it uses NFS Server and this will mess with MatrixStore communications between nodes.

I’m testing this system exhaustively to see how stable it is, and currently I`m convinced that it could be used for small video production companies.

If this solution appears to you as too much of a “Frankenstein”, you can also use the newly released “The MatrixStore” (http://object-matrix.com/thematrixstore/ ) a pre-configured plug-and-play device designed especially for archiving important assets.

So, these are the solutions that Object Matrix came up, which answered my wishes, and stopped letting the video/IT market disappoint me anymore.

Lucas Saldanha Werneck.

The View From OM

We are delighted that Lucas has chosen MatrixStore to protect his content and we look forward to working with him and his colleagues/clients in the very near future.

So, to reiterate I have pinched some details from other websites (wiki, Apple, Black Magic, Microsoft) and am happy to stand/sit corrected if any of the data below is incorrect. The information on 3D came from the talented Mr 3D.

SD

HD

3D

No table here but the following information is straight from the horses mouth so to speak…

“60 minutes of RAW data for dual digital projection amounts to 1.5 Terabyte for the imagery alone,  although a DCI-compliant master is smaller in size due to the Jpeg-2000 compression used”.

For more info on 3D Stereoscopic movie making a visit to the3drevolution.com site is a must.

Assumptions

1024 bytes in MB

References

Alexander Lentjes

Ben Waggoner at Microsoft

Apple

Wiki

Black Magic

The View from OM

Foggy! Please do let me have any sources you may have for this information and I will keep this post up to date.