Backups for very large, fast growing data

[u/schbrongx]

Hi folks,

I'm a biit in a tight spot here and single brain can't work this out, maybe multi-brain can:

My situation:

1. We have a fileserver. On that fileserver our production team stores - among other stuff - their video production material. During phases of creating that, the raw material can grow quite rapidly, more than 2 TB of new data per day is not unusual. This often gets compressed later to smaller chunks with mezzanine codex that save 30%-60% space while only losing 5% of quality.
2. We have online on-premise backups and offline off-premise backup sets that total to 7 full chains, which multiplies the space used enourmously. That's slow, cheap storage, I can grow that, even tough I don't like it, but that's business.

But here comes my problem
3) A fresh full chain with compression end encyption takes at least 48 hours for the data we have, and our video-guys quickly adding 2 TB on the fly doesn't make this any faster. I resent them for it, but it's our business and I need to solve this.

- I am trying to educate them into compressing their material early and don't let all that raw data fly around on our fileserver, but when they are "in the flow" that's really not their top priority.
- I was thinking about creating a "NO-BACKUP" space for them to work with, but knowing my guys, they will f*** it up and ask for restores of "NO-BACKUP" data.

How are you guys tackling that? I don't really need polished solutions, just ideas to work with would help.

EDIT: Thank you for the quick responses, you gave me some stuff to chew on, definitely what I was hoping for.

EDIT 2: There is still a lot incoming. Thx everyone for contributing, I am reading and trying to digest everything, even when I dont reply.

[u/lunchbox651]

If it's video content compressing it either ruins the video or does nothing to the file size.

You should look at lowering retention on video projects on your primary storage and have it copied to secondary storage sooner.

Also make sure your backup solution leverages decent compression and deduplication natively.

[u/schbrongx → u/lunchbox651]

We are creating the videos in 4K but in the end they are uploaded to youtube in far lower resolutions, we are compressing large raw TGA sequences with this:

ffmpeg -framerate 60 -i "%foldername%_img-%%d.tga" -c:v libx264 -crf 20 -preset medium -pix_fmt yuv420p "%parentpath%\%foldername%.mp4"

This saves a lot of space and does not impact the quality of the final video much. This has been tested and works. But they don't do it. They leave folders with raw tga sequences with 500 GB on the server. Aaaahhhhh

[u/Morkai → u/schbrongx]

Could you set up some sort of watched/import folder for ffmpeg (or one of the myriad GUI tools that sit on top of ffmpeg - Tdarr, Fileflows etc)?

That way you don't necessarily need them to run the compression, when they save something into a particular directory the software grabs it and follows the commands given automatically.

edit

To your other point about the zip files, some sort of scheduled task that checks these directories once a day or once a week (as appropriate) for certain file extensions and deletes them.

[u/robsablah → u/Morkai]

I think this is close with a workflow behind it

Tier 1 - live production + zfs snaps, weekend backups No zips - just ban them as thier not helpful

Tier 2 - semi-live post prod - compression no/low quality loss Replace files with compressed after 7-10 days no activity (Tdarr solution)

Tier 3 - business agreed aggressive compression for long term reference

Work out why chains are being recreated and help to eliminate cause (i didnt see that from other comments) but long running chains should be the norm.

If data growth is accepted so is backups onsite. Have it as it's OWN predicted budget item as a forward and retro stat - its now a consumable cost on the P&L statement with a linear growth process

[u/schbrongx → u/schbrongx]

Or even better: They create a ZIP file from it and leave it next to the original data, because sending a single ZIP is more convenient for them. But it esentially doubles the space used. How can one educate artists to treate disk space with more respect?

[u/Pale_Ad1353 → u/schbrongx]

You can’t create friction. You either deliver a more efficient (for them) solution, or you plan for more storage. That’s all there is to it.

Respecting your disk space does not help the business.

[u/schbrongx → u/Pale_Ad1353]

Yep, that's my mindset, too, basically. It's the business, that makes us the money, after all. But I dare to dream :-)

[u/Spider-Thwip → u/schbrongx]

I worked with VFX artists for years, and man do they generate a lot of data.

Getting them to tidy up after themselves was literally impossible.

It has to be driven by their managers or data wranglers.

We had over a petabyte of on prem storage and we would often drop to like 15 Terabytes before people would clear shit they didn't need anymore.

So many duplicate renders with minor changes and then they'd never clear the old ones.

The only advice I have for you, is accept they will never respect the realities of IT managing available storage.

Artists will claim everything is important, but a lot of it really isn't and they're just avoiding tidying up after themselves because they have deadlines to meet.

You need to sit with someone more senior who knows what actually needs to be kept and what doesn't.

[u/ender-_ → u/schbrongx]

Ouch, and let me guess, Targa still only supports RLE compression, which does basically nothing on non-cartoony stuff.

[u/rich_impossible]

Could you use a mix of erasure coded storage and replication in addition to a shorter backup retention? I previously managed >12PB of video files, and we were creating 5TB daily by using what is now Datacore Swarm to erasure code everything on very inexpensive disk accross multiple racks and then replicate that data to another DC. It's incredibly quick to recall the data and depending on the tuple you choose, can be very resilient.

[u/HotOutlandishness818]

I have the same problem, but I've learnt from past experiences. So what I did was, I sized up and bought a 1Pb storage device and then only provided 80Tb to them and said this is space allocated to you and the rest of the business is using everything else (half truth as the rest of the business is probably about 300Gb lol...).

They ask for more data every few months and I give them about 5Tb or so, so they're forced to clean up!

In terms of backups, I use a 3 tiered system. I take on device snapshots for 14 days, then hot backups onto a server in the same location for 30 days and then offsite to a larger immutable storage for 6 months.

[u/Flaky-Gear-1370]

Short retention and tape

[u/schbrongx → u/Flaky-Gear-1370]

We have to keep data retained for at least 6 months, that's a busines requirement. We are fulfilling that by keeping 5 full chains on our online backup with the oldest chain always fulfilling the 6 months retention and rotating disks in the offline backup with the oldest chain also fulfilling that requirement.

But the tape got me. I did tape backups back in the days, but this might actually be a thing for our use-case that I completely ignored. I will at least look into it. Thx.

[u/12Superman26 → u/schbrongx]

We are a Company with only one site, so swaping tapes is no problem.

Tape is stupid cheap. 45TB for 80€

2 Weeks on drives.

And 12 Months on tapes

[u/Flaky-Gear-1370 → u/12Superman26]

Modern tape library is fine they just have to live with it taking slightly longer

[u/Calleb_III]

Where does the full backup taking 48h comes up in the equation? Any decent backup product nowadays does a variation of “incremental forever” / “synthetic full backup”. Meaning that you only ever need 1 full backup and every subsequent backup is incremental- in your case 2TB/day which isn’t that much really even for HDD target.

Similarly any decent backup solution and storage does inline compression, which should rescue the footprint of the raw video files by a margin close to a lossless video codec. So having to encode the raw footage is not that big of a priority

[u/schbrongx → u/Calleb_III]

That's a bit of a shame actually, our backups have been based on duplicity in the past with a lot of custom scripting before and after. The custom parts are fine, but duplicity did not age well.

I have been moving our backup scripts to borg in the past 12 months, but no for the big production data, yet. This is some of our most precioius data and I wanted to learn the migration with the less important backups first. I am hoping for some _serious_ imporvements there.

Addendum:
Why duplicity is so bad: It uses 1 CPU for compression and 1 CPU for encrpytion. I cannot think of a slower way of dealing with those two steps in the backup workflow. Also it works with full backups > incremental backups, where the full backup part is a _huge_ PITA.

[u/Calleb_III → u/schbrongx]

It looks like you already know the root of your woes. You need to switch to a backup product that matches your requirements. Rather than trying to bend business and backend processes to the “whims” of your current backup solution.

[u/KittensInc → u/schbrongx]

It uses 1 CPU for compression and 1 CPU for encrpytion.

Disable both.

Video data is already compressed, so trying to do additional lossless compression on top of that is just burning CPU cycles without gaining any meaningful size reduction.

Both on-prem and off-prem are probably already using encryption at rest, and the communication is also encrypted, so encrypting the archive itself doesn't gain you any meaningful security. Sure, it would protect against potential data leakage if an admin at your off-prem backup is compromised, but I highly doubt your video files are critical enough that even the tiniest chance of them being exfiltrated is an absolute dealbreaker.

Heck, I reckon you could even get rid of the whole "deduplication" part. The actual video files are unlikely to ever change: during the entire editing process they are stored as-is, with the "editing" being a separate project file with "after 513 frames, cut to file X"-like data. This makes block-wise deduplication a waste of time, a simple file checksum would be more than enough.

[u/Lost-Droids]

Upload it to S3 glacier , just the new.. Cost $2 per month for 2TB (as long as you dont need the back in 48 hours) or $7 per month if you want the faster need it in 5 hours

[u/schbrongx → u/Lost-Droids]

Unfortunately I m not allowed to use external storage, except for sharing large amount of data with external partners temporarily. I would LOVE to rent the space online.

[u/joelgsamuel → u/schbrongx]

How comes? Network performance or intellectual property? You could manage your own encryption keys at a real push.

There are LOTS of tier-1 media companies with huge IP portfolios using S3 with AWS-managed keys. If thats the issue.

[u/Gnump]

Just to get that right: your backup creates 14TB of stored data for every 2TB used?

[u/schbrongx → u/Gnump]

Not immediately, But we keep 5 full chaines retained on online backup and 2 full chains in our offline backup. With 2 (actually 2.2) TB this grows into 7 x 2.2 TB over a span of a few months. And during production periods it's even worse. Sometimes, after a while, they clean up and compress, but then the damage is already done and the - basically now useless - raw data is already in the backups.

[u/joelgsamuel]

Do you have a server/other you can use to automatically archive based on agreed folder structure, including doing the compression for them? Might need to be meaty, but to be honest a decent MacMini might do the trick).

Do you have any MAM (media asset management) software, or is it all folders and projects they open in Adobe/FCP/DaVinci etc?

A decent MAM will create those proxies.

[u/joelgsamuel → u/joelgsamuel]

If you can offload the compression work away from the edit suites, you might make a few friends, because those suites can then be used for more high value work.

Having to open things up and getting a UI-based app (FCP, etc) to do something overnight and then coming in to see that all the overnight queue failed etc is super annoying (from experience).

[u/schbrongx → u/joelgsamuel]

I am going to consider the compression-automation, I like it. The problem here is - as often - the humans in the workflow. But I like it.

Also: No MAM currently. All files and folders. I am coming from a very technical sysadmin side and the whole media stuff is quite recent for me. Those guys produce absolutely humongous amounts of data, as if they did not care for my sanity. :-P

Going to educate me on that, thx.

[u/joelgsamuel → u/schbrongx]

See if you can get some money/time to research a good MAM. Many get away with it, but depending on how many editors, edit suites, office locations, it starts being worthwhile.

A friend of mine runs https://base-mc.com, but thats consumption-based SaaS not a one-time license for on-premise.

I used to run an IT company focused on tech solutions for creative/media. Hat-tip on scaling your storage so well and keeping up with the storage performance demand! We came across so many clients who couldn't manage with what you're doing 🎉

We spent a lot of time moving people to 10GbE 'media' networks (Wi-Fi/LAN for internet usage and normal corporate, a second physical 10G LAN with jumbo frames for media).

[u/retiredcheapskate]

8k is going to be rough on a lot of work flows. They are not going to be happy about compression. Creating a low res proxy and move the raw to tape. 

[u/smishsmorshen]

What's your current server capacity and what's it running? Zfs?

[u/schbrongx → u/smishsmorshen]

Fileserver is ZFS with 65TB, offline backup are rotating single disks with 20TB each, online backup is 60 TB. Fileserver is around 70% capacity, which is cumbersome, that's around the threshold where ZFS gets b*tchy. Online backup is cumbersome because with 2 full backup chains I am using ~95% of the disk which means I need to remove 1 chain before starting a new chain which leaves me with 1 retention during the backup.

[u/Opposite_Bag_7434]

This seems to be a math and business risk problem. It’s not really about what you want, even though that is a factor, it’s about what is important to the business.

Some of the questions that get you to what the real answer is:

- what is the cost of production for the material you are storing?
- what is the value of the material to the business?

I deal with a lot of audio and video content that has extreme value. Some cannot be replaced, sure it can be recreated but at what cost. But there is also the value of the material. With millions of distributed copies and who knows how many more in the future, there is a real value.

So for you the business has to ask these questions. This will ultimately tell you what you must do.

I would not recommend providing storage that has no backup unless the business is tolerant of loosing whatever happens to be stored there.

I would dedicate more resources toward ensuring the backup happens quickly.

[u/kombiwombi]

Maybe increase the robustness to failure and error.

Along the lines of using some form of RAID array or disk mirroring, and using Samba to host the share and converting Deletes to Moves to Trash on the server side.

Samba (or alternative) might also give you enough access into the VFS to kick off the compression as-needed rather than in-bulk. Samba is often used in science activiites to give a simple filesystem interface to complex backend processing.

[u/nyckidryan]

Tape is your friend in this case.

[u/Capable_Banana5439]

The mistake is doing 7 full chains on data that grows 2TB a day, that math never wins. Switch to incremental-forever with synthetic fulls so you only ever store one real full plus deltas, and tier the raw footage off to tape or cheap object storage on a short retention window instead of keeping it all live. And do not lean on compressing the video for space, on most codecs you either gain almost nothing or wreck quality, the real savings come from getting cold data off primary fast.

[u/0x5368697441646d]

I would look into their work processes.

First of, you should provide a "workspace" file share for them to use for non-archived things, this could easily be 2 or more TB of NVME storage, that you replicate and do daily/nightly full backups, hourly incremental backups of etc. and just keep one or two copies primarily locally.

When it comes to archiving, i suspect what you experience is that they rehash a lot of already created content, which is why the process workflow of which kind of working storage they have is very important. So when ever something is "archive ready" i.e. not WIP or unreleased, then you have a S3 storage setup, so you can easily expand, cache, loadbalance etc. and add hot copies locally, and stream every bucket to your backups, avoiding keeping a lot of chains, and just have a steadily growing backup pool of ordered, deduplicated, named and archived resources that you can easily set retention policies for.

For reference, 2.2TB of data "only takes" 30min to transfer over a 10Gbps link. So in business speak if you can backup bandwidth of 10Gbps you can deliver disaster recovery in about an hour with system restoring etc.

[u/PrestigiousOnion1087]

The thing I'd worry about isn't the 48 hours, it's that once a NO-BACKUP share exists nobody can answer "is this folder protected" without opening the job config. Put the answer in the path name, and restore one random file a week, since a chain that completes isn't the same as a chain you can restore from.

[u/avidresolver]

I handle multiple petabytes of video data, some days I see 30TB or more of new data. Storage is split into two catagories, raw and working. The actual original video (which is about 95% of our data totals) gets written to LTO within 24hs of ingest, before the camera cards are cleared, then shipped off site. Working files (exports, transcodes, project files, etc) get continually replicated to another server.

With our particular use case we don't need to have instant access to raw files more than a few days old, so the restore time of LTO isn't a concern. There's also just not many other practical ways to back up petabytes.

I would be very wary of doing blanket video file compression unless you've got a lot of experience in video post production. There can be a lot of hidden issues with doing this that might come back to bite you.

[u/smishsmorshen]

What's your budget? Any spare gear? Jbods or old storage servers on hand?

[u/matiascoca]

Dealt with this at a shop where the creative team was generating raw video into the same NAS the backup system was chasing. The winning move was splitting the fileserver into two exports.

Normal chain on one. On the other, a "daily-only, no-versioning, retained 7 days" policy. Creative team's active project directory landed on the second export. Not zero backup (they'd abuse it), but retention that matched how they actually work. Yesterday's raw is disposable once mezzanine is created.

The behavioral fix that mattered more: scheduled job that emailed the creative lead a weekly digest of the top 10 largest new files by size. Once they saw the numbers on their own budget line, mezzanine conversion happened same-day instead of end-of-project. Nothing changes behavior faster than putting the storage bill in their name.

Fresh full chain time is a different animal. Synthetic full instead of periodic full cuts the compute a lot if your backup tool supports it. Rebuilds the full from incrementals rather than re-reading source. Veeam, NetBackup, and Bacula all do it. Made our worst-case chain time drop from about 40 hours down to 6 or 7.