Put this together after going down a rabbit hole on why cloud mount tools feel terrible for editing vs. LucidLink style streaming. Here is an open implementation of the pattern (HuskHoard), so full disclosure I contribute to the project— but the first two-thirds is just how this class of system actually works, independent of any specific tool. Why You Can't Just Mount an S3 Bucket and Start Editing
Streaming edit access to S3 Cloud — stubbing, byte-range fetch, and why directory listings are the hard part
- Thread starter Joichiro Mitaka
- Start date
Jason Myres
Administrator
Nice article. Does EBS (or FSx) have a place in all this? In my mind S3 is a large, flexible nearline, but EBS/ FSx would do the real work.
Thanks for reading! Historically, yes, S3 is your cheap, deep nearline archive, and you have to incur the time and cost of moving data to an active tier like EBS, FSx, or a local NAS/SAN to do the real work.
Today, people generally avoid the S3 bottleneck in one of two ways:
The Download approach: Provision an EBS/FSx volume or a local NAS, copy the entire 50GB ProRes file out of S3, do the work, and sync it back (incurring large egress fees and waiting on transfers).
The Cloud Only Workstation approach: Avoid downloading entirely by renting a high end cloud computer (EC2 instance) via Teradici/NICE DCV that sits in the same data center as an FSx volume. You pay zero egress, but you pay an hourly tax for renting cloud compute and hot FSx storage.
The entire goal of HuskHoard is to challenge whether that data movement, or that expensive cloud lock in, s actually necessary.
HuskHoard is designed to let you do the real work on your existing local machines, directly against your S3 (or MinIO/TrueNAS) backend, without needing a dedicated block/file storage tier in the middle. Here is how it replaces the need for an EBS/FSx tier:
fanotify turns your local drive into a dynamic block cache
Instead of provisioning a whole EBS volume, HuskHoard uses your machine's existing local storage as a smart cache via Linux’s fanotify API. When your NLE scrubs a timeline, HuskHoard intercepts the read and fetches only the requested byte ranges from S3. Once those bytes hit your local drive, any subsequent reads are served directly by the kernel's normal page cache. You get EBS like local SSD speeds for the parts of the file you're actively working on, while the 90% of the file you aren't using stays safely (and cheaply) in S3.
Local SQLite defeats S3 APi latency
The other reason people rely on FSx is that S3 is terrible at filesystem operations. If an NLE tries to scan a directory with thousands of media files, S3 list objects calls will choke it. HuskHoard solves this by decoupling the metadata. It uses a local SQLite catalog to mirror the S3 namespace. Your application gets instant ls and stat() responses locally, making your S3 bucket feel exactly like a mounted FSx drive, but without the hourly AWS bill for a managed filesystem.
LTO and SMR integration
This architecture solves the same problem for local cold storage. If you have an LTO tape shelf, you'd normally have to restore the whole tape to a RAID array before you could view the footage. HuskHoard applies the exact same stubbing and byte range fetching logic to tape. Its native SCSI driver and local HTTP gateway (StreamGate) allow you to plug an LTO tape into your system and scrub through a multi terabyte file in Plex, mpv, or an NLE within seconds, completely skipping the need to stage it to a disk array first.
The weaknesses and trade offs
To be completely transparent, this architecture isn't magic, and it doesn't defeat physics. There are two main trade offs:
First Byte Latency: The very first time you scrub to a completely un cached section of a file, you still have to wait for the network (S3) or the physical seek time (Tape) to deliver those bytes. HuskHoard makes subsequent reads instant, but that first touch will always be bound by your internet connection or tape mechanics.
Write Operations: Because S3 and Tape strictly forbid random in place overwrites, HuskHoard has to enforce log structured, sequential writes. It's good for non destructive editing (where source files are read only and new renders are written as continuous files), but it is not a drop in replacement for a multi user SAN where dozens of people are constantly modifying the same live databases.
For the vast majority of media workflows, though, partial hydration and a local metadata catalog give you the performance of a local block drive while leaving the heavy data exactly where it is.
Today, people generally avoid the S3 bottleneck in one of two ways:
The Download approach: Provision an EBS/FSx volume or a local NAS, copy the entire 50GB ProRes file out of S3, do the work, and sync it back (incurring large egress fees and waiting on transfers).
The Cloud Only Workstation approach: Avoid downloading entirely by renting a high end cloud computer (EC2 instance) via Teradici/NICE DCV that sits in the same data center as an FSx volume. You pay zero egress, but you pay an hourly tax for renting cloud compute and hot FSx storage.
The entire goal of HuskHoard is to challenge whether that data movement, or that expensive cloud lock in, s actually necessary.
HuskHoard is designed to let you do the real work on your existing local machines, directly against your S3 (or MinIO/TrueNAS) backend, without needing a dedicated block/file storage tier in the middle. Here is how it replaces the need for an EBS/FSx tier:
fanotify turns your local drive into a dynamic block cache
Instead of provisioning a whole EBS volume, HuskHoard uses your machine's existing local storage as a smart cache via Linux’s fanotify API. When your NLE scrubs a timeline, HuskHoard intercepts the read and fetches only the requested byte ranges from S3. Once those bytes hit your local drive, any subsequent reads are served directly by the kernel's normal page cache. You get EBS like local SSD speeds for the parts of the file you're actively working on, while the 90% of the file you aren't using stays safely (and cheaply) in S3.
Local SQLite defeats S3 APi latency
The other reason people rely on FSx is that S3 is terrible at filesystem operations. If an NLE tries to scan a directory with thousands of media files, S3 list objects calls will choke it. HuskHoard solves this by decoupling the metadata. It uses a local SQLite catalog to mirror the S3 namespace. Your application gets instant ls and stat() responses locally, making your S3 bucket feel exactly like a mounted FSx drive, but without the hourly AWS bill for a managed filesystem.
LTO and SMR integration
This architecture solves the same problem for local cold storage. If you have an LTO tape shelf, you'd normally have to restore the whole tape to a RAID array before you could view the footage. HuskHoard applies the exact same stubbing and byte range fetching logic to tape. Its native SCSI driver and local HTTP gateway (StreamGate) allow you to plug an LTO tape into your system and scrub through a multi terabyte file in Plex, mpv, or an NLE within seconds, completely skipping the need to stage it to a disk array first.
The weaknesses and trade offs
To be completely transparent, this architecture isn't magic, and it doesn't defeat physics. There are two main trade offs:
First Byte Latency: The very first time you scrub to a completely un cached section of a file, you still have to wait for the network (S3) or the physical seek time (Tape) to deliver those bytes. HuskHoard makes subsequent reads instant, but that first touch will always be bound by your internet connection or tape mechanics.
Write Operations: Because S3 and Tape strictly forbid random in place overwrites, HuskHoard has to enforce log structured, sequential writes. It's good for non destructive editing (where source files are read only and new renders are written as continuous files), but it is not a drop in replacement for a multi user SAN where dozens of people are constantly modifying the same live databases.
For the vast majority of media workflows, though, partial hydration and a local metadata catalog give you the performance of a local block drive while leaving the heavy data exactly where it is.
Jason Myres
Administrator
Where does HuskHoard live so to speak?
When I read this...."It acts like an Enterprise Tape Library, but built for the modern hybrid cloud user." .... it sounds like a server, but when I read your description ("on your existing local machines, directly against your S3") it sounds like a client install, which then mounts the required S3 volume directly, and allows S3 to simulate an online volume. Any thoughts on where it actually sits would be helpful.
Also, has anyone completed real shows with it yet?
When I read this...."It acts like an Enterprise Tape Library, but built for the modern hybrid cloud user." .... it sounds like a server, but when I read your description ("on your existing local machines, directly against your S3") it sounds like a client install, which then mounts the required S3 volume directly, and allows S3 to simulate an online volume. Any thoughts on where it actually sits would be helpful.
Also, has anyone completed real shows with it yet?
Technically, HuskHoard is a service daemon that runs on Linux. Two possible architectures
The server way. It lives on a dedicated storage gateway or your head unit, managing your disk array, LTO library and S3 buckets, presenting them to the network. You set it up as an SMB file share where an admin sets the rules for the server and the users just see a normal file tree.
The self host way. It can run locally on your Linux workstation or via a VM to manage your local NVMe usb attached storage and S3 directly.
How it handles S3 , It’s not just a mount.
You’re right to suspect it’s more than a client install. Unlike simple FUSE mounts, like rclone mount, which often struggle with NLE directory listings and header reading, HuskHoard is a data tiering archive.
It keeps the stubs metadata and file structure on your fast local NVMe. To Resolve or Premiere, the file looks like it's 500GB and sitting right there. When you hit play, HuskHoard uses StreamGate to perform those byte range fetches from S3 or LTO in real time. It only pulls the bits the NLE is actually asking for. It effectively turns S3 into a nearline tier that feels like an online volume.
Has anyone completed real shows with this... I don't know, we have around 500 downloads per month at this point, but only a handful that have reached out for assistance or made suggestions for improvements. I would say HuskHoard is currently in the hands of early adopters at this stage, the project has been 3 years of development and launched 6 months ago.We are seeing it used to manage archives across disk arrays, LTO and Cloud. it’s open source and high performance, but it’s for the user who wants total control over their data lifecycle rather than a black box subscription service.
If you’re comfortable with a Linux environment, it’s designed to stop the Online/offline media dance by keeping everything visible, regardless of whether it's on a tape in the basement or a bucket in AWS. you can set it up on any free machine you have available to run a test environment, it will adress any attached storage or cloud buckets as archive volumes, you set the archive rules, SMB share it to your network and you have your own cheap and deep archive. if you want to understand the underlying process the blog is a good place to start https://huskhoard.com/blog.html
The server way. It lives on a dedicated storage gateway or your head unit, managing your disk array, LTO library and S3 buckets, presenting them to the network. You set it up as an SMB file share where an admin sets the rules for the server and the users just see a normal file tree.
The self host way. It can run locally on your Linux workstation or via a VM to manage your local NVMe usb attached storage and S3 directly.
How it handles S3 , It’s not just a mount.
You’re right to suspect it’s more than a client install. Unlike simple FUSE mounts, like rclone mount, which often struggle with NLE directory listings and header reading, HuskHoard is a data tiering archive.
It keeps the stubs metadata and file structure on your fast local NVMe. To Resolve or Premiere, the file looks like it's 500GB and sitting right there. When you hit play, HuskHoard uses StreamGate to perform those byte range fetches from S3 or LTO in real time. It only pulls the bits the NLE is actually asking for. It effectively turns S3 into a nearline tier that feels like an online volume.
Has anyone completed real shows with this... I don't know, we have around 500 downloads per month at this point, but only a handful that have reached out for assistance or made suggestions for improvements. I would say HuskHoard is currently in the hands of early adopters at this stage, the project has been 3 years of development and launched 6 months ago.We are seeing it used to manage archives across disk arrays, LTO and Cloud. it’s open source and high performance, but it’s for the user who wants total control over their data lifecycle rather than a black box subscription service.
If you’re comfortable with a Linux environment, it’s designed to stop the Online/offline media dance by keeping everything visible, regardless of whether it's on a tape in the basement or a bucket in AWS. you can set it up on any free machine you have available to run a test environment, it will adress any attached storage or cloud buckets as archive volumes, you set the archive rules, SMB share it to your network and you have your own cheap and deep archive. if you want to understand the underlying process the blog is a good place to start https://huskhoard.com/blog.html
Last edited:
Jason Myres
Administrator
Thanks for the detailed description. It clarifies a lot. One theme that keeps recurring as I read about cloud-based post production is there are a bunch of ways to approach it, particularly where you divide cloud and on-prem equipment. It sounds like HuskHoard is ideal for those who want cloud storage, but retain their local workstations. The other option might be storage and workstations in the cloud, with local KVM and reference monitoring. Those fees could really add up though.
Have you seen any patterns in terms of which model (all cloud vs hybrid) users are finding the most appealing for bandwith heavy use cases like post-production?
Have you seen any patterns in terms of which model (all cloud vs hybrid) users are finding the most appealing for bandwith heavy use cases like post-production?
There are good cases for using the cloud and also good cases for on prem. What we have seen over the last 10+ years is an everything must be in the cloud mentality, and the bill arrives, and it is not a small bill. The pendulum just swung a little too far to the cloud side and it is starting to swing back to on prem. Hybrid configurations are what is driving most of interest we are seeing in multiple industries.
Post production for big studios will likely remain in the cloud for the near term, they have control of their entire eco system and can secure it and they don't pay retail cloud prices. For smaller shops and independents hybrid setups, I believe they will turn to proxies and byte range requests on full size originals so they can keep the downloads small, fast and use hardware they already own rather than paying to rent cloud hardware.
With the adoption of 8K cameras, something has to change because the economics of cloud start to break at that level for everyone but large studios. This hardware switch is giving everyone an opportunity to look at their existing workflows. There will be people who change their workflow and make money in this environment and those that make enough to pay their cloud bill and pay their rent.
Post production for big studios will likely remain in the cloud for the near term, they have control of their entire eco system and can secure it and they don't pay retail cloud prices. For smaller shops and independents hybrid setups, I believe they will turn to proxies and byte range requests on full size originals so they can keep the downloads small, fast and use hardware they already own rather than paying to rent cloud hardware.
With the adoption of 8K cameras, something has to change because the economics of cloud start to break at that level for everyone but large studios. This hardware switch is giving everyone an opportunity to look at their existing workflows. There will be people who change their workflow and make money in this environment and those that make enough to pay their cloud bill and pay their rent.
Jason Myres
Administrator
The bill, along with a lack of tools to handle post in the cloud has been the barrier up until recently. There are a few things changing the picture....
1) Hardware is so expensive now that the ROI on compute and storage purchases is extremely challenging even for very large facilities.
2) Amazon has created new EC2 instances that are actually comparable to workstations you might deploy in a large post facility.
3) (The big one) Amazon (MGM) and Netflix are starting to offer cloud production resources in packages to their own shows as an alternative to budgeting for traditional post vendors.
The capital expense of traditional post has risen very quickly due to the AI hardware boom, and tech facing studios like Amazon and Netflix are filling that void with "free" cloud resources as an incentive for their own productions using hardware those studios have already paid for. They are essentially double-ending their own deals now in what could be called the virtual version of bringing post "in-house." The difference is that they can still take advantage of external talent (editors, VFX artists, colorists, etc) through remote access while spooling up whatever "hardware" they need for each show virtually.
"They are essentially double-ending their own deals now in what could be called the virtual version of bringing post 'in-house.'"
I think you have a good understanding of what is going on here, and I think it's specific to studios that already own the infrastructure rather than a sign of where cloud pricing broadly is headed. My honest expectation is the market sorts this out over time. As more hybrid, byte range, stub based architectures prove out (and HuskHoard is just one of several taking this approach), cloud vendors feel the pressure and respond with better price performance rather than watch workloads walk. We're already seeing early signs of it, Fsx shipped an Intelligent Tiering class specifically to compete on cost, and S3 Express One Zone exists because Standard wasn't fast enough for some workloads. Competition tends to be a pretty effective forcing function.
Worth keeping in mind though, cloud's business model is built around data capture. Get your data to live there, and it stays convenient right up until you need to move it or work with it in real time. That's by design, not an oversight, so I wouldn't expect it to soften as fast as the "compute is comparable now" argument might suggest.
Some real numbers on current AWS pricing, S3 Standard roughly $23-24/TB/month. The momen that same data needs to sit on an active tier like Fsx for Lustre so an NLE can scrub it, you're at $0.145-0.30/GB month depending on throughput, 6 to 13x the S3 rate. Pulling it out entirely costs more still, egress is about $0.09/GB for the first tier, so a single 1TB Prores master costs roughly $90 to download before anyone touches a color wheel. Compare that to a 14-18TB enterprise drive on your own shelf right now, running $9-15/TB as a one-time buy rather than a monthly rent, and you can see why the math gets interesting once you're managing real archive volumes of 8K raw.
I think you have a good understanding of what is going on here, and I think it's specific to studios that already own the infrastructure rather than a sign of where cloud pricing broadly is headed. My honest expectation is the market sorts this out over time. As more hybrid, byte range, stub based architectures prove out (and HuskHoard is just one of several taking this approach), cloud vendors feel the pressure and respond with better price performance rather than watch workloads walk. We're already seeing early signs of it, Fsx shipped an Intelligent Tiering class specifically to compete on cost, and S3 Express One Zone exists because Standard wasn't fast enough for some workloads. Competition tends to be a pretty effective forcing function.
Worth keeping in mind though, cloud's business model is built around data capture. Get your data to live there, and it stays convenient right up until you need to move it or work with it in real time. That's by design, not an oversight, so I wouldn't expect it to soften as fast as the "compute is comparable now" argument might suggest.
Some real numbers on current AWS pricing, S3 Standard roughly $23-24/TB/month. The momen that same data needs to sit on an active tier like Fsx for Lustre so an NLE can scrub it, you're at $0.145-0.30/GB month depending on throughput, 6 to 13x the S3 rate. Pulling it out entirely costs more still, egress is about $0.09/GB for the first tier, so a single 1TB Prores master costs roughly $90 to download before anyone touches a color wheel. Compare that to a 14-18TB enterprise drive on your own shelf right now, running $9-15/TB as a one-time buy rather than a monthly rent, and you can see why the math gets interesting once you're managing real archive volumes of 8K raw.
