Data Transfer

Ric Murray Jan 10, 2017

  1. I have 9T to transfer for an indie feature project. The data is on 3 USB3 drives and I am trying to transfer to a 4 Disk RAID5 box and the finder is telling me 3 Days to transfer! I am only getting 30-40 MBS transfer speed. I tried copying a single folder to a USB3 SSD and got 100+ MBS, not great, but at least I know the originals can be read at over 100MBS. I then tried transferring from the SSD (the SSD's are connected with hi speed USB controllers that I have tested to almost 500MBS) to my RAID, and got 200MBS. Pretty respectable. I then reconnect the originals to the RAID and still only get 40MBS. This is all on the same USB3 Bus. Tried it on both a Mac Pro (yes I know the USB bus sucks) and a Macbook Pro, same results. Can anyone explain why if I can read the disks at 100 plus, and my RAID is capable of writing 200+ I can only get 40MBS?
  2. Can you run a speed test with BlackMagic or AJA disk test and see the results?

    Are you plugging the USB one by one? Some ports are on the same bus and that can create a bottleneck sucking up all the bandwith

    Also use a checksum software like Silverstack or yoyotta to avoid data corruption
  3. 40 MB/s sounds like a USB2 controller bandwith limitation.
    Are you sure the disks are connected via USB3?

    I have an external SSD with more than 400MB/s read speeds.
    Regular cheap 5400 RPM USB drives can be much slower. I had external drives with speeds under 80MB/s but 40MB/s is pretty slow indeed.

    I have to say I don't like USB at all, because it is CPU dependent.
    Thunderbolt is always so much faster.
    Samuel Ouimet L. likes this.
  4. I will run some BM tests when the current transfer is over, but the source disks can read at 100+ and the RAID can write 200+.

    The RAID is plugged directly into a USB3 Port on the Mac Pro, the source disk into a USB hub. I used that path with the other disks and easily got 200+.

    I have Shotput Pro, but won't that just slow it down more?
  5. 40 MB/s sounds like a USB2 controller bandwith limitation.
    Are you sure the disks are connected via USB3?

    Yes same path with different disks yielded 200+

    I have an external SSD with more than 400MB/s read speeds.
    Regular cheap 5400 RPM USB drives can be much slower. I had external drives with speeds under 80MB/s but 40MB/s is pretty slow indeed.

    80MBS would be great! Would cut transfer time by half!

    I have to say I don't like USB at all, because it is CPU dependent.
    Thunderbolt is always so much faster.

    Agreed, but this RAID was bought before T-Bolt and has SATA, USB3, & FW800. It's the only thing I have that can hold 9T on one Volume.
  6. Doesn't what you are copying also affect the speed. Copying a feature film DPX sequence will be alot slower than copying a single QT of the same film because the drive is constantly start/stopping to copy each individual file.

    Could that be an issue here?
    jamie dickinson likes this.
  7. Thanks all, but I just ran a BM Disk test and yes indeedy the pocket USB drives only read about 48-50 MBS. Must be 5200 RPM cheapos. Looks like three days of watching the finder creep along. In 2017, who'd a thunk it.
  8. A fast interface doesn't help if the drives are shitty.
    Marc Wielage and Margus Voll like this.
  9. You write that your RAID id FW800, so why not use FW800 instead?

    I have some old USB, FW800 + SATA drives and they all have USB2 only!

    And be sure to check your HUB!!
    Most HUBS are just bad. Try it without.
  10. I just checked the speed of two of my externals:

    2TB Samsung EVO960 SSD via USB3:


    500GB El Cheapo HDD drive via USB3:

  11. Can you skip the Hub? Plug the source drive directly into the computer, just like you have the RAID plugged in?

    Are you copying from multiple source drives simultaneously? The way Finder calculates remaining time is a bit . . . not-accurate if you are copying multiple things at a time. First and foremost, copying from multiple sources simultaneously tends to eat up more overhead, so you're losing time compared to if you are sitting there right when one ends in order to queue up the next source. The Mac Pro only has one USB bus, remember, so you're total 5Gb/s speed is divided between all USB devices.

    Though even at that, in many cases, the bus speed is not the limitation these days, but rather the hard drive itself. The BEST spinning disk on the market gives around 220MB/s transfer speed, and the average is around 120MB/s. Most external drives are around that average, with cheaper ones dropping to around 70MB/s. USB3.0 maxes at 5Gb/s, which is around 640MB/s (b=bit, B=byte, so divide 'b' by 8 to get useful numbers). If the drive is not the limiting factor, reading from a source on USB and writing to a destination on the same USB bus, the theoretical max is around 320MB/s, but you'll usually get lower than that because of overhead and whatnot. Still, if you have 4 sources and 1 destination, all on the same bus, that could easily drop you to 30-40MB/s on each. But, if you limit it to one source at a time, it will probably take you far less overall time. Your best bet *might* be 2 sources at a time, but 1 is probably ideal.
    Andrea Bonomelli likes this.
  12. Thanks all. It's been 3 days and the data is on the RAID. As I said, 40MBS in 2017?, who'd a thunk it.
  13. OK, after all the time spent transferring data, the files aren't linking up in Premiere. It turns out that some of the old OWC RAIDs are not recognized by the Premiere Media Browser. Long story short, a wasted week, so I just ponied up for a 12T ThunderBay4. It tests out at 700+ MBS in and out configged as RAID 0. Don't worry I'm using the old OWC USB3 RAID5 as a nightly backup. Now my question is this...
    When doing a large transfer like this (almost 8 TB) is there any speed difference in having the source and target drives in a T-Bolt daisy chain or plugged into different outlets on a Mac Pro 6,1?
  14. I solve this in Resolve by generally conforming the session first from the client's drives, then copying only the used takes in Media Management. That way, I'm not copying all the files -- only the 4 or 5TB used in the session. In best-case scenarios, it might only be like 2 or 3TB.

    If this is not Resolve, there are media-management apps out there that will essentially do the same things from XML. I've seen this topic come up a few times from Premiere editors. I generally ask my clients to media-manage the material first, but about 90% of the time it doesn't happen and I wind up doing it myself. The good news is, when I do it, it's done right and I don't have to worry about it. I just set it up before I go to sleep and most of it's done by the next morning; the transfer rates for USB 3 and TB2 are somewhere around 1TB/4 hours.

    I think Apple has said there's no difference in daisy-chaining Thunderbolt 2 for drives when copying, but I don't trust them -- I just use two different ports.
  15. Trim option also reduces media size a lot.

    I have found myself working with client drives connected to my machine and the cache media and not copy at all. Specially with Arri Raw
    but this is for short form of jobs like adverts and such. After cache is on my drive all is golden and only needed media is cached.
  16. Ric, if you can detail out EXACTLY how you have everything plugged in, I can give you an idea of whether or not there's a better way. Daisy chaining is helpful, to a point. When data is transferred, it passes through the CPU. So if you have your computer, connected to your destination drive, and daisychained to your destination drive is your source drive, what's happening is that the data is going from your Source, through your Destination (just through the circuitry and Thunderbolt cable, not the media of the drive itself), through your CPU, then back to your Destination to finally be written.

    If you extend this out, and have 6 drives daisy chained together, with a bunch of stuff reading and writing, you could very quickly saturate your throughput and limit your speed.

    But again, I can only spot the issue if you tell me what drive is plugged into what port, what is daisy chained into what, and if there are any Hubs in between port and drive.
  17. I've tried to stop using the trim or media manage workflow - in the NLE or Resolve - only because it's so difficult to relink a new version of the edit to the Media Managed media. It can avoid problems if you copy what's used in the edit but don't trim or rename the clips. If you know for sure your edit is locked, that's fine but...
    Marc Wielage likes this.
  18. Ok then it seems that i 99% of the time work with locked edit. If you have original stuff around and new edl then jumping on new timeline with color trace should be a breeze but totally different need then. I just wanted t point out that some cases you can work without media on your drive.
    jamie dickinson likes this.
  19. Thanks all. I have set up the new Thunderbay 12TB as a RAID0 for speed. BM Disc test shows 700+MBS (of course w empty discs). I have handed off the drive to my director and he will do the transfer this time to get an exact mirror of his directory set-up and project file. The edit is NOT locked, so we will be working back and forth a bit in Premiere to get some sample scenes cut to a trailer and promo (we need to show some work for additional fund raising). We are a ways from final color work, and we are in an extreme low budget production, so it's more about whats' possible than best practices. I will attach to separate T-Bolt connectors on my trashcan. Thanks again for input!

Share This Page