Moving Large Amounts of Data

Ric Murray Sep 24, 2016

  1. I am doing some testing for very basic DIT duties on an feature length indie shoot, 11-12 days, in about 6 weeks. Mostly single camera (BM Production Camera) ProRes HQ, 4K. Recording to 256G and 480G SSD's. DP would like to offload both audio (24bit, 96K, 3-4 tracks from Zoom H6) and video by scene, then copy the days' work to secured storage at 2 locations each night.
    See images below for what I am suggesting. Ingest is on the left, a Convergent Design USB3 card reader that can read about 500MBS. Using Shot Put Pro 6 on a MacBook Pro, I am transferring simultaneously to (2) 1T SSD's, using the XXHash 64 Checksum verification. The 1T SSD's are in a OWC disc caddy via Thunderbolt. After ordering the OWC unit I was disappointed to find that it only contains 1 SATA controller for the 2 slots, resulting in a 250 MBS write speed combined when writing to both discs or when transferring between the two in the caddy. Is there a better way to write to two discs simultaneously that I have missed? I'm getting close to 1:1 copy time, that is, 30 minutes of footage takes about 30 minutes to transfer (getting 250-280MBS) which means the tech will have time to do the copy then review files briefly, be sure they open in QT 7 for example before we move company. Then the (2) 1T SSD's will go to two different locations to be backed up over night. On my end storage is to a RAID5 SmallTree 8T Server that gets backed up to a NAS (also Raid 5, 8T) every 24 hours. Files checked again before SSD's get formatted for the next day. If there are any corrupted files we could go back to shooting SSD's in the AM and off load again before format for the days work. Comments and suggestions welcomed.

    IMG_1894.JPG IMG_1895.JPG IMG_1896.JPG
     
  2. Black magic SSD's are often long to copy, do to a lack of good card reader. Well in my opinion.
    If there is only one SATA bus un your backup "toaster", I would recommand to move a faster storage solution. Two drives with each one a Thunderbolt connector would be faster. There is no way you can run a 500mb/s two bakcup with that setup, specially coming from there Black Magic SSD's Card.
     
  3. Are BM SSD's slower to read? I was testing with my Convergent Design SSD's and get a steady 500MBS out with the card reader pictured. I get 500 MBS from the CD SSD to the Samsung 1T if doing a single copy. It's when I add another drive to the OWC and copy to both simultaneously that I drop to 250MBS (which is not terrible for this workflow).

    I looked for an SSD T-Bolt drive with additional T-Bolt connection for daisy chaining but could find none. That would mean going to a T-Bolt Hub and connecting each receiving T-Bolt SSD individually, or copying to T-Bolt RAID 0 drives with HDD's, no?
     
  4. I dont think that their SSD are slower than others, its just that a USB 3 reader is often not enough. If the camera runs a lot on a day, downloading 200-300Gb of Data with a USB 3 reader is often really long.

    I guess it's normal that the 500mb/s is split in half with only one Sata Bus on the toaster. If you have two seperate drives with each is connector, you should be able to get a better speed transfer. But hey, 250bs/s can be ok for a short shoot like this, given the fact that you have only one camera to offload.
     
  5. I don't understand the question.

    A 20Gbps Thunderbolt dock splits the bandwidth among the USB 3.0 connections, so theoretically you are able to attach 4 SSDs via USB 3.0 like you did with the 256G drive pictured above. All 4 SSDs would transfer at full speed at around 500MB/s each.
     
  6. I actually did testing with a bunch of USB card readers like the Atmos readers that had blue USB3 connectors, and some other SATA cabling again- supposedly USB3, and only got 250-280 MBS read. After conversation with Mitch Gross at Convergent Design I was advised that just because a reader has a blue USB3 cable doesn't mean it has a controller inside that can do a full speed read of an SSD at 500MBS. He recommended the CD brand reader, a deal at $53, and moves data consistently over 500MBS.
    https://www.bhphotovideo.com/c/prod...gent_design_150_10060_100_usb_3_0_to_ssd.html
     
  7. The reason is some USB 3 to SATA adapters use SATA 3Gbps bridge, which causes speed bottleneck. SSDs are usually SATA 6Gbps, the CD brand adapter uses SATA 6Gbps to USB 3.0 interface.
     
  8. Sorry I was unclear. Yes a T_Bolt Dock with 2-3 USB3 buses, if discreet, would work. It looks like a lot of these devices take a single USB3 bus and split it over several outlets. I will do a little research in that area.
     
  9. A hub splits the 5Gbps bandwidth between available slots. A dock is more likely to offer you independent 5Gbps bandwidth.
     
  10. we have used this setup with great effect

    this

    https://eshop.macsales.com/shop/Thunderbolt/Dock/OWC/Thunderbolt2-Dock/

    plugged into TB2 port on the mac book pro

    for a SSD shoot we use these or the older version ....so a BMPC4k/BMC

    https://www.amazon.co.uk/Inateck-Drive-Enclosure-Support-Optimized/dp/B00KW4T69A

    all the SSD,s are loading into these before the shoot and all marked with a big green bit of gaffer tape

    BTW the DIT keeps all of these and hands them out as required

    ok so the moment it come out the camera in goes back into its enclosure green gaffer comes off ( anything this with out the green gaffer tape means it,s been written on or don,t use it)

    we have a 2 x TB2 12TB raid 5 units for offload, daisy chained to the thunderbolt dock

    and we rent a license of silver stack each shoot as it has cascade copy which mean you can do a fast dump of the card and then it will do the second copy to the other raid at a slower pace

    normally we can offload 400 to 500 gb from the card in about 40 mins

    we do the same set up with REDS but just stick the gaffer tape over the front of the mini mags and use the red Usb 3 reader

    same thing with alexa just using the CF card reader and green tape

    this setup works well as silver stack allow you to view the footage even have play with a grade and export CDL /luts for later use
     
  11. Thanks for the outline above. SilverStack looks like a great tool. I might look more into that.

    The fastest read/write from the Samsung EVO 1T SSD's seems to be about 500MBS, so I don't see what advantage I would get from more elaborate dock arrangements and 10 or 20 GBS Thunderbolt bus. I think I will order in the USB3 Hub and 2 CD USB3 cables and see what I get. I will report back.
     

Share This Page