Hi there,
I’ve been trying to upload about 1.5TB of files to Amazon Drive via odrive. I even purchased the pro plan.
I let my iMac run 24/7 so odrive can upload constantly. My upload speed is usually quite good - it only takes a couple of minutes to upload over 1 GB via wetransfer, dropbox etc…
I checked Amazon Drive for the last two days and each day odrive uploaded only 200MB per day!! I already set the upload speed to unlimited etc… is this normal? That would be quite unusable…
Miri
Ah and when I restart my iMac I can access the odrive menu in my top bar, but after the first click it’s completely unresponsive… I know there are a ton of files in the queue, but I never had these issues with any other provider 
Hi @bambi14,
1.5TB is a lot of data to upload at once, so it is likely that all of that activity, scanning, and queuing can make the odrive menu unresponsive. Chunking up the bulk import can help to reduce overhead and easy the data import effort.
As for speed, Amazon is fairly notorious for large file upload issues. It could be that the larger files are recycling because of Amazon errors. If the files are very small, it could also be that the speed seems slow because all of the current sync activity is with very numerous, but very small files.
Do you know approximately how many files and folders are in that 1.5TB?
Well I compared uploading stuff via browser, good sync, the native amazon app and odrive. I get good upload speed results with the first three options… so I guess the speed problem is not on Amazon’s side.
The 1,5 TB contain a lot of folders and I guess maybe 500k files. All mixed from very small files up to a bit larger video files… maybe 4GB.
I uploaded these to Dropbox about a year ago to a business account, and it took about a week.
odrive didn’t even upload 100MB last night! I’m sorry but for a 99$ software, this is unacceptable. At this rate my files won’t be online until next year. I’ve been reading through other support threads and speed seems to be an issue since last year with a lot of answers like “biggest issue we’re working hard on to improve”. Well… if there isn’t a way to improve that, I guess I have to look for another solution.
Just to give you an example. Look at the time in the top bar and at the upload progress in % … the jpg has probably 3MB and the CR2 files maybe 30MB…
oh and the ones who aren’t in the upload queue anymore in the second pic aren’t uploaded but back in the “waiting” list…
Hi @bambi14,
Can you send a diagnostic from the odrive menu so I can take a closer look? If files are going back into waiting then that indicates that they are failing when trying to upload. If there are a lot of failures that would definitely slow things down.
odrive will generally prioritize the smallest files first, then move to the larger ones, so I would expect performance to be lower, but the speeds you are seeing sound unexpectedly low.
odrive pushes a single file up as fast as Amazon will allow via their API interface. The trick is in concurrency, where we upload more than one file at a time to increase speed. odrive’s concurrency is limited for files larger than 100MB, and that is generally where folks complain about speed, because Amazon’s single-file upload speed is pretty low compared to other providers.
Hi,
I just sent the diagnostics.
Last night I uploaded 2 files directly to Amazon Drive (while odrive was uploading simultaneously) via browser. Both files were around 2GB. Took about 4hrs maybe – still slow but lighting speed compared to what odrive is doing atm.
Didn’t upload more than 500MB over the last few days…
Hi @bambi14,
Thanks for sending over the diagnostic.
Taking a look, odrive is extremely busy. It already has ~6000 folders in scope, and is uploading many files. The files appear to be smaller, which will slow things down. There will always be a very big speed difference between uploading a single 1GB file vs 100,000 10KB files or 1,000,000 1KB files, for example.
I would still recommend breaking up the dataset, if possible, so that odrive doesn’t have to deal with the full data set, while it is trying to upload everything. You can also try unsyncing folders that have finished uploading to reduce the scope and overhead.
Things do look like they are moving, in general, but there is a lot to move. Amazon is also restricting the flow by rate limiting the uploads (Amazon will restrict the amount of API calls that can be made), which can also slow things down quite a bit.
I can’t stop any folders from uploading (and thus break up the dataset), as the odrive interface when right clicking a folder doesn’t show up anymore.
I’ve been constantly uploading files for weeks now. And my best guess is, it’s a max of 40-50GB that were uploaded through odrive. Just for a comparison: I upload pictures through the amazon web interface and it doesn’t take forever… a couple of GB (which consist of hundreds of small files) are uploaded in under an hour.
It’s not like the upload is “kind of slow” and it takes maybe twice or three times as long as it should.
It just doesn’t work.
The odrive interface / menu is not accessible at all and it’s pretty much uploading nothing.
There is no progress (if you don’t see a couple of MB per day as a process…).
Hi @bambi14,
You would need to move folders out of odrive’s scope to break up the dataset. Once the data is within odrive’s view, it will try to scan, index, and sync.
To break it up just requires that you move a folder out from odrive’s view, then move it back when you are ready to process it. Try exiting odrive before you do this, then starting it again after you’ve moved the folders. If the odrive menu is unresponsive you may need to kill the app from the activity monitor.
Yeah I kind of don’t see me moving around 1.xTB of files, especially when a lot of these are used in After Effects projects, InDesign or light room and when I work on these projects, I’d have to reconnect all the files over and over again.
I guess I’ll just upload everything through the Amazon Web interface and hope that someday odrive will fix that, cause like this it’s not worth the 99$ per year.
Hi @bambi14,
At this point odrive might not be the best solution for your use case. We have done a lot of work for bulk processing on the next generation of odrive, so there will be some big improvements, but processing massive amounts of data is always going to be a tough challenge.
If you are going to use another means, I would suggest trying Amazon’s desktop client instead of their web browser. The web browser is not going to be great for importing the amount of data you have. Their desktop client at least has the ability to resume uploads, which, unfortunately, they have still not released as an API that we can use.