Hi, I’ve been syncing my Amazon Drive to move away from it (using an EC2 Windows VM).
Since it’s a lot of files, and 5TB+ of data, I’ve noticed a worrying issue with ODrive itself:
When looking at the Amazon CloudDrive web client, a folder has X files, but on ODrive it has less files than that (sometimes a lot less).
The things that worried me:
- There were not .cloud files in that folder at all.
- ODrive showed that folder as synced successfully.
Only when I open that folder in Windows explorer locally, then ODrive started syncing and download the missing files.
I found several cases of this issue.
So I’ve done “Force refresh during sync” just in case - will that 100% guarantee that nothing will be missed?
Performing a “Force refresh during sync” should take care of it (so should the periodic background remote scan), but I worry about Amazon Drive’s aggressive throttling and general quirkiness. My guess is that is how it got into this situation to begin with.
There are a few optimizations that are being done that may be able to be improved. There is already a refresh that is scheduled for any folders that have not been completely downloaded, like the ones you are describing. That scheduled refresh is meant to be a “clean-up” of the folder, to ensure everything is downloaded. If that refresh fails, then the periodic background remote scan, or direct folder navigation, is meant to pick-up any stragglers. This all works really well in general use cases, but may need more tweaking for these types of cases, where there is a massive scale of data and the remote storage tends to be uncooperative for large operations (like Amazon Drive).
I’ll see what can be done to improve odrive behavior in this type of situation.
Thanks for the detailed answer -
Is there a way to get total file count from Amazon Drive? Whether through ODrive / APIs / Amazon Drive web client?
This way at least I can compare downloaded files (since Windows Explorer can show me that for the entire folder) vs. what CloudDrive reports it has
BTW - what do you think about the following solution as well:
After supposedly finishing downloading all Amazon files → I’ll move them all (not copy) to Dropbox using ODrive → then empty trash with ODrive once the move is finished. If any files were not downloaded from Amazon originally → those will still show up in Amazon Drive, right? (since they were not marked as moved by ODrive)
Amazon Drive doesn’t provide that information via the API, unfortunately. Their storage reporting is pretty generalized, too, as you’ve likely seen.
If you move folders then the entire folder will be deleted in Amazon, so this won’t work.
I am hoping to have a new release for you to try soon that is more persistent in how it deals with rate limiting during bulk download. It will also have better logging so that you can tell when a folder did not download entirely. More info to come, once it is ready…
OK, thank you so much! When do you think this new version will come out, more or less?
Shooting for this week. It won’t be an official release, as that will take longer to go through the whole QA release process, so it will be a one-off for you to try until we can get it through the official cycle.
Here is the new version.
- Exit odrive
- Download the new version from here: odrive
- Install to update
If you do another right-click->sync, including subfolders and forcing refresh, it should ensure that you get everything down.
The sync activity log will provide information on everything, including anything that runs into an issue. It will likely take a long time to finish…
Thanks so much!!!
I downloaded it and started syncing
I just wanted to check-in and see how things are going. Did you get everything migrated over?
Thanks so much for your help - managed to download all Amazon CloudDrive files, now in the process of moving to Dropbox, almost done