Hi, I’ve been syncing my Amazon Drive to move away from it (using an EC2 Windows VM).
Since it’s a lot of files, and 5TB+ of data, I’ve noticed a worrying issue with ODrive itself:
When looking at the Amazon CloudDrive web client, a folder has X files, but on ODrive it has less files than that (sometimes a lot less).
The things that worried me:
There were not .cloud files in that folder at all.
ODrive showed that folder as synced successfully.
Only when I open that folder in Windows explorer locally, then ODrive started syncing and download the missing files.
I found several cases of this issue.
So I’ve done “Force refresh during sync” just in case - will that 100% guarantee that nothing will be missed?
Hi @budowski,
Performing a “Force refresh during sync” should take care of it (so should the periodic background remote scan), but I worry about Amazon Drive’s aggressive throttling and general quirkiness. My guess is that is how it got into this situation to begin with.
There are a few optimizations that are being done that may be able to be improved. There is already a refresh that is scheduled for any folders that have not been completely downloaded, like the ones you are describing. That scheduled refresh is meant to be a “clean-up” of the folder, to ensure everything is downloaded. If that refresh fails, then the periodic background remote scan, or direct folder navigation, is meant to pick-up any stragglers. This all works really well in general use cases, but may need more tweaking for these types of cases, where there is a massive scale of data and the remote storage tends to be uncooperative for large operations (like Amazon Drive).
I’ll see what can be done to improve odrive behavior in this type of situation.
Thanks for the detailed answer -
Is there a way to get total file count from Amazon Drive? Whether through ODrive / APIs / Amazon Drive web client?
This way at least I can compare downloaded files (since Windows Explorer can show me that for the entire folder) vs. what CloudDrive reports it has
BTW - what do you think about the following solution as well:
After supposedly finishing downloading all Amazon files → I’ll move them all (not copy) to Dropbox using ODrive → then empty trash with ODrive once the move is finished. If any files were not downloaded from Amazon originally → those will still show up in Amazon Drive, right? (since they were not marked as moved by ODrive)
Amazon Drive doesn’t provide that information via the API, unfortunately. Their storage reporting is pretty generalized, too, as you’ve likely seen.
If you move folders then the entire folder will be deleted in Amazon, so this won’t work.
I am hoping to have a new release for you to try soon that is more persistent in how it deals with rate limiting during bulk download. It will also have better logging so that you can tell when a folder did not download entirely. More info to come, once it is ready…
Hey @budowski,
Shooting for this week. It won’t be an official release, as that will take longer to go through the whole QA release process, so it will be a one-off for you to try until we can get it through the official cycle.
