I have one more question. I went in a couple days ago and unsynced several different folders on my drive to free up space on my drive. I now have a large amount of items sitting in the odrive trash, when I attempt to “Empty trash and sync all deletes” I am getting the below error… and have been for two days.
additionally… in my attempt to unsync some items I unsynced a folder that I actually need back. When I right click and attempt to sync the folder back including all the subfolders as you described to me above. I am getting the following error message. Can you tell me when this limit will be up and when I will be able to get this folder back??
Hi @mandyhart80,
My apologies for the trouble. Very recently Amazon has been responding back with these rate limit errors. I am trying to get in touch with them to see what is going on (I posted about it here last night: Amazon Drive/Photos Rate Limiting).
When you empty the trash, odrive will send the delete commands to Amazon. In this case they are responding with a rate limit error. The same applies when trying to download a folder. Can you try accessing the data you need from the Amazon web client? https://www.amazon.com/clouddrive
You mentioned unsyncing and trash above and I want to make sure you are not deleting items to unsync them. When you unsync, the items should not appear in the trash. Instead, the unsync action will turn your “real” files and folders back into placeholder files. I just want to make sure that you are taking the appropriate actions for unsync.
I’m using Amazon Photos/Drive provisioned from Amazon Japan, and I experience the sync to be quite slow.
I’m trying to sync my directory structure (sync all/download nothing) to a new internal SDD, and watching main.log shows a pattern like below:
Expand 3 cloudf files
2 minutes wait
Expand 3 cloudf files
2 minutes wait
…
Of course, there are no log entries stating that processing is waiting for 2 minutes, but watching the log timestamps this pattern appears to be the norm.
I also get lots of errors on syncing deletes like below: 03 Jan 12:43:02PM ERROR Failed Delete Folder (Local to Remote) for C:\Users\xxxxxx\odrive\Amazon Cloud Drive\Backup\xxxxxxxx\yyyyyyyy\ Error: code CD_RATE_LIMIT_EXCEEDED - {"message":"Rate exceeded"}
I’m currently on v6769 according to the Windows tray pop-up menu.
Is this rate normal? And, if not, is there some configuration I could try to make it faster and reduce delete sync errors?
Hi @belegring,
The issue in this thread was something Amazon had introduced that made the integration basically unusable. What you are describing sounds like “normal” throttling, although it does sound aggressive.
There isn’t anything we can do about rate-limits except back-off and retry the operation, which it sounds like you are seeing with the delays between expands. Our back-off is potentially too “careful”, however, which was a reaction to the previous incident, so we can probably stand to reduce the wait time a bit.
The trash routine for Amazon is also very intensive because of the way they need deletes processed. For example, you can’t just send a delete request for a folder to them. You need to delete every individual item inside a folder and then delete the folder. If you have lots and lots of files to delete that could be heavily contributing to the rate limiting. I would suggest only performing the recursive folder expand and then try emptying the trash after that.
Can you send a diagnostic from the odrive tray menu so I can take a closer look? I will see if we can tweak our rate-limit response and provide a test build for you in the next couple days.
I’ve sent the diagnostics report now.
Also, to provide some more background information; I’m not trying to delete folders with tons of files in them, but I am trying to delete tons of empty folders (mostly temporary folders created by other applications that were unfortunantely backed up by earlier versions of the Amazon sync client).
My initial tests looks promising. It still “hangs” every few cloudf files, but the waits appear to have been substansially reduced (from my visual inspection, maybe 20s~30s at most).
I haven’t tried simultaneous deletes yet, but the reason I did that in the first place was that the recursive expansion was slow. I was thinking that reducing the directory hierarchy might help the synchronization performance.
When I read the original threads on this issue, I did have a suspicion that using Amazon Japan might be why the problem appeared to linger for me.
I don’t know about the underlying infrastructure, but Amazon Japan and Amazon US/UK/Germany/etc. accounts are separated in other areas. I can for instance not login with my Amazon US account on Amazon Japan or vice versa. I can also not subscribe to Amazon Drive from my Amazon US account (although I can use the original free storage).
I’ll post an update when I’ve tried delete as well to let you know whether the limit rate error messages are also reduced.
Hi @belegring,
Yeah, I’ve seen via user feedback that different regions tend to behave differently, and seem to have different limits.
I updated the shared link with an updated version that will populate an amazondrive.log file in C:\Users\[your user name]/.odrive/log with throttling log entries, so we can get a better idea of what is happening.
Hi @belegring,
Can you send the diagnostic and the amazondrive.log file? Given what you’ve been seeing with the high level of rate limiting, I was expecting a lot of entries in the log, but I would like to see the details/pattern of the errors Amazon is throwing.
Hi @belegring,
You can send it to me via a direct message by clicking on my name and then “Message” and then using the upload button, or uploading it via odrive and creating a shared link and posting it.
I took a look and Amazon is definitely rate limiting hard in your region. The client changes seem to be handling it pretty well.
I saw that you have Lightroom structures that you are expanding, which are notoriously painful to sync because of their ridiculous sprawling hierarchies (there are typically tens of thousands of folders in the lrdata structures, for example).
Sometimes I recommend adding lrdata to our custom exclusions list because they can be so problematic with ongoing sync. This may not be the case for you since you are just exposing the folders and not actively working out of them, but it can be a big drag on the periodic remote scans odrive does (where it will scan through the remote structure every 12 or so hours). Just something to keep an eye on.
Yeah, I noticed before that the Lightroom directory structure slowed down the Amazon sync client, so I’ve already moved it out to a separate directory structure. Actually, removing the directory hierarchies that were backed up before I moved it out is one of the things I’m trying to accomplish now.
Is there any way in which one can deal with the rate-limiting in this region?
Unfortunately there isn’t really anything that can be done aside from dealing with the errors when they occur or reducing the amount of API requests being made.
To that second point, while you are performing your sync, you can “Disable background scanning” under “Ready to sync new changes / Syncing changes” at the top of the odrive tray menu. This will stop odrive from doing its periodic scans on the remote storage, which will lessen the total API calls made. It may help a bit.
OK, I’ll give it a try turning off background scanning temporarily, and then I’ll wait for all my directories to download before cleaning up unnecessary ones.
Thank you so much for all your support.
I should add that I really appreciate odrive.
In my current situation, the sync client provided by Amazon just hangs, and deleting empty folders one-by-one via the web interface is not really feasible either. If it weren’t for odrive I would probably have to resort to deleting higher level directories, but since I store some data in cloud only, this would have been somewhat risky.
