Amazon Drive Rate Limiting

Troubleshooting
1

Hi, I have the latest MacOS client (7193) - and it seems like sync is stuck (Amazon CloudDrive).
I think they might be rate-limiting me, since I installed a Windows client on an Amazon EC2 VM - basically syncing/downloading all files (about 5TB of data), so I can then move to Dropbox before Amazon CloudDrive shuts down.
Can I send you diagnostic logs so you can see if there are any errors in the Amazon API side?

2

Hi @budowski,
Sure you can send over a diagnostic. Just let us know when it has been sent.

You can also look at the sync activity log to see all of the actions odrive has taken:
sync activity log

sync_activity_log_mac

3

Thanks, just sent the logs.
Also looked at the activity log - does seem like rate limitations:

Error: code CD_RATE_LIMIT_EXCEEDED caused by CloudDriveRateLimitingException(code CD_RATE_LIMIT_EXCEEDED - {} - )

Question is - from your experience, how long does it take until they stop with this? It’s been almost 24 hours and still very slow

4

Hi @budowski,
Amazon has never provided any specifics on their rate limits, so it is hard to say. They may be more impacted right now since lots of users are trying to pull their stuff off (they likely tightened things up, too).

Previously we have seen rates vary quite a bit. In some cases we’ve even seen what appeared to be a “soft ban”, where all downloads were halted for 24-48 hours. It doesn’t look like this is happening in your case, as I do see some downloads squeaking through amid the rate limit errors.

Here are a couple things you can do that may help:

  • Disable background scanning. Since you are purely focused on downloading, you can disable odrive’s background scanning. You can find this option at the top of the odrive menu under “Ready to sync new changes”/“Syncing changes”. This will stop odrive from performing its infrequent, but intensive, remote scans, which will reduce the number of requests being sent to Amazon.

  • Reduce the number if concurrent downloads (requires Premium subscription). This will slow down the rate of downloads (when you aren’t being rate-limited), but will hopefully ease the aggressiveness of Amazon’s throttling, giving you better throughput overall.

    To do this:

    1. Double-click to open the odrive_user_premium_conf.txt file in the root of the odrive folder.
    2. Change "maxConcurrentDownloads": 4, to "maxConcurrentDownloads": 2,
    3. Change maxConcurrentJobs: 4, to maxConcurrentJobs: 2,
    4. Save the file

  • Only run one client, or only let one client actively access Amazon Drive (exit odrive on all other machines, or unsync the Amazon Drive folder on all machines except the one you are performing the bulk downloading on). Since the rate-limit is applied to your user account, you will want to reduce the calls you are making to Amazon to just the essentials