Amazon Drive API limits? Unexpected error. Please try again later

Hello,
I started using odrive agent to make sync faster.
I have done a very large backup on “Amazon Drive” and as amazon decided to cancel the “unlimited” accounts I am downloading everything I had there to my machine.
I decided to run 32 downloads in parallel using the following code:

exec 32>&1;
num_procs=32;output=“go”;
while [ “$output” ];
do output=$(find “/Volumes/WARRIOR/odrive/Amazon Cloud Drive/Arq Backup Data” -name “.cloud” -print0 | xargs -0 -n 1 -P $num_procs python “$HOME/.odrive-agent/bin/odrive.py” sync | tee /dev/fd/32);
done

Everything was working perfectly and odrive agent was really fast, till I started getting those messages here:
“Unable to sync nameOfMyFile.cloud. Unexpected error. Please try again later.”

I wonder if I have been blocked for excessive usage. If that was the case, I would like to know what is the limit for parallel downloads using the odriveagent.

Regards,

Hi @pedrolimao,
There isn’t a limit for parallel downloads, but I recommend keeping it to around 5-6 instead of 32 for more reliable functionality.

You can run into issues trying to have that much going at once, especially with the storage provider (Amazon in this case), clamping down on your API activity. Amazon doesn’t have any “official” limits, but they will definitely impose rate limiting after a certain amount of activity.

Hi Tony,
Thank a lot for the attention given.
The problem is that now I can’t sync to my Amazon Drive any longer, even when I try 6 syncs instead of 32.
I am still using my Amazon account perfectly from other apps, but I am not able to do so from odrive. Are you 100% sure the black-listing comes from Amazon?
Is there a way to get me out of the black-list?

Hi @pedrolimao,
odrive doesn’t impose any rate limiting or blacklisting, so it sounds like an Amazon issue.

You are using the CLI to interface to the odrive desktop client, correct? If so, can you send a diagnostic from the odrive menu?

Can you also see if you are able to download and upload using the odrive web client? (https://www.odrive.com)

Yes. I m using CLI, and I have just sent you a “diagnostic”.
I just tried the web client and it is possible to download files there, but the web client is very slow loading the contents of each folder.
For example, I have one folder open it’s been more than 5 minutes, and I still can’t see all the files that I can see using the Amazon Drive web client, for example.

Hi @pedrolimao,
I took a look at the diagnostic… Amazon is returning 404 (typically means “not found”) errors on many of the requests, which is not something we normally see. Unfortunately they are not providing any additional information with these errors.

It also looks like you have the odrive desktop client and the odrive agent running at the same time. Is that correct?

To clarify, the odrive Desktop client and the odrive Agent are both sync engines. They basically do the same thing. The CLI, which is separate from the odrive Agent, can be used to issue commands to either of those sync engines, but you do not want them both running at the same time. Let me know if you have any questions about this.

In your case, you really should just have the odrive Desktop client running and you can send commands to that via the CLI. Can you make sure the odrive agent is shut down and then restart the odrive Desktop client and try a few basic operations in the root of the Amazon Drive folder (create folder, upload file, download file, expand folder, etc)? A system reboot may be the best option to make sure everything is initialized properly.