Recurring problem with upload to Amazon drive

For quite a few times I have encountered a very specific (and a very annoying) glitch.
I’m currently uploading large files, around 20 Gb each. As there’s no option currently to instruct Odrive to upload them one by one, I do it manually by adding them to the sync directory one by one.
In many cases the upload reaches 99% (sometimes even 100%), and then the upload gets reset to zero.
No explanation of any kind is provided. Just goes from 100 back to 0.
Does anybody know what this issue is caused by?

Thanks!

P.S. I’m also going to open a feature request to do something about the utter lack of information regarding the failures

Hi @alexey,

Pretty much every time I’ve investigated something like this it turns out that Amazon is returning 504s and 408s for repeatedly uploading file. Here is a post that goes over similar issues: Amazon Drive Sync to 98% - Repeat

This is a common issue I’ve seen with Amazon. It really throws a wrench in the works. Essentially what happens is Amazon throws an error back to us on upload, telling us that it failed. We ack that and place the file back in the queue. Sometime later the file actually shows up on Amazon’s side and was, apparently, actually uploaded.

This tends to lead to a race condition where we get the error and retry before the file actually shows up on Amazon’s side.

We are going to try to combat this as much as possible in future revs with more intelligent back-offs, better queuing management, and more robust exception handling, but there are limits to what we can do when Amazon is the one producing the errors. I think if Amazon Drive was to release an upload resume API, that would also help tremendously.

If you would like to send over a diagnostic from the odrive menu and the name of the file, I can take a look at confirm.

For this particular file, see if you see it actually in Amazon Drive already. The file may actually be showing up sometime later, but we have already initiated a subsequent upload of the file.

Something to try if the above is true:

  1. If the file looks to be making it up, eventually, move the local file outside of odrive’s view, so it stops trying to upload it.
  2. Wait for the .cloud file to show up locally and then download it and make sure it ends up correct/whole on the download.
  3. It is possible that moving the local file out will be picked up as a delete. If so, make sure you restore the item from the odrive trash (which will create the placeholder for the file), then download it.

Hi Tony,

Thanks for an incredibly detailed reply!
Actually, now it makes sense. Indeed, on one occasion after the upload restarted and the progress was around 1%, I suddenly noticed that the badge on the file in my local directory had turned into a blue V-mark. I then accessed the storage from another computer, synced it there and made sure the file was indeed intact and valid. Back then I didn’t know what to make of it…

Specifically in this latest case I complained about the workaround you suggested didn’t work, but I’ll keep trying if it happens again.

Generally, I agree, of course, that this problem should be fixed by Amazon, but I also agree that modifying your back-off policies and queue management might help, too. For one, I’d suggest not re-attempting the upload for at least half an hour in case you have uploaded the whole file and only got an error at the very end of the process. But I’m sure your SW guys have thought of this as well.

So just one more question (an expected one): when are you guys planning to release your next major version? There are more fixes I need badly that are supposed to be included in it.

Thanks again,
Alexey

Thanks for the reply @alexey.
We are trying to get out the new stuff as soon as possible. Here is a response I wrote to that question in another post:

1 Like

Hey @alexey! I was having the same problem with Odrive but I found a work around. Take the file you’re trying to sync out of Odrive and upload it through Amazon app directly. As soon as it reaches 100% it’s synced and didn’t went back to 0%. Hope it works with you too! And hope Odrive looks at it because apparently it’s working on Amazon’s end.

Thanks for updating to this thread @besugophoto. I posted a reply to your other thread to explain this a bit more.