Can't upload 1 TB file to Amazon Cloud Drive

Troubleshooting
#1

I’m trying to upload large file(1 TB) with split by 100mb. First time odrive stuck on 59 chunks(5.9GB), after that it keep consume bandwidth, but no new chunks added. Then I renamed this file, it’s started to uploading again and then stuck on 28 chunks. Same as before odrive keep consume a lot of bandwidth but no new chunks added. Restarting odrive/mac doesn’t help

using version 5944 on macOS El Capitan

#2

Hi @drapl,
Can you send a diagnostic from the odrive tray menu while this is happening, please? Please also let me know what the name of the files/file is that is have the issue.

Thanks!

#3

Done, the file name is “1tb_justrename.dmg”

This night odrive was auto updated without any asking to version 5983 and start previously stopped syncing. But still same behavior

#4

Thanks. We are investigating.

Since this is a very large file, there may be issues with the number of chunks allowed in a folder. Can you try something while we are going through this?

  1. Move the file out of odrive, so it stops syncing
  2. On Amazon Drive’s web interface, remove the partially uploaded IFS folders
  3. Set the split to 2GB instead of 100MB
  4. Rename the file to another name
  5. Move the file back into odrive

Thanks!

#5

it’s not related to number of chunks in a folder. odrive stop syncing(but still show activity) on random number of chunks, tested 3 times: 59, 28, 103

#6

I am running a series of tests to try to reproduce this.

Can you try something for me? When it gets stuck , can you restart odrive and see if that allows it to resume?

#7

As I say before restart odrive(or even mac) doesn’t help

#8

I’m in the process of uploading a 300GB to Amazon Drive. I am seeing that some exceptions are hit during the upload that can cause the file to be put back into waiting or into not allowed for a time. But, it recovers and continues syncing after a little a delay. I am currently at 2,133 100MB segments.

The problem with hitting Amazon Drive exceptions is that continuing the upload on such a large file is expensive because odrive has to figure out where it left off, which means it needs to hash the file. Hashing a 1TB file will take quite a while during which it will look like nothing it happening. A single 1TB file is incredibly large.

We are looking at ways to optimize the handling of this, but this size of a file is really pushing the limits.

Do you still have an upload running currently?

#9

Just an update. I was able to upload my 300GB file to Amazon Drive. It was a total of 3072 segments. It definitely took some time, but it did make it.

#10

No, I’ve stopped upload because it eat all bandwidth without any progress for few days

#11

If you would like to try again, I would like to analyze one more run to see where it stops making progress.

Also, can you tell me where you are located and what region your Amazon Cloud Drive account is in?

Thanks!

#12

Just tryed build 6002 - same behavior. Eated 700mb of bandwidth and no one chunk was added(split size: 100mb). My region is South-East Asia. I think region of Amazon account is same

#13

Hi @drapl,
Can you please send over a diagnostic so we can take a look?

#14

Done, the file name is 1tb_justrename2.dmg

#15

Hi @drapl,
When looking at your drive from the Amazon Drive web client: https://www.amazon.com/clouddrive/folder/root you dont see the 2 TB/1tb_justrename2.dmg.464a7e8426072c69a8ae71d925fcfe07.xlarge ?

I am expecting that folder to be there along with many files inside.

I also see that Amazon is complaining about a duplicate name for one of the files in the package: 2f282b84e7e608d5852449ed940bfc51. This may be causing things to hang up. If you delete that file inside (from the Amazon Drive web client), I think it will continue on. I am not sure why that exception is happening. I will need to look into it.

#16

Hi @Tony, 2f282b84e7e608d5852449ed940bfc51 it is first uploaded chunk. Looks like odrive sync client reupload file from start

#17

Hi @drapl,
How many files are in the 1tb_justrename2.dmg.464a7e8426072c69a8ae71d925fcfe07.xlarge folder?

#18

103 chunks. do you need some additional info?

#19

Thanks @drapl,

Can you delete the 2f282b84e7e608d5852449ed940bfc51 file from the web client and then wait for odrive to try to sync the file again (after falling back into waiting), then send another diagnostic?

What is odd about this is that odrive should be seeing the contents of the xlarge folder, including the files it has already uploaded, and skipping them. The fact that it is not could mean that it isn’t getting the proper listing from Amazon, but I don’t know why that would be. In my own test against Amazon Drive, I resumed several times, with up to 3000 chunks, and it was able to pick up where it left off.

#20

Deleting file help upload more ~100 chunks. Now it stuck again at 220 chunks, sometime goes to “Not allowed” state

Diagnostic sended