Can't upload 1 TB file to Amazon Cloud Drive

drapl · September 21, 2016, 7:14am

I’m trying to upload large file(1 TB) with split by 100mb. First time odrive stuck on 59 chunks(5.9GB), after that it keep consume bandwidth, but no new chunks added. Then I renamed this file, it’s started to uploading again and then stuck on 28 chunks. Same as before odrive keep consume a lot of bandwidth but no new chunks added. Restarting odrive/mac doesn’t help

using version 5944 on macOS El Capitan

Tony · September 21, 2016, 3:21pm

Hi @drapl,
Can you send a diagnostic from the odrive tray menu while this is happening, please? Please also let me know what the name of the files/file is that is have the issue.

Thanks!

drapl · September 22, 2016, 3:04am

Done, the file name is “1tb_justrename.dmg”

This night odrive was auto updated without any asking to version 5983 and start previously stopped syncing. But still same behavior

Tony · September 22, 2016, 9:16pm

Thanks. We are investigating.

Since this is a very large file, there may be issues with the number of chunks allowed in a folder. Can you try something while we are going through this?

Move the file out of odrive, so it stops syncing
On Amazon Drive’s web interface, remove the partially uploaded IFS folders
Set the split to 2GB instead of 100MB
Rename the file to another name
Move the file back into odrive

Thanks!

drapl · September 24, 2016, 1:55pm

it’s not related to number of chunks in a folder. odrive stop syncing(but still show activity) on random number of chunks, tested 3 times: 59, 28, 103

Tony · September 27, 2016, 3:24pm

I am running a series of tests to try to reproduce this.

Can you try something for me? When it gets stuck , can you restart odrive and see if that allows it to resume?

drapl · September 28, 2016, 4:43am

As I say before restart odrive(or even mac) doesn’t help

Tony · September 28, 2016, 4:53pm

I’m in the process of uploading a 300GB to Amazon Drive. I am seeing that some exceptions are hit during the upload that can cause the file to be put back into waiting or into not allowed for a time. But, it recovers and continues syncing after a little a delay. I am currently at 2,133 100MB segments.

The problem with hitting Amazon Drive exceptions is that continuing the upload on such a large file is expensive because odrive has to figure out where it left off, which means it needs to hash the file. Hashing a 1TB file will take quite a while during which it will look like nothing it happening. A single 1TB file is incredibly large.

We are looking at ways to optimize the handling of this, but this size of a file is really pushing the limits.

Do you still have an upload running currently?

Tony · September 29, 2016, 2:40pm

Just an update. I was able to upload my 300GB file to Amazon Drive. It was a total of 3072 segments. It definitely took some time, but it did make it.

drapl · September 29, 2016, 2:53pm

No, I’ve stopped upload because it eat all bandwidth without any progress for few days

Tony · September 29, 2016, 3:01pm

If you would like to try again, I would like to analyze one more run to see where it stops making progress.

Also, can you tell me where you are located and what region your Amazon Cloud Drive account is in?

Thanks!

drapl · October 17, 2016, 6:37am

Just tryed build 6002 - same behavior. Eated 700mb of bandwidth and no one chunk was added(split size: 100mb). My region is South-East Asia. I think region of Amazon account is same

Tony · October 17, 2016, 6:39am

Hi @drapl,
Can you please send over a diagnostic so we can take a look?

drapl · October 17, 2016, 12:12pm

Done, the file name is 1tb_justrename2.dmg

Tony · October 17, 2016, 3:21pm

Hi @drapl,
When looking at your drive from the Amazon Drive web client: https://www.amazon.com/clouddrive/folder/root you dont see the 2 TB/1tb_justrename2.dmg.464a7e8426072c69a8ae71d925fcfe07.xlarge ?

I am expecting that folder to be there along with many files inside.

I also see that Amazon is complaining about a duplicate name for one of the files in the package: 2f282b84e7e608d5852449ed940bfc51. This may be causing things to hang up. If you delete that file inside (from the Amazon Drive web client), I think it will continue on. I am not sure why that exception is happening. I will need to look into it.

drapl · October 18, 2016, 4:16am

Hi @Tony, 2f282b84e7e608d5852449ed940bfc51 it is first uploaded chunk. Looks like odrive sync client reupload file from start

Tony · October 18, 2016, 4:28am

Hi @drapl,
How many files are in the 1tb_justrename2.dmg.464a7e8426072c69a8ae71d925fcfe07.xlarge folder?

drapl · October 18, 2016, 4:34am

103 chunks. do you need some additional info?

Tony · October 18, 2016, 5:02am

Thanks @drapl,

Can you delete the 2f282b84e7e608d5852449ed940bfc51 file from the web client and then wait for odrive to try to sync the file again (after falling back into waiting), then send another diagnostic?

What is odd about this is that odrive should be seeing the contents of the xlarge folder, including the files it has already uploaded, and skipping them. The fact that it is not could mean that it isn’t getting the proper listing from Amazon, but I don’t know why that would be. In my own test against Amazon Drive, I resumed several times, with up to 3000 chunks, and it was able to pick up where it left off.

drapl · October 19, 2016, 11:50am

Deleting file help upload more ~100 chunks. Now it stuck again at 220 chunks, sometime goes to “Not allowed” state

Diagnostic sended