But why is it uploading to 100% then?
Why does it take all the space required at Amazon drive?
What I’ve seen quite a bit is that Amazon can timeout after receiving the whole file.
Amazon Drive is actually pushing the data into S3, and there is a hand-off during the upload transmission process, including a hashing process. On large files Amazon can not respond for an extended period of time after we have transmitted all of the data. We tried increasing our timeout to accommodate this and found that Amazon would timeout the connection themselves because we hadn’t sent anything within 60 seconds (lol). It is possible they have removed their own timeout now, so I will put this on the list for revisiting. The current solution, though, is IFS.
@pkoenighofer if you are still seeing this, can you send over another diagnostic? I am going to investigate the timeouts a bit more and I need a good example to take a look at.
Thanks!
I experimented a bit further.
I uploaded a folder consisting of two 8.1GB sized files. It looped them after each other! FILE1 reached 100%, then FILE2 started and FILE1 was placed as “Waiting” in the odrive queue again.
After FILE2 reached 100% it started re-uploading FILE1 again, placed FILE2 in the queue and so on.
Since it took already all the spaced need on amazon drive. I decided to quit the odrive client and deleted the local odrive folder. (even while uploading the file again at 37% already)
after restarting the odrive client I was able to download both FILE1 and FILE2 again. I checked the hash values. They matched.
This problem is re-produceable with large files on unencrypted and encrypted spaces. Just upload a folder with two large files to an amazon backed folder.
It seems the client does not mark the file as finished and re-adds it to queue forever. So I think there is no timeout problem with Amazon, since the data got stored. It is a problem with odrive.
Even if there is a timeout. Amazon seems to do it’s part correctly. Since I was able to retrieve both files, after they started looping. Just find a way to properly detect the upload finished. As far as I noticed Amazon only reports a filesize via API, when the file is stored entirely. So there should be a way to recheck the state of a file after a timeout.
Another bug report:
After deleting the local odrive folder, it recreated it.
but it trashed “Amazon Cloud Drive”, “Encryptor” and “Spaces”.
I was able to untrash them via the odrive client. But still, I don’t think they should be trashed since they are system-defined folders and present on the web client anyways.
Thanks. Did you send a diagnostic during this time so I can take a look?
We are supposed to account for this scenario. What happens is Amazon times out, but ends up processing the file. We are supposed to query Amazon for another listing before starting the file from waiting again. The hope is that Amazon will now list the file as there and we can mark it as synced. Either this is not working as expected, or it is taking Amazon longer to list the file than it is for us to restart syncing it.
The diagnostic may be able to give us a clue. My own testing tonight is not hitting this issue, so I’m hoping to look at yours. Can you also tell me what region/country your Amazon account is in?
Thanks!
Thank you for your patience.
all times are CEST. I am located in Austria. I noticed some connections to fls-eu.amazon.com.
09:49 Upload start. odrive/Spaces/TestSpace/W.D
At first it showed the entire directory as “Syncing” item with percentage count.
10:17 4%
10:43 9%
11:21 17%
11:51 23%
15:23 Reports WD1.iso in “Syncing Queue” with 33%
WD2.iso is visible on AMAZON DRIVE
15:51 Reports WD1.iso 44%
18:44 odrive finished uploading both files.
send diagnostic report
18:45 started same folder with same two files to sync to encrypted space odrive/encryptor/software/W.D
show W.D directory with percentage count
19:19 WD2.iso uploading 1%
WD1.iso not present on amazon yet.
19:45 WD2.iso 12%
22:58 WD1.iso 2%
WD2.iso is on amazon drive now. but already in the waiting queue again.
22:59
send diagnostic report
Will send another one as soon as the already present file, start uploading again.
Any ETA on the release of the windows client?
If IFS is working properly with encryption, that would be sufficient for my use case, no need to upload large files directly
What’s the hold on? Maybe you can compile an intermediate release?
I would also be willing to try out a new build and report back.
update:
23:02 WD1.iso 4%
00:05 26%
01:15 Lost internet connection. ISP modem firmware flash/reboot. ordrive goes idle
01:20 internet connection restored. odrive start syncing again
REstarts with WD2.iso (which is already present on amazon drive!)
does not show any percentage
01:27 start with 1% of the already present file. (maybe is unable to detect presence with encryption enabled?)
WD1.iso which is NOT on amazon drive, is not even in “Waiting” queue (yet!)
send diagnostic report
Thanks. Our next release, which should be out in the next couple of days will have the IFS + encryption download fix.
23:02 WD1.iso 4%
00:05 26%
01:15 Lost internet connection. ISP modem firmware flash/reboot. ordrive goes idle
01:20 internet connection restored. odrive start syncing again
REstarts with WD2.iso (which is already present on amazon drive!)
does not show any percentage
01:27 start with 1% of the already present file. (maybe is unable to detect presence with encryption enabled?)
02:05 13%
06:26 WD2.iso at 38% (should have looped once. i guess)
06:29 send diagnostic
as far as i see it. first it started alternating again, then after loosing internet connection it completly forgot about the other file.
now it just loops on the file present at amazon drive.
i would have understood if it now tries to loop the missing file and fail, but not the other way around.
so detection may be broken in two ways. 1. cannot detected finished encrypted files 2. after loss of connection forgets about other file.
do you need anything else on this issue?
Thanks for the additional data and information. I will be taking a deeper look at this tomorrow.
I don’t need anything else at this time. Thanks for the help!
Since I have no production data on those accounts yet, I could also grant you access to the system running the client if that is any help.
Thank you for looking into this.
@Tony
What to expect from todays client release?
Changelog did not state any changes regarding this matter explicitly.
ok. 2 large files in folder uploaded to encrypted space on amazon drive >8.1GB.
client 5656.
file 1 present at Amazon Drive, got re-added to the queue again. 
now testing with IFS enabled to encrypted space.
Hi @pkoenighofer,
The IFS download from encrypted folders should be working now, which had issues before.
The large file upload cycling is still outstanding and something I will be digging into today to figure out what is going on there.
first tests with IFS on encrypted folders with unsync and re-sync are successful.
I am going to run some more complex folders now. I will report back in a few days.
1 Like
I wanted to update on the IFS + encryption behavior. Currently resume of an uploaded IFS file in an encrypted folder does not work as it does in non-encrypted folders. This is due the the randomized nature of the encrypted filenames being generated, which resets when an exception is hit. When IFS is engaged it is not finding a match to resume from.
Sorry for the confusion here.
after some testing, it does NOT work.
Please see my reply in the other thread:
I have the same issue.
Something is wrong with the detection of existing files on ACS.
Uploading to amazon partly works, partly not.
If the odrive client (on windows) gets in an upload loop, I often can see the file in a mac client and it is present on the amazon drive. But the windows client restarts the action and tries to upload the file over and over again.
I use an encryption folder for this.
I downloaded the file in the mac client, it is exact same file as on the windows side. So upload succeeded, but the client does not detect it correctly.
I stopped the windows odrive client, restarted it, still the same behavior: trying to upload already existing files.
I stopped the windows odrive client, deleted the .odrive/db/tracking files, restarted. still the same behavior: trying to upload already existing files.
I moved out the files, added empty .cloud files for each of the already uploaded files…And tada… the client recognizes the files on ACS…
So there is a bug in there in detecting a successful upload!
PLEASE CHECK, this leads to a lot of these looping issues.
Appreciate the detail and the report. We will take another look at this looping behavior in encrypted folders.