Sync stopping or re-uploading data

Troubleshooting
1

Hi, i have odrive and i have drop box etc to the account, but i use amazon the most as there is so much data i can have.

i am encrypting all my files apart from some.

I’m finding that it starts to upload some files and then it stopped at some point and restarts them and then i look again and its stopped all together it says its doing other files but i can see on the network that this isnt the case and a restart of odrive will start this all over again, I’ve left it for a week before still just stopped and didnt keep on going.

2

Hi @leo_beaumont,
Can you send a diagnostic from the tray menu and describe the behavior you are seeing right now, including a couple of file names I can reference?

3

****Keep that is one folder to look at

4

Having same problem here. @Tony

5

Hi @leo_beaumont,
The diagnostic didn’t actually make it through, but a thread dump did and it looks like odrive may be overwhelmed with the amount of data it is processing.

Can you tell me a little bit about the data you are uploading? Things like:

  • How much data, in total size
  • Approximately how many files?
  • Approximately how many folders?
  • Where is the data located?
  • Where is the odrive folder located?
  • Are there any symlinks, external drives, or network drives involved?
  • Are the files you are seeing repeat upload large files?

To get a better sense of what odrive is doing, we can use the CLI to query its status using this method:

  1. Open a terminal session (type “terminal” in Spotlight search):

  2. Copy and paste the following command into the terminal and hit enter:
    python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status

This will return a summary of odrive status. To get more detailed you can add parameters to the status command like this:
python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --uploads
python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --waiting
python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --trash

@lee.jake This may be helpful to you, as well.

6

Hi Tony,

i would say 1 TB i move them into the encryptor folder this is on a 8 TB hard drive on inside of the computer. when putting them commands into the terminal the waiting one comes back with amazon cloud drive error at the end of everything.
yes they are large files.

7

Thanks @leo_beaumont,
Unfortunately we’ve seen a lot of upload troubles with Amazon Drive. Since they do not offer a resumable upload API and they have some quirky behavior on their large file uploads, we have seen these cases where large files can get stuck retrying uploads due to recurring Amazon Drive errors.

Here is some background on the issue:

Can you send a diagnostic from the odrive tray menu so I can take a look?

8

okay no update on a fix for this then? or a work around?
and i will do but when i click on the icon it just highlights and goes away i have this problem at times too.

9

Hi @leo_beaumont,
The proper fix would need to be something on the Amazon side of things, although we are looking at ways to combat this on our side.

A large reason for creating IFS was because of these Amazon Drive upload issues, actually: https://docs.odrive.com/docs/manage-sync#section-upload-large-files-ifs-split-large-files-

The caveat to using IFS with encryption, currently, is that there is a bug in the download processing that can make it difficult to complete a download of a large encrypted IFS file, once it is uploaded. It is not a problem with the file, but rather with the way we are handling the multi-part download. This will be fixed in our next major revision however, and it is safe to upload your data if you goal is to archive it.

If you get a chance to run the status command above to list waiting items, please paste the output here so I can take a look: python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --waiting

I’m not quite sure what you mean here. When you click on the odrive icon is ends up disappearing?

10

Hi @Tony

So if you click on the icon in the top bar to get its menu, it highlights and then doesn’t do anything.

as i had to quit and reopen odrive there is nothing come up on that command,
this is what i have to do to get the menu to come back up to see if its uploading or send you that data.

11

Hi @leo_beaumont,
I think that odrive may be processing too much and it is causing the odrive menu to have this issue. Even though that is the case, you should be able to follow these instructions to get some status from odrive.

Make sure odrive is running before running these commands.

  1. Open a terminal session (type “terminal” in Spotlight search):

  2. Copy and paste the following command into the terminal and hit enter:
    python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status

This will return a summary of odrive status.

  1. (Optional) To get more detailed you can add parameters to the status command like this:
    python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --uploads
    python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --waiting
    python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --trash

After going through steps 1 and 2, please post a screenshot of what you see.

Thanks!

12

odrive Make Cloud Storage THE WAY IT SHOULD BE.

isActivated: True hasSession: True
email: accountType: CloudDrive
syncEnabled: True version: prod 6149
placeholderThreshold: large autoUnsyncThreshold: day
downloadThrottlingThreshold: unlimited uploadThrottlingThreshold: unlimited
autoTrashThreshold: never Mounts: 2
xlThreshold: never Backups: 0

Sync Requests: 0
Background Requests: 7
Uploads: 11
Downloads: 0
Trash: 2
Waiting: 3
Not Allowed: 0

13

Thanks @leo_beaumont,
Modifying that command a bit and using the ones listed in step 3) will give us more information about the queues.

You can monitor uploads using:
python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --uploads

You can see what is on the waiting list with:
python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --waiting

If uploads are repeating, you should see them move from the uploads list to the waiting list. The waiting may then show why it failed upload.

Can you try monitoring those for a bit and see what you can find out there?

14

i can do yes. just look for errors?

15

That and just any repeating patterns you may see. The important thing is getting your files uploaded, so we want to identify anything impeding that.