Downloaded via odrive Win, can I continue download on linux?

voarsh · December 5, 2020, 2:20am

Hi.
So, I’m downloading 8 TB’s onto my USB HDD via oDrive on Windows.
I want to download the linux CLI, mount the USB to a folder, and basically continue the odrive sync from where it left off with Windows.
E.G. mount onto the USB folder, re-sync, leaving the existing content there and downloading things that hasn’t synced.

Is this doable?

Tony · December 5, 2020, 3:51am

Hey @voarsh,
Yes, this is doable. On the agent you specify the mount locations (both local and remote). This will work similarly to how “sync to odrive” works. Look at step 4. here: https://docs.odrive.com/docs/odrive-sync-agent#using-the-cliagent-all-platforms

Keep in mind that it may take a while for odrive to slog through all of the data after you create the mount.

voarsh · December 5, 2020, 8:45am

Irrelevant for you, but my use case is that I started my downloading of Google Drive (8
TB’s) on Windows, was using the downloaded data as I got it.

I don’t fancy waiting until I download it on Windows to them mount it later.

But ultimately it’s going on a LXC/container as a mount for NextCloud (my Google Drive replacement).

How will the mount point work when I already have the data in:
/shared/odrive ? (/shared/odrive being the existing synced stuff)

python “$HOME/.odrive-agent/bin/odrive.py” authenticate 00000000-0000-0000-0000-000000000000-00000000

“$HOME/.odrive-agent/bin/odrive” mount “/shared/odrive” /
Like this?

python “$HOME/.odrive-agent/bin/odrive.py” sync “shared/odrive/Google Drive.cloudf” —recursive
(FYI, my storage cloudf is already synced so I am not even sure if it’ll be there as “.cloudf”?)

I am ever so nervous of messing up what I’ve already downloaded - it’ll take about 1 month to download the rest.

Tony · December 5, 2020, 3:43pm

Hi @voarsh,

This sounds like the correct pathing based on what you’ve described above. odrive will perform a merge of the two folders and determine what is already synced locally.

To confirm the approach and behavior and make sure it is what you want and expect, you can start by mounting a subfolder and observing the result. For example:
"$HOME/.odrive-agent/bin/odrive" mount "/shared/odrive/Google Drive/subfolder1/subfolder2" "/Google Drive/subfolder1/subfolder2"

What you should see is that odrive will:

Scan through the items on both the local and remote sides and determine what is already the same on both sides.
If odrive sees anything locally that is not on the remote, or is different than the remote (a file has a different size locally, for example), it will upload it.
Anything on the remote side that is not on the local side will show up as a new placeholder file.
Anything on the remote side that already has a corresponding placeholder file on the local side will remain that way.

Once odrive is done scanning through, you can start a recursive sync:
python "$HOME/.odrive-agent/bin/odrive.py" sync "/shared/odrive/Google Drive/subfolder1/subfolder2" --recursive and odrive will download all of the remaining items.

Once done with the test, you can unmount "/shared/odrive/Google Drive/subfolder1/subfolder2" and go for the full mount and sync:

"$HOME/.odrive-agent/bin/odrive" mount "/shared/odrive" "/"

The scan that is kicked off on the full mount will likely take a long time to complete since odrive will need to go through every folder and look at the local and remote listings and do a comparison. Once done, you can kick off the recursive sync.

For the recursive sync, since you have so much data, you can also use a more advanced command to increase performance by using 4 concurrent processes (the CLI --recursive is single-threaded):
exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "/shared/odrive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive" sync | tee /dev/fd/6); done

voarsh · December 5, 2020, 5:38pm

I’m not quite getting what I expect:

“$HOME/.odrive-agent/bin/odrive” mount “/shared/odrive/Google\ Drive/asus” “/Google Drive/asus”
Unable to sync /shared/odrive/Google\ Drive/asus Path not found.

On another computer I did sync (freshly)
With something like this:
“$HOME/.odrive-agent/bin/odrive” refresh ~/odrive-agent-mount/Google\ Drive/Personal/TV\ Shows/

I’ve done:
cd /shared/odrive
Then executed:
“$HOME/.odrive-agent/bin/odrive” mount “/shared/odrive” /

I am not actually sure what it is syncing, if it’s going to do all my storage inside /odrive/ or just the Google Drive one that is already started from Windows?

odrive Make Cloud Storage THE WAY IT SHOULD BE.

isActivated: True hasSession: True
email: XX@XX.co.uk accountType: OneDrive
syncEnabled: True version: prod 999
placeholderThreshold: neverDownload autoUnsyncThreshold: never
downloadThrottlingThreshold: unlimited uploadThrottlingThreshold: normal
autoTrashThreshold: never Mounts: 1
xlThreshold: never Backups: 0

Sync Requests: 0
Background Requests: 1
Uploads: 0
Downloads: 0
Trash: 0
Waiting: 0
Not Allowed: 1

I ran this:
“$HOME/.odrive-agent/bin/odrive” sync “/shared/odrive/Google Drive” --recursive

It seems to be syncing, haven’t checked for data loss yet.

I can confirm this is a new document I created for testing.

Seems to working just fine.

Tony · December 5, 2020, 6:00pm

Hey @voarsh,

I believe you were getting this error because you quoted the path but still had the \ to escape the space between Google and Drive. When you use quotes you don’t need to use the \ escape character for spaces.

This will mount your entire remote odrive to /shared/odrive. The background request shown is likely the scan of all of the exposed folders.

This will kick off a recursive download of Google Drive folder. There shouldn’t be any data loss. Are you seeing something unexpected?

voarsh · December 5, 2020, 6:01pm

While I can see /asus/test in that screenshot, the file is basically empty and it hasn’t downloaded the actual file.
Do I need to wait a long time until I find out it completes it?

Tony · December 5, 2020, 6:04pm

It will depend on the amount of data odrive needs to go through. Since you started at the top of the Google Drive folder it may be traversing through other folders first and hasn’t gotten to asus/test yet.

You can tail the sync log with tail -F ~/.odrive-agent/log/main.log to see the current activity.

voarsh · December 5, 2020, 6:07pm

root@nextcloud ~# tail -F ~/.odrive-agent/log/main.log
05 Dec 05:55:00PM INFO Successful Expand Folder for /shared/odrive/Google Drive/MacriumAsus.cloudf
05 Dec 05:55:01PM INFO Successful Expand Folder for /shared/odrive/Google Drive/Uber.cloudf
05 Dec 05:55:02PM INFO Successful Expand Folder for /shared/odrive/Google Drive/Fitbit.cloudf
05 Dec 05:55:08PM INFO Successful Expand Folder for /shared/odrive/Google Drive/QuickBooks.cloudf
05 Dec 05:55:12PM INFO Successful Expand Folder for /shared/odrive/Google Drive/asus.cloudf
05 Dec 05:55:13PM INFO Successful Expand Folder for /shared/odrive/Google Drive/LAPTOP-External-Syncs.cloudf
05 Dec 05:55:15PM INFO Successful Expand Folder for /shared/odrive/Google Drive/IFTTT.cloudf
05 Dec 05:55:28PM INFO Successful Expand Folder for /shared/odrive/Google Drive/support@profanity-blocker.com.cloudf
05 Dec 05:55:52PM INFO Successful Expand Folder for /shared/odrive/Google Drive/Shared with Me.cloudf
05 Dec 06:01:28PM ERROR Failed Upload File for /shared/odrive/Google Drive/netsh advfirewall firewall add rule name=＂Allow from 192.gdocx Error:  Gdocs cannot be uploaded.

I think it’ll be a while until I know if it will download the files. Nothing meaningful in the logs.

The bit I am curious about is when it did this:
05 Dec 05:55:12PM INFO Successful Expand Folder for /shared/odrive/Google Drive/asus.cloudf

Why did it not download the test doc?

Tony · December 5, 2020, 6:15pm

Hi @voarsh,
Sync traversal isn’t necessarily linear, so it may be operating on some other paths.

You can use this command to get a good idea of the current activity:
watch -n 5 'date && ~/.odrive-agent/bin/odrive status --sync_requests && ~/.odrive-agent/bin/odrive status --downloads && ~/.odrive-agent/bin/odrive status --uploads && ~/.odrive-agent/bin/odrive status --background && tail ~/.odrive-agent/log/main.log'

Is the recursive sync command still running or did it return? It is also possible that the sync command was issued too early and odrive wasn’t finished scanning.

voarsh · December 5, 2020, 6:18pm

I can’t seem to run this

I can’t actually say at the moment, I closed the screen for that.
But I guess so:
Sync Requests: 1
Background Requests: 1
Uploads: 0
Downloads: 0
Trash: 0
Waiting: 0
Not Allowed: 1

Should I kill odrive and start it again?

I keep seeing new .cloudf files but no new files that it actually downloads.

Tony · December 5, 2020, 6:20pm

Hi @voarsh,
The command in the screenshot is missing the final single quote '

If you closed the screen then the syncing will stop. You will want to leave it up.

voarsh · December 5, 2020, 6:22pm

I was ran with nohup &
It’s still adding placeholders, so it hasn’t stopped.

voarsh · December 5, 2020, 6:23pm

watch -n 5 'date && ~/.odrive-agent/bin/odrive status --sync_requests && ~/.odrive-agent/bin/odrive status --downloads && ~/.odrive-agent/bin/odrive status --uploads && ~/.odrive-agent/bin/odrive status --background && tail ~/.odrive-agent/log/main.log'

Every 5.0s: date && ~/.odrive-agent/bin/odrive status --sync_requests && ~/.odrive-agent/bin/odrive status --downloads && ~/.odr… nextcloud: Sat Dec 5 18:23:19 2020

Sat Dec 5 18:23:19 UTC 2020
/shared/odrive/Google Drive/Shared with Me 0%
No downloads.
No uploads.
Google Drive 0%
05 Dec 05:55:02PM INFO Successful Expand Folder for /shared/odrive/Google Drive/Fitbit.cloudf
05 Dec 05:55:08PM INFO Successful Expand Folder for /shared/odrive/Google Drive/QuickBooks.cloudf
05 Dec 05:55:12PM INFO Successful Expand Folder for /shared/odrive/Google Drive/asus.cloudf
05 Dec 05:55:13PM INFO Successful Expand Folder for /shared/odrive/Google Drive/LAPTOP-External-Syncs.cloudf
05 Dec 05:55:15PM INFO Successful Expand Folder for /shared/odrive/Google Drive/IFTTT.cloudf
05 Dec 05:55:28PM INFO Successful Expand Folder for /shared/odrive/Google Drive/support@profanity-blocker.com.cloudf
05 Dec 05:55:52PM INFO Successful Expand Folder for /shared/odrive/Google Drive/Shared with Me.cloudf
05 Dec 06:01:28PM ERROR Failed Upload File for /shared/odrive/Google Drive/netsh advfirewall firewall add rule name= Allow from 192.gdocx Error: Gdocs cannot be u
ploaded.
05 Dec 06:15:34PM ERROR Failed Upload File for /shared/odrive/Google Drive/netsh advfirewall firewall add rule name= Allow from 192.gdocx Error: Gdocs cannot be u
ploaded.
05 Dec 06:21:34PM ERROR Failed Upload File for /shared/odrive/Google Drive/netsh advfirewall firewall add rule name= Allow from 192.gdocx Error: Gdocs cannot be u
ploaded.

Tony · December 5, 2020, 6:26pm

Ahh, okay, then it should be alright. The status output shows it is working on Google Drive as a background request and is actively syncing /shared/odrive/Google Drive/Shared with Me (doing folder expands). It is probably just going to take a long time to get through everything.

voarsh · December 5, 2020, 6:33pm

Alrighty. Thanks.
I’ll report back when I find it does/doesn’t actually download any of the files, just placeholders.

Tony · December 5, 2020, 6:51pm

With the amount of data you have, you may also want to try this command in another window to speed up sync by quite a bit:

exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "/shared/odrive/Google Drive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive" sync | tee /dev/fd/6); done

voarsh · December 5, 2020, 6:55pm

Something I noticed:

“$HOME/.odrive-agent/bin/odrive” sync “/shared/odrive/Google Drive” --recursive

this doesn’t seem… to be running in background.
Only now when I run it in console with it open do I see:

image1366×768 90.5 KB

(notice: 5.4MB remaining (83%))

How can I get THIS command to run in background?

voarsh · December 5, 2020, 6:56pm

Can you elaborate on what this does?

Thanks.

Tony · December 5, 2020, 7:09pm

I would actually suggest not running it in the background. Instead I would open up some additional putty sessions to the server so you can monitor progress and behavior. In the background you lose some visibility.

I suggest:

1 window for the status/logging monitor command
1 window for the recursive sync command (either the --recursive or the scripted one above)
1 window for misc stuff you may need to do on the server while things are running

This is a scripted way to perform a multi-threaded recursive sync. It basically will do the same thing as --recursive, except it will do it in 4 concurrent threads (so 4x).