Haha. During my sync I got:
There was an error sending the command, please make sure odrive agent or desktop is running.
It got me this:
find: ‘Drive\342\200\235’: No such file or directory
xargs: “/root/.odrive-agent/bin/odrive”: No such file or directory
Hey @voarsh,
I noticed some of the commands above are formatted with the “fancy” quotes (“ ” instead of " "), which can screw things up. Can you try this command, as pasted below? The quotes should be correct here so you don’t run into that issue.
exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "/shared/odrive/Google Drive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive" sync | tee /dev/fd/6); done
Hey @voarsh,
Hmm… Can you try running the odriveagent in a separate window in the foreground (just run $HOME/.odrive-agent/bin/odriveagent without nohup or output redirection) so we can see if/when it exits if it is spitting anything out?
I’ll do it next when it fails. It’s making some progress and I’m on a schedule to get this stuff downloaded ASAP. 
I just raised memory as well.
I am running:
$HOME/.odrive-agent/bin/odriveagent
I am not sure it’s syncing anything with just this command so I’ll also run your additional multithread sync
So now I have them both running. Unfortunately I have to sleep, so any output will likely be lost if there’s lots of console spam.
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
Traceback (most recent call last):
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 877, in emit
File “/home/ubuntu/hudson/workspace/Odrive_Lnx_Agent_64/deploy/output/artifact s/build/odriveagent/LinuxOdriveAgent/out00-PYZ.pyz/logging”, line 837, in flush
IOError: [Errno 28] No space left on device
Logged from file src.odrive_app.controllers.utility.Apply, line 86
That’s annoyed it died at 7 AM.
I only saw it at 2 PM.
It seems to be syncing again when I run the command inside the odrive folder, but I don’t know why it keeps crashing.
I don’t have this problem on Windows.
I noticed that the LXC/Container was reporting space issues, but I didn’t think that syncing would cause issues if the host was on low disk. I’ve increased disk by another 1 G and will see if I get another error.
It also seems odriveagent is causing huge I/O delay on the box (among another process that uses the HDD (not the USB HDD), this I/O activity is on a USB device, I am not sure why it’s affecting the host, but it’s blocking other processes from running at normal times.
Hi @voarsh,
Disk usage will definitely increase as you go because of the local tracking databases and logging. Since you have a lot of data I would expect the tracking database to grow fairly large.
The I/O activity could also be due to logging and sync database activity (reads/writes during the bulk download process).
The bulk processing you are doing is definitely going to need resources since it is pretty intensive, especially with the higher-performance script running to do multi-threaded folder expands and downloads. If possible, I would make sure you have a decent amount of disk and CPU available to go through the 8TB of data.
What are you running this on and how constrained are the resources?
Just an update on my sync.
It’s going slowly indeed, from the high I/O usage and grinding my machine to a halt, restarts, weird behaviour. I’ve had to use 1/2 processes.
I’m now experiencing an unusual thing where I needed to reinstall the odrive agent, and when I run a sync in the same directory I am seeing:
Unable to sync init.pyc.cloud. This file is not inside an odrive folder.
but for everything…
I re-mounted ("$HOME/.odrive-agent/bin/odrive" mount “/shared/odrive” /) , just in case and sync ("$HOME/.odrive-agent/bin/odrive" sync “/shared/odrive/Google Drive” --recursive):
and get:
Unable to sync Finale Fireworks.mp3.cloud. This file doesn’t exist.
for loads and loads of files. Should I be concerned?
Hi @voarsh,
If you reinstalled odrive you will need to let odrive scan through the entire mount to re-index everything. This could take quite a while, unfortunately. Until odrive is able to index the structure it won’t “know” about it, so you may see errors like this.
If you are concerned about any files in particular, you can verify them via the odrive web client or the Google web client.
So, I suppose that it is finding files that are in the folder, but not on the remote?
I don’t know why I needed to re-install it - it just wouldn’t let me run odriveagent, it didn’t think it was running. 
Thanks, I guess I’ll just continue running it over night. I think it’s only done 2 TB’s so far, I’ve actually delete a portion of data it downloaded that was unnecessary as well. I have a feeling it’ll take another 1/2 months - even though my internet speed is 3-5 MB/s - the client/system instability is delaying me.
Hey @voarsh,
The errors you posted previously indicated lack of drive space, which could also cause local tracking database corruption if odrive is unable to properly write to the database files. It is possible it hit something like that that caused an issue with it running. It is hard to tell without seeing any logs or output.
Can you tell me a bit about what you are running on and what the resources dedicated to it look like?
A large-scale bulk operation like this will need a good amount of resources to run comfortably, which means you’ll want to be generous with disk space, CPU, and memory allocation, however that is being done.
Can you be more targeted in your download commands to avoid unnecessary sections?
The LXC container has 6 GB’s disk space free, I haven’t seen it max out, the external drive has 11 TB’s free. There’s no space issue.
Debian flavoured host, with a Turnkey Core flavour container, 7 GB’s dedicated RAM to this instance, USB 2 (external drive), unfortunately.
It has been obvious to me that CPU has never been an issue:
(container)
The extreme load is taken into account with high I/O delay/wait time, not CPU. The CPU is mostly idle:
The container is hosted by PROXMOX (Debian) and the host is passing the mounted drive to the LXC container. Raising the odrive process count by 4 thrashes the I/O. Not sure if it is USB 2 issue, PROXMOX or the something else. But I had to lower it.
I had initially tried it on USB 3, and the CPU on that host was slightly less and it took a beating so I opted for the one I am using, which only has USB 2 - so I don’t think that’s the cause.
I did have that thought. Eventually all will need to be downloaded. I suppose what I should have done is have a clean up of non-essential stuff before syncing…
Hey @voarsh,
I had a thought about the high I/O. The CLI is packaged in a way where it needs to “unpack” when it is executed. When you are using it in a scripted fashion it could raise I/O when it is called over and over again. This hasn’t been an issue on the systems I have tested on, but it could be impacting your system more than usual.
To avoid this you can use the pure python CLI version. You would just need to have python installed on the system to do this. This would avoid the binary executable unpacking and may alleviate some of the I/O pressure you are seeing. To be clear, this is only for the CLI client, the odriveagent is continuously running, so is not affected in this way.
I had another system crash, which wasn’t entirely related to oDrive and I had to forcibly turn off the computer. Talking about a load of over 200, completely out…
Running oDrive gets me:
root@nextcloud ~# “$HOME/.odrive-agent/bin/odrive” > /dev/null 2>&1
root@nextcloud ~# “$HOME/.odrive-agent/bin/odrive” status
There was an error sending the command, please make sure odrive agent or desktop is running.
root@nextcloud ~# python “$HOME/.odrive-agent/bin/odrive” status
File “/root/.odrive-agent/bin/odrive”, line 1
SyntaxError: Non-ASCII character ‘\x97’ in file /root/.odrive-agent/bin/odrive on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
Really annoying, because the only way I know to sort this is to re-download/install odrive, resetting the DB… which will take over 6 hours to re-scan. 
Interesting.
exec 6>&1;num_procs=3;output=“go”; while [ “$output” ]; do output=$(find “/shared/odrive/Google Drive” -name “.cloud” -print0 | xargs -0 -n 1 -P $num_procs python “$HOME/.odrive-agent/bin/odrive” sync | tee /dev/fd/6); done
So, like this?
— it doesn’t seem to like “python” included"
Or:
python “$HOME/.odrive-agent/bin/odrive.py” mount “/shared/odrive” /
python “$HOME/.odrive-agent/bin/odrive.py” sync “/shared/odrive/Google Drive” --recursive
Hi @voarsh,
Yeah you will want to target the odrive.py file instead of the odrive binary. So:
exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "/shared/odrive/Google Drive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs python "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done
Make sure you are running odriveagent instead of odrive to run the agent process. If that is still not working, then try running it without piping the output to null. So just: $HOME/.odrive-agent/bin/odriveagent
Seems the Python version may be helping a bit, plus some changes on my side.
I’ve bumped into a bit where lots of files are showing as this:
update
Crud:
Drive site
Another question, do you know why I cannot download Google documents?
I always get an error for these types of docs:
Unable to sync ttw-election-branch-… details.gdocx.cloud. Google error.
Hey @voarsh,
It looks like the errors you were seeing may be related to Google issues, is that correct? The log files can also shed more light on the errors that might be seen.
For thee Google Docs files, I would probably need to see the logs, or even the diagnostics if Google is returning an unexpected error.
The next time you see that, can you send a diagnostic (diagnostics command)?
