Odrive Sync Agent: A CLI/scriptable interface for odrive's Progressive Sync Engine for Linux, OS X, and Windows

Tony · March 25, 2016, 11:02pm

Documentation also available here: https://docs.odrive.com/docs/odrive-sync-agent

Overview
Installing odrive Sync Agent
Windows
OS X
Linux/RPi
odrive Sync Agent quick start guide
Available commands
Helpful tips
Direct download links

Overview

The odrive Sync Agent provides advanced users with a flexible, scriptable interface to odrive’s powerful Progressive Sync engine. Use it to bring cloud access to almost any environment.

The odrive Sync Agent can be used on Windows (7+), OS X (10.7+), Linux (glibc 2.11+), Raspberry Pi/ARMv6 (glibc 2.13+) operating systems.

####Sync Agent service (odriveagent/odriveagent.exe)
This is the “engine” and runs persistently. It will react to filesystem and remote changes and listen for commands from the odrive CLI.

####odrive CLI ( odrive.py/odrive/odrive.exe )

This is the client interface to the Sync Agent server. The odrive CLI is used from the command-line or from shell scripts to control and enhance Sync Agent behavior. The client is written in python and can be extended and customized to add additional functionality and automation, as well.

Installing odrive Sync Agent

##windows installation

Open up powershell and run the following command to download and install the odrive Sync Agent to .odrive-agent in your user home directory. This will download the odrive Agent service, the odrive Agent python client, and the odrive Agent binary client. The command doesn’t give much feedback, so just give it some time:

$o=“$HOME\.odrive-agent\bin”;mkdir “$o” -FORCE;(New-Object System.Net.WebClient).DownloadFile(“https://dl.odrive.com/odrive-py","$o\odrive.py”);(New-Object System.Net.WebClient).DownloadFile(“https://dl.odrive.com/odriveagent-win”, “$o\o.zip”);$shl=new-object -com shell.application; $shl.namespace(“$o”).copyhere($shl.namespace(“$o\o.zip”).items(),0x10);del “$o\o.zip”;(New-Object System.Net.WebClient).DownloadFile(“https://dl.odrive.com/odrivecli-win”, “$o\oc.zip”);$shl=new-object -com shell.application; $shl.namespace(“$o”).copyhere($shl.namespace(“$o\oc.zip”).items(),0x10);del “$o\oc.zip”;

To run the odrive Sync Agent server in the background, use the following command in powershell:

Start-Job { & “$HOME\.odrive-agent\bin\odriveagent.exe” }

Now the odrive Sync Agent server is running and you can use the odrive CLI to control it. For example, the following command will display the available client commands:

Python client

& python “$HOME\.odrive-agent\bin\odrive.py” -h

Binary client

& “$HOME\.odrive-agent\bin\odrive.exe” -h

The following command will show the current status of the odrive Sync Agent:

Python client

& python “$HOME\.odrive-agent\bin\odrive.py” status

Binary client

& “$HOME\.odrive-agent\bin\odrive.exe” status

###OS X installation

Run the following command in your terminal to download and install the odrive Sync Agent to ~/.odrive-agent. This will download the odrive Agent service, the odrive Agent python client, and the odrive Agent binary client.:

od=“$HOME/.odrive-agent/bin” && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-osx” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-osx” | tar -xvzf- -C “$od/”

To run the odrive Sync Agent server in the background, use the following command in your terminal:

nohup “$HOME/.odrive-agent/bin/odriveagent.app/Contents/MacOS/odriveagent”>/dev/null&

Now the odrive Sync Agent server is running and you can use the odrive CLI to control it. For example, the following command will display the available client commands:

Python client

python “$HOME/.odrive-agent/bin/odrive.py” -h

Binary client

“$HOME/.odrive-agent/bin/odrive” -h

The following command will show the current status of the odrive Sync Agent

Python client

python “$HOME/odrive-agent/bin/odrive.py” status

Binary client

“$HOME/.odrive-agent/bin/odrive” status

##linux/RPi installation

Run the following command in your terminal to download and install the odrive Sync Agent to ~/.odrive-agent. This will download the odrive Agent service, the odrive Agent python client, and the odrive Agent binary client.:

32bit Linux

od=“$HOME/.odrive-agent/bin” && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-lnx-32” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-lnx-32” | tar -xvzf- -C “$od/”

64bit Linux

od=“$HOME/.odrive-agent/bin” && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-lnx-64” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-lnx-64” | tar -xvzf- -C “$od/”

RPi

od=“$HOME/.odrive-agent/bin” && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-rpi” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-rpi” | tar -xvzf- -C “$od/”

To run the odrive Sync Agent server in the background, use the following command in your terminal:

nohup “$HOME/.odrive-agent/bin/odriveagent”>/dev/null&

Now the odrive Sync Agent server is running and you can use the odrive CLI to control it. For example, the following command will display the available client commands:
Python client

python “$HOME/.odrive-agent/bin/odrive.py” -h

Binary client

“$HOME/.odrive-agent/bin/odrive” -h

The following command will show the current status of the odrive Sync Agent

Python client

python “$HOME/odrive-agent/bin/odrive.py” status

Binary client

“$HOME/.odrive-agent/bin/odrive” status

#odrive Sync Agent quick start guide
Now that you have the odrive Sync Agent installed and running, it is time to start playing with it!

Create an odrive account
If you don’t have an odrive account yet, you can get one here: https://www.odrive.com
Create an auth key for odrive Agent
Once you have created an account/signed-in, navigate to odrive | Sign in and click on “Create Auth Key” to create an auth key that you can then use to authenticate your odrive Sync Agent install.
Authenticate odrive Sync Agent

authenticate [auth key]

Python client

python “$HOME/.odrive-agent/bin/odrive.py” authenticate 00000000-0000-0000-0000-000000000000-00000000
```
_OS X/Linux/RPi binary client_
```
“$HOME/.odrive-agent/bin/odrive” authenticate 00000000-0000-0000-0000-000000000000-00000000

Windows binary client

& “$HOME\.odrive-agent\bin\odrive.exe” authenticate 00000000-0000-0000-0000-000000000000-00000000
Create an agent mount
This will mount the root of your odrive to the local folder “odrive-agent-mount”. Make sure the local folder exists before attempting to mount.
Do not use a local folder that is currently used/monitored by the odrive Desktop Client (if you are running both on the same system). Local paths for odrive Agent and odrive Desktop Client need to be separate:

mkdir “$HOME/odrive-agent-mount”

You can then mount and see your odrive links listed in ./odrive-agent-mount

>mount [local folder] [remote odrive folder]

_Python client_
>python "$HOME/.odrive-agent/bin/odrive.py" mount "$HOME/odrive-agent-mount" / 

_OS X/Linux/Rpi binary client_
 >"$HOME/.odrive-agent/bin/odrive" mount "$HOME/odrive-agent-mount" /

_Windows binary client_

>& "$HOME\\.odrive-agent\bin\odrive.exe" mount "$HOME\odrive-agent-mount" /

Sync an odrive folder

sync [path to .cloudf placeholder file]

Python client

python “$HOME/.odrive-agent/bin/odrive.py” sync “$HOME/odrive-agent-mount/Dropbox.cloudf”

OS X/Linux/RPi binary client

“$HOME/.odrive-agent/bin/odrive” sync “$HOME/odrive-agent-mount/Dropbox.cloudf”

Windows binary client

& “$HOME\.odrive-agent\bin\odrive.exe” sync “$HOME\odrive-agent-mount\Dropbox.cloudf”
Sync a file

sync [path to .cloud placeholder file]

Python client

python “$HOME/.odrive-agent/bin/odrive.py” sync “$HOME/odrive-agent-mount/Dropbox/myfile.txt.cloud”

OS X/Linux/RPi binary client

“$HOME/.odrive-agent/bin/odrive” sync “$HOME/odrive-agent-mount/Dropbox/myfile.txt.cloud”

Windows binary client

& “$HOME\.odrive-agent\bin\odrive.exe” sync “$HOME\odrive-agent-mount\Dropbox\myfile.txt.cloud”

#Available commands

authenticate - authenticate odrive with an auth key
mount - mount remote odrive path to a local folder
unmount - remove a mount
sync - sync a placeholder
stream - stream a placeholder eg. stream path | appExecutable -
refresh - refresh a folder
unsync - unsync a file or a folder
syncstate - get sync status info (blue text is Synced, Pink is Active)
status - get status info
deauthorize - deauthorize odrive to unlink the current user and exit
empty trash - empty odrive trash
shutdown - shutdown odrive

status also has the following flags for details:
–mounts get status on mounts
–sync_requests get status on sync requests
–uploads get status on uploads
–downloads get status on downloads
–background get status on background requests
–trash get status of trash items
–waiting get status of waiting items
–not_allowed get status of not allowed items

#Helpful tips
Detailed help
You can get more detailed help on individual commands. For example:

authenticate -h

Watch realtime status

Windows

while(1){& “$HOME\.odrive-agent\bin\odrive.exe” status;start-sleep -seconds 1}

OS X/Linux/RPi

while true; do “$HOME/.odrive-agent/bin/odrive” status; sleep 1; done

Logs
An odrive Sync Agent log file is located at “$HOME/.odrive-agent/log/agent.log”

Streaming
To stream a file, use standard piping commands such as:

“$HOME/.odrive-agent/bin/odrive” stream “placeholderPath” | “playerExecutablePath” -

On windows this works in cmd.exe, but not in powershell

Recursive Sync

Copy, Sync, Unsync Automation

#Direct download links
If you use the install instructions above you will get these versions
: Agent/CLI (Linux, MacOS, Win)

Tony · September 22, 2016, 12:19am

79 posts were merged into an existing topic: Odrive Agent Troubleshooting, Feedback, and Questions

Tony · April 5, 2016, 4:47pm

Linux/MacOS bash-based:

This uses find to recurse and sync. The output is not ideal, though, since it is all queued up. You would need to change the path from ~/odrive-agent-mount/Dropbox/ to whatever you require, of course:

output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;); echo $output; done

This second one uses an extra file descriptor trick to push the output out in realtime. Again, you will need to change the path:

exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;|tee /dev/fd/6); done

If you only wanted to sync the files immediately inside the specified folder:
exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -maxdepth 1 -name "*.cloud" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;|tee /dev/fd/6); done

Adding parallelization to the mix with xargs, contributed by @stuckj. num_procs = max # of parallel processes (set to 4 in the example):
exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done

Dynamically adjust parallel processing “pool” using SIGUSR1(10) and SIGUSR2(12), contributed by @paradox606:
Decrease parallelism by 1 (down to 1 process): kill -12 $(pidof xargs)
Increase parallelism by 1 (up to max specified in xargs -P): kill -10 $(pidof xargs)

Windows Batch:

Batch via @bojandjuric:

Place into a .bat file

@echo off cd /d %~dp0 set FOLDERPATH="Folder path to sync here" set ODRIVEBIN="odrive cli binary client path here" for /r "%FOLDERPATH%" %%i in (*.cloudf) do "%ODRIVEBIN%" sync "%%i" for /r "%FOLDERPATH%" %%i in (*.cloud) do "%ODRIVEBIN%" sync "%%i"

Windows Powershell
One-liner script (run via a standard command prompt):

powershell -command "& {$syncpath=\"Folder path to sync here\";$syncbin=\"odrive cli binary client path here\";while ((Get-ChildItem $syncpath -Filter \"*.cloud*\" -Recurse | Measure-Object).Count){Get-ChildItem -Path \"$syncpath\" -Filter \"*.cloud*\" -Recurse | % {echo \"Syncing: $($_.FullName)n";& “$syncbin” “sync” “$($_.FullName)”;}}}"`

Python:
I forked the official odrive CLI and added recursive sync here:

shawnholt · April 13, 2016, 8:53pm

Will a folder be automatically syncd? What is the interval for agent to check? What does refresh do?

shawnholt · April 13, 2016, 8:59pm

Feature request:
I’d like to use folder for linux backups that will sync to cloud and then unsync from local (when operation is complete) Ideally this would be an option on the folder. So any items put in the folder will automatically sync (ASAP after file is written/closed) and then automatically unsync. I assume this could be scripted but lot’s of crazy polling of status.

Tony · April 15, 2016, 2:16am

Folders within the scope of agent will automatically sync. Refresh will request that odrive refresh the targeted directory, which will scan for changes and sync if any are found.

Tony · April 15, 2016, 5:28am

For your backup and unsync use case, you can script something like this in bash:

#!/bin/bash
agentdir="$HOME/.odrive-agent/bin"
export agentdir
agentmount="$HOME/odrive/Dropbox (Personal)"
targetdir="$agentmount/temp"
sourcedir="$HOME/Downloads/test_directory"
python "$agentdir/odrive.py" sync "$targetdir.cloudf"
cp -R "$sourcedir" "$targetdir/"
python "$agentdir/odrive.py" refresh "$targetdir"
echo Waiting for odrive to start syncing $targetdir ...
while [ "$(python "$agentdir/odrive.py" syncstate "$targetdir" | head -n 1)" == "Synced" ]; do
    sleep 5
done
echo odrive is syncing changes in $targetdir ...
while [ "$(python "$agentdir/odrive.py" syncstate "$targetdir" | head -n 1)" != "Synced" ]; do
    find "$targetdir" ! -name '*.cloud*' -type file -exec bash -c 'thepath="$1"
    echo "$(python "$agentdir/odrive.py" syncstate "$thepath" | head -n 1)"
    if [[ "$(python "$agentdir/odrive.py" syncstate "$thepath" | head -n 1)" == "Synced" ]]; then 
    	echo "$thepath" is synced! Unsyncing...
    	python "$agentdir/odrive.py" unsync "$thepath"
    fi 
    ' _ {} \;
    sleep 5
done
echo Sync complete! Unsyncing $targetdir ...
python "$agentdir/odrive.py" unsync "$targetdir"

This will first sync the target directory residing inside the agent mount (since it is assumed that it is starting in an unsynced state) and then copy a source directory into the target directory. It will then wait until syncing starts. Once syncing starts, it will begin looking for files that have finished syncing and unsync them if they have. At the end it unsyncs the whole target directory.

I did this quick and only tested a couple of times on OS X, so think of it more as a starting point, but it may get you to a simple solution. It does do some polling, but it it isn’t a problem.

nevrax · May 28, 2016, 11:58am

odrive sync agent is awesome!

this is pure gold! i always wanted something like odrive. is exactly as described the way it sould be

here i’ve made few osx bash scripts for some heavy duty amazon cloud drive sync. i need to be in control of what i sync at some point. here’s my magic. use it. adapt it. make it better. share it

scripts

~/odrive-sync/odrive-sync.sh finds .cloudf folders

#!/bin/sh
find ~/odrive/amzdrive-sync/cloud-sync -type f -name *.cloudf -exec echo odrive sync \"{}\" ";"

~/odrive-sync/odrive-sync-log.sh logs output of sync script and compress the log

#!/bin/sh

log=`date "+/full/path/log/dir/odrive-sync/logs/%Y%m%d%H%M%S.odrive.log"`

echo sync started. check $log
time ~/odrive-sync/odrive-sync.sh | sh >> $log 2>&1

echo sync log
echo -----
cat $log
echo -----
echo compressing log
gzip $log
echo sync completed

usage example

mbp2k9:~ nevrax$ ~/odrive-sync/odrive-sync-log.sh 
sync started. check /full/path/log/dir/odrive-sync/logs/20160528143123.odrive.log

real	0m55.131s
user	0m4.013s
sys	0m2.393s
sync log
-----
/secret/path/2014-10 auckland new zealand
/secret/path/2014-10 boat trip riding bicycles gili trawangan gopro
/secret/path/2014-10 gili trawangan sergiu gopro
/secret/path/2014-10 gili trawangan
/secret/path/2014-10 glowing worms new zealand
/secret/path/2014-10 hobiton new zealand
/secret/path/2014-10 hot water beach new zealand
/secret/path/2014-10 kuta bali
/secret/path/2014-10 pool timelapse sankara ubud
/secret/path/2014-10 reverse bungee hot water beach hobiton gopro
/secret/path/2014-10 romantic timelapse sankara ubud
/secret/path/2014-10 sam nada bali
/secret/path/2014-10 sanur bali
/secret/path/2014-10 seawalk bali indonesia extra
/secret/path/2014-10 seawalk bali indonesia gopro
/secret/path/2014-10 snorkeling gili trawangan gopro
/secret/path/2014-10 sydney australia
/secret/path/2014-10 tamaki maori village new zealand
/secret/path/2014-10 taupo new zealand
/secret/path/2014-10 timelapse sankara ubud
/secret/path/2014-10 ubud bali
/secret/path/2014-10 wai o tapu geothermal wonderland new zealand
/secret/path/2014-10 wai o tapu waikite thermal pool gopro
/secret/path/2014-10 wellington nelson new zealand
-----
compressing log
sync completed

check the logs

mbp2k9:~ nevrax$ ls ~/odrive-sync/logs
20160528140745.odrive.log.gz	20160528142143.odrive.log.gz
20160528140758.odrive.log.gz	20160528142211.odrive.log.gz
20160528141344.odrive.log.gz	20160528142217.odrive.log.gz
20160528141407.odrive.log.gz	20160528142454.odrive.log.gz
20160528141639.odrive.log.gz	20160528142508.odrive.log.gz
20160528141640.odrive.log.gz	20160528142515.odrive.log.gz
20160528141652.odrive.log.gz	20160528142547.odrive.log.gz
20160528141655.odrive.log.gz	20160528142549.odrive.log.gz
20160528141732.odrive.log.gz	20160528142553.odrive.log.gz
20160528141757.odrive.log.gz	20160528142557.odrive.log.gz
20160528141904.odrive.log.gz	20160528142612.odrive.log.gz
20160528141924.odrive.log.gz	20160528142829.odrive.log.gz
20160528141936.odrive.log.gz	20160528142907.odrive.log.gz
20160528141943.odrive.log.gz	20160528143118.odrive.log.gz
20160528142002.odrive.log.gz	20160528143123.odrive.log.gz
20160528142040.odrive.log.gz	20160528143223.odrive.log.gz
20160528142112.odrive.log.gz	20160528143245.odrive.log.gz
20160528142118.odrive.log.gz	20160528143332.odrive.log.gz
20160528142125.odrive.log.gz	20160528143407.odrive.log.gz
20160528142129.odrive.log.gz

i’m so happy i can finally do my photo stuff and spend less time on api bla bla stuff or stupid amazon “not for mortals” cloud drive api access key. seems like you need a separate api key and wait for their approval to access your own data. wtf amazon?

thanks odrive

Tony · May 30, 2016, 3:49pm

Thanks for the feedback and for contributing with your own use case and script!

graham.spinks · June 12, 2016, 9:33pm

Hi all,
In case it helps anybody, here is a simple recursive refresh script that will recursive run through all your directories and subdirectories and upload any files that might have been changed:
find ~/odrive-agent-mount/ -type d -exec python "$HOME/.odrive-agent/bin/odrive.py" refresh "{}" \;

(change ~/odrive-agent-mount/ to wherever your odrive is mounted)
schedule the task in cron on linux to run it every 15 minutes or so.

I did run into a strange issue where I renamed a folder and the text file in it, and the textfile didn’t upload anymore. Any new text files I added to that folder didn’t upload either. I tried to reproduce the problem with another folder but wasn’t able to. If anybody knows what the issue might have been, let me know!

Tony · June 18, 2016, 12:09am

Just want to let folks know that we’ve released the CLI to the desktop client, which many have requested. I just added a blog post that dives into it a bit. The content in there also applies to Agent, so double whammy there.

We also added a CLI section in our usage guide:

stuckj · July 1, 2016, 5:39pm

FYI, if you want parallelized recursive downloads, here’s a modification to the recursive oneliner mentioned above. If you have a LOT of files you’re trying to sync this will be much faster.

exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P 10 "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done

paradox606 · July 8, 2016, 11:56am

stuckj,

I too am using xargs to parallelize the syncs. One useful tidbit, if you are saturating your internet connection, and want to ease off during work hours, you can fire a SIGUSR2 at the xargs process to tell it to reduce its parallelism by one, so hit it 9 times with kill -12 $(pidof xargs) (on linux) and any xargs processes will be reduced by 9 processcount.

When you are ready to ratchet it back up, use kill -10 $(pidof xargs) 9 times to get back to 10.

I’ve had a sync running for a few days and this way it doesn’t stop in the middle of a file, rather xargs just stops running the parallel job count when current jobs finish.

Tony · July 8, 2016, 3:13pm

Thanks for the contributions @stuckj and @paradox606. I added them to the post:

eprisencc · July 17, 2016, 1:54pm

How do I uninstall the Linux sync agent? I cannot find a command to uninstall this.

Tony · July 18, 2016, 4:29pm

If you followed the instructions then the only 2 folders you would need to remove are:
~/.odrive-agent (which contains the application information)
~/odrive-agent-mount (which contains the odrive files/folders)

Tony · August 9, 2016, 12:37am

A post was split to a new topic: Peer to peer transfer capabilities (LAN sync)

Tony · September 22, 2016, 12:21am

In an attempt to declutter this thread a bit, I have moved a lot of the Questions and Troubleshooting to this thread:
https://forum.odrive.com/t/odrive-agent-troubleshooting-feedback-and-questions/1297

If you have issues or questions, please post there.

Thanks!

Tony · October 1, 2016, 12:58am

New release of Agent is available to bring it into parity with the fixes that have been released for the Desktop. We also now have a Raspberry Pi (ARMv6) compatible build!