Odrive Sync Agent: A CLI/scriptable interface for odrive's Progressive Sync Engine for Linux, OS X, and Windows


#1

Overview

The odrive Sync Agent provides advanced users with a flexible, scriptable interface to odrive’s powerful Progressive Sync engine. Use it to bring cloud access to almost any environment.

The odrive Sync Agent can be used on Windows (7+), OS X (10.7+), Linux (glibc 2.11+), Raspberry Pi/ARMv6 (glibc 2.13+) operating systems.

####Sync Agent service (odriveagent/odriveagent.exe)
This is the “engine” and runs persistently. It will react to filesystem and remote changes and listen for commands from the odrive CLI.

####odrive CLI ( odrive.py/odrive/odrive.exe )

This is the client interface to the Sync Agent server. The odrive CLI is used from the command-line or from shell scripts to control and enhance Sync Agent behavior. The client is written in python and can be extended and customized to add additional functionality and automation, as well.

Installing odrive Sync Agent


###Windows installation

Open up powershell and run the following command to download and install the odrive Sync Agent to .odrive-agent in your user home directory. This will download the odrive Agent service, the odrive Agent python client, and the odrive Agent binary client. The command doesn’t give much feedback, so just give it some time:

$o="$HOME\.odrive-agent\bin";mkdir “$o” -FORCE;(New-Object System.Net.WebClient).DownloadFile(“https://dl.odrive.com/odrive-py","$o\odrive.py”);(New-Object System.Net.WebClient).DownloadFile(“https://dl.odrive.com/odriveagent-win”, “$o\o.zip”);$shl=new-object -com shell.application; $shl.namespace("$o").copyhere($shl.namespace("$o\o.zip").items(),0x10);del “$o\o.zip”;(New-Object System.Net.WebClient).DownloadFile(“https://dl.odrive.com/odrivecli-win”, “$o\oc.zip”);$shl=new-object -com shell.application; $shl.namespace("$o").copyhere($shl.namespace("$o\oc.zip").items(),0x10);del “$o\oc.zip”;

To run the odrive Sync Agent server in the background, use the following command in powershell:

Start-Job { & “$HOME\.odrive-agent\bin\odriveagent.exe” }

Now the odrive Sync Agent server is running and you can use the odrive CLI to control it. For example, the following command will display the available client commands:

Python client

& python “$HOME\.odrive-agent\bin\odrive.py” -h

Binary client

& “$HOME\.odrive-agent\bin\odrive.exe” -h

The following command will show the current status of the odrive Sync Agent:

Python client

& python “$HOME\.odrive-agent\bin\odrive.py” status

Binary client

& “$HOME\.odrive-agent\bin\odrive.exe” status


###OS X installation

Run the following command in your terminal to download and install the odrive Sync Agent to ~/.odrive-agent. This will download the odrive Agent service, the odrive Agent python client, and the odrive Agent binary client.:

od="$HOME/.odrive-agent/bin" && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-osx” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-osx” | tar -xvzf- -C “$od/”

To run the odrive Sync Agent server in the background, use the following command in your terminal:

nohup “$HOME/.odrive-agent/bin/odriveagent.app/Contents/MacOS/odriveagent”>/dev/null&

Now the odrive Sync Agent server is running and you can use the odrive CLI to control it. For example, the following command will display the available client commands:

Python client

python “$HOME/.odrive-agent/bin/odrive.py” -h

Binary client

“$HOME/.odrive-agent/bin/odrive” -h

The following command will show the current status of the odrive Sync Agent

Python client

python “$HOME/odrive-agent/bin/odrive.py” status

Binary client

“$HOME/.odrive-agent/bin/odrive” status


###Linux/RPi installation

Run the following command in your terminal to download and install the odrive Sync Agent to ~/.odrive-agent. This will download the odrive Agent service, the odrive Agent python client, and the odrive Agent binary client.:

32bit Linux

od="$HOME/.odrive-agent/bin" && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-lnx-32” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-lnx-32” | tar -xvzf- -C “$od/”

64bit Linux

od="$HOME/.odrive-agent/bin" && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-lnx-64” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-lnx-64” | tar -xvzf- -C “$od/”

RPi

od="$HOME/.odrive-agent/bin" && curl -L “https://dl.odrive.com/odrive-py” --create-dirs -o “$od/odrive.py” && curl -L “https://dl.odrive.com/odriveagent-rpi” | tar -xvzf- -C “$od/” && curl -L “https://dl.odrive.com/odrivecli-rpi” | tar -xvzf- -C “$od/”

To run the odrive Sync Agent server in the background, use the following command in your terminal:

nohup “$HOME/.odrive-agent/bin/odriveagent”>/dev/null&

Now the odrive Sync Agent server is running and you can use the odrive CLI to control it. For example, the following command will display the available client commands:
Python client

python “$HOME/.odrive-agent/bin/odrive.py” -h

Binary client

“$HOME/.odrive-agent/bin/odrive” -h

The following command will show the current status of the odrive Sync Agent

Python client

python “$HOME/odrive-agent/bin/odrive.py” status

Binary client

“$HOME/.odrive-agent/bin/odrive” status


#odrive Sync Agent quick start guide
Now that you have the odrive Sync Agent installed and running, it is time to start playing with it!

  1. Create an odrive account
    If you don’t have an odrive account yet, you can get one here: https://www.odrive.com

  2. Create an auth key for odrive Agent
    Once you have created an account/signed-in, navigate to https://www.odrive.com/account/authcodes and click on “Create Auth Key” to create an auth key that you can then use to authenticate your odrive Sync Agent install.

  3. Authenticate odrive Sync Agent

    authenticate [auth key]

    Python client

    python “$HOME/.odrive-agent/bin/odrive.py” authenticate 00000000-0000-0000-0000-000000000000-00000000

    _OS X/Linux/RPi binary client_
    

    “$HOME/.odrive-agent/bin/odrive” authenticate 00000000-0000-0000-0000-000000000000-00000000

    Windows binary client

    & “$HOME\.odrive-agent\bin\odrive.exe” authenticate 00000000-0000-0000-0000-000000000000-00000000

  4. Create an agent mount
    This will mount the root of your odrive to the local folder “odrive-agent-mount”. Make sure the local folder exists before attempting to mount.
    Do not use a local folder that is currently used/monitored by the odrive Desktop Client (if you are running both on the same system). Local paths for odrive Agent and odrive Desktop Client need to be separate:

mkdir “$HOME/odrive-agent-mount”

You can then mount and see your odrive links listed in ./odrive-agent-mount

>mount [local folder] [remote odrive folder]

_Python client_
>python "$HOME/.odrive-agent/bin/odrive.py" mount "$HOME/odrive-agent-mount" / 

_OS X/Linux/Rpi binary client_
 >"$HOME/.odrive-agent/bin/odrive" mount "$HOME/odrive-agent-mount" /

_Windows binary client_

>& "$HOME\\.odrive-agent\bin\odrive.exe" mount "$HOME\odrive-agent-mount" /
  1. Sync an odrive folder

    sync [path to .cloudf placeholder file]

    Python client

    python “$HOME/.odrive-agent/bin/odrive.py” sync “$HOME/odrive-agent-mount/Dropbox.cloudf”

    OS X/Linux/RPi binary client

    “$HOME/.odrive-agent/bin/odrive” sync “$HOME/odrive-agent-mount/Dropbox.cloudf”

    Windows binary client

    & “$HOME\.odrive-agent\bin\odrive.exe” sync “$HOME\odrive-agent-mount\Dropbox.cloudf”

  2. Sync a file

    sync [path to .cloud placeholder file]

    Python client

    python “$HOME/.odrive-agent/bin/odrive.py” sync “$HOME/odrive-agent-mount/Dropbox/myfile.txt.cloud”

    OS X/Linux/RPi binary client

    “$HOME/.odrive-agent/bin/odrive” sync “$HOME/odrive-agent-mount/Dropbox/myfile.txt.cloud”

    Windows binary client

    & “$HOME\.odrive-agent\bin\odrive.exe” sync “$HOME\odrive-agent-mount\Dropbox\myfile.txt.cloud”


#Available commands

authenticate - authenticate odrive with an auth key
mount - mount remote odrive path to a local folder
unmount - remove a mount
sync - sync a placeholder
stream - stream a placeholder eg. stream path | appExecutable -
refresh - refresh a folder
unsync - unsync a file or a folder
syncstate - get sync status info (blue text is Synced, Pink is Active)
status - get status info
deauthorize - deauthorize odrive to unlink the current user and exit
empty trash - empty odrive trash
shutdown - shutdown odrive

status also has the following flags for details:
–mounts get status on mounts
–sync_requests get status on sync requests
–uploads get status on uploads
–downloads get status on downloads
–background get status on background requests
–trash get status of trash items
–waiting get status of waiting items
–not_allowed get status of not allowed items


#Helpful tips
Detailed help
You can get more detailed help on individual commands. For example:

authenticate -h

Watch realtime status

Windows

while(1){& “$HOME\.odrive-agent\bin\odrive.exe” status;start-sleep -seconds 1}

OS X/Linux/RPi

while true; do “$HOME/.odrive-agent/bin/odrive” status; sleep 1; done

Logs
An odrive Sync Agent log file is located at “$HOME/.odrive-agent/log/agent.log”

Streaming
To stream a file, use standard piping commands such as:

“$HOME/.odrive-agent/bin/odrive” stream “placeholderPath” | “playerExecutablePath” -

On windows this works in cmd.exe, but not in powershell

Recursive Sync

Copy, Sync, Unsync Automation


#Direct download links
If you use the install instructions above you will get these versions
: https://docs.odrive.com/docs/odrive-sync-agent#section--download-sync-agent-


Needs to get Linux support (Ubuntu, CentOS, RedHat )
Using Plex with oDrive and CloudStorage? Encrypted Cloud Storage?
Amazon Cloud Drive as "Live Drive", not synced?
Stream media as it syncs
Feature request: allow scripting odrive folder
Odrive Agent/CLI Troubleshooting, Feedback, and Questions
FTP Sync LAN Only
Using Plex with oDrive and CloudStorage? Encrypted Cloud Storage?
Multiple Clients to same Sync Location
File Size Limit for Google Drive
Sync Agent RPI Urllib3-Error
Odrive Agent/CLI Troubleshooting, Feedback, and Questions
Odrive Agent/CLI Troubleshooting, Feedback, and Questions
Reset my odrive install and getting "This file does not exist" messages anytime I try to sync anything
Odrive Agent/CLI Troubleshooting, Feedback, and Questions
I want to stop using Amazon Cloud Drive, how do I download all .cloud files
ARM Linux installation
Odrive Agent/CLI Troubleshooting, Feedback, and Questions
Suggested feature enhancements for encryption
macOS Sierra compatibility
Install as admin
Download links broken in usage guide
Recursive refresh script using linux python agent
New desktop release today! (06-10-2016) - Support for Agent CLI on Desktop Sync Client
Does Amazon Cloud with Odrive function like Drop Box?
Is it Possible to Manually throttle odrive even more?
#4

79 posts were merged into an existing topic: Odrive Agent Troubleshooting, Feedback, and Questions


#13

Linux/MacOS bash-based:


This uses find to recurse and sync. The output is not ideal, though, since it is all queued up. You would need to change the path from ~/odrive-agent-mount/Dropbox/ to whatever you require, of course:

output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;); echo $output; done


This second one uses an extra file descriptor trick to push the output out in realtime. Again, you will need to change the path:

exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;|tee /dev/fd/6); done

If you only wanted to sync the files immediately inside the specified folder:
exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -maxdepth 1 -name "*.cloud" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;|tee /dev/fd/6); done


Adding parallelization to the mix with xargs, contributed by @stuckj. num_procs = max # of parallel processes (set to 10 in the example):
exec 6>&1;num_procs=10;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done

Dynamically adjust parallel processing “pool” using SIGUSR1(10) and SIGUSR2(12), contributed by @paradox606:
Decrease parallelism by 1 (down to 1 process): kill -12 $(pidof xargs)
Increase parallelism by 1 (up to max specified in xargs -P): kill -10 $(pidof xargs)


Windows Batch:

Batch via @bojandjuric:

Place into a .bat file

@echo off cd /d %~dp0 set FOLDERPATH="Folder path to sync here" set ODRIVEBIN="odrive cli binary client path here" for /r "%FOLDERPATH%" %%i in (*.cloudf) do "%ODRIVEBIN%" sync "%%i" for /r "%FOLDERPATH%" %%i in (*.cloud) do "%ODRIVEBIN%" sync "%%i"


Windows Powershell
One-liner script (run via a standard command prompt):

powershell -command "& {$syncpath=\"Folder path to sync here\";$syncbin=\"odrive cli binary client path here\";while ((Get-ChildItem $syncpath -Filter \"*.cloud*\" -Recurse | Measure-Object).Count){Get-ChildItem -Path \"$syncpath\" -Filter \"*.cloud*\" -Recurse | % {echo \"Syncing: $($_.FullName)n";& “$syncbin” “sync” “$($_.FullName)”;}}}"`


Python:
I forked the official odrive CLI and added recursive sync here:


In Linux agent how can I sync an entire OneDrive folder with one command?
Xubuntu 16.04 odrive not working: SOS!
Method to SPEED up syncs/unsync with your cloud (almost!)
Extensions not loading in OS X Sierra
macOS Sierra compatibility
Local to remote syncing problems
Odrive Agent/CLI Troubleshooting, Feedback, and Questions
Linux (using cli): how to sync (and keep it synced) a selected folder and all its content
#23

Will a folder be automatically syncd? What is the interval for agent to check? What does refresh do?


#24

Feature request:
I’d like to use folder for linux backups that will sync to cloud and then unsync from local (when operation is complete) Ideally this would be an option on the folder. So any items put in the folder will automatically sync (ASAP after file is written/closed) and then automatically unsync. I assume this could be scripted but lot’s of crazy polling of status.


#25

Folders within the scope of agent will automatically sync. Refresh will request that odrive refresh the targeted directory, which will scan for changes and sync if any are found.


#26

For your backup and unsync use case, you can script something like this in bash:

#!/bin/bash
agentdir="$HOME/.odrive-agent/bin"
export agentdir
agentmount="$HOME/odrive/Dropbox (Personal)"
targetdir="$agentmount/temp"
sourcedir="$HOME/Downloads/test_directory"
python "$agentdir/odrive.py" sync "$targetdir.cloudf"
cp -R "$sourcedir" "$targetdir/"
python "$agentdir/odrive.py" refresh "$targetdir"
echo Waiting for odrive to start syncing $targetdir ...
while [ "$(python "$agentdir/odrive.py" syncstate "$targetdir" | head -n 1)" == "Synced" ]; do
    sleep 5
done
echo odrive is syncing changes in $targetdir ...
while [ "$(python "$agentdir/odrive.py" syncstate "$targetdir" | head -n 1)" != "Synced" ]; do
    find "$targetdir" ! -name '*.cloud*' -type file -exec bash -c 'thepath="$1"
    echo "$(python "$agentdir/odrive.py" syncstate "$thepath" | head -n 1)"
    if [[ "$(python "$agentdir/odrive.py" syncstate "$thepath" | head -n 1)" == "Synced" ]]; then 
    	echo "$thepath" is synced! Unsyncing...
    	python "$agentdir/odrive.py" unsync "$thepath"
    fi 
    ' _ {} \;
    sleep 5
done
echo Sync complete! Unsyncing $targetdir ...
python "$agentdir/odrive.py" unsync "$targetdir"

This will first sync the target directory residing inside the agent mount (since it is assumed that it is starting in an unsynced state) and then copy a source directory into the target directory. It will then wait until syncing starts. Once syncing starts, it will begin looking for files that have finished syncing and unsync them if they have. At the end it unsyncs the whole target directory.

I did this quick and only tested a couple of times on OS X, so think of it more as a starting point, but it may get you to a simple solution. It does do some polling, but it it isn’t a problem.


Best way to sync over 400,000 files/directories
Reset my odrive install and getting "This file does not exist" messages anytime I try to sync anything
#53

odrive sync agent is awesome!

this is pure gold! i always wanted something like odrive. is exactly as described the way it sould be :relieved:

here i’ve made few osx bash scripts for some heavy duty amazon cloud drive sync. i need to be in control of what i sync at some point. here’s my magic. use it. adapt it. make it better. share it :blush:

scripts

~/odrive-sync/odrive-sync.sh finds .cloudf folders

#!/bin/sh
find ~/odrive/amzdrive-sync/cloud-sync -type f -name *.cloudf -exec echo odrive sync \"{}\" ";"

~/odrive-sync/odrive-sync-log.sh logs output of sync script and compress the log

#!/bin/sh

log=`date "+/full/path/log/dir/odrive-sync/logs/%Y%m%d%H%M%S.odrive.log"`

echo sync started. check $log
time ~/odrive-sync/odrive-sync.sh | sh >> $log 2>&1

echo sync log
echo -----
cat $log
echo -----
echo compressing log
gzip $log
echo sync completed

usage example

mbp2k9:~ nevrax$ ~/odrive-sync/odrive-sync-log.sh 
sync started. check /full/path/log/dir/odrive-sync/logs/20160528143123.odrive.log

real	0m55.131s
user	0m4.013s
sys	0m2.393s
sync log
-----
/secret/path/2014-10 auckland new zealand
/secret/path/2014-10 boat trip riding bicycles gili trawangan gopro
/secret/path/2014-10 gili trawangan sergiu gopro
/secret/path/2014-10 gili trawangan
/secret/path/2014-10 glowing worms new zealand
/secret/path/2014-10 hobiton new zealand
/secret/path/2014-10 hot water beach new zealand
/secret/path/2014-10 kuta bali
/secret/path/2014-10 pool timelapse sankara ubud
/secret/path/2014-10 reverse bungee hot water beach hobiton gopro
/secret/path/2014-10 romantic timelapse sankara ubud
/secret/path/2014-10 sam nada bali
/secret/path/2014-10 sanur bali
/secret/path/2014-10 seawalk bali indonesia extra
/secret/path/2014-10 seawalk bali indonesia gopro
/secret/path/2014-10 snorkeling gili trawangan gopro
/secret/path/2014-10 sydney australia
/secret/path/2014-10 tamaki maori village new zealand
/secret/path/2014-10 taupo new zealand
/secret/path/2014-10 timelapse sankara ubud
/secret/path/2014-10 ubud bali
/secret/path/2014-10 wai o tapu geothermal wonderland new zealand
/secret/path/2014-10 wai o tapu waikite thermal pool gopro
/secret/path/2014-10 wellington nelson new zealand
-----
compressing log
sync completed

check the logs

mbp2k9:~ nevrax$ ls ~/odrive-sync/logs
20160528140745.odrive.log.gz	20160528142143.odrive.log.gz
20160528140758.odrive.log.gz	20160528142211.odrive.log.gz
20160528141344.odrive.log.gz	20160528142217.odrive.log.gz
20160528141407.odrive.log.gz	20160528142454.odrive.log.gz
20160528141639.odrive.log.gz	20160528142508.odrive.log.gz
20160528141640.odrive.log.gz	20160528142515.odrive.log.gz
20160528141652.odrive.log.gz	20160528142547.odrive.log.gz
20160528141655.odrive.log.gz	20160528142549.odrive.log.gz
20160528141732.odrive.log.gz	20160528142553.odrive.log.gz
20160528141757.odrive.log.gz	20160528142557.odrive.log.gz
20160528141904.odrive.log.gz	20160528142612.odrive.log.gz
20160528141924.odrive.log.gz	20160528142829.odrive.log.gz
20160528141936.odrive.log.gz	20160528142907.odrive.log.gz
20160528141943.odrive.log.gz	20160528143118.odrive.log.gz
20160528142002.odrive.log.gz	20160528143123.odrive.log.gz
20160528142040.odrive.log.gz	20160528143223.odrive.log.gz
20160528142112.odrive.log.gz	20160528143245.odrive.log.gz
20160528142118.odrive.log.gz	20160528143332.odrive.log.gz
20160528142125.odrive.log.gz	20160528143407.odrive.log.gz
20160528142129.odrive.log.gz

i’m so happy i can finally do my photo stuff and spend less time on api bla bla stuff or stupid amazon “not for mortals” cloud drive api access key. seems like you need a separate api key and wait for their approval to access your own data. wtf amazon? :confused:

thanks odrive :heart_eyes:


#54

Thanks for the feedback and for contributing with your own use case and script!


#62

Hi all,
In case it helps anybody, here is a simple recursive refresh script that will recursive run through all your directories and subdirectories and upload any files that might have been changed:
find ~/odrive-agent-mount/ -type d -exec python “$HOME/.odrive-agent/bin/odrive.py” refresh “{}” ;

(change ~/odrive-agent-mount/ to wherever your odrive is mounted)
schedule the task in cron on linux to run it every 15 minutes or so.

I did run into a strange issue where I renamed a folder and the text file in it, and the textfile didn’t upload anymore. Any new text files I added to that folder didn’t upload either. I tried to reproduce the problem with another folder but wasn’t able to. If anybody knows what the issue might have been, let me know!


I am seriously confused how Odrive Agent is suppose to work in linux
#64

Just want to let folks know that we’ve released the CLI to the desktop client, which many have requested. I just added a blog post that dives into it a bit. The content in there also applies to Agent, so double whammy there. :slight_smile:
https://medium.odrive.com/sync-client-magic-602d858731de

We also added a CLI section in our usage guide:


#68

FYI, if you want parallelized recursive downloads, here’s a modification to the recursive oneliner mentioned above. If you have a LOT of files you’re trying to sync this will be much faster.

exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P 10 "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done


#69

stuckj,

I too am using xargs to parallelize the syncs. One useful tidbit, if you are saturating your internet connection, and want to ease off during work hours, you can fire a SIGUSR2 at the xargs process to tell it to reduce its parallelism by one, so hit it 9 times with kill -12 $(pidof xargs) (on linux) and any xargs processes will be reduced by 9 processcount.

When you are ready to ratchet it back up, use kill -10 $(pidof xargs) 9 times to get back to 10.

I’ve had a sync running for a few days and this way it doesn’t stop in the middle of a file, rather xargs just stops running the parallel job count when current jobs finish.


#70

Thanks for the contributions @stuckj and @paradox606. I added them to the post:


#81

How do I uninstall the Linux sync agent? I cannot find a command to uninstall this.


#83

If you followed the instructions then the only 2 folders you would need to remove are:
~/.odrive-agent (which contains the application information)
~/odrive-agent-mount (which contains the odrive files/folders)


#84

#88

A post was split to a new topic: Peer to peer transfer capabilities (LAN sync)


#89

In an attempt to declutter this thread a bit, I have moved a lot of the Questions and Troubleshooting to this thread:

If you have issues or questions, please post there.

Thanks!


#90

New release of Agent is available to bring it into parity with the fixes that have been released for the Desktop. We also now have a Raspberry Pi (ARMv6) compatible build!