Odrive Sync Agent: A CLI/scriptable interface for odrive's Progressive Sync Engine for Linux, OS X, and Windows

79 posts were merged into an existing topic: Odrive Agent Troubleshooting, Feedback, and Questions

Linux/MacOS bash-based:


This uses find to recurse and sync. The output is not ideal, though, since it is all queued up. You would need to change the path from ~/odrive-agent-mount/Dropbox/ to whatever you require, of course:

output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;); echo $output; done


This second one uses an extra file descriptor trick to push the output out in realtime. Again, you will need to change the path:

exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;|tee /dev/fd/6); done

If you only wanted to sync the files immediately inside the specified folder:
exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -maxdepth 1 -name "*.cloud" -exec python "$HOME/.odrive-agent/bin/odrive.py" sync "{}" \;|tee /dev/fd/6); done


Adding parallelization to the mix with xargs, contributed by @stuckj. num_procs = max # of parallel processes (set to 4 in the example):
exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done

Dynamically adjust parallel processing “pool” using SIGUSR1(10) and SIGUSR2(12), contributed by @paradox606:
Decrease parallelism by 1 (down to 1 process): kill -12 $(pidof xargs)
Increase parallelism by 1 (up to max specified in xargs -P): kill -10 $(pidof xargs)


Windows Batch:

Batch via @bojandjuric:

Place into a .bat file

@echo off cd /d %~dp0 set FOLDERPATH="Folder path to sync here" set ODRIVEBIN="odrive cli binary client path here" for /r "%FOLDERPATH%" %%i in (*.cloudf) do "%ODRIVEBIN%" sync "%%i" for /r "%FOLDERPATH%" %%i in (*.cloud) do "%ODRIVEBIN%" sync "%%i"


Windows Powershell
One-liner script (run via a standard command prompt):

powershell -command "& {$syncpath=\"Folder path to sync here\";$syncbin=\"odrive cli binary client path here\";while ((Get-ChildItem $syncpath -Filter \"*.cloud*\" -Recurse | Measure-Object).Count){Get-ChildItem -Path \"$syncpath\" -Filter \"*.cloud*\" -Recurse | % {echo \"Syncing: $($_.FullName)n";& “$syncbin” “sync” “$($_.FullName)”;}}}"`


Python:
I forked the official odrive CLI and added recursive sync here:

1 Like

Will a folder be automatically syncd? What is the interval for agent to check? What does refresh do?

Feature request:
I’d like to use folder for linux backups that will sync to cloud and then unsync from local (when operation is complete) Ideally this would be an option on the folder. So any items put in the folder will automatically sync (ASAP after file is written/closed) and then automatically unsync. I assume this could be scripted but lot’s of crazy polling of status.

Folders within the scope of agent will automatically sync. Refresh will request that odrive refresh the targeted directory, which will scan for changes and sync if any are found.

For your backup and unsync use case, you can script something like this in bash:

#!/bin/bash
agentdir="$HOME/.odrive-agent/bin"
export agentdir
agentmount="$HOME/odrive/Dropbox (Personal)"
targetdir="$agentmount/temp"
sourcedir="$HOME/Downloads/test_directory"
python "$agentdir/odrive.py" sync "$targetdir.cloudf"
cp -R "$sourcedir" "$targetdir/"
python "$agentdir/odrive.py" refresh "$targetdir"
echo Waiting for odrive to start syncing $targetdir ...
while [ "$(python "$agentdir/odrive.py" syncstate "$targetdir" | head -n 1)" == "Synced" ]; do
    sleep 5
done
echo odrive is syncing changes in $targetdir ...
while [ "$(python "$agentdir/odrive.py" syncstate "$targetdir" | head -n 1)" != "Synced" ]; do
    find "$targetdir" ! -name '*.cloud*' -type file -exec bash -c 'thepath="$1"
    echo "$(python "$agentdir/odrive.py" syncstate "$thepath" | head -n 1)"
    if [[ "$(python "$agentdir/odrive.py" syncstate "$thepath" | head -n 1)" == "Synced" ]]; then 
    	echo "$thepath" is synced! Unsyncing...
    	python "$agentdir/odrive.py" unsync "$thepath"
    fi 
    ' _ {} \;
    sleep 5
done
echo Sync complete! Unsyncing $targetdir ...
python "$agentdir/odrive.py" unsync "$targetdir"

This will first sync the target directory residing inside the agent mount (since it is assumed that it is starting in an unsynced state) and then copy a source directory into the target directory. It will then wait until syncing starts. Once syncing starts, it will begin looking for files that have finished syncing and unsync them if they have. At the end it unsyncs the whole target directory.

I did this quick and only tested a couple of times on OS X, so think of it more as a starting point, but it may get you to a simple solution. It does do some polling, but it it isn’t a problem.

1 Like

odrive sync agent is awesome!

this is pure gold! i always wanted something like odrive. is exactly as described the way it sould be :relieved:

here i’ve made few osx bash scripts for some heavy duty amazon cloud drive sync. i need to be in control of what i sync at some point. here’s my magic. use it. adapt it. make it better. share it :blush:

scripts

~/odrive-sync/odrive-sync.sh finds .cloudf folders

#!/bin/sh
find ~/odrive/amzdrive-sync/cloud-sync -type f -name *.cloudf -exec echo odrive sync \"{}\" ";"

~/odrive-sync/odrive-sync-log.sh logs output of sync script and compress the log

#!/bin/sh

log=`date "+/full/path/log/dir/odrive-sync/logs/%Y%m%d%H%M%S.odrive.log"`

echo sync started. check $log
time ~/odrive-sync/odrive-sync.sh | sh >> $log 2>&1

echo sync log
echo -----
cat $log
echo -----
echo compressing log
gzip $log
echo sync completed

usage example

mbp2k9:~ nevrax$ ~/odrive-sync/odrive-sync-log.sh 
sync started. check /full/path/log/dir/odrive-sync/logs/20160528143123.odrive.log

real	0m55.131s
user	0m4.013s
sys	0m2.393s
sync log
-----
/secret/path/2014-10 auckland new zealand
/secret/path/2014-10 boat trip riding bicycles gili trawangan gopro
/secret/path/2014-10 gili trawangan sergiu gopro
/secret/path/2014-10 gili trawangan
/secret/path/2014-10 glowing worms new zealand
/secret/path/2014-10 hobiton new zealand
/secret/path/2014-10 hot water beach new zealand
/secret/path/2014-10 kuta bali
/secret/path/2014-10 pool timelapse sankara ubud
/secret/path/2014-10 reverse bungee hot water beach hobiton gopro
/secret/path/2014-10 romantic timelapse sankara ubud
/secret/path/2014-10 sam nada bali
/secret/path/2014-10 sanur bali
/secret/path/2014-10 seawalk bali indonesia extra
/secret/path/2014-10 seawalk bali indonesia gopro
/secret/path/2014-10 snorkeling gili trawangan gopro
/secret/path/2014-10 sydney australia
/secret/path/2014-10 tamaki maori village new zealand
/secret/path/2014-10 taupo new zealand
/secret/path/2014-10 timelapse sankara ubud
/secret/path/2014-10 ubud bali
/secret/path/2014-10 wai o tapu geothermal wonderland new zealand
/secret/path/2014-10 wai o tapu waikite thermal pool gopro
/secret/path/2014-10 wellington nelson new zealand
-----
compressing log
sync completed

check the logs

mbp2k9:~ nevrax$ ls ~/odrive-sync/logs
20160528140745.odrive.log.gz	20160528142143.odrive.log.gz
20160528140758.odrive.log.gz	20160528142211.odrive.log.gz
20160528141344.odrive.log.gz	20160528142217.odrive.log.gz
20160528141407.odrive.log.gz	20160528142454.odrive.log.gz
20160528141639.odrive.log.gz	20160528142508.odrive.log.gz
20160528141640.odrive.log.gz	20160528142515.odrive.log.gz
20160528141652.odrive.log.gz	20160528142547.odrive.log.gz
20160528141655.odrive.log.gz	20160528142549.odrive.log.gz
20160528141732.odrive.log.gz	20160528142553.odrive.log.gz
20160528141757.odrive.log.gz	20160528142557.odrive.log.gz
20160528141904.odrive.log.gz	20160528142612.odrive.log.gz
20160528141924.odrive.log.gz	20160528142829.odrive.log.gz
20160528141936.odrive.log.gz	20160528142907.odrive.log.gz
20160528141943.odrive.log.gz	20160528143118.odrive.log.gz
20160528142002.odrive.log.gz	20160528143123.odrive.log.gz
20160528142040.odrive.log.gz	20160528143223.odrive.log.gz
20160528142112.odrive.log.gz	20160528143245.odrive.log.gz
20160528142118.odrive.log.gz	20160528143332.odrive.log.gz
20160528142125.odrive.log.gz	20160528143407.odrive.log.gz
20160528142129.odrive.log.gz

i’m so happy i can finally do my photo stuff and spend less time on api bla bla stuff or stupid amazon “not for mortals” cloud drive api access key. seems like you need a separate api key and wait for their approval to access your own data. wtf amazon? :confused:

thanks odrive :heart_eyes:

1 Like

Thanks for the feedback and for contributing with your own use case and script!

Hi all,
In case it helps anybody, here is a simple recursive refresh script that will recursive run through all your directories and subdirectories and upload any files that might have been changed:
find ~/odrive-agent-mount/ -type d -exec python "$HOME/.odrive-agent/bin/odrive.py" refresh "{}" \;

(change ~/odrive-agent-mount/ to wherever your odrive is mounted)
schedule the task in cron on linux to run it every 15 minutes or so.

I did run into a strange issue where I renamed a folder and the text file in it, and the textfile didn’t upload anymore. Any new text files I added to that folder didn’t upload either. I tried to reproduce the problem with another folder but wasn’t able to. If anybody knows what the issue might have been, let me know!

1 Like

Just want to let folks know that we’ve released the CLI to the desktop client, which many have requested. I just added a blog post that dives into it a bit. The content in there also applies to Agent, so double whammy there. :slight_smile:

We also added a CLI section in our usage guide:

FYI, if you want parallelized recursive downloads, here’s a modification to the recursive oneliner mentioned above. If you have a LOT of files you’re trying to sync this will be much faster.

exec 6>&1;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P 10 "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done

1 Like

stuckj,

I too am using xargs to parallelize the syncs. One useful tidbit, if you are saturating your internet connection, and want to ease off during work hours, you can fire a SIGUSR2 at the xargs process to tell it to reduce its parallelism by one, so hit it 9 times with kill -12 $(pidof xargs) (on linux) and any xargs processes will be reduced by 9 processcount.

When you are ready to ratchet it back up, use kill -10 $(pidof xargs) 9 times to get back to 10.

I’ve had a sync running for a few days and this way it doesn’t stop in the middle of a file, rather xargs just stops running the parallel job count when current jobs finish.

Thanks for the contributions @stuckj and @paradox606. I added them to the post:

How do I uninstall the Linux sync agent? I cannot find a command to uninstall this.

If you followed the instructions then the only 2 folders you would need to remove are:
~/.odrive-agent (which contains the application information)
~/odrive-agent-mount (which contains the odrive files/folders)

A post was split to a new topic: Peer to peer transfer capabilities (LAN sync)

In an attempt to declutter this thread a bit, I have moved a lot of the Questions and Troubleshooting to this thread:

If you have issues or questions, please post there.

Thanks!

New release of Agent is available to bring it into parity with the fixes that have been released for the Desktop. We also now have a Raspberry Pi (ARMv6) compatible build!

1 Like

5 posts were merged into an existing topic: Odrive Agent/CLI Troubleshooting, Feedback, and Questions