Hi @damianhenry,
That is within the range of expected. The “recursive” directive is single threaded, so not ideal for large bulk downloads like you are trying to do. You can use a one-line script to allow for faster downloads by using concurrent threads.
Example command: exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Amazon Drive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs python "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done
This will use 4 concurrent processes to perform the download in the given folder (in this case $HOME/odrive-agent-mount/Amazon Drive).
Yes I want to run the command but how do I do this: I am a newbie so I am sorry for the questions. I can tell you I have successfully pulled down a 50 gig folder from amazon with the CLI tool… So far so good. My attempt in other programs and amazons tool failed
Hi @damianhenry,
You can just copy and paste the exact command I have above to the terminal. You will just want to make sure the folder targeted for download is correct. Right now it is "$HOME/odrive-agent-mount/Amazon Drive", so you will need to change that to the correct path if it is not correct for your setup.
exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Amazon Drive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs python3 "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done
The path you are using looks wrong (the $ at the beginning). Instead, is it just /media/Joe/USB STICK/odrive-agent-mount? So: "$HOME/.odrive-agent/bin/odrive" mount "/media/Joe/USB STICK/odrive-agent-mount" /
Is this going through the entire Amazon Drive Folder on Amazon and creating place holder files on my local machine ? If so if I want to download everything why wouldn’t I just run recursive
I am getting SyntaxError: invalid Syntax
exec 6>&1;num_procs=4;output=“go”; while [ “$output” ]; do output=$(find “/media/bob/Doe/odrive-agent-mount/Amazon Drive” -name “.cloud” -print0 | xargs -0 -n 1 -P $num_procs python3 “$HOME/.odrive-agent/bin/odrive.py” sync | tee /dev/fd/6); done
Yes, it is just exposing the entire structure, but not downloading anything. If you wanted to download you would remove the --nodownload parameter. For your bulk downloading needs, however, the script below will be faster.
exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "/media/bob/Doe/odrive-agent-mount/Amazon Drive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs python3 "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done
Try this command, exactly as it is above (just copy, paste, and hit enter). Copying from the forum here can get a little weird because the quotes are converted into fancy opposing quotes (“”), which won’t work within the terminal. The commands I am pasting in are designated as preformatted text, so that they are shown exactly as they should be (this is done by adding a ` character to the beginning and end of the line).
If you are still getting an error, can you take a screenshot of the terminal with the command and the error output?
Yeah for smaller files it is likely going too fast to show the full progression. You can tail the main log to see the sync actions: tail -F "$HOME/.odrive-agent/log/main.log"
Yes
You can increase the num_procs=4 to num_procs=6 or num_procs=8, for example. The risk will be that Amazon will start to throw rate limit errors and they can get progressively worse if you continue after they start, so I would say to increase gradually and be on the look-out for any errors they start throwing.
I am sure you know this but there are may companies who are facilitating the transfer of data from Amazon to various new storage companies. aka Wasabi. Many of them are charging big money to handle this transfer. This tool would be a great solution to a growing market.
I tired so many tools on maxOS, and Windows and I just kept getting errors and slow downloads. This tool just keeps jugging along.
I appreciate the patience on this and the feedback. I have been wanting to create some “simple-ish” scripts/utilities to aid folks with migration from one storage to another (handle the download from one and upload to another automatically), but I haven’t had a chance to focus on it yet, unfortunately.
So how can I set this up to to backups all the time. I want to setup my folders local and remote and have it sync at all times. I want the process to start when I start my computer.
Do I need to purchase the software to do these things ?
Hi @damianhenry,
You can run odriveagent automatically a few different ways. There was a post here where someone used what looked like a simple solution on Mint:
Other options would be to create a systemd service, which is a bit more involved.
When odriveagent is running, it will automatically upload items that it find during its routine local scans.
For downloads, you could add a cron job (crontab -e) to run periodically to recursively download any files it finds, setup like this (will run every hour): 0 * * * * "$HOME/.odrive-agent/bin/odrive" sync "/media/bob/MAGA/odrive-agent-mount/Amazon Drive" --recursive
The above stuff doesn’t require a Premium subscription.
