Hi @spacecommguy,
Thanks for the extensive writeup and contributing to the forum. I love the enthusiasm .
You are correct that Amazon Drive seems to be susceptible to slow single-instance downloads/uploads and benefits significantly from concurrent transfers, if your bandwidth can support it.
I haven’t actually tested running multiple agent instances as the same time, but I would guess that there could be problems doing so. All of the instances will share the same persistence and config, which may be causing some race conditions and locking out of the instances while one tries to write to the persistence layer. The same applies to the downloads. Downloads are done to a temporary file and then renamed once the file has finished. I am guessing that you may end up with parallel downloads of the same files as each instance is blissfully ignorant that another instance has already started downloading that file. It will also increase the overhead quite a bit.
Have you taken a look at this post here?
These are a few iterations of some one-liners, including running parallel downloads. You could parse one of the examples out into proper script form, with better feedback and logging.
I actually just used a variant of this one, yesterday, to download 2TB of data from Google Drive in one go:
exec 6>&1;num_procs=10;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive.py" sync | tee /dev/fd/6); done
I was averaging about 200MB/sec download with 10 concurrent transfers against a single agent instance.