I’d like to see a parameter to the CLI shutdown option that waits for all uploads to complete before shutting down.
I’m running odrive only 8 hours per day and we know that Amazon doesn’t support file resume. I’m often uploading files as large as 12 GB to Amazon Cloud, meaning a lot of unuploadable files and a lot of wasted bandwidth.
I appreciate the Infinite File Upload solution to this problem but what I really need are the files fully intact on Amazon Cloud. Since these are MP4 files, I’m able to view them directly from the cloud- provided they are not broken up into smaller chunks.
As a compromise, I’m willing to let the pending uploads go to completion and then shut down odrive for the day when issuing the odrive shutdown command. It may run several more hours but at least I’m getting the files up there fully intact and no more wasted bandwidth for abandoned uploads.
I hope you’ll consider this. It would make a huge difference for me.
Hi @jeff.lawlor,
This is not going to be perfect, but it may help. Currently odrive should usually upload only 1 file at a time if the file is greater than 100MB. With that in mind here is a quick script I wrote up that will defer exiting odrive if there are any files uploading greater than a set threshold (5% in this example).
$cli_bin = "$HOME\.odrive-agent\bin\odrive.exe"
$upload_thresh = 5
$odrive_not_running = "There was an error sending the command, please make sure odrive agent or desktop is running."
while ($(& "$cli_bin" status) -ne $odrive_not_running){
$cur_status = $(& "$cli_bin" status --uploads)
if ($cur_status -eq "$odrive_not_running") {
echo "odrive is not running!!"
}
else {
$can_exit = 1
foreach ($line in $cur_status){
if ($line -eq "No uploads."){
#No uploads
}
else {
[string]$up_perc = $line.Substring($line.Length - 3,2)
if ([int]$up_perc -gt $upload_thresh -or $up_perc -eq "00") {
#Uploads in progress. Need to wait.
$can_exit = 0
}
}
if ($can_exit) {
echo "No uploads beyond $upload_thresh percent. Exiting"
$(& "$cli_bin" shutdown)
}
else {
echo "There are uploads beyond $upload_thresh percent. Deferring Exit."
}
}
}
sleep 5
}
You can put this into a .ps1 script and run it instead of your current “shutdown” call.
I’ve been experimenting with the shutdown script. It does work when the circumstances are right. Unfortunately, a couple of issues preventing those ideal circumstances.
First, the assumption that when a large file is uploading, no other file will upload simultaneously isn’t playing out. When a several gigabyte file is uploading, I can watch as smaller files complete and new ones start in their place.
Second, sometimes the item it’s uploading is a folder, not a file. In the screenshot, “Cosmos” is an entire directory. Some of my directories are hundreds of gigabytes.
Under controlled circumstances, the script works great. But with the above issues, odrive is never shutting down under practical circumstances.
Hi @jeff.lawlor,
When you run the script under those circumstances are you seeing that it is detecting several files that are uploading and past the 5% threshold?
Currently odrive can have some issues syncing one large file at a time. It depends on the folder structure and on user activity. Our next major version will be fixing this so that it will always respect limits. For the files you are seeing uploading, how distributed are they across your folder structure?
Yes, that’s exactly what’s happening. In fact, I just made the script dump the status --uploads output and this is what I got. That’s a lot of simultaneous uploads!
Friends - S03E03 The One With The Jam - Episode Promo.m4v 78%
Friends - S07E07 The One With Ross' Library Book - Episode Promo.m4v 46%
Friends - S03E10 The One Where Rachel Quits - Episode Promo.m4v 48%
Friends - S08E23 The One Where Rachel Has A Baby - Episode Promo.m4v 22%
Friends - S08E03 The One Where Rachel Tells Ross - Episode Promo.m4v 40%
TORTILLA_HEAVEN.m4v 4%
Friends - S03E25 The One At The Beach - Episode Promo.m4v 80%
Dead Like Me - TV Series Trailer.m4v 43%
Friends - S09E04 The One With The Sharks - Episode Promo.m4v 33%
Friends - S04E13 The One With Rachel's Crush - Episode Promo.m4v 42%
Friends - S08E19 The One With Joey's Interview - Episode Promo.m4v 43%
Friends - S03E13 The One Where Monica And Richard Are Just Friends - Episode Promo.m4v 62%
Friends - S01E02 The One With The Sonogram At The End - Episode Promo.m4v 33%
Ep 12 The World Set Free.mp4 0%
Friends - S02E19 The One Where Eddie Won't Go - Episode Promo.m4v 93%
Friends - S08E22 The One Where Rachel Is Late - Episode Promo.m4v 90%
Silicon Valley - S02E07 Adult Content.m4v 1%
Last Man on Earth, The - S01 - Bloopers.m4v 16%
- Commercial.m4v 21%
Friends - S01E22 The One With The Ick Factor - Episode Promo.m4v 35%
Family Guy - S09E18 It's a Trap!.m4v 1%
2015_0606_104135 Santa Barbara Zoo and Shawn's Graduation.MOV 1%
Friends - S09E01 The One Where No One Proposes - Episode Promo.m4v 40%
Friends - S08E13 The One Where Chandler Takes A Bath - Episode Promo.m4v 58%
Friends - S08E16 The One Where Joey Tells Rachel - Episode Promo.m4v 39%
Friends - S01E19 The One Where The Monkey Gets Away - Episode Promo.m4v 59%
Friends - S03E01 The One With The Princess Leia Fantasy - Episode Promo.m4v 39%
Friends - S07E12 The One Where They're Up All Night - Episode Promo.m4v 89%
I have probably 12 root folders. In this sample, it’s clearly fixated on the “TV” root and the “Friends” folder within it in particular. However, there are several files from different folders in the TV root and a couple more files from entirely different roots.
There are at least 3 files that are over 100MB but most of those files are under, if that helps.
Is there a way to limit the overall number of simultaneous uploads?
I did fix what looked like a bug in the script- the check for $can_exit seems to belong after the foreach loop. Otherwise, it exits when it encounters an individual file is less than 15% uploaded, irregardless of the upload state of the remaining files.
I pasted the version I’m running. It has the bug fix, a lot more output to the screen and a configurable interval.
$d = Get-Date
echo "$d Starting odrive yielding shutdown script`n"
$cli_bin = "$PWD\odriveCLI.exe" #$HOME\.odrive-agent\bin\odrive.exe #"$HOME\.odrive\bin\6083\odrive.exe"
$upload_thresh = 15
$update_interval = 300
$odrive_not_running = "There was an error sending the command, please make sure odrive agent or desktop is running."
while ($(& "$cli_bin" status) -ne $odrive_not_running){
$d = Get-Date
echo "$d Starting odriveCLI status --uploads"
$cur_status = $(& "$cli_bin" status --uploads)
echo $cur_status
$d = Get-Date
echo "$d Completed odriveCLI status --uploads"
if ($cur_status -eq "$odrive_not_running") {
echo "odrive is not running!!"
break
}
else {
$can_exit = 1
foreach ($line in $cur_status){
$d = Get-Date
if ($line -eq "$a No uploads."){
echo "No uploads in progress."
}
else {
[string]$up_perc = $line.Substring($line.Length - 3,2)
if ([int]$up_perc -gt $upload_thresh -or $up_perc -eq "00") {
#Uploads in progress. Need to wait.
$can_exit = 0
}
}
}
if ($can_exit) {
echo "$d No uploads beyond $upload_thresh percent. Exiting"
$(& "$cli_bin" shutdown)
break
}
else {
echo "$d There are uploads beyond $upload_thresh percent. Deferring exit for $update_interval seconds.`n`n"
}
}
sleep $update_interval
}
Yup, you are right, the shutdown should be after the foreach.
For the simultaneous uploads, there isn’t currently a way to limit the number of total uploads, but the next major release will do a better job of this. I think the simultaneous uploads overrun will generally happen right on startup and then level out after, but that may not help if it takes all day to make it through the initial uploads…
For now you may just need to make the threshold looser if stopping is more important than finishing uploads in progress.
Yes, you’re right- that long list of files was within an hour of starting odrive. Here’s what it looks like after running more than a day. Quite a few less.
Silicon Valley - S02E09 Binding Arbitration.m4v 76%
Portlandia - S03E11 Blackout.m4v 5%
Ep 7 The Clean Room.mp4 28%
SYRIAN_BRIDE.m4v 79%
Simpsons, The - S04E02 A Streetcar Named Marge.m4v 92%
Friends - S06E06 The One On The Last Night - Episode Promo.m4v 18%
Dead Like Me - S02E05 Hurry.m4v 83%
Last Man on Earth, The - S01E11 Moved to Tampa.m4v 48%
I have the threshold set to 15% but there’s still little chance all of these uploads will be below 15% at any given run of the script.
Making it less likely is that I had to increase the delay to 5 minutes. When my PC is under load (which is most of the time), it takes 30 seconds or longer to complete the script.
I have a 4 Mbps upload.
I’m looking forward to that next release. I’m happy to beta test it.
Thanks for the feedback @jeff.lawlor. I appreciate the details. I would expect the number of uploads to dwindle as time goes on, but it may not help all that much if it takes longer than a day to do that.
