How do I limit the number of parallel upload streams

Hi,

I’m new to Odrive. I have a premium subscription yet my experience so far has been a nightmare.
The product simply seems unusable since I can not get it to upload my files to the storage. The files are often huge, 15-20Gb. Somehow, without asking my permission, Odrive decides to start a parallel upload of a dozen of such files, then it works for days only managing to upload a single digit percentage of each file’s volume and then something happens. Sometimes it’s a reboot, sometimes a momentary connection loss, sometimes a power outage. And sometimes I just have no idea what happened - I couldn’t find logs of any kind,- the files just get thrown to the waiting list.

Yes, I’m aware of the option to split files. But I don’t want to do that. All I want is to let the program concentarte on one or two files at a time, get the job done and move on to the next files - just like I have always successfully done with Dropbox’s or Amazon’s native clients. But here, having paid $99, I seem to have lost this crucial and basic functionality. I’m extremely dissapionted… I hope it’s just the stupid me and there’s some way to do what I need to do.
If not, obviously, this post should be marked as a “Feature request” rather than “I have a question”, but I just won’t stay Odrive’s client long enough to see this feature implemented, as I’m not used to paying money for reducing the existing functionality…

Hi @alexey,
Finely tuning concurrency is something that is being worked on. There is a feature request for it here: Limit Amount of Files being Uploaded at a Time

Are you uploading these large files to Amazon Drive? If so, it is one of the services that has proven to be pretty temperamental with large file uploads. It is actually one of the major reasons we made IFS.

As for concurrency, currently odrive is supposed to limit concurrency to just 1 file if the file is over 100MB. However, there are certain cases and structures that result in this not being hit, especially on initial odrive startup. As stated in my responses to the feature request above, this is on of the major things being overhauled for the next big release.

For your current situation, can you send a diagnostic while it is trying to concurrently upload many of these large files at once? I am curious what it looks like and would like to see if there is a way we can coax a more serial upload flow.

Hy Tony,
Thanks for the detailed reply.
I have just selected the “Send diagnostics” option as you requested. I hope you can see what’s going on based on this report.
I’m also attaching the screenshot showing the files beeing uploaded as well as the waiting list.

And yes, I’m currently uploading to Amazon Drive. It does seem a bit unreliable, but whenever just a single file is uploaded, it eventually gets done. With a bunch of files it simply never happens.
Thanks again,
Alexey

Hi @alexey,
For now I think you can mitigate the concurrency by trying to limit the number of items inside “D:/odrive/Amazon Cloud Drive/VIDEO/SD/”. If you were to move out all of the subfolders in there to a non-odrive location, except for one (keep PAPA, for example) odrive should sync against this, and then you can move another folder in, etc.

This is certainly more work, but I think it is something that is simple enough to execute and should allow you to upload things much more reliably and consistently.

Thanks, Tony.
I really appreciate your trying to help - and I’m not being cynical about that.
But it’s not about finding a way to upload my files - I was perfectly able to do that before switching from Amazon’s client to Odrive and I can always go back to that option.
It was rather about getting my money’s worth from Odrive. I know you’re working on a solution but I just want to make sure you guys at Odrive really understand how outrageous it is from customer’s perspective to see a basic and an absolutely critical feature available for free removed from an expensive commercial alternative.
Anyway, could you give me a hint as to when this feature is going to be released?

Best regards,
Alexey

1 Like

What’s IFS do or about?

Hi @slowdriver,

Here is more details from our usage guide regarding IFS

UPLOAD LARGE FILES (IFS/SPLIT LARGE FILES)

Storage is pretty useless if you can’t upload your files. That is why we created the odrive Infinite File Size (IFS) feature. IFS gives users a sure-fire way to get even the largest files uploaded. If you find that you are experiencing issues with larger files uploading successfully, you should adjust your IFS settings to compensate.

IFS is available from the odrive tray menu under “Split Large Files”.

You have the option of choosing a threshold and segment size between 100MB and 2GB, or turning off IFS completely with the “Nothing” setting.

Set the size you wish to split large files in the tray menu

Once set, the odrive desktop client will automatically engage IFS when uploading any files that exceed the selected size option. The file will be split into chunks, according to the size selected, and uploaded to your storage.

For example, if you have set the IFS option to 100MB and you have added a 650MB file into an odrive folder, the odrive desktop client will automatically split the file into 7 different segments (6 x 100MB + 1 x 50MB) and upload it. If an IFS file upload is interrupted at any point, odrive will resume the upload from where it left off, automatically.

Important: The files uploaded with IFS will remain “split” on the remote storage. These files will look just like any other file, both as a placeholder and locally cached, when viewed through the odrive desktop client. If you try to access these files outside of odrive, they will appear in their “split” state.

Please also look at:
https://medium.odrive.com/cloud-upload-was-a-struggle-until-today-bb267cb5566e#.rhba1ajes

Thanks,
Asif

Thanks Asif,

As I mentioned in my opening message, I’m aware of the IFS feature.
But while it might be useful in some use cases, I don’t think it’s a reasonable solution to my specific problem, for at least two reasons:

  1. I know for a fact that splitting files is not necessary in my case. I used to work with Amazon’s native client and it did a reasonable job without splitting the files.
  2. The more important consideration is that having the files split on the cloud makes me totally dependent on Odrive for retrieving them. And what if I don’t renew my subscription one year from now and one day later want to access my files?

All in all, while some workarounds might be available, I’m absolutely sure the right thing to do is for Odrive to implement the parallel upload control as soon as possible, issue a sincere apology to the customers and from now on refrain from imposing crucial decisions on the users while taking the control they used to have away from them. This is completely unacceptable.

@AsifNisarr,

I didn’t notice at first your response was meant for @slowdriver rather than me.
Anyway, I still mean everything I said, I’m just sorry I mistook your general reply for a recommendation to use IFS in may case.

Thanks @Alexey for your feedback. Unfortunately, currently there is no option available to throttle the parallel upload as per user selection =(

We have noted this as feature request internally and hopefully this is something we would like to offer in future to give more control over how user want to sync their files in particular situations / scenarios.

Thanks,
Asif