Bandwidth Throttling Unlimited

Hello,

I’ve searched through the forums and noticed that some users complained about Odrive upload saturating their connection preventing them from doing their normal work via internet, so to solve that, you’ve set the upload bandwidth to about 80% of the users’ max speed. I don’t know if this is for the “Normal” or “Limited” setting in the Throttling option or if it also applies to “Unlimited”.

I’ve set the Throttling options for download and upload to “Unlimited”, however it doesn’t seem to saturate or upload my files at my full speed which is 10 Mbits/sec. (I can’t test out the Normal and Limited mode now that the trial expired) Unlike some other users, I actually WANT to utilize my full upload speed especially when I leave my PC on overnight dedicating it to uploading files.

So my question is:
Is the “Unlimited” mode in Throttling supposed to fully saturate/use my full upload speed of 10 Mbits/sec?
Or is it STILL limited to the 80% of max speed throttle?
If it is limited to the 80% of max speed throttle, can you implement a feature for us users who would prefer to fully make use of our max upload speed?

Thank you

Hi,
The throttling we employ is actually not a set value. It is dynamically calculated, based on evaluated performance for the last few file transfers.

Even if you set to “Unlimited”, it is not a guarantee that you will be able to saturate your internet connection, since the speed is highly dependent on service you are connecting to, among many other factors.

What kind of speed are you seeing and what service is it to?

My internet connection speed is 50 Mbits download and 10 Mbits upload.
The upload speed that I’m seeing odrive use is around 5-7 Mbits. VERY rarely will it go up to 10 Mbits (this happened only once in 4 days and only for around 15 minutes. I’m uploading 24/7 and no other applications are using my bandwidth. To be fair, I didn’t monitor it all day and night so this 10 Mbits upload rate may occur more than I noticed, but it definitely is very rare.)
I’m syncing it to only the Amazon Cloud Drive.
I also noticed that it says it’s syncing 10 items simultaneously.

Is the “Unlimited” mode STILL subjected to your dynamic throttling too? Or does this throttling only occur for “Normal” and “Limited” mode?

Unlimited disables our dynamic throttling, although it will still limit the number of concurrent transfers so it doesn’t spin out of control.

I have noticed in my own testing that Amazon Cloud Drive can give some inconsistent speeds, depending on which endpoint is being hit. It is also influenced by the number of concurrent transfers. For example, a single transfer of a large file will show lower performance than 10 concurrent transfers of smaller files.

Where are you located?

I’m in Thailand. Most of the 10 concurrent uploading files are large files of about 500MB to 5GB. I did not activate the IFS mode because it changes the file names when I occasionally want to browse the files through the Amazon webpage. Anyways, I understand the risk of uploading large files and having it fail (then having to re-upload , etc.). I do not mind that, as long as Odrive can fully utilize my full upload speed.

I just find it weird that I’ve been able to reach my max upload speed for 15 minutes, but was unable to sustain it. I guess you might be right that Amazon Cloud Drive gives some inconsistent speeds.

Thanks for the additional info.

Amazon also gives odrive the endpoint to connect to based on the account region. For example, if you are signed-up against a U.S. Amazon Drive account you may be hitting far-off servers.

For us, we consistently see odrive’s initial upload speeds as being competitive with a native cloud client. On the commercial side, odrive matches or exceeds other clients uploading to Google Cloud Standard or DRA. Amazon S3 is hit-or-miss. We’re located close (from a network hop perspective) to both Amazon and Google’s US-West server farms. At times odrive seems to pick a more circuitous route than a direct access to an S3 bucket.

SMB or consumer could storage is another matter. Amazon Cloud Drive appears to quickly throttle uploads if too many connections are made, so we’ll leave that one aside as it is too inconsistent. Uploading to either Dropbox for Biz or Box for Biz proceeds at least as quickly with odrive as with the native client initially. After a few 10’s of GB however, odrive slows to a crawl. I am see it using under 0.5% of the upload bandwidth even when there are essentially no other active connections. The Box and Dropbox clients consistently utilize far more, maintaining minimum speeds of between 600 - 1800 Mbps (20-60% capacity). That’s a difference between uploads lasting a few hours vs. days.

Interesting feeback. I like to see this type of information.

I have noticed, in general, that 3rd parties integrating to these services APIs can, at times, get treated as second-class citizens compared to the native clients. This manifests in the form of api rate-limiting and throttling.

I can’t say for sure what is happening with Dropbox and Box, but it sounds like throttling may be kicking in at a certain point there. The underlying transfer mechanism we use for standard REST-based integrations is the same across the board, so if it was something in our implementation (a leak, for example), I would expect to see consistent behavior across all REST-based integrations, in terms of sudden transfer slowdown.

I suppose this could be tested by restarting odrive, once this massive slowdown occurs, and seeing if the transfer rate improves. Of course, as per your recent post in another thread, if the locally exposed odrive structure is massive, the time to upload start could be delayed enough that throttling is lifted by the time it kicks in, so this test would need to be performed on a local structure small enough to ensure near-immediate transfer after restart.

I have another data point. Previous large uploads were only for a single folder. The files were backups, divided into 2GB chunks. Odrive only uploaded a single file at a time.

Trying the same steps (uploading to Dropbox, 400+ GB total transfer) with two folders being synced at once causes odrive to perform two simultaneous uploads. Total throughput more than doubled - averaging between 3x - 5x what odrive managed with a single folder syncing. Adding another folder increased throughput by 40-50% as odrive starts transferring 3 files simultaneously.

Something weird is going on here. Can odrive be convinced to upload more than one (large) file per folder?

Yes. odrive can currently be manipulated/triggered to upload more than one large file at a time.

There is a balance we are trying to create where user actions are not “delayed” and remain “snappy”. This means we do not globally block actions, currently. So, if you navigate into other folders structures and trigger a refresh or sync, you can initiate an additional upload.

That is the behavior I see. My question was whether it is possible to configure odrive to upload more than 1 file per folder. When odrive is using less than 5% of the available bandwidth “snappy” does not come into play.

From what I’ve seen, creating additional concurrent uploads results in better than a 1:1 increase in transfer speed - assuming one has the available bandwidth. If odrive is targeting business customers rather than only folks looking for a free solution, one assumes that these clients will also arrive with more than bare-bones connectivity.

Another ping on this topic. I started a sync for infrequently used files from one of our servers. Odrive had been plugging away on the machine uploading a set of backups that were made to a single folder. Uploads were running with the standard odrive limit of one file a a time. Odrive was averaging between 3-5 Mbps of upload bandwidth.

I then synced folder trees containing several thousand files in some hundreds of folders. Odrive bounced up to 19 simultaneous uploads (maybe more - I didn’t logon to the server for a few minutes) and utilized 1.5 - 2 Gbps of bandwidth. That’s nearly a 1000x increase in upload speed and still did not strain our capacity.

Can the hard limit of 1 upload slot per folder please be relaxed if the network can handle it? The artificial cap seriously limits odrive’s functionality in real world applications.

Thanks @Ethan,
There is a bunch of work currently being done in this area and all of the feedback here is being considered as part of that development work.

Fantastic news! Being able to set a more aggressive sync policy would aid folks with higher bandwidth connections.

Sorry to revive an old thread. But in reading your posts I am trying to figure out how you can tell how many concurrent uploads you have? I don’t see any indicator. Or are you just counting the pink sync icons next the files?