Prioritize syncing

Hi,

Somehow a certain folder won’t sync, even when I try to force it to. I just tried again and sent a diagnostics report, expecting you’d need it anyway :wink:

tvnviewer_2019-12-17_15-25-08

Restart of Odrive didn’t seem to help. It seems that other, more recently created files, are syncing without issue.

Please let me know what could be the issue or could fix it.

It seems to be the case that this problem occurs with any yet unsycned folder (queued for sync).

Maybe it’s placebo but previously I had the idea I could prioritize syncing of any yet unsynced folder (queued for sync) by manually selecting it to be synced. This can sometimes be handy when Odrive is stuck on syncing new folders with huge files, while I just need a folder or 2 with some few megabytes of files to be synced quickly in between so my colleagues can resume their work from the cloud.

Hi @skander,
There isn’t currently a way to prioritize syncing. odrive will detect changes and start syncing those, so it ends up being the files/folders it first identifies as needing to be synced. The right-click sync action is only used for syncing down content from your remote storage.

You might be able to get odrive to see the changes you want to upload first by performing a right-click->refresh where the content is, immediately after starting odrive.

You can also put a ~ character in front of the folders or files you do not currently want to upload, which will cause odrive to ignore them. Once you remove the ~ and refresh, odrive will pick it up again and start uploading.

Hi Tony, thanks for the explanation! I have some follow up questions:

  • Is there any way to explain that Odrive “deviated” from what we experience as its usual behavior? That is to say: In our experience, even when there’s loads of files left to sync, Odrive seemed toprioritize “simple/small” changes like newly created folders or small image files over the larger, time consuming changes.
  • Shouldn’t the error message be more user friendly in this case? E.g. “file already queued for sync” or something?
  • Given our usecase where we are syncing gigabytes of files 24/7 usually, it can be very annoying to have to wait for a small file to finally appear in the cloud storage. Does Odrive2 perhaps offer any functionality in terms of prioritization? Given the fact that we’re working with so many folders, using the ~ workaround is not practical for us.

Hi @skander,

It is possible to “kick-off” an additional upload task, which I think is what you may have seen. The following things need to be true:

  1. Available upload slots (default is 4)
  2. The items you want to start uploading are inside a folder that has already synced (it exists on the remote).
  3. A refresh is done on the folder the items are in (a sync action will also kick off a refresh).

There are some other quirks and caveats to the above, but it should work that way, generally.

I’ll look into improving the error message here. I believe the error you are seeing is caused by the parent folder not existing yet on the remote side.

odrive2 won’t have what you are looking for, but we are working on some other products that might fit your use case better.

Can you tell me a bit about how you are currently using odrive and what use cases you are trying to solve with it (data access/management/movement needs, workflow, sharing, etc…)?

Hi @Tony,

The folder not yet existing on the remote is definitely the common factor amongst our “problem cases”. The inverse seems to also be true: Any newly created folder on the remote doesn’t seem to be able to be “kicked-off” so it syncs faster to our local storage. This is also causing us inconvenience.

Is it possible to increase the available upload slots? I cannot find the option for it. Is it some variable that has to be adjusted in a configuration file?

As for our usecase, I’ll gladly elaborate:

We’re a small video production company with team members often having to collaborate remotely from outside of our office. Our remote storage is both used as a general backup and as a way for collaborators to access files remotely. We use odrive and gsuite for this.

Since we are working with big video files, our local storage is usually busy syncing 24/7. This has usually not been an issue for our workflow, and files used to be available remotely (or conversely when the file creator was working remotely) on time for further usage. We have been growing however, and running into the problem of the syncing not catching up with our work more often and often.

We’re now considering setting up something for direct remote access (ftp or something) to our local storage to circumvent this problem. Obviously we’d prefer to stick to one solution instead of having 2 types of remote access to our file, for simplicity’s sake and to save the bandwidth of having to transfer the same file multiple times (remote + collaborator(s)) instead of just one time to the remote storage.

In my view, having a solution like gsuite+odrive WITH the possibility of being able to prioritize which folders/files sync first would perfectly solve our problem. Then again, I’m not the company’s dedicated IT guy and haven’t invested any deep brainstorm sessions into the best solution for our usecase, so maybe I’m approaching it completely from the wrong angle and am I using odrive in a way that was completely not intended :sweat_smile:

Thanks @skander! I appreciate the additional details.

I am exploring what we can do in the future for this, so I sent you a follow-up direct message with some additional questions, if you can take a look when you get a chance.

As for what we can try right now, we can increase the number of uploads available on the client.

  • Open the odrive_user_premium_conf.txt in the root of the odrive folder
  • For maxConcurrentUploads, increase the value (from 4 to 12, for example)
  • Save the file and then restart odrive.

I have a couple other ideas, but I wanted to start with this one, since it is very easy to try.

Hey @Tony

I have applied this config change, however, what I have noticed now but also in the past (with the default 4 max concurrent uploads) is that sometimes odrive “decides” to use less than the max allowed upload slots, even though there are way more files/folders yet to be synced.

Is there any reason for this behavior? I proactively sent a diagnostic report just now, while odrive was using only 3 upload slots.

edit: I now see that I didn’t press the confirm button to send the report until just now, when odrive actually IS using 4 slots… But the max was already increased to 12 so something still seems up?

Hi @skander,

There are probably a few things at play here. odrive tries to prevent uploads from spinning out of control, especially for large files, so it has some internal limiters in place. For example, the vast majority of folks shouldn’t upload more than one large file at a time since upload speeds, in general, are fairly slow. Connection failures become a big problem as you increase concurrency with large files.

Here is what you may be seeing:

  • odrive generally starts off with a single upload, by default, and then ramps it up over time, once it can see that you have sufficient bandwidth to support multiple concurrent uploads.

  • odrive will try to limit the number of concurrent uploads based on a total size of the current uploads. If it tries to add another upload and the total exceeds 250MB it will not try to add it until finishing the currently transferring item(s).

I’ll look at the possibility of adding a few more advanced controls to tweak these limiters, for folks like you who have sufficient bandwidth and resources to allow for more large file concurrency.

I did think of another idea that may work for your use case. I think we can setup a “priority” folder for the uploads that you want to get up quickly. This would be done by taking advantage of the “sync to odrive” feature. Here is how you do this:

  • Create a local folder somewhere on your system with the same name as a folder on your remote storage. You can even do this at the storage root level. For example, you can create a folder named “Google Drive” and then right-click->“sync to odrive” on that folder. You will be asked to choose its corresponding remote location and you would pick “Google Drive” folder in the root of your odrive. (This is assuming that you have Google Drive linked and the folder name is “Google Drive”).

  • This will create a new “sync mount”, and it is treated independently from the default odrive folder in several respects. You can actually create any number of these relationships.

  • Once you’ve done this you will have 2 (or more) folders that you can add/edit items into that will sync to the same remote location. You can then use one as your “priority” folder. For example, if odrive is busy uploading a large folder somewhere inside your default odrive folder and you need to get some smaller files up, you can go to the “priority” folder and drop those files/folders in. odrive should detect these changes on this other mount and start immediately syncing those items, as well.

Here is some additional information on “sync to odrive” folders: https://docs.odrive.com/docs/manage-sync#section-sync-external-folders

Also, I sent you a direct message I was hoping you could find the time to look at. If you click on your profile image at the top you should see it listed.

Hi @Tony, thanks for the further explanation, and for thinking along with our usecase.

My first objection as to why your proposed solution (a priority folder) wouldn’t work for us, is because we use a very specific folder structure, where each of our projects are split into their own parent folders. The project folders are then divided into folders representing each quarter of a year, which are in their turn divided into yearly folders. This allows us to build a user friendly and accessible archive, and keeping all materials seperater project allows us to share only the required things to any remote user. Having a general priority folder that would contain all the files that required fast remote access wouldn’t work, as we’d have clients and contractors being able to access each others files. The workaround I could think of is to put files in the priority folder first, and once they are synced, move them to the proper folder. But this would increase manually required operations significantly. Too big of a hassle, I estimate.

As for your direct message: Yes, I was notified, but simply didn’t have the time to reply yet. I will get around to that asap, because I really appreciate the attention you’re giving to our usecase and very idiosyncratic grievances with your product :slight_smile:

Hi @skander,
Thanks for the feedback!

I just wanted to make sure we were on the same page with this. Here is a simple diagram of what I am talking about:
sync_to_odrive

  • In this diagram you have two local folders that point to the same remote folder location. The “Default Folder” which is the normal odrive folder, and the “Priority Folder”, which is a new “sync to odrive” folder.

  • In this example, let’s say you have a few large files uploading in the “Default Folder”.

  • While that is happening, you can add/edit files in the “Priority Folder” and have them start uploading immediately, as long as there are upload slots available.

  • This wouldn’t affect what the users who you are sharing with will see. They will still be looking at the “Remote Folder”, so the structure will remain the same.

Does this make sense? I may be missing some details about your workflow, so just let me know.

Happy Holidays!

Hi @Tony

My apologies, I read your earlier post too hastily. I didn’t catch the “local folders sync to the same remote location” part :slight_smile:

As I understand it now, your proposed solution would certainly work on the remote user side of our workflow. On the local side, however, we’d still have the same problem of non-adherance to our folder structure. To illustrate, our regular folder structure looks like this:

explorer_2019-12-24_11-58-00

If a local user needs to access this project’s renders, the user knows to navigate the folder structure by going to the project’s year, project’s quarter, project name, and finally, the folder “renders”. I’m not sure how we’d not bump into an issue of not being able to intuitively find files when we’d be using a separate priority folder.

Oh, also, to clarify, we always keep everything locally synced on our local storage; I just realised that a lot of people use odrive precisely for the selective local syncing - we don’t.

I guess what we could do is to set the priority folders to not sync anything locally, and then actually copy files from our standardized folder hierarchy to the priority folder whenever we need a fast remote sync. That would temporarily create a locally redundant duplicate file, which would be “removed” once syncing is complete, right?

Questions regarding this:

  • How does odrive handle the fact that there are duplicate files (one in the correct folder structure, one in the priority location)?
  • How does odrive handle the fact that once priority upload is complete, preferrably we’d get rid said priority folder, as it would be locally useless at that point?

Obviously, working with local copies of files still seems kind of a hassle for us. It requires extra manual action and causes unnecessary load on the computer (mainly the HDD) by having to make those redundant copies of files.

Happy holidays!

Hi @skander,
Thanks for the additional details and discussion.

Correct. For anything that needed to be uploaded immediately, you could copy that into the corresponding location in the “Priority” folder by expanding the appropriate placeholder folders to navigate to the proper location. Once it was done syncing you could right-click->unsync to remove the redundant storage.

  • odrive doesn’t relate the two local folders. It just has two sync mounts that happen to point to the same remote storage. It looks at each mount, and its remote counterpart, individually. You could actually have 10 or more mounts pointing to the same remote folders, if you wanted to.

  • Once a file has been uploaded in one mount, the other mount(s) will see the new file on the remote side.

    • If it is a “new” file in that mount (meaning there is no corresponding local file in that mount, yet), it will create a placeholder file to represent it.
    • If there is a local file there already (like the yet-to-be-synced files that are sitting the “Default” folder that you copied into your “Priority” folder), odrive will check if the remote file matches the one it has locally. If it does it just considers it synced and moves on.

Similar to the above, it treats the local mounts independently. You can download or unsync any content in one mount and the other mounts won’t be impacted by it.

The flow would look something like this:

  • “Default” folder path: C:\Users\You\odrive\My GDrive → points to your remote Google Drive account

  • “Priority” folder path: C:\Users\You\Documents\My GDrive → also points to your remote Google Drive account

  • Let’s say you have several items in C:\Users\You\odrive\My GDrive\_PROJECTEN_\2019\2019_KW4\Example project folder that are currently uploading and blocking any additional uploads in this location.

  • You just finished adding some new files in that same folder that you want to start uploading immediately.

  • You can go to C:\Users\You\Documents\My GDrive and expand the placeholder folders down to _PROJECTEN_\2019\2019_KW4\Example project folder and then copy the priority items into there.

  • The copied items should start uploading immediately. Once they are done, you can right-click->unsync the C:\Users\You\Documents\My GDrive\_PROJECTEN_ folder to collapse the whole structure back down into a placeholder file and leave it that way until you need to prioritize an upload in the future.

  • The “Default” folder mount will eventually see the “new” remote files that were uploaded in the “Priority” folder mount. It will check them against the files it has locally, see they are the same, and consider them already synced.

Absolutely. Our product team is looking if anything can be done about this from a product feature/behavior standpoint. Since I can’t provide any ifs or whens for that type of enhancement, I wanted to see if we could come up with a viable alternative for your use case.

Happy holidays! :smile:

Hi Tony,

Sorry for the late reply again, holidays & daily business getting in the way :blush:

Thanks for the elaborate explanation. What i didn’t get before is that the priority folder can basically contain the full cloud storage structure, where you only temporarily expand the folders where you need priority upload. I was still assuming that every time you’d need a priority upload, you’d need to manually create a new folder and mount it to the right remote location. Good to know that that wouldn’t be necessary :slight_smile:

I’d like to try and test your proposed workaround. Is the priority folder function already available, or is it yet to be implemented?

1 Like

Hi @skander,

Yes, this is just using the “sync to odrive” feature to create the new local folder to remote folder relationship. You can set it at a top-level folder so that you have the whole structure available. Just let me know if you have any questions. Manage Sync

Right, thanks. Perhaps I forgot to mention that we do not use the default odrive folder at all. All our remote syncs are sync mounts on our local storage. Would your way of using a second sync mount for the same remote location still work in this case?

Hi @skander,

Yes, it should work. Give it a shot once you get a chance and just let me know if you encounter any issues or have any questions.