Thanks for the information! Everyone’s data is different and you look to have a lot of what we call “sprawling” folder structures. These are typical in photo and video editing type of software, where they tend to have these ridiculously complex data structures with thousands and thousands of folders. You’ll see these, for example, in software like Adobe Lightroom, Apple Photos, and Scrivener.
The most difficult use cases to make really smooth are those dealing with bulk operations, like bulk downloading, especially against sprawling structures, as mentioned above.
One of the big ideas behind odrive’s sync engine is the idea of “progressive” sync, where you can get to and sync the specific data you need at that time without having to sync down the entirety of your remote data. The ability to do this can cut out a ton of unnecessary overhead because it allows odrive (and users) to focus on a specific subset of the data instead of having to monitor and track everything. Because this is sync and not basic import/export, during and after bulk operations odrive still needs to constantly look at everything, on both the local and remote side, to make sure it is all in sync. This is a constant, never-ending task that odrive has to perform. The larger the scope of the data that odrive needs to monitor, the more overhead you are likely to have.
This all being said, we still want to be able to make these difficult scenarios smooth. The information you provided yesterday actually pointed out a situation where we can potentially get into a feedback loop when bulk downloading thousands of these sprawling types of folders in rapid succession. Here is a test version that addresses that situation. I am hoping you can run to see if it solves a lot of the CPU heat you are seeing: https://www.odrive.com/s/aea2a683-cacc-4d4c-b5e9-44ea967fd55b-5ef2179d
So that I can further understand your method of odrive use and use cases better, can you walk me through what actions you are taking prior to recognizing that things are not syncing?
Can you also describe the behavior when you notice that syncing is not working? Is this for downloads, uploads, or both?
The status files you sent over actually indicate that things are syncing, but it is a bit deep in the bowels of some of these folder structures, so it may not be readily apparent that something is actually happening. For example, in the last odrive status file you sent, it looks like it was working its way through one of these Lightroom structures (you can see it under “Active (4):” in the status file).
There are some tricks we can employ, as well, to cut down on unnecessary processing, like identifying caches or what I call “throw-away” files and folders that could be excluded from sync using our custom blacklist feature. It’s just a matter of identifying the right stuff.