Just started using this yesterday. Worked great when I was only syncing a few files.
Then today, I attempted to sync 100,000+ files, and OS got slower and slower and slower as I went. I checked mem usage, and it was 60.0 GB! I only have 16 in ram; the rest was being swapped.
Is this an issue just with first sync or will this continue to use so much memory?
Thank you!
Hi @michael.steinberg.ir,
Bulk sync operations can be very intensive, but I have never seen 60GB of memory utilized, even from folks reportedly uploading over a million objects.
The complexity of the structure can impact performance. Is there anything of note about the structure of the 100,000+ files?
In any case, a bulk operation like this with over 100K objects will have a lot of overhead. We recommend breaking up the total into smaller chunks to alleviate some overhead. We also recommend unsyncing sections that you may not need immediate access to so that odrive can focus on just the items you do need to access/sync on that machine. Since odrive is a full bi-directional, near real-time sync engine, the larger the data set, the more work it will continually have to do to make sure everything is in sync on both the local and the remote sides.
Once the bulk operation has completed, you should see standard behavior, without excessive memory or CPU usage.
Sorry for the delay. Finally getting back to this.
There are two factors that may be contributing to the issue, but I am not sure:
Thank you
Hi @michael.steinberg.ir,
That should be okay. Syncing your Documents folder is a typical use case for folks using the “sync to odrive” feature. There shouldn’t be this level of memory consumption, unless there is something weird about the structure (ridiculous number of folders/very complex folder structure, a huge amount of objects, or a self-referencing symlink inside that is causing infinite recursion when we are scanning and syncing).
Are you still seeing the same behavior? Is odrive still in the process of syncing it?