Best way to sync over 400,000 files/directories

Hi @toupeiro,
So it sounds like you are wanting to do a one-time bulk import of all of this stuff, is that correct? I will assume it is, for now, and continue.

As you may have read in other threads, sync is a somewhat heavy process compared to traditional backup, and certainly to a one-time import. There are things we can do to try to alleviate the sync overhead and simulate more of an import flow, however.

The most basic flow would be:

  1. Copy/move the data in in batches
  2. Wait for sync to complete on the batch
  3. Unsync the completed batch
  4. Repeat

This constrains the scope of work odrive needs to do to the size of the current batch. You could do this “by hand” or script it up using something like this example with the CLI:

How is the data structured? Is it naturally segmented in a way that would allow for easy batching?