Odrive hangs uploading a large number of files

Troubleshooting
, ,
1

I’m in the process of uploading an archive of an old linux machine to Google Drive. It’s somewhere around 200GB, contains somewhere around 1 million files inside 100k folders. What appears to happen is the odrive process hits approximately 1.8GB of physical memory usage and approximately 2GB virtual memory usage (“Private Bytes” and “Working Set” in Sysinternals’ Process Explorer), and locks up. My only resort is to kill it and restart. Upon restarting, it appears to run for awhile without issue, but then pocks up again some time later. When it’s locked up, the icon has turned white and no longer responds to any clicks. It appears that this is a secondary symptom, as I resumed the upload (kill/restart) about 20 minutes ago and the process is back up to 1.8GB in memory usage, but the icon is still animating and pink. What I suspect will happen is that it will finish whatever tasks it’s currently performing and then hard lock at the white icon, as that’s the state I keep finding it in.

The 2GB virtual memory is eerily close (I don’t recal the literal size and I didn’t screenshot it last time, so I can’t be exact) to the 2GB single file size limit of 32-bit processes. Is odrive running out of memory? Have you considered switching to a 64-bit process? I suspect a number of these types of issues might go away with a 64-bit executable, able to use more than 2GB of memory.

2

Hi @ndgeek07,
64bit Windows binaries is something we are looking at for the next major release.

For now I highly recommend trying to break up the set. That is a pretty massive data set you are bulk uploading. odrive needs to scan and track that entire set in real-time, both on the local and remote sides. Since odrive is full sync rather than traditional backup, the processing is heavier. Breaking up the dataset and letting the “segments” sync, before adding the next can help a lot.

Another thing that can be extremely helpful is to unsync sections that have already completed syncing to the cloud. This reduces the terrain that odrive needs to cover and relieves overhead processing.

Another option is to use odrive on the Linux box, itself. It may help performance and the Linux agent is built with a 64bit option. I would still recommend breaking up the set, if possible, though.