is there a functional limit to number of files in a folder supported by the client?
There aren’t any limits that odrive imposes, but you can start running into limitations and performance issues, such as:
- Windows Explorer/MacOS Finder delays or increases in CPU/memory overhead when browsing and listing all of the objects along with their badge overlays
- Increases in odrive CPU/memory overhead as it maintains sync tracking and scanning on all of the objects
- Storage service delays when trying to get the current listing of the folder
- Storage service object limitations (both global and per folder)
ok, well… i have a backup directory on my amazon drive that’s got over 600,000 objects in a single folder… (that’s how Mylio did it)… and i’m trying to pull it all down locally to no avail as the client crashes when enumerating the folder.
It gets to 349,525 and falls over.
01 Dec 04:33:39PM INFO Failed Folder refresh attempt (R) for: D:/odrive/Amazon Drive/Mylio/26f5a5046b77b2bd08a1f252c17680cb2d27a6f9_U. Error: code stopped
My guess is that it is running into some form of a memory error. We haven’t tested with a single folder of nearly that size before. A folder listing works as a single internal operation and we are likely running into problems with the in-memory tracking as it processes that gargantuan folder.
I will see if anything can be done, although it may take a little while to get it into the queue.
Thanks… I upped the memory limit to 8gb in conf and it was able to get up to 448,600.
Also disabled onaccess EDR. Upping to 16gb to see where that gets me.
I started testing this a bit over the weekend and then populated an Amazon Drive folder with 500,000 files (which took a couple of days with Amazon’s rate limits … heh).
I’ll try to dig into it more this week.
Thanks @Tony! Guess i need to have a little more patience due to 429s eh?
@Tony, how can I turn up log verbosity to see what’s really going on?
07 Dec 10:02:18AM INFO Start scan and refresh mount D:/odrive
07 Dec 10:09:34AM ERROR Encountered error scanning: D:/odrive/Amazon Drive/Mylio/26f5a5046b77b2bd08a1f252c17680cb2d27a6f9_U during local scan (mount).
There isn’t a way to enable more verbose logging than what you are seeing, right now.
With the current client you are running there are going to be issues trying to download a single folder with 600,000 items in it, as you are experiencing. It’s a pretty crazy and impractical organization scenario and the client just wasn’t designed for it.
I recommend waiting for me to provide a new version to test before continuing, as it is likely just going to just lead to more frustration, otherwise. It may eventually finish, but the cards are stacked against it with the current client.
Thanks @Tony. I wish Mylio had done their backup structure differently, for sure.
Totally understand how impractical it is… Odrive is my best shot at recovering some lost images since Amazon killed Drive client. Please let me know when there’s a build i can test with. I have plenty of local compute and storage to throw at it.
- Exit odrive
- Download the installer from the link above
I’ve tested this version with a single folder of 500000 files, so I am hoping that it will allow you to get your data downloaded.
My suggestions to help things along:
Once the new version starts and you can interact with the odrive tray menu, go to the top of the tray menu under “Ready to sync new changes” and click on “Disable background scanning”
Do not navigate into the large folder in Windows Explorer. We don’t want an automatic refresh to be triggered.
Right-click on the large folder, select sync, and then set the options as follows:
- Slide the slider all the way to the right for, “Everything”
- Check “Include Subfolders”
- Check the “Force refresh during sync” checkbox.
- Click the silver sync button
Once the sync is kicked-off, close any open Explorer windows
Monitor the status via the “sync activity log”
6. Wait for a long time
Thank you! Taking it for a spin now!
For anyone who may stumble on this thread later… easy way to monitor the sync log is w/ powershell.
Get-Content "$($env:USERPROFILE)\.odrive\log\main.log" -Tail 20 -Wait
I just wanted to check-in on this and see how is going/went. Did the new version help?