Files in a folder item limit

Troubleshooting
1

is there a functional limit to number of files in a folder supported by the client?

2

Hi @warren,
There aren’t any limits that odrive imposes, but you can start running into limitations and performance issues, such as:

  • Windows Explorer/MacOS Finder delays or increases in CPU/memory overhead when browsing and listing all of the objects along with their badge overlays
  • Increases in odrive CPU/memory overhead as it maintains sync tracking and scanning on all of the objects
  • Storage service delays when trying to get the current listing of the folder
  • Storage service object limitations (both global and per folder)
3

ok, well… i have a backup directory on my amazon drive that’s got over 600,000 objects in a single folder… (that’s how Mylio did it)… and i’m trying to pull it all down locally to no avail as the client crashes when enumerating the folder.

It gets to 349,525 and falls over.

01 Dec 04:33:39PM INFO Failed Folder refresh attempt (R) for: D:/odrive/Amazon Drive/Mylio/26f5a5046b77b2bd08a1f252c17680cb2d27a6f9_U. Error: code stopped

4

Hi @warren,
My guess is that it is running into some form of a memory error. We haven’t tested with a single folder of nearly that size before. A folder listing works as a single internal operation and we are likely running into problems with the in-memory tracking as it processes that gargantuan folder.

I will see if anything can be done, although it may take a little while to get it into the queue.

5

Thanks… I upped the memory limit to 8gb in conf and it was able to get up to 448,600.
Also disabled onaccess EDR. Upping to 16gb to see where that gets me.

6

Hi @warren,
I started testing this a bit over the weekend and then populated an Amazon Drive folder with 500,000 files (which took a couple of days with Amazon’s rate limits … heh).

I’ll try to dig into it more this week.

7

Thanks @Tony! Guess i need to have a little more patience due to 429s eh?

8

I get a bunch of these too… which is very confusing.

image
image827Ă—430 22.3 KB

image
image1752Ă—1005 130 KB

9

@Tony, how can I turn up log verbosity to see what’s really going on?

07 Dec 10:02:18AM INFO Start scan and refresh mount D:/odrive
07 Dec 10:09:34AM ERROR Encountered error scanning: D:/odrive/Amazon Drive/Mylio/26f5a5046b77b2bd08a1f252c17680cb2d27a6f9_U during local scan (mount).

10

Hey @warren,
There isn’t a way to enable more verbose logging than what you are seeing, right now.

With the current client you are running there are going to be issues trying to download a single folder with 600,000 items in it, as you are experiencing. It’s a pretty crazy and impractical organization scenario and the client just wasn’t designed for it.

I recommend waiting for me to provide a new version to test before continuing, as it is likely just going to just lead to more frustration, otherwise. It may eventually finish, but the cards are stacked against it with the current client.

11

Thanks @Tony. I wish Mylio had done their backup structure differently, for sure.
Totally understand how impractical it is… Odrive is my best shot at recovering some lost images since Amazon killed Drive client. Please let me know when there’s a build i can test with. :slight_smile: I have plenty of local compute and storage to throw at it.
Much appreciated!

12

Hi @warren,
Here is the new version

To install:

  1. Exit odrive
  2. Download the installer from the link above
  3. Install

I’ve tested this version with a single folder of 500000 files, so I am hoping that it will allow you to get your data downloaded.

My suggestions to help things along:

  1. Once the new version starts and you can interact with the odrive tray menu, go to the top of the tray menu under “Ready to sync new changes” and click on “Disable background scanning”

  2. Do not navigate into the large folder in Windows Explorer. We don’t want an automatic refresh to be triggered.

  3. Right-click on the large folder, select sync, and then set the options as follows:

    1. Slide the slider all the way to the right for, “Everything”
    2. Check “Include Subfolders”
    3. Check the “Force refresh during sync” checkbox.
    4. Click the silver sync button

    recursive_sync_win_force_new

  4. Once the sync is kicked-off, close any open Explorer windows

  5. Monitor the status via the “sync activity log”

sync activity log
6. Wait for a long time :wink:

13

Thank you! Taking it for a spin now!

For anyone who may stumble on this thread later… easy way to monitor the sync log is w/ powershell.

Get-Content "$($env:USERPROFILE)\.odrive\log\main.log" -Tail 20 -Wait

1 Like
14

Hey @warren,
I just wanted to check-in on this and see how is going/went. Did the new version help?