Synced folders have different numbers of files

I am syncing an encrypted folder (with many subfolders) across three Windows computers, but when I look at the File Properties for the main folder on each computer, Windows reports a different number of files on each computer. Computer A reports 13,993 files and 2,866 folders. Computer B reports 13,798 files and 2,875 folders. And computer C reports 13,962 files and 2,875 folders. And it has been at least 12 hours since I changed anything in the main folder (or any of the subfolders). What should I do, please?

Hi @mbarre2004,
Are you sure that all folders are expanded (no more .cloudf files)? Typically any discrepancy like this is due to having some placeholder folders still around.

Can you do a search for *.cloudf on each computer and see what it comes back with?

Thanks for the prompt response, Tony.
There are no cloudf files under the main folder on Computer B.
I’ll check computers A and C when I get back to the office tomorrow,

Hi @mbarre2004,
Are these all the same version of Windows too? I’m just thinking its possible there may be system files (like thumbs.db) that are on one system, but not another. There could also be a dependency on which options you have set in Explorer (hide system and hidden files, for example).

Computers A and C now agree on 13,961 files under the main folder.
Unfortunately, computer B still has 13,977 files, or 16 extra.
And none of the computers have any hidden files under the main folder, according the the folder properties.
And Computer C does not have any cloudf files under the main folder.
Computer C is Windows 10, Computer B is Windows 8.1, and Computer A is even older than B and C (maybe Windows 7? - I could check if that would help.)

Got any more advice on how to find the 16 extra files on computer B, or some other way to resolve this issue, please?

Hi @mbarre2004,
Did you do a search for thumbs.db or desktop.ini? These two files are Windows-generated and could be throwing off the count. If you still can’t find it then we can probably run a script to list all files and then diff the list to see what the difference is.

I searched for thumbs.db and desktop.ini, but did not find any.
And on two of the computers, I copied the files to a folder that is not being synced, and I split the contents into 4 subfolders.
On computer C, nothing strange happened.
But on computer B (with Windows 8.1), something strange did happen.

  • Not very strange: the properties of the new main folder report a total of 13,977, which matched the count of the synced folder on computer B, but not the count on the other two computers.
  • Strange: the properties of the four subfolders report four numbers that don’t add up to 13,977. Instead, they add up to 13968.
    Additionally, three of the four subfolders have more files on computer B than computer C, and one folder has fewer files on computer B than on C: 660 v. 654; 7,773 v. 7,770; 4,890 v. 7884; and 645 v. 653.

About your script idea, since the files are encrypted, even if you could see which files don’t match, wouldn’t the file names be gibberish?

If you have other ideas, I’m all ears.

And in the meantime, I’m going to look into the advice here:
http://www.overclock.net/t/927069/folder-properties-showing-invalid-information

Edit:

  1. I turned off “Hide system files” on computers B and C, and the only change was computer B now reports 13,978 files, instead of 13,977.
  2. I found a clerical error. All 4 folders on computer B report more files than computer C, include the 4th folder, which reports 654 files (not 645).

Hey! After turning off “Hide system files,” a search on computer B turns up 11 thumbs.db files!
I’ll delete them, do another search etc., and get back to you.

1 Like

Tony, in addition to the thumbs.db files, I found 4 files with names that started with ~.
Three of them were temporary files (two ending in “.doc” and one ending in “.tmp”).

The fourth was actually a PDF file that I received from a colleague in Europe. I don’t know why he used ~ to start the file name, but it is a file I wanted synced.

Does ODrive ignore all files that start with ~ (tilde)?

Hi @mbarre2004,
Yes, files starting with ~ are ignored since they are almost exclusively temporary files. Here is a full list of ignored files, extensions, and prefixes. https://docs.odrive.com/docs/sync-changes#section--ignore-list-

Thanks, Tony. BTW, you might want to edit that list. At least on Firefox, part of it looks like this:
com1
com2
com3
com
com
com
com
com8

Thanks for pointing that out @mbarre2004. I have corrected it.

Hi @Tony!

I just wanted to follow up on this topic. I just finished a 2-month project (you helped me before here: Google Drive (G Suite) on macOS High Sierra) and I have some questions relating to these hidden files and count discrepancies.

I currently have uploaded ~11.56TB out of my ~11.57TB (on a Drobo, not that it matters). I have uploaded 847,441 items out of the 850,323 items counted on my Drobo. I was able to get these counts because I am using Google Drive’s File Stream, and they let you do item counts on the entire volumes as if they were local, really amazing stuff! :slight_smile:

So clearly there is a mis-match of ~2,882 items between the two folders. I found this topic after searching online, and when doing a test, I actually found 3,055 “Thumbs.db” files (I migrated from Windows to macOS a few years back), and 22 “desktop.ini”/“Desktop.ini” files. If I deleted those files, then I would actually have more files on the cloud than I would locally which doesn’t add up… Some folders seem to have more items after they synced to the cloud than they did before. Lots of little inconsistencies like this, which is understandable given the massive amount of data I have. I am curious, is there a way to turn off or edit the ignored files list? I’d rather have a few thousand useless/messy files on there than have mis-matched item counts. I’m just hoping to have more confidence that all, or nearly all of my files and folders have been synched. I have also looked into using the Kaleidoscope app, but I’m not sure how helpful it will be just yet…

It’s worth noting that before I started this process, I had over 4 million items. I spent a few weeks zipping up old folders with tons of files that I won’t likely ever need to access again. I will also do a few more passes on this since zipping up files is an effective way of masking these types of messy/ignored files. It also greatly speeds up the processing/syncing process after each restart.

It is unreasonable for me to shoot for nearly-exact-or-exact item counts for peace of mind? Should I just not worry about it? I’d love to hear your thoughts and hear any tips you may have to get closer to 100% accordance. I am also now more careful about adding ignored files and incomplete files/downloads with the leading “~” character from now on. Are there any tools that you’d recommend that are practical/safe?

Oh! I forgot this related question. Why do you think that Google Drive, macOS’s “Get Info”, and my Drobo all give different size counts? This is how my data is counted across devices/counters:

11.57 TB -> My single top-level folder on my Drobo being counted by macOS’s “Get Info”.
11.56 TB -> My single top-level folder on my Google Drive File Stream volume being counted by macOS’s “Get Info”.
10.7 TB -> Google Drive’s website.
10.71 TB -> Google Drive’s iOS app.
10.53 TB -> Drobo Dashboard on macOS.

How can all of these counts be so different? The difference between my macOS “Get Info” count and the Drobo software’s count is ~1.04 TB. Given your team’s intimidate knowledge of data, I’m curious if you have any insights as to how this could be explained. It reminds me of currency exchange rates, and the relationship between megabits and megabytes.

Thanks so much! :slight_smile:
Tim

Hi @timsilva,
There isn’t a way to turn off the exclusion, unfortunately. I can certainly understand the desire to make everything line up, so I don’t think it is unreasonable.

In addition to ignored files, symlinks could also explain the difference in the number of files and total size. odrive will follow symlinks. This means you could have the same file uploaded twice, once for the symlink and once for the actual file, if both are somewhere inside the odrive folder. The same would apply to folders symlinks.

Are you able to pinpoint any specific examples where there is a difference in the cloud vs the local system?

As for storage reporting differences, you can get these depending on what is being reported and how the storage is being calculated. The apparent size of a file (the amount of data that a file contains, as it appears to applications) may be different than the actual disk usage for storing that file. They can differ because of things like disk compression, sparse files, or partial block usage (“empty space”) when a file requires an additional storage block, but only uses part of it. You could also have differences if symlinks are used and they are followed when calculating sizes, which would make it seem like you have more data than you actually do.

@Tony Woo, another detailed response! :slight_smile:

I will probably just be okay with what I have uploaded now. My current upload is ~99.94%, so I think that should be fine and I should stop worrying about it :blush:

I have been looking for specific examples. I’ve found a few Thumbs.db files, then there are some small folders with a handful of files that report different sizes/items. I might continue to dig around for more examples and attempt to fix the differences by zipping more folders, but that could take days so I will probably just let it go.

I wasn’t familiar with the formal term symlinks, new word to me! It’s like “tissue paper” is to “Kleenex”. I’ve always just said shortcuts/aliases (Win/Mac), glad to know there is a non-branded word to describe these types of files/folders. But yes, I use TONS of aliases. I don’t have a way to count them, but I must have at least 10-15k of them sprinkled throughout my ~850k items.

Thanks for also explaining the storage differences. That is best explanation I’m aware of, I appreciate it. :slight_smile: I wonder how common it is for these differences to appear. It also makes sense that the Drobo software reports the smallest count because of their proprietary RAID technology. I have 5x4TB hard drives in there, so perhaps their super-efficient software somehow makes saves ~1TB of space by not wasting any clusters/blocks… I have files ranging from ~1B to ~30GB zip files, so there’s lots of opportunities for these discrepancies to occur I suppose.

Thanks again for sharing the knowledge!

No worries @timsilva!

These types of operations are always stressful, but it sounds like you’ve got a good handle on it. Archiving areas of your storage into zip files sounds like a good way to declutter and help organize , too.

1 Like

Thanks for the suggestions.

1 Like