Rebuilding a sync from backup which has .cloud files in it

Troubleshooting
, , ,
1

I had to preform a full rebuild of my MacBook Pro when I upgraded to High Sierra because the existing install was performing terribly because of kernel module problems. I made a full backup of the MBP’s SSD onto my local NAS so that I could copy my very large (300-400 GB) file repository that I sync with Google Drive from a local location rather than have to download the whole thing from the cloud which would take days.

I now have copied that repository that is in Drive back to my fresh installation of High Sierra and I am about to install oDrive, to sync the folders up, but I discovered something slightly unfortunate. It appears that a large number of files in the Google Drive folder are .icloud placeholders and not the actual files hemselves. When I made the backup, I didn’t notice this, so when I sync the folder up with its counterpart in the cloud, I will need to download these files. This is not that big of a deal and it will only take a few hours, but I want to make sure that I do this correctly. Before I install oDrive and set the folder to sync version on Drive in the cloud, should I:
a) Leave the .cloud placeholder files in place and direct the oDrive to download the real files wherever placeholder files exist
OR
b) put all of the .cloud files in the trash, install oDrive, and configure the the folder to get synced with the cloud counterpart and let it fill in the missing files with whats in the cloud?

I am not sure whether keeping the .cloud version will cause weird behavior by not recognizing the existing .cloud files are placeholder files which results in duplicates of the files, or placeholder files that totally confuse oDrive, or if my deleting the placeholder files will confuse oDrive when I install it and wind up deleting the versions which are on the cloud! Both scenarios seem plausible so I want to make sure that I do the right thing which is either keep the .cloud files when I install oDrive and configure it to sync, or if I should deltete the .cloud files before installing oDrive and syncing them.

Finally, should I consider using the CLI agent versus the desktop app? I realize that the desktop app is scriptable using the python client script, but I am wondering if the core function of the CLI agent vs the desktop app is different enough that I might consider using it. I generally prefer the command line versus a GUI as I want things that I can script and capture specific programatic behavior by supplying switches at command invocation rather than poking around a GUI configurator. Also, I am wondering if by using the CLI agent I might be able to explicitly command oDrive to do exactly what Iwant it to do in this case- that is replace any gaps in my backup with whatever is in the cloud.

Thanks in advance!

2

Recently I performed a reinstall of odrive on my Windows desktop, and can confirm the following observations:

  • On install, odrive doesn’t sync into any providers, they’re all .cloudf files in the odrive root
  • You can manually create .cloud and .cloudf files that odrive will pick up and use if they have the right name and are in the right place (they’re just zero-size files, you could create them with touch)
  • You can close the client, copy a backup of the folder with some previously synched stuff in it, then restart the client and it will recognise all the stuff you put in there (uploading things that didn’t exist before, etc).

That said, I don’t know how it works on a Mac and might be different. Good luck!

1 Like
3

Hi @DarfNader,
@yukihyou pretty much covered it, but you can take a look at this post here for some additional details (especially the note concerning Encryptor content):

Mac and Windows will work the same, in this respect.

4

@Tony I don’t think that I was clear enough in my request. On my mac I have a folder that was formerly synced with oDrive and is now out of sync. There are significant changes since the last sync. The folder is about 120 GB and there are 14K files. The changes maybe make up 500 MB of of data tops. What I need to do is sync this folder with the folder in oDrive (in Google Drive) so that I only download or upload changes, and so deletes and moves are recognized since the updat(The last part might be tricky, but the files are the same name. I realize that I moved some of them.)

In short, I want to:

  • Resync the folder with the cloud version
  • Limit uploads and downloads just to deltas

My concern is that since the offline folder is complete that ODrive will need to upload every single file again anyway to determine if there is a change. What I hope will happen it would just perform a checksum or even note the name, size, and last modified timestamps match what’s up in the cloud and move along.

So that all said, I still am unclear on how to do this. I have the Odrive folder with all the cloud folders in it which are mostly full of .cloud files. Do I drop the contents of my local copy of the Google Drive folder into the oDrive version and just hope ODrive will remove all of the .cloud versions of things and rectify just the changes?

Thanks,
Matthew

5

Hi @DarfNader,
Do you have a mix of changes both in the cloud and on the local system that you are considering the ‘latest’ version? If that is the case it is going to require manual finessing. The reason being that odrive will consider any newly introduced local content as content that needs to be synced as long as it is different than the current remote content and upload them, even if they are older or smaller in size.

For the same reason, it is also tricky if you want to only download changed data from the remote to the local, keeping the rest of the local files the way they are.

If you just want to make sure you upload any local data that has changed to the remote storage, then the above posts should fit that scenario.

6

@Tony yes, there is a good chance there is a mix, but not many. I might try to find files in the replica that are newer since the sync break happened. If I copy the local static files over to the Odrive location and the file that is in ODrive is newer, will it win out?

Also, and the is the big one, HOW do I set up a sync folder now? It looks like I can’t and that’s very odd.

7

Hi @DarfNader,
The local file will always win out if it doesn’t match what is in the cloud. odrive assumes your intention is to sync the local file you put in.

For creating a sync folder, you should be able to right-click on the local folder you want to sync (outside of the default odrive folder) and select ‘sync to odrive’. Do you see that option?

8

Hi @Tony , when I right click on the folder in Finder I do not see any of the oDrive integration options. I thought this is because the folder is not in the oDrive folder. That would certainly be the easiest sort of a aproach. The best option that I have is to rsync with another local computer that has a synced version of the folder which happens to have the entire cloud drive localized (it is a direct sync with Google Drive and not oDrive so I don’t have to worry about the placeholder file issue.

9

Hey @Tony, before I attempted the rsync I found that my Google Drive is full of .cloud files. The are thousands of them. I am trying to write some commands to get rid of them but since there are single quotes in names and such trying to deal with it using a find command is having issues with xargs. I’m not exactly sure how it happened, but I am using another sync mechanism on a NAS to sync files with the drive and it is is possible .cloud files got in there and since it doens’t know that .cloud files are to be ignored I there are everywhere like a fungus. Is there some way to get oDrive to clean this up when this happens?

10

Hi @DarfNader,

Is the folder you are right-clicking on a network share? Or are you not seeing any odrive options on any folders?

These are placeholder files that are not in odrive folders? odrive can’t work on any items that are not within the odrive scope. Although, if it was in the odrive’s view then you’d see a bunch of placeholders anyway :wink:

11

Hi @Tony

Is the folder you are right-clicking on a network share? Or are you not seeing any odrive options on any folders?

No, it is a local file. I only see oDrive options for items within the oDrive folder but nowhere else.

These are placeholder files that are not in odrive folders? odrive can’t work on any items that are not within the odrive scope. Although, if it was in the odrive’s view then you’d see a bunch of placeholders anyway :wink:

No, the placeholders were copied to Google Drive when I was rysncing a backup of my oDrive contents which I thought was fully synced but unfortunately wasn’t. My Google Drive was infested with them and it took a while it clean them out because of files with single quotes in them, etc… I think oDrive was confused seeing them on the other side of the equation and basically hid them from on the oDrive representation of the Google Drive. I think I was able to restore everything without data loss, but it took an afternoon to recover. When oDrive loses sync with a folder that was previous in sync, things really can go sideways. This has happened to me twice now on separate machines. :worried:

12

Hi @DarfNader,
That is really odd. If you create a test folder on your desktop and click on that new folder, so you see any options?

You mentioned its a local “file”. I just want to make sure you are doing this with folders and not files. If you click on a file I would expect something in the context menu to say “No options available”.

Do you have an odrive icon in the top of your Finder window (usually between the tag icon and the search field). If so, does that show any options for non-odrive folders? If not, can you try toggling the odrive extension in System Preferences -> Extensions?

13

Yes, it was odd as I can see on my other Mac that indeed there are oDrive options for files and folders outside of the odrive folder. I have no idea why this machine is different, but it’s pretty frustrating how I am having a host of issues across all three of my machines like this. And no, to be clear, what I mean is there is literally no oDrive-related menu options at all, as if the Finder plugin was not present when selecting those files. Normally, I would expect the oDrive options to be greyed-out like you said, but not here. They just aren’t there. Items in odrive have the menu items there so it is not like the Finder plugin has failed completely- it just doesn’t have any context function outside odrive. I have restarted Finder and deselected/reselected the oDrive extension for Finder to no effect. It just doesn’t work right. oDrive menu items only show up for files/folders inside odrive. One thing I will mention is that I do not see anything in Finder’s preferences that indicate any configuration for Finder Plugins. What do you suggest that I do to fix this? Considering the problems with odrive getting out of sync on a regular basis, this is something I need working as I otherwise have to spend a long time resyncing files using rsync.

Another problem which I am having on my work machine is my odrive folder will, without any warning whatsoever, get decoupled. It becames a static folder called odrivebackup-<date-and-time-stamp> and I am then putting my files into this static folder until two weeks later I realize it isn’t syncing and I now have a divergence. I cannot count the number of times this has happened and not one time has there been a clear reason for it happening. It is transcending being a nuisance and become what I would consider a serious problem.

I have to tell you @Tony, in spite of your excellent support for this product, at some point I will have to abandon oDrive because after over a year of struggling with it, it just as flaky as it was when I bought it and frankly I don’t feel like my data is safe using it at all. Any convenience and reduction of overhead using a consolidated cloud client is overshadowed by the endless problems I have with it. That is aside from the long awaited features like better integration with Dropbox which never seem to materialize. Unless I can isolate both of these problems I am not going to have any choice but to cancel my subscription. I feel like I have been excdeedingly patient and positively focused about using oDrive as genuinely want your product to succeed, I realize that the amount of time I have spent dealing with the fallout when it fails, which seems to be monthly, is ridiculous. To know that I am paying for what I feel like is being a member of your QA team actually makes me kind of mad. Please forward this to your development and management team. I know that you can vouch that I have been a loyal and patient customer for nearly two years but I simply cannot waste any more time with it. I harbor no ill will- it’s simply a matter of practicality.

I hope that we can get this resolved for the sake of your product’s ongoing success.

Thanks,
Matthew

14

Hi @DarfNader,

Is your odrive folder in the normal place (in your user home folder)? The only thing I can think of is that there would be something different about where your odrive folder is vs the folders you are clicking on elsewhere. I saw a case once where the user’s name and the user’s home folder were different cases and it it caused some inconsistencies in the context menus and icon overlays, but that is about all I can think of for this issue. Since we know the Finder extension is loaded, it should be displaying something.

Can you clarify what you see from this menu option in folders that are not the odrive folder?

27 AM
27 AM1088×56 8.68 KB

Is it just empty when clicking on it?

Can you clarify this? Do you mean you do not see the “Extensions” section in System Preferences?
What is the current version of High Sierra on this machine?

This should only happen if the client has been uninstalled, odrive was deauthorized, or the local databases had an issue and weren’t able to be read properly. In the last case a new odrive folder would be created with all placeholders. In the deauthorized case, odrive would be set back to a vanilla state where you would need to log in again.
When you see this happen, is a new odrive folder created?
What type of system is this on?

I have flagged this post for the team to review. I can certainly couch, and am very appreciative, of your loyalty and patience and willingness to investigate and I understand your position. The next generation product, which resolves and improves on so much, is on the horizon, but it is coming much later than we wanted, and I realize that it may be too late for some who have been waiting for it…

15

Thanks @Tony for your continued support on this. Sorry that I am such a “problem child” but things have gotten really messy and I want to just clean things up and “start fresh” as best that I can.

Is your odrive folder in the normal place (in your user home folder)? The only thing I can think of is that there would be something different about where your odrive folder is vs the folders you are clicking on elsewhere. I saw a case once where the user’s name and the user’s home folder were different cases and it it caused some inconsistencies in the context menus and icon overlays, but that is about all I can think of for this issue. Since we know the Finder extension is loaded, it should be displaying something.

It’s in my home directory. I am assuming at this point there is some messed with my configuration. Is there a process that oDrive prescribes for wiping my machine clean and then resyncing the drive in a manner that ensures I don’t overwrite newer files wile older versions? Things are getting so precarious at this point I worry about the integrity of all of my files and data loss is imminent. If things get jumbled, having a backup is not going to be much good because it’s going to be a mish-mash of various versions with the risk of having inaccurate time stamps. If that happens, I honestly have no idea how I would recover the thousands of files that I have collect over that past 20 years.

One thing I will mention is that I do not see anything in Finder’s preferences that indicate any configuration for Finder Plugins.

Can you clarify this? Do you mean you do not see the “Extensions” section in System Preferences?
What is the current version of High Sierra on this machine?

Normally, under the “Extensions” panel for “oDrive” I see a checkbox for “oDrive” in here which presumably is what turns on these menu items in Finder. This begs the question why I see the menu items within the oDrive folder at all, but I don’t profess to understand how oDrive integration works.

It becames a static folder called odrivebackup-

This should only happen if the client has been uninstalled, odrive was deauthorized, or the local databases had an issue and weren’t able to be read properly. In the last case a new odrive folder would be created with all placeholders. In the deauthorized case, odrive would be set back to a vanilla state where you would need to log in again.
When you see this happen, is a new odrive folder created?

This is not what happened. In fact, I am pretty sure I was not even using the computer the date that is put on the backup (12/27/17). It would appear that a new odrive folder was created, but I have no idea if the new drive (which I realized that I was using several weeks after the backup was made) has is missing any files that are only in this backup. I would try to use rsync to compare them but with all of the .cloud placeholder files this process becomes challenging because you can’t compare file dates to placeholder files.

What would be helpful is a tool that would traverse a backup folder like this can compare data stamp, checksum, etc… to see if there are versions in the backup that are newer and different than the version currently in the cloud. Considering how frequently this happen (at least for me of three separate computers) this kind of maintenance tool would would amazingly helpful. In fact, it would be useful for whenever you want to sync a local version of a folder with a folder in the cloud. (Of course there should be a way to do the compare first and then apply the changes so you don’t accidentally smear files all over the place in case you accidentally get the directory level wrong or something. Please suggest this to the devs. It would save me a great deal of hair-yanking and downloading ENTIRE cloud directories locally in order to perform a validation.

What type of system is this on?

High Sierra

I don’t want to be too melodramatic but I really need to get all of my cloud versions in sync. I realize your dev team is busily working on the new version, but I hope that there is time to perform a little damage control over the current version for when oDrive fills its shorts. I’d be happy with just some quick utilities like what I suggested that resync orphaned folders with the cloud version so no updates get lost.

Thanks again for you support.

16

Hi @DarfNader

You can definitely perform a clean uninstall and reinstall, but there isn’t a way for odrive to reconcile file differences that extend in both directions (i.e. remote has a most updated version of one file while local has the most updated version of another file). odrive can perform a merge of local and remote, but local will be “dominant” version of the file if there are any differences, regardless of the timestamp.

I made a quick bash script to compare non-placeholders files in the current folder (the odrive backup folder, for example) with ‘real’ files or ‘placeholder’ files in a target folder (the odrive folder). It only compares the modtimes for the files, so it may not work in all cases. Most sources should have the correct modtime reflected in the placeholder files so you can see if there are instances where the files in the backup folder are newer than those in the remote storage.

#!/usr/bin/env bash
target_folder='/Users/yourusername/odrive/'
find . -type f ! -name '*.cloud*' ! -name '.DS_Store' ! -name '.odrive' -print0 | while IFS= read -r -d $'\0' line; do 
	source_path_modtime=$(stat -f "%m" "$line" | cut -f2-)
	cur_tar_path=$(echo $line | sed "s#./#${target_folder}#")
	target_path_modtime=0
	if [ -f "$cur_tar_path" ]; then
		target_path_modtime=$(stat -f "%m" "${cur_tar_path}" | cut -f2-)
	elif [ -f "${cur_tar_path}.cloud" ]; then
		target_path_modtime=$(stat -f "%m" "${cur_tar_path}.cloud" | cut -f2-)
	fi
	if [ $target_path_modtime -ne 0 ] && [ $target_path_modtime -lt $source_path_modtime ]; then
		echo "$line is newer!: Source time: $source_path_modtime Target time: $target_path_modtime"
	elif [ "$target_path_modtime" == "0" ]; then
		echo "$line doesn't exist on target!"
	fi
done

The way I was using this was to cd into the backup folder (cd ~/odrivebackup-2017-12-25) then run this script. Make sure you change the target_folder variable accordingly to point to your odrive folder. i’m not sure how useful this will be for you, but maybe it will help.

1 Like
17

Thanks! Love me some bash scripts. Not exactly knowing the behavior I am much happier that you made this. I really appreciate it and I will let you know how it pans out!

1 Like