ODrive Git Support

[Tony]

So how are you using sync with your git projects? What is the use case?

Configurable exclusions is something we have planned, but it will take some time to release it.

[gullitmiranda]

I use odrive to sync the folder where I put all the projects that I’m working on. Many of these folders has a .git. I don’t want that the folder .git is synchronized by the odrive already that I use the services of git (github, gitlab, bitbucket, …).

The 2 main reasons I do not want to sync .git are:

1. conflicts that break the git;
2. because of the number of files, there is high cpu usage;

The 2nd case is also the reason I need .odriveignore, because the folders of the dependencies, builds, (node_modules for example).

[hpg]

Please let us configure the exclusions. The suggested .odriveignore seems to be a flexible solution to this.

My use case are Unity projects. The project folders contain folders like “Library” that could be completely excluded without loosing any relevant data.
If .odriveignore will work similar to .gitignore those contents could simply be copied.

[Tony]

Hi,
Custom exclusions can be discussed in this thread:

Custom exclusion lists for sync (blacklist control, simple file-extension exclusion, etc.)

Starting this thread from thoughts by @EthanH so that others can engage, too… Can odrive please give the ability to add per-user file exclusion masks? Better still would be the option to specify exclusions on a per-share basis as well as globally. Do others want this capability as well? What are your specific needs? Is a simple exclusion list based on file extensions good enough (probably best bang for the buck and least complexity to understand/use)? Or do you need the ability to filte…

For the current thread, I would like to hear from folks who were previously complaining about blacklisting .git and .gitignore, which has now been removed.

[Smudge]

BTW, since .git and .gitignore are no longer ignored, they should be removed from the Ignore List in the User Guide

[Tony]

Thanks. I updated here: https://docs.odrive.com/docs/sync-changes#section--ignore-list- in our new documentation. We will be phasing out the old stuff soon.

[dubaaron]

Yeah, OK, syncing .git does turn out to be somewhat of a pain, because it takes SO long, and uses so much CPU, because there are so many files … it takes so long it’s sometimes useless. I’m ending up using rsync to do my .git folder instead.

It’s mostly so I can share my working tree, commit status, etc., between laptop and desktop, home and work.

It still makes sense to be able to turn it on and off, though, because not all use cases are created equal, and not all conditions are ideal (ie., using git itself to sync between multiple locations, sometimes is difficult due to a particular setup.)

[dubaaron]

I call uncle. I want to turn it off. I have seen the error of my ways! I have seen the light! I was wrong, you were right!

Problem is, it just takes so long, and has potential of polluting the git repo with " (conflict)" and “.cloud” files.

I still think it ideally would be allowed to be turned on, for those who know what they’re doing, the downsides, and want to enable it anyway.

[Tony]

Appreciate the feedback. I was hoping that some others would chime in with their own best practices for using git in this way so that others could benefit.

I believe the key is careful coordination: syncing down everything before local changes are made and making sure you are not making changes across machines “simultaneously”.
The last one can be tricky because simultaneous in sync terms doesn’t just mean that the actions happen simultaneously in “real time”. The delays in pushing changes can be quite dramatic if the system is offline, or odrive is not running, for example. If a change is made on the 1st system and that change is not brought down, in its entirety, on the 2nd system before changes are made locally on the 2nd system you will start hitting issues.

[enrico.castella]

Hi there, I’m just evaluating and found this thread, while deciding my best practice workflow when syncing folder which could have git repos inside.
Sorry for the long post and for cross-talking issues/featRequests I find related.

Let me explain some of my scenarios with some folder tree.

1. A folder which is a local repo created with “git init”:
   • I have no internet repository (github,gitlab,onPremise) for this gitRepo;
   • should be up-synced with cloud overwrite form the master PC1;
   • on PC2 the folder should be always down-synced (the cloud is the “master”)
   • I see this is a wanted feat by other user, which I could be left w/out (my personal opinion), I document it here for the sake of completeness;

• I understand that other users already tried this and found that in this case odrive is a CPU hog, I haven’t tried it;
  Folder1 |-- .git | |-- COMMIT_EDITMSG ...snip.........other git specific files/folders | |-- objects | | |-- 00 | | | `-- 9595005cb291047e6533e69bc8e1111a9037b7 ...snip.........very long list of objects |-- .gitignore ...snip.........actual content of the folder |-- .hiddenfile |-- .hiddendirectory |-- Docs | `-- Architecture.txt |-- DB_SCHEMA_1.0 | `-- samples.sql |-- DB_Schema_pre_1.0 | |-- SQL_Queries1.sql | `-- SQL_Queries2.sql ...endsnips

1. The following is a folder which contains sub-folders which I would like to include/exclude from up/down sync:

• non-git-versioned folders
• git versioned folders
• other content .
  Folder2 |-- .hiddenfile <----------- I do not want to up/down sync this hidden file |-- .hiddendirectory <------ yes please, up/down sync this hidden folder |-- Folder3 <--------------- yes please, up/down sync this sub-folder | `-- Architecture.txt |-- Project1.git <---------- I do not want to up/down sync this sub-folder | |-- .git | |-- README.md | `-- sample.txt |-- Project2.git <---------- yes please, up/down sync this sub-folder |-- .git |-- README.md `-- sample.txt

I do see this “feature” of .git folders to be linked to the following:

• ability to custom create white/black lists on folder base (in my opinion this could address all issues on .git)
• custom white/black lists should be customizable differently on each folder
• ability to choose the preferred “direction” of up/down sync for each folder (maybe with some default for sub-folders that inherit behavior from the parent)

Let me add that definitely this issue (and for sure white/black listing) would be a decision point on wether to pay or not the subscription.

Thanks for listening

[Ethan]

Odrive would benefit from something along the lines of Apache’s <Directory> block syntax and/or .htaccess files. This would allow defining both global and per-folder white/black list entries.

I suspect that as with Apache, defining the exclusion masks in a global config file would cause less of a performance hit than needing to parse individual files for each folder. That would also help in multi-user environments where exclusion masks may well differ between users and systems.

[jordanbtucker]

You stopped ignoring .git folders! This is amazing news! I can almost start using ODrive again. I stopped once I realized that it was ignoring .git folders by default with no way to include them.

This makes a lot of sense since “Git” does not equal “backed up or synced.” By definition, Git is a _de_centralized version control system, which means not all Git repositories are synced with a server and not all repositories exist in more than one place. I don’t sync all of my repositories with GitHub or BitBucket. Some of them are just local and that’s all they’ll ever be.

Now if you would just add a custom exclusion list (your second most requested feature) I could move away from Dropbox for good and stick with ODrive.

[alex.hakso]

looks like this ignore list needs to be updated with the .git and .gitignore changes!

[Tony]

ODrive Git Support

Thanks. I updated here: https://docs.odrive.com/docs/sync-changes#section--ignore-list- in our new documentation. We will be phasing out the old stuff soon.

The old usage guide still exist, although it shouldn’t be linked to directly by anything anymore. Are you still seeing it linked to somewhere on our pages or in the app?

[alex.hakso]

Just in this thread via comments (you Aug 4, Smudge Aug 30), probably not important

[dubaaron]

Has there been any change on this topic/feature? This seems to be the most recent posting/documentation on this topic.

I was one of the ones originally asking for .git files not to be ignored; at the time, I was a relative newbie in my usage of git.

Since then, I have been using git a lot more and have gotten more comfortable with git workflow, and also still using odrive, but keeping them strictly separate. I found that syncing is much faster and trouble free by just pushing/pulling to/from a remote repository.

Now I am experimenting with git annex, to try and help organize the files that are across my different cloud providers, and running into this problem again – syncing the .git directories is slow because there are so many files, and am afraid it’s going to cause errors in the repo.

Do we now have a way to toggle which files we can ignore for syncing purposes? I would rather not sync .git for now, but I can see where one might want them to be included, so it seems better to allow this to be toggled by the user vs. hardcoded in the program.

I notice there is a .odrive file in the root of my odrive folder which seems to be in JSON format; is there some way something like this can be used to toggle options like that?

Thank you very much, Cheers,

aaron

[Tony]

Hi @dubaaron,
There hasn’t ben any change to this behavior.

Syncing git in with odrive is something I would say is definitely on the advanced side of things. I work on several repos on a daily basis and I don’t use odrive sync on any of them. Instead, as you said above, I use git in a centralized workflow.

Custom exclusions is something that will be coming in the next generation odrive product. I don’t have a timeline to give on that, however.

[dubaaron]

Thank you very much @tony for the information. : )

[aaron.caito]

I don’t know if this is a necro but I’m curious if there have been developments here? For most my git repos i probably only want to keep my .git/config so i can clone back down, perhaps others i want to ignore the same things .gitignore ignores - or not. I have millions of files being backed up by odrive and my syncs are very slow because of so many developer deps / libraries/frameworks clogging up the works.

[Tony]

Hi @aaron.caito,
We did add an advanced feature that allows for a globally-applied blacklist. Take a look at this post for more information:

Custom exclusion lists for sync (blacklist control, simple file-extension exclusion, etc.)

My apologies for not updating this thread sooner… We added an advanced Premium feature to the product a little while ago that can be used to add items to our global ignore list. Documentation for this feature can be found here: https://docs.odrive.com/docs/sync-changes#section--custom-ignore-list-blacklist-