Custom exclusion lists for sync (blacklist control, simple file-extension exclusion, etc.)

[JeffL]

Starting this thread from thoughts by @EthanH so that others can engage, too…

Can odrive please give the ability to add per-user file exclusion masks? Better still would be the option to specify exclusions on a per-share basis as well as globally.

Do others want this capability as well?

What are your specific needs?

1. Is a simple exclusion list based on file extensions good enough (probably best bang for the buck and least complexity to understand/use)? Or do you need the ability to filter on file/folder name filters/regex?
2. Do you need exceptions to the exclusion list? (e.g. whitelist on top of the blacklist)
3. Is this something you are looking for on an individual basis, or are you trying to set this up for a group of people (so that having each user custom-configure their exclusion list is inconvenient)? There are different ways that configuration could be done: on an individual basis, at a share-basis (e.g. as part of Spaces settings), and on a group/organization level (configured for all users in an organization).
4. Do you have any other considerations that we should be aware of? Or do you want to exclude files based on something else (e.g. file size, custom file metadata, etc.) other than file name or type?

Thanks!
-Jeff

[EthanH]

Jeff,

Thanks for opening up this discussion. As the complaining customer who launched it, I’ll add details as to our organization’s needs. What brought this topic to my attention was experimenting with syncing a folder of test images. Our latest cameras are 100MP, producing tiff images over 500MB each. The temp files were also large size and there was no reason to transfer them as well.

1: More than just an extension is needed. For example, one software package we use names temporary files as <filename>.$#$.<ext> where filename and ext are the original name and extension. A simple regex supporting * and ? wildcards would suffice. Ideally this would support folder and/or file names, e.g. *.tmp and /Scratch/*.jpg.

2: An additional whitelist could be useful, but is lower priority. As long as there is not an overly aggressive built-in blacklist, it should not be a major concern.

3: Yes. All of the above, please. Ideally we would be able to generate shared blacklists on a group/org level and also let individual users maintain personal lists. There should be the option to have global blacklists as well as ones on a per-share basis. By share, I am thinking of the odirve link rather than through the Spaces add-on. That might work as well, although I admit to not seeing how Spaces meshes with existing cloud account access permissions.

4: I can see the appeal of adding additional filtering options, but the low-hanging fruit is based on filename.

Thanks!

[greenish]

Hi,

I am really looking forward to this feature as it is currently a deal breaker for me to move everything to the cloud!

Here is my use case:
I’m a web developer and have a lot of installed node applications on my computer. Each node application can have a huge dependency tree sitting in the node_modules/ folder within each project. This folder is created when installing the project and can contain thousands of files which I don’t need synced. The amount of small files creates an error/hitting some limit when syncing with odrive saying “Google is busy”.

Being able to exclude/ignore node_modules/ folder globally would hopefully solve this issue.

For my use case something like .gitignore for my odrive account would be enough.

Thanks

[Michael_Wolfe]

Hi Jeff,

I’d like to see something that works like gitignore, where I can specify file or folder paths, with optional wildcards.

Thanks.

[mike]

I too would like the blacklist on file extension. I like some of the dreaming of much more than that but would be pleased if I could exclude movie extensions from my Amazon sync as they churn and fail all day over and over.

[skifreak85353]

I would love to have this feature. Like others, I am trying to backup my Lightroom catalog and images to my Amazon Cloud account. Movie files just churn and churn and churn. Would love for the ability to exclude file types. Please!!

[robertrudolph1978]

Please let us exclude files. Same scenario: Want to exclude video files from uploading to Amazon Cloud drive. Would be a feature that would lead to a positive decision to buy the paid version.

[peter]

We will make this a higher priority and deliver the capability before summer. Thanks everyone for the feedback!

[geofftech]

This would be a great feature and one I am explicitly looking for in a sync tool. You have so many great features that put you ahead of the pack. Especially the unsync option. The only thing stopping me is the mass of dev and temp file I don’t want synced.

[nam]

That’s great, I m adding my vote here - willing to exclude video and temp files to sync on Amazon
My workflow for now forces me to separate file types using SyncToy before odrive sync - not really practical and time consuming

[roland1]

I say yes to all your choices as I either have a need for them now, or could have a need for them in the future. You have a multi-national user base of folks who get new ideas of how to use your service more efficiently and effectively everyday. It seems like little thought needs to go into allowing a person more flexibility in how they manage their storage. Implementing all the features would be great for me, but if I have to place a priority, it would be filters, regex, specific file choosing/naming, blacklist/whitelist, listed in order of importance and requested on an individual basis for me.

Thank you for a great product and for listening to your user base!

[nam]

Hi there, any news for this feature ?
Would be already great to have a file type exclusion field that we could set (i.e. “prel; mov; mp4; f4v” to exclude some video file types)

[geofftech]

I would just like to add another ping to this thread. Odrive is excellent, but this single feature is stopping me from rolling it out to customers. They have extensive temporary files created (over 1GB) and these are being uploaded and causing issues.

And I would like it for myself as well!!!

Thanks

[gltjunk]

I have loved using the odrive service for the last week, my amazon cloud drive is in perfect sync with my local directory. Although I had to uninstall odrive today due to the lack of this feature. I plan to use the service again in the future when the feature is available. Because of video files in my local directory that are not in my amazon drive, it continuously attempts to upload the videos into amazon. This has created over 100 GB of traffic per day, causing me to go over the limit put on me by my ISP. We need to have the ability to blacklist file extensions, or the service is useless. Please move this feature up the priority list. Thanks for your consideration.

[JeffL]

Hi @gitjunk,

We’ll definitely keep you updated in this thread once we are able to implement the exclusion list. There’s currently a logjam of stuff (a fair deal of which is prerequisite to offering the feature, too) that we’re trying to get out, so it’s taking us a little bit longer than expected.

Please note that if you actually do want to have the videos go up but are just having problems completing the uploads due to network faults, etc. then you can try our Infinte File Size (IFS) feature. Our usage guide has more information about it. The gist is that you can set a threshold so that files larger than a certain size get split up behind the scenes. So we’ll upload them more reliably as smaller partial files. When using the odrive desktop client, the files get reassembled properly seamlessly. You wouldn’t be able to consume the files outside of odrive since they are split up, but if you are mainly interested in archiving them, that may be good enough. There’s nothing proprietary about the file splitting so you can easily reconstruct them outside of odrive if you had to. Scripts to do that and full transparency of how it works can be found in this medium article.

Waiting for the exclusion list is the best thing to do if you simply don’t care about keeping those particular video files in the cloud, of course. =)

Best,
-Jeff

[jordanbtucker]

We developers definitely need a custom exclusion list as well as a whitelist.

My current need is to ignore ‘node_modules’ folders for certain sources. This is the most requested feature for Dropbox right now. It would be awesome if odrive could be the savior for us poor Dropbox users.

In addition I don’t want to ignore ‘.git’ folders and ‘.gitignore’ files.

I can see the reasoning behind ignoring .git folders, however Git is not synonymous with cloud storage. Git is a distributed version control system that works very well as a local-only version control system. Sometimes I just want to keep a local history of changes made to a folder without storing it on GitHub or collaborating with another Git user. But I still want to sync that history between my laptop and desktop. I can’t currently do that with odrive.

[nam]

Hi @JeffL,

I think @gltjunk is facing the same issue that I do, and it is not about file size.
Amazon cloud offer in Europe (at least in France) is unlimited storage for pictures only - with a limited storage for any other file types.
A very large PSD file (photshop) will backup with no issue (no storage limit), but any small MP4 file will continuously attempt to sync and fail if the storage limit has already been reached.
So the issue is definitely about filtering media types here and not file size

Should the team have any idea of delay for implementing this feature, I would adapt my backup strategy accordingly

Thanks

[gullitmiranda]

I talked about this in the another topic, but as soon as the @greenish. I believe that the use of a file by following the standard .gitignore or .rsync_ignore would be of great value, for being a pretty standard used.

topic: ODrive Git Support
reference: http://sync-help.bittorrent.com/customer/portal/articles/1673122-ignoring-files-in-sync-ignorelist-

[Smudge]

Please add a +1 for a need for custom exclusion lists.

Until they are available, please add .*.swp to the internal ignore list. These are created by the vi/vim editor as temporary swap buffer files. They are then uploaded by odrive but when I’m done editing the file and the swp file is deleted, odrive then puts it into the trash and pops up a notification message. This is very annoying.

I would like to have a .odriveignore file similar to .gitignore, which you can read about the format here.

(1 and 2) The git ignore format is quite flexible. It can exclude files, folders, but then also the ability to negate a previous exclusion. It allows the use of multiple ignore files in different directories for granularity. For example, in the user home dir could be a .odriveignore file to exclude mp4 files with *.mp4. Further down, in a project subdirectory where you need to include .mp4 files, you could create another .odriveignore file in the project directory containing a '!*.mp4' rule.

(3) Using the above method of hierarchy-based ignore files, a company could create a .odriveignore file above the user’s home dir (in /Users/ or even at /) with some default ignore rules.

The user could negate those rules by creating their own .odriveignore file. However, for corporate security reasons to never upload certain files, you could create a custom feature where they could add a line where any remaining rules can’t be overridden.

*.mp4
*.mp3
#permanent
*.exe

(4) The ability to exclude based on metadata would be great! In MacOS, one of the big features is custom tags and automatically created metadata via Spotlight. It would be nice if odrive could exclude/include based on custom Spotlight search/rules or even the output of a script/command. For example, sudo mdfind "com_apple_backup_excludeItem = 'com.apple.backupd'" finds all files that are tagged as excluded from TimeMachine backups (and other 3rd party backup utilities honor these tags too)

Thanks for listening. I really hope this feature happens soon as it is way overdue.

[hpg]

Please add anoter +1 for custom exclusions.

Like “gullitmiranda” and “Smudge” suggested before a system similar to the one GIT is using would be great.
No Need for a complicated GUI to configure this. Concentrate on functionality.