Download of encryption volume triggers full second resync of same volume

Troubleshooting
, ,
1

When I saw that files within my Encryptor Volume on odrive were not synced and were all just placeholder .cloud files, I performed a “sync” of the entire volume and specified that it sync the volume in its entirety. This is because the whole .cloud placeholder system has some serious flaws:

While the .cloud model is a good idea in theory, the fact that it doesn’t resolve at key situations, like when you are in a system dialogue box trying to open a file that is still in .cloud form, odrive doesn’t go get the actual file that the .ocloud placeholder is keeping a seat warm for. Attempting to open a .cloud file within an application using the system’s dialogue box for opening files simply fails and does nothing. You have to go into finder and explicitly sync the files which is way more trouble than it’s worth. If you are going to have a system that offloads your files to the cloud and leaves placeholders behind, the placeholder management system should actually work just above the file system level so that if you make any call to access a file that odrive has ofloaded and put a .cloud placeholder for, odrive should kick in and download the file from the cloud source automatically. Instead, it is just a finder plugin so you can only resolve the placeholders in Finder. It doesn’t work in any other context- not even the system dialogue boxes which would seem the most likely place that you would encounter placeholders- when you need to access the files! It’s risky enough to offload your files just for the fact that you might need them at some time when you don’t have internet connectivity to the cloud, but to add these problems too makes it an impractical method.

Anyway, the point of this post is not just to rip on ocloud’s placeholder model on the whole, but that there seems to be some additional problems using it with it Encryptor Volumes, and to to point out that the simple act of running a sync on your Encryptor Volume to resolve all of the .cloud files so that you have the actual unencypted files locally on your system, it would appear that not only does odrive have to download all of the files one time (which is to be expected) but after I complete that process, odrive is now uploading the entire Encryptor Volume a second time for some reason. To boot, it would seem that not only are the unencrypted versions of the files kept locally, but it seems that I have to keep local copies of the encrypted files as well, thereby doubling the amount of disk space on your local drive for any file stored an Encryptor Volume. (Maybe this is why odrive has the .cloud placeholder system at all- to make up for this doubling of EV-stored files!)

I just sent another diagnostic so you can see this in action as it is presently re-syncing (uploading) my entire Encryptor Volume “Encryptor-AmazonDrive01” a second time after it completed the download of all of my unencrypted files when I told it to sync the .cloud files. I have no idea why it would be doing this. I made no changes to these files whatsoever other than replacing the .cloud placeholders with actual files. What’s more, I can see that odrive also has downloaded the encrypted version of each file along with the unencrypted files that I just synced within the Encryptor Volume. While I can’t be sure that it is in fact two distinct files, it sure does look that way, and considering that the encryption/decryption appears to happen between odrive and the cloud store, and not locally, having unencrypted and encrypted files cohabitate the same space on disk (meaning the decryption of the EV is happening live on the client machine) seems very unlikely simply because if that were the case, odrive would natively support mounting Encryptor Volumes, which it doesn’t! The way EVs work is therefore wasteful on two fronts (1) It is double syncing my encryptor volume- once it downloads files to replace the .cloud placeholders, it then uploads those same files BACK up to odrive for reasons that I can’t even imagine and (2) it keeps two copies of any file in an Encryptor Volume within the odrive structure after you sync the .cloud files- one, it seems that the raw encrypted file is downloaded locally to the local location corresponding to the cloud datastore that you used to make the Encryptor Volume, and second- there is the unencrypted volume as seen inside the Encryptor Volume, which in my case is called “Encryptor-AmazonDrive01”. Aside from this needless upload of the Encryptor Volume, once again burning up my metered bandwidth for the month, it is also keeping the encrypted version of the local files on my machine which seems completely pointless since the unencrypted versions are right there in plain view anyway. Why bother filling the disk with encrypted versions as well? The whole thing makes no sense.

Please let me know if this corresponds with the way odrive is supposed to work or not, and what I can do to limit this waste of resources, other than using another tool altogether like Cryptomator?

Thanks,
Matt

2

Hi @DarfNader,
What you are describing is not expected. Let’s see if we can make sense of what is happening.

When you download an encrypted file, it actually decrypts it, in-line, as it is downloading, so the files are never written to the local disk in an encrypted format. Also, Encryptor volumes can only show decrypted content, as it is all run through the decryption scheme and checked for format validity.

Since you are seeing what look to be encrypted files being brought down locally, it seems like you may have somehow uploaded already-encrypted content via an Encryptor volume at some point in time. What I mean by this is that encrypted files were added into an Encryptor folder, encrypted again, and then uploaded. I’ve seen someone do this before when trying to reconcile the Encryptor volume with the encrypted data that is visibile via the “standard” paths. They ended up copying the data from the “standard” path back into their Encryptor folder, creating double-encrypted duplicates of the original files.

I am trying to think of how else this might have happened…

When a file is downloaded, a temporary file is created which contains the downloaded data being streamed to the local system. Once the download is complete, this temporary file is renamed to the proper name. They start with a .~ (period and tilde). During download, the placeholder file will be listed with a pink “syncing” badge and the temporary file (which is normally hidden), will be created for the download.

Some questions:

  • Is it possible that you are seeing the temporary download files and were mistaking them for encrypted files?
  • Are you using any symlinks or aliases in the odrive folder that you have setup to point to other locations?
  • As for the uploads, in the diagnostic provided I don’t see any files in the Encryptor volume being uploaded. How are you determining that files are re-uploading?
  • Can you provide a screenshot of what you are looking at in Finder in the Encryptor folder? This should give me a better idea of what you are looking at.
3

The simple answer is that the EV resides within my Amazon Cloud Drive. It is a folder at the top level. Therefore, what appears to have happened is that when i synced all of the files that were within my EV, it would appear to have forced a sync of all of the encrypted versions of the same files, but in the context of the Amazon Cloud Drive. That’s because within the odrive, there is the volume for the EV, but there is also a folder which represents the the my entire Amazon Cloud Drive. The behavior is not all that surprising, really, except that I don’t believe that I manually synced the encrypted versions of the EV files at any point so I don’t know why it would try to do that now.

4

Hi @DarfNader,
Ahh, I see now. You are correct. Since the Encryptor folder is just a “virtualized” view of the encrypted data in the target storage, that data is still accessible the “standard” way, via normal paths.

I know some folks have set a sync folder rule ( https://docs.odrive.com/docs/sync-source-changes#section-configure-folder-sync-rules) on folders in their “standard” paths that specify to expand/sync any new remote content that shows up. New encryptor uploads would then end up triggering folder expands or full file downloads, depending on how the rules were set. Your scenario doesn’t really fit here, though, at least on the surface, since you were performing a download. Is it possible those files have been there for a while and were unnoticed until you started digging into things?

5

Hi @Tony,

I’m not fully sure what you are asking- do you mean are the encrypted version of the EV and its contents (encrypted files with encrypted names) there from a previous sync that I performed of that directory explicitly? If so, then yes, that is certainly possible, but I don’t see why I would have done that. One thing this makes me consider is that there are folders in my Amazon Cloud Drive (the location of the EV in question) that I simply do not want odrive to even look at for the fact it would be wasting cycles on their contents. For instance, I use ACD for my backups using a tool called ARC. That directory has files both voluminous and multitudinous. In fact, I really don’t want odrive in there at all because it is just going to be bogged down tracking enormous amount of files and god forbid I accidentally try to sync it! It has to be about 3 TB of data at this point. (I am not sure how large it might get- I am kind of running an experiment to see if Amazon really means it when they say “Unlimited”. :wink:) Anyway, the point is I probably just want to have odrive sync a few folders in ACD or specifically exclude the syncing of specific folders. Is that possible?

But back to the issue at hand, I think the reason that this is a problem is that the EV is contained in another cloud drive that odrive syncs. However, I am still wondering why the act of syncing the unencrypted versions (which is basically just decrypting and downloading all of the file contents) caused the encrypted version of the files to resync. I didn’t make any changes to any of the files other than “touch” them by accessing them. If odrive is that “sensitive” that would be worrisome.

6

Hi @DarfNader,

You can usync any folders you want odrive to remove from odrive’s active monitoring. this will collapse them and remove the contents from odrive’s view.

It shouldn’t cause an re-uploading of Encryptor content or downloading of content in other areas. If it did it is unexpected. When I looked at the diagnostic, though, I only saw download activity in the Encryptor folder you were syncing. As we discussed above, maybe the data was already there and was just unnoticed until that time?

7

You can usync any folders you want odrive to remove from odrive’s active monitoring. this will collapse them and remove the contents from odrive’s view.

I have done that, but I wasn’t sure if that was the definitive means of taking those files out of odrive’s database so they don’t “clog up the works”. So once a top-level folder is collapsed so that only the .cloud version of that folder remains, odrive no longer has any knowledge of its contents until you open the folder forcing a sync? I will be interested to see if I see that behavior of the encrypted version of files coming down when I sync the decrypted version of it. I will test that and report back.

It shouldn’t cause an re-uploading of Encryptor content or downloading of content in other areas. If it did it is unexpected. When I looked at the diagnostic, though, I only saw download activity in the Encryptor folder you were syncing. As we discussed above, maybe the data was already there and was just unnoticed until that time?

Yes, as I said, it is possible that I manually synced the encrypted versions of the files at one point, though I don’t know why I would have. Still it doesn’t explain why the act of downloading the decrypted versions of the files would cause the encrypted versions of the files to sync as well. That is what I was seeing and one of the reasons for my support request in the first place. It may just be that I misread what odrive was reporting what it was doing. The thing is, from what I was seeing, all of the files had already been fully downloaded (it’s a very fast process because I have like a 200 Mbit download connection), but odrive was busily uploading the same files. (I only have a 30 Mbit upload.) At this point, whether it was uploading the encrypted or decrypted versions of the file is outside of what I remember exactly, but the fact that it was uploading anything didn’t make sense to me and the other big part of my inquiry. Actually, this was probably the more pressing question, as I am more concerned about odrive generating excess sync traffic. For starters, getting rid of the AFS share from odrive might be a culprit as I may have some of those directories synced. (It would be impossible to have it fully synced as I don’t have enough space on any of my other computers.)

I am not sure where we go from this point as for tracing whether EV’s create extra traffic, but I think need to do some testing.

Thanks again!

8

Correct. All previous sync tracking data of that area is removed from the persistence layer and odrive will not longer scan that area of the storage.

I have asked the QA team to take a look at a few scenarios around the encryption stuff, as well.