I’ve come across this kind of issue in quite a few forms, I wonder if someone can help me understand the logic / algorithm so I know what to expect in the future.
I’ll try do describe the most recent use case in which I encountered this problem.
I’m currently migrating from the unusable Amazon Drive to a seemingly better OneDrive cloud storage. Certain files are already present in both cloud storages, but not necessarily on each client PC. So on one of them I simply want to copy the actual video files from the Amazon Directory to the OneDrive one manually instead of downloading them by syncing (we’re talking about hundreds of gigabytes here). So I copy the files. And then something very strange happens. Some of them are immediately identified as similar to the cloud copy, the corresponding .cloud file disappears and the one I just copied gets the blue check mark badge. Wonderful, just as expected.
But then with some of the copied files (roughly one third of them) it doesn’t happen. Instead, they get the red “syncing” badge and under the “syncing changes” I start seeing some slow progress, per cent after per cent.
Obviously, there is no apparent reason for transferring any data here, since the local and the cloud copies are exactly the same. But worse still, due to the lack of differentiation between download and upload icons (see issue Different Icons for Upload and Download in Odrive context menu) I don’t even know what Odrive is doing: uploading the newly copied files to the cloud or downloading the cloud copies to my local PC.
Any help understanding what and why is happening and how this could be dealt with will be greatly appreciated.
Hi @alexey,
The determination is pretty straightforward. odrive will compare the attributes of the file as it is seen locally with the attributes of the file, as they are reported by the remote storage. If they differ then the local file is uploaded, with the assumption that it has changed somehow. Depending on the storage source and the atributes of the local files, and where they came from, there can be some variation in values compared.
We have accounted for your scenario, however, and there there is a subtle variation in the comparison logic, depending on the type of “merge”. The comparison is more strict if the remote files have been scanned and indexed prior to the local files being scanned and indexed. In this case it is more likely that a re-upload will occur. The comparison is less strict if the remote and local indexing happens at the same time. I know this description may not be clear, so here is an example:
Let’s say you have the “odrive\OneDrive\A” folder expanded via the odrive Desktop application and “odrive\OneDrive\A\file.zip.cloud” is listed. If you copy a local “file.zip” into the “odrive\OneDrive\A” folder, (alongside file.zip.cloud) there will be a “strict” comparison between the remote OneDrive\A\file.zip and the local file.zip. In this case chances are higher that a re-upload occurs.
Now, let’s say you have “odrive\OneDrive\A.cloudf” listed locally, via the odrive Desktop Application. In the cloud there is a “OneDrive\A\file.zip” file, but odrive has not yet seen that file because the local A.cloudf placeholder has not been expanded and indexed. If you take a local “A” folder (that has a file.zip inside) and copy that into the local “odrive\OneDrive” folder, it will replace A.cloudf. The contents of the remote “OneDrive\A” folder and contents of the local "odrive\OneDrive\A\ folder (that you just copied in) are scanned and indexed at the same time. In this case, the comparison is relaxed a bit because there are certain assumptions that can be made, and it is very likely that this is a “local data migration” scenario, like the one you are going through right now.
To summarize, if you replace placeholder folders (.cloudf) with local folders, where the local folder contains the same data as the remote folder, you are much more likely to get a “clean” result, without re-upload.
In my case I didn’t copy any directories, just copied some files into a folder that I had expanded merely a few seconds earlier. Maybe the local directory was still being indexed which caused some race conditions leading to some files being treated differently than the others.
Anyway, I wonder what file attributes and why could have been deemed different in my scenario. The files’ content, sizes and dates are the same. I think it could be great if there were a way for the user to force odrive to apply this more lenient comparison logic when necessary. Otherwise, I’m wasting many hours and even days of time and gigabytes of traffic on an operation that I know for a fact is useless. Do you think it makes sense for me to open a feature request for this?
Hi @alexey,
I saw your post in feature requests around this scenario. Can you send a diagnostic from the odrive menu and give me a couple of (very recent) files that should not re-upload, but are? I would like to see if I can determine why.
I just sent the diagnostics.
Each of the 8 files currently being uploaded is the same on the cloud storage (unless some of them have been ruined by the recent unsuccessfull upload attempts, of which there have been a few since the folders were copied, but I think it’s unlikely). Below is the screen snapshot taken a couple of minutes ago, right after the diagnostics were sent:
Hi @alexey,
I took a look, but it doesn’t look like we captured when the files entered the odrive folder and the comparison was made. I think it must’ve rolled off. The diagnostics don’t capture an extensive window. The next time you see this happen on a file you just dropped in, can you send and provide that file name?
The upload was taking ages anyway, so I tried copying one of the folders again, and the problem was reproduced.
What I did was the following:
I stopped the automatic syncing of the folder
Created a local backup of the folder
Unsynced the whole folder
Turned the automatic syncing back on
Moved the folder from its backup location to the odrive folder
As you can see, odrive decided to start uploading at least one of the files from the folder back to the cloud. Ironically, a longer file that odrive had tried to upload on my previous attempt, was recognized as being identical this time and marked with a blue check mark.
I’m considering more experimenting, for instance, switching the order of 4) and 5), but in any case I will appreciate it if you have a look at the diagnostics I’ve just sent and maybe figure out what’s wrong.
Hi @alexey,
This is for the 1997_2000_1.avi in the Dropbox/VIDEO/SD/1997_2000_RENTED_CAMERAS/RAW/ folder? Do you know when (what time) you performed those 5 actions?
For the drive you have the odrive folder on (D: drive), is that a local drive or a remote/external drive? Is that an NTFS-formatted volume?
Yes, this is the folder. The actions were performed a few minutes before I sent the diagnostics and wrote the message. The drive is a local NTFS-formatted hard drive.
In the meantime, though, it seems like I’ve found a reliable workaround. All it takes for the operation of this type to succeed is to replace action 4 with exiting the odrive client altogether and then restarting it after action 5 is complete (after the files have been copied).
Anyway, thanks for helping me with this investigation!
