I first verified that the issue is still present on build 7145, and indeed it is. My test:
move a PDF to the slow folder
wait until odrive has synced
rename file
After 11 seconds, it was renamed back to the original filename whereas both Backup and Sync and the web version of Google Drive kept showing the renamed filename.
Then I installed build 7150 and retested it a few times. The filenames remained stable each time, that’s a win!
Then I proceeded to install build 7150 on another Mac so I can test this build on two computers. The filenames remained stable on the computer that performed the local renames, but the other computer reverted back to the old name 11 seconds later. So the issue is not fully resolved.
Also, I kept an eye on Backup and Sync during all tests, and noticed that it picks up all remote changes a lot sooner. So that might be something to look into as well.
Hey @richard ,
Sorry for the late reply. I haven’t forgotten about this. We may have another test to try to see if we can improve the behavior on the observing clients. There is a limit to what can be done because of the context (or, rather, missing context).
The root problem is that listing the folder in Google has the possibility of stale data, especially on these older structures. We can’t afford to do to much second-guessing because the service provider is the authority in terms of telling us what’s what.
For the observing system we should be able to gate consecutive renames on the same files, which should help to keep thing from flip-flopping. We won’t have a silver bullet, but it should be much better, overall with all of these changes.
Google’s Backup and Sync is my benchmark. It does not have any such issues in the same situation, so there must be a solution out there. I’m happy to do more testing if it helps.
I wonder, could the parent directory listing step right after processing a change be omitted altogether? It seems that all of the required information is already available from the update notification and subsequent download.
Just had another instance of reverting a rename on Google drive. Here are the events from the odrive log:
09 Sep 02:52:06PM INFO Successful Move (Local to Remote) for File: /Users/oscar/odrive/Google Drive/MergeBase/finance/receipts/Receipts-MB/bill_9912.pdf to /Users/oscar/odrive/Google Drive/MergeBase/finance/receipts/Receipts-MB/2021-09-07-Golbey.pdf - Size: NA - Date: NA
09 Sep 02:52:17PM INFO Successful Move (Remote to Local) for File: /Users/oscar/odrive/Google Drive/MergeBase/finance/receipts/Receipts-MB/2021-09-07-Golbey.pdf to /Users/oscar/odrive/Google Drive/MergeBase/finance/receipts/Receipts-MB/bill_9912.pdf - Size: NA - Date: NA
In the past we tried to rely more heavily on the change API, for the services that supported them, but it wasn’t reliable enough and calling it too often resulted in a higher degree of rate limiting. The only truly reliable way to see what is in a folder on the remote side is to look at it on the remote side. Unfortunately if there is significant delay before consistency you can see things flip-flopping around. Things will eventually become consistent and when that happens odrive will be able to reconcile any discrepancy through normal sync processing.
This delay on Google is surprisingly long. I am not sure what Google is doing to combat it. They must be ignoring subsequent folders listings, to some degree.
In any case, this new version attempts to combat delay on the observing clients, too, so hopefully it will provide a “good enough” solution for this issue.
Thanks again. I performed some tests using build 7152 on two Macs. It took:
on average 46 seconds (19, 35, 53 and 76) seconds for a new file to appear on the observing system
until the next remote scan (potentially hours) for the renames to appear on the observing system.
I’m sorry but that’s not usable. I think using the changes API is really needed to speed this up. Google Backup and Sync handles this within seconds and is probably using it.
Test case 1 as seen from Mac I (the acting client):
15 Sep 08:25:08AM INFO Successful Upload File for Item: /Users/richard/odrive/Google Drive/A/B/1.pdf - Size: 136801 Bytes - Date: 2021-09-15 08:24:47
15 Sep 08:25:22AM INFO Successful Move (Local to Remote) for Item: /Users/richard/odrive/Google Drive/A/B/1.pdf to /Users/richard/odrive/Google Drive/A/B/2.pdf - Size: NA - Date: NA
15 Sep 08:27:40AM INFO Successful Upload File for Item: /Users/richard/odrive/Google Drive/A/B/5.pdf - Size: 6039 Bytes - Date: 2021-09-15 08:26:08
15 Sep 08:27:57AM INFO Successful Upload File for Item: /Users/richard/odrive/Google Drive/C/D/E/3.pdf - Size: 35402 Bytes - Date: 2021-09-15 08:27:38
15 Sep 08:28:11AM INFO Successful Move (Local to Remote) for Item: /Users/richard/odrive/Google Drive/C/D/E/3.pdf to /Users/richard/odrive/Google Drive/C/D/E/4.pdf - Size: NA - Date: NA
Test case 1 as seen from Mac II (the observing client):
15 Sep 08:25:27AM INFO Successful Download File for Item: /Users/richard/odrive/Google Drive/A/B/1.pdf - Size: 136801 Bytes - Date: 2021-09-15 08:24:47
15 Sep 08:28:32AM INFO Successful Download File for Item: /Users/richard/odrive/Google Drive/C/D/E/3.pdf - Size: 35402 Bytes - Date: 2021-09-15 08:27:38
15 Sep 08:28:33AM INFO Successful Download File for Item: /Users/richard/odrive/Google Drive/A/B/5.pdf - Size: 6039 Bytes - Date: 2021-09-15 08:26:08
15 Sep 09:52:23AM INFO Start scan mount (periodic) /Users/richard/odrive
15 Sep 09:53:53AM INFO Scanned (periodic) 1570 local folders
15 Sep 09:53:53AM INFO End scan mount (periodic) /Users/richard/odrive
15 Sep 09:53:53AM INFO Local scan complete. Will run again at the scheduled interval.
15 Sep 01:38:41PM INFO Start scan mount (periodic) /Users/richard/odrive
15 Sep 01:40:05PM INFO Scanned (periodic) 1570 local folders
15 Sep 01:40:05PM INFO End scan mount (periodic) /Users/richard/odrive
15 Sep 01:40:05PM INFO Local scan complete. Will run again at the scheduled interval.
15 Sep 04:25:47PM INFO Start scan mount (periodic) /Users/richard/odrive
15 Sep 04:27:11PM INFO Scanned (periodic) 1570 local folders
15 Sep 04:27:11PM INFO End scan mount (periodic) /Users/richard/odrive
15 Sep 04:27:11PM INFO Local scan complete. Will run again at the scheduled interval.
15 Sep 07:25:52PM INFO Start scan mount (periodic) /Users/richard/odrive
15 Sep 07:27:22PM INFO Scanned (periodic) 1570 local folders
15 Sep 07:27:22PM INFO End scan mount (periodic) /Users/richard/odrive
15 Sep 07:27:22PM INFO Local scan complete. Will run again at the scheduled interval.
15 Sep 09:30:59PM INFO Start scan mount (periodic) /Users/richard/odrive
15 Sep 09:32:28PM INFO Scanned (periodic) 1570 local folders
15 Sep 09:32:28PM INFO End scan mount (periodic) /Users/richard/odrive
15 Sep 09:32:28PM INFO Local scan complete. Will run again at the scheduled interval.
15 Sep 09:54:44PM INFO Start remote scan /Users/richard/odrive
15 Sep 09:56:01PM INFO Successful Move (Remote to Local) for Item: /Users/richard/odrive/Google Drive/A/B/1.pdf to /Users/richard/odrive/Google Drive/A/B/2.pdf - Size: NA - Date: NA
15 Sep 10:04:58PM INFO Successful Move (Remote to Local) for Item: /Users/richard/odrive/Google Drive/C/D/E/3.pdf to /Users/richard/odrive/Google Drive/C/D/E/4.pdf - Size: NA - Date: NA
15 Sep 10:07:21PM INFO End remote scan /Users/richard/odrive
Test case 2 as seen from Mac II (the acting client):
16 Sep 04:24:12PM INFO Successful Upload File for Item: /Users/richard/odrive/Google Drive/A/B/6.pdf - Size: 60598 Bytes - Date: 2021-09-16 16:13:45
16 Sep 04:24:31PM INFO Successful Move (Local to Remote) for Item: /Users/richard/odrive/Google Drive/A/B/6.pdf to /Users/richard/odrive/Google Drive/A/B/7.pdf - Size: NA - Date: NA
Test case 2 as seen from Mac I (the observing client, current time is 06:33 PM):
16 Sep 04:25:47PM INFO Successful Download File for Item: /Users/richard/odrive/Google Drive/A/B/6.pdf - Size: 60598 Bytes - Date: 2021-09-16 16:13:45
Were you able to confirm that the main issue reported in this thread (names reverting back after a rename) has been addressed with the new build?
For the second issue (speed for odrive pick-up of a rename): We are actually using the change API, which is why the downloads are happening fairly quickly. The issue here is that we get the change event for the rename, see that there has been a reported change in the specified folder, go take a look in that folder, and see that the name is actually unchanged (because Google is delayed and is still reporting the old name back to odrive).
I don’t think we will be able to optimize the change detection further to account for Google’s lag in consistency, at least not in the immediate future.
Keep in mind that change detection is considered an “optimization” for odrive, since real-time refreshes of folders happen when a user is browsing around. If a user was to browse into the folder after Google has finally reflected the change on their side, then odrive will pick up that rename immediately and does not need to depend on the periodic remote scan. To a typical user this is a pretty seamless reflection process because the item ends up being renamed locally as soon as they enter the parent folder on their local system.
Hi @tony,
Sorry for the late reply. Version 7152 was very bad (as you can see the log I posted above) in that it took almost forever for the new name to appear locally unless I performed manual refreshes.
Now I am using version 7167 and it’s still not as fast as Backup and Sync (seconds) but it is acceptable (<2 minutes) and the file names remain stable so far.
Regarding browsing around to trigger a refresh, that is not at all how we use it. We use Spotlight to find things (typing something we remember about the file contents or file name) and mostly don’t care where items are located. So therefore it is important that things sync quickly and reliably, especially when working with other people.
Hi @richard,
Thanks for the feedback! I’m glad to hear things are working better with the new version.
The longer reflections times you were seeing were not because odrive was failing to detect the change, but rather due to the lag in Google’s reflection of the change in its listing. Basically odrive saw a reported change in a location, went to look at that location, and acted on the information it was given for the current listing. Unfortunately the information Google was providing was stale.
We tried to account for this a bit more in our latest version. That being said, it will not be bulletproof. There may still be occasions where you will need to resort to a manual refresh in a directory, if you do not see an expected update.