Problem - Random, Selective Sync of the Folder Contents

Hello, guys! I’d like to as for your help.

Recently I started having a peculiar sync issue. Only some random selection of the new folder uploads gets synced to my computer(s). Here are the details:

  • I have the following folder D:\odrive\Google Drive\Photos\Android
  • I sync all the new photos from my phone to the Android folder on Google Drive via IFTTT
  • All the photos from the smartphone were successfully sent by IFTTT to the destination, Android folder (checked IFTTT logs)
  • all the photos from the smartphone are mirrored on GD and visible in the web interface (both GD and Odrive)
  • Only only a fraction of all the photos reaches the Odrive folder on my computers, not even the placeholders.

Problem occurs both on my desktop and laptop
Both computers run on Windows 10

I tried the following:

  • ending the Odrive process and restarting
  • rebooting computer
  • unsyncing the Android folder and syncing again

Nothing worked so far.

I made a screenshot showing side by side, from left: Google Drive web interface, Odrive folder on PC and Odrive web interface. I marked with red outline a range of photos illustrating the problem. https://www.odrive.com/s/2298fd18-ccdc-4797-b7d3-fcf6a84b1e52-5882cd7b

Hope you’ll be able to help.

Kind regards,
Igor

Hi Igor,
Usually the cause of this is illegal characters in the filenames. Since you can see the items in the odrive web client, that means that odrive is capable of seeing the items (logic is the same between clients). I cannot see the full file names of the missing items. Is it possible they have “:” characters in the times?

Can you send a diagnostic from the odrive tray menu so I can take a deeper look?

Hi Tony, thank you for getting back to me.

I just did the “Send Diagnostics” from the odrive tray menu. Is this what you needed?

The file names are generated by IFTTT applet, please see below

January 19, 2017 at 0132AM.jpeg
(one of the files that didn’t make it through to the desktop folder)

(screenshot from my odrive web interface)

(screen from my Google Drive)

All best,
Igor

Hi @idesignwebsites,
I took a look at the diagnostic. I see many instances of Google Drive rate limiting the API calls. Google Drive is probably the most strict when it comes to enforcing API limits.

I noticed that you have over 13,000 folders synced in odrive and the vast majority seem to be inside “1 IDW Google Drive/3-Toolkit/”. When odrive is refreshing the contents of that account, it is hitting many rate limit exceptions because of the number of items in that location.

Here is what I would recommend trying:

  1. Unsync the D:/odrive/1 IDW Google Drive/3-Toolkit/ folder, or at least the “D:/odrive/1 IDW Google Drive/3-Toolkit/FONTS” folder
  2. Right-click->“∞ refresh” on the “D:/odrive/1 IDW Google Drive/Photos/Android” folder

It may take some time after the unsync in step 1 before the API limits are relaxed, so it would be good to wait several minutes.

If you are still seeing a discrepancy after that, send another diagnostic. Also let me know if you see any errors when performing the right-click->“∞ refresh” on the “D:/odrive/1 IDW Google Drive/Photos/Android” folder

Hi Tony,

It’s the second time I needed help with odrive and it’s the second time I’m extremely impressed. Thank you! The quality of the help you provide here is really exceptional.

Now that you explained what is the issue, I think I even have an Idea what exactly happened.

Recently I installed Extensis Suitcase Fusion - a font manager. I placed its library in the D:/odrive/1 IDW Google Drive/3-Toolkit/FONTS

Apparently, Suitcase Fusion has its own cloud sync. But I thought that it wouldn’t hurt to have an extra copy of the library, just in case. Well… looks like this time it did hurt:)

I imagine that Suitcase Fusion must have been making all these calls to its own cloud sync and these must have been done on some insane rates.

Now it seems simple but I really wouldn’t be able to figure it out myself.

Thanks for your help!

Hi @idesignwebsites,
Thanks for the follow-up. Let me know if removing that folder (with unsync, or just transplanting it elsewhere), clears up the issue you are seeing. It sounds like the likely culprit in a few ways ;).

Hello Tony,

I followed all the steps that you suggested, plus I tried some additional things like renaming the folder, moving the folder, removing files in the folder, but my Photos/Android still is not in sync. Not only older photos are missing (still) but also new uploads are not appearing there. I’ m just pinging diagnostic again, hoping that you’ll be able to help.

All best,
Igor

Hi @idesignwebsites,
I took a look. I see some activity in the Photos/Android folder, so it seems to be doing some syncing in there. It is hard to gauge it, though, because there is still a lot of noise from the odrive/1 IDW Google Drive/3-Toolkit folder.

If possible, can you do the following?

  1. Unsync odrive/1 IDW Google Drive/3-Toolkit to remove the noise
  2. Restart odrive (this will ensure we have a clean diagnostic)
  3. Wait about 10 minutes
  4. Add a file, locally, to “D:/odrive/1 IDW Google Drive/Photos/Android” and make sure it syncs
  5. Add a file remotely to 1 “IDW Google Drive/Photos/Android” from the Google web client
  6. right-click->refresh “D:/odrive/1 IDW Google Drive/Photos/Android”
  7. Send another diagnostic

After step #7, let me know if the files added in #4 and #5 synced as expected. Please also let me know the names of those files.

Thanks!

Hi Tony,

Once again, thank you for your time. I completed steps 1-7 as listed above.

Test files synced both ways instantly. Names of these files are: “test-web” and “test-desktop”

I was playing a bit with this stuff yesterday by the way. I tried renaming the folder, moving it, etc. What worked was renaming the files. Like this one: January 19, 2017, at 0133AMxx - after I renamed it in the GD web client it appeared on PC. New files are still syncing to PC randomly.

Thanks a tonne!

Hi @idesignwebsites,
The behavior you are describing sounds like something in the name is preventing sync. The names you are giving as examples seem fine. It is possible they have any whitespace at the beginning or end of the file name?

Are you consistently seeing that renaming a file allows it to show up? Can you give me an example of the rename (from and to)?

Is this the IFTTT applet you are using? https://ifttt.com/applets/164503p-back-up-your-new-android-photos-to-google-drive

Hey @Tony

I think I mentioned already the file naming stuff in one of my first messages. So - all the files from IFTTT have the same naming structure.

Example of file that synced successfully: January 19, 2017 at 0133AM
Example of file that didn’t: January 19, 2017 at 0133AM.jpeg

file extensions are automatically being removed by Windows I guess?

the only change I made to the file name was adding “xx” to the end of the name - like this: January 19, 2017 at 0133AMxx

The applet you linked is exactly the one I’m using.

@Tony I just found the essence of the problem! :slight_smile: …you wouldn’t believe …I can’t believe :slight_smile: It was hidden in plain sight. Do you want to make a guess? :wink:

Do tell!
Unless it was some whitespace, or a hidden character like that, I don’t know…

Look at the file names I listed above:) Aren’t they… the same?:slight_smile:

I’ve selected “Date Taken” ingredient in the IFTTT applet builder to be used as the file name. The problem is that IFTTT engineers didn’t think through the date format too well. And that actual file name is not created based on date taken, but based on upload (I guess) date. (I noticed it because there were two photos I know they couldn’t be taken within the same minute)

So - the date format on IFTTT lacks precision. Within the same minute, there could be several pictures taken - and they all will have the same file name. GD and Odrive don’t care - I guess there must be some checksum that is taken into account? But Windows - Windows can’t handle two files with the same name in one folder. And that’s the reason for the patchy sync.

So obvious, right?:slight_smile: But I noticed it only because you asked me to provide file name samples and linked to the applet. I went to IFTTT to double check whether I didn’t have any extra spaces (as you suggested it could be an issue) and then it hit me :slight_smile:

1 Like

Ahh… yes, that certainly explains it.

I didn’t realize that there were multiple files with the same name. Indeed, that would cause a problem when pulling them down to the local system. It’s too bad that the timestamp doesn’t provide granularity to the second. It also doesn’t appear that there are any other ingredients that could be used, aside from possibly the URLs, which would make the file name very ugly.

What happens if you do not enter an ingredient for the file name?

Yes - certainly the file naming options are not the best at the moment. There are easy fixes - like you mentioned - with seconds, but I wonder if they would care to implement it. And as I said - it looks like they are faking “Taken Date” which in reality looks more like a “Pull Date” - so seconds might not fix this issue. So maybe simply n+1 option is lacking.

These are the alternatives by the name:

public link: httpslocker.ifttt.comf30d58817-17f1-4364-a760-c2dd80df8987.jpg
when you leave the file name field empty: 22ec365f-b490-49b8-a0c1-85aed0c7a691

Yes, not pretty:)

Luckily batch file renaming is not a big issue and the date is saved in metadata anyway

@Tony once again, thanks for all your help. I really appreciate. The level of service I’m getting here is top notch (consistently) and odrive itself, well - just an epic bit of software. I think it may be time to get the paid premium. Despite the fact, that feature-wise there is nothing really, that would tempt me …you made the free plan just too good :slight_smile:

1 Like

Thanks for helping me work thought this issue and for the swift follow-up. It is much appreciated. We are working on some alternate purchase plans that you may be interested in. I expect them to be available within the next 2 months or so (although we are not very good at estimates… :wink: )

Hi,
I’m having the same issue with filenames that contain colons (in Google Drive in this case). They show up in the web client, but not in windows client. Most files are created via the web interface, but that is slow from South Africa, so I’m trying odrive - which is much quicker - but is losing these files:

Getting everyone else to change to suit me is unlikely - is there anything in the odrive works to handle this?

Thanks
Grant H

Hi @g.hillebrand,
Unfortunately there isn’t a better way to handle this, at the moment.

We will list these items in the “not allowed” section of the menu so that they can at least be identified, but we don’t currently have a way of converting these names into something Windows will accept.

It is something we would like to support in the future, however. We just need a reliable, bulletproof method of doing this. Our previous attempt ended up creating more problems than it solved…

Hi,
Thanks for the quick response. I figured out the “not allowed” list after I had posted this.
I don’t know what you guys tried, but hashing the not-alloweds and dropping the illegal characters (which is what Google seem to do), or escaping using ~'s (ala the DOS 8.3 mapping) come to mind. I’m sure you have smart people on it. It was just awkward that the first file I looked for had a colon!

Grant H

1 Like