Basically, at times, Google Drive API will seemingly not deliver a proper listing of files to odrive – therefore the odrive interface does not display all files when this happens. It’s for example, missing one to three files typically in one of my directories of interest containing about 1200 files.
This problem also happens with the command line app rclone as well as the file sync app Syncovery. I have contacted GSuite tech support front line, and am trying to escalate up the ladder, but I strongly suggest odrive to look at this as well. It potentially affects all of us.
The problem seems to be related to the number of files fetched through the API to get a directory listing, or listing of md5sums. I believe the default fetch of files is 1000. If you change that value to fetch an “ls” or group of md5sums in smaller groups (rclone developer has helped track this down), the files that “go missing” are different. Again, please see the thread below.
The implications are not very good because this messes up the filesystem integrity, causing things “go missing” and potentially “re-appear” later, while yet another different file goes missing.
Please see the following thread on the rclone site:
Hi @roy,
This is interesting. I have suspected that this type of behavior happens, but I have never been able to reproduce an instance, myself. I even went so far as to create an internal dev ticket, proposing a method to try to combat theoretical intermittent, incomplete listings.
What has Google said about this? Do they acknowledge the issue or are you hitting a wall with them?
When you say internal dev ticket – you mean within the odrive dev team? As opposed to Google, correct? Just making sure I understand.
I have finally sent Google an email thread yesterday that sorta left off where the info. on the rclone posting stopped. I had taken some of this to an offline exchange. I suspect frontline support will reply soon (they were quite persistent actually), but I don’t really think this is probably the best way to get their attention.
I assume the GDrive developer forum (at stackoverflow.com) is the best place to post the info, but I’m not sure. I’m a little out of my element here.
That is great – those links that you sent – I will soon try and see what happens.
Being intermittent, despite me having problems with two directories all yesterday and part of today, it has settled down right now. I’m curious if the v3 API produces any difference.
When I get a repro case (probably tomorrow) I will send you an odrive diagnostic, and yes, can let you know the path to the folder.
I assume for debugging purposes you can currently get into that directory? If not, I can also share the directory with you through Google and not through odrive. It is a directory full of encrypted backups. The files are not therefore sensitive.
Finally, do you have an email address such that I could forward you some test results and methodology?
BTW, what is the pagesize that odrive is using for the directory listings? 1000?
I guess that because that number would be consistent with what I was seeing with odrive – i.e. the file(s) that would go “missing” through odrive were consistent with a pagesize of 1000. The rclone developer was kind enough to make a custom build that let me query with different page sizes.
Hi @roy,
You can feel free to send any info you don’t want posted publically to me via private message here. Just click on my username and click the “Message” button.
Yes, we use a page size of 1000 for folder listings.
Yes. The ticket ticket was internal to odrive. Hopefully Google will find something on their side about this, though.
I’ve been trying to use the API “try it” pages. Of course a bit difficult on my end, as I’m trying to understand how to get at the data in the first place. I see that folders are dealt with at least on the surface differently in v3 vs. v2 API. Which API are you guys using?
Trying to hone in on a directory using this “try it” interface, and without being able to really specify path, it’s taking a bit more time for me to wade through this. Are you guys somehow giving the parent folder ID and having GDrive list the contents of that directory? I’m trying to find the parent ID of a folder of interest.
I just came across a problem case in a couple of my folders, but now those are indeed showing up in odrive (unlike before), so I will kick the tires further. If I can get you a smoking gun case I will give you the path to the folder, so you can hopefully view the behavior directly.
Yes, all of the objects are referenced by ID, rather than path, so it takes some work to get those IDs.
If you recently performed a sync on the folder, and sent at diagnostic, I may be able to extract the ID for that folder from there, to help out. Otherwise you should be able to use the query string (q parameter) to search for the parent folder with “name”, grab the ID from there, then query using the “parents” search parameter. https://developers.google.com/drive/v2/web/search-parameters
I managed to use the “Try this API” tools you made reference to in order to demonstrate the file listing problem purely with the Google API, and not involving any 3rd-party tools (odrive & rclone). It’s a Google problem.
If you can provide a way for me to send you my outputs of these tests, I would be happy to forward your way.
I’m also hoping that you / odrive can become involved with helping motivate Google to fix this terrible bug which undermines the utility of the entire Google Drive API for all of us that use it.
If you have the output of a demonstrable case, I can work on contacting Google to see if we can make any headway. Go ahead and PM me the results (or a shared link to the results) and we can go from there.