Odrive not syncing timely with google drives

[erikal]

We have 6 different ODrive accounts and each of them contain the same (300+) google drives for daycare centers that we support. I am finding that the odrive is not syncing or catching up with the changes that all of the google drives are doing. We use the odrive to have a central location for all of our google drives because they all have scanners that are connected to their google drive and odrive allows us to search for a certain folder that is created when a scan comes through, this is a time-saver, otherwise, we would have to open each individual google drive to see if anything came in.

I have deleted odrive off of our computers and put it back on and it still will not catch up. I have deauthorized all accounts except one, it will not catch up.

O drive is also creating back ups on our computers that are taking up enourmous amount of space and then we have to go in and delete the back up to even be able to work…PLEASE HELP!

[Tony]

Hi @erikal,

Wow, 300+ links! That is probably the most we have ever seen.

Your use case may require us to tweak the configuration and approach a bit to make sure we can satisfy your use case.

From what I understand:

• Each google account is used by a specific daycare center
• Someone at each daycare center is scanning documents, which are uploaded to Google Drive
• You have 6 computers that you use to centrally manage these scanned documents, and each of these computers has odrive installed with the 300+ Google Drive accounts linked.
• You are performing searches within odrive for a specific folder name. If it is located you know that a new scan has been added

Is the above all correct so far?

If so, I have some follow-up questions (sorry there are so many, but I just want to make sure I am clear on the issue so I can come up with an appropriate solution):

• Is this a recent problem? If so, could it be that the number of links has reached some sort of critical mass that is now causing delays? (In other words have you been continuing to increase the number of linked drives, over time?)
• Are all of the odrive installations experiencing the same delay issues?
• Are you seeing that changes are showing up, but they are just not showing up fast enough?
• How often are documents being scanned and uploaded?
• What software is being used at the daycare centers to upload the scanned documents?
• How are the folders named? (What type of designations are you using to search for new folders)?
• Is a new folder created with every scanned document?
• What is the acceptable window of time for having the new folders be available within odrive?
• Are you only concerned with the new folders that are created each day, or do you also need to have all of the previous folders synced, from all of the previous scans?

Can you please send a diagnostic from the odrive tray menu from one of the computers? This will allow me to gather some additional data on the configuration.

@erikal

O drive is also creating back ups on our computers

Can you clarify what you mean by this? Uninstallation will create a backup of the data, so this may be the backups you are talking about?

[erikal]

Yes, it is a recent problem last couple of weeks. We have experienced a lag before but not to this extreme. Yes we add more centers often. Would it be beneficial if we separated the centers and not add them all on the same odrive?
Yes, they all experience a delay…some will be the same and get the information and others it will take a couple of hours if not days.
Recently we have used the odrive to sweep our accounts for scanned documents and we put them in the correct folder to be worked…well since the odrive is not syncing we went into the individual google drives and the scans that had been moved through the odrive on October 19 were still in the google drive today.
The daycares use a Neat Scanner or a Raven Scanner to upload documents.
For the Neat Scanners when something is scanned it actually creates a folder called “From NeatConnect” once we move the scans from that folder we delete the folder, the new folder actually acts like a flag to let us know that that center scanned something in to us. With the Raven Scanners we had to create a “RAVEN SCANS” Folder in the google drive and then point the scanner to put forms there. We just empty that folder and leave it there since those scanners will not create a folder and need an exact path to put the upload.
Acceptable time should be within minutes. It always has before. I could be talking to a center and have them scan something and I would see it almost instantly.
I need all folders to sync constantly. We “sweep” our centers 4 times a day to get new scans.
Sending diagnostic now.

[erikal]

Ok, so it looks like when I uninstalled the odrive I created the back up…I was just trying to get it up and running.

[Tony]

Thanks @erikal!
I am looking through the diagnostic, and I do actually see a lot of auto-syncing for the mentioned scanned folders (From NeatConnect and RAVEN SCANS), but I am guessing there are specific ones that you are not seeing.

There are a few things I see that could be influencing the behavior here:

1. You have folder sync rules set on most of the linked drives, but not all. This can influence how odrive is expanding/downloading items. We can try set a consistent rule on the entire odrive folder so that everything is synced the same.

2. You do not have any auto empty trash rules set. Since you are moving folders and files quite a bit, it is possible that odrive is not able to detect the full move and optimize it and will pick up these moves as add and deletes instead of moves. This can explain why you may see items that you moved in odrive that still exist in the old location in Google Drive

3. There are a number of files in “Not allowed” because the remote version has become older than the local version on the computer. This is a safeguard settings that we recently set to prevent users from accidentally copying over newer files with older ones (for example, if a user uploads an older version of a file on another system). odrive will detect that the file on Google storage is now older than it was before and halt the download, to prevent any data loss on the local machine. This setting can be turned off if this is expected behavior as part of your workflow. I will cover how to do this below.

Here are some things I recommend:

1. It looks like you want all of the Google Drive links to always be synced, with everything downloaded. We can set a sync rule on the odrive folder, itself, so you don’t have to set it on each individual link. To do this, right-click on the odrive folder and select “sync”. Then slide the slider all the way to the right for “Everything”, check both checkboxes, and click the sync button.
   a.
   b.
   c.
   d. Note: It may take a while to go through all of the folders once you’ve done this

2. As mentioned above, since you are performing a lot of moves of folders, you may sometimes hit an instance where odrive isn’t able to see the move from one place to another, and ends up picking it up as an upload to the new place and a delete from the old place. When this happens the both the new and old place will have the files that were moved, until you empty the odrive trash, at which point the files in the old place will be deleted. You can also have odrive auto-empty the trash using the autotrash rules in the odrive tray menu
   a.

3. To address the “older file” warning, you can disable this check in our advanced settings.
   a. Go to the root of the odrive folder and open the odrive_user_general_conf.txt file
   b. Change false to true for the allowOldDownload setting
   c. Save the file

After the above I am hoping you will see better behavior. Please let me know if you have any questions on the above items.

If you are still seeing delays after these adjustments, then please let me know. I will need the following to properly assess the issue:

1. A new diagnostic sent
2. A few example paths that are not showing up when expected

If you are still seeing large delays there are a few other things I can take a look at from the product side, but I want to make sure things are all dialed-in before we start that process.

[erikal]

Okay, I put those fixes in place and it is syncing right now.

When you ask for a path, please talk to me like a kindergartner and tell me exactly what you mean! LOL

[Tony]

Thanks @erikal!

For a path, I just mean the location of the items that are not showing up when you expect them too. For example:
C:/Users/[your user name here]/odrive/[Daycare center name here]/RAVEN SCANS

You can send this info to me directly by clicking on my name and selecting “Message” if you don’t want it in the post here.

Additionally, if/when you send example paths, if you could also note the time the files were uploaded, so we can gauge the amount of delay up to that point.

[faithg]

Well, yes and no on better behavior. I reloaded the odrive on my machine and so right now we have only 2 of the 6 up and going and even though they are connected to the exact same google drives and her computer is maybe 15 feet away from me, it still is syncing slowly.

Any extra ideas would be great! I would really like to get all of our odrives back up and running properly.

[Tony]

Hi @faithg,
I moved this post here to put it back into the context of the original post.

I’ll also repost some of the info I posted previously in our direct messages, for reference:

Things to keep in mind:

1. odrive will only know about remote changes (new file or folder added to a remote Google Drive) once it asks Google Drive for a list of changes, since it last looked. We ask Google Drive for changes every 1.5 minutes. However, since you have ~350 Drives, the routine to check for changes across all of the drives will take a while to complete. At a minimum it will be about 15-20 minutes to complete the whole circuit. If there are changes in drives to process, then it will take longer because odrive will need to process all of the changes in each drive. The extra time will depend on the number of changes to process across all of the drives.

2. When you restart odrive it can cause large delays in seeing new changes. odrive will scan all of the folders when it starts-up. This means it will go into each folder and ask Google for a listing of the contents. You have over 6000 folders exposed on your desktop, so this routine can take quite a while, especially if there are changes found to sync down (could be as much as a few hours).

3. If you ever want to get an immediate, new listing of a folder, you can right-click on the folder and select “refresh” (this is the odrive refresh, not the standard Windows refresh). This will immediately ask Google for a currently listing of the selected folder.

@faithg

they are connected to the exact same google drives and her computer is maybe 15 feet away from me, it still is syncing slowly

Since the get changes calls will be on different cycles, you may see that some computers will pick-up changes before others and vice-versa. Since there are so many Google Drive accounts to go through, there will likely always be an offset and you could see differences of up to 20 minutes or more. (See 1. above for more on why this is.)

Restarting odrive will cause delays because it needs to perform a full scan of all of the Google Drives to see what has changed. (See 2. above). Depending on the number of changes to process it could take on the order of hours to finish this process. So, if you have cases where the computers are shutting down each day and then starting-up in the morning, this can definitely delay things.

Some questions to help guide potential improvements to the process:

• Is odrive being restarted often (every day), either through computer shutdowns and start-ups or other means?
• Is there a reliable location for the scan folders? It seems like they may always be in the root of the Google Drive. Is that correct? For example: odrive\[Google Drive Account\RAVEN SCANS
• When you say that your desktop is syncing slowly, can you clarify the behavior you are seeing? (What you are seeing vs what you are expecting).

[faithg]

Thank you! I have just changed my unsync to after a month and we will see what that will do.

Would it be helpful to split the google drives onto two main computers to do a “sweep” of the accounts and then the other employees only have the centers that they are responsible for on their odrive instead of all google drives on all odrives?

This is what a sweep looks like when we search the odrive for folders called “FromNeatConnect”

The scanners that we use will create a new folder called From NeatConnect and that is our flag that something has been scanned over. We then go to that center on the odrive and move it to a folder that is shared with different people through shared drives in Google. It looks like this… once the files are moved to expenses or received we delete the From NeatConnect folder, so next time they scan something it will show up as a new folder.

They are then able to work the forms and then move them into a subfolder under these main folders.

This used to be a really fast action to where we would see it within 30 seconds to 1 minute in the google drive once moved in the folders in the odrive…I don’t know why all the sudden it is so slow and some will not sync at all.

Our computers do go to sleep at night. Do you suggest to keep them up and running?

[faithg]

As of 4:40 Pm I have 14 From NeatConnect Folders and Erika has 45. We do a sweep at 8, 11,2 and 4 of all accounts. Why again is it so different?

[Tony]

Hi @faithg,
Thanks for this information!

@faithg

Would it be helpful to split the google drives onto two main computers to do a “sweep” of the accounts and then the other employees only have the centers that they are responsible for on their odrive instead of all google drives on all odrives?

This is a great idea and can definitely help speed up reflection on the computers. odrive’s perfomance is directly tied to the amount of work it needs to do. By reducing the number of folders it has to worry about you can speed up the overall time to reflection considerably. This can actually be done very simply by selecting the folders you aren’t concerned with and then using right-click->unsync on them.

I suggest you do this on any computers that do not need access to every Google Drive you have linked. If they do need access later, they can always right-click->sync those folders again.

@faithg

This used to be a really fast action to where we would see it within 30 seconds to 1 minute in the google drive once moved in the folders in the odrive…I don’t know why all the sudden it is so slow and some will not sync at all.

After changes have been made in odrive and have synced, you should always be able to see them immediate by navigating to the location where the changes were made and performing a right-click->refresh on the folder.

The search you are performing for FromNeatConnect or RAVEN SCANS will depend on when odrive can detect those changes on its own. (See 1. and 2. in my previous post). I would expect that changes you are looking for via search will take, on average, about 20 minutes to show up, but could be much longer if there are a lot of changes to go through in some Google Drives.

@faithg

Our computers do go to sleep at night. Do you suggest to keep them up and running?

Normal sleeping should be more or less okay, since odrive can ask the Google Drives for changes when it wakes up. However, if there are a lot of changes to process it can slow things down quite a bit. Shutting down the computer so that odrive needs to start up again (or just restarting odrive) will really slow things down.

Keeping the computer on all the time will provides the fastest reflection because odrive will be continually processing changes and not having to play “catch-up”.

@faithg

As of 4:40 Pm I have 14 From NeatConnect Folders and Erika has 45. We do a sweep at 8, 11,2 and 4 of all accounts. Why again is it so different?

In the screenshot you posted a good majority of those folders are showing times after 4:00, which means that odrive just recently picked them up on that computer. I assume the screenshot was from Erika’s computer since it has more than 14 From NeatConnect folders. It may be that your computer is still in the processes of getting these changes.

If you are able to send a diagnostic from both systems when you see a difference like this, I can analyze them and see what is causing it and if it is what we expect.

Another thing I can do is create a simple script that can run every minute or so that will focus just on the locations you are concerned with and directly refresh them. This can help to speed up the pick-up of the new scan folders so that they show up quicker in your periodic searches. This may not be needed if you implement your first idea of unsyncing drives that users don’t need to monitor, but it is something we can try if we want to increase the performance further.

[Tony]

Hi @faithg and @erikal,
I just wanted to check-up on this and see how things are going. Were able to take a look at my previous post above?