Here’s the problem statement: As we are experimenting with odrive for more usage, we have synced several large data stores (>20TB each, with some millions of files). These contain files that we are migrating from local storage to the cloud and are - obviously - not all needed at any one time. These data change less frequently.
Other shares are far smaller and contain “hot” data that need to be accessed by multiple users frequently. The data are updated more frequently, usually multiple times an hour.
If odrive is running on a server containing both types of storage buckets with active uploads in process from each, the files in the small bucket are often delayed in uploading for long times while other folders are processed.
A useful option for this type of situation would be the ability to assign variable sync priorities to each odrive folder. Low, default, and high would suffice. Changes to a high priority folder would take the next available sync slot. Low priority folders would only sync if no other upload/downloads were in process. Default would use the existing odrive priority system.
I’ll add to this request that the problems we see when syncing large folder trees is worse from a download sync perspective. On computers that have even moderately large shares synced to the cloud (one example has the two largest folder trees with 200GB and 40K files between them, a handful of other shares with <200 total files combined) smaller trees do not update at all quickly when files are changed in the cloud. Even 5 hours later odrive has not picked up that a remote file changed. It requires manually going into the local folder and performing a refresh from the odrive context menu to get the updated remote file.
Servers with the large data stores rarely update local files to reflect cloud changes. I’ve tried syncing a folder containing a single file (all shares are encrypted, in case this makes a difference from odrive’s perspective) and after 12 hours the local file was still not updated.
In these scenarios, is odrive a client on a user’s computer or a client on a shared/IT server moving data between the cloud and the server? Also, is sync for backing up data or for users to get files?
Both cases. Odrive is installed on servers pointing at multi-TB buckets. The smaller cases (200GB/40K files synced to cloud) are on workstations.
The large buckets are used for backups and infrequent retrieval for assets not stored locally. Small buckets are for files actively shared between collaborators. We’re seeing too many instances where a file is not updated from the cloud in a timely manner.
This issue has become a showstopper for us. Even with the latest (Win v5354) client, there are significant delays in syncing files. A PC with only a few odrive managed shares usually but not always syncs within 10 minutes without needing to manually navigate to the odrive folder. Browsing to the specific local shared folder in Windows Explorer is usually enough to trigger an odrive sync but this is not an acceptable option for production work.
A computer with a large number of files (100K - 100M+) synced via odrive is another matter entirely. At the lower end of the range, it takes hours to a day for odrive to pick up a single modified file remote in a small (<15 total files/folders) share. At the upper end, we’ve gone for over a week without odrive grabbing the modified remote file. In this case, a manual right click-sync command in the folder in question is required before odrive actually syncs,
For now, we’ve given up on odrive for work purposes where maintaining timely synchronization is a requirement. None of the provider’s own apps from Box, Google, Amazon, or Dropbox suffer from this problem. With a fast connection, syncs rarely take more than 30 seconds.
Google Drive, S3, Box, ACD, and Dropbox all show the same behavior. With v5354, the Windows odrive client appears even less responsive for querying remote storage for updates. Even on a computer with minimal synced shares (3 synced encrypted shares, less than 100 total files) it either takes navigating to the local synced folder in Windows Explorer or, if there already, issuing a sync request to odrive to pick up files changed even 4 or 5 hours ago.
Hi Ethan,
I would like to get a better understanding of the issue here. Can you clarify the flow of changes and reflection where this is an issue? For example:
Computer 1 makes a change to a file in a local folder that is syncing to Dropbox.
Computer 1’s change takes X time to sync up to Dropbox (the change is not visible via Dropbox’s or odrive’s web client for X amount of time)
Computer 2 does not see the change on the local file system until Y amount of time
In the above example case, I could anticipate a large delay in reflection on Computer 2 if the local folder on Computer 1 is actually on a network share, or external volume that does not allow for optimized change detection.
@peter: Indeed we are using encryption on all affected shares. That’s a killer business feature of odrive: not only does odrive allow businesses to use any storage provider without installing multiple interface tools, but it supports zero knowledge encryption.
@Tony: The flow is largely as you outline. We’ve seen this problem occur when all odrive storage is on local disks for both computers 1 and 2.
1: Computer 1 changes a file in a local, encrypted odrive folder.
2: Changes are synced by odrive within a small (<3 sec) time to the cloud provider. Both odrive’s interface and looking at the cloud storage directly via the provider web interface show the updated file.
3: Computer 2 sees the change only by manually browsing to the local synced folder in Windows Explorer or in some cases after a manual odrive refresh is issued. If Explorer is not used, the updated file does not sync to computer 2 for a time ranging from tens of minutes to never. There appears to be a correlation to how many other encrypted files and folders odrive is monitoring. Sync times range from ~30 minutes if <10 files are synced to days for 10’s of thousands to never for 1M+.
This thread should possibly be refiled under troubleshooting. The problems we see with odrive not detecting modified files without manual intervention makes odrive useless as a collaboration tool in our environment.
Edit to above. After 5 days, two Windows machines running odrive v5391 have not picked up a changed file. Odrive only syncs to a single encrypted folder containing one file on each of these computers. The share on one computer is hosted on Box, the other uses Google Drive. The native clients synced the changes within seconds of the files being modified.
@peter - I may open a new thread on this, but wanted to say that aside from the other problems Ethan was having, very generally the ability to give a folder or file “priority” would be extremely useful.
I have a similar issue on a smaller scale. I have a vast amount of design assets and closed projects that I am syncing along with data that may be considered, as Ethan put it, “hot”. I would like to be able to keep all the backups going without having to intervene by just letting the odrive client know which of the files/folders in the sync queue are a priority for me.
Exactly. I made the mistake of using odrive to upload ~300K files from the same computer that I modified a single file in another sync account. While odrive ever so slowly processes the large batch of files, the single file never updated. Odrive on the source computer claims that the file in question is synced. No other computers see the update after more than 20 hours. As the file in question is in an encrypted folder, there is no independent means of verifying its status.
Manually copying the file from computer to computer resulted in odrive creating numerous (conflict) files. I’m willing to let odrive poke away through a large upload, but it would be ideal if those folders could be assigned a lower priority.