Edit: oDrive DIDN'T delete any wrong data from the cloud storage, despite failing drive making it think that I was deleting files. Post left for awareness to of the danger if you auto-empty trash is on

Troubleshooting
1

EDIT 2: Odrive’s support team was very speedy, and confirmed that odrive had not deleted anything from my cloud drive because of this. So I didn’t lose any data, and am happy. Leaving this here because I think this is still a theoretical possibility, so it may not hurt to be aware of it if you start seeing odrive sending items to your trash that you didn’t actually delete. It could be your drive failing.

EDIT: It may not be as bad as I thought, or even exactly what I thought. oDrive’s defaults don’t empty the trash and commit the deletes unless the user manually selects empty trash. Support has been super responsive and I’ll delete or update the thread as appropriate as soon as we know more.

PUBLIC SERVICE ANNOUNCEMENT: I’m posting this so that others can be aware that it can happen. If I’d known what I was seeing earlier, I would have been able to save a lot more data months ago.

I have used oDrive as a backup solution for years, and loved it. I still love it. Now, it’s like my friend has become my enemy.

A few days ago I found out that one of my data drives may be failing. It’s throwing bad sector errors, and failed to clone the disk. I wasn’t worried about this. I use oDrive to sync between a few systems, and my data drive is just a terminal, essentially. I thought all my data was backed up and safe on Google Drive.

Apparently my hard drive started failing a while back, because I’ve been seeing local deletion notices for items I didn’t change for a while, but assumed they were the auto-unsync tools doing their thing. It turns out that it was oDrive deleting files from the cloud as those files failed on my local drive (I think).

I checked the logs today to make sure that my backups were up to date, so I didn’t have to worry that the drive was failing. What I found were long lists of random files that I haven’t touched in a long time being marked for deletion from the cloud, because they had been deleted locally. These were mostly larger files, like old movies, old photo archives, but also smaller ones. I looked locally and they were indeed gone. I believe that as the drive has failed and the larger files have been more likely to be corrupted, they were being “lost” by the operating system, and essentially deleted.

I believe that oDrive has unfortunately detected those corruptions and disappearance as the file being deleted locally. It’s then been sending instructions to the cloud to delete the file there, as well.

The result has been that as my local drive has failed, file-by-file, oDrive has dutifully carried that drive failure to my backup drives in the cloud, and wiped out each of those files in my backup.

I’m not 100% sure this is the cause, and I’m still investigating. I don’t have an easy way to know how much data I have lost - how do you know if you’ve lost 1 photo out of 100, or 10 out of 100. I do know I’ve seen long lists of files queued for deletion on various days for a long time, when I delete almost nothing from my system.

So, here is the service warning. Unless oDrive has some sort of solution, or some sort of technical way to catch the difference between a slow drive failure vs. your local deletion, having oDrive installed has the potential to silently wipe out your online backups.

Please periodically check to make sure it’s deleting files that you have actually deleted. :frowning:

2

Hi @ar.stanton,
I am very sorry to hear this. Data loss is something that we take extremely seriously and we have tried to make it as difficult as possible to unintentionally delete data.

If your drive was providing a listing of local directories, but those listings were suddenly absent of files that were listed before, odrive will pick those absences up as local deletes. This is actually the only way odrive can detect local deletes.

I wanted to clarify a few things, if you are willing to provide some additional details:

  1. You mentioned “backup”. The backup functionality cannot delete remote data.
    Were you using odrive’s backup functionality, or its sync functionality?

  2. odrive will consider an item deleted if it is no longer listed on the local filesystem. By default it will queue those detected deletes in the odrive trash, and it will not delete any remote data until the odrive trash is emptied.
    Were you emptying the odrive trash or have a rule set to automatically empty the trash?

  3. The sync activity log will list all of the odrive sync activity, including deletes. It will show when a local delete is detected and when it is emptied from the odrive trash and deleted on the remote.
    Do you see these events in the sync activity log?

  4. odrive will always attempt to “soft delete” remote items, if the remote storage supports it. This means, if you are using Google Drive, for example, the items will be put in the Google Drive trash to allow for recovery.
    Does your cloud storage have a trash feature?

In the case of a slow drive failure where files start to disappear, I’m not sure how odrive sync can protect against this aside from the features listed above. odrive is meant to be a full bi-directional sync engine, including syncing local deletes. That is why the odrive trash was created, to provide a safety net for accidental or unintended local deletes. The odrive backup feature was also created to provide true one-way backup functionality, if folks did not want the full bi-directional sync.

Again, I am sincerely sorry that this happened. Please let me know if I can provide any assistance with anything. Please also let me know if you feel that odrive is behaving differently than described above. I want to make sure we identify and fix any possible issues.

3

Hi, Tony, thank you. You’re support has always been excellent, and I’m not doubting that now.

However, this seems like a pretty clear fatal flaw to using oDrive for storing data in the cloud. The fact that a local drive failure can wipe out my cloud storage really makes the idea of offsite storage a bit pointless. It undermines the primary reason people have cloud storage, I think.

May I ask - if a file becomes corrupted do to read errors on the local drive, will odrive detect that as a file change, and replace the cloud version with the corrupted version, as well?

To answer your questions:

  • Sync feature, not backup.
  • I quit odrive and deauthorized it as soon as I realized it was happening. I’m scared to turn it back on again for fear that it will send more delete commands to my remaining files. That said, I’m pretty sure that my trash settings were default, so I think it auto deleted every 30 days? I’m not 100% sure.
  • I submitted a copy of my logs and sent an email with a copy about the same time I wrote this post, so if you look you may be able to find my logs. Unfortunately, I’ve been having a regular issue with odrive where it will unauth itself every day, so there’s only about 5 days of logs in it. The rest is just tons of failure to auth messages. I wasn’t able to make sense of it - I would really love it, actually, if someone with more knowledge could look at it and help me understand if I’ve really lost as much as I think. I’d love to know how long it’s been deleting cloud files.
  • I have Google Drive, with a trash feature. However, it’s completely empty. If there is some way for me to recover something that’s been soft deleted, that would be great. But again, I’m afraid to boot up and re-auth the software because of any delete commands.

If I may suggest:

I understand that odrive doesn’t have any way to identify the difference between a file that’s lost due to data corruption vs. a manual delete. But regardless of the technical limit, this seems like a legitimate threat to the viability of your product. If connecting odrive puts all my cloud data at risk, that’s a really big deal. And if a corrupted file on a bad drive would also be considered a changed file and somehow damage the cloud files, it invalidates the major use of the tool, I’d think.

What I would suggest is that the pattern of deletion from a failing drive would look completely different from a human deleting files. Whereas a human would likely delete files together, and in inconsistent intervals, it looks to me in the limited log files that they were being deleted from random places, one file here, one file there, and groups of 10 or so at a time, on a pretty predictable trajectory over time. I would think it would be pretty easy to flag deletes that are happening at all similar to that. If you flag them over a couple days, pop up a message to the user saying, “This is your data collector. We’ve noticed a deletion pattern on your hard drive that suggests your hard drive may be failing. Please confirm your recent deletes, and … etc.”

If odrive could have warned me it was happening, it would have given me time to cancel or check any trash before it emptied. Also, it would be a feature, a data guard. It would have been incredibly valuable, and would protect users from local data corruption spreading to the cloud storage.

Because while yes, it’s designed to by a sync function and I understand that, I’m pretty sure that the average consumer would not use a tool that has the potential to wipe out their online storage due to local hardware failure. That’s really bad.

4

I’ll add though that your point about the Google Drive trash is a good one, and gives me some hope. I just double checked and the Drive policy is to empty the trash automatically after 30 days. I never do this manually. The fact that it’s not full of files that have been recently removed by odrive would suggest that it’s not doing this as often as the log file would suggest.

Again, if someone could look at the log file I submitted and help me understand it, that would be greatly appreciated. You said the log shows both when it’s marked for removal, and when it’s actually removed… maybe it’s just doing one of the two and something else is going on.

I did check one of the items that were marked for removal in the log, and it’s gone from the local drive, but present on the cloud. So, that change has not been committed yet.

5

Ok, Tony - so it’s possible I have no idea what is going on, and happy to remove or update this if that’s the case.

I braved starting and reauthing the app again, assuming that anything removed recently would be in the trash of odrive or google drive… my odrive trash is set to never auto-empty. Not sure if those settings could be reset by my logging out, but I don’t see why they would have been. So, it didn’t empty my trash. But also, there’s no files in my trash. BUT… the log does say it has a lot of files listed for being in the trash.

So, I’m confused and possibly over racted. I don’t know what is or isn’t missing from the cloud.

6

Hi @ar.stanton,
Thanks for the reply!

The default behavior for the odrive trash is to never empty it automatically. I see that you sent a diagnostic yesterday and in that diagnostic the app is still at the default value for never emptying the trash. This is good news! If this has always been set this way then odrive will have never deleted any remote data unless you explicitly emptied the odrive trash through a manual user action.

I can do some further analysis with the full set of logs. They will be located in the following directory:

C:\Users\Aaron\.odrive\log\

The .odrive folder has a leading period and is hidden by default, so you may need to enable viewing hidden items in Windows Explorer to see it.

If you can zip up the log folder and send it to me via a private message (click on my avatar and select “Message”), I can take a look at the historic sync activity, as far back as the logs allow, and try to gain more insight into what was going on.

The diagnostics are showing ‘cyclic redundancy check’ errors, where odrive is trying to read a file but is unable to. CRC errors are good indicators of disk issues, so I am going to look into the possibility of detecting this type of error and providing a user alert if it is encountered by odrive.

I strongly advise replacing that drive as soon as possible! I also strongly advise that you do not add or change any data on that drive until it can be replaced.

7

Great, thank you. This is good news, and despite my complaining, it sounds like odrive has intelligent defaults to keep damage from going beyond the local. I think the error detection is an excellent idea; at this point, I would have considered it a huge service to have something catch that.

Let me send you the logs in a PM. Thank you again.

1 Like
8

Hi @ar.stanton,
Thanks!

The logs go back a couple of weeks and there aren’t any remote delete actions shown, so it looks like all local deletes were being kept in the odrive trash and never hitting the remote storage.

For the Google authentication failures, I’m not sure exactly what is happening. It is like Google is not allowing us to get a new access token when the old one expires.

Here are my recommendations for next steps:

  1. Replace the failing drive
  2. Login to odrive again and set it up, as before (When you deauthorize the settings all revert to defaults)
  3. Revoke odrive access to Google by doing the following:
  • Go to https://myaccount.google.com/permissions
  • Click on any ‘Oxygen Cloud | odrive’ listings under ‘Third-party apps with account access’ or ‘Signing in with Google’.
  • Click on ‘Remove Access’ for any of those listings found.
  1. After step 3, odrive will prompt for authentication at some point. Reauthenticate with Google and wait a few minutes for the new information to be pulled. See if that improves the reauth prompting behavior.
1 Like
9

Thank you, Tony. That’s great news re. the deletes from the cloud. Sorry to have raised a false alarm.

I will try that with the auth. after I replace the drive. It’s been driving me nuts, to be honest. It didn’t used to be a problem, but now I have to reauth every morning as part of my routine.

1 Like
10

Hey @ar.stanton,
I just wanted to let you know that we released a new version that will alert users with a pop-up if a CRC error is encountered:

1 Like