Deleted files moving to root of current "account" and impossible to delete

mac
amazon-drive

#1

Hello -

I have an Amazon Drive based store, when I delete files anywhere in that tree structure, they wind up under the main “Amazon Drive” folder.

When I try to delete them there, many of them have already synced to the cloud drive location, and as a result even when I delete them from my command line, they persist on cloud side and eventually resync.

I have what I think to be around a million files in this state. It is currently making odrive unusable as it is constantly syncing the .cloud files and using ~100% CPU in process. I let it run for 4-5 days and finally turned it off.

Need help!!

Ryan


#2

Hi @ryanhoium,
It sounds like you have some sync folder rules set on the root of the Amazon Drive folder to automatically download any new remote files, is that correct? You can check this by right-clicking on the Amazon Drive folder and selecting “sync”. The settings on the windows that pop-ups will indicate the current settings on that folder. If “Save and apply to new files and folders” is checked, then you have that folder set to auto-sync any new content on the remote side.

If it is set this way, you can slide the slider all the way to the left for “Nothing”, uncheck both checkboxes, and then slick the sync button to remove the rule on that folder and prevent further auto-download.

The other setting to adjust is in the odrive menu under “Set auto download”. Make sure this setting is set to “Never”

There are also two settings at the very top of the odrive menu that can disable automatic scanning/processing, which will help calm things down. These settings are called “Stop automatic sync” and “Disable background scanning”.

Once that is all set we can concentrate on the issue of trashed files in the root of the Amazon Drive.

This is a problem we’ve seen before. There is actually a looong thread on it. Here is a post that summarizes the Amazon issue and our attempt to work around it:

So far we have only ever seen this issue happen after and Amazon Drive trash purge, where the files in the Amazon Drive remote trash are deleted permanently (every 30 days, or when triggered manually by the user). Can you clarify if this is the case in your situation, or if you are seeing these files end up in root immediately after deleting them from their original locations?

Once the above settings have all been set, you should have better luck trying to clean up the foler, although trying to operate on ~1 million files may be a little tricky. Do you have the “Auto empty trash” setting enabled in the odrive menu, currently?


#5

OK, I was able to follow your instructions.

It may very well be that the files are showing back up when the purge happens. I just did a test of this to see if the files show up in the root of ACD after the purge.

That said, I also cleaned up all the files that had showed up in root of ACD, some from command line and a few from Finder.

There is nothing yet showing up in my odrive trash. So I enabled background scanning again, and now I’m at 100% CPU again… there should be no files to sync at this point. After 20 minutes, no trashed items to sync and nothing new in my trash in ACD web client.

UPDATE::: used CLI to also check status of trash, CLI output as follows:

odrive Make Cloud Storage THE WAY IT SHOULD BE.

isActivated: True hasSession: True
email: xxxxxxxxxxxxxxxx accountType: CloudDrive
syncEnabled: True version: prod 6497
placeholderThreshold: neverDownload autoUnsyncThreshold: never
downloadThrottlingThreshold: unlimited uploadThrottlingThreshold: normal
autoTrashThreshold: day Mounts: 1
xlThreshold: small Backups: 0

Sync Requests: 0
Background Requests: 0
Uploads: 0
Downloads: 0
Trash: 0
Waiting: 0
Not Allowed: 0

UPDATE #2: now the cli status command is showing 205k items in teh ‘Trash’

Going to let this run for a bit here and see if these continue to fill on the ACD webclient trash. up to 607 items in ACD trash now. It seems they are moving very very slowly - like 20 files / second… gonna take a very long time at this rate :slight_smile:

Ryan


#6

Hi @ryanhoium,
Try disabling background scanning but enabling automatic sync. The automatic sync should pick up the deleted items, although it may take a while.

You still see the deleted items in the root of your Amazon Drive from the Amazon Drive web client? https://www.amazon.com/clouddrive/all


#7

Hi @ryanhoium,
I saw your updates. It looks like odrive is trucking through the tracking of those deletes. Turning off background scanning should reduce additional overhead, since this turns off our scanning/querying for remote changes, which you don’t care about right now.

There is also a post here that goes over using the CLI to “brute force” the trash operation, since it is likely to abort due to errors with so many items.


#8

Hey Tony -

Just FYI, I’m still trying to work through the ~200k or so files that need to be moved from my local trash to the Amazon Drive trash; the odrive app crashes ever handful of hours when I let it try to work through things, and often I don’t realize its not going for a while.

That said, I’m starting to get files that were deleted when this post originated showing back up in the root of my Amazon Cloud Drive folder. Not sure why that is happening here (again). any ideas?

ryan


#9

Hi @ryanhoium,
I’ve tried to contact Amazon about the trash items moving to root 5 times now, but they have not yet responded.

The problem is that the files/folders seem to become orphaned when Amazon goes through its trash purging cycle (after 30 days of the items being in the trash, or when the user initiates the purge manually). This means it can continue to happen as long as you have items in the Amazon trash that are purged.

Are you using the CLI script to cycle through the trash?


#10

Thanks - the CLI script you shared depends on odrive being up and running; so when odrive crashes, the script does too. So I’m not seeing that it makes a difference versus just letting odrive run… maybe I’m not doing it correctly. I can try again


#11

Thanks for the update @ryanhoium.

We’ll need to take a look at how we can better optimize for so many odrive trash items. Hopefully this won’t be something that you run into often.


#12

Thanks @Tony - it shouldn’t be something I run into often, however, it seems onedrive or ACD keeps putting items back in that root folder. hoping when I get these to delete they don’t come back :slight_smile:
In truth what I really want based on the seeming flakiness of ACD is to move everything from there to Dropbox or another storage service (perhaps S3 or Box as well). but i don’t see a way to do that easily until after I get this cleaned up…

… then I assume I have to do a ‘sync all’ and get a local copy, then re-upload - correct? any direct transfer or duplicate option?


#13

Hi @ryanhoium,

That is correct. We do not have a direct cloud-to-cloud option, although it is something we’d like to offer in the future.


#14

@Tony - FYI, so I was able to get everything deleted using your procedure, however, now I’m already starting to see files go back into the root of ACD.

I guess my best option is to open another storage account with DropBox or S3 or Box, so a sync-all including all these extraneous files, and then do the delete in the new account - and stop using ACD.

Is that my best / only course of action here?


#15

Hi @ryanhoium,
We have one final thing to try in the Amazon trash issue. I think we should be able to roll it out next week, and it may greatly mitigate the issue, if not solve it, so stay tuned.


#16

OK very cool. Will sit tight


#17

Hi @ryanhoium,

We’ve released a new desktop client version that has the fix mentioned above.

Even though Amazon Drive’s API allows for moving folder nodes to the Amazon Drive trash, when they process that folder during their purging routine, they seem to tend to orphan the files inside unless those files were also explicitly trashed. We don’t see the behavior every time, so it is unclear what is wrong with their method, but we’ve tweaked our trash routine to go through each trashed folder and recursively trash the items inside before finally trashing the folder, itself. Unfortunately this means that the trash routine for Amazon Drive can be very long and intensive. If there are a lot of items to trash, you may end up needing to issue the odrive “empty trash and sync all deletes” command multiple times.


#18

tried this a few weeks ago, and it didn’t work… the items actually sync’d back while I was watching :slight_smile:

trying again now, so far it seems not to be syncing back the files that were deleted. Will let you know.