I have an Amazon Drive based store, when I delete files anywhere in that tree structure, they wind up under the main “Amazon Drive” folder.
When I try to delete them there, many of them have already synced to the cloud drive location, and as a result even when I delete them from my command line, they persist on cloud side and eventually resync.
I have what I think to be around a million files in this state. It is currently making odrive unusable as it is constantly syncing the .cloud files and using ~100% CPU in process. I let it run for 4-5 days and finally turned it off.
It sounds like you have some sync folder rules set on the root of the Amazon Drive folder to automatically download any new remote files, is that correct? You can check this by right-clicking on the Amazon Drive folder and selecting “sync”. The settings on the windows that pop-ups will indicate the current settings on that folder. If “Save and apply to new files and folders” is checked, then you have that folder set to auto-sync any new content on the remote side.
If it is set this way, you can slide the slider all the way to the left for “Nothing”, uncheck both checkboxes, and then slick the sync button to remove the rule on that folder and prevent further auto-download.
The other setting to adjust is in the odrive menu under “Set auto download”. Make sure this setting is set to “Never”
There are also two settings at the very top of the odrive menu that can disable automatic scanning/processing, which will help calm things down. These settings are called “Stop automatic sync” and “Disable background scanning”.
Once that is all set we can concentrate on the issue of trashed files in the root of the Amazon Drive.
This is a problem we’ve seen before. There is actually a looong thread on it. Here is a post that summarizes the Amazon issue and our attempt to work around it:
So far we have only ever seen this issue happen after and Amazon Drive trash purge, where the files in the Amazon Drive remote trash are deleted permanently (every 30 days, or when triggered manually by the user). Can you clarify if this is the case in your situation, or if you are seeing these files end up in root immediately after deleting them from their original locations?
Once the above settings have all been set, you should have better luck trying to clean up the foler, although trying to operate on ~1 million files may be a little tricky. Do you have the “Auto empty trash” setting enabled in the odrive menu, currently?
It may very well be that the files are showing back up when the purge happens. I just did a test of this to see if the files show up in the root of ACD after the purge.
That said, I also cleaned up all the files that had showed up in root of ACD, some from command line and a few from Finder.
There is nothing yet showing up in my odrive trash. So I enabled background scanning again, and now I’m at 100% CPU again… there should be no files to sync at this point. After 20 minutes, no trashed items to sync and nothing new in my trash in ACD web client.
UPDATE::: used CLI to also check status of trash, CLI output as follows:
odrive Make Cloud Storage THE WAY IT SHOULD BE.
isActivated: True hasSession: True
email: xxxxxxxxxxxxxxxx accountType: CloudDrive
syncEnabled: True version: prod 6497
placeholderThreshold: neverDownload autoUnsyncThreshold: never
downloadThrottlingThreshold: unlimited uploadThrottlingThreshold: normal
autoTrashThreshold: day Mounts: 1
xlThreshold: small Backups: 0
UPDATE #2: now the cli status command is showing 205k items in teh ‘Trash’
Going to let this run for a bit here and see if these continue to fill on the ACD webclient trash. up to 607 items in ACD trash now. It seems they are moving very very slowly - like 20 files / second… gonna take a very long time at this rate
I saw your updates. It looks like odrive is trucking through the tracking of those deletes. Turning off background scanning should reduce additional overhead, since this turns off our scanning/querying for remote changes, which you don’t care about right now.
There is also a post here that goes over using the CLI to “brute force” the trash operation, since it is likely to abort due to errors with so many items.
Just FYI, I’m still trying to work through the ~200k or so files that need to be moved from my local trash to the Amazon Drive trash; the odrive app crashes ever handful of hours when I let it try to work through things, and often I don’t realize its not going for a while.
That said, I’m starting to get files that were deleted when this post originated showing back up in the root of my Amazon Cloud Drive folder. Not sure why that is happening here (again). any ideas?
I’ve tried to contact Amazon about the trash items moving to root 5 times now, but they have not yet responded.
The problem is that the files/folders seem to become orphaned when Amazon goes through its trash purging cycle (after 30 days of the items being in the trash, or when the user initiates the purge manually). This means it can continue to happen as long as you have items in the Amazon trash that are purged.
Are you using the CLI script to cycle through the trash?
Thanks - the CLI script you shared depends on odrive being up and running; so when odrive crashes, the script does too. So I’m not seeing that it makes a difference versus just letting odrive run… maybe I’m not doing it correctly. I can try again
Thanks @Tony - it shouldn’t be something I run into often, however, it seems onedrive or ACD keeps putting items back in that root folder. hoping when I get these to delete they don’t come back
In truth what I really want based on the seeming flakiness of ACD is to move everything from there to Dropbox or another storage service (perhaps S3 or Box as well). but i don’t see a way to do that easily until after I get this cleaned up…
… then I assume I have to do a ‘sync all’ and get a local copy, then re-upload - correct? any direct transfer or duplicate option?
We’ve released a new desktop client version that has the fix mentioned above.
Even though Amazon Drive’s API allows for moving folder nodes to the Amazon Drive trash, when they process that folder during their purging routine, they seem to tend to orphan the files inside unless those files were also explicitly trashed. We don’t see the behavior every time, so it is unclear what is wrong with their method, but we’ve tweaked our trash routine to go through each trashed folder and recursively trash the items inside before finally trashing the folder, itself. Unfortunately this means that the trash routine for Amazon Drive can be very long and intensive. If there are a lot of items to trash, you may end up needing to issue the odrive “empty trash and sync all deletes” command multiple times.