Ghost-Files and Random Downloads with Amazon Cloud Drive

Hey guys,

I’ve come across a very strange problem which seems to be caused by Amazon Cloud Drive itself or odrive’s internals.
First of all: I use odrive to manage … about 3 Million files and folders (between 1KB and 50GB each) in Amazon Cloud Drives Unlimited Storage (germany still have the offer). This is my offsite-backup solution for my personal NASes.
The ability to unsync certain files, like some raw video data (~225GB per hour on my setups) or old incremental backups is awesome… also the .cloudf placeholders, which allows a wonderful navigation through thousands of gigabytes without having all the stuff stored locally is a true killer-feature!

Ehm, some informations which may be useful:
Im using exclusively Windows Server (2012 R2 and 2016) to run the odrive agent.
Im using Amazon Cloud Drive AND the Encryption function!

But… there is a little and silly phenom which ive been watching for about 4 weeks now, and it starts to annoy me a bit.
Everytime i put something into trash, it deletes it AND puts it as a placeholder in the Amazon Cloud Drive root folder - when i try to download this placeholder… nothing happens. Also inside Amazon Cloud Drive, it is marked as deleted.

My amount of files was growing very fast and odrive was always able to deal with it - for months!
Since i passed the 3 Million mark, i have this weird ghost-placeholders trashing my file structure…

– Another problem is, that when i try to download things from Amazon Cloud Drive, a whole folder or multiple files for example, it only downloads ONE randomly picked file of everything i marked before.
Well… why? :smile:

Don’t get me wrong, odrive is awesome and i’ll not stop using it (as long as you dont add some more zeros to the price-tag), but these problems are really getting annoying over time and slowly starts to defeat the use of odrive, so i would really like you to look into these mysteries and try to solve them.

If you need any additional informations, just ask me and I’ll provide them! :slight_smile:
Fixing these strange bugs would really help me and (as i saw) some other people to continue enjoying odrive!

Best wishes,
Felix

Ehm… maybe i should add, that im not using the split feature. I let odrive sync all files as whole ones.
And, the ghost-files also seems to still have some relation to the cloud drive… because deleting them actually blows holes into their original destination.
Fortunately I’ve realised that before deleting any files newer than a few months ago…

@Tony
I just tried to switch the odrive folder to another location and back. Now odrive turns into an black hole.
Auto-Empty Trash Bin and Automatic Sync off stops it from imploding my data… Amazons Webinterface shows me thousands of deletes in their own trash-bin… why is odrive so fragile when used with Amazon Cloud Drive at the moment?
Thats absolutely new to me.

Hi @fastfelix771,

There have been some recent oddities revolving around the Amazon Drive trash. Here is a thread for reference: Mysterious unknown files/folders under root Amazon Drive folder

What you are describing sounds different though. To clarify:

  • You are seeing that files that are deleted locally, with odrive trash emptied, end up showing up directly in the root of the Amazon Drive folder?
  • These files are not visible in the root of the folder from within the Amazon Drive web client (https://www.amazon.com/clouddrive) or the odrive web client (https://www.odrive.com), but do show up in the Amazon Drive root on odrive desktop client?
  • You see these same files within the Amazon Drive trash (https://www.amazon.com/clouddrive/trash)?

I think you are talking about multi-select on several files/folders and then right-click->sync. Is that right? That is a current limitation in the odrive desktop interface. You cannot multi-select items to sync them all. You can right-click->sync on a single folder and bulk download the items inside, however: https://docs.odrive.com/docs/sync-your-odrive#section--sync-all-files-and-subfolders-

Can you walk me through this? I don’t understand the sequence of events or the results afterwards.

Mostly yes. I haven’t checked that in detail, but it looks like this only happens for some random files, but not all of them.

That is correct.

Yes.

That works quite good, but it misses about 10% of the files.
Randomly it just “forgets” that it have to download file “x” and “y” and just leave them as placeholders… i have to sync the whole folder multiple times to have them ALL synced.
This may sound funny, but actually, odrive sometimes behave like its having alzheimer disease. :smirk:

Honestly, i don’t understand it either.
This may also be related with multiple odrive clients running on different machines, all accessing the same cloud for some time.
Shutting down some of the other clients and processing the trash-bin on a single machine helped a bit - but it’s already starting filling it again… far slower though.

Ehm, what i tried:

I moved all important things to its local storages, outside of odrives sync-folder to prevent data-loss, if i break something.

I searched around a bit and found some intresting articles complaining about NTFS’ path-length limitations which can lead to errors, when NTFS long-paths are not exclusively enabled.
Well, at some point i was actually using NTFS and i wanted to fully switch to ReFS anyways, so i gave it a shot and switched the filesystems.
To prevent odrive from going crazy because all the data got locally deleted from it’s point of view, i moved the odrive folder to some fresh desktop folder and disabled the odrive agent.

After the ReFS setup was done, i started odrive, switched the odrive folder back, disabled and re-enabled odrive aaaand it worked just fine. Only thing i had to do was re-adding all the links in the odrive web interface and re-adding all the encryption folders. That was no problem.

Then i triggered a full sync of ALL files from the cloud drive and it started downloading, smoother than usual by the way.
Now, about 24 hours later, it finished the download task - for most files… some got skipped, as mentioned on top of the post.

The trash-bin behaves quite quiet now, but 1 or 2 files per hour still appear there, and i didn’t delete anything in the meanwhile. Amazon Cloud Drive trash-bin is empty though. Perhaps because i disabled the automatic upload and trash-bin emptying.


Strange is that all these problems only appear with Amazon Cloud Drive…
Google Drive, FTP/SFTP servers etc. don’t have this issue and they also contain quite some data.

So, i don’t think it’s odrives mistake, but Amazons. Maybe odrives ACD integration can become enhanced with some more fault-tolerance against Amazons interface and some validation that a special action really succeeded?

Hi @fastfelix771,
Thanks for the details!

We are currently going through our integrations, like Amazon Drive, and trying to apply some “defensive” coding to account for some of the quirks.

Do you still see items in the root of Amazon, only on the desktop client? If so, can you send a diagnostic from the odrive menu and provide a screenshot of the root folder from the odrive web client and the odrive desktop client? I would like to look into that further, as this is behavior I haven’t see before.

Yep, and during the last hours, some hundred of such files joined the root folder again.
I just sent a diagnostic, i hope it helps!

I’ll also make some screenshots and post them, but ehm… since all these files are stored in the Encrypted Folders, it’s not very informative i think. Well, if you have any use for them, you’ll get them.

Hi @fastfelix771,
I have the diagnostic, but I think I will need the screenshots so I can understand what you are seeing. Can you provide the following screenshots:

  • A screenshot of what you see in the root of Amazon Drive of the desktop client
  • A screenshot of what you see in the root of Amazon Drive on the odrive web client
  • A screenshot of what you see in the root of Amazon Drive on the Amazon Drive web client

You can send these to me by direct message if you do not want to post here.

Can you also walk me through the steps you take to reproduce this issue and the full Amazon Drive paths for the actions and results?

Thanks for the help, @fastfelix771