Amazon Cloud Drive internal server error

Troubleshooting
21

Thanks.

We may need to increase our timeout to accommodate AD’s long response times for folder enumeration. However, we use pagination, so it doesn’t make a whole lot of sense why Amazon would be so slow to respond. I will do some testing to see what we are looking at here.

Are you able to list the folder in the AD web client or the odrive web client?

On the bright side, at least you uploaded a 131.8GB file to Amazon Drive, which has a file size limit of 50GB :slight_smile:

Now we just need to make sure you can get it back down.

22

Right! Exactly why I love odrive.

I think this is what you mean by listing the folder?

Pasted image1094×551 41.2 KB

And you mention odrive web client… let me see, here we go:

Pasted image1315×683 37.8 KB

In the meantime, should I stick with using 2 GB split? Try that again with a new file I have?
Thanks

23

I actually prefer 100MB split because its got less risk of individual segment failure. Let me take a look and see what I can find. If we put it up there, we’ll make sure you can get it down.

24

okay, I started a 2 GB upload again of the new file before reading your message… as it takes 4 days, I’ll just let that keep running for now, to see if that works which I imagine would be informative. Thanks for looking into this! :grin:

25

Just to update, we’ve identified the cause of the troublesome download and are working on a solution.

26

Awesome, thanks for being so responsive. Looking forward to trying it out.

27

Okay, don’t shoot the messenger, but I have new issues now. The 2 GB upload I started again of the same file completed apparently successfully, and then I first unsynced to a placeholder state, then synced it down again from Amazon Cloud Drive (ACD), also apparently successfully. However, when I went to restore the disk image from that disk image file (dmg) I found it kept failing. My son needed his computer so I had to give up for the night. When I returned to it this morning, I found odrive had started syncing the same file again from scratch. So, the same file, which I had “resynced” down from ACD in it’s same place in the same folder, had started to sync all over again and had the pink syncing icon next to it. (I’ll try to update later with images once my son returns with his laptop.) When I went out to ACD I found a brand new IFS file folder for 0703b.xxxxxxx where xxxxx is the unique hash appended whenever an IFS file upload occurs. This hash was different from the 0703b.xxxxx file sitting next to it in ACD, the original file that had already completed uploading a day or two ago and which I had already “successfully” resynced. I stopped the new sync as I don’t need it uploading again and don’t have another three days to wait for it to complete on it’s own, and couldn’t think of any value in letting it complete.

My suspicions were confirmed when I finally got to the step in my testing process where I compare the resynced file with a local copy I made of the original dmg file, a copy saved before any odrive syncing experiments. I use TidyUp and set it to compare all file contents, so it performs a bit by bit comparison. I used it on other resync tests previously and it successfully found both the original and the resynced files to be identical in past tests. Unfortunately, this time TidyUp found that the resynced file was NOT identical anymore after IFS split and upload to ACD and then round trip reassembly back down to my local machine during resync. That seems to indicate that something went awry this time in either the split and upload process, or during reassembly upon resync.

I’ve asked my son to send diagnostics to you, but I don’t know if he has had the chance. If not, I will send them as soon as I lay hands on the laptop again.

I hope this detail is helpful, but please feel free to focus the conversation by asking specific questions you may have. I think this might be a new issue actually, and perhaps it’s a new topic worthy of its own separate thread. Let me know if you agree and you or I can cut and paste this over to a new thread if you do.

28

Thanks for reporting this.

I’m trying to think of the best way to investigate this. I just had a lengthy talk with the lead dev for IFS and we have a ton of checks in place to prevent any deviance between source->cloud->source. The file won’t actually even download if it is detected to differ from the cloud when it is downloaded. Each hash is checked and then the cumulative hash is checked.

So, if we were to take this as fact, it would mean that the file was changed prior to upload, or it was changed after re-download. The fact that it started uploading again would indicate that some change was made somewhere…

Some questions:

  • What is this .dmg? (app, file format, purpose, etc…) I believe these are Time Machine backups, as per the previous conversation. Are you choosing a dmg as the backup target for your Time Machine backups and then placing those into odrive?
  • Is the dmg read-only, or is it a read-write, meaning it can be altered? If you are targeting a dmg for TM backups, I assume its read-write and can be modified at any time.
  • Can you walk through the steps from dmg creation->placing in odrive->upload->unsync->download->compare for this file, including if/when/how it was knowingly copied, moved, or modified between folders/systems/drives/etc…

Thanks!

29

dmg file is read-only, and is not an active target of Time Machine. Time Machine has also been turned off all this time, since dmg creation. Interestingly, the dmg does exactly match a prior dmg created from the same Time Machine, showing that the disk utility is working perfectly, creating an exact same dmg each time, also showing that the TM backup has not changed either. All of which proves that the TM backup is not changing, nor is the dmg being accessed and changed. As far as all the steps, I believe you listed them all! Except that this never happened: “if/when/how it was knowingly copied, moved, or modified between folders/systems/drives/etc…” It is on an external hard drive that has been ejected and or restarted, but that is about all I could think of for why odrive might think it ever “moved.” I sent diagnostics just now, but my son will be traveling with his laptop for a few days so I won’t have access for a bit. Thanks

30

Thanks Paul.

Can you step me through how you are creating these dmgs? I would like to test it, but I want to make sure I perform the same steps you are.

31

I realize I should have immediately run the comparison via TidyUp immediately after completing the resync (download). Now, there are valid questions about what happened after that and before odrive started thinking it was a different file. I did compress and encrypt the dmg file, so that may no longer mean it is “read-only?” And after re-sync, I attempted the restore as I said. And, I opened the dmg directly by doubleclicking the dmg file, whereby the Mac recognizes it essentially as a mounted drive, and maybe the Mac modifies it slightly, even if only the modified date. Come to think of it, I’ll need to check that out, as TidyUp won’t even compare files with different modified dates. Let me get back to you after trying WinDiff or something. Either way, the re-synced dmg file is failing to restore properly. Let me check these things out, and do you have any other suggestions for me to try?

I will paste the steps I use to create the dmg shortly, but I think you have what you need to know when I say I select the only options, i.e. compress and encrypt, there really isn’t anything else to choose I don’t think.

32

Thanks Paul. I’m just trying to figure out how the dmg is created. Are you creating your own sparse disk then choosing it as a target for TM or is TM creating ti automatically? I believe when you target a network destination TM will automatically create a sparse bundle disk image, but I’m not sure if that is happening here.

Thanks!

34

Sorry, post is accepting pasted text as an image, that’s why I deleted the previous. Trying other things (paste and match style worked):

When I created the new disk image dmg, I simply went to disk utility, highlighted the separate TM partition, went to file menu, new image, and created the image from the partition level.

Pasted image720×177 83.7 KB

I then selected Save as as “TIMEMACHINE1 -LUKE MBP BAK 16-0703b,” no tags, saved it to the external HD location, format compressed, and set up for encryption at 128 bit.
Pasted image720×334 107 KB

I then did round trip up to ACD and back as described earlier.

Restore steps: Disk Utility, create new volume on target hard drive (different from source HD), worked with more than double the space, as it needs more space than the size of the image to work with. Select target, Edit > Restore, click image button, then choose the dmg file image source you are restoring to the target. Target volume will be erased and become identical to source dmg image (except for partition size).

Unfortunately, when I go back to test restoration of the original dmg created out of disk utility and undergoing no round trip to ACD and back, that one ALSO FAILS. I am crestfallen. :stuck_out_tongue_winking_eye: It gets the following error, the same as with the round-trip file:

Pasted image720×427 88.8 KB

I wonder if that has something to do with the encryption, but the encryption is already authorized to grab password from my keychain, so I don’t know what to say. I’m going to have to start all over with no encryption this time. What a colossal waste of my time to find out Apple apparently hasn’t gotten this to work properly, or at the very least, not intuitively. I receive no messaging that would tell me I need to handle an encrypted dmg any differently from an unencrypted one. Sorry to waste your time thus far.

File sizes are technically different after round-trip, minimally, but still different, which is not surprising. I can’t force TidyUp to ignore file size when comparing file contents, and the original dmg doesn’t work either, so I have no way of knowing if the file’s SHOULD work the same after round trip:

Pasted image1093×398 195 KB

35

BTW, all of this was done with TM turned off, so TM is not writing anything to the backups.backupdb, so no changes would ever have occurred from TM activity, not during dmg creation, nor after resync, etc.

36

Hi @paulschroeder1
I was able to do a little testing here. I didn’t realize that you were using standard disk imaging instead of time machine.

For restoring a disk image like this, you actually need to mount the dmg and then restore from that (choose the mounted dmg volume as the “restore from:”) instead of trying to restore from the dmg directly. When restoring directly from the dmg file I encountered the same error you did.

After an image is created, you can choose “Scan Image for Restore” in the Images section of Disk Utility, which will tell you if it is valid. You can also “Verify” it to make sure it is still valid and correct and it will also display a checksum, which you can use for later verification.

I haven’t reproduced the re-upload scenario yet, but I’m hoping this can give you a little more to work with.

37

Aha, interesting, let me try that out tomorrow. I did try those “Scan Image for Restore” and “Verify” options but got no result, probably because the Image was fine. This seems like a major bug in Apple’s utility that you have to mount the dmg first, and of course, as I mentioned earlier, very unintuitive as it’s not mentioned anywhere (except now, here, by you… thank you for recreating that so I know I’m not insane!)

I’ll report back on that when I can, but for now, some fairly significant other news: I repeated the sync, unsync, resync test using a compressed but not encrypted dmg file, and all was well. TM worked perfectly once the file had been reassembled and restored from the Amazon Cloud.

I’ll try to mount the encrypted dmg first per your suggestion tomorrow (only problem is with encrypted dmg, and the problem exists locally with encrypted dmgs, without ever going anywhere near odrive), but I may end up keeping the dmgs unencrypted and just use odrive encryption instead at this point. This process has been so long (but ultimately successful, thankfully).

Thank you for riding along with me.

38

SUCCESS! :tada: :triumph:

The encrypted dmg version restored successfully now as well, using your tip Tony about mounting/opening the dmg within Disk Utility first(File > Open Disk Image), and then using that mounted drive as the source for the restore. This fixed the issue seen when trying to restore directly from dmg per Apple’s instructions, which worked on unencrypted dmg files only. This is huge. I’m all set now to use this approach for large TM file storage (and other files). Now, I can encrypt the actual dmg files, and they will be protected locally on the device as well as out in the cloud. I like to think I also helped prove that your IFS works swimmingly, so hopefully that’s a silver lining for you given all the time you’ve spent helping me/us. Thank you! :slight_smile:

And now, back to our regularly scheduled program: please keep me apprised of the original issue in this thread, i.e. once you complete your tweaks of the 100 MB split for IFS on Amazon Drive. I look forward to trying it out, but for now, I’m able to use the 2 GB setting quite reliably, and can get everything I need. Thanks again

39

Appreciate the update. Glad things are working for you now!

Our IFS fix is still outstanding, but its on the list.

1 Like