Amazon S3: Dos and Don'ts

Hi there,

First time poster, long-time user of Odrive. :slight_smile:

I recently decided to try using the AWS S3 service with Odrive, as I am a big of S3. I learned a couple of things that I wanted to share. (This MAY be Mac OS/X specific, I warned ya!)

DO: Make sure you have a security policy set up in IAM. I would recommend creating a brand new user account in IAM so that it has the minimum privileges necessary (good system admin practice). Once the account is created, attach the following IAM policy to it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::yourusername-odrive"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::yourusername-odrive/*"
            ]
        }
    ]
}

That should allow that account to access Odrive.

DON’T: try to sync tens of thousands of files to Amazon S3.

I learned this one the hard way. I do some software development, and have a few Node.js projects that, when all of their modules (and sub modules) are added up, came out to about 40,000 files across 900 Megs of data. It turns out that S3 isn’t the best when it comes to uploading many small files. I ended up having to let Odrive run overnight to sync up all those files, then the following evening let it run another night to unsync those files.

SUGGESTION: Use something like Duplicity to tar up your entire development directory into just a few large files, and write a script that does that and puts those files into the Odrive directory that gets synced to S3. You’ll thank me later. :slight_smile:

To get to this point, it took me a few evenings, so I thought I would share what I learned, and hopefully it will save anyone else wanting to use S3 with Odrive some frustration.

If anyone has questions about S3, feel free to hit me up in the comments, I’m a huge fan of AWS and its services.

– Doug

1 Like

Thanks for the feedback @dmuth!

Coincidentally, I am just writing up an S3 how-to:

Syncing 40k files will probably take a while, just because there are so many individual API calls. Unsyncing should be pretty quick though. Did you find that the unsync op took a really long time?

1 Like

Yeah, the unsync seemed to (emphasis there) take a long time, so I left it running overnight. It’s hard to say, since (as I mentioned elsewhere), there isn’t any detailed logging that is exposed.

If I ever wind up in this situation again, I could see about sending deletion events off to an SQS queue for monitoring or something…

Hi @dmuth,
The right-click->unsync is really just a local action, which uncaches the local data and leaves it in the cloud. It would be unexpected for it to take so long, so just let me know if you see that behavior again.

Thanks!

1 Like

A post was merged into an existing topic: Wasabi policy requirements