Support for 'S3-like' endpoints (compatible with S3 APIs)


#1

While I’m happy that Odrive supports S3, webdav, ftp, and others: it appears that the S3 integration is for AWS only, and not for non-Amazon managed S3 implementations. Other applications ( like Cyberduck and Arq for OSX) support “S3 Compatible” interfaces, where you can specify:

server/host address
port
access key
secret key

The only thing that ODrive would need to change or ammend would be the ability to specify server/host address (and possibly port #)

Is this something that has been considered?


S3 compatible service support
#2

Hi,
We’ve discussed it but we haven’t yet allocated any resources for further investigating and testing. If it is truely plug-and-play with our existing S3 implementation then it should be pretty straight forward, but experience has shown us that nothing is ever as simple as it seems.

Can you tell me what particular s3-like services you are looking to link?

Thanks!


#3

That was a fast reply!

@Tony : so, I’m in the process of validating various tools (both on the service side, as well as on the client side). On the service side, AWS-S3 has definitely been top of mind, but there is the possibility that we’ll want to use an on-premise S3-like provider (like Cloudian, for example).

In my poking around, I’ve found that most of the other client-side applications (such as cyberduck) don’t expose a ton of options which would be provider specific (besides port #, ability to toggle encryption on/off, and ability to disable multi-part). That said, its not always just plug-and-play, of course: if O-Drive were to provide ability to simply specify server:host (and perhaps have a couple mentioned toggles), and label the plugin as ‘S3-like’, it would give folks like myself the ability to try it out. If something specific didn’t work, it wouldn’t necessarily be O-Drive’s responsibility, since the provider of the storage service could also be contacted to improve their S3 support.


#4

I was surprised to discover it looks like you don’t have support for S3 compatible storage services (DreamHost, Wasabi, and many more). Especially due to the growth of S3 compatible private clouds using OpenStack, minio, Zenko, or other open source projects.

I see this requests for this go back to at least Feb’16, almost 2 1/2 years later and no s3 compatible support.

In theory it should be all of your same code for S3 support, configuration would just require the URL endpoint for the service.


#5

I will add there are many tools that support 3rd party S3 solutions through their S3 connector by simply allowing editing of the host URL. Goodsync, Cloudberry, S3Browser, and many more support this. Most s3 utilies will have a drop-down for S3 ‘account type’ with predefined providers and ‘S3 Compatible’ where you can put your own REST Endpoint.

Even Amazon’s own CLI tools work with any S3 compatible service !


#6

Hi @mark.ericson,
This isn’t supported yet, but I agree it would be nice to have. I have asked the product team to revisit this in the next few weeks to see what could be easily done here.


#7

I would love to be able to connect my Wasabi storage to Odrive. It is 100% bit-compatible with Amazon S3 API (uses same AWS API constrcuts for storage & identity management) *per their site. So I believe as someone else mentioned, it should simply be a matter of being able to specify the server host/end-point.


#8

Morgan, not exactly what I want, but I’m using ftp for one-way support of Wasabi, writing to Wasabi.


#9

Any updates on this?

It shouldn’t be very hard to introduce a new “standard protocol” storage type for S3. The code only has to change to support a user-provided url from the configuration.


#10

Hi @mark.ericson,
We haven’t been able to get this into development amidst all of the other priorities, but I am hopeful it is not too far off. I’ll definitely update here when I have some news.


#11

The feature will significantly increase the potential market base for odrive as there are literally dozens of storage services with S3 compatible APIs. Everything built on OpenStack is S3 compatible! And Wasabi which is S3 compatible just got another round of funding!