Concurrency for small files in linux not works well

Troubleshooting
1

My environment is
Ubuntu Xenial 16.04.1
The odriveagent I’m using is
d5d911ea950c98c32e9c2f92f07e68cc odriveagent-linux-64bit-916_64bit.tar.gz
The cloud storage I’m using is google drive.

So, my scenario is to mount a cryfs [1] to one folder under google drive folder in odrive mount point.
Then cryfs will generate a lot of 32kb files scattered in lot of folders.
ex. http://paste.ubuntu.com/23861300/

And from $ odrive.py status --uploads shows it actually have concurrency but seems only tackle one folder one time, say if a folder have 9 files, then 9 process shows in status, but if another folder has only 2 files, then it shows only 2 currency process. So, overall the upload speed in my case is about 1~8 kb/s.

So, I guess it would be better to also implement concurrency on multiple folders.

BTW, because I can not find the packed debian package for odrive, so I plan to pack one so that user could easily install and run the odrive agent automatically by just installing package. But I’m confused about what the version of odriver agent is ? is the version just be 916 in the tarball name string?

[1] https://www.cryfs.org/

2

Hi @daidailin003,
Currently concurrency can be limited, as you have seen. For small files scattered throughout lots and lots of folders this can result in slower overall upload speeds. This is something we are planning on improving in the future.

I appreciate the offer for creating a deb package, but the agent app may be changing significantly enough soon that it is probably better to wait.

3

ok, got it, and thanks.

1 Like