So it seems odrive is getting caught up on this large database catalog. I am worried about turning the IFS function because I am not sure what it will do to file I store locally. Honestly I dont understand IFS too well.
I need to share files I upload and unsync so they have to be functional to people downloading them and I share multiple gb files to people often.
Will IFS make it so people have to download multiple files via odrive downloader and then it will reconstitute the original file?
I want to look around all the corners before I start using these other features. Main thing right now is to keep lightroom working as normal while it syncs with odrive.
Hi @conor,
If you are planning on sharing, it may be better to zip up the catalog before uploading and sharing. You can split the zip file into “bite-sized” chunks to facilitate easier uploading, since a 20GB, single file upload can be problematic.
IFS is cool, in that it handles the file splitting for you and makes the experience seamless, but you need an odrive client to download the file again. IFS stores the data in a split format on the remote storage, so it is not ideal for sharing with folks in an ad-hoc manner, and is more applicable for self-storage/archiving.
If you need some info on creating split zip archives, just let me know.
So I have let this run for awhile now, about two days and odrive is stuck on my lightroom catalog and will not sync anything past it.
Odrive client seems to sorta die as well - all of the finer integrations no longer work after it has run for awhile.
I have attached images of what I see on my end and hopefully, we can troubleshoot this together.
The IFS is a problem because I am hoping to use odrive as an easy way to share links to large files with zero footwork to the person on the other end. Just a link to download and done.
Hi @conor,
Can you send a diagnostic from the odrive menu? It is possible that an error is being hit for this large file.
Also, what do you see in the odrive menu for current status? Are things currently uploading? Are you noticing things restarting?
Since these are very large files, it can be challenging to upload them. It can also be challenging to download them because of the risk of a break in the download during transmission. You could zip these up with a split at around 500MB to 4GB, depending on transmission speeds you and your recipients have. You can then shared the folder and the recipients can download the zips and extract. The only problem is that it is not dead simple to extract since it can’t be done with the default Mac Archive Utility from Finder or the built-in unzip capabilities of Windows. The viability of this will depend on your audience.
odrive can become a little bit unresponsive when it is working on syncing large amounts of content (something we are actively working on improving in the next major release). During these times the right-click menus can be laggy or show no options available. Sometimes it can help to run some CLI status commands to get a view of what’s happening, vs the traditional UI.
To use the CLI commands from Mac:
Open a terminal session (type “terminal” in Spotlight search):
Copy and paste the following command into the terminal and hit enter: python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status
This will return a summary of odrive status. To get more detailed you can add parameters to the status command like this: python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --uploads python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --downloads python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --waiting python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --trash python $(ls -d "$HOME/.odrive/bin/"*/ | tail -1)odrive.py status --not_allowed
Also - this lightroom archive is actively in use, I am considering restructuring how I organize these. I could possibly keep the active catalog outside odrive but the RAW files inside it so they are backed up and secure. I can save backups of the catalog to odrive at regular intervals.
Hi @conor,
Okay. I wasn’t thinking this through properly before. I forgot that these lightroom projects were actually made up of a ridiculous number of files and folders.
Finder shows .lrdata it as a single “package” file, but it is actually a giant, nested folder structure. Currently odrive is tracking close to 60,000 folders, which accounts for the sluggish response from the menu.
This also will make it next to impossible to share with others over the web, as is, because the lrdata structure is so sprawling. Users would need to use the odrive client to access lrdata file in this structure, properly.
So here are the things we need to account for:
.lrdata “files” are actually folders that contain thousands of files and folders inside.
Sharing these is going to be difficult because of the structure. It will probably require you to zip up the lrdata package, upload that, and share it to your collaborators.
Actively using a lightroom project while it is being monitored/synced by odrive is going to cause some issues with data ever-changing, and files being held onto, preventing sync.
I think the main issue is the lrdata Preview folders, which contains lots and lots of data. The lrcat files are large, but single files and much easier to manage. I am not familiar with your full use case, so I don’t know if all of the data is required for your sharing. I am also, I must admit, not very familiar with lightroom. Is it possible to only share the lrcat files?
Here is what I propose:
Work on your lightroom catalogs outside of odrive, if possible so that things are not constantly changing underneath. Or, pause sync while you are actively working and then resume when done.
For sharing, you will most likely need to create an archive of the .lrdata folders, for users to download efficiently, unless the users are going to use the odrive desktop client to sync the contents down.
These lightroom projects are pretty brutal on sync, so you can expect some high CPU, longer processing, and a less-responsive interface while odrive is cranking through things.
This is working now, the lightroom catalogs are outside the folder and things are moving along again.
So how this works is I would definitely share single raw files that the catalog links to and stores previews of. Though I would never share the catalog so this works very well to operate this way.
A lot of photo DAM or databases work this way. Static links and all.
I am adapting to your system and i really enjoy it
