After running the odriveagent for ~12 hours, I begin getting sync errors. All of the items that I try to sync return “Busy - try again later”. I can restart the agent and it works again but is there a reason this is happening? It’s annoying to run a batch process overnight and come back with not everything synced.
Thanks!
Hi @jedhassell,
I haven’t seen this behavior before. I have a few agents running on Linux boxes for several months without interruption and they perform as expected. I don’t think I have one running on a Pi at this time, though.
Can you describe what you have the agent doing? Is it after a sustained period of activity or when idle, or does it matter?
Do any commands work when this happens, if so, which commands are showing the issue?
Hi @Tony,
The odriveagent uses a ton of CPU while idle (see screenshot).
When I do recursive sync calls like
exec 6>&1;num_procs=2;output="go"; while [ "$output" ]; do output=$(find "/media/4tb/odrive" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs odrive sync | tee /dev/fd/6); done
it works for a few hours and then begins with the Busy - try again later errors until I restart odriveagent. I have only tried the sync command. Which other commands would you like me to try?
Thanks,
Jed
Hi Jed,
There are several status command that can be run that may give us some information on what is happening. Take a look at this post here.
Once you get this busy error I would be curious to know what is reported (or not reported) from the status commands.
I’m also surprised to see three odriveagent procs. I would expect only two. That is something else to take a look at. If there is a rogue instance running that could cause some issues with conflicting behavior between the processes.
Hi Tony,
I’ll try to reproduce it tonight. htop shows 3 for some reason but ps only shows 2. Not sure how htop behaves but I’ve seen similar things with other processes.
I’m now getting the errors after running for ~12 hours.
For every file I see this error:
Unable to sync temp.cloudf. Busy - try again later
status:
odrive Make Cloud Storage THE WAY IT SHOULD BE.
isActivated: True hasSession: True
email: jedhassell@gmail.com accountType: CloudDrive
syncEnabled: True version: prod 1
placeholderThreshold: neverDownload autoUnsyncThreshold: never
downloadThrottlingThreshold: unlimited uploadThrottlingThreshold: normal
autoTrashThreshold: never Mounts: 1
xlThreshold: never Backups: 0
Sync Requests: 0
Background Requests: 0
Uploads: 0
Downloads: 0
Trash: 3
Waiting: 0
Not Allowed: 0
other status:
pi@raspberrypi:~$ odrive status --uploads
No uploads.
pi@raspberrypi:~$ odrive status --downloads
No downloads.
pi@raspberrypi:~$ odrive status --waiting
No waiting items.
pi@raspberrypi:~$ odrive status --not_allowed
No not allowed items.
Anything I missed?
Thanks,
Jed
Thanks @jedhassell.
It looks like odrive thinks there is something already processing on the path hierarchy that is being requested and blocking the request, waiting for completion. The status commands don’t show anything processing, though…
There hasn’t been exhaustive testing on RPi devices, and its possible the agent is hitting some sort of internal resource limit after an extended period of time that is creating a lock condition. I will continue to investigate, but for now it may be best to restart the agent every 12 hours or so to clear things out (using a cronjob, for example).
Thanks @Tony. Does restarting during transfers mess anything up or will it retry and overwrite successfully?
Hi @jedhassell,
It will restart the transfer. Partial transfers should not actually ever make it up since it is never completed before interruption.
Since it looks like it may at least be completing transfers before it gets into this state, you could also gate the restart on odriveagent being idle (no activity), if you wanted to get a little more fancy.
Thank you @Tony.
It seems odriveagent is constantly at high cpu while idle. Do you know the reason for this?
Hi @jedhassell,
Since odrive is a bidirectional sync engine that tries to achieve near real-time remote<->local consistency, it ends up doing a fair amount of work while not actively uploading and download stuff. It has to make sure that everything stays in sync, at all times.
This means the more stuff you have that odrive needs to worry about (basically the number of files and folders), the more work it has to do, in general. Since the Pis are not powerhouse boxes, this background “idle” activity could manifest as high CPU if there is a fair amount of data to monitor.
If you don’t require that type of “vigilant” behavior, you can shutdown the agent until you need it, in which case odrive scans for changes on startup.
You can also remove items from scope by unlinking storage or unsync certain folders, which can reduce overhead.