Hi @D_Sec,
A refresh on the folder containing any local changes that you want to send to the cloud should trigger an immediate scan for odrive in that location (unless odrive is busy doing something else in another location), which should then send any “dirty” files to the remote storage.
So you are adding or changing files in a folder that is mounted against Google Cloud Storage and you are not seeing the files upload, even after issuing a refresh command? Can you tell me what the path of the file is?
Backup and stream are alpha-state features that are more there for experimentation. Backup will create a one-way sync (local to remote) relationship with the specified local and remote folders. Stream was an attempt to allow streaming to an application as the file is being downloaded.
Refresh will tell odrive to scan the specified location (both locally and remotely) for changes.
After some tests here’s the behaviour:
If I add files to comp1:
C:\Users\Me\My Pictures\Xyz\DIR2
then issue a refresh in powershell, it says: Synced and list all the folders
Now, on comp2:
I issue a refresh, no matter how many times and how long I wait, no .cloud files are created BUT if I unmount, then mount again, files will appear (as .cloud) in C:\Data\Pictures\Xyz\DIR2
So on “Google Cloud Storage” refresh seems to work from local to storage but not from storage to local.
Hi @D_Sec,
Yes this is odd. What if you add a file on Computer 1, refresh to sync, then add a file in the corresponding location on Computer 2 and refresh. Does that file on Computer 2 upload and then is the file from Computer 1 shown?
What if you add a file from the odrive web client and then refresh on Computer 2?
Is the behavior the same in the opposite direction (Computer 2 -> Computer 1)?
I’m just trying to figure out where the root cause lies here.
Can you also clarify the configuration on both systems for me? You are using the odrive agent on both Windows machines, or are you using the desktop client on one Windows machine and the agent on another?
Your first question sounds kinda strange to me, how can I verify that something actually happened?
(only thing that could be verified is that file from Computer 1 is actually well synced to cloud).
Refresh after adding a file from odrive web client generate the same problem. No .cloud file is downloaded.
And yes, just tested the other way around (not working either).
Both system are Windows7 with only odrive agent installed (so without the normal client).
NB. I don’t know if it matter, but while testing I’ve created a multi-regional bucket.
Hi @D_Sec,
Sorry that the first question was confusing. I was wondering if an upload on Computer 2 would trigger reflection of the file from Computer 1. When odrive syncs it needs to “come current” before it uploads, so I would expect it to see the new file from Computer 1 when it is trying to upload the file on Computer 2.
I can’t reproduce the problem with the agent, so I’m not sure what is happening.
Are you able to install the desktop client as opposed to the agent? The CLI can be used for both, so you can still use the CLI to control the desktop client, but we can at least get some better diagnostics from the desktop client if it exhibits the same behavior.
Are you also testing on “Google Cloud Storage” bucket?
Because I suspect more a behavior like you explained in this post (i.e. refresh asking for change instead of full listing):
I will unlink my Google Drive from account and see if it changes something… If not working, I’ll try full client with CLI.
With odrive client installed (and odriveagent not running), I just sent those commands in CLI:
authenticate
status --mounts (only default $HOME\odrive active)
mount localpath "/Google Cloud Storage"
status --mounts (ok localpath is added)
refresh localpath (was already busy syncing… but it was the observed behaviour on mount or re-mount)
When sync was completed, I changed, some files on the Computer 1, and without having to re-issue any command the changes were reflected on Computer 2 (files were correctly deleted and new ones appeared as .cloud)
But the problem is that odrive full client need a user logged in (and sync automatically periodically).
The agent (in theory) was perfect for intended use. When computer starts agent starts (even without users). Then some scheduled tasks instruct him what to sync and when.
Hi @D_Sec,
When you switch back to the agent, are you still seeing the placeholder reflection issue?
If so:
Is this running as a normal user under a user login, or is it running as a service?
If you run the agent in the foreground, instead of as a background task, do you see anything output to standard out? I’m wondering if there is an error being hit on the attempt to write out the placeholder files.
Yup, if I switch back to agent, cloud storage to local isn’t working anymore even on refresh.
During test period it’s running under a domain admin user which have of course (direct) write access on any location used with odrive (so not a SYSTEM account).
How am I supposed to run agent in the foreground? even if I start it manually from a command prompt it’s forking in the background.
NB. When I killed process tree odriveagent, to restart it from command line, he picked up the changes. Anyway after the fresh start, he continues to behave just like the one I start during boot (if I add new files, they are not transferred on local even with refresh).
So, so far, only way to pick up new files from storage are:
Hi @D_Sec,
Apologies for the late response on this.
I’ve been trying to reproduce what you are seeing without any success. I’ve tested the Windows agent against Google Cloud Storage using both simple delimited and enhanced and a refresh always picks up new changes. One thing I want to mention is that I noticed that the output of the refresh command doesn’t actually list the new items, but they will show up when you issue an ls or dir after the refresh.
I’m testing as my logged-in, local user on a Windows 10 system under powershell.
As for running in the foreground, I forgot that the agent is set to run in the background by default. You can redirect stdout and stderr to a log file, in case something is spit out with a command like this: & "$HOME\.odrive-agent\bin\odriveagent.exe" 2>&1 > D:\odriveagent-log.txt
There are also logs in the $HOME.odrive-agent\log folders, although they are not very verbose. Do you see anything in those logs?
Is the behavior the same if you run as a local machine user?
I’m giving up tests with Google Cloud Storage, I absolutely need to move on now.
On my side, nothing was showing up after a dir…
Will run odriveagent with the redirect to a log file from now on (just in case).
As you said nothing interesting in the default log files agent.log and backup.log, and nothing at all in error.log.
Will keep you posted, have upgraded my subscription, took 1Tb on Google Drive and will run it on production server with dedicated user. Now initial sync just needs to replicate a few hundreds Gb first…
I’m not sure what could be happening here. I tried this on Windows 7 and Windows 2012R2 and still wasn’t able to reproduce it. Let me know if you see any issues when using Google Drive.
Funnily enough now it seems that it’s the opposite behaviour
When I start the agent and authenticate:
mount is directly there (no need to mount /localpath/ /cloudstorage/)
on refresh it says that it’s active (yet to be confirmed that it’s syncing without issuing any command)
Anyway looks that it works fine, so that’s the most important.
Thanks for your help and debugging all along!
[EDIT] Just maybe one last question, as now odriveagent will be running all the time in background.
Is there any other way than unmount to pause replication during the day?
In reference to your statement:
“
A refresh on the folder containing any local changes that you want to send to the cloud should trigger an immediate scan for odrive in that location (unless odrive is busy doing something else in another location), which should then send any “dirty” files to the remote storage.
”
Does this mean refresh can be used to delete files from a remote? or are “changed” files only files that exist and are modified? but I can’t sync a directory then sync all the files in it, then move those files away leaving the directory empty and then refresh that folder manually to delete it from the cloud? (I’d actually want this to happen, but it doubt it works this way? I assume refresh works about the same as killing the agent and restarting odrive? or rebooting and restarting odrive right?) not this alternate bizarre function I just made up? (I’ve never run a refresh command manually, and given what I just said, I’m slightly afraid to).
I didn’t see your edit here. You can use the shutdown command to stop the agent, and then launch the agent again when you want to resume. Is that what you are looking for?
Hi @bouya.daman,
A refresh is really just a manual way to tell odrive to look at the current folder and make sure the sync engine is up-to-date. For some integrations and configs odrive cannot see change events, so a change will not be picked up until odrive gets around to doing its periodic walk-and-check. A refresh will shortcut this on the target folder.
Deletes will be picked up in this case, and any local files that have been deleted in that folder will be picked-up and those delete actions will be held in the odrive trash until the user empties the odrive trash, which will sync those deletes to the remote storage.
