How do I Move or Migrate files from one storage provider to another?

How do I move of migrate my file from one storage provider to another?

3 Likes

Currently odrive cannot move files between remote storage, directly, but it can still facilitate movement/migration with the following steps:

  • Once you have logged in and linked your desired storage providers at https://www.odrive.com/ you can download and install the desktop client from here: Set Up Your odrive

  • If you already have the odrive desktop client installed, make sure you are running the latest version. To do this:
    1 . Exit odrive
    2. Download the latest version from here: Set Up Your odrive
    3. Install

  • To copy your files and/or folders from one cloud storage provider to another you must first download all of the files from the source. This can be a very lengthy operation, so patience will be needed.
    Important Note: During this process you should not add any new data or change any data on the storage you are downloading from.

  • To download everything in a chosen folder:

    • Right click on the source cloud folder and select Sync

      1. Slide the slider all the way to the right for, “Everything”
      2. Check “Include Subfolders”
      3. Click the silver sync button
        recursive_sync_win_new
  • The above action will tell odrive to download all of the items in the selected folder.
    Important Note: Depending on factors such as the amount of data being download, the storage service being downloaded from, network conditions, API rate limits, etc… odrive may not be able to download everything on the first pass, so more than one attempt may be needed.

  • When it looks like everything has been fully downloaded from the source, we recommend the following additional actions, to double-check:

    • Trigger another full download, but this time also check the “Force refresh” option (if available).
      1. Slide the slider all the way to the right for, “Everything”
      2. Check “Include Subfolders”
      3. Check the “Force refresh during sync” checkbox (Only available on Windows right now)
      4. Click the silver sync button
        recursive_sync_win_force_new
  • When the above action completes you will want to look at the odrive “sync activity log” to see what the final status of the sync operation was:
    sync activity log

    You should see a line like the following:
    INFO Right-click->sync action on [Chosen folder here] completed. Some refresh jobs may still be active/queued.

    Important Note: If you see either of the following messages, you will want to perform this action again, after waiting at least two hours (to help avoid storage service API rate limits):
    Found incomplete folder downloads and scheduled refreshes for the the following locations
    or
    Error encountered during right-click->sync action

  • Once you see the sync completed message without either of the “Error” or “Found incomplete” messages in the sync activity log you can proceed with the copy operation.

  • Copy the folder you fully downloaded to the new storage provider and wait for odrive to finish uploading the entire folder.

    You will know odrive is done when:
    * odrive is idle (not actively syncing)
    * There are no items listed in the “Waiting” or “Not allowed” queues in the odrive tray menu
    * There is a turquoise checkmark on the folder you are uploading

  • You can now optionally unsync (right click > Unsync) the source folder you had downloaded, to free-up local space.

  • For an example using the CLI/agent (for example, to use cloud infrastructure to facilitate a storage migration), take a look at this post here:
    How do I Move or Migrate files from one storage provider to another? - #9 by Tony

1 Like

Can you guys create a function to move files between drives online?
If I have a TB of data to transfer that is going to kill my download for the month.

There is a feature request for this here: Moving, Copying and Auto-Syncing Files Between Services (Clouds)

One possible workaround for files not synced on your local machine is to use a cloud instance, which usually has screaming download and upload speeds (in the order of 100’s MB/s). In AWS, Google, Azure, etc. create a Windows instance, which you can access via RDP. Once spun up, you can install the desktop client, move files from storage provider A to storage provider B without burdening your machine or bandwidth. Once done make sure to shutdown and delete the instance, and you’re done.

For files that are locally synced, you will have to re-download them from the new provider to your machine.

In terms of specific Cloud instances:

  • I have found Google cloud to require the least upfront overhead, and it does offer a $300 starting credit, which I cannot imagine will not support a full transfer.
  • AWS is pretty powerful, but requires one to go through the full IAM exercise, including RSS keys and what not, so probably too much overhead if you are going to use it for transfers only.
3 Likes

I migrated from Amazon Cloud Drive (now branded as Amazon Photos) to Google Drive today. Even with relatively fast home internet bandwidth downloading and uploading 1TB of data is time consuming. Here is how I did it

  1. Spin up a bare metal Windows Server on your favourite hosting service. I used packet.net
  2. Login to Amazon Cloud, download and install their photos App.
  3. Use the app to restore all your Amazon cloud folders to the local drive.
  4. Login to Google Drive and upload your data
  5. Unsync your odrive folders from Amazon Drive
  6. Resync your odrive folders to Google Drive

This worked great for me and I migrated the data in hours instead of days at a cost of about $30.

2 Likes

These are some good pointers.

I am having trouble downloading all my files on to my limited-storage local drive to switch cloud providers.

If all my files were unsynced on my local machine, would I be able to sync all my files onto a windows machine in the cloud using the odrive client and switch cloud providers? Can the odrive employees or users provide support with step by step procedures?

Thanks.

Hi @digiform,
Yes you could do this. The procedure would be similar to what @paj posted above.

I’ve done this a few times, as well, and can provide the steps I took.

  • I’ve used a few different types of services, including Amazon Lightsail, Amazon EC2, and DigitalOcean. They are all going to be the same process, so it is just going to depend on your needs in terms of storage and pricing. The storage will be the most important, as you will want enough storage to perform the migration.

  • On the service you choose, spin-up a Linux instance. I usually go for the latest Ubuntu LTS instances they offer (probably 18.04 right now).

  • Install the Linux odrive sync agent (64 bit) and CLI: https://docs.odrive.com/docs/odrive-sync-agent

  • Setup odrive and create the mount: https://docs.odrive.com/docs/odrive-sync-agent#section-using-odrive-sync-agent

  • Sync (expand) the root-level folder for the storage you want to download. For example:

    "$HOME/.odrive-agent/bin/odrive" sync "$HOME/odrive-agent-mount/Dropbox.cloudf" This will expand the Dropbox folder in my odrive.

  • Recursively sync the expanded Dropbox folder, downloading everything in it until there are no more placeholder files. The command for this example would be:

    exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive" sync | tee /dev/fd/6); done

    More details on what this command is doing here, if you are curious: Odrive Sync Agent: A CLI/scriptable interface for odrive's Progressive Sync Engine for Linux, OS X, and Windows

  • Once the download has completed, you will want to expand the destination storage folder to the appropriate location and then move the downloaded data from the source folder into the destination folder. For example:

    • "$HOME/.odrive-agent/bin/odrive" sync "$HOME/odrive-agent-mount/Google Drive.cloudf"

    • "$HOME/.odrive-agent/bin/odrive" sync "$HOME/odrive-agent-mount/Google Drive/Destination folder for migration.cloudf"

    • mv "$HOME/odrive-agent-mount/Dropbox/*" "$HOME/odrive-agent-mount/Google Drive/Destination folder for migration/"

  • Once the move is complete, perform a refresh command on that folder to kick-off an immediate odrive scan in that location, which will start the upload process. For example:

    "$HOME/.odrive-agent/bin/odrive" refresh "$HOME/odrive-agent-mount/Google Drive/Destination folder for migration"

  • At this point the data will be uploading to your new destination. You then need to decide what to do with the data in your old storage. I would suggest leaving it there until all of the data has been uploaded to the new location and you are satisfied with the migration. If you plan to delete the data, it will probably be more efficient to do so from the storage’s web client, directly, than through the odrive client’s “emptytrash” comand.

Thanks Tony.

  1. Could you provide the latest instructions for this from Amazon to OneDrive. (I am using encrypted storage from odrive.)
  2. Is it straightforward to purchase these services (Amazon Lightsail, Amazon EC2 or Digital Ocean)?
  3. Once we use these services like Amazon Lightsail, Amazon EC2 or Digital Ocean, do we delete these instances and is the risk for data leakage low? What will help reduce the risk?
  4. Why do you recommend using Linux? Are there advantages?
    Thanks for your help.

Hey @digiform,

  1. The instructions should be very similar, but you would need to account for Encryptor, which the CLI/Agent can handle via the relevant CLI command ( encpassphrase).

  2. I think that it is pretty straight forward. Spinning up Lightsail instances or Digitalocean Droplets is pretty painless.

  3. Correct, you can just terminate the instances. There shouldn’t be any risk of data leakage once the instance and any attached storage are terminated/deleted.

  4. Linux is generally cheaper, for one. It was also easier to lay out the step-by-step instructions above when put in terms of CLI commands. You could do this with Windows, as well, if you are more comfortable with that.

As you start going through the process, feel free to post any questions here and I can assist.