How do I Bulk Download Everything or Move/Migrate/Backup files from one storage provider to another?

I Have a Question
1

How do I move of migrate my file from one storage provider to another?

4 Likes
2

Depending on the amount of storage you have and the storage service being used, bulk downloading can be something that requires a fair bit of patience, diligence, and forethought. This guide will help you with the process.

NOTE: Currently odrive cannot move files between remote storage, directly, but it can still be used with great success to facilitate movement/migration.

SETUP:

  • Once you have logged in and linked your desired storage providers at https://www.odrive.com/ you can download and install the desktop client from here: Set Up Your odrive

  • If you already have the odrive desktop client installed, make sure you are running the latest version. To do this:
    1 . Exit odrive
    2. Download the latest version from here: Set Up Your odrive
    3. Install

  • To copy your files and/or folders from one cloud storage provider to another you must first download all of the files from the source. This can be a very lengthy operation, so patience will be needed.

    Important Note: During this process you should not add any new data or change any data on the storage you are downloading from.

DOWNLOAD:

  • To download everything in a chosen folder:

    • Right click on the source cloud folder and select Sync

    right_click_sync
    right_click_sync808×250 18.9 KB

    1. Slide the slider all the way to the right for, “Everything”
    2. Check “Include Subfolders”
    3. Click the silver sync button

    recursive_sync_win_new

  • The above action will tell odrive to download all of the items in the selected folder.

  • Keep an eye on the odrive “sync activity log” to watch the progress and verify the the final status of the sync operation (see screenshot below for where to find the sync activity monitor):

    sync activity log

Important Note:
Depending on factors such as the amount of data being downloaded, the storage service being downloaded from, network conditions, API rate limits, etc… odrive may not be able to download everything on the first pass, so more than one attempt may be needed.

You want to see a line like the following in the sync activity log:
INFO Right-click->sync action on [Chosen folder here] completed. Some refresh jobs may still be active/queued.

If you see either of the following messages:
Found incomplete folder downloads and scheduled refreshes for the the following locations
or
Error encountered during right-click->sync action

Then perform the same sync action again, after waiting at least two hours (to help avoid storage service API rate limits/issues)

Important Note:
There may be scenarios where the amount of data you need to migrate will not all fit on the available local storage you have, at one time. In these cases you can perform the download and upload in “stages”, where you download a section of the data, copy it over, and then unsync that completed section and then move on to the next section.
_
Alternatively, you could make use of external storage to expand your available local storage capacity, using the “sync to odrive” feature: Manage Sync
_
See this post for more details: Migration with limited local storage - #2 by Tony

  • When it looks like everything has been fully downloaded, verify the following:
    • odrive is idle
    • The last right-click sync operation completed successfully (See above Important Notes for how to verify this).
    • The sync activity log shows no new activity for at least 2 hours

  • As a final sync pass, trigger another full download with the following options.

    1. Slide the slider all the way to the right for, “Everything”
    2. Check “Include Subfolders”
    3. Check the “Force refresh during sync” checkbox
    4. Click the silver sync button

    recursive_sync_win_force_new

  • Like before, when the above action completes you will want to look at the odrive “sync activity log” to see what the final status of the sync operation was.

  • Once you see the sync completed message without either of the “Error” or “Found incomplete” messages in the sync activity log, we recommend you do a final “spot check” of your data, to make sure it looks the way you expect:

    • The downloaded content looks correct.
    • The number of files and folders looks correct.
    • The amount of data downloaded is consistent with the data in your remote storage.

COPY AND UPLOAD TO NEW STORAGE:

  • Once you are satisfied with the download, you can proceed with the copy operation to the new storage (if needed).

  • Copy the folder you fully downloaded to the new storage provider and wait for odrive to finish uploading the entire folder.

    You will know odrive is done when:
    * odrive is idle (not actively syncing)
    * There are no items listed in the “Waiting” or “Not allowed” queues in the odrive tray menu
    * There is a turquoise checkmark on the folder you are uploading

  • You can now optionally unsync (right click > Unsync) the source folder you had downloaded, to free-up local space.

  • For an example using the CLI/agent (for example, to use cloud infrastructure to facilitate a storage migration), take a look at this post here:
    How do I Bulk Download Everything or Move/Migrate/Backup files from one storage provider to another? - #9 by Tony

Tips and Recommendations:

  • Migrating data can be a very intense and time-consuming process that should never be taken lightly. Patience and diligence are going to be essential to a successful migration.
  • Be prepared for hiccups along the way and make sure to follow our recommended process above. Pay special attention to the “Important Note” sections.
  • Some services can have trouble with large scale operations, like the ones needed for a bulk migration. Pay attention to the “sync activity log” and make sure you end up with a “clean” sync run before you copy your data to the new service.
  • Once you have copied all of your data to the new service, we recommend keeping your data on the old service for at least 30 days. This is to make sure that you have a path for recovery, just in case anything was accidentally missed or excluded along the way.
1 Like
3

Can you guys create a function to move files between drives online?
If I have a TB of data to transfer that is going to kill my download for the month.

4

There is a feature request for this here: Moving, Copying and Auto-Syncing Files Between Services (Clouds)

5

One possible workaround for files not synced on your local machine is to use a cloud instance, which usually has screaming download and upload speeds (in the order of 100’s MB/s). In AWS, Google, Azure, etc. create a Windows instance, which you can access via RDP. Once spun up, you can install the desktop client, move files from storage provider A to storage provider B without burdening your machine or bandwidth. Once done make sure to shutdown and delete the instance, and you’re done.

For files that are locally synced, you will have to re-download them from the new provider to your machine.

In terms of specific Cloud instances:

  • I have found Google cloud to require the least upfront overhead, and it does offer a $300 starting credit, which I cannot imagine will not support a full transfer.
  • AWS is pretty powerful, but requires one to go through the full IAM exercise, including RSS keys and what not, so probably too much overhead if you are going to use it for transfers only.
3 Likes
6

I migrated from Amazon Cloud Drive (now branded as Amazon Photos) to Google Drive today. Even with relatively fast home internet bandwidth downloading and uploading 1TB of data is time consuming. Here is how I did it

  1. Spin up a bare metal Windows Server on your favourite hosting service. I used packet.net
  2. Login to Amazon Cloud, download and install their photos App.
  3. Use the app to restore all your Amazon cloud folders to the local drive.
  4. Login to Google Drive and upload your data
  5. Unsync your odrive folders from Amazon Drive
  6. Resync your odrive folders to Google Drive

This worked great for me and I migrated the data in hours instead of days at a cost of about $30.

2 Likes
8

These are some good pointers.

I am having trouble downloading all my files on to my limited-storage local drive to switch cloud providers.

If all my files were unsynced on my local machine, would I be able to sync all my files onto a windows machine in the cloud using the odrive client and switch cloud providers? Can the odrive employees or users provide support with step by step procedures?

Thanks.

9

Hi @digiform,
Yes you could do this. The procedure would be similar to what @paj posted above.

I’ve done this a few times, as well, and can provide the steps I took.

  • I’ve used a few different types of services, including Amazon Lightsail, Amazon EC2, and DigitalOcean. They are all going to be the same process, so it is just going to depend on your needs in terms of storage and pricing. The storage will be the most important, as you will want enough storage to perform the migration.

  • On the service you choose, spin-up a Linux instance. I usually go for the latest Ubuntu LTS instances they offer (probably 18.04 right now).

  • Install the Linux odrive sync agent (64 bit) and CLI: https://docs.odrive.com/docs/odrive-sync-agent

  • Setup odrive and create the mount: https://docs.odrive.com/docs/odrive-sync-agent#section-using-odrive-sync-agent

  • Sync (expand) the root-level folder for the storage you want to download. For example:

    "$HOME/.odrive-agent/bin/odrive" sync "$HOME/odrive-agent-mount/Dropbox.cloudf" This will expand the Dropbox folder in my odrive.

  • Recursively sync the expanded Dropbox folder, downloading everything in it until there are no more placeholder files. The command for this example would be:

    exec 6>&1;num_procs=4;output="go"; while [ "$output" ]; do output=$(find "$HOME/odrive-agent-mount/Dropbox/" -name "*.cloud*" -print0 | xargs -0 -n 1 -P $num_procs "$HOME/.odrive-agent/bin/odrive" sync | tee /dev/fd/6); done

    More details on what this command is doing here, if you are curious: Odrive Sync Agent: A CLI/scriptable interface for odrive's Progressive Sync Engine for Linux, OS X, and Windows

  • Once the download has completed, you will want to expand the destination storage folder to the appropriate location and then move the downloaded data from the source folder into the destination folder. For example:

    • "$HOME/.odrive-agent/bin/odrive" sync "$HOME/odrive-agent-mount/Google Drive.cloudf"

    • "$HOME/.odrive-agent/bin/odrive" sync "$HOME/odrive-agent-mount/Google Drive/Destination folder for migration.cloudf"

    • mv "$HOME/odrive-agent-mount/Dropbox/*" "$HOME/odrive-agent-mount/Google Drive/Destination folder for migration/"

  • Once the move is complete, perform a refresh command on that folder to kick-off an immediate odrive scan in that location, which will start the upload process. For example:

    "$HOME/.odrive-agent/bin/odrive" refresh "$HOME/odrive-agent-mount/Google Drive/Destination folder for migration"

  • At this point the data will be uploading to your new destination. You then need to decide what to do with the data in your old storage. I would suggest leaving it there until all of the data has been uploaded to the new location and you are satisfied with the migration. If you plan to delete the data, it will probably be more efficient to do so from the storage’s web client, directly, than through the odrive client’s “emptytrash” comand.

11

Thanks Tony.

  1. Could you provide the latest instructions for this from Amazon to OneDrive. (I am using encrypted storage from odrive.)
  2. Is it straightforward to purchase these services (Amazon Lightsail, Amazon EC2 or Digital Ocean)?
  3. Once we use these services like Amazon Lightsail, Amazon EC2 or Digital Ocean, do we delete these instances and is the risk for data leakage low? What will help reduce the risk?
  4. Why do you recommend using Linux? Are there advantages?
    Thanks for your help.
12

Hey @digiform,

  1. The instructions should be very similar, but you would need to account for Encryptor, which the CLI/Agent can handle via the relevant CLI command ( encpassphrase).

  2. I think that it is pretty straight forward. Spinning up Lightsail instances or Digitalocean Droplets is pretty painless.

  3. Correct, you can just terminate the instances. There shouldn’t be any risk of data leakage once the instance and any attached storage are terminated/deleted.

  4. Linux is generally cheaper, for one. It was also easier to lay out the step-by-step instructions above when put in terms of CLI commands. You could do this with Windows, as well, if you are more comfortable with that.

As you start going through the process, feel free to post any questions here and I can assist.

15

A post was split to a new topic: Migration with limited local storage