How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.
-
So, you have a bunch of cloud storage accounts? and you have a Local network with several computers? and you want to sync it all to your Linux Based NAS? And you want local changes to be synched up to the cloud? But you don't want to have all of the sync apps for all of those providers installed, not to mention the hassle of having different accounts for the same provider or a provider that doesn't have a client for your target OS (i'm looking at you google drive with your lack of linux client)?
Well, do I have the solution for you?
...
Well? Do I?
...
what?
OH! I'm writing this so I'm supposed to answer my own question.
...
Of course I do!
Step 1: Sign up for Odrive
This is the provider i'm using to handle all this. I'm paying for pro version, but everything i'm writing about works on the free version too.
It's pretty cool actually, absolutely give them a look, even if you're not interested in setting up the setup i have
Step 2: Link your Cloud Storage Spaces
Odrive handles quite a few different providers including instagram, gmail (for downloading contacts) onedrive, google drive, box, drop box and more. I'm a little disappoitned in the lack of support for Mega, but whatever, i don't actually use mega for anything other than downloading backups of our minecraft server that the server owner uploads to Mega periodically, so no big loss for me there.
After linking all my storage (At least my primary ones. I'll add the others later) my storage spaces look like this:
Step 3: Install the Odrive Sync Agent
We'll be installing the Command line sync agent for this as it supports mounting your storage to a network drive (something not supported by the desktop agent) and it's scriptable (something we'll want to be doing to make this maintenance free).
You can find instructions on how to set this up in the Odrive docs and the instructions are pretty awesome if i do say so myself.
Step 4: Make the agent start at boot.
For linux this is easy, crontab to the rescue (adjust the command line to your particular tastes):
# Elided the rest of my crontab... @reboot nohup /home/accalia/.odrive-agent/bin/odriveagent &>/dev/null &
i mean, you could use systemd if you really wanted to, but why would you want that?
For windows, Task Scheduler is your friend here. I'm running my agent on windows because i already had an always on auto-login windows server and i'm fresh out of rpi's
My configuration was to set the task to trigger on login (because i already autologin and tasks that trigger before login get weird sometimes), and start the program
C:\Users\accal\.odrive-agent\bin\odriveagent.exe
with a working directory ofC:\Users\accal\.odrive-agent\bin
Step 5: Mount your storage spaces.
For this mount i chose
L:\Cloud
. It's a NFS mount to my linux based NAS that the desktop client wouldn't touch but the CLI agent is A-OK with. Obviously substitute the path for wherever you want to put it. Annoyingly it will refuse to sync to the root of a drive so i couldn't just have it mount toL:\
which happens to be a mount for192.168.1.1:Cloud
in NFS speak or//192.168.1.1/Cloud
in UNC path.Where was I?
Oh. yes.
For Linux the command is simple:
$> ~/.odrive-agent/bin/odrive.py mount /path/to/mount /remote/mount/path
For Windows I used this command:
PS > C:\Users\accal\.odrive-agent\bin\odrive.exe mount L:/Cloud /
Step 6: Write a Sync Script
so, now you are syncing your cloud storage. you're all done now right? WRONG!
Odrive doesn't sync all your data by default because when you have 90GB of combined storage space that's a lot to drop on a HDD. instead it downloads placeholder files
*.cloud
and placeholder "folders"*.cloudf
We want to download EVERYTHING, because we want all our data.
Python to the rescue (remember to install Python on the windows PC if you're using this on a windows server like I am.)
Script for Linux:
#!/usr/bin/env python from subprocess import call import os from os import walk from os.path import join, dirname, abspath, exists from sys import argv, exit lockfile = join(dirname(abspath(__file__)), 'sync.lock') try: import fcntl fp = open(lockfile, 'w') fp.flush() try: fcntl.lockf(fp, fcntl.LOCK_EX | fcntl.LOCK_NB) except IOError: print ('Another instace of sync is already running, aborting.') exit(0) raise def sync(path): call(['/home/accalia/.odrive-agent/bin/odrive', 'sync', path]) print(argv[1]) for root, dirs, files in walk(argv[1]): for file in files: if file.endswith('.cloudf'): sync(join(root,file)) files.remove(file) dirs.append(file[:-7]) if file.endswith('.cloud'): sync(join(root,file))
Script for Windows:
#!/usr/bin/env python from subprocess import call import os from os import walk from os.path import join, dirname, abspath, exists from sys import argv, exit, exc_info lockfile = join(dirname(abspath(__file__)), 'sync.lock') try: # file already exists, we try to remove (in case previous # execution was interrupted) if exists(lockfile): os.unlink(lockfile) os.open(lockfile, os.O_CREAT | os.O_EXCL | os.O_RDWR) except OSError: type, e, tb = exc_info() if e.errno == 13: print ('Another instace of sync is already running, aborting.') exit(0) raise def sync(path): call(['C:\\Users\\accal\\.odrive-agent\\bin\\odrive', 'sync', path]) print(argv[1]) for root, dirs, files in walk(argv[1]): for file in files: if file.endswith('.cloudf'): sync(join(root,file)) files.remove(file) dirs.append(file[:-7]) if file.endswith('.cloud'): sync(join(root,file))
Obviously i hardcoded the paths top the odrive executable. Naughty fox! go ahead and update those paths to the ones you need.
Step 7: Run that sync job on a schedule!
I chose every five minutes. Every five minutes new files and folders will be downloaded. You want this at fast as you are comfortable with to keep absolutely up to date, and don't worry about running multiple instances. that's why there are windows and mac/linux versions of the script above. they detect when another instance is already running and abort if it's still trying to finish a prior sync.
For linux this is again a no brainer:
# hey! this is my crontab again! */5 * * * * python /home/accalia/.odrive-agent/bin/sync.py /path/to/mount
For Windows, another Scheduled task.
This one runs at logon, and every 5 minutes thereafter, indefinitely.
the executable it runs is
C:\Users\accal\.odrive-agent\bin\sync.py
with working directoryC:\Users\accal\.odrive-agent\bin
and arguments ofL:\Cloud
Huzzar! we're syncing!
now just a liiiiitle bit of housekeeping.
Step 8: Run Refresh and Empty Trash on a schedule.
We want to periodically refresh our odrive folders to pick up changes that might have missed the SyncAgent's attention and to pick up new storage spaces as those don't always sync automatically in a timely manner.
The command for that is simple,
*/30 * * * * python /home/accalia/.odrive-agent/bin/odrive.py refresh /path/to/mount
And similarly in task scheduler, though i'll let you figure out the exact configuration by now, we've done it twice together after all.
Got it? Good!
Now we want to empty the trash regularly too, so files we delete locally get removed from cloud storage too.
0 * * * * python /home/accalia/.odrive-agent/bin/odrive.py emptytrash /path/to/mount
Step 9: ???
???
????????????????!
Step 10: Share your results with friends and enemies online!
Which is what you've been reading!
I hope it helps!
-
Alternative solution: Just use one service
(says the hedgie with two OneDrives and one Google Drive)
-
@RaceProUK said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:
Just use one service
hard to do when School requires you to use their gdrive instance, and work pfovides you with another and you already had your own personal, and you need access to all three because shared computer.
@RaceProUK said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:
(says the hedgie with two OneDrives and one Google Drive)
see?
-
Redirecting errors to /dev/null is TRWTF
-
@JazzyJosh said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:
Redirecting errors to /dev/null is TRWTF
that's why i'm only redirecting stderr. odrive logs errors to
(odriveagent binary directory)/../logs/error.log
:-P
-
@accalia To be fair, the GDrive doesn't really get used, and I don't mind having my personal and work OneDrives separate. Actually, considering the amount of not-quite-legally-procured comic book scans on there, it's probably better that way
-
@accalia
Hello Mr. or Mrs. Accalia,
I have followed your 10 steps but Cloud Storage is still a thing that exists.
This tutorial solves nothing!
1/10 (1 effort point!)
-
@Kuro said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:
I have followed your 10 steps but Cloud Storage is still a thing that exists.
hmm.... how unusual. Are you certain that you performed step 9 correctly?
-
@accalia But can it be done without a live chicken and a rabbi?