How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.


  • SockDev

    So, you have a bunch of cloud storage accounts? and you have a Local network with several computers? and you want to sync it all to your Linux Based NAS? And you want local changes to be synched up to the cloud? But you don't want to have all of the sync apps for all of those providers installed, not to mention the hassle of having different accounts for the same provider or a provider that doesn't have a client for your target OS (i'm looking at you google drive with your lack of linux client)?

    Well, do I have the solution for you?

    ...

    Well? Do I?

    ...

    what?

    OH! I'm writing this so I'm supposed to answer my own question.

    ...

    Of course I do!

    Step 1: Sign up for Odrive

    This is the provider i'm using to handle all this. I'm paying for pro version, but everything i'm writing about works on the free version too.

    It's pretty cool actually, absolutely give them a look, even if you're not interested in setting up the setup i have

    Step 2: Link your Cloud Storage Spaces

    Odrive handles quite a few different providers including instagram, gmail (for downloading contacts) onedrive, google drive, box, drop box and more. I'm a little disappoitned in the lack of support for Mega, but whatever, i don't actually use mega for anything other than downloading backups of our minecraft server that the server owner uploads to Mega periodically, so no big loss for me there.

    After linking all my storage (At least my primary ones. I'll add the others later) my storage spaces look like this:
    0_1479838252639_upload-8a6f5d8c-5420-42e2-8496-92b0411fa0d0

    Step 3: Install the Odrive Sync Agent

    We'll be installing the Command line sync agent for this as it supports mounting your storage to a network drive (something not supported by the desktop agent) and it's scriptable (something we'll want to be doing to make this maintenance free).

    You can find instructions on how to set this up in the Odrive docs and the instructions are pretty awesome if i do say so myself.

    Step 4: Make the agent start at boot.

    For linux this is easy, crontab to the rescue (adjust the command line to your particular tastes):

    # Elided the rest of my crontab...
    @reboot nohup /home/accalia/.odrive-agent/bin/odriveagent &>/dev/null &
    

    i mean, you could use systemd if you really wanted to, but why would you want that?

    For windows, Task Scheduler is your friend here. I'm running my agent on windows because i already had an always on auto-login windows server and i'm fresh out of rpi's

    My configuration was to set the task to trigger on login (because i already autologin and tasks that trigger before login get weird sometimes), and start the program C:\Users\accal\.odrive-agent\bin\odriveagent.exe with a working directory of C:\Users\accal\.odrive-agent\bin

    Step 5: Mount your storage spaces.

    For this mount i chose L:\Cloud. It's a NFS mount to my linux based NAS that the desktop client wouldn't touch but the CLI agent is A-OK with. Obviously substitute the path for wherever you want to put it. Annoyingly it will refuse to sync to the root of a drive so i couldn't just have it mount to L:\ which happens to be a mount for 192.168.1.1:Cloud in NFS speak or //192.168.1.1/Cloud in UNC path.

    Where was I?

    Oh. yes.

    For Linux the command is simple:

    $> ~/.odrive-agent/bin/odrive.py mount /path/to/mount /remote/mount/path
    

    For Windows I used this command:

    PS > C:\Users\accal\.odrive-agent\bin\odrive.exe mount L:/Cloud /
    

    Step 6: Write a Sync Script

    so, now you are syncing your cloud storage. you're all done now right? WRONG!

    Odrive doesn't sync all your data by default because when you have 90GB of combined storage space that's a lot to drop on a HDD. instead it downloads placeholder files *.cloud and placeholder "folders" *.cloudf

    We want to download EVERYTHING, because we want all our data.

    Python to the rescue (remember to install Python on the windows PC if you're using this on a windows server like I am.)

    Script for Linux:

    #!/usr/bin/env python
    from subprocess import call
    import os
    from os import walk
    from os.path import join, dirname, abspath, exists
    from sys import argv, exit
    
    lockfile = join(dirname(abspath(__file__)), 'sync.lock')
    
    try:
    	import fcntl
    	fp = open(lockfile, 'w')
    	fp.flush()
    	try:
    		fcntl.lockf(fp, fcntl.LOCK_EX | fcntl.LOCK_NB)
    	except IOError:
    		print ('Another instace of sync is already running, aborting.')
    		exit(0)
    	raise
    
    def sync(path):
    	call(['/home/accalia/.odrive-agent/bin/odrive', 'sync', path])
    
    print(argv[1])
    for root, dirs, files in walk(argv[1]):
    	for file in files:
    		if file.endswith('.cloudf'):
    			sync(join(root,file))
    			files.remove(file)
    			dirs.append(file[:-7])
    		if file.endswith('.cloud'):
    			sync(join(root,file))
    

    Script for Windows:

    #!/usr/bin/env python
    from subprocess import call
    import os
    from os import walk
    from os.path import join, dirname, abspath, exists
    from sys import argv, exit, exc_info
    
    lockfile = join(dirname(abspath(__file__)), 'sync.lock')
    
    try:
            # file already exists, we try to remove (in case previous
            # execution was interrupted)
            if exists(lockfile):
                    os.unlink(lockfile)
            os.open(lockfile, os.O_CREAT | os.O_EXCL | os.O_RDWR)
    except OSError:
            type, e, tb = exc_info()
            if e.errno == 13:
                    print ('Another instace of sync is already running, aborting.')
                    exit(0)
            raise
    
    def sync(path):
    	call(['C:\\Users\\accal\\.odrive-agent\\bin\\odrive', 'sync', path])
    
    print(argv[1])
    for root, dirs, files in walk(argv[1]):
    	for file in files:
    		if file.endswith('.cloudf'):
    			sync(join(root,file))
    			files.remove(file)
    			dirs.append(file[:-7])
    		if file.endswith('.cloud'):
    			sync(join(root,file))
    
    

    Obviously i hardcoded the paths top the odrive executable. Naughty fox! go ahead and update those paths to the ones you need.

    Step 7: Run that sync job on a schedule!

    I chose every five minutes. Every five minutes new files and folders will be downloaded. You want this at fast as you are comfortable with to keep absolutely up to date, and don't worry about running multiple instances. that's why there are windows and mac/linux versions of the script above. they detect when another instance is already running and abort if it's still trying to finish a prior sync.

    For linux this is again a no brainer:

    # hey! this is my crontab again!
    */5 * * * * python /home/accalia/.odrive-agent/bin/sync.py /path/to/mount
    

    For Windows, another Scheduled task.

    This one runs at logon, and every 5 minutes thereafter, indefinitely.

    the executable it runs is C:\Users\accal\.odrive-agent\bin\sync.py with working directory C:\Users\accal\.odrive-agent\bin and arguments of L:\Cloud

    Huzzar! we're syncing!

    now just a liiiiitle bit of housekeeping.

    Step 8: Run Refresh and Empty Trash on a schedule.

    We want to periodically refresh our odrive folders to pick up changes that might have missed the SyncAgent's attention and to pick up new storage spaces as those don't always sync automatically in a timely manner.

    The command for that is simple,

    */30 * * * * python /home/accalia/.odrive-agent/bin/odrive.py refresh /path/to/mount
    

    And similarly in task scheduler, though i'll let you figure out the exact configuration by now, we've done it twice together after all.

    Got it? Good!

    Now we want to empty the trash regularly too, so files we delete locally get removed from cloud storage too.

    0 * * * * python /home/accalia/.odrive-agent/bin/odrive.py emptytrash /path/to/mount
    

    Step 9: ?‌??‌

    ?‌??‌

    ?‌??‌??‌??‌??‌??‌??‌??‌?!

    Step 10: Share your results with friends and enemies online!

    Which is what you've been reading!

    I hope it helps!


  • SockDev

    Alternative solution: Just use one service :slight_smile:

    (says the hedgie with two OneDrives and one Google Drive)


  • SockDev

    @RaceProUK said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:

    Just use one service

    hard to do when School requires you to use their gdrive instance, and work pfovides you with another and you already had your own personal, and you need access to all three because shared computer.

    @RaceProUK said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:

    (says the hedgie with two OneDrives and one Google Drive)

    see?



  • Redirecting errors to /dev/null is TRWTF


  • SockDev

    @JazzyJosh said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:

    Redirecting errors to /dev/null is TRWTF

    that's why i'm only redirecting stderr. odrive logs errors to (odriveagent binary directory)/../logs/error.log

    :-P


  • SockDev

    @accalia To be fair, the GDrive doesn't really get used, and I don't mind having my personal and work OneDrives separate. Actually, considering the amount of not-quite-legally-procured comic book scans on there, it's probably better that way :slight_smile:


  • Winner of the 2016 Presidential Election

    @accalia
    Hello Mr. or Mrs. Accalia,
    I have followed your 10 steps but Cloud Storage is still a thing that exists.
    This tutorial solves nothing!
    1/10 (1 effort point!)

    Filed Under: I didn't actually follow any of these steps but I am pretty sure 90% of ratings are from people who haven't done anything with the product!


  • SockDev

    @Kuro said in How Accalia Solved the whole Cloud Storage thing in Ten Simple Steps.:

    I have followed your 10 steps but Cloud Storage is still a thing that exists.

    hmm.... how unusual. Are you certain that you performed step 9 correctly?


  • SockDev

    @accalia But can it be done without a live chicken and a rabbi?


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.