Backup strategies


  • Discourse touched me in a no-no place

    Got to work to be greeted by an email saying our Trac[1] website was reporting disk full. I start digging. The main problem at the moment can probably be summarised by:


    [root@trac ~]# crontab -l
    [...]
    01 00 * * *     trac-admin /srv/trac/site/ hotcopy /trac/backups/$$
    
    1. At one minute past midnight
    2. every day
    3. a full backup is made of the Trac database in
    4. a randomly named directory that
    5. could clash with a previous incarnation,
    6. is never compressed and
    7. no deletions of any previous backups are performed and
    8. this has been in place for over two years.



    [1] http://trac.edgewall.org/ for those that haven't come across it - it's basically a ticketing system. That link is not our Trac website.


  • Is the backup at least stored on a seperate disk from the production database?


  • Discourse touched me in a no-no place

    @cdosrun said:

    Is the backup at least stored on a seperate disk from the production database?
    Don't be silly, of course it wasn't. It was even in the same partition.



  • This is one of the most awesome WTFs ever posted to this site, particularly since it means that (9) the backups have probably never been used before (if they had someone would've noticed that the backup schedule was created by a moron) and (10) the backups are useless (storing a backup on the same physical disk shows that the creator had no idea of the actual concept of "backup").

    Find the person who setup that backup schedule and beat him to death with a hard disk, while shouting "BACK THIS UP, BITCH!"

    Also, embedding links into your text is infinitely preferable to footnoting them. :p


  • Discourse touched me in a no-no place

    @The_Assimilator said:

    Find the person who setup that backup schedule and beat him to death with a hard disk, while shouting "BACK THIS UP, BITCH!"
    The person concerned no longer works for us.
    @The_Assimilator said:
    Also, embedding links into your text is infinitely preferable to footnoting them. :p
    The link was largely irrelevant to the post, and had I linked at the top I'd be getting replies thinking that the Trac website itself was the server I was talking about.


  • Discourse touched me in a no-no place

    Anyway, to follow up, I deleted all the backups since they were essentially worse than useless, put in a saner backup script (based largely on AutoMySQLBackup) and just need to organise a different server to move the backups to:

    [root@trac backups]# df -m
    Filesystem           1M-blocks      Used Available Use% Mounted on
    /dev/mapper/VolGroup00-LogVol00
                             73012      1922     67382   3% /
    /dev/sda1                   99        13        81  14% /boot
    none                       505         0       505   0% /dev/shm
    


  • @PJH said:

    Got to work to be greeted by an email saying our Trac[1] website was reporting disk full. I start digging. The main problem at the moment can probably be summarised by:


    [root@trac ~]# crontab -l
    [...]
    01 00 * * *     trac-admin /srv/trac/site/ hotcopy /trac/backups/$$
    

    .

    That's horrible.

    Why not just make a symlink to it instead, it'd be a lot faster...



  • @PJH said:

    a randomly named directory that

    To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?

    Keeping a day-old mirror on the same partition isn't that big a deal to me, frankly, because it can solve a lot of easy backup problems without having to go to your "real" backup system. Stupid stuff like, "oh shit I hit delete too fast, and it's a network drive so there's no recycle bin".



  • @blakeyrat said:

    @PJH said:
    a randomly named directory that

    To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?

    Keeping a day-old mirror on the same partition isn't that big a deal to me, frankly, because it can solve a lot of easy backup problems without having to go to your "real" backup system. Stupid stuff like, "oh shit I hit delete too fast, and it's a network drive so there's no recycle bin".


    something like
    ls -l | grep "insert date here"


  • @PJH said:

    @The_Assimilator said:
    Find the person who setup that backup schedule and beat him to death with a hard disk, while shouting "BACK THIS UP, BITCH!"
    The person concerned no longer works for us.

    You have an HR department. Your HR department has employment records. Make a plan.

    @Lord abletran said:

    @blakeyrat said:
    @PJH said:
    a randomly named directory that

    To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?

    Keeping a day-old mirror on the same partition isn't that big a deal to me, frankly, because it can solve a lot of easy backup problems without having to go to your "real" backup system. Stupid stuff like, "oh shit I hit delete too fast, and it's a network drive so there's no recycle bin".


    something like
    ls -l | grep "insert date here"

    I had the same fucking idea... that definitely means I've been on this site (or in the industry) for too long...



  • ls -lt gives you the dates and sorts them from most youngest to oldest. Of course, instead of $$, the author could have just used date +%Y%m%d, but that would make it too easy.



  • @Xyro said:

    ls -lt gives you the dates and sorts them from most youngest to oldest. Of course, instead of $$, the author could have just used common sense, but that would make it too easy.

    FTFY


  • ♿ (Parody)

    @Xyro said:

    ls -lt gives you the dates and sorts them from most youngest to oldest.

    Pshaw...that's all well and good for people who only have a few things in the directory. For lots of stuff, you need ls -lrt so the stuff you're interested in doesn't scroll off the screen. (blakey, that's like clicking the column header twice when you're trying to sort it so you don't have to deal with the scroll bar).



  • @boomzilla said:

    ls -lrt

    I used to habitually use -tr because I was too lazy to just | head the output. Since switching jobs a few years ago and dealing with directories so freaking humongous that it could take 30-60 for ls to return (and that's not even screen-draw time), I have since switched my habit to just | head or more everything. (Yes, more, these HP-UX machines don't have fancy things like less.)

    But since the "backup" script has been only a few hundred times, I guess paging the output would be unnecessary...



  • @Lord abletran said:

    @PJH said:
    Got to work to be greeted by an email saying our Trac[1] website was reporting disk full. I start digging. The main problem at the moment can probably be summarised by:

    [root@trac ~]# crontab -l
    [...]
    01 00 * * *     trac-admin /srv/trac/site/ hotcopy /trac/backups/$$
    


    .
    That's horrible. Why not just make a symlink to it instead, it'd be a lot faster...

     

    And wouldn't have caused the out of space error, either.



  • @The_Assimilator said:

    This is one of the most awesome WTFs ever posted to this site
    Agreed - this really should have gone on the front page.  Please submit it!


  • Discourse touched me in a no-no place

    @hoodaticus said:

    @The_Assimilator said:

    This is one of the most awesome WTFs ever posted to this site
    Agreed - this really should have gone on the front page.  Please submit it!

    Done. (Didn't think it front-page-worthy myself.)


  • Discourse touched me in a no-no place

    @blakeyrat said:

    To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?
    When I was considering only deleting the older directories, rather than blowing away the whole lot, I used ls -lart to find out which were which.


Log in to reply