Backup strategies
-
Got to work to be greeted by an email saying our Trac[1] website was reporting disk full. I start digging. The main problem at the moment can probably be summarised by:
[root@trac ~]# crontab -l [...] 01 00 * * * trac-admin /srv/trac/site/ hotcopy /trac/backups/$$
- At one minute past midnight
- every day
- a full backup is made of the Trac database in
- a randomly named directory that
- could clash with a previous incarnation,
- is never compressed and
- no deletions of any previous backups are performed and
- this has been in place for over two years.
[1] http://trac.edgewall.org/ for those that haven't come across it - it's basically a ticketing system. That link is not our Trac website.
-
Is the backup at least stored on a seperate disk from the production database?
-
@cdosrun said:
Is the backup at least stored on a seperate disk from the production database?
Don't be silly, of course it wasn't. It was even in the same partition.
-
This is one of the most awesome WTFs ever posted to this site, particularly since it means that (9) the backups have probably never been used before (if they had someone would've noticed that the backup schedule was created by a moron) and (10) the backups are useless (storing a backup on the same physical disk shows that the creator had no idea of the actual concept of "backup").
Find the person who setup that backup schedule and beat him to death with a hard disk, while shouting "BACK THIS UP, BITCH!"
Also, embedding links into your text is infinitely preferable to footnoting them. :p
-
@The_Assimilator said:
Find the person who setup that backup schedule and beat him to death with a hard disk, while shouting "BACK THIS UP, BITCH!"
The person concerned no longer works for us.
@The_Assimilator said:Also, embedding links into your text is infinitely preferable to footnoting them. :p
The link was largely irrelevant to the post, and had I linked at the top I'd be getting replies thinking that the Trac website itself was the server I was talking about.
-
Anyway, to follow up, I deleted all the backups since they were essentially worse than useless, put in a saner backup script (based largely on AutoMySQLBackup) and just need to organise a different server to move the backups to:
[root@trac backups]# df -m Filesystem 1M-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 73012 1922 67382 3% / /dev/sda1 99 13 81 14% /boot none 505 0 505 0% /dev/shm
-
@PJH said:
Got to work to be greeted by an email saying our Trac[1] website was reporting disk full. I start digging. The main problem at the moment can probably be summarised by:
[root@trac ~]# crontab -l [...] 01 00 * * * trac-admin /srv/trac/site/ hotcopy /trac/backups/$$
.That's horrible.
Why not just make a symlink to it instead, it'd be a lot faster...
-
@PJH said:
a randomly named directory that
To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?
Keeping a day-old mirror on the same partition isn't that big a deal to me, frankly, because it can solve a lot of easy backup problems without having to go to your "real" backup system. Stupid stuff like, "oh shit I hit delete too fast, and it's a network drive so there's no recycle bin".
-
@blakeyrat said:
@PJH said:
a randomly named directory that
To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?
Keeping a day-old mirror on the same partition isn't that big a deal to me, frankly, because it can solve a lot of easy backup problems without having to go to your "real" backup system. Stupid stuff like, "oh shit I hit delete too fast, and it's a network drive so there's no recycle bin".
something like
ls -l | grep "insert date here"
-
@PJH said:
@The_Assimilator said:
Find the person who setup that backup schedule and beat him to death with a hard disk, while shouting "BACK THIS UP, BITCH!"
The person concerned no longer works for us.You have an HR department. Your HR department has employment records. Make a plan.
@Lord abletran said:
@blakeyrat said:
@PJH said:
a randomly named directory that
To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?
Keeping a day-old mirror on the same partition isn't that big a deal to me, frankly, because it can solve a lot of easy backup problems without having to go to your "real" backup system. Stupid stuff like, "oh shit I hit delete too fast, and it's a network drive so there's no recycle bin".
something like
ls -l | grep "insert date here"I had the same fucking idea... that definitely means I've been on this site (or in the industry) for too long...
-
ls -lt
gives you the dates and sorts them from most youngest to oldest. Of course, instead of$$
, the author could have just used
, but that would make it too easy.date +%Y%m%d
-
@Xyro said:
ls -lt
gives you the dates and sorts them from most youngest to oldest. Of course, instead of$$
, the author could have just used common sense, but that would make it too easy.FTFY
-
@Xyro said:
ls -lt
gives you the dates and sorts them from most youngest to oldest.
Pshaw...that's all well and good for people who only have a few things in the directory. For lots of stuff, you needls -lrt
so the stuff you're interested in doesn't scroll off the screen. (blakey, that's like clicking the column header twice when you're trying to sort it so you don't have to deal with the scroll bar).
-
@boomzilla said:
ls -lrt
I used to habitually use-tr
because I was too lazy to just| head
the output. Since switching jobs a few years ago and dealing with directories so freaking humongous that it could take 30-60 forls
to return (and that's not even screen-draw time), I have since switched my habit to just| head
ormore
everything. (Yes,more
, these HP-UX machines don't have fancy things likeless
.)But since the "backup" script has been only a few hundred times, I guess paging the output would be unnecessary...
-
@Lord abletran said:
@PJH said:
Got to work to be greeted by an email saying our Trac[1] website was reporting disk full. I start digging. The main problem at the moment can probably be summarised by:
That's horrible. Why not just make a symlink to it instead, it'd be a lot faster...[root@trac ~]# crontab -l [...] 01 00 * * * trac-admin /srv/trac/site/ hotcopy /trac/backups/$$
.And wouldn't have caused the out of space error, either.
-
@The_Assimilator said:
This is one of the most awesome WTFs ever posted to this site
Agreed - this really should have gone on the front page. Please submit it!
-
@hoodaticus said:
Done. (Didn't think it front-page-worthy myself.)@The_Assimilator said:
This is one of the most awesome WTFs ever posted to this site
Agreed - this really should have gone on the front page. Please submit it!
-
@blakeyrat said:
To ask a possibly stupid question, how did the person who implemented this "system" intend to restore (say) yesterday's backup if they weren't named by date?
When I was considering only deleting the older directories, rather than blowing away the whole lot, I usedls -lart
to find out which were which.