[Hoax] Dude ruins company who had never heard of proper backups

[swayde]

Apr 18, 2016  /  Tech

Man accidentally deletes his entire company with one line of bad code

‘I feel sorry to say that your company is now essentially dead,’ one person on a coding forum advised Marco Marsala

[swayde]

Missed the text.
It does seem unlikely, but damn...

[fbmac]

Poor guy even had backups, but it was connected with the same ansible network and were deleted with it.

[blakeyrat]

@swayde He deleted his entire company by using an OS with terrible design. Don't blame the human here; blame the software with zero respect or compensation for human error.

[swayde]

@fbmac backups should never be writable. Why not mount a new folder each day? And why no offsite backup?
Sorry this just doesn't pass muster when you are a damn data based company. Offsite backups are for precisely this reason.

I do wonder what he was running that doesn't have --no-preserve-root

[fbmac]

@swayde I agree the guy is incompetent, but I pity him destroying his business like this.

Usually we hope to learn our lessons in less catastrophic ways.

[Rhywden]

@swayde Probably a really old version of some Linux distribution?

[mnm1992]

@Rhywden if you use rm -rf /* you can delete everything without --no-preserve-root just tested it on ubuntu.

[Rhywden]

@mnm1992 said in Dude ruins company who had never heard of proper backups:

@Rhywden if you use rm -rf /* you can delete everything without --no-preserve-root just tested it on ubuntu.

Well, the article says that he used rm {argument1}/{argument2} with both arguments being null. So that can't be it.

[anotherusername]

Here's the serverfault question...

Monday morning mistake: sudo rm -rf --no-preserve-root /

Please note: The answers and comments to this question contains content from another, similar question that has received a lot of attention from outside media but turned out to be hoax question in...

Apparently there might be a miraculous save at the end?

No word on how, yet.

[blakeyrat]

@Rhywden It's also the Internet. He could simply be a fucking liar.

[accalia]

@swayde i've long been tempted to replace rm with a tiny batch file that looks for the recursive and force flags in the command line and prints a rude comment before exiting with error if found and forking the real rm if not recursive and force.

I haven't yet mostly because i'm a little afraid of how many parts of the system would break because they use rm -rf somewhere in their code.

[mnm1992]

@Rhywden He also tests his scripts directly on production... I won't trust his statements. He might aswell have mis analysed his problem.

[ben_lubar]

@accalia if I want to delete a git folder, I need to use those flags.

[accalia]

@ben_lubar said in Dude ruins company who had never heard of proper backups:

@accalia if I want to delete a git folder, I need to use those flags.

no, you don't.

$ yes | rm -r ~/projects/sockbot/.git

see? problem solved.

[ben_lubar]

@accalia

yes | rm -r /*

[accalia]

@ben_lubar your point being?

[ben_lubar]

@accalia how are you running a batch file on Linux?

[accalia]

@ben_lubar said in Dude ruins company who had never heard of proper backups:

how are you running a batch file on Linux?

dude.... what are you getting at?

/me is confused

[RaceProUK]

@ben_lubar said in Dude ruins company who had never heard of proper backups:

@accalia how are you running a batch file on Linux?

Foxily

[ben_lubar]

@accalia said in Dude ruins company who had never heard of proper backups:

/me is confused

me too. Let's go eat some cupcakes.

[Polygeekery]

@blakeyrat said in Dude ruins company who had never heard of proper backups:

He deleted his entire company by using an OS with terrible design. Don't blame the human here; blame the software with zero respect or compensation for human error.

You are completely insufferable for thinking that running on Windows would always save people from doing completely stupid shit like this.

[accalia]

@Polygeekery said in Dude ruins company who had never heard of proper backups:

@blakeyrat said in Dude ruins company who had never heard of proper backups:

He deleted his entire company by using an OS with terrible design. Don't blame the human here; blame the software with zero respect or compensation for human error.

You are completely insufferable for thinking that running on Windows would always save people from doing completely stupid shit like this.

ah yes... deltree /y c:\ the rm -rf / of windows

[swayde]

There is a thread on reddit now.
Didn't we have a FP story with the same as this user is posting ? some pet shop or something ?
https://www.reddit.com/r/news/comments/4erjgp/man_accidentally_deletes_his_entire_company_with/d22t3h1

[flabdablet]

@swayde When using a production environment with all backup devices mounted simultaneously to develop scripts that invoke rm, best practice is to preconfigure it with sudo touch /-rf /--no-preserve-root first.

[accalia]

@flabdablet said in Dude ruins company who had never heard of proper backups:

@swayde When using a production environment with all backup devices mounted simultaneously to develop scripts that invoke rm, best practice is to preconfigure it with sudo touch /-rf /--no-preserve-root first.

..... i am now curious if that would work and actively fearful to test it.....

/me starts QEMU cloning one of her test VMs for an experiment

[flabdablet]

@accalia Test it with cd /; rm * run from the root account. Safest way is to use a blank VM booted from a live CD image.

[accalia]

@flabdablet that only makes me more afraid.

i shall report results once the VM finishes cloning and i get to somewhere where i can watch from teh console as i suspect SSH is likely to crap out at some point in the process.

[loopback0]

When I saw the title of the article, I hoped it was Jeff.

[fbmac]

some people are suspicious this is a fake on slashdot

[accalia]

@flabdablet said in Dude ruins company who had never heard of proper backups:

cd /; rm *

ah.

oh that's evil.

and of course it's triggered contextually too.

rm -- * won't trigger it

nor will rm ./*

[swayde]

@fbmac said in Dude ruins company who had never heard of proper backups:

some people are suspicious this is a fake on slashdot

@swayde said in Dude ruins company who had never heard of proper backups:

Missed the text.
It does seem unlikely, but damn...

I'm suspicious too. 1300 customers and not one has confirmed? Not one screenshot? Has data back after only hours? Story goes viral of it's own volition? In hours?
It's not impossible, just unlikely.

[anonymous234]

@Rhywden said in Dude ruins company who had never heard of proper backups:

Well, the article says that he used rm {argument1}/{argument2} with both arguments being null

If only people would STOP USING SHITTY OLD SCRIPTING LANGUAGES BASED ON STRING REWRITING this could have been avoided.

[flabdablet]

@Polygeekery said in Dude ruins company who had never heard of proper backups:

@blakeyrat said in Dude ruins company who had never heard of proper backups:

He deleted his entire company by using an OS with terrible design. Don't blame the human here; blame the software with zero respect or compensation for human error.

You are completely insufferable for thinking that running on Windows would always save people from doing completely stupid shit like this.

To be fair, Windows command shells leave glob expansion to the executables they invoke, and switches for its rmdir are introduced by a character that can't occur in filenames; either of these is enough to guarantee that rmdir C:\* is never going to pick up a spurious /S option from a maliciously crafted filename.

The idea that these design decisions flowed from a deliberate attempt to build a system more tolerant of human error is of course completely unhinged. Putting rmdir /S/Q %foo% in a Windows command script is every bit as potentially destructive as putting rm -rf $foo in a Unix one.

[loopback0]

@swayde There's a Reddit onebox?

Also he changed this script and decided not to test it before running it on Production?

[swayde]

@loopback0 said in Dude ruins company who had never heard of proper backups:

@swayde There's a Reddit onebox?

Also he changed this script and decided not to test it before running it on Production?

1. I did not see that when posting.
2. are emoji case sensitive ? :WTF: does noot seem to work in preview

[FrostCat]

@ben_lubar said in Dude ruins company who had never heard of proper backups:

if I want to delete a git folder

I solve that problem preemptively by not using git.

[flabdablet]

@accalia said in Dude ruins company who had never heard of proper backups:

@flabdablet said in Dude ruins company who had never heard of proper backups:

cd /; rm *

ah.

oh that's evil.

and of course it's triggered contextually too.

rm -- * won't trigger it

nor will rm ./*

Quite so.

I think my favourite thing about this particular piece of vandalism is that if the rm * is allowed to run to completion, the only files remaining in the filesystem will be /-rf and /--no-preserve-root.

[blakeyrat]

@Polygeekery said in Dude ruins company who had never heard of proper backups:

You are completely insufferable for thinking that running on Windows would always save people from doing completely stupid shit like this.

I don't think that.

So I'm insufferable because of a thing you pulled out of your ass entirely.

Awesome.

[DCRoss]

@swayde said in Dude ruins company who had never heard of proper backups:

@fbmac said in Dude ruins company who had never heard of proper backups:

some people are suspicious this is a fake on slashdot

@swayde said in Dude ruins company who had never heard of proper backups:

Missed the text.
It does seem unlikely, but damn...

I'm suspicious too. 1300 customers and not one has confirmed? Not one screenshot? Has data back after only hours? Story goes viral of it's own volition? In hours?
It's not impossible, just unlikely.

Not only that, but he said he was running CentOS 7, which, according to the man pages here, has --no-preserve-root set by default in rm. That means his "rm -rf {foo}/{bar}" command should have failed with an error unless he had somehow explicitly sabotaged it with this end in mind.

He claims to have run this on every machine at once, during the backup window, while his backup drives were mounted on every machine, and then while recovering the data did his best impression of the "Well what if you don't have that..." interviewer by accidentally running "dd if=blankdisk of=sourcedisk" instead of "dd if=sourcedisk of=blankdisk", destroying whatever was left of his drives, and then... recovered all of the data anyway.

All this without a single customer speaking up, or any coverage of this disaster that wasn't a single anonymous post on serverfault.

It's not impossible, but there are a few bits missing from the story which would make it more believable.

[Tsaukpaetra]

@DCRoss said in Dude ruins company who had never heard of proper backups:
and then... recovered all of the data anyway.

You don't need the url= there...

[DCRoss]

Changing forums makes by brain hurt.

[CreatedToDislikeThis]

Wow, a lot of "professional programmers' here.
Replace 'OS' with 'OS-endorsed scripting language' and Blakey got it right here - being able to delete an entire folder implicitly (by avoiding specifying something) is a serious failure of the language that makes it plain dangerous to use.
That this can delete the root folder in some cases (rm -rf /* ?) is even worse.
Backups are important but are no excuse for dangerous languages.

[Grunnen]

@CreatedToDislikeThis As if it is any different on Windows. If you put something like deltree %foo%\*.* in a bat file the result is about the same.

Well, on Windows you have the option of mounting a backup disk using a different drive letter rather than under a subfolder, but that is just something specific to this particular case.

Anyways, I guess the lesson is that a backup maintenance script should simply not delete files as long as it has the backup medium mounted.

[flabdablet]

@Grunnen said in Dude ruins company who had never heard of proper backups:

I guess the lesson is that a backup maintenance script should simply not delete files as long as it has the backup medium mounted.

I think that's the wrong lesson. I think the right lesson is that if you ever mount all your backup media at once, you're doing it wrong.

[CreatedToDislikeThis]

@Grunnen said in Dude ruins company who had never heard of proper backups:

@CreatedToDislikeThis As if it is any different on Windows. If you put something like deltree %foo%\*.* in a bat file the result is about the same.

Well, on Windows you have the option of mounting a backup disk using a different drive letter rather than under a subfolder, but that is just something specific to this particular case.

Anyways, I guess the lesson is that a backup maintenance script should simply not delete files as long as it has the backup medium mounted.

I never claimed BAT files were better. (Though, at least, I doubt "deltree %foo%*.*" would delete too much(/anything?) gven that "" isn't "root folder" on windows).

They are less endorsed for automation on Windows, though. Does Powershell have the same issue (asking - I never used it and have no idea)

[Grunnen]

@flabdablet said in Dude ruins company who had never heard of proper backups:

@Grunnen said in Dude ruins company who had never heard of proper backups:

I guess the lesson is that a backup maintenance script should simply not delete files as long as it has the backup medium mounted.

I think that's the wrong lesson. I think the right lesson is that if you ever mount all your backup media at once, you're doing it wrong.

That as well. Luckily at our company we have something like 10 backup disks which are rotated for daily and monthly backups. Recently I needed to restore something from a backup and that actually even worked too. I think it's something they/we actually got right. _{I only sometimes wonder whether we have any off-site backups.}

[asdf]

@mnm1992 said in Dude ruins company who had never heard of proper backups:

if you use rm -rf /* you can delete everything without --no-preserve-root just tested it on ubuntu.

ZSH will give you an "are you sure" prompt.

[fbmac]

@CreatedToDislikeThis said in Dude ruins company who had never heard of proper backups:

I doubt "deltree %foo%." would delete too much(/anything?) gven that "" isn't "root folder" on windows).

you forgot the backslash after foo. deltree works exactly the same way

[Polygeekery]

@flabdablet said in Dude ruins company who had never heard of proper backups:

I think that's the wrong lesson. I think the right lesson is that if you ever mount all your backup media at once, you're doing it wrong.

That and his backup strategy was fucked to begin with. It sounded as though he did not really have an on-site backup, only off-site. They should have an on-site backup that aggregates everything and then transports a versioned copy off-site.

Without versioning, you do not really have backups. He never should have been able to delete all of his backups, period. End of story. If that is even possible, you are doing it wrong.