[Hoax] Dude ruins company who had never heard of proper backups
-
@Polygeekery said in Dude ruins company who had never heard of proper backups:
Without versioning, you do not really have backups.
And the best way to implement versioning is to have each version on its own medium.
-
@flabdablet said in Dude ruins company who had never heard of proper backups:
And the best way to implement versioning is to have each version on its own medium.
I would strongly disagree with you there.
-
@swayde said in Dude ruins company who had never heard of proper backups:
@fbmac why no offsite backup?
According to the article he in fact have offsite backup, just that since the backup media is mounted during the accident, those backup are cleaned by this command too.
-
@CreatedToDislikeThis honestly, the way the filesystem works internally in Windows is much more UNIXy than you might expect. There is something very rootlike, and drive letters are simply a convenience. But few things interact with it at that level.
-
"Mr Masarla runs a web hosting company, which looks after the servers and internet connections on which the files for websites are stored."
nope, he doesn't. not anymore. :-D
-
@Polygeekery The trouble with "smart" backup solutions that implement versioning on a single medium is that people start relying on them, and then act all surprised and hurt when something trashes the versioning mechanism.
Windows provides a volume shadow copy mechanism that implements file and folder versioning on NTFS volumes, and it's really tempting to use that to implement a versioned backup scheme; Apple's Time Machine uses a slightly modified Unix filesystem that allows the creation of hardlinks to directories to do much the same thing.
The trouble with all of these schemes is that mounting the drive containing a versioned filesystem does amount to having all those versions online at the same time, which means that a rogue process running on the host where they're mounted can destroy them all. Most of the current crop of crypto ransomware, for example, is aware of both Windows VSC and Time Machine and quite capable of trashing both.
Versioned backups that live behind a tightly specified API on some other server - an API that specifically doesn't allow clients any control over version maintenance - are safer than a versioned filesystem mounted directly on the host being backed up, but are still susceptible to rogue code that finds its way into the backup server.
If your backup strategy involves leaving most of your backups offline most of the time, and only ever mounting media containing the single version you're about to discard anyway, it's pretty hard to fuck it up by any means.
-
@flabdablet said in Dude ruins company who had never heard of proper backups:
Windows provides a volume shadow copy mechanism that implements file and folder versioning on NTFS volumes
You are generally right. But volume shadow copies can be exported. Say incremental every 12 and full backup every 48. This works well. I've even had a workplace do incremental each hour - remember, only changed files are take up space.
-
@swayde said in Dude ruins company who had never heard of proper backups:
volume shadow copies can be exported.
Sure. But I've seen Windows arbitrarily decide to drop all the shadow copies on a given volume just because it felt like it, which means that there's a possibility of that happening on the volume you export your shadow copies to.
De-duping is neat, and it saves money on backup media, but it also introduces failure modes that don't exist if you just do things the old slow stupid expensive way.
-
I was about to post this, but something didn't feel right with this story.
Did some research (also known as checked out reddit). And, yup.
It's a troll.
-
@flabdablet said in Dude ruins company who had never heard of proper backups:
The trouble with "smart" backup solutions that implement versioning on a single medium is that people start relying on them, and then act all surprised and hurt when something trashes the versioning mechanism.
Well, they should only be part of your backup strategy. Any good backup strategy should have multiple layers. Simply mounting a volume locally and copying things over can get you situations like this person (who it appears was a troll account) can do. The key is to put it behind an API that does not allow any control over versioning or deletion of objects. Make it entirely incapable of the client ever touching that, at least not through automated processes.
@flabdablet said in Dude ruins company who had never heard of proper backups:
Versioned backups that live behind a tightly specified API on some other server - an API that specifically doesn't allow clients any control over version maintenance - are safer than a versioned filesystem mounted directly on the host being backed up, but are still susceptible to rogue code that finds its way into the backup server.
There are ways to mitigate that, as well as anything can be mitigated. I think we have done about as good a job as we can.
For the offsite backup solution that we sell, there is a client executable that transports the data to our datacenter. From the client installable, there is absolutely no mechanism that would ever allow for the changing of versioning or deletion of backups. The service itself is actually pretty dumb. Cataloging, transport and logging. That's it, if you boil it down to one sentence.
Now that we are out of beta, and on a more stable development track, everything goes through a pretty extensive shakedown in our sandbox. Hotfixes in production are not something we ever want to have to do, and we have not had to. If we did, there would be a very extensive autopsy to find out went wrong and we would do our best to make sure it never happened again. In this (fictional) scenario, he had no testing area. Everything was done in production. That is a really goddamn bad place to find out if you have unintended consequences.
The last line of defense though is that every machine that houses customer data is N*2. We have enough margin, and our CapEx costs on deploying a machine are low enough that every machine in production has a redundancy. The redundant machine does not even have our code running on it. Once per day, an rsync operation synchronizes data between the servers. Each redundant machine has a snapshot run on it before rsync and no operations are allowed that would reduce the number of snapshots below a set level. That level guarantees us that, if massive amounts of shit were to hit the fan, and every single machine in production were to be wiped completely clean of customer data, we could failover to the redundant servers with a theoretical loss of no more than 24 hours of data.
The failover process is manual, and is intentionally so and would require us installing our software on the machines which is not to be done until we are sure of what the fuck went wrong.
There are still people involved in the system, so it is far from infallible, but it is as failsafe as I could think to make it.
@flabdablet said in Dude ruins company who had never heard of proper backups:
If your backup strategy involves leaving most of your backups offline most of the time, and only ever mounting media containing the single version you're about to discard anyway, it's pretty hard to fuck it up by any means.
Perhaps. But you are losing out on deduplication and consolidation of resources. We decided to go the route of not ever mounting the media at all.
-
@cartman82 As far as trolls on the Internet go, I'd rank him above average on the entertaining scale.
-
@Polygeekery said in Dude ruins company who had never heard of proper backups:
There are still people involved in the system, so it is far from infallible, but it is as failsafe as I could think to make it.
Having a redundant datacenter that is located in a different part of the city (physical separation, independent power) is about the only thing missing. It's also where costs start to pick up. :(
-
@dkf said in Dude ruins company who had never heard of proper backups:
Having a redundant datacenter that is located in a different part of the city (physical separation, independent power) is about the only thing missing. It's also where costs start to pick up.
That is on the development track. We are looking at how to do it economically as a value added service to customers. $X/GB for single location, $X + $Y for geographic diversity. It should lower our costs and also give customers options and keep their costs down. Not everyone needs geographic diversity, really. There are a shitload of businesses out there that if overwhelming catastrophe were to strike our area and wipe out all of the infrastructure, would never be rebuilt and would never need access to their data.
For now, with this being an offsite backup, not a primary backup for customers, and with the datacenter we are in, I am happy enough with the arrangement for now. Our servers are sitting in a building that used to be a nuclear bunker. A tornado would not even take them offline.
I can be that sure because the owners of the datacenter are friends of mine, I know the care that goes in to the place, and I genuinely trust their processes and equipment because I have been there and seen it built from the beginning.
If I were in one of Expedient's datacenters I would not be able to sleep at night, because I also know their processes and equipment and they are absolute shit.
-
@swayde Like I said before, troll post. I do not believe someone is actually incompetent enough to rm -rf /* and also dd if=/dev/null of=/dev/sda0
-
@Grunnen said in Dude ruins company who had never heard of proper backups:
As if it is any different on Windows.
NOBODY IS SAYING IT IS ANY BETTER IN WINDOWS. IT'S POSSIBLE FOR BOTH LINUX AND WINDOWS TO BE SHITTY AS SOMETHING SIMULTANEOUSLY.
-
@Polygeekery said in Dude ruins company who had never heard of proper backups:
There are a shitload of businesses out there that if overwhelming catastrophe were to strike our area and wipe out all of the infrastructure, would never be rebuilt and would never need access to their data.
QFT. For my current project, we're looking to survive a fire or flood taking out the building but nothing more catastrophic. Anything worse than that will cause sufficient damage that recovering the data will be knocked off the first hundred pages of worries. (If nothing else, we'd have to rebuild the labs in order to sensibly use the data and that'd be lot of millions of expenditure right there.)
-
@blakeyrat You don't need to shout. Especially because it would have been quite easy to prove me wrong: AFAIK Windows has user roles specifically designed for backups, e.g. allowing to read any protected file but not to delete them.
-
@Grunnen You're not wrong; you're just replying to something NOBODY SAID, then acting all smug like, "oh yeah, buddy, I demolished that point, didn't I? I am the greatest".
Look, if I said: "Linux is shitty at X", that only means "Linux is shitty at X." It doesn't mean, "Linux is shitty at X and oh BTW Windows is amazing at X I saw Windows do X once and I came right in my pants". Just read the words people type and reply to those. Don't reply to the words your defective moron brain made-up and attributed to them.
-
@cartman82 said in Dude ruins company who had never heard of proper backups:
It's a troll.
That didn't stop The Register from posting it as news just now.
-
@NedFodder The Register is the Daily Fail of online tech journalism. Not impressive unless techdirt gets fooled as well.
-
@NedFodder said in Dude ruins company who had never heard of proper backups:
The Register
Just look at the URI:
[…]man_deleats_customers_running_script
Doesn't that say everything about the quality of the article?
-
The real investigative journalists at El Reg post exclusively in the comments sections.
Most of them got the story right.
-
@blakeyrat said in Dude ruins company who had never heard of proper backups:
@swayde He deleted his entire company by using an OS with terrible design. Don't blame the human here; blame the software with zero respect or compensation for human error.
It isn't an OS, it's drivers.
-
@blakeyrat when you say A is shitty at X it hints you think A is doing it worse than it's main competitors.
-
@fbmac said in Dude ruins company who had never heard of proper backups:
when you say A is shitty at X it hints you think A is doing it worse than it's main competitors.
Nothing I say "hints" at anything.
-
-
@Polygeekery said in Dude ruins company who had never heard of proper backups:
@blakeyrat said in Dude ruins company who had never heard of proper backups:
He deleted his entire company by using an OS with terrible design. Don't blame the human here; blame the software with zero respect or compensation for human error.
You are completely insufferable for thinking that running on Windows would always save people from doing completely stupid shit like this.
Exactly. I have, in fact, done exactly what this guy did except it was a) on my home machine and b) on Windows.
From memory (it was a looooooooong time ago and, yes, I did learn my lesson) it was something like;
net use X: \\someserver\share
X:
REM Of course the net use failed...
rd /s /q *.*
-
rm just modifies the file table, it doesn't destroy any of the underlying data, so as long as nothing else was written to those locations, most of it can easily be restored. Calling bullshit on the whole story.
-
@Ashley_Sheridan said in Dude ruins company who had never heard of proper backups:
rm just modifies the file table, it doesn't destroy any of the underlying data, so as long as nothing else was written to those locations, most of it can easily be restored. Calling bullshit on the whole story.
Not sure I concur with your use of the word "easily", given the likely size of the data and (IIRC) spans across mounted filesystems, but I do agree that we're missing some chunks of info.
-
@skotl said in Dude ruins company who had never heard of proper backups:
I do agree that we're missing some chunks of info.
Like it being bullshit.
-
@skotl Somewhere in the comments he indicates that they were able to get the data restored by a data rescue company.
-
Haha, Engadget too.
-
-
@skotl said in [Hoax] Dude ruins company who had never heard of proper backups:
Now, i'll have to believe you and SO that the article actually says what they claim. I've edited the title.
To be fair i pointed the unlikelyness out in my second post.
-
@skotl Why don't they just knee-jerk vote to close it. Oh wait, they do that to ALL questions already.
Only scammers and trolls, apparently, are able to successfully keep a question open.
-
@swayde The thing I don't understand is that the discussion I linked to there suggests that the guy/company created the hoax "for marketing reasons". What possible good reasons are there for this kind of hoax?
Hmmm... unless he works for the data recovery company and cited them as his saviour...?
-
@skotl any attention is good attention?
Now pulling a trick like this is likely to make people actually ruin his Company. 🤓
-
@swayde said in [Hoax] Dude ruins company who had never heard of proper backups:
🤓
Curse my Windows and not having proper Emoji support!
-
@Tsaukpaetra said in [Hoax] Dude ruins company who had never heard of proper backups:
Curse my Windows and not having proper Emoji support!
Seems it's a new one, it looked geeky. It's not on my android 5 device either.
-
@skotl said in [Hoax] Dude ruins company who had never heard of proper backups:
What possible good reasons are there for this kind of hoax?
I don't know what the word ‘good’ means, but I understand you can often get good seo scores out of viral stunts like this.
-
@blakeyrat said in [Hoax] Dude ruins company who had never heard of proper backups:
@swayde He deleted his entire company by using an OS with terrible design. Don't blame the human here; blame the software with zero respect or compensation for human error.
@Polygeekery said in [Hoax] Dude ruins company who had never heard of proper backups:
@blakeyrat said in Dude ruins company who had never heard of proper backups:
You are completely insufferable for thinking that running on Windows would always save people from doing completely stupid shit like this.
I have to agree with Blakey on this one.
A single command line that's only a few characters, where a single space determines whether you delete a targeted location or the root and everything mounted to the machine....
That's fucking stupid.
Now as far as Windows being better, the statement was rather atomic, throwing Windows in there is a false dichotomy.