RAID is the best backup
-
@PJH how am I doing on attendance?
[postgres@sofa ~]$ NAME=mott555 sql_tdwtf attendance # Record of continuous days of attendance # Most continuous days of attendance WITH StartingPoints AS ( SELECT user_id, visited_at, ROW_NUMBER() OVER(ORDER BY user_id, visited_at) AS rownum FROM user_visits AS A WHERE NOT EXISTS ( SELECT * FROM user_visits AS B WHERE B.visited_at = A.visited_at - 1 AND B.user_id = A.user_id ) ), EndingPoints AS ( SELECT user_id, visited_at, ROW_NUMBER() OVER(ORDER BY user_id, visited_at) AS rownum FROM user_visits AS A WHERE NOT EXISTS ( SELECT * FROM user_visits AS B WHERE B.visited_at = A.visited_at + 1 AND B.user_id = A.user_id ) ) SELECT u.username, S.visited_at AS start_range, E.visited_at AS end_range, (E.visited_at - S.visited_at +1) AS Days FROM StartingPoints AS S JOIN EndingPoints AS E ON E.rownum = S.rownum JOIN users u ON u.id=S.user_id AND u.username like 'mott555' username | start_range | end_range | days ----------+-------------+------------+------ mott555 | 2014-05-20 | 2014-06-04 | 16 mott555 | 2014-06-11 | 2014-06-13 | 3 mott555 | 2014-06-16 | 2014-06-27 | 12 mott555 | 2014-06-30 | 2014-07-03 | 4 mott555 | 2014-07-07 | 2014-07-12 | 6 mott555 | 2014-07-14 | 2014-07-18 | 5 mott555 | 2014-07-21 | 2014-08-01 | 12 mott555 | 2014-08-04 | 2014-11-08 | 97 (8 rows) Elapsed: 0.596s Backup taken: 2014-11-08 13:05:21.472597 [postgres@sofa ~]$ 5~
-
> a process that can take a surprisingly short time and is quite computationally inexpensive
8/10 for the dark humor, would read again
-
You don't encrypt backups stored on 3rd party servers?
Encryption of backups on 3rd party servers is definitely a good idea; what I'm saying is that it may not quite be enough in all circumstances...
-
my googlefu has failed me for the last time
You're going to quit searching for things on Google?
-
How am I doing attendance-wise? I have a bad feeling I missed a day somewhere.
-
Mobile, so excuse formatting..
assword: postgres@sofa ~]$ sql_tdwtf attendance | grep Frost FrostCat | 2014-07-09 | 2014-07-11 | 3 FrostCat | 2014-07-13 | 2014-07-14 | 2 FrostCat | 2014-07-17 | 2014-07-17 | 1 FrostCat | 2014-07-19 | 2014-07-19 | 1 FrostCat | 2014-07-21 | 2014-07-22 | 2 FrostCat | 2014-07-24 | 2014-07-24 | 1 FrostCat | 2014-07-28 | 2014-07-28 | 1 FrostCat | 2014-08-02 | 2014-08-03 | 2 FrostCat | 2014-08-05 | 2014-08-05 | 1 FrostCat | 2014-08-07 | 2014-08-07 | 1 FrostCat | 2014-08-09 | 2014-08-15 | 7 FrostCat | 2014-08-18 | 2014-08-18 | 1 FrostCat | 2014-08-21 | 2014-09-13 | 24 FrostCat | 2014-09-15 | 2014-11-08 | 55 postgres@sofa ~]$
-
That's good enough, thanks.
-
Another (bigger) issue with a RAID 5/6 is rebuild mechanics.
Because of the parity bit(s), any rebuild on RAID 5/6 must thrash ALL OTHER drives in the array in order to rebuild the data. If you're using a RAID 10, only the "matching" drive on the other side has to be read, which leads to less heat generated during the rebuild, faster rebuild times (no parity calculations, fewer reads, etc etc), and generally lower risk of "OMG everything is gone" when you have to rebuild a drive.
-
-
Well, yes, obviously any RAID works better when fully submerged in water (do you use salt water? it absorbs 77.2% more heat!).
</sarcasm>
We have had customers that do lose the RAID due to drive #2 (or even #3) failing during the rebuild process. Always fun to hear the fallout when they started the rebuild over our specific recommendation to take a backup first (which, of course, has its own "perils", but much lower risk operation than launching straight into the rebuild).
-
-
Aside: what's really disturbing is, in that blog post, he mentions "instead of horse laxative like others have done." I felt that deserved to be called out. You're welcome, everyone. Enjoy the rest of your weekend.
-
Aw, Discourse messed up. I've been intentionally online every 12 hours to avoid DiscoRounding and apparently it didn't work
97!!! I was so close.
-
NVM I'm TRWTF...it's backwards from what I expected. The most recent intervals are at the end.
-
Niiiice one.
-
Dude, just leave the browser open and never shut down your computer. I got like 100 days when I wasn't even reading or fucking posting here from "doing" that.
"doing" being in quotes because it doesn't involve actually doing anything at all.
-
@PJH, when you get a chance, could I see how I am doing on attendance? No rush.
-
It also costs more than.... okay, most of the machines in my home right now. I bet my work machine costs more if I were to buy it myself without the benefit of a corporate discount that I'm sure they've negotiated.
You could also pick up something more inexpensive like:
http://www.newegg.com/Product/Product.aspx?Item=N82E16822122135&ignorebbr=1&cm_re=PPSSPRONAS4--22-122-135--Product
Then if you add your own drives that you have laying around, you could consolidate all of your media in one spot for a fraction of what the other suggestion was.
-
I've been looking at the ReadyNAS 104 for a while - as I'm getting short of space on the 2x 1TB USB drives I have, it's making more and more sense.
-
NewEgg is running a special on Seagate 2TB drives until Sunday. They have them for $75 each (with promo code). You could have a NAS up and running for pretty cheap.
-
are they consumer grade or NAS/enterprise grade?
it makes a HUGE difference to a NAS solution.
-
Doesn't look like the same offer this side of the Atlantic, hard drives seem to have dropped a bit since the last time I looked though.
are they consumer grade or NAS/enterprise grade?
it makes a HUGE difference to a NAS solution.
I only use it for storing films and music and streaming over the network, and that's currently done from consumer USB drives. Can't see why it wouldn't still be fine on consumer drives.
-
if you are using them as really massive external storage, fine, but don't (please don't) get consumer grade hard disks for any sort of RAID setup.
you really are just asking for trouble.
reason being that consumer disks try to recover from errors by remapping sectors, this is not always seamless and those remapped sectors cause havoc on RAID settups and can even end up having massive corruption spread through the array before it finally falls over and nothing is recoverable.... not good..
-
Hadn't thought of that.
Seems like WD Red aren't that much more expensive, so isn't a massive pain to get those instead.
-
yeah. WD red are an excellent choice, as are the seagate NAS offerings and.... that third one that does the halfway decent drives that i never remembner the name of but always recognize....
-
-
It'll be the WD Red - have always bought WD drives and have only had one failure in about 16 years.
-
hat third one that does the halfway decent drives that i never remembner the name of but always recognize....
Shitatchi. Formerly IBM.
-
The people at BackBlaze, and myself, would disagree with you about using consumer drives in RAID.
-
consumer disks try to recover from errors by remapping sectors
My friend says all modern storage devices do that, including server-grade hard disks and SSDs and USB drives. Do you have a source saying otherwise?
-
the NAS ones do that too, yes. the difference is in how they do it.
NAS units will only attempt to recover a sector for a limited number of times and then signal read failure so the NAS can calculate what the sector that failed should have been from the parity informtation. the NAS drive will the remap the sector so that when the array writes that sector back out (because it constructed from parity) the information gets written to the new sector.
a consumer drive will keep trying for much longer causing one of two things to happen:
- the consumer drive will eventually report success and recover the sector but not remap it so that the NAS doesn't know the secotr is going bad and can handle appropriately, this is particularly bad when the recovered sector actually was corrupt but passed the checksum tests (that happens, not too hard when the checksum is only 8-32 bits of entropy depending on the HDD firmware)
- The consumer drive will take longer to service the request than the NAS is willing to wait and the NAS will mark the drive offline prematurely causing a drive failure. This is actually the most favorable situation as you r data is safe, even if you didn't need to replace that drive yet It is also far more rare.
-
It'll be the WD Red - have always bought WD drives and have only had one failure in about 16 years.
I've never had a HD of ANY brand last longer than 3 years. (So far so good on SSDs, though.)
I think they have a circuit in them that detects Blakeyrat-like activity and self-destructs. I have no idea why, but spinning HDs (and magnetic storage in general) hates me*.
* Pre-emptive DailyWTF joke: "haha just like all of us hate you, Blakeyrat! Seriously you are the worst, kill yourself now! Haha!"
-
I've never had a HD of ANY brand last longer than 3 years.
The older USB drive is coming up to 6 years old. Possibly another reason it might be beneficial to get the data moved onto something else.
-
I've never had a HD of ANY brand last longer than 3 years.
External drives around me (either spinners or flash (usb thumb drives) tend to last about a year before they crap out. almost always a controller issue. in most of the cases with the externals i can open the case, extract the HDD and plug it into a case with a free bay and get another year of service out of them.
Internal drives tens to last 4-5 years, because they tend to get tossed out after a thorough session with DBAN with the computer itself as i sell it to pay for part of the new one. no idea how much life is left in those guys but with the way i use them and after a double session with DBAN there's probably not much (which i do responsibly disclose to the person that buys the thing from me)
edit: for those that don't know: http://www.dban.org/
-
I had a colleague who used to keep a massive wall of old HDDs so that he could transplant the controllers from them in the event of a failure and at least recover the data. I personally think that keeping sane backups and testing them periodically to ensure that they're actually recoverable is preferable to breaking out the soldering iron but then I have shitty hand-eye coordination.
First thing I did when I took over his area was schedule those disks for a DBAN session (so glad I'm not an intern) and throw them out.
-
First thing I did when I took over his area was schedule those disks for a DBAN session (so glad I'm not an intern) and throw them out.
You know a ball-peen hammer's a lot quicker.
-
True but that's like two weeks of time I need to fill for said intern. If they'd hire dumber, slower ones I wouldn't have to find new things for them to do when they master the usual tasks.
-
Is it a secret test? Like the first one who pulls out a ball-peen hammer get a full-time position?
-
If he can find the shop and the tool chest and knows which end to hold, he's doing better than some of the maintenance people we used to hire.
-
We have had customers that do lose the RAID due to drive #2 (or even #3) failing during the rebuild process
I had a drive on a RAID 5 fail and then another drive failed during rebuild. Thankfully the second drive later decided to come back to life and the thing rebuilt successfully (avoiding a restore from tape).
The other drives were NOT replaced (on management orders) and were still spinning just fine when the thing was retired.Edit: I put a couple of years ago and my email history says 2006. Bloody hell time flies.
-
@Intercourse said:
@PJH, when you get a chance, could I see how I am doing on attendance? No rush.
u.username like 'Intercourse' username | start_range | end_range | days -------------+-------------+------------+------ Intercourse | 2014-06-18 | 2014-06-18 | 1 Intercourse | 2014-06-20 | 2014-06-22 | 3 Intercourse | 2014-06-25 | 2014-06-26 | 2 Intercourse | 2014-06-30 | 2014-07-04 | 5 Intercourse | 2014-07-06 | 2014-07-25 | 20 Intercourse | 2014-07-27 | 2014-07-27 | 1 Intercourse | 2014-07-29 | 2014-08-01 | 4 Intercourse | 2014-08-03 | 2014-08-28 | 26 Intercourse | 2014-09-01 | 2014-11-08 | 69 (9 rows) Elapsed: 0.348s Backup taken: 2014-11-08 13:05:21.472597
-
Hard Drive Reliability Update: September 2014
At Backblaze we now have 34,881 drives and store over 100 petabytes of data. We continually track how our disk drives are doing, which ones are reliable, and which ones need to be replaced.
BackBlaze uses nothing but consumer level drives to keep prices down on their backup service and they have the data to back them up.
As for myself, we tend to use whatever we can get cheapest for sandbox environments and even through all of the FUD about running consumer drives in RAID arrays, I have yet to have a RAID array catastrophically fail when using them. Storage servers with client data on them all use WD Reds or better, but honestly I would not have a problem building one with consumer drives. In my experience, if a drive is going to fail on us, it will do so when we burn the array in and just as BackBlaze says, it will fail hard.
Doesn't look like the same offer this side of the Atlantic, hard drives seem to have dropped a bit since the last time I looked though.
I keep forgetting that a lot of the people on here are not from the USA. You know how we colonists are, we think we are the world and all.
Try this link: http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834&utm_medium=Email&nm_mc=EMC-EXPRESS110814&cm_mmc=EMC-EXPRESS110814--EMC-110814-Index--InternalHardDrives-_-22148834-L01C
With this promo code to see if it works: EMCWWPB23
On this side of the Atlantic, it brings the price down to $74.99.
-
All drives will eventually fail. The main difference between consumer drives and enterprise drives is the rate of failure. Therefore, any comparison between systems built with consumer vs. enterprise drives will only be an economics question - does the increased failure rate cost us more in labor and downtime than is saved by buying drives cheaper.
A RAID array will only catastrophically fail when too may of it's components fail or when a component fails while it is rebuilding. If the rate of failure is higher, then the likelihood of this happening is higher. The fact that you haven't experienced one has more to do with your volume of experience rather than the relative likelihood of the event. It's more likely for someone to die in an aircraft accident that to win the lottery - if you met someone that won the lottery, it wouldn't change that fact. The point is that one person's experience isn't enough to get a handle on the risks associated with RAID failure.
-
The main difference between consumer drives and enterprise drives is the rate of failure.
Yeah, and if you had read the BackBlaze blog link I posted, their dataset shows that "Enterprise" drives fail more often than consumer drives in their application. A lot of this "common knowledge" that you espouse is just FUD.
The fact that you haven't experienced one has more to do with your volume of experience rather than the relative likelihood of the event.
So fucking what? BackBlaze has 34,881 drives, nearly all of them consumer drives at this point. I would never rely entirely on one RAID array to hold important data. I have also not said that I have never had a RAID array shit itself on me, I have, but I do not like it when people speak in absolutes. I have had a consumer level RAID array running for a few years now and it has not shit itself. I have others with WD Reds, several with SAS drives, etc. For home use, a RAID array with consumer drives will likely function just fine. But you should also have it periodically backed up somewhere else.
-
@Intercourse said:
Yeah, and if you had read the BackBlaze blog link I posted, their dataset shows that "Enterprise" drives fail more often than consumer drives in their application. A lot of this "common knowledge" that you espouse is just FUD.
Why would you insult someone who is agreeing with you? Beside, I haven't espoused any "common knowledge" in this thread. My only contribution was to mention that you were using a mismatched set of drives, which isn't supported by any RAID standard. Of course, after that, you asked me if I knew about RAID 2 through 4 - none of which support mismatched drive sizes.@Intercourse said:
So fucking what?
It means that your anecdote is pointless and you should stick to the BackBlaze data.Large corporations use enterprise drives because their labor is expensive and they are generally very conservative about spending money to protect data. By enterprise drives, I mean the ones hand selected by HP that sell for five times the cost of consumer drives, not the ones BackBlaze referenced. Besides, BackBlaze didn't say enterprise drives failed, they said that even if they had 99.999999999999% reliability, they still wouldn't be worth the extra cost. BackBlaze is also in a unique position where they will never lose data that valuable to them - only data that's valuable to their customers.
@Intercourse said:
but I do not like it when people speak in absolutes
I was trying to pipe in on your side -- then you shit on me. Good job.
-
Why would you insult someone who is agreeing with you?
Welcome to TDWTF. (Yes, I realize you have been around for a while, it is sarcasm)
asked me if I knew about RAID 2 through 4 - none of which support mismatched drive sizes.
RAID 4 does, it just cancels out the size of largest drive as that will be the parity disk. No distributed parity, so it can handle mismatches drives. While it is not very good for enterprise use, it is great for consumer level systems. UnRAID built a business upon it, althought I do not think they ever made much money with it?
Besides, BackBlaze didn't say enterprise drives failed
There was another link to follow there:
Enterprise Drives: Fact or Fiction?
Last month I dug into drive failure rates based on the 25,000+ consumer drives we have and found that consumer drives actually performed quite well. Over
I was trying to pipe in on your side -- then you shit on me. Good job.
I never saw you pipe in on my side, but I also never shit on you. If your skin is that thin, you might have a tough time around here.
-
I was trying to pipe in on your side -- then you shit on me. Good job.
@Intercourse said:
Welcome to TDWTF
the power of Jeff is strong with these two.....
-
I was trying to pipe in on your side -- then you shit on me. Good job.
That is the power of not speaking absolute, you can shit on both sides.
-
-
Recordable optical media (WORM) might work, but RW optical media has not had a particularly good retention track record...
Not to mention what is available has quite a low capacity at the moment. I was considering write-once bluray disks when I was looking for a home backup solution, though it starts stacking up (pun intended) quite a bit when you need to backup TB worth of data.
My current backup solution is just single drives. Data gets dumped onto them. Once a drive is full, it gets pulled, stored, and a fresh blank drive is slotted in.
