“One of the greatest miscarriages in British judicial history”
-
Forgive the hyperbolic headline but it’s fairly accurate. I’m just amazed there isn’t a topic about this yet because it’s absolutely the sort of shitshow we talk about here.
I don’t have a good place to start with this so I guess I’ll start with the take from one of the satirists at the Guardian - https://www.theguardian.com/commentisfree/2022/feb/15/post-office-scandal-workers-computer-system
Some stories feel so unbelievable that every time you think of them again, you have to sit with the basic concept for a few moments just to remind yourself how truly, staggeringly outrageous the whole business is.
This absolutely is a shitshow. We’re at the stage where 736 people in various parts of the country running what amounts to small businesses that collectively support the physical mail infrastructure were routinely being accused of fraud, and later prosecuted for it. Plus many many hundreds more people caught up in this mess but weren’t serious enough to get charged with something.
Individual cases where the Royal Mail were accusing these small businesses of owing thousands of pounds, which they were personally expected to make up the “shortfall”.
Except they didn’t. Computer glitches, so many glitches. Only now, 20 years on in some of the cases, is there even recognition that the computer was wrong.
But those people, their families, their livelihoods ruined. Some imprisoned for fraud. Some of those took their own lives.
I’m glad that nothing I do is important in the scheme of things. I don’t know if I could deal with being responsible for the sorts of mess this produced. But the CEO of this… is fine. Not even an actual apology but a classic non-apology - as though setting some money aside for damages claims is adequate.
As for the wider lessons of the scandal, what a lot it says about a society crossing the threshold of the third millennium that thousands of entirely upstanding human beings were disbelieved in favour of trusting a computer.
It should be noted that there was a lot of evidence that the computing system wasn’t working correctly but the Post Office ignored it entirely, a move later noted by one of the judges presiding over the eventual inquiry as a comparison to Flat Earthers’ maintained outlook on the universe.
So that’s the short version, but of course it all feels inadequate. Even simply listing every individual injustice in the sparsest possible terms would take far more space than I have available; at even cursory depth, every single story is utterly heartbreaking and utterly extraordinary.
I don’t believe the human race, collectively, deserves to live if this is how we behave, bereft of thinking, compassion, integrity and ethics, when we will assume the computer is always right.
-
@Arantor It's been reported on by The Register for years. Not a mainstream outlet by any means. But, well, you can get the picture of how long this has been going on. There's even more if you bother to search their arhchives. I think one even had a description of what exactly went wrong in the technical sense.
2013:
2015:
2017:
2019:
2021:
39 Post Office convictions quashed after Fujitsu evidence about Horizon IT platform called into question
Japanese firm has a reckoning to face
-
@Arantor said in “One of the greatest miscarriages in British judicial history”:
I don’t believe the human race, collectively, deserves to live if this is how we behave, bereft of thinking, compassion, integrity and ethics, when we will assume the computer is always right.
QFT.
Reminds me of the famous quote/question about whether inputting the wrong figures will still give the right answer. It feels like a lot of people are assuming that it does, and missing the point that even inputting the right figures might give you the wrong answer, because machines aren't made perfect. Hell, even if one inputs the right figures and get the right answer, one might not be understanding what the right answer is. (And, one might argue, that even if the right figures produce the right answer and the answer is understood fully, it might still point towards the wrong course of action.)
-
@acrow yeah, I’ve known about it for a while (and I have been reading El Reg since… 2000?) but it kind of took seeing the summary article to really ran it home how fucked up this really is.
-
@Arantor Yes.
There was another similar case. Several people were imprisoned for fraud. They had purchases on their card statement, but claimed that they never made these purchases. Then finally one of these people's lawyers found tech assistance, and they could prove to the court that the purchases had been accepted without the PIN code, or something. Which was supposed to be impossible. Several people had already been jailed based on this "impossibility". Some for a long time.
Can't find that article right now. But, y'know, it's not the first or the last time, even among cases we know of...
-
@Arantor said in “One of the greatest miscarriages in British judicial history”:
I’m glad that nothing I do is important in the scheme of things.
I usually tell myself that (though not when I'm dealing with clinical software). But those engineers at Fujitsu probably thought the same thing about their tasks.
-
@Watson probably, though in my defence what I do is not unimportant either. It’s just a matter of scale; if I get something wrong, people will be inconvenienced and in some cases that could be profoundly life altering - or it might not.
My current world involves such lofty heights as whether luggage is getting shipped on the correct day - and while it is possible for something to go markedly wrong, it isn’t the kind of thing where this kind of thing can be systematically compromised in this way. We’re a little company shipping software for other little companies and none of it is on the scale that the world burns if I screw up.
But I have friends who do medical software and I know some folks in oil and gas - I couldn’t do what they do because I’d be forever scared of “but what if I make a mistake and people die?”
And yet in those industries, some level of “accidental death by software” is priced in, as I understand it. I can safely say that PHP shouldn’t ever be the cause of that…
-
@Arantor said in “One of the greatest miscarriages in British judicial history”:
But I have friends who do medical software and I know some folks in oil and gas - I couldn’t do what they do because I’d be forever scared of “but what if I make a mistake and people die?”
And yet in those industries, some level of “accidental death by software” is priced in, as I understand it. I can safely say that PHP shouldn’t ever be the cause of that…Having dealt with software people in the oil industry and in the railway signalling business, where I happen to know that some of my software is used, they said quite clearly that it was part of their jobs to show that the overall system was fit for purpose, not that each and every part had to be specifically certified. Which is good; while I try to make sure my bits don't leak memory or have unexpected behaviours, no way I'm I ever going to certify their fitness for any purpose (other than to be bits of source code you can look at if you print it out).
Money is another thing I don't want to write software to work with.
-
These are all very good reasons for having a whoopsie insurance as a consultant. If you don't, shit could get very expensive.
-
@dkf I used to work for a part of Lehman Brothers. I made sure to keep out of any code wrangling other than the occasional report (with clearly explained SQL signed off by the MIS folks). Better life that way.
-
@Arantor said in “One of the greatest miscarriages in British judicial history”:
I couldn’t do what they do because I’d be forever scared of “but what if I make a mistake and people die?”
When I was in university, I was offered a paid internship with a company that was making (or trying to make; I suspect they failed) active suspension for motorcycles; I would have been working on the control electronics. This internship had the distinct advantage, relative to other internships I was offered, of being able to live at home and thus save more money for school. However, I declined it and accepted one of the others for exactly this reason.
(It turned out there were other good reasons, which I didn't know until later, for having declined it. Their entire engineering department consisted of interns, with only one real, degreed engineer supervising them. They did not actually pay the "paid" interns. And they also didn't fill out the paperwork needed for the interns to get academic credit for their internships. They wound up getting banned from the internship program at that university.)
-
@HardwareGeek said in “One of the greatest miscarriages in British judicial history”:
active suspension
Reminds me. Bose sold the invention some years ago. I wonder, has anything come from it? Didn't seem to have any downsides other than being bulky and power-hungry, which might have been a problem in the 80s and 90s, but shirley not anymore? If anything, His Muskiness should have taken a note.
-
-
this hit the market last year: https://www.ridefox.com/content.php?c=livevalve-bike
-
@cvi said in “One of the greatest miscarriages in British judicial history”:
@Arantor said in “One of the greatest miscarriages in British judicial history”:
I don’t believe the human race, collectively, deserves to live if this is how we behave, bereft of thinking, compassion, integrity and ethics, when we will assume the computer is always right.
QFT.
Reminds me of the famous quote/question about whether inputting the wrong figures will still give the right answer. It feels like a lot of people are assuming that it does, and missing the point that even inputting the right figures might give you the wrong answer, because machines aren't made perfect. Hell, even if one inputs the right figures and get the right answer, one might not be understanding what the right answer is. (And, one might argue, that even if the right figures produce the right answer and the answer is understood fully, it might still point towards the wrong course of action.)
That's what founds my opinion that "black box decisions" by computers for anything important should be illegal. Period.
If you can follow the data trail and arrive at the same conclusion (albeit more slowly), fine. If you cannot explain to at least a domain expert how exactly your system arrived at its conclusion: Hard nope.
-
@cvi said in “One of the greatest miscarriages in British judicial history”:
Reminds me of the famous quote/question about whether inputting the wrong figures will still give the right answer
-
I've been waiting to go off about the Enclosure Act for some time and this thread is... oh.
-
@Gribnit said in “One of the greatest miscarriages in British judicial history”:
I've been waiting to go off about the Enclosure Act for some time and this thread is... oh.
Oh, is that why it seems like all of the UK is criss-crossed with little stone walls all over the place.
-
@Rhywden said in “One of the greatest miscarriages in British judicial history”:
That's what founds my opinion that "black box decisions" by computers for anything important should be illegal. Period.
And that's theoretically covered by the GDPR. But only theoretically.
-
@Rhywden said in “One of the greatest miscarriages in British judicial history”:
That's what founds my opinion that "black box decisions" by computers for anything important should be illegal. Period.
If you can follow the data trail and arrive at the same conclusion (albeit more slowly), fine. If you cannot explain to at least a domain expert how exactly your system arrived at its conclusion: Hard nope.
How did you arrive at that conclusion?
I put the numbers in the magic box?
That's illegal. How exactly did you arrive at that conclusion?
I applied this machine learning algorithm that uses this set of matrices with 100k elements, and that popped the answer out of the end. It was trained on this set of 13 million prejudiced and biased data items, so its decisions are obviously perfect.
-
@dkf came here to say this, but for once read to the end.
I would just like to add to "prejudiced and biased" data items those that they were never supposed to have in the first place and maybe even deleted ("voluntarily" or by court order) later, but still remain enshrined in their ML matrices.
-
@topspin said in “One of the greatest miscarriages in British judicial history”:
@dkf came here to say this, but for once read to the end.
I would just like to add to "prejudiced and biased" data items those that they were never supposed to have in the first place and maybe even deleted ("voluntarily" or by court order) later, but still remain enshrined in their ML matrices.
So no different than right now with HL except maybe more consistent.
-
@boomzilla said in “One of the greatest miscarriages in British judicial history”:
So no different than right now with HL except maybe more consistent.
Except that everybody knows that humans are full of shit. Some people haven't quite figured that out w.r.t. computers/ML.
-
@cvi Some people haven't quite figured it w.r.t. humans, too
-
@Applied-Mediocrity said in “One of the greatest miscarriages in British judicial history”:
@cvi Some people haven't quite figured it w.r.t. humans, too
Tbf we're mostly working on the error bars now.
