COVID-19 CovidSim Model
-
Major
s here:
Code Review of Ferguson's Model – The Daily Sceptic
by Sue Denim [Please note: a follow-up analysis is now […]
(h/t @Benjamin-Hall)
-
@boomzilla Already discussed
, but that's in the Garage. I agree it does warrant more public ridicule. That discussion isn't really Garage-y; maybe Jeff it here. 
-
I read/skimmed the article, and it sounded pretty damning.
However, then I read John Carmacks actual tweets that the article very selectively quoted. Here's the text from the four tweets:
The Imperial College epidemic simulation code that I helped a little on is now public: https://github.com/mrc-ide/covid-sim I am a strong proponent of public code for models that may influence policy, and while this is a "release" rather than a "live" depot, it is a Good Thing.
Before the GitHub team started working on the code it was a single 15k line C file that had been worked on for a decade, and some of the functions looked like they were machine translated from Fortran. There are some tropes about academic code that have grains of truth, but \
it turned out that it fared a lot better going through the gauntlet of code analysis tools I hit it with than a lot of more modern code. There is something to be said for straightforward C code. Bugs were found and fixed, but generally in paths that weren't enabled or hit. \
Similarly, the performance scaling using OpenMP was already pretty good, and this was not the place for one of my dramatic system refactorings. Mostly, I was just a code janitor for a few weeks, but I was happy to be able to help a little.
Now, that suddenly sounds a lot less bad. So, who do I trust?
Well, looking back at the article (and I feel bad for missing this initially), it's by an anonymous author at "lockdownsceptics.org". I'm sorry, but yeah, those guys sound like they have an axe to grind. I wouldn't count them as a reliable (or even useful) source of information at this point.
-
@cvi he didn't seem to say anything about the issue of "stochastic means we can never reproduce anything!" That alone is worthy of tar and feathers (of the modelers, not Carmack).
-
@boomzilla said in COVID-19 CovidSim Model:
@cvi he didn't seem to say anything about the issue of "stochastic means we can never reproduce anything!" That alone is worthy of tar and feathers (of the modelers, not Carmack).
Anyone doing Monte Carlo simulations or similar should know how to get exact reproduction of results. Seed those PRNGs. (Trivial for code in a single thread of a single process. Quite a lot more tricky with a distributed multithreaded application.)
OTOH, you probably don't want to do that the whole time (or even most of the time!); when exploring the outcomes of a model and working out the implications for policy, locking down the model to the point where it effectively only ever considers a single case is worse than useless.
-
@boomzilla I assume that is referencing one of the comments in this issue? Yeah, I can see why that doesn't sound great.
I can also understand what he's saying, and even potentially agree to it - the discrepancy in the example is only really a problem if it is consistently biased into a certain direction (which I would not think to be the cased based on my very brief perusal of the info in the bug reports). Don't get me wrong, yeah, that's pretty bad, and as @dkf says, if you do serious stochastic simulations, you should know enough to keep track of RNG states. (The good thing is that the bugs are being fixed, this one in particular seems to be here.)
I still maintain that the article you posted is so biased that it should be taken with a pinch of salt large enough to cause a 2-week long shortage at the supermarkets wherever you live.
Edit: Additionally, from those bug reports, it does not seem to be true that they can never reproduce anything. The RNG states were not properly saved when creating a cache file and later loading it. As far as I understood, rerunning the model from scratch (i.e., without using previously generated caches) seemed to be reproducible.
-
@cvi said in COVID-19 CovidSim Model:
I’m sorry, but yeah, those guys sound like they have an axe to grind. I wouldn't count them as a reliable (or even useful) source of information at this point.
On a personal level, I’d go further and suggest that all academic epidemiology be defunded.
You think so?
-
@topspin Missed that one when skimming the "article". (Glad I didn't waste more time on it, though.)
-
@cvi said in COVID-19 CovidSim Model:
rerunning the model from scratch seemed to be reproducible
That's good enough for scientific purposes. You can't transfer checkpoints between independent implementations of a model anyway. Or, usually, not even between runs of the same model on different hardware.
-
Pretty ridiculous that this so-called "Google engineer" doesn't know the first thing about Monte Carlo simulation, or how to use the damn Google to learn about it.
-
@Captain said in COVID-19 CovidSim Model:
Pretty ridiculous that this so-called "Google engineer" doesn't know the first thing about Monte Carlo simulation, or how to use the damn Google to learn about it.
That's not true--they even mention it and explain why it doesn't apply. They're explicitly not talking about the overall running of the program multiple times stochastically. That's normal. What isn't is not having replay ability. Even Monte Carlo simulations (which I've written a few of) can do that, if you re-seed it with the same thing. This one...can't. Run the same thing twice with the same seed (even single-threaded) and you get different answers. Like...quite a bit different. 80k deaths different. Which means you can't even test it after making refactoring changes, because you don't know which sources of randomness were intended and which were just brokenness.
That's bad. That's real bad. Add in the rest and you have a total mess where it's impossible to judge whether they actually implemented things correctly at all. And note, the released code is not the original (used to actually derive the published results). It's a rewrite/refactoring done by Microsoft guys. They've refused to release the original, actual code.
-
@Benjamin-Hall see what @cvi wrote (assuming he’s correct). It’s just the checkpoint restart that’s broken, starting from scratch is reproducible.
-
We, the undersigned software engineers, call for any papers based on this codebase to be immediately retracted. · Issue #165 · mrc-ide/covid-sim
The tests in this project, being limited to broad, "smoke test"-style assertions, do not support an assurance that the equations are being executed faithfully in discrete units of logic, ...
At risk of sounding repetitive), submit an issue reporting your observations so we can work on what you have observed. The code, as we run and test it, for given seeds, is deterministic when run single-threaded. Post-initialisation, it is deterministic for a given number of threads. There is one section of SetupModel.cpp function, well-discussed in other issues, which can produce statistically equivalent stochastic realisations, but not binary identical ones, using multi-threading. If determinism is required there, commenting out one line can achieve it.
The rest of the thread is pure Garage.
-
@topspin said in COVID-19 CovidSim Model:
@Benjamin-Hall see what @cvi wrote (assuming he’s correct). It’s just the checkpoint restart that’s broken, starting from scratch is reproducible.
That's not what I read, but I haven't actually done much beyond that. So maybe. Which would be better. But the rest of the code (based now on many independent people, including generally liberal types) is so poor in quality[1] as to render confidence in their models rather minimal. And you can look at the track record of Fergeson's group. So far they've been wrong on every single pandemic. Often by orders of magnitude, always high. So I'm much less disposed to give them the benefit of the doubt.
Basically, it sounds like the code I wrote as a graduate student. Totally hacked together, frequently modified, no tests other than running it and looking at the output, etc. Which is totally understandable as the basis for scientific exploration but rather less so as the basis for fundamentally transforming the world's economy and society.
Basically, I have zero trust in academic modeling work as applied to the real world. Economists? basically don't believe them. Epidemiologists? Data may be fine, but their models so far have sucked. Badly. Etc. I won't say anything about climate, as that's garage material. And I don't even believe that models are that useful for making decisions even when implemented properly (which few are), because they're so tied to their assumptions about values. And when that hits politics, the mess is even worse. Backward-looking "it seems to be spreading this fast" data collection is generally fine IMO, if subject to substantial uncertainties. Couple that with poor code and poorly-qualified models and you have
uncertainty!and you can basically get whatever you want out.[1] and remember, this isn't the original code. The original is by their own reports much much much worse.
-
@Benjamin-Hall Yes. But the way the guys on that Github thread are going about it? The endresult will be that no one will make their code public anymore unless forced to because those guys are way too hostile.
-
@Rhywden said in COVID-19 CovidSim Model:
@Benjamin-Hall Yes. But the way the guys on that Github thread are going about it? The endresult will be that no one will make their code public anymore unless forced to because those guys are way too hostile.
Sounds like some people need to put on their big persyn pants and deal with it. Scientists are brutal to each other on a daily basis. Have you read any referee reports? That rough-and-tumble is how the whole system is supposed to work. Science stops working well when people sugar-coat criticisms.
I'm in the "publishers and grant sources should require all code be public if your publication depends on that code" camp. And I do mean all. Code, documentation, data, etc. Unless it's under an NDA, in which case you should have to put it in a data escrow so that once the NDA runs out, others can check. Science needs to be replicatable if it wants to be trusted. "Trust us guys, we're the experts" doesn't fly with me. It breaks the whole basis for trust in science.
-
@Benjamin-Hall Yeah, insulting people will make them trust more. Just look at Trump - so much trust!
Great way to solve problems!
Also, I'm not talking about "sugar-coating". There's constructive criticism and then there's that. If you haven't learned the difference by now then I'm kind of doubting your teaching credentials.
-
@Rhywden said in COVID-19 CovidSim Model:
@Benjamin-Hall Yeah, insulting people will make them trust more. Just look at Trump - so much trust!
Insulting people? No. Insulting their work? You've never been in academia, have you? That's the norm there. Anyone offended by that can't have made it that far.
Great way to solve problems!
Also, I'm not talking about "sugar-coating". There's constructive criticism and then there's that. If you haven't learned the difference by now then I'm kind of doubting your teaching credentials.
Oh, there are certainly better and worse ways. But nothing I saw there was anything out of the norm for that environment. Which is totally toxic, to be sure. But that's nothing new. And publishing all your data and models would certainly not make it any more toxic. Trust has to be earned. And so far none of the models or "experts" have even attempted to do so--they've been working fully on credentialism. "I'm the expert, you have to believe me and do what I say because I have a degree." And that's even more toxic than any of the insults. Because that actually destroys lives and justifiably destroys trust in the work of genuine experts doing their best.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
@Rhywden said in COVID-19 CovidSim Model:
@Benjamin-Hall Yeah, insulting people will make them trust more. Just look at Trump - so much trust!
Insulting people? No. Insulting their work? You've never been in academia, have you? That's the norm there. Anyone offended by that can't have made it that far.
And you don't see how that kind of prevents people from cooperating? Up-to-date my results of obtaining cooperation through insults have been ... zero. Less than zero if you count adverse results.
Trust has to be earned.
You do not get people to trust you by slinging insults. You obviously fail to see the massive problems inherent with your way things ought to work.
Again: You want better software in science? Get them to cooperate. Because otherwise you get exactly the same results as now. When have insults ever worked for you when you wanted help?
All you get with this approach is a siege mentality.
-
@Rhywden said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
@Rhywden said in COVID-19 CovidSim Model:
@Benjamin-Hall Yeah, insulting people will make them trust more. Just look at Trump - so much trust!
Insulting people? No. Insulting their work? You've never been in academia, have you? That's the norm there. Anyone offended by that can't have made it that far.
And you don't see how that kind of prevents people from cooperating? Up-to-date my results of obtaining cooperation through insults have been ... zero. Less than zero if you count adverse results.
Trust has to be earned.
You do not get people to trust you by slinging insults. You obviously fail to see the massive problems inherent with your way things ought to work.
Again: You want better software in science? Get them to cooperate. Because otherwise you get exactly the same results as now. When have insults ever worked for you when you wanted help?
Telling people that their work is wrong is not an insult. It may be false, but it's not an insult. Unless your ego is so bound up in your work that you can't separate the two. And that's a serious problem in academia--the idea that criticism of your work is tantamount to an insult to your person is totally wack and totally common there. And is one of the key reasons that academic work is so often faulty. Because no one is willing to listen to criticism no matter how nicely put. The only people that academics listen to are the sources of their funding. So any change has to come from that angle. No one else's criticisms even matter at a functional level.
And several big medical funding sources have already done this and have gotten results. No reason the rest of the agencies couldn't do the same. That's political side, not science at all.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Telling people that their work is wrong is not an insult.
There's "This is wrong!" and there's "This is an utter failure, you should be ashamed and kill yourself!" The latter is hyperbole, yes, but it should finally get the point across where the difference lies.
Which is what I was going on about right from the start. If you cannot understand that then get the fuck out of teaching. I mean that. Because you should not be let near pupils within shouting distance.
Can you tell me that you understand that?
-
@Rhywden this isn’t the garage. Try to do as you say and tune down the insults.
-
@topspin I actually did that in full knowledge of the contradiction because I hope it drives the point across on how this particular model of communication is problematic.
-
@Rhywden said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
Telling people that their work is wrong is not an insult.
There's "This is wrong!" and there's "This is an utter failure, you should be ashamed and kill yourself!" The latter is hyperbole, yes, but it should finally get the point across where the difference lies.
Which is what I was going on about right from the start. If you cannot understand that then get the fuck out of teaching. I mean that. Because you should not be let near pupils within shouting distance.
Can you tell me that you understand that?
And where have I supported the latter? You seem to believe that I fully endorse everything that random people put on those things. I don't. Not at all. As I said, it's toxic. But normally so for academia. That's bad. Academia is toxic. And I never act like that--that's one major reason I'm glad I'm out of academia. But if you hide your data because you don't want to be insulted, you won't make it in academia as it is right now. Because that's how it is on a day to day basis. I've read referee reports that were way worse than any of that. And no one turned a eyelash at it.
I'm not saying it's right, I'm saying it's the norm. And I don't think it makes any difference to the people actually doing the research. They're doing it just as much. The complaints are purely tactical. Any change will have to happen at the level of the funding agencies.
And anyway, this is a total digression, so I'll stop there.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Sounds like some people need to put on their big persyn pants and deal with it. Scientists are brutal to each other on a daily basis. Have you read any referee reports? That rough-and-tumble is how the whole system is supposed to work. Science stops working well when people sugar-coat criticisms.
I doubt they are too fazed by a bunch of randos commenting on Github. For instance, the initial bugs that I linked to above seem to be from a review coordinated with/by e.g. Microsoft, where others were invited to vet the code.
I'm in the "publishers and grant sources should require all code be public if your publication depends on that code" camp. And I do mean all. Code, documentation, data, etc. Unless it's under an NDA, in which case you should have to put it in a data escrow so that once the NDA runs out, others can check. Science needs to be replicatable if it wants to be trusted. "Trust us guys, we're the experts" doesn't fly with me. It breaks the whole basis for trust in science.
I agree, except scratch the NDA part. If you publish something, then it's not under NDA. If there is an NDA preventing you from publishing, by definition, you're not. I would go one step further and require that the code is runnable and includes instructions+sample data to do so.
Much like democracy, academia might be the worst, except for all the other options. In this particular instance, I'm not seeing anybody else coming forward with their models.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Insulting people? No. Insulting their work? You've never been in academia, have you? That's the norm there. Anyone offended by that can't have made it that far.
Didn't we go through this exact thing decades ago with Linus Torvalds and eventually decide that you can criticize without being a dick?
-
@anonymous234 said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
Insulting people? No. Insulting their work? You've never been in academia, have you? That's the norm there. Anyone offended by that can't have made it that far.
Didn't we go through this exact thing decades ago with Linus Torvalds and eventually decide that you can criticize without being a dick?
Oh. You certainly can. But a lot of academia hasn't quite figured that out yet. A more nasty, undercutting, poisonous environment I have yet to find elsewhere. Rife with internal politics, unspoken rules of behavior (that usually reinforce existing power structures), and where subtext and "who knows who" is more important than actually doing the right thing.
And I still maintain that criticisms of work should not be taken as personal attacks. One of the only productive study groups I've ever been in was with a bunch of other junior/senior physics majors. We were brutal toward each others work, but everyone accepted that we weren't attacking each other. Everyone outside thought we were fighting tooth and nail. But it was super effective at teaching us how to do the work and how to collaborate/do group work effectively. Because everyone makes idiot mistakes. So being called out on doing something dumb isn't the same as being called dumb. And it's effective at teaching you to make fewer (not zero, but fewer) stupid mistakes. And being totally open with no sugar-coat makes it efficient. If you see a mistake, you point it out quick so that it doesn't become a habit. And you fully expect and appreciate others doing it back to you, so you don't get into ruts. It's a hard lesson to learn, to separate one's own ego from the work, but I think it's vital to being good at things and learning and growing.
If everyone has to dance around any criticism of work because it might be offensive, then lots and lots of good is lost. And people use that defensiveness as a weapon, to silence justified criticism. Or even critique. I maintain that people should just get over it. Your work is not you. You are not your work. Your value is entirely separate from the value of your work.
-
@topspin said in COVID-19 CovidSim Model:
@Benjamin-Hall see what @cvi wrote (assuming he’s correct). It’s just the checkpoint restart that’s broken, starting from scratch is reproducible.
How soon people forget the lessons we learned from Edward Norton Lorenz... Um. The lesson he learned nearly sixty years ago.
-
@Steve_The_Cynic it would be much easier to remember if not for all that bullshit about butterflies and tornadoes.
-
@cvi said in COVID-19 CovidSim Model:
I still maintain that the article you posted is so biased that it should be taken with a pinch of salt large enough to cause a 2-week long shortage at the supermarkets wherever you live.
It read to me like a rant someone here would make.
-
@Rhywden said in COVID-19 CovidSim Model:
@topspin I actually did that in full knowledge of the contradiction because I
hope it drives the point across on how this particular model of communication is problematic.can't control myselfYour explanation would be more believable on a day not ending in Y.
-
@boomzilla said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
I still maintain that the article you posted is so biased that it should be taken with a pinch of salt large enough to cause a 2-week long shortage at the supermarkets wherever you live.
It read to me like a rant someone here would make.
I find the code-related rants here typically to be better, perhaps because they're often written by people who actually have to deal with the code in question. And, thankfully, the code-rants here tend to be more out of genuine frustration and less as a political hit-piece.
-
@cvi said in COVID-19 CovidSim Model:
@boomzilla said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
I still maintain that the article you posted is so biased that it should be taken with a pinch of salt large enough to cause a 2-week long shortage at the supermarkets wherever you live.
It read to me like a rant someone here would make.
I find the code-related rants here typically to be better, perhaps because they're often written by people who actually have to deal with the code in question. And, thankfully, the code-rants here tend to be more out of genuine frustration and less as a political hit-piece.
Hmm...I didn't see it as a political hit piece, but I guess I can see why someone might have that view.
-
@Gąska said in COVID-19 CovidSim Model:
@Steve_The_Cynic it would be much easier to remember if not for all that bullshit about butterflies and tornadoes.
That's one confusing downvote.
-
@boomzilla said in COVID-19 CovidSim Model:
@Rhywden said in COVID-19 CovidSim Model:
@topspin I actually did that in full knowledge of the contradiction because I
hope it drives the point across on how this particular model of communication is problematic.can't control myselfYour explanation would be more believable on a day not ending in Y.
Sonnabend?
-
@boomzilla said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
@boomzilla said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
I still maintain that the article you posted is so biased that it should be taken with a pinch of salt large enough to cause a 2-week long shortage at the supermarkets wherever you live.
It read to me like a rant someone here would make.
I find the code-related rants here typically to be better, perhaps because they're often written by people who actually have to deal with the code in question. And, thankfully, the code-rants here tend to be more out of genuine frustration and less as a political hit-piece.
Hmm...I didn't see it as a political hit piece, but I guess I can see why someone might have that view.
It's definitely a hit piece, although not necessarily political.
-
@Gąska said in COVID-19 CovidSim Model:
@boomzilla said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
@boomzilla said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
I still maintain that the article you posted is so biased that it should be taken with a pinch of salt large enough to cause a 2-week long shortage at the supermarkets wherever you live.
It read to me like a rant someone here would make.
I find the code-related rants here typically to be better, perhaps because they're often written by people who actually have to deal with the code in question. And, thankfully, the code-rants here tend to be more out of genuine frustration and less as a political hit-piece.
Hmm...I didn't see it as a political hit piece, but I guess I can see why someone might have that view.
It's definitely a hit piece, although not necessarily political.
Yep.
-
@cvi said in COVID-19 CovidSim Model:
I would go one step further and require that the code is runnable and includes instructions+sample data to do so.
Steady on there! Having it so that the code is easily runnable by others and includes both (non-
) instructions and sample data is not something that just happens by magic. For a very large fraction of scientific code, it most certainly isn't that by default; it's often full of assumptions that work well on one specific system but which don't apply anywhere else (e.g., hard coded paths, assumed library versions). It takes effort to make the code portable enough to reliably run on someone else's computers (that's a large part of what my sort of job is really about). Writing decent documentation is hard too (and that's not usually a part of the scientist's job; they get career advancement from the published results, not the README of the code that produced them). A sample dataset is a good thing to have (assuming there's some systematic testing; a great deal of code for PhDs lacks that before the final six months writeup) but the one they're using internally might be encumbered by PII or, more commonly, be too damn large to ship about a lot. That's tricky, because all too often you get significant problems in codebases that simply don't show up at all with small inputs.Right now, we're talking about code that's making the journey from being a codebase used in a single situation to something more generally valuable, work that is the kind of job I do and isn't usually the job of the original author of the code (different career path). They're doing it more in the open than usual, which is laudable in my eyes. What they do not need is politically-motivated know-nothing carping from asshats who hate the policy implications of the results.
(FWIW, my response to checkpointing not working would be to just delete the checkpointing support. Want reproducibility? Rerun from the beginning like Real Men, mothafukkas!)
-
@Gąska said in COVID-19 CovidSim Model:
It's definitely a hit piece, although not necessarily political.
Oh, I think it is political, to support someone who really desperately wants an end to economic restrictions (probably in the US, but such people are elsewhere in the world too so it isn't certain), though not necessarily on the standard Left/Right axis you see in Europe or Dem/Rep axis you see in the US. However, the playbook (discredit all scientists who produce evidence for policy you don't like, often by finding and encouraging useful cranks) is one that's been established well for quite a few industries; similar approaches have been seen with tobacco, energy, and sugar for sure, and probably others. I don't know which scumbag thinktank/lobbying firm is involved this time, and as I regard them as interchangeable, I don't care that much either. Some investigative journalist can hunt that down…
-
@dkf This is the same code that has been used (by this group) for all the recent pandemics. And has been wrong each and every time.
I'm more interested in having the code/data as a fixed point, so that people can't go back later and "change the code/data to fit later data". Because replication is important, and these code-based models never can have enough details in the papers to replicate independently. They're so tightly implementation-dependent. Or at least the ones I used in quantum chemistry were. If you tried to re-implement our mixed quantum-classical model (which had exhaustive papers about the math), you'd end up somewhere similar...ish. Maybe.
Instead, I'd have everything available on a public repo, hosted by someone else. You want to publish? Publish everything.
-
@dkf said in COVID-19 CovidSim Model:
@Gąska said in COVID-19 CovidSim Model:
It's definitely a hit piece, although not necessarily political.
Oh, I think it is political, to support someone who really desperately wants an end to economic restrictions (probably in the US, but such people are elsewhere in the world too so it isn't certain), though not necessarily on the standard Left/Right axis you see in Europe or Dem/Rep axis you see in the US. However, the playbook (discredit all scientists who produce evidence for policy you don't like, often by finding and encouraging useful cranks) is one that's been established well for quite a few industries; similar approaches have been seen with tobacco, energy, and sugar for sure, and probably others. I don't know which scumbag thinktank/lobbying firm is involved this time, and as I regard them as interchangeable, I don't care that much either. Some investigative journalist can hunt that down…
Meh. We know the model was garbage as a policy tool (aside from its propaganda value if people believed it) already. This just gives us a peek under the covers at some of the technical details.
-
@Benjamin-Hall Dude, you still. do. not. get it.
Jesus Christ. What do they teach you guys over there? Don't you have courses on how to communicate in a classroom? I mean, practical ones where your speech patterns and way of communicating get analyzed so you get feedback on how not to "kill" a student?
Again: Criticism is one thing. The way you criticise is another. And if you choose the wrong tone then you might as well discuss with a wall for all it's worth. Or you may just have turned off a student from Science or Math forever because you just reinforced their belief that they're too dumb - so instead of working on a problem you'll simply get someone who does his or her (mostly her) best to pass unnoticed. Or will drop out. If that's your goal then ... well, there's no nice way to say it: But you're not a teacher in that case.
I'm NOT advocating for no criticism.
-
@Rhywden I have never said anything about classrooms. Those are completely different from academia (my topic). It's only your own shoulder aliens that have made this anything about classrooms.
The appropriate mode of criticism varies tremendously based on context (something you are completely 100% immune to). Colleagues and peers are (and should be) treated differently than students. The kinds of criticism I see there are what peers do to other peers every single day in academia. Sure. They're not appropriate for dealing with students. But these aren't students. They're world-famous professors, the "cream of the crop". And they did a boner. Several, in fact. And their model hasn't ever worked. At that point, being harsh and direct has value.
-
@dkf said in COVID-19 CovidSim Model:
Steady on there! Having it so that the code is easily runnable by others and includes both (non-) instructions and sample data is not something that just happens by magic. For a very large fraction of scientific code, it most certainly isn't that by default; it's often full of assumptions that work well on one specific system but which don't apply anywhere else (e.g., hard coded paths, assumed library versions). It takes effort to make the code portable enough to reliably run on someone else's computers (that's a large part of what my sort of job is really about). Writing decent documentation is hard too (and that's not usually a part of the scientist's job; they get career advancement from the published results, not the README of the code that produced them). A sample dataset is a good thing to have (assuming there's some systematic testing; a great deal of code for PhDs lacks that before the final six months writeup) but the one they're using internally might be encumbered by PII or, more commonly, be too damn large to ship about a lot. That's tricky, because all too often you get significant problems in codebases that simply don't show up at all with small inputs.
The state of academic code reminds me of stories about, say, chemistry papers where knowledgeable chemists can't figure out how to repeat an experiment in a published paper, as I've previously read Derek Lowe bitching about.
It's understandable that academics can't write tidy code. It's difficult enough for people who do nothing but write code for a living. Nevertheless, it means that any peer review (which is really not worth very much) is not very valid and prevents anyone attempting to reproduce the results (how we really separate the scientific wheat from the chaff).
I think it's good for people to get an occasional peek behind the curtain like this. We place way too much trust in this sort of thing as it is.
-
-
@error thanks for reminding me.
-
@boomzilla said in COVID-19 CovidSim Model:
it means that any peer review is not very valid
There's been a very long argument about reproducibility in academic circles across many disciplines, with such fun things as journals having public policies to require it (at least minimally) yet not actually taking any steps to enforce it even when presented with flagrant disregarding of the rules. When there are rules in place, but effectively systematic exceptions are granted to the rules, what point is there in having those rules in the first place?
It seems to be one of these generational things. The only way this is going to be fixed is by having senior scientists retire. (I'd guess that similar things were present in software development, except we went through it earlier; the incentives for software devs are much more slanted towards being open these days, even though obviously not everything can be open.)
-
@Benjamin-Hall
statements as "“Stochastic” is just a scientific-sounding word for “random”." makes me dislike the tone of the article. It's the level of "yo mama"-jokes.To top it all off: It was ignorance from the writer how the software (or any software) operates under specific conditions.
Happens to everyone. But in this case, would bring shame on the writer, because of said tone.
-
@Flips Yeah, plus there is a misunderstanding of how models are even used for policy decisions.
Even a flawed model can provide information. For example, the Atlanta Fed has a GDP growth model that is "notorious" for being systematically over the real figure. It's "reliably" wrong, and actually provides a decent upper bound.
Models work for the same reason markets and democracy do. Because even though each can get it wrong, they can still broadly agree.
-
@Captain said in COVID-19 CovidSim Model:
Yeah, plus there is a misunderstanding of how models are even supposed to be used for policy decisions.
I agree that bad models can tell you interesting things if you understand their limitations. I don't think that scenario applies here.
