COVID-19 CovidSim Model
-
Re-open and unlock issue #144 (Publish original source code) · Issue #179 · mrc-ide/covid-sim
Issue #144 has been prematurely closed as the original C source code has not been published. Please re-open and unlock the issue so the community can provide feedback and discuss the comments made ...
full of responses from someone I can only assume (based on time zones) is a Russian troll
-
@boomzilla said in COVID-19 CovidSim Model:
@Captain said in COVID-19 CovidSim Model:
Yeah, plus there is a misunderstanding of how models are even supposed to be used for policy decisions.
I agree that bad models can tell you interesting things if you understand their limitations. I don't think that scenario applies here.
And many of the bad models that're good for policy decisions (but shouldn't be let out into public because the media/people will completely ignore any caveats) become actively harmful if used as political fuel. For example, presenting a known "high-side estimate, if everything goes wrong (plus usual caveats)" as "this will absolutely happen unless we do THAT". Which is how this particular model was used in practice. To stampede politicians into doing sudden, harsh lock-downs to prevent the otherwise-inevitable millions of deaths.
-
@Flips quoted in COVID-19 CovidSim Model:
"“Stochastic” is just a scientific-sounding word for “random”."
At that level of discussion it might as well have been “Oh, I've put a random number generator in this. I'm so stochastic now!”
In reality, it's about not just having some random numbers, but also thinking about the quality of the random numbers (making a good RNG is really hard, and there are a lot of crappy ones about), reproducibility when requested (typically by supplying seeds), and ensuring that there's no correlations between the various random sources. And then you have to sample the result many times to get at least a statistical prediction of the likely span of outcomes; the more runs you do, the better the characterisation of the true behaviour of the model itself, at least at a statistical level. The detail will be lacking, but you probably don't want every last detail of the internals of the model when taking decisions; you won't have (most of) that detail with real life either.
We'll often replace a complex model with a stochastic input instead. But it'll not be just random, but actually random with the right statistical patterns: for example, if a complex model effectively just produces output that is a sequence of essentially independent events with an average rate of arrival, a simple Poisson event source is a good approximation, and that works very well… until you start dealing with correlations in the input, but then the basic assumption (no correlations) doesn't hold any more. In simple studies, that's not too important. In complex ones, you instead do the work to couple up multiple models so that you get the interesting correlations in the outputs of one part that the the next part has to deal with. But complex studies take an absolute load more work to set up, especially as it is rare indeed to find anyone who is an expert on all the parts of the problem. So you have to collaborate, and that immediately quadruples the organisational complexity at least.
Which is all very well… until you get some random yahoo come along and claim that, because they can write a fucking login page for a tiny charity website, they are then obviously experts on writing complex scientific codes and that it's being done wrong because its using too much Fortran and not enough jQuery.
-
@dkf said in COVID-19 CovidSim Model:
Which is all very well… until you get some
randomstochastic yahoo come along
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
And many of the bad models that're good for policy decisions (but shouldn't be let out into public because the media/people will completely ignore any caveats) become actively harmful if used as political fuel. For example, presenting a known "high-side estimate, if everything goes wrong (plus usual caveats)" as "this will absolutely happen unless we do THAT". Which is how this particular model was used in practice. To stampede politicians into doing sudden, harsh lock-downs to prevent the otherwise-inevitable millions of deaths.
So the harsh lock-downs are instead preventing the otherwise-inevitable death of hundreds of thousands.
Luckily, this is America so we'll soon be able to see what happens when states open too early, like what's going on in Sweden these days. :-/
But what do I know? Maybe the Governor of Florida has better data than I do (probably). I don't know that I'd trust Florida man with making health choices for me (lmao).
-
@Captain said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
And many of the bad models that're good for policy decisions (but shouldn't be let out into public because the media/people will completely ignore any caveats) become actively harmful if used as political fuel. For example, presenting a known "high-side estimate, if everything goes wrong (plus usual caveats)" as "this will absolutely happen unless we do THAT". Which is how this particular model was used in practice. To stampede politicians into doing sudden, harsh lock-downs to prevent the otherwise-inevitable millions of deaths.
So the harsh lock-downs are instead preventing the otherwise-inevitable death of hundreds of thousands.
Luckily, this is America so we'll soon be able to see what happens when states open too early, like what's going on in Sweden these days. :-/
But what do I know? Maybe the Governor of Florida has better data than I do (probably). I don't know that I'd trust Florida man with making health choices for me (lmao).
considering florida is doing real darn well compared to other states that did full heavy lockdowns....
-
@dkf said in COVID-19 CovidSim Model:
Steady on there! Having it so that the code is easily runnable by others and includes both (non-) instructions and sample data is not something that just happens by magic.
Yeah, I know. I've been on both sides (All three even: producing unmaintainable research code, fixing up other people's unmaintainable code, and finally, fixing up my own unmaintainable research code for general consumption. The last one really is salt in the wounds.)
Right now, we're talking about code that's making the journey from being a codebase used in a single situation to something more generally valuable, work that is the kind of job I do and isn't usually the job of the original author of the code (different career path). They're doing it more in the open than usual, which is laudable in my eyes. What they do not need is politically-motivated know-nothing carping from asshats who hate the policy implications of the results.
Agreed.
As an aside, frankly, we need more people like you. As far as I understand, the UK is actually ahead on this front. Fortunately, other places are starting to copy that model (or are at least discussing doing so).
@dkf said in COVID-19 CovidSim Model:
It seems to be one of these generational things. The only way this is going to be fixed is by having senior scientists retire. (I'd guess that similar things were present in software development, except we went through it earlier; the incentives for software devs are much more slanted towards being open these days, even though obviously not everything can be open.)
There is some additional noise. Journals start rewarding releasing (working) code; I think the ACM has an initiative for this. Right now it's at the level of badges, but at least something. I've seen funding agencies specifically asking for reporting of open source contributions as part of their evaluation and reports.
-
@Captain said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
And many of the bad models that're good for policy decisions (but shouldn't be let out into public because the media/people will completely ignore any caveats) become actively harmful if used as political fuel. For example, presenting a known "high-side estimate, if everything goes wrong (plus usual caveats)" as "this will absolutely happen unless we do THAT". Which is how this particular model was used in practice. To stampede politicians into doing sudden, harsh lock-downs to prevent the otherwise-inevitable millions of deaths.
So the harsh lock-downs are instead
preventingdelaying the otherwise-inevitable death of hundreds of thousands.
If you believed the "flatten the curve" strategy. Then, of course, there's the recent revelation in NYC that most new cases were people who from all accounts had stayed at home.Luckily, this is America so we'll soon be able to see what happens when states open too early, like what's going on in Sweden these days. :-/
Another example demonstrating that old people homes really should be locked down, true, but not the point that I think you were making, which also ignores places like South Dakota that also never shut down.
\
-
@boomzilla said in COVID-19 CovidSim Model:
@Captain said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
And many of the bad models that're good for policy decisions (but shouldn't be let out into public because the media/people will completely ignore any caveats) become actively harmful if used as political fuel. For example, presenting a known "high-side estimate, if everything goes wrong (plus usual caveats)" as "this will absolutely happen unless we do THAT". Which is how this particular model was used in practice. To stampede politicians into doing sudden, harsh lock-downs to prevent the otherwise-inevitable millions of deaths.
So the harsh lock-downs are instead
preventingdelaying the otherwise-inevitable death of hundreds of thousands. If you believed the "flatten the curve" strategy. Then, of course, there's the recent revelation in NYC that most new cases were people who from all accounts had stayed at home.That was an interesting factoid, but it's meaningless without more data. Of course the largest population was the largest source. But on a "per capita" basis?
Luckily, this is America so we'll soon be able to see what happens when states open too early, like what's going on in Sweden these days. :-/
Another example demonstrating that old people homes really should be locked down, true, but not the point that I think you were making, which also ignores places like South Dakota that also never shut down.
\I'm glad South Dakota didn't shut down if it didn't have to. Florida, on the other hand, has around 5 times the population of Oregon, 15 times the infection rate, and close to 50 times the death rate.
-
@Captain said in COVID-19 CovidSim Model:
@boomzilla said in COVID-19 CovidSim Model:
@Captain said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
And many of the bad models that're good for policy decisions (but shouldn't be let out into public because the media/people will completely ignore any caveats) become actively harmful if used as political fuel. For example, presenting a known "high-side estimate, if everything goes wrong (plus usual caveats)" as "this will absolutely happen unless we do THAT". Which is how this particular model was used in practice. To stampede politicians into doing sudden, harsh lock-downs to prevent the otherwise-inevitable millions of deaths.
So the harsh lock-downs are instead
preventingdelaying the otherwise-inevitable death of hundreds of thousands. If you believed the "flatten the curve" strategy. Then, of course, there's the recent revelation in NYC that most new cases were people who from all accounts had stayed at home.That was an interesting factoid, but it's meaningless without more data. Of course the largest population was the largest source. But on a "per capita" basis?
Huh? Why "of course?"
Luckily, this is America so we'll soon be able to see what happens when states open too early, like what's going on in Sweden these days. :-/
Another example demonstrating that old people homes really should be locked down, true, but not the point that I think you were making, which also ignores places like South Dakota that also never shut down.
\I'm glad South Dakota didn't shut down if it didn't have to. Florida, on the other hand, has around 5 times the population of Oregon, 15 times the infection rate, and close to 50 times the death rate.
And? I posted an article (over in the General thread) showing negative (weak) correlations between locking down and stuff like death rates. Maybe in discussions like this it gets used as shorthand for other social distancing and hygiene measures.
-
@Captain said in COVID-19 CovidSim Model:
\
I'm glad South Dakota didn't shut down if it didn't have to. Florida, on the other hand, has around 5 times the population of Oregon, 15 times the infection rate, and close to 50 times the death rate.
Uh...wat?
Let's use cases and deaths/1M population:
Oregon: 1224 cases, 29 deaths (both /1M)
Florida: 1825 cases, 78 deathsThat's...1.5x as many cases and 2.7x as many deaths. In a much (much much much much much) older, much denser population. And those deaths are clustered basically entirely in two counties (both in the Miami area). And then largely in nursing homes and care facilities (see "much older population"). As well as having much looser restrictions and much much much more international travel and cruises.
You're only off by an order of magnitude there buddy.
-
@Captain said in COVID-19 CovidSim Model:
Of course the largest population was the largest source. But on a "per capita" basis?
Some quick napkin math
Population of New York State: about 20M (Wikipedia)
The closest is Florida with about 21M (Wikipedia)
Covid cases/deaths (from here)
New York: 340k/26k
Florida: 39k/1.6k
-
@boomzilla said in COVID-19 CovidSim Model:
Huh? Why "of course?"
Maybe "of course" was too strong, but it shouldn't be surprising that 2/3 of infections/hospitalizations are coming from 2/3 of the population.
Especially since the remaining third is "expected" to have a high concentration of infections, so that a smaller population of essential workers are slowly infecting a larger population of WFH and laid off etc.
-
@boomzilla said in COVID-19 CovidSim Model:
@Captain said in COVID-19 CovidSim Model:
Of course the largest population was the largest source.
Huh? Why "of course?"
Maybe "unsurprisingly" would a more precise way of saying it.
-
@Benjamin-Hall I only compared the daily infections and death rates as raw numbers and said so. So chill out.
You're still proud of 3x as many deaths huh.
-
@Captain said in COVID-19 CovidSim Model:
@Benjamin-Hall I only compared the daily death rates as raw numbers and said so. So chill out.
You mean you presented knowingly misleading data. And even then, you're cherry picking individual days (where the noise is nearly 50%). So yeah, you fit in this thread just fine. Misleading, tendenciously-presented data used to slime people you don't like. Great job.
-
@Benjamin-Hall lol go make your accusations somewhere else loser.
Still proud of 3 times the death huh.
-
@cvi said in COVID-19 CovidSim Model:
As far as I understand, the UK is actually ahead on this front.
Yes. My boss has been working very hard on that front, along with people at other UK universities. One of the key things they've done has been to persuade universities that what they were really just doing was officially recognising the facts on the ground: there were already numerous people acting as staff programmers for researchers (often in research assistant posts, but also with quite a lot of abuse of the roles of people doing PhDs) so having it as an actual career path with the usual metrics of all that is good for staff retention (a problem many places were having; it's still hard even with that due to the sheer demand for software engineering from the rest of the global economy). We've got as far as having a professional association and getting everything on the official rules for HR and so on, though I don't think we've had a formal round of role grade fitting yet (that sort of thing is done on a multi-decade frequency so that's not surprising, and I wouldn't expect much change to the facts on the ground there).
It's been working out very nicely, but it's been helped a lot by being able to describe it as recognition of reality rather than saying it's something that doesn't exist but should. Everything in my career indicates that “Paving the Cowpaths” arguments have far more force than almost everything else you can possibly raise; as the saying goes, “it's better to ask for forgiveness than for permission” and that's got the precise same root. (If this helps others get their situations sorted out, so much the better.)
-
@Flips said in COVID-19 CovidSim Model:
@Benjamin-Hall
statements as "“Stochastic” is just a scientific-sounding word for “random”." makes me dislike the tone of the article. It's the level of "yo mama"-jokes.
It means more than just "random", it means something that's random but reproducibly probabilistic.If you flip a coin, I have no way of predicting whether you'll get heads or tails. That's random. If you flip a coin 100 times, I can predict with a pretty high degree of certainty that you'll get approximately 50 heads and 50 tails, as will anyone else who flips a coin 100 times. That's stochastic.
-
@Mason_Wheeler No it doesn't. You don't know what you're talking about. A stochastic process is nothing but a time dependent random process.
-
@Captain First rule of holes, dood...
-
@Mason_Wheeler JFC
In probability theory and related fields, a stochastic or random process is a mathematical object usually defined as a family of random variables. Historically, the random variables were associated with or indexed by a set of numbers, usually viewed as points in time, giving the interpretation of a stochastic process representing numerical values of some system randomly changing over time, such as the growth of a bacterial population, an electrical current fluctuating due to thermal noise, or the movement of a gas molecule.[1][4][5][6]
I'll give you one thing, though. Ergodicity is a property of some (many) stochastic processes that in some sense does what you ask of it. It's long term behavior will tend towards uniform entropy.
-
@Captain said in COVID-19 CovidSim Model:
A stochastic process is nothing
I took a graduate-level Stochastic Processes class in uni. The last word in the quote is how much I remember of it.
-
@HardwareGeek said in COVID-19 CovidSim Model:
I took a graduate-level Stochastic Processes class in uni. The last word in the quote is how much I remember of it.
Cool! What was the context? Signal processing?
-
@Captain Yeah, or at least mostly. I remember having once learned something about spread spectrum radio and autocorrelation. That pretty much sums up all that I remember.
-
@HardwareGeek said in COVID-19 CovidSim Model:
@Captain Yeah, or at least mostly. I remember having once learned something about spread spectrum radio and autocorrelation. That pretty much sums up all that I remember.
I remember in maths class, when they were teaching us exponential growth and decay, they gave us a formula for the average rate a human forgets information.
They then asked us, if we got n percentage on an exam today, what score should we expect if we took the same exam m years later.
It was the most depressing answer to calculate (i.e., not a passing grade).
-
@error said in COVID-19 CovidSim Model:
It was the most depressing answer to calculate (i.e., not a passing grade).
And now you can empirically validate it!
In case you want to get even more depressed. Or something.
-
@Gąska said in COVID-19 CovidSim Model:
And now you can empirically validate it!
I've not only forgotten a bunch of stuff, by this point I've probably forgotten about forgetting about it. (On the positive side, that makes the whole ordeal slightly less depressing, I guess.)
-
@error said in COVID-19 CovidSim Model:
It was the most depressing answer to calculate (i.e., not a passing grade).
One reason they should mostly test reasoning abilities on a subject, and less memorization of raw facts.
-
@jinpa the problem is, there's only so much reasoning you can do in history or biology. In subjects where reasoning is readily applicable, it's already being favored in tests over raw facts (maths, physics, chemistry etc.)
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
these code-based models never can have enough details in the papers to replicate independently. They're so tightly implementation-dependent. Or at least the ones I used in quantum chemistry were.
On of my friends had to trudge through a 30-year-old FORTRAN codebase for his MSc thesis in quantum chemistry. He had made a point of having a code repo available with every code-based paper even before the ordeal, but now it seems to be a matter of principle.
-
-
@Gąska said in COVID-19 CovidSim Model:
the problem is, there's only so much reasoning you can do in history or biology
At university level, there's a lot more. But some subjects have an awful lot of detail that you need to understand before you can start building on them; both history and biology are like that (so much so that most people don't take on learning the whole of those subjects). And yes, some people are looking for organizing principles behind them so that the detail can become not the bulk of the subject but rather more of a case study (as is the case in most of the physical sciences).
-
@boomzilla said in COVID-19 CovidSim Model:
@Captain said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
And many of the bad models that're good for policy decisions (but shouldn't be let out into public because the media/people will completely ignore any caveats) become actively harmful if used as political fuel. For example, presenting a known "high-side estimate, if everything goes wrong (plus usual caveats)" as "this will absolutely happen unless we do THAT". Which is how this particular model was used in practice. To stampede politicians into doing sudden, harsh lock-downs to prevent the otherwise-inevitable millions of deaths.
So the harsh lock-downs are instead
preventingdelaying the otherwise-inevitable death of hundreds of thousands. If you believed the "flatten the curve" strategy. Then, of course, there's the recent revelation in NYC that most new cases were people who from all accounts had stayed at home.That's where most people are.
ETA: Most people staying home, are still checking their mail, getting deliveries, probably going to the local bodega. I consider myself staying home (I have not been in the subway since March 12 and the farthest I've been from my apartment has been on bike), but I've been in Rite-Aid and the bodega (I do keep it to a bare minimum). If you have a dog to walk you have to go through the common areas.
Also, everybody lies.
Luckily, this is America so we'll soon be able to see what happens when states open too early, like what's going on in Sweden these days. :-/
Another example demonstrating that old people homes really should be locked down, true, but not the point that I think you were making, which also ignores places like South Dakota that also never shut down.
\Coumo and deBlasio fucked the nursing homes.
-
@Karla said in COVID-19 CovidSim Model:
Coumo and deBlasio fucked the nursing homes.
Reminds me of us…
-
Really bad/stupid conspiracy theory: They saw an opportunity to gain more control (in an evil villain laugh sort of way) and solve the looming pension crisis at the same time. Maximize the deaths of people who just were a drain on the public fisc while also centralizing power!
Note: I totally don't believe this theory. It's stupid and presumes a level of combined evil and competence that I think is lacking, even in NYC politicians. Especially the competence part.
-
@Benjamin-Hall I really don't believe that here. The people most likely to die were overwhelmingly the group most likely to vote for the government. This is a case where simple ignorance and incompetence are sufficient to explain it all.
-
@dkf said in COVID-19 CovidSim Model:
@Benjamin-Hall I really don't believe that here. The people most likely to die were overwhelmingly the group most likely to vote for the government. This is a case where simple ignorance and incompetence are sufficient to explain it all.
Yeah. That's why I said it's a stupid theory. Like many.
I agree that it was basically a pileup of a bunch of thoughtless decisions only considering first-order consequences.
if people test positive, they'll be kicked out and not have anywhere to go!
then let's mandate that they be readmitted/not refused admission!without thinking of the rest of the people they'd infect. So yeah. Ignorance + incompetence all around.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Really bad/stupid conspiracy theory:
There's a thread for that
.
-
@HardwareGeek said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
Really bad/stupid conspiracy theory:
There's a thread for that
.Yeah, but I couldn't figure out how to copy the context as well. Because
can't 
.
-
Back to the original topic:
Here's someone else doing a review of the model itself, as well as the code. Warning: facebook.
https://www.facebook.com/scarlett.strong.1/posts/252437219500976
Edit: after reading it--it's not favorable. Basically, even with the updated code its' impossible to determine if the model is meaningful or implemented correctly. The model itself seems flawed (due to treating individuals as "zombies" (their word, not mine) who don't change behavior except due to legal restrictions)--when applied to Sweden, even accounting for the imposed restrictions, it overpredicts by orders of magnitude the deaths (and thus the lives saved by stronger ones).
As a note, those same model deficiencies are in the other SIR models as well. So not just the IC, but the broader community has this bias toward over-prediction.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Warning: facebook.
I can tell by the fact that it utterly refuses to embed at all on my desktop, probably due to an ad blocker.
-
I've long felt that scientific papers based on computer modelling aren't really science if they don't publish their code. How are you supposed to reproduce the results if you don't have the methodology? And for numerical modelling the code is the methodology.
Peer review should also be a code review in this case but I suspect that in most fields the peer reviewers don't know the first thing about software so that can't really work.
-
@bobjanova said in COVID-19 CovidSim Model:
I've long felt that scientific papers based on computer modelling aren't really science if they don't publish their code. How are you supposed to reproduce the results if you don't have the methodology? And for numerical modelling the code is the methodology.
Peer review should also be a code review in this case but I suspect that in most fields the peer reviewers don't know the first thing about software so that can't really work.
As someone whose PhD was based on computer modelling (very different field), your suspicion is correct in my experience. We had a computer guy on our team whose job it was to actually write the core simulation code. He was Belgian, but I don't hold that against him.
-
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
PhD was based on computer modelling ... He was Belgian, but I don't hold that against him.
You probably should, since it casts doubt on the validity of your degree.
-
@HardwareGeek said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
PhD was based on computer modelling ... He was Belgian, but I don't hold that against him.
You probably should, since it casts doubt on the validity of your degree.
TBQH, I already have strong doubts about the validity (more precisely the utility and meaning) of my degree. Getting a PhD was taking the easy road, and my dissertation topic was, looking back, rather pointless. Probable sum addition to the knowledge of mankind? ε.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Getting a PhD was taking the easy road, and my dissertation topic was, looking back, rather pointless. Probable sum addition to the knowledge of mankind? ε.
I would argue that is true for a lot of dissertation topics.
-
@Dragoon said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
Getting a PhD was taking the easy road, and my dissertation topic was, looking back, rather pointless. Probable sum addition to the knowledge of mankind? ε.
I would argue that is true for a lot of dissertation topics.
Pretty much all of them in the social sciences, and definitely all of them in {Gender|Ethnic|etc.} Studies.
-
@HardwareGeek said in COVID-19 CovidSim Model:
@Dragoon said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
Getting a PhD was taking the easy road, and my dissertation topic was, looking back, rather pointless. Probable sum addition to the knowledge of mankind? ε.
I would argue that is true for a lot of dissertation topics.
Pretty much all of them in the social sciences, and definitely all of them in {Gender|Ethnic|etc.} Studies.
The physical sciences are definitely not immune either. Lots of stuff written just to meet the requirements for a degree (or for tenure).
