COVID-19 CovidSim Model
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
@HardwareGeek said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
PhD was based on computer modelling ... He was Belgian, but I don't hold that against him.
You probably should, since it casts doubt on the validity of your degree.
TBQH, I already have strong doubts about the validity (more precisely the utility and meaning) of my degree. Getting a PhD was taking the easy road, and my dissertation topic was, looking back, rather pointless. Probable sum addition to the knowledge of mankind? ε.
Huh, you're (by common convention) > 0 then? Show-off.
Filed under:
-
@topspin said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
@HardwareGeek said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
PhD was based on computer modelling ... He was Belgian, but I don't hold that against him.
You probably should, since it casts doubt on the validity of your degree.
TBQH, I already have strong doubts about the validity (more precisely the utility and meaning) of my degree. Getting a PhD was taking the easy road, and my dissertation topic was, looking back, rather pointless. Probable sum addition to the knowledge of mankind? ε.
Huh, you're (by common convention) > 0 then? Show-off.
Filed under:
Positive, but to so small a degree as to be infinitesimal. I did do something that no one else has done. And I don't think it was wrong (just pointless). So that's something.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
The physical sciences are definitely not immune either. Lots of stuff written just to meet the requirements for a degree (or for tenure).
I would imagine that most people would prefer, everything else being equal, to do a dissertation (or magnum opus) on something that would at least have a chance to be a stepping stone for their colleagues. At the beginning, when they're picking the topic, wouldn't they try to pick one that left open that possibility?
-
@jinpa sadly not all else is equal. And there are lots of marginal people, even in PhD programs. And lots of advisors doing marginal research.
-
@jinpa said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
The physical sciences are definitely not immune either. Lots of stuff written just to meet the requirements for a degree (or for tenure).
I would imagine that most people would prefer, everything else being equal, to do a dissertation (or magnum opus) on something that would at least have a chance to be a stepping stone for their colleagues. At the beginning, when they're picking the topic, wouldn't they try to pick one that left open that possibility?
I think you would be surprised at how many people (even at the PhD level) are happy just skating by.
-
@Dragoon said in COVID-19 CovidSim Model:
I think you would be surprised at home many people
I'd be surprised if many people showed up at my home.
Filed under: Not social distancing
-
@dkf said in COVID-19 CovidSim Model:
OTOH, you probably don't want to do that the whole time (or even most of the time!); when exploring the outcomes of a model and working out the implications for policy, locking down the model to the point where it effectively only ever considers a single case is worse than useless.
Hurricane prediction models manage to do this. They source several different sims and draw each line.
The author suggested just "taking an average", which to me is just omission.
@cvi said in COVID-19 CovidSim Model:
the discrepancy in the example is only really a problem if it is consistently biased into a certain direction
You're forgetting the human component. A simulator that can't reproduce results allows a human to select using their own bias.
EDIT: Any mode of using the software that can't reproduce results...
@Rhywden said in COVID-19 CovidSim Model:
The endresult will be that no one will make their code public anymore unless forced to because those guys are way too hostile.
It shouldn't be a question.
If it's used for input into policy, it's public. Period.
@Rhywden said in COVID-19 CovidSim Model:
This is an utter failure, you should be ashamed
If the student isn't honest about their work. Then I'm sorry, but this SHOULD be the reaction.
-
Edit: 450 free parameters. All of which are arbitrary and most of which cause significant differences in the output (ie the output is non-linear in changes in the input parameters). No explanation of where the parameters are coming from or why they have the values that they do.
-
How dare you doubt the magic of those numbers.
-
@xaade said in COVID-19 CovidSim Model:
You're forgetting the human component. A simulator that can't reproduce results allows a human to select using their own bias.
EDIT: Any mode of using the software that can't reproduce results...I'm not familiar with the code, other than what I saw from briefly browsing the guthub repo, but I got the impression that they were running ensembles anyway (which would make sense, since I would guess that the system that they try to simulate is quite chaotic/complex). That means that you expect to get a lot of different results to begin with (just from using slightly different starting conditions), and then you analyze the whole bunch of them.
That's why it's important that there are no consistent biases into a certain direction, and why it would matter less if there were random discrepancies, since those are just as likely to follow the trends in the system.
-
@cvi said in COVID-19 CovidSim Model:
That's why it's important that there are no consistent biases into a certain direction
Yes
@cvi said in COVID-19 CovidSim Model:
why it would matter less if there were random discrepancies
No.
This is the point of my previous post. If rerunning it with the same inputs without restarting gives different results, then they can just run it again and again until it agrees with them, and there's no way to prove they tampered in that manner.
It should give the same output with the same input, and then you can prove there wasn't human intervention in this manner because anyone can run it with those inputs.
Without that feature, we run it independently and we don't know if they just happen to get those results, or there was bias in result selection.
-
Total deaths right now is 50k, halfway through the year.
80k margin of error?? Really?
-
@Benjamin-Hall Holy 'ol WTF, I think this reply is borderline other thread material:
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Edit: 450 free parameters. All of which are arbitrary and most of which cause significant differences in the output (ie the output is non-linear in changes in the input parameters). No explanation of where the parameters are coming from or why they have the values that they do.
That's the sort of thing that is very interesting and fun to play with but it utterly unreliable since it's practically guaranteed that you're getting at least some of the interactions wrong. Any "correct" predictions are pure luck at that point.
And thinking about what the model supposedly does (simulates people doing stuff) it probably couldn't be anything else.
The curve fitting exercise is actually starting to sound appealing now.
-
@hungrier said in COVID-19 CovidSim Model:
@Benjamin-Hall Holy 'ol WTF, I think this reply is borderline other thread material:
Cool, so the message is that we should ignore the scientists then, because they're pants on head retarded.
I could accept a statement like, "This is the best model out there." But to say that it's a good model for modeling the real world takes some real chutzpah.
-
@hungrier said in COVID-19 CovidSim Model:
@Benjamin-Hall Holy 'ol WTF, I think this reply is borderline other thread material:
Yeah, never in the history of science has the consensus been wrong. Never has a valiant scientist looked at the "consensus" and seen all of its sharp edges and say "Something is wrong,something is amiss" and then go about figuring out what is wrong. Sure, many (most?) times they come back having further strengthened the consensus, but not always. Sometimes they prove the consensus wrong.
Blind acceptance is not, nor will ever be, the way forward for science. Anyone who suggests otherwise is not worthy of the term scientist.
-
@xaade said in COVID-19 CovidSim Model:
No.
This is the point of my previous post. If rerunning it with the same inputs without restarting gives different results, then they can just run it again and again until it agrees with them, and there's no way to prove they tampered in that manner.
It should give the same output with the same input, and then you can prove there wasn't human intervention in this manner because anyone can run it with those inputs.
Without that feature, we run it independently and we don't know if they just happen to get those results, or there was bias in result selection.First, as mentioned upthread, the model did give the same results if one weren't caching the network file across runs (although that was apparently fixed).
Secondly, I don't see why being reproducible helps against a malicious actor as you describe. The reason the program produced different answers was that random number generator states weren't properly replicated between generating versus using the cached stuff.
But that means that the program depends on the random seed as an input.
Now, if I wanted to be clever, I could just randomly generate a whole bunch of random number seeds, and see which one end up with simulations that go in whatever direction I favour. I discard the other ones. To reproduce my results exactly, you would have to use the seeds that I provide, in which case you end up with my results too. If the RNG that uses the seeds is decent, the seeds should be indistinguishable from a random set, so you can't easily prove that I picked them maliciously.
That's why you run a whole pile of simulations and look at the aggregate data. There one should find similar tendencies. Individual runs don't matter that much. That's also why a bias would be bad, but a random discrepancy wouldn't necessarily.
-
@Benjamin-Hall Haven't followed this closely, and I'm sure I could find out myself (
), but is that 450 free parameters, or 450 hyper-parameters? Sure, 450 is a lot of free parameters, but the real question is can they be estimated with any precision? For example, 450 free parameters is peanuts for a neural network, but usually the training data is large enough to be somewhat confident your estimates aren't bonkers. Maybe they have enough constraint from the data to justify 450 parameters?On the other hand, if they have 450 hand-picked hyperparameters, then
, Lucy has some 'splainin to do.
-
@HannibalRex said in COVID-19 CovidSim Model:
@Benjamin-Hall Haven't followed this closely, and I'm sure I could find out myself (
), but is that 450 free parameters, or 450 hyper-parameters? Sure, 450 is a lot of free parameters, but the real question is can they be estimated with any precision? For example, 450 free parameters is peanuts for a neural network, but usually the training data is very enough to be somewhat confident your estimates aren't bonkers. Maybe they have enough constraint from the data to justify 450 parameters?On the other hand, if they have 450 hand-picked hyperparameters, then
, Lucy has some 'splainin to do.As far as I can tell, they have 450 free parameters in text files in the source. Each is a float, pulled...from thin air with no documentation. supposedly they're all significant in one way or another and you can manually go in and change any of them without the model saying "no, you can't do that". And there's no neural network in sight--this is a massively-coupled differential equation solver, not machine learning.
-
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
@HannibalRex said in COVID-19 CovidSim Model:
@Benjamin-Hall Haven't followed this closely, and I'm sure I could find out myself (
), but is that 450 free parameters, or 450 hyper-parameters? Sure, 450 is a lot of free parameters, but the real question is can they be estimated with any precision? For example, 450 free parameters is peanuts for a neural network, but usually the training data is very enough to be somewhat confident your estimates aren't bonkers. Maybe they have enough constraint from the data to justify 450 parameters?On the other hand, if they have 450 hand-picked hyperparameters, then
, Lucy has some 'splainin to do.As far as I can tell, they have 450 free parameters in text files in the source. Each is a float, pulled...from thin air with no documentation. supposedly they're all significant in one way or another and you can manually go in and change any of them without the model saying "no, you can't do that". And there's no neural network in sight--this is a massively-coupled differential equation solver, not machine learning.
Not only are there 450 parameters, some of those 450 parameters are lists of numbers.
-
@Carnage or even matrixes.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
@Carnage or even matrixes.
Someone didn't get the memo about the kiss rule.
You might just as well go build an ai for all the transparency that hodgepodge has. It'd probably even be more accurate.
-
@Carnage more like "Kiss of death" rule
-
-
@HardwareGeek I think you might have misclicked. Let me fix that for you:
@HardwareGeek said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
machine learning.
-
@HardwareGeek said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
massively-coupled differential equation solver
I have actually used this very gif for a code review.
-
@cvi said in COVID-19 CovidSim Model:
but a random discrepancy wouldn't necessarily.
No.
You don't get it. If there is any way to produce something non-deterministic from a single seed, then you have a problem.
Running multiple seeds to get multiple deterministic results is not bad, because the seeds could be publicized and the results reproduced.
If you have a certain number of identical input and steps that produce different outcomes, the program is defective, and the models it produces are suspect. End of story.
-
@cvi said in COVID-19 CovidSim Model:
To reproduce my results exactly, you would have to use the seeds that I provide, in which case you end up with my results too. If the RNG that uses the seeds is decent, the seeds should be indistinguishable from a random set, so you can't easily prove that I picked them maliciously.
No, because I can peer review the results by running your seeds, getting your results, then running other seeds.
If I can't get your results from those seeds using identical steps, then I can't peer review it.
And we're not talking about a small difference here. We're talking about twice the current number of deaths difference.
-
@Carnage said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
@Carnage or even matrixes.
Someone didn't get the memo about the kiss rule.
You might just as well go build an ai for all the transparency that hodgepodge has. It'd probably even be more accurate.I've got the bestest model in the whole wide world then!
echo "You're going to die."
-
@dcon Sooner or later, yes.
-
@cvi said in COVID-19 CovidSim Model:
Now, if I wanted to be clever, I could just randomly generate a whole bunch of random number seeds, and see which one end up with simulations that go in whatever direction I favour. I discard the other ones.
You could, but I'd bet nobody does this because that's a lot of work and
-
@HardwareGeek said in COVID-19 CovidSim Model:
@dcon Sooner or later, yes.
Speak for yourself, I intend to live forever.
-
@xaade said in COVID-19 CovidSim Model:
No.
You don't get it. If there is any way to produce something non-deterministic from a single seed, then you have a problem.No.
I think you don't get it.
I've not run the code, nor worked on it (and I bet you haven't either), so I can only speak from related experiences. Nevertheless, in general, this kind of simulation code tends to be highly sensitive to any small differences (which is no surprise in simulations of complex/chaotic systems). In fact, as far as I understood, the authors above only guarantee reproducibility only on a single machine/compiler setup. Anything different is largely impossible anyway, i.e., different compilers may just apply slightly different optimizations that affect precision just that slight amount that causes the simulation to behave just a bit differently.
If you have a certain number of identical input and steps that produce different outcomes, the program is defective, and the models it produces are suspect. End of story.
And I would guess it does. Just not for individual runs, but only for the aggregated results.
And we're not talking about a small difference here. We're talking about twice the current number of deaths difference.
If this was the result of one individual simulation run and not an aggregate (which I would assume it Is), then it's meaningless to discuss without knowing the overall distribution of the results. Was this a very unfortunate outlier? Or is the spread in the outcomes actually that large? Or are there perhaps multiple different modes? .
-
@dkf said in COVID-19 CovidSim Model:
You could, but I'd bet nobody does this because that's a lot of work and
Exactly. I would guess that it would about as much effort as rerunning a hypothetically non-reproducible program and then cherry-picking the results that show what you wanted like @xaade suggested. Both are a lot of work, and a bit like playing the lottery.
If somebody wanted to fake / bias the results into a certain direct there are much more direct ways that take way less effort.
-
@cvi said in COVID-19 CovidSim Model:
If this was the result of one individual simulation run and not an aggregate (which I would assume it Is), then it's meaningless to discuss without knowing the overall distribution of the results.
Not at all. There is much that can be learned from tying the inputs to the outputs. In fact, with a model like this I would argue that is one of the most important things that can be learned.
-
@cvi said in COVID-19 CovidSim Model:
If somebody wanted to fake / bias the results into a certain direct there are much more direct ways that take way less effort.
Yes.
The problem is that it can't be verified.
I can say, "I used this seed, and got this result." And that can't be contested. "I ran it twice, and the second time, it produced the results I have after I averaged the two runs."
If you always get the same result with the same seed every time, you'd be able to say, "That's not what these 10 independent runs show."
-
@cvi said in COVID-19 CovidSim Model:
In fact, as far as I understood, the authors above only guarantee reproducibility only on a single maching/compiler setup. Anything different is largely impossible anyway, i.e., different compilers may just apply slightly different optimizations that affect precision just that slight amount that causes the simulation to behave just a bit differently.
This sounds like an academic curiosity rather than something that would have any value when reality hits.
-
@Carnage said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
In fact, as far as I understood, the authors above only guarantee reproducibility only on a single maching/compiler setup. Anything different is largely impossible anyway, i.e., different compilers may just apply slightly different optimizations that affect precision just that slight amount that causes the simulation to behave just a bit differently.
This sounds like an academic curiosity rather than something that would have any value when reality hits.
"Academic curiosity" sounds like most simulations done at universities. And most research at them as well.
-
@Carnage said in COVID-19 CovidSim Model:
This sounds like an academic curiosity rather than something that would have any value when reality hits.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
@Carnage said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
In fact, as far as I understood, the authors above only guarantee reproducibility only on a single maching/compiler setup. Anything different is largely impossible anyway, i.e., different compilers may just apply slightly different optimizations that affect precision just that slight amount that causes the simulation to behave just a bit differently.
This sounds like an academic curiosity rather than something that would have any value when reality hits.
"Academic curiosity" sounds like most simulations done at universities. And most research at them as well.
Yes, and there is no problem with that. The research with apparent value is mostly done by the companies that can profit from the value.
But at the very least it should be on a bit less flimsy ground that looking wrong at it makes it collapse before it's inflicted upon the world. Flimsy madness is for researchers.
-
@Carnage said in COVID-19 CovidSim Model:
@Benjamin-Hall said in COVID-19 CovidSim Model:
@Carnage said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
In fact, as far as I understood, the authors above only guarantee reproducibility only on a single maching/compiler setup. Anything different is largely impossible anyway, i.e., different compilers may just apply slightly different optimizations that affect precision just that slight amount that causes the simulation to behave just a bit differently.
This sounds like an academic curiosity rather than something that would have any value when reality hits.
"Academic curiosity" sounds like most simulations done at universities. And most research at them as well.
Yes, and there is no problem with that. The research with apparent value is mostly done by the companies that can profit from the value.
But at the very least it should be on a bit less flimsy ground that looking wrong at it makes it collapse before it's inflicted upon the world. Flimsy madness is for researchers.Agreed. And even I, as a totally naive researcher (back in the day), had heard the saying
With four parameters I can fit an elephant, and with five I can make him wiggle his trunk. (attributed to Von Neumann).
Seeing 450 parameters makes me BSOD...especially at the thought that this was taken seriously.
-
@Benjamin-Hall said in COVID-19 CovidSim Model:
Seeing 450 parameters makes me BSOD...especially at the thought that this was taken seriously.
It makes sense given the complexity of what they were simulating. I mean...what's the alternative, hard code a ton of different things? But, yeah, it just points out that the results shouldn't be trusted to resemble reality. Sell it as a video game, maybe.
But it seems like we don't want to accept that we can't use a computer to predict something and we refuse to take the appropriate caveats into account when we do.
-
@boomzilla said in COVID-19 CovidSim Model:
But it seems like we don't want to accept that we can't use a computer to predict something and we refuse to take the appropriate caveats into account when we do.
This to some degree. Turns out that once you move away from spherical vacuum cows isolated in their pocked universes, this become difficult. Doubly so if you end up trying to simulate a chaotic system.
I guess I'm having my "somebody is wrong on the internet" when people pick on issues that don't really matter and then believe they've just invalidated something that they are (I'm relatively sure) completely clueless about. (Fuck, I know I don't know anything about simulating pandemics, either.)
...
Look guys, if you want to attack something like this, then read up on the underlying models. I'm sure you can find a pile of assumptions there that are completely laughable. (But that would be work, I guess.)
-
@Carnage said in COVID-19 CovidSim Model:
Yes, and there is no problem with that. The research with apparent value is mostly done by the companies that can profit from the value.
You'd normally want academia to focus on a bit more fundamental stuff, which companies can then pick up and build on / refine. Surprisingly few companies have the resources to fully fund any kind of in-depth research.
-
@cvi said in COVID-19 CovidSim Model:
@Carnage said in COVID-19 CovidSim Model:
Yes, and there is no problem with that. The research with apparent value is mostly done by the companies that can profit from the value.
You'd normally want academia to focus on a bit more fundamental stuff, which companies can then pick up and build on / refine. Surprisingly few companies have the resources to fully fund any kind of in-depth research.
Yeah, and many companies seem hell bent on core business, even selling off bits that make a profit but aren't core business. And with that attitude, spending money on fundamental research is just not going to happen.
-
@Carnage said in COVID-19 CovidSim Model:
@cvi said in COVID-19 CovidSim Model:
In fact, as far as I understood, the authors above only guarantee reproducibility only on a single maching/compiler setup. Anything different is largely impossible anyway, i.e., different compilers may just apply slightly different optimizations that affect precision just that slight amount that causes the simulation to behave just a bit differently.
This sounds like an academic curiosity rather than something that would have any value when reality hits.
Not really, there’s nothing unusual about that.
Unless you enable strict math, even release vs. debug runs of floating point math might not create the exact same binary result in the unit in the last place of precision. Different implementations differ too. I’m not sure which C functions are required to be correctly rounded to 0.5 ulp, but I don’t think all of them are, and “exotic” cluster hardware might not even use IEEE754.
Then there’s random number implementations which usually are not reproducible either. Take C++ <random> for example: I think the random generators are completely defined and thus portable, but the distributions are only defined up to, well, which distribution to generate (or was it the other way round?). Not the implementations details, though, i.e. which bit sequence gets transformed to which output. So a “portable” program drawing random numbers with a PRNG seeded from a given fixed number will give different outputs on MSVC/Windows vs. gcc/Linux.And that’s not usually a problem either, unless you require exactly that for some reason (multi player games yes, Monte Carlo simulations no). The “result” of such a simulation isn’t the output of a single run, but the ensemble distribution.
-
@topspin said in COVID-19 CovidSim Model:
Unless you enable strict math, even release vs. debug runs of floating point math might not create the exact same binary result in the unit in the last place of precision.
Ok, when it's off by half the value on a 5 digit precision value, then I'll maybe give you this point. Maybe.
After which I'll immediately distrust all computations ever made.
-
@cvi said in COVID-19 CovidSim Model:
I guess I'm having my "somebody is wrong on the internet" when people pick on issues that don't really matter and then believe they've just invalidated something that they are (I'm relatively sure) completely clueless about. (Fuck, I know I don't know anything about simulating pandemics, either.)
Maybe so.
I DO know that if this was banking software, there would be lawsuits.
But hey, what's a ruined economy?
I mean, we're only just complaining about it being completely off by the entire value of magnitude on a second run, when it's not even correct when it's run without the bug.
-
If GitHub was trying to federate with lubar.me, I would have completely banned it by now for lack of moderation. Three days for a form letter response and no action?
x.com
Ig Nobel Prize - Wikipedia
