Soft Science Gives Under Pressure



  • What a surprise this is to hard science researchers. Who would have guessed that psychologists did not have replicable results. Alas. I am shocked. Original paper here.

    Ignore scaremongering headline. Scientists have nothing to fear. Apparently, psychologists do.

    ( If your sample size is like 10, you can't prove shit. If your participants are aware of what you're trying to test, you can't prove shit. If your 'random sample' is self-selecting in any way, you can't prove shit. This is why we have double-blind experiments. Psychologists have outed themselves as Not Real Scientists, and I feast on their tears.)


  • Discourse touched me in a no-no place

    @AyGeePlus You probably don't want to know how much medical “science” is little better than that.



  • @dkf Ugh, tell me about it.

    At least biomedical researchers have an excuse. It's expensive to install artificial hearts or whatever.



  • @dkf While we're at it, let's maintain our gentleman's agreement never to look too closely at economics either.



  • Fuck that noise, when economics produces a falsifiable prediction I'll eat my hat. Any of my hats. All of my hats.

    I'm under no such gentleman's agreement: ENGINEERS ARE NOT GENTLEMEN.


  • Discourse touched me in a no-no place

    @AyGeePlus said in Soft Science Gives Under Pressure:

    At least biomedical researchers have an excuse.

    No, they don't. Certainly not for the level of crappy reporting that some of my colleagues uncovered.

    You'd think that, if you're testing drugs for some condition in mice and it is known that the age and sex of the mouse matters for how the condition progresses, you'd at least report the ages and sexes of the mice that you used? Better yet, not just report but control for it? Really basic stuff. The journals and funders say that you must do it, and it's obvious that you ought to.

    You might think that, but you'd be (mostly) wrong. Vast amounts of research is just bunk because there's no way to truly evaluate it or reproduce it because there's just too little info reported. There are people about who try to work around these things with meta-analyses, but without the basic info in the original works they're likely to be just piling turds together to make a bigger pile of excrement.


  • Trolleybus Mechanic

    In the interest of fairness, I must point out that the "hard" sciences aren't necessarliy much better in that respect. They do have an advantage, however, because compared to anything that humans (or other life-forms) do, physics is easy.

    As I understand it, it's not at all a simple matter to secure funds to replicate a published result - after all, why waste money on studying something someone else already has. What's worse, if your experiment fails to replicate the published result, most people's initial reaction is that they did something wrong.

    I don't have any references on hand - nor time to look for them now - but I am aware of several cases where a wrong result persisted in literature for a long time, simply because nobody got around to verifying it.

    Just sayin'.


  • Discourse touched me in a no-no place

    @GOG said in Soft Science Gives Under Pressure:

    it's not at all a simple matter to secure funds to replicate a published result

    True, but if the original result doesn't have enough info published that you could in principle reproduce it, that's a bit shoddy…


  • Trolleybus Mechanic

    @dkf Agreed.



  • @dkf said in Soft Science Gives Under Pressure:

    You'd think that, if you're testing drugs for some condition in mice and it is known that the age and sex of the mouse matters for how the condition progresses, you'd at least report the ages and sexes of the mice that you used? Better yet, not just report but control for it? Really basic stuff. The journals and funders say that you must do it, and it's obvious that you ought to.

    If you'd have seen some of the protocols I've seen you'd have a stroke. 'Add an appropriate amount of X', 'trypsinize cells in the usual way', 'grow cells on normal medium', 'until confluency'. WHICH CONFLUENCY. IT IS A SCALE.

    In the case I am most intimately familiar with the published info is insufficient to reproduce the result because a) there are bugs b) the data is inconsistent and in places just wrong c) <everything I talked about in the lounge>



  • @dkf said in Soft Science Gives Under Pressure:

    piling turds together

    👴 I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a procedure.


  • Grade A Premium Asshole

    @flabdablet said in Soft Science Gives Under Pressure:

    While we're at it, let's maintain our gentleman's agreement never to look too closely at economics either.

    Judging by your proposals to put the world on welfare, I feel it safe to assume you will never look at economics at all.



  • @Polygeekery Judging by your standard propensity to distort and misrepresent any position you feel uncomfortable with, I feel it safe to assume that giving more than 0.000001 shits about what you think would be a waste of time.


  • Dupa

    @AyGeePlus it's always funny to me, when people say shit like this.

    Age of the earth, anyone? Theory of animal migration?

    Yeah, cause "hard" science was NEVER wrong. NEVAH.



  • I'm sure you feel comfortable now that physics has been shown to be fallible, that any level of fallibility is acceptable.



  • @flabdablet @Polygeekery This is a thread for science. Economics posts will be jeffed with extreme prejudice.

    (by asking a mod nicely, but still! Science here. Economics elsewhere. Welfare politics in the appropriate thread, u guyz)


  • BINNED

    @AyGeePlus said in Soft Science Gives Under Pressure:

    Economics posts will be jeffed with extreme prejudice.

    Game theory has deep root in economy, the part of economy that is shared with math is solid. This guy knows how to make it into good use:

    Billionaire Mathematician - Numberphile – 18:39
    — Numberphile

    @AyGeePlus said in Soft Science Gives Under Pressure:

    Psychologists have outed themselves as Not Real Scientists,

    And so do sociologists, political science and recent philosophers (not natural philosophy, or what later merged into real sciences and math). In fact most of humanities, and arts, all are not real science.


  • Discourse touched me in a no-no place

    @AyGeePlus said in Soft Science Gives Under Pressure:

    If you'd have seen some of the protocols I've seen you'd have a stroke.

    No I wouldn't. I've talked to working bench scientists before. ;)

    More to the point, it's nice when the scientists are thinking about moving the physical part of what they do into robotics, since then the program that drives the robot is itself the protocol. There'll be a higher level SOP too, but at some point it hits software and hardware and then there needs to be no uncertainty (though whether the real observations meet what people think is going on… well, That's Different.)

    We're not to that point yet when working with animals (let alone people). The lab I'm working with just uses bacteria, so genuine reproducibility is a goal. :)


  • ♿ (Parody)

    @AyGeePlus said in Soft Science Gives Under Pressure:

    Scientists have nothing to fear. Apparently, psychologists do.

    Lots of medical studies seem to be in the same boat of not being reproducible. IIRC, this was the paper that kinda blew the first big whistle recently:


  • ♿ (Parody)

    @GOG said in Soft Science Gives Under Pressure:

    As I understand it, it's not at all a simple matter to secure funds to replicate a published result - after all, why waste money on studying something someone else already has. What's worse, if your experiment fails to replicate the published result, most people's initial reaction is that they did something wrong.

    Also, publishers don't want negative results. They want positive and novel results.


  • I survived the hour long Uno hand

    ....We all work in software, right? Our house is made of the most fragile glass in existence as far as cargo culting and refusing to even study our assumptions goes.


  • BINNED

    @Yamikuronue Sure, but I don't see many people claiming that software development has anything to do with science. It happens occasionally but it seems to me that people who say that tend to be regarded as idiots by almost everyone else.


  • I survived the hour long Uno hand

    @blek I'm just saying, it seems a bit hypocritical to come down on psychologists for not doing enough research when we boldly make claims about usability without doing studies all the damn time. After all, what is usability if it's not intrinsically tied to psychology?


  • BINNED

    @Yamikuronue Not being a scientist yourself doesn't prevent you from criticizing people who are (or claim to be) scientists, does it? Car mechanics, plumbers, construction workers or architects, all of which are IMO professions closer to software devs than to scientists, can point out flaws in science just fine, I'd say - and I don't think it even matters how good they are at doing their own jobs.

    Also, since apparently psychology is bogus to a large degree, I guess it's a good thing that software developers, interface designers etc. don't take it into account :)


  • ♿ (Parody)

    @Yamikuronue said in Soft Science Gives Under Pressure:

    when we boldly make claims about usability

    Can you provide an example of this? That wasn't mercilessly mocked here?


  • I survived the hour long Uno hand

    @boomzilla I'm sure you know the sorts of things I mean: Marketing says users want one thing, we protest they want another, but neither side has any evidence to back it up. The most egregious examples end up mocked here, but smaller examples go on every day all across the industry unchecked.

    @blek said in Soft Science Gives Under Pressure:

    Car mechanics, plumbers, construction workers or architects, all of which

    use sound scientific research to back their fields. Architects don't suddenly decide people really respond better to buildings shaped like cats, let's build a few like that.



  • @blek said in Soft Science Gives Under Pressure:

    I don't see many people claiming that software development has anything to do with science.

    Except all those CS grads applying for jobs as software engineers.


  • BINNED

    @Yamikuronue Yeah, and programmers and other assorted computer folk use sound scientific research in computer science and physics.

    @flabdablet What do you mean? I know plenty of psychology grads who went on to work as HR people, but I don't see how that would prove anything.


  • ♿ (Parody)

    @Yamikuronue said in Soft Science Gives Under Pressure:

    I'm sure you know the sorts of things I mean: Marketing says users want one thing, we protest they want another, but neither side has any evidence to back it up. The most egregious examples end up mocked here, but smaller examples go on every day all across the industry unchecked.

    OK, sure, but I don't think anyone is taking those things to be grand claims to make broad generalizations worthy of capital letters. Well, I sure don't, and I've never seen anything like that first hand.

    We're just all trying to do what we think is best.


  • I survived the hour long Uno hand

    @boomzilla said in Soft Science Gives Under Pressure:

    We're just all trying to do what we think is best.

    Are we? Maybe it's been a rough week, but I have my doubts.


  • ♿ (Parody)

    @Yamikuronue said in Soft Science Gives Under Pressure:

    Are we? Maybe it's been a rough week, but I have my doubts.

    I assume some people are just really terrible. I suppose malice could be involved sometimes, too.



  • @blek said in Soft Science Gives Under Pressure:

    What do you mean?

    I mean that substantial numbers of CS grads would claim that software development has a scientific basis.


  • I survived the hour long Uno hand

    @boomzilla I think a lot of devs, particularly webdevs, are more motivated by feeling like they're "winning" and/or seeming like they're smart than they are by honestly trying to make a good product for the end users.


  • BINNED

    Oh and also:

    @Yamikuronue said in Soft Science Gives Under Pressure:

    Architects don't suddenly decide people really respond better to buildings shaped like cats,

    Well, that's because it costs a shitload of money to build something, and you have all sorts of laws you have to maneuver around. However, that kind of thing does actually happen from time to time: http://img.blesk.cz/img/1/normal620/602055-img-blob-chobotnice-kaplicky.jpg

    This was supposed to be a new library in Prague. The city was holding a competition and that... thing won, despite not even fulfilling all of the competition's criteria, because the author was apparently "world famous". Then the dude died or something and the whole thing got canned, the last time I heard about it... and someone turned a scaled down model of the building into a bus station in another city.

    Also, this thing actually exists: http://www.praguecityline.com/wp-content/gallery/dancing-house/praha-2-tancici-dum-16.jpg


  • BINNED

    @dse said in Soft Science Gives Under Pressure:

    And so do sociologists, political science and recent philosophers (not natural philosophy, or what later merged into real sciences and math). In fact most of humanities, and arts, all are not real science.

    Who thinks philosophy is a science?

    @Yamikuronue said in Soft Science Gives Under Pressure:

    ....We all work in software, right? Our house is made of the most fragile glass in existence as far as cargo culting and refusing to even study our assumptions goes.

    Most of us aren't putting our conclusions out there as absolute gospel truth and calling anyone who disagrees ASSholes.

    @Yamikuronue said in Soft Science Gives Under Pressure:

    we boldly make claims about usability without doing studies all the damn time

    Is anyone here doing that other than @blakeyrat?


  • Discourse touched me in a no-no place

    @antiquarian said in Soft Science Gives Under Pressure:

    Who thinks philosophy is a science?

    Nobody. But science used to be called natural philosophy, and there are philosophers of science.


  • BINNED

    @antiquarian said in Soft Science Gives Under Pressure:

    Who thinks philosophy is a science?

    People who believe Philosophy should still investigate the edge of science, where science has not yet found a satisfactory answer. When we did not know atoms philosophizing about its existence played the role that nowadays theoretical physics plays explaining the unknowns of today. Philosophy as a field of study should be merged with history.



  • @boomzilla said in Soft Science Gives Under Pressure:

    @AyGeePlus said in Soft Science Gives Under Pressure:

    Scientists have nothing to fear. Apparently, psychologists do.

    Lots of medical studies seem to be in the same boat of not being reproducible. IIRC, this was the paper that kinda blew the first big whistle recently:

    My point is that psychologists don't design double-blind studies, don't have good control groups, and don't include unpublished data in their meta-analyses. They lead studies with small sample sizes even though it costs them like 10$ a data point. There's no excuse.

    The crisis in replication in medical studies has a lot to do with statistics, the cost of running trials, p-hacking, and so on and so forth. It's embarrassing but it's not a damning indictment on an entire field. This ego depletion shit is like if DNA methylation was discovered not to be real.



  • @blek said in Soft Science Gives Under Pressure:

    Car mechanics,

    Mechanics aren't equivalent to developers; car designers are. And you bet your ass they study the usability of their dashboard designs. (Which, BTW, I have a lot of respect for because making one configuration that serves most human beings in the space allotted is really difficult. People come in thousands of shapes and sizes. If both Shaq and Carla from Cheers can drive your car safely and comfortably, you've made something pretty amazing.)



  • @boomzilla said in Soft Science Gives Under Pressure:

    Can you provide an example of this? That wasn't mercilessly mocked here?

    Markdown is one of those things that multiple people have somehow decided is desirable without, AFAICT, studying that assertion in the slightest.

    And while, sure, a lot of people here mock that (although a lot don't), but not enough people in our industry are mocking that to prevent GitHub from adopting it as a standard, or it being the default choice in every forum software apparently.

    And while I'm sure some wag will come in here and say, "well GitHub is for developers who are smart enough to get Markdown," which completely misses the primary point, it also misses the secondary point that software like Discourse and NodeBB aren't intended for developers. And yet made that same unstudied knee-jerk decision to adopt Markdown.


  • kills Dumbledore

    @blakeyrat said in Soft Science Gives Under Pressure:

    well GitHub is for developers who are smart enough to get Markdown

    Also, isn't Markdown meant to be a simple markup language? If it was done properly there wouldn't be anything to get. Like a WYSIWYG with clearly labelled buttons



  • @Yamikuronue said in Soft Science Gives Under Pressure:

    use sound scientific research to back their fields. Architects don't suddenly decide people really respond better to buildings shaped like cats, let's build a few like that.

    Architects also don't suddenly decide to use 18" tall steps on their stairs, and utterly ignore that there's a sizable percentage of the population who can't use stairs with 18" steps. Which I see as a better analogy. (A lot of buildings are weirdly-shaped and just fine in the important aspects of usability. The Experience Music Project building in Seattle is a good example. Or, heck, the Space Needle even.)



  • @boomzilla said in Soft Science Gives Under Pressure:

    We're just all trying to do what we think is best.

    Right; so three guys (the designer of GitHub, NodeBB and Discourse) think Markdown is "best". But the problem is: they have no fucking clue what's "best" because they haven't studied it. They picked it because they have a personal preference and they have absolutely no clue how many other people share that preference.

    That's not doing what they think is best; that's doing what they personally like. That's selfish. That's almost the exact opposite of doing what they think is best.



  • @Jaloopa said in Soft Science Gives Under Pressure:

    Also, isn't Markdown meant to be a simple markup language? If it was done properly there wouldn't be anything to get. Like a WYSIWYG with clearly labelled buttons

    A lot of things that are simple aren't usable. Assembly language is simple. One line = one processor instruction. Even the geekiest of geeks wouldn't use it however unless they had no other choice. I've had a hell of a lot of people tell me Git is simple.

    Here's a simple way to implement a drop-down menu:

    • If the mouse cursor leaves the menu's drawing rectangle, collapse it

    The problem is that "simple" way isn't at all usable, because as it turns out human beings don't move their mouses in perfectly straight horizontal and vertical lines. So a much more usable drop-down menu implementation might look like this:

    • If the mouse cursor leaves the menu's drawing rectangle:
      • If the the cursor has been outside for less than 100 milliseconds, do nothing, maybe they're elderly and/or don't have steady hands
      • If the cursor is in a triangle drawn from the bottom edge of the menu to the left or right of the top-most menu item of an adjacent menu, do nothing because they likely didn't find what they wanted and are just moving the mouse up to open the next top-level menu item
      • Except check that same region but pad it by 20% of the menu width because human beings are also pretty bad at drawing perfectly straight lines
      • If all those are false, then the user intent is probably to stop using the menu bar, collapse the menu, but even then you're likely to piss someone off who thought it collapsed too soon
      • And of course you can also end up with pissed off users if it stays open too long when they think it should be collapsed, so make sure you tune the shit out of this hypothetical algorithm on a thousand different users until they're all happy with it

    Remember when you were learning web programming and you came across that article that was like, "how to make drop-down menus out of CSS only!" and you were like, "wow that's nifty!" but then you tried it and the menus were fucking awful and didn't fucking work right? Yeah. That was the "simple" implementation. Sure, it "works". The bear's dancing. It's just not dancing very well.

    ... anyway, as far as Markdown specifically go, I'd call any markup language that changes the text I type into something else a failure, personally. I don't even see that as bad usability; I see that as a simple failure to perform the function the markup language is meant to perform. (And that's not getting into the lack of features-- how do I use Markdown to justify a line of text? To put text under a spoiler hider? To make text underlined, for God's sake? There's many reasons Markdown sucks.)



  • @blakeyrat said in Soft Science Gives Under Pressure:

    Even the geekiest of geeks wouldn't use it however unless they had no other choice.

    Lies.



  • @Yamikuronue said in Soft Science Gives Under Pressure:

    Architects don't suddenly decide people really respond better to buildings shaped like cats, let's build a few like that.

    Denied!



  • Wait, I'm confused. Wasn't being able to stand up and say, "This whole theory we've been using is terrible and doesn't work, even though we relied upon it for many years," something that Physics, Chemistry, and Biology do quite a bit? And historically, did a whole hell of a lot?

    I mean, first light was a wave, then it was a particle, then it was a wave, and then it was both? And it took a couple hundred years to sort out? And the physicists are bitching about the psychologists?

    Anyone ever hear of the bad smell theory for disease? Anybody ever look at how long Aristotle's theories -- like crystal spheres supporting stars and planets -- were considered unquestionable truth? Ignorance and error is a hallmark of science. That's why we keep studying shit!

    Why isn't the goddamn title of the article, "Groundbreaking Study Redefines Understanding of Human Psychology"?



  • @BaconBits It 'redefined' the understanding of human psychology back to where it was in the eighties.

    The photoelectric effect and the double-slit experiment show conflicting results, which can be resolved by advances in theory. Every time you do the photoelectric effect experiment, you get the same result. Every time you do the double-slit experiment, you get the same result.(Barring some weird quantum things you can do that are hilarious, but I digress.)

    This study doesn't demonstrate anything new. It tries to repeat a famous study with a more stringent set of controls and showed no result. Science advances by moving from 'kinda right' to 'mostly right'. Light-as-particle is 'kinda right'. Light-as-probability-wave is 'mostly right'.

    Ego depletion is fictional. The people who most understand the placebo effect and the importance of controls, hell, the people who discovered the placebo effect can't or won't do science stringently enough to avoid it. This isn't really about ego depletion, it's about psychologists. This replicability problem shows that psychology researchers are not doing science.

    Soft sciences are harder to do that hard science. Protons don't behave differently if you've had a bad day or it's raining outside. Controlling for confounding variables is 95% of doing good science, and a fundamental part of scientific practice for good reason. We have advanced statistical methods for dealing with this sort of thing!


  • area_pol

    For the hypothesis being "people sometimes get mentally tried", I do not think they will ever prove it entirely true or entirely false.


  • area_can

    @blakeyrat said in Soft Science Gives Under Pressure:

    So a much more usable drop-down menu implementation might look like this

    Related: here's a look into Amazon's nifty dropdown menu implementation


Log in to reply