@Gaska said:
@EvanED said:I will try to find the paper this evening; I think the fastest way will be for me to look through old emails on an account I don't have good access to currently.
I'd be very thankful.
So this turned out to be a much more complicated story than I thought.
The paper was about CS learning, so I assumed it was something I encountered while participating in the teaching & learning CS reading group I was in as a grad student. But I looked at the list of papers we covered, and unless it's incomplete or has a rather different title, I apparently invented the discussion. There are two possibilities for where I was familiar with it from. The first possibility is that I ran across some paper discussing this experiment when I was looking for a paper to moderate a discussion on for the reading group. This is totally feasible; I was skimming through a bunch of papers on mental models and this research falls squarely into that category.
However, I suspect the truth is different, and here's where things get hairy. From what I can reconstruct, the following is what happened. As a note, I haven't read most of the links I'm about to give you.
In the mid-2000s, Saeed Dehnad (a grad student of Richard Bornat) carried out the aforementioned experiment. They wrote up their experiments in a paper called "The Camel Has Two Humps." You can easily find it, but I won't link to it because of what I describe next. This paper, as I remembered, does actually control for prior programming experience (namely, they claim a cohort that doesn't have any). This paper made the rounds at the time, including showing up on our own @codinghorror's blog. (@codinghorror, you may want to amend your post according to the following. :-)) There's actually a reasonable chance this is where I originally came across it.
Later research has had... mixed success bearing out the conclusions. This includes a paper (ASE 2008) by the authors for which the abstract says "We now report that after six experiments, involving more than 500 students at six institutions in three countries, the predictive effect of our test has failed to live up to that early promise." Another study (ITiCSE 2007) by other authors also didn't substantiate the results. This alone isn't such a strike against them; that's just how science works. And this sort of subject is psychology, which by its nature is very difficult to study.
Furthermore, it sounds like the case isn't even completely closed; a number of researchers have tried to reproduce the results, and Dehnadi apparently conducted a meta-analysis (PPIG 2009) of these that has more positive results -- including that the test is a stronger predictor of end-of-course success than is prior programming experience.
(Part of the reason for this is the interpretation of Dehnadi's test hasn't been accurately described in this thread yet. The predictor is not whether the person guesses the operation of =
correctly. Rather, they are asked a number of similar questions, and the answers are compared. Subjects are grouped depending on whether they have consistent or inconsistent answers between questions. If you guessed wrong about how =
behaves but guessed wrong in the same way across all questions, you would be put into the "consistent" pile -- and it's whether you appear in the consistent pile that predicts your end-of-course success.) [Typo fixed]
Here's the problem with that paper. Actually here is not the problem because there are multiple:
- "The Camel Has Two Humps" version was not and has not been published in a scientific sense (i.e. peer reviewed).
- The 2006 paper is actually quite awful on a number of fronts; the most offensive (even if their experiments had been substantiated) is that it goes on to claim not only that the current courses aren't teaching the people who "fail" the pre-test CS, but that those who fail the pre-test are unteachable. They dramatically overclaim in many other ways as well.
For the above reasons, the authors (or at least Bornat) have actually issued a formal retraction of the original, 2006 paper. The retraction has more, but effectively it boils down to that he was being treated (and for a while, suspended from his job) for depression; here is his explanation:
I took the SSRI for three months, by which time I was grandiose, extremely self-righteous and very combative – myself turned up to one hundred and eleven. I did a number of verysilly things whilst on the SSRI and some more in the immediate aftermath, amongst them writing “The camel has two humps”. I’m fairly sure that I believed, at the time, that there were people who couldn’t learn to program and that Dehnadi had proved it. The paper doesn’t exactly make that claim, but it comes pretty close. Perhaps I wanted to believe it because it would explain why I’d so often failed to teach them. It was an absurd claim because I didn’t have the extraordinary evidence needed to support it. I no longer believe it’s true.
I also claimed, in an email to PPIG, that Dehnadi had discovered a “100% accurate” aptitude test (that claim is quoted in (Caspersen et al., 2007)). It’s notable evidence of my level of derangement: it was a palpably false claim, as Dehnadi’s data at the time showed.
So anyway, there are some references to chase. :-)