Taking me to task

The Family Social Scientist takes me to task in the comments on my post on Decision Fatigue:

while this is an interesting quirk of the data and warrants further exploration, its hardly conclusive . To say that the marshmallow test is "known to predict future success in life" is a little misleading and perhaps a misinterpretation of the results.

Touché. I really cannot complain that the FSS is calling me out in the same way that I do to others all the time. It is indeed perilous to try to glean science out of popular articles. I am a rank amateur in the field, and I know how that seems to an expert because I have not always had the grace to deal lightly with those who have trespassed into my own technical specialty.

Yet, nevertheless, I will persist on this topic, despite the many landmines, because it is absolutely fascinating to me. The FSS brings up some really good points in his comment that need to be considered:

grades are hardly a proper measurement of academic progress and intelligence

Quite true. This was arguably less true in the past, but grades, both high school and undergraduate, have a pretty loose connection with both academic progress and intelligence. This is actually one of the things that first attracted me to psychometrics, because it gave me the mental tools to understand why grades aren't a measure of intelligence. 

By way of example, consider this work by Steve Hsu and Jim Schombert on college GPA and SAT scores. Hsu and Schombert explain some of the complexities their work demonstrated in an interview:

“Freshman GPA is not a satisfactory metric of academic success,” Hsu explains. “There is simply too much variation in the difficulty of courses taken by freshmen.” More able freshmen typically take more difficult courses, whereas less able freshmen take introductory courses “not very different from high school classes,” he says. Under these circumstances, academic success—an “A” in an introductory course versus a “B” in an advanced course—becomes too relative to accurately measure. Course variation decreases in later years, as students settle into their respective majors, working hard in required classes.

The new approach bore fruit: SAT and ACT scores, their analysis showed, predict upper-level much better than lower-level college grades, “a significant and entirely new result,” Schombert says. 

Hsu and Schombert are now working on including personality inventories in this assessment to see whether they can improve their model. As a guess, conscientiousness will probably be a big hitter. But, there is a difficulty here. How do you measure conscientiousness? The short answer is: we don't know how. The longer answer is we try various techniques to quantify a quality, such as personality inventories or the marshmallow test. Personality inventories are easy, but they are also easy to game. If you know what the questions are getting at, you can manufacture any result you want. The marshmallow test, and the ice bath test, are a little better in this respect because they push up against a hard limit that we hope is correlated with the thing we are interested in. Thus, even if you knew that holding your hand in the water was going to be used to judge your mental toughness, this would be a good thing because your ability to endure unpleasantness for a positive social judgement is exactly what the test is after.

This is also related to why grades aren't the best predictor either: the system is easy to game. In college admissions, this is part of the reason grades have become de-emphasized. Good grades in high school aren't by themselves a good predictor of doing well in college, but if you factor in participation in sports and other extracurriculars, you can get a rough estimate of a student's ability to stick something through and their ability to manage competing priorities. This can be gamed too, as Amy Chua demonstrates, but if you can successfully game this system, it means you are probably smart and likely to be wealthy, which is something colleges want anyway. 

Comments

[Tom]

I greatly dislike it when someone says "correlation does not imply causation" to discourage a person from changing habits intended to improve some aspect of their life.

For example, I read one study correlating high fructose corn syrup to heart disease. I saw another study correlating soda to heart disease. So in 2006 I stopped drinking soda with the intent of helping to avoid heart disease. And everybody and their dog tells me "correlation does not imply causation".

Of course they are right, but they are also assholes. My premise isn't that avoiding soda with HFCS will for sure prevent myself from getting heart disease. Instead, we live life by making choices every day, and I don't need to know with certainty what the outcome of my choice will be before I make a decision. If instead I choose to use potentially flawed data with the best intent to achieve a goal then I am doing things right.

One might say that I'm exercising my conscientiousness.

If I want to train myself to not drink soda, to hold my hand in ice water for a long time, and to get good grades, then I don't need some outsider telling me that correlation doesn't imply causation, that my choices and beliefs won't make me successful. Far better to make the best choices with the best data available.

Sorry for the rant, I'm not 100% on topic but this is what your post made me think.

[Ben Espen]

Haha Tom! That's awesome!

Certainty is overrated, we just have to do the best we can.

P.S. I hope I didn't tell you that correlation is not causation!

[The Family Social Scientist]

Thats fair enough Tom, and I try to avoid doing the same. The issue I took with the article, and indeed most mass media interpretations of a scientific study (especially in the field of psychology), is that a single data point (or in this particular case, a single flimsily designed study) was used to essentially substantiate an entire broad claim. This is seen with depressing frequency in psychology, and especially psychometrics, especially in the 80's and 90's. Self help books and news stories, magazine articles and television specials all foaming over whatever new study happened to have come out that was marketable. The issue became, however, that most of these studies showed weak correlation (i.e. Sig= ~.06 -.051) at best and the theories behind them were discarded by the scientific community before the story even hit the media. This leads to pop-psych ideas like the ever persistent notion that we only use x% of our brain. While mostly innocuous, they can lead to plenty of problems down the road if they are allowed to pile up, as laymen more often than not hold the purse strings.

Like I said in the original comment, the marshmallow study in and of itself does show that inhibition deficits and decision fatigue are real and measurable things. That much has been reproduced and studied over and over again, verifying the claim. The issue was with the notion that a single instance of them doing some shoddy post-hoc analysis on a study that was never made to be longitudinal in nature could extrapolate some larger truth about the influence of these factors on future success. Having more control is a good thing, and learning to improve yourself in that regard is also good. Assuming that because a kid ate a marshmallow once the tester's back is turned means he is more likely to be a washout is not so good, and could potentially lead to some very negative effects down the line. Parents do some crazy things. If it changes anything, it should change our research focus to see if it can be verified and clarified so that if we are going to use it to better ourselves, we can do it with the utmost efficacy.

Thats the core of my issue. Not that correlation does not equal causation and therefor you should not change your life. We arent certain about a damned thing if you come right down to it. No, the issue I take is that both the article and Ben took this to be good science and presented it as a well researched and solid scientific theory. You can use pelvic thrusts as your primary means of locomotion if you think it improves your life for all I care, but if you claim that a single study you performed shows with a large degree of certainty it predicts how good your life will be, I will probably call shenanigans.