Do People Change?

Thoughts on sibling comparison analyses and overconfidence

You wouldn’t know it from looking up or down here in Laubzeit (foliage season) Berlin, but change is hard. Just ask someone trying to quit smoking — or a soul-killing job, desperately depressing grad program, or crappy relationship. Mary Oliver agrees, with her typically gentle devastation: 

 V.

We do one thing or another; we stay the same, or we

 change.

Congratulations, if

 you have changed.

We’re off the hook here, it seems. Change is hard! Congratulations if you have changed; no worries if not. That’s normal, because it’s hard. No guilt. No pressure. Carry on with the quiet desperation and self-sabotage. 

It’s good, in fact, for us methodologists — especially those interested in things having to do with moms, which is everyone interested in humans. Because then we can just compare you and your siblings to measure causal effects of stuff your mom did differently with different kids. Right? 

Not if you ask the ancient Greek philosopher Heraclitus, who said no one steps in the same river twice. It’s always a different person because we’re all changing constantly, just as it’s always a different river because it’s constantly flowing and shifting. “It is not possible to step twice into the same river according to Heraclitus, or to come into contact twice with a mortal being in the same state” (Plutarch).

In sibling comparisons, this means we still don’t know if we correct for all the confounds (that is, factors that could causally contribute to both the exposure and outcome of interest) that matter. There could be nonshared confounding, in addition to random measurement error of exposure, leading to increased bias in these analyses — when the whole point is to correct for confounds to decrease bias. 

It’s the river of Heraclitus problem: Moms at time zero (T0), when they had baby #1, may have differed in important, unobserved ways from the same moms at T1 (when they had baby #2). After all, if they didn’t, then why would they have made different parenting choices on stuff like breastfeeding and taking antidepressants? Maybe they lived and learned (iterative process). Maybe life came at them hard and it had a knock-on effect (exogenous shock). Maybe, just maybe, the same women were different moms when they had different babies.

You don’t know their truth without asking people what it is. And there is not enough asking women (or anybody else) to explain their experiences in research. Actually listening to other people would, you know, take too long when we’re busy sciencing the science with big, important numbers. And how would you code it? People and their stories are messy. Listening, synthesizing, and learning take a lot of time that most scientists don’t have amid increasing precarity. We call data “hard,” but people are harder. And you have to talk to people to hear what data have to say.

When I spoke with a researcher about her sibling comparison analysis last week, she pointed out that health administrative datasets don’t specify whether sibling pairs are full or half siblings. This means that many of the sibling comparisons that researchers make using them — aren’t.

It makes sense: Health administrative databases contain information on what diagnoses were coded per patient. This is often for insurance reimbursement; indeed, they’re often insurance databases, state registries, or a bit of both. Unsurprisingly, your standard, epic tale of love lost and new love (eventually) found doesn’t get coded by Medicaid. So data on how many siblings are full versus half siblings tend to be missing from health administrative databases.

This is a problem for sibling comparisons like those in two recent studies I’ve been looking at lately, on antidepressants during pregnancy and offspring autism risk, by Brown et al (JAMA 2017) and Suarez et al (JAMA 2022). And this problem really matters in these sorts of studies in particular, because recent research by Sebat et al (Science 2018) changed our understanding of genetic autism risk. We used to think that parental contribution to autism risk was 50-50, or (as usual) more Mom’s fault. But it seems that autistic children tend to inherit relevant DNA mutations from their (non-autistic) fathers. So of course kids with different fathers would have different autism risks. This means that a large enough proportion of half-siblings with different fathers could bias these analyses towards finding no effect where there is one.

According to the U.S. Census Bureau, one in six children live with a half-sibling under 18. The proportion appears smaller in Canada, so this may matter more for Suarez et al’s U.S. sample than Brown et al’s Canadian one. Perhaps not coincidentally, Suarez et al’s adjusted sibling comparison analysis results find what looks considerably more like a null effect than Brown et al’s, with hazard ratios of .86 [95% CI .60-1.23] versus 1.60 [95% CI 1.17-2.17].

Some surveys have a less wonky form of the same problem, like this one that underpinned Der et al’s 2006 BMJ sibling comparison analysis of breastfeeding’s effects on IQ. In it, the 1979 National Longitudinal Survey of Youth asked young people who weren’t sure about their family background to just talk about the people they thought of as their siblings, when asked about siblings. While nice for holiday dinner planning, this is obviously a bad way to account for genetics in statistical analyses. 

A related but distinct problem arises when researchers say they’re able to make direct causal inferences from sibling comparison analyses. For example, Suarez et al say “Sibling comparisons control for shared mediators by design and therefore provide an estimate of the direct causal effect.” This considerably overstates the method’s power. Moms who take antidepressants during one pregnancy and not another may be different in other potentially relevant ways, such as partner (and father of child), diet, lifestyle, inflammatory status, and social engagement, not to mention having a kid at home in the subsequent pregnancy and not in the first — which could have numerous potentially relevant effects such as more processed food consumption due to less time for cooking, and more illnesses from exposure to sick little kids. 

This sort of misrepresentation is common across methods, especially quantitative ones. It seems to be part of the widespread confusion of certainty with expertise. There’s a practically important and empirically well-supported movement to recognize and educate scientists about how cognitive bias shapes our work in relevant ways. Dichotomania in the statistical significance test misuse / reform campaign springs to mind, and is a manifestation of grasping for certainty through p-value thresholding, the most common way of misusing statistical analyses to tell us more than they really do.

But focusing on the cognitive side of things risks ignoring the somatic and emotional roots of cognition. We need to talk about power and its felt effects. Fear tells us to dominate or submit, fight or flee (or freeze) — to not think, and to reject uncertainty and ambiguity (which require thinking).

The unknown is scary, and admitting that you don’t know a lot as an expert can be threatening, too. I think this fear gets used as a hack to make people who know that they don’t really know things, pretend that they do — like agreeing that they see the emperor’s new clothes for fear of being the only one who doesn’t. So researchers make pronouncements that they should (or, deep down, really do) know they shouldn’t be making, because they think they’re supposed to do it to be good scientists. Fear makes them destroy what they’re trying to protect (their own professionalism, along with the integrity of the literature, for instance).

Talking about emotions in professional contexts is taboo. But we have to talk about fear of uncertainty when we talk about improving methods, because so much of bad science involves (at best) powerful social networks hacking this fear, and scientists not seeing that they’ve been engineered into doing propaganda. As Upton Sinclair said, “It is difficult to get a man to understand something when his salary depends on his not understanding it.” But it is not impossible. 

Change is hard. Science is hard. There’s a lot we don’t know and can’t control, in trying to account for confounding and otherwise. We’re all human beings, prone to mistakes, plagued by uncertainty and our own reactions to it, and embedded in webs of power. Admitting that, and trying to live and work as honestly as possible — as a person and as a researcher — is hard. Like everything else I’ve written here so far, this is about being mindful of limitations and perspectives (ours and others’), the better to represent what we actually know instead of what we think we’re supposed to say we know.

Congratulations, if you have told the truth.