science made more stupid than usual
Dec. 7th, 2013 11:27 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
twoelevenPNAS recently published a paper purporting to show large differences between adult male and female brains. the popular press -- and AFA dïe überblönde and i CT, the bio-med bloggers -- took this at face value, though the latter pointed out a number of problems in the analysis. they all appear to missed a big one: the paper studied teenagers, not adults.
while the authors do say that, they tend to bury that key fact in sweeping generalizations, such as
and
so what did they really find? frankly, i'm not quite sure. i'm not familiar at all with the analysis method they used, nor the statistical test(s) they applied to the data. a lot of the statistical analysis appears to be missing: it's not in the paper, nor does there seem to be any supplementary information. i could go hunting around for it, but the onus is is on them to defend their claims, not me.
also frankly, i'm not up to applying my usual rigor to this paper. i'm too sick and too far behind with real work to spend the time and attention it would take. so, i'm gonna be a bit sloppy, and use weasel-words myself... though i'll have the decency to italicize my handwaves. my apologies in advance. however, this one appears to be a real winner, so i don't think i can let it pass entirely unchecked.
ok, so what do the authors report?
they take their raw data -- which was NMR images of the brain, which kinda-sorta show brain activity, with the usual stars and daggers of outrageous caveat about how imaging studies relate to what the brain is really doing -- and apply some fancy math to turn that a list of regions in the brain that appear to be connected. i don't understand that math. they then statistically analyze the differences in the connections between the sexes, grouped by ages.
that's the first part that they don't seem to like to talk about, but i'll point it out:
i'd emphasize that "roughly". based on the age ranges, i'd say the groups correspond to "mostly children, but some entering adolescence", "all adolescents", and "mostly adolescents with some adults". to the best of my knowledge, the human brain starts undergoing significant changes right around puberty, and finishes up maturing between 19 years old and the early 20s. so, the study is really about kids and adolescents, despite the researchers' labels.
even within those limits, people mature at different rates, and girls tend to mature earlier than boys, so the groups may not really be comparable. if i wanted to do what the authors seem to want to, i'd make the age limits for the groups 8-11, 14-18, and 22-25. the gaps between the age ranges are deliberately to avoid the transitional periods between childhood, adolescence, and adulthood. (if i wanted to study those, i'd have five groups rather than three.)
the authors say "Linear regression was applied to each of the connections in the SC [structural connection] matrix on sex, age, and age-sex interactions." seem to be saying they looked at each of those three variables one at a time, which would be stupid beyond description, so they probably mean they used multiple regression on all of them at once. either way, the details of whatever they did are missing, which makes their analysis impossible to check. all they report are results, using a statistic called "T", which is apparently popular in brain imaging. they refer to a couple of papers in the brain-imaging-processing literature, but i lack the time/attention to check them. let's pretend all the data processing is correct.
it turns out not to matter, because while they report big differences between the sexes, when they break the results down by age ranges, they find:
what's that mean? it means that its easier to see statistical differences between "adolescent" (13.4 - 17 year old) boys and girls than it is to see them between younger ones (8- 13 years old) and older ones (17.1 - 22 years old). or in other words, the statistical differences appear easiest to find during the middle adolescence, but become harder to find at either end (and for "modularity", there apparently were no differences between boys and girls entering adolescence).
none of these results say how big differences are, just how easy they are to find with statistical tests. they're related, of course, but not simply, since it depends on how clean/noisy the data are. nor do they really say anything about adulthood, because the authors' oldest group is too young.
the authors don't report the same age-based analysis for their headline results about which regions connect to which. why? did they not check? did it also show that it seems to be going away as children mature? we'll never know, but selective reporting of results tends to catch my attention, especially when it's the splashy results missing.
so, i can only conclude that they've shown some differences between the sexes during puberty -- which i doubt will surprise anybody -- but haven't made their case for adults. which is, unfortunately, what they want play up.
years ago, happy_dr_friend@lj lamented to me that the really splashy papers about new cancer treatments tend to be the ones that go nowhere, but the unreported incremental advances in treatments are the ones that really do the work of curing people. there are, of course, breakthroughs in science, but there are many fewer of them than it seems from reading the headlines, or even reading what the researchers say in interviews. unfortunately, there's a good razor for general press reporting of scientific results: the bigger the splash, the smaller the rock.
the really ground-breaking stuff tends to be so technical and so weighted down with caveats that the press doesn't understand it. those papers get the specialists excited, but even when reporters catch that it's big news, they have trouble explaining it.
i urge non-scientists looking for a more thorough discussion of how to read bio-medical papers to take a look at tylik@lj's, On medical research and not being an idiot.
while the authors do say that, they tend to bury that key fact in sweeping generalizations, such as
The observations suggest that male brains are structured to facilitate connectivity between perception and coordinated action, whereas female brains are designed to facilitate communication between analytical and intuitive processing modes(PNAS paper, "significance" side-bar)
and
“So, if there was a task that involved logical and intuitive thinking, the study says that women are predisposed, or have stronger connectivity as a population, so they should be better at it,” [lead author Ragini] Verma said.(LA times interview for the paper)
“For men, it says they are very heavily connected in the cerebellum, which is an area that controls the motor skills. And they are connected front to back. The back side of the brain is the area by which you perceive things, and the front part of the brain interprets it and makes you perform an action. So if you had a task like skiing or learning a new sport, if you had stronger front-back connectivity and a very strong cerebellum connectivity, you would be better at it.”
so what did they really find? frankly, i'm not quite sure. i'm not familiar at all with the analysis method they used, nor the statistical test(s) they applied to the data. a lot of the statistical analysis appears to be missing: it's not in the paper, nor does there seem to be any supplementary information. i could go hunting around for it, but the onus is is on them to defend their claims, not me.
also frankly, i'm not up to applying my usual rigor to this paper. i'm too sick and too far behind with real work to spend the time and attention it would take. so, i'm gonna be a bit sloppy, and use weasel-words myself... though i'll have the decency to italicize my handwaves. my apologies in advance. however, this one appears to be a real winner, so i don't think i can let it pass entirely unchecked.
ok, so what do the authors report?
they take their raw data -- which was NMR images of the brain, which kinda-sorta show brain activity, with the usual stars and daggers of outrageous caveat about how imaging studies relate to what the brain is really doing -- and apply some fancy math to turn that a list of regions in the brain that appear to be connected. i don't understand that math. they then statistically analyze the differences in the connections between the sexes, grouped by ages.
that's the first part that they don't seem to like to talk about, but i'll point it out:
We present results from a cohort of 949 healthy subjects aged 8-22 [years old] ... including 428 males ... and 521 females (demographic details are provided in Table 1).(my emphasis and eliding)
...
Because the age range in this population is so large, to examine developmental sex differences the population was divided into three groups such that they have balanced sample sizes: group 1 (8-13 [years old], 158 females and 156 males), group 2 (13.4-17 [years old], 180 females and 141 males), and group 3 (17.1-22 [years old], 183 females and 141 males). These groups roughly correspond to the developmental stages of childhood, adolescence, and young adulthood.
i'd emphasize that "roughly". based on the age ranges, i'd say the groups correspond to "mostly children, but some entering adolescence", "all adolescents", and "mostly adolescents with some adults". to the best of my knowledge, the human brain starts undergoing significant changes right around puberty, and finishes up maturing between 19 years old and the early 20s. so, the study is really about kids and adolescents, despite the researchers' labels.
even within those limits, people mature at different rates, and girls tend to mature earlier than boys, so the groups may not really be comparable. if i wanted to do what the authors seem to want to, i'd make the age limits for the groups 8-11, 14-18, and 22-25. the gaps between the age ranges are deliberately to avoid the transitional periods between childhood, adolescence, and adulthood. (if i wanted to study those, i'd have five groups rather than three.)
the authors say "Linear regression was applied to each of the connections in the SC [structural connection] matrix on sex, age, and age-sex interactions." seem to be saying they looked at each of those three variables one at a time, which would be stupid beyond description, so they probably mean they used multiple regression on all of them at once. either way, the details of whatever they did are missing, which makes their analysis impossible to check. all they report are results, using a statistic called "T", which is apparently popular in brain imaging. they refer to a couple of papers in the brain-imaging-processing literature, but i lack the time/attention to check them. let's pretend all the data processing is correct.
it turns out not to matter, because while they report big differences between the sexes, when they break the results down by age ranges, they find:
Global transitivity [connections between adjacent brain areas] was higher in males among all three groups (children: T = 3.1, P = 0.003; adolescents: T = 4.9, P < 0.0001; young adults: T = 3.7, P = 0.0003), whereas global modularity [the presences of big "chunks" in the networks, like groups of close friends with only a few acquaintances connecting them] was significantly higher in adolescents and young adult males (T = 5.1, P < 0.0001 and T = 2.7, P = 0.005, respectively).
what's that mean? it means that its easier to see statistical differences between "adolescent" (13.4 - 17 year old) boys and girls than it is to see them between younger ones (8- 13 years old) and older ones (17.1 - 22 years old). or in other words, the statistical differences appear easiest to find during the middle adolescence, but become harder to find at either end (and for "modularity", there apparently were no differences between boys and girls entering adolescence).
none of these results say how big differences are, just how easy they are to find with statistical tests. they're related, of course, but not simply, since it depends on how clean/noisy the data are. nor do they really say anything about adulthood, because the authors' oldest group is too young.
the authors don't report the same age-based analysis for their headline results about which regions connect to which. why? did they not check? did it also show that it seems to be going away as children mature? we'll never know, but selective reporting of results tends to catch my attention, especially when it's the splashy results missing.
so, i can only conclude that they've shown some differences between the sexes during puberty -- which i doubt will surprise anybody -- but haven't made their case for adults. which is, unfortunately, what they want play up.
years ago, happy_dr_friend@lj lamented to me that the really splashy papers about new cancer treatments tend to be the ones that go nowhere, but the unreported incremental advances in treatments are the ones that really do the work of curing people. there are, of course, breakthroughs in science, but there are many fewer of them than it seems from reading the headlines, or even reading what the researchers say in interviews. unfortunately, there's a good razor for general press reporting of scientific results: the bigger the splash, the smaller the rock.
the really ground-breaking stuff tends to be so technical and so weighted down with caveats that the press doesn't understand it. those papers get the specialists excited, but even when reporters catch that it's big news, they have trouble explaining it.
i urge non-scientists looking for a more thorough discussion of how to read bio-medical papers to take a look at tylik@lj's, On medical research and not being an idiot.