Measuring the Reliability of Self-Reported Behavior

Many studies make use of self-reported behavior, nutritional studies asking what you are eating, or surveys asking about how much exercise you do. These self-reports are flawed, subject to our memories and perhaps by an unconscious desire to appear giving the “right” answer. And when this data forms the primary data of analysis, it weakens if not eliminates the certainty of the findings. Despite these flaws, they remain attractive measures because they are easily obtained. A group of researchers reports on trying to measure the gap between self-reported and actual.

The authors made use of SHARE, the Survey of Health, Ageing, and Retirement in Europe, a dataset of roughly 200K adults age 50 to 94 from 19 European countries. In addition to capturing participants, age, education, gender, and nationality, they looked at self-reported cognitive and physical behaviors – the difference, they then tested for those behaviors.

For performance, “participants were asked to stand up from a chair without using their arms;” for cognition, a memory test in which the observer listed ten words and asked the participant to repeat them. Before testing, participants were asked whether they had difficulty getting out of a chair and how they rated their current memory. Participants’ self-reports were consistent with their performance, or if not, were over or underestimates. Prior work suggested that bias in self-reports was related to the four demographic features they considered, all of which are commonly included in scientific research.

Mobility

Roughly 80% of the time, self-reports matched up with actual behavior. Women tend to underestimate their ability more so than men, although both groups over and underestimate. Overestimation increases with age more than underestimation. The “highly” educated are more likely to be concordant, then “less” educated [1]; with “the less educated often overestimating their health, whereas the highly educated more often underestimate their health.” (Is this a manifestation of the Dunning–Kruger effect?[2]) The concordance of self-reports and performance also varied, often by 10 to 15%, by nationality.

The researchers found that about 35% of the variation they demonstrated was due to nationality, and another 32% due to age.

Memory

Self-reports matched up with actual behavior about 70% of the time; in this instance, participants were more likely to overestimate their impairment. There was little gender-related bias, but the age and education discordance were replicated. When weighing the contribution of the demographics, nationality explained 44% of the mismatch, followed by 29% for age, and 22% for education.

Individuals over and underestimated their abilities in the same direction in both physical and cognitive measures. 

Self-reporting introduces additional uncertainty into a study’s outcome. The researchers demonstrated that the bias was a complex mix of cultural expression (nationality), education, age, with a little gender thrown in. The goal of finding some mathematical formulation to improve the veracity of self-reporting, even if not explicitly stated, was not met.

“In conclusion, self-reported measures of mobility and cognition have to be treated cautiously, in particular when comparing health across countries and age groups.”

When it comes to studies involving self-reporting, especially studies of health, caveat lector – let the reader beware. 

[1] Education was categorized as high-school graduates, college students or graduates, and individuals with more advanced degrees.

[2] The Dunning-Kruger effect is the inability of people to recognize their lack of ability.

Source: Who is telling the truth? Biases in self-reported physical and cognitive health status of older Europeans PLOS One DOI: 10.1371/journal.pone.0223526