Yes, it happened again. Twitter has just gotten a new favorite study. The study “shows” that “we subconsciously consider cycling more dangerous when we see a cyclist decked out in safety gear.” And that of course means we should “forget the hi-vis and helmet argument.”
In this text, I offer my understanding of the study. First I describe the study and the attention it has received, and then I explain why in my view the study provides no support for the above conclusions.
1. The study. The idea was to “determine whether the clothing a cyclist is shown wearing in an image would affect respondents’ judgement of the safety and social acceptability of cycling, and whether it would affect the likelihood that they will associate the image with exercise or sport.”
The task was to sort five cards (“with a photo of the same model either walking, cycling, on a bus, in a taxi, or driving a car”) in order from the least safe to the most safe. All cards were identical except for the photo of the cyclist.
“Card Set A showed the cyclist in clothing comparable to that worn by the model in the RSA videos: a high visibility jacket, helmet and sports clothes (from here on referred to as RSA Style for convenience). Card Set B showed the cyclist in everyday clothing (Appendix 7).”
The main result is depicted in the figure below. Note that numbers are actually percentages.
2. The text for RTÉ (the website of Raidió Teilifís Éireann, Ireland’s National Public Service Media)
The study was a bachelor thesis ‘published’ in 2017 and it was apparently presented on several conferences/meetings. However, it seems it got the most attention after the author published a text for RTE.
The author has summarized her thesis, offered a few analogies, argued that “The predictable arguments about the merits of cycling safety gear is a red herring” and concluded “Let’s forget the hi-vis and helmet argument.”
3. Twitter. In less than 24 hours, the RTE text has received a lot of attention on Twitter. It is used as an argument against using hi-vis and helmets. I predict this study will be repeatedly used against any promotion of cycling safety gears.
My view about the study.
1. Randomization. The participants were recruited at five different locations:
“The first batch of responses came from a knitting group.
The second batch of responses were from the eating area in the nursing building in DkIT at 11am.
The third batch of responses was from parents waiting for their children to finish a lesson at a martial arts school on a Saturday morning.
The fourth batch of responses came from visitors to the food court in a shopping centre on a Friday morning, again with good variety in ages
The final batch of responses were gathered from students and staff in the main restaurant in DkIT at lunchtime on a Monday.”
I have found no information about how these participants were assigned to one of the two groups. The number of participants was 73 so it would be of interest to know more about the “randomization.”
2. How instructions were administered. I am tired of writing about the experimenter effect, but if you have a fully informed experimenter who recruits people and verbally (?) instructs them how to perform a task…
3. Descriptive data. The author keeps writing that there was a difference between two conditions (i.e., safety gears make cycling look more dangerous); however, this conclusion is based on descriptive statistics only.
I was not able to find the exact number of participants in each condition, but assuming the number was very similar, I “reproduced” her data and performed a statistical analysis.
Neither of two applied tests showed association between the rating and condition: P-values were 0.5144 (gamma test) and 0.6290 (Cochran-Armitage Trend Test). As a funny observation (if we focus only on percentages and ignore statistical tests), it is actually that the cyclist in ordinary clothes was judged to be less safe: 92% (22+70) vs. 89% (11+78) :)))))
In my view, the study provides no evidence that “We subconsciously consider cycling more dangerous when we see a cyclist decked out in safety gear.” Furthermore, cycling, irrespective of the look of the cyclist, was considered least or next least safe. This accounted for about 90% of both groups, so I think it’s quite misleading what the author and Irish media have claimed. Unfortunately, I predict that anti helmet (law) people will endlessly share this study on Twitter and elsewhere. It has happened before.