UKH

Getting outdoors may be a rewarding way to boost health and wellbeing, but the benefits are not shared evenly, with people from an ethnic minority background, people with disabilities, and women, among those less likely to enjoy the benefits. A new research project, Your Movement Matters, is investigating participant demographics in walking and climbing. This includes a public survey - and here's where you can get involved, says Emily Ankers, one of the researchers behind the project.

This looks like it might be a politically commendable but methodologically questionable exercise. It’s foreseeable that the resulting statistics will only reflect the demography of self-selecting respondents – those motivated to respond to such a survey. Just what this sample may be representative of could be hard to unravel. I’m not convinced it will reflect the population of participants in (a very wide range of) outdoor activities.

Starting out with a conclusion in mind isn’t generally the best way to frame the intro to a survey questionnaire as it sets up a proposition bias from the outset. I‘m referring to...

“the benefits are not shared evenly, with people from an ethnic minority background, people with disabilities, and women, among those less likely to enjoy the benefits.”

If the proportions in the hills were to be the same as the proportions in society, how many of each minority should represent ‘fair representation’. Scotland, for example, had 2.6% Asian and 0.12% Black population in 2011. Broadly 1 in 50 and 1 in 1000 resp. Numbers like these would be ‘noise’ even in most quantitative sample surveys.

Does the survey interest me as a (probable) majority demographic to participate? Or am I demotivated by the meta-influence I think a response like mine will have? How many would-be respondents are thinking the same? How are tailored individual responses (essay answers) incorporated statistically? Context, and any inferences premised on randomisation of samples, are going to be the missing pieces for interpretation of your responses.

Another issue is the categorisations suggested. Being ‘a minority’ is a pretty broad qualification, and not all minorities are simple or unique for analysis purposes. Some are meaningful and some less so. There will always be minorities (a statistical truism) and departures from proportional representation in any population are not always significant or blameworthy.

So I think I see why your article is getting some 'down arrows'.

Very well said. It is a shame as this is potentially very valuable research. I would add two additional points:

Firstly I’ve seen this 60% figure before but it seems to me that it is not comparing like for like. As I understand it minority groups are much more likely to be younger and cluster in urban areas. It seems to me that to draw any clear conclusions you would need to compare minority groups to comparable majority populations, not majority populations as a whole. It may be that in comparable majority groups participation is also low. 

Secondly there appears to be an assumption that walking and climbing are the principle ways in which minority groups should be accessing the outdoors. This appears at best presumptuous and at worst chauvinistic. From my personal experience outdoor picnics are popular throughout the world, whereas hill-walking is not. These would seem to me an equally legitimate way of enjoying the outdoors but would be missed in your data.  

Also there is a spelling mistake in the final paragraph which does not help the overall presentation. 

Your criticisms of the survey methodology are mostly quite fair, but I really hope they don't stop you or anyone else from completing the survey. No research is perfect and this survey is what we have right now, so why not use it? I did the survey yesterday, I personally quite enjoyed it.

One thing I would say about the incorporation of essay answers statistically: this is nowadays a relatively straightforward task to perform in a statistically rigorous way, using machine learning tools to perform sentiment analysis and text classification along with a numeric assessment of the uncertainty in any data or models from which inferences are made. The underlying biases in interpretations made from this information are overall similar to the biases made in quantitative questioning, as far as I can see, and essay answers are useful in other ways (they can provide a diversity, specificity and richness of data that more quantitative approaches might not). So the essay questions are worth including in my view.

"...You don't sign in to a National Park, the mountains or your local park and therefore it is difficult to accurately know who is taking part in activities."

Of course the researchers could simply go out at a weekend to the local park car parks in Leeds or the Peak District car parks and TALK to people. Collect some (arguably) more meaningful statistics. But I guess that's not the modern way of doing social science (sic) research. Sigh.

Excellent point. And pretty obvious (in retrospect) when you look at the ethnic mix in the Chew Valley car park (for example) at a weekend and compare it to that on the moors and edges.

I thought the survey was OK but I couldn't be bothered to answer the essay questions. I find it quite difficult to pin down, let alone put into words, what I get from climbing and the outdoors.

I found the repeated assurances that I was "doing really well" a bit patronising and irritating.

that is my take as well – no data collecting effort is perfect to begin with, and equally, no data set is approached with the assumption of being so. (by anyone worth their salt anyway.)

heck, sometimes, you even need that initial flawed, imprecise, broadside data set to give you pointers as to what could be a subject to collect more specific data for.

Perfection is not the issue. It is about what inferences you can draw from the data you have with any degree of confidence. To evaluate this self-selection survey data, you would need to know the selection criteria, i.e., who actually saw the survey, who answered it (and why) and who chose not to answer it (and why) in order to understand how your sample represents the whole. Without that information you cannot extrapolate beyond the sample. And without extrapolation it's a pretty pointless exercise (from a research point of view).

I completed this survey.

I do not think it is a very good survey. It starts by telling us what the result is they want. Many of the questions are essay questions with a very broad or vague baseline. Some of the questions are difficult to answer with any sense of objectivity. Some only allow one answer when a broader response is more accurate.

All in all, this survey feels like an attempt to justify a pre-existing opinion!

Agreed which is why I seldom fill them in.  If not used for that they are sometimes used to justify specific actions which is just as bad. But diversity appears to be a popular bandwagon to jump on at the moment.

Al

I think the problem with these open questions is that it gives you a wide variety of answers to cherry pick supportive quotes for your final report from. Not specific to this research, its a problem with any qualitative survey. It does solve the ..."but the choice of answers doesn't precisely fit my circumstance"... response, though

I completed it but did point out that being urban (and young) may well be a significant factor. It was for me, a late comer to hillwalking and even later to climbing. 

Slightly aside from that..

The main problem I had with the choice of answers not precisely fitting my circumstances was that the range of 'outdoor activities' you get to choose from has some glaring omissions.  (Such as mountain biking, trail/fell running or kayaking.) 

And that one of the ones you can choose is indoor climbing, which (if it happens to be the remaining option that you are "most passionate" about) makes subsequent questions about the natural environment nonsensical.

"It is clear that we have a problem with equity and diversity outdoors".

Sorry I had to give up after this.  It is clear to me that certain sections of the population are less prevalent in the outdoors, but its a massive leap to declare this a problem.  As others have already said, it appears as though you already know your answer.

[Warning: Slightly off-topic]

Hmm, my work experience of sentiment analysis (OK, a few years back) was that it was pretty hit-and-miss (aka too unreliable to be useful). Perhaps thing have improved.

In that spirit, I ran one of the current breed of generalised SA tools (Sentigem) on the responses in one recent thread on here (https://www.ukhillwalking.com/forums/walls+training/better_indoor_wall_route_i...) where the responses were pretty one-sidedly negative. Result? Sentigem considered the responses "positive", largely - I think - because it couldn't distinguish between sentiments expressed about the original suggestion and sentiments about better alternatives.

Oh well, back to the drawing board.

Unless I missed it, I didn't see a question about my children's ethnicity.  My answers will therefore have skewed the whole purpose of this survey.  Oops. (they're not the same ethnicity as me)

That's a problem with bad/dishonest research, rather than a problem with qualitative research specifically. If a dishonest researcher cherry-picks or falsifies their data then obviously it is crap or fraudulent research. That's got nothing to do with the methodology. Quantitative researchers could just as easily bin any questionnaires that don't fit what they want to find, but I reckon it's probably fair to assume that most researchers are basically honest and don't do this.

Agreed. And qualitative surveys have the big advantage of getting responses the researcher hadn't previously considered and therefore wouldn't have put in their list of options in a quantitative survey. 

I'm still stuck at the identification of sections of the population as somehow cohesive groups. Some are, but many would resent being ascribed to the very group they'd be categorised in and point out all sorts of distinctions. There are individuals who enthuse over cultural distinctives and individuals who castigate them, from within the same notional 'group'. Some folk cherish heritage, others despise it. So what validity have the broad categories got?

And if the under-represented 'group' was actually the majority... by which I mean a group comprising 75% of the population was only making up 70% of outdoors users, should there be a push to remediate this somehow? Or does this example serve to highlight GrahamD's point..

I filled it out. At every question I felt that they had already got their conclusion and wanted the data to back it up. Overly long and steered you towards the answer desired 

Yup, same here.   So I'm afraid I stopped completing it.  Shame as my kids are non-white people who go out into the countryside, so part of the story this survey is trying to investigate.  

You say the methodology criticisms are fair, then invite us to complete it anyway??? So, a potentially valuable exercise turned into tickbox compliance. No thanks!

The trouble is that by not completing the survey you will be providing valuable high-quality data to the researchers confirming that there really is a problem with diversity in the outdoor pursuits.

OK, I'm joking. I hope.

As a single white Dad in a multi-ethnic home (my daughter is mixed heritage), surveys like this never capture this data and probably make assumptions based upon initial questions as to my background.

It's the narrow-minded idea of "outdoors" that freaks me, I'm a climber but there's other things I do and am sometimes more passionate about. Even gardening on the odd occasion not to mention why don't I see too many ethnic minority motocross riders?

A typical survey from students that might view the world in less black and white fifty years on.