Research on Research – to agree or not to agree? That is the question…

  1. Research on Research – the awareness illusion
  2. Research on Research – to agree or not to agree? That is the question…
  3. Research on Research – uncovering what really matters

We like a Likert scale!

Welcome to the second post in our Research on Research series. Naturally we love a good survey, so in the interest of quantifying how seemingly small questionnaire design decisions can impact your results, this week we’re looking at data from a range of attitudinal statements asked in our proprietary survey in order to explore the importance of context.

Using a 5-point Likert scale to assess agreement (i.e., 1 being strongly disagree through to 5 being strongly agree) is a well-established question format for understanding attitudes and psychographic characteristics. It lends itself nicely to attention grabbing stats, such as 78% agree watching TV is one of the best ways to unwind, or 48% agree there are better ways to unwind than watching TV.

Hang on, how can both these statements be possible if 78% + 48% sum to more than 100%? The answer is twofold; the first watch out is that the proportion who disagree with a statement is rarely as simple as 100% minus the proportion who agree, because there is usually a neutral mid-point which people may have chosen. For the statement, ‘watching TV is one of the best ways to unwind’ only 8% actually disagreed, the remaining 14% were neutral. The other consideration is that the wording of the statement being assessed will subconsciously influence the way a respondent processes and assesses it because it’s impossible to avoid framing when wording these statements.

The ease of agreement

The statement, ‘Watching TV is one of the best ways to unwind’ is a relatively easy statement to assess and determine your personal view on. A respondent might think, do I watch TV to unwind? Does it help me relax? And from those thoughts determine the extent that they agree or disagree, with our data showing that most people agree in this scenario. In contrast, the statement, ‘There are better ways to unwind than watching TV’ is more cognitively challenging. A respondent has to evaluate how well TV performs as a way to unwind, and then consider alternative activities and how TV might compare to these. Unless you have a strong belief that TV is not a good way to unwind, the answer to this question might not feel as intuitively obvious for a respondent.

Don’t make your data work too hard

Another common misstep when interpreting attitudinal agreement questions is to try and make the data work too hard. With limited survey space, you may be tempted to infer attitudes based on the results to the statement that’s been asked; but while these types of questions are incredibly useful, they are not nuanced enough for us to start reading between the lines.

In the media sector, binge watching is a hot topic and one that has big implications for distribution and windowing as platforms seek to balance series drops that appeals to  viewers’ desire to binge with weekly release strategies that can aid retention. To better understand this, we can ask respondents their viewing preference, but because the desire for instant gratification is a central part of the human condition, when asked explicitly people invariably will choose the option for as much as possible, as soon as possible, over alternatives. This can be seen in our survey where 3 in 4 respondents (74%) agreed with the statement, ‘I prefer it when all episodes of a TV series are available from the start’.

Three-quarters is a strong result, and one you think it might be important to act on if we inferred from this that these people will be put off from watching if they can’t access the whole series from day one. However, 63% agreed with the statement ‘I’m happy to watch one episode a week of a TV series I like if all of the episodes aren’t available from the start’, providing useful context to the strength of the stated preference to have all episodes available, and indicating the importance of deploying the right content across series drops and weekly releases.

How to choose your words

As well as considering how individual attitudinal statements relate to your objectives, it may also be relevant to consider how the attitudes asked about relate to each other, and the bigger picture you’ll capture on respondents’ outlook by considering them side-by-side. For example, if your objective is to find variation in attitudes that allow you to segment a broad audience into distinct groups, each one will need to be suitably different and worded strongly enough to ensure they reflect a range of distinct opinions that aren’t easy for the majority to agree with. Alternatively, you may want to layer together several more similar attitudes to build up a more nuanced picture of a narrower target audience, allowing you to identify where boundaries and thresholds lie.

When attitudinal statements are key for audience understanding you can’t overestimate the importance of semantics. Where possible, draw on existing research (ideally qual) to phrase statements in easy to answer, consumer friendly language that aligns with your end goals. As with all good research, considering how you want to be able to report the results and what they will be used for when designing your survey is crucial in order to capture reliable data that you can report cleanly, without making it work too hard – you can manipulate data to say almost anything, but eventually it will scream!

Our survey was completed online by ~1000 respondents, representative of the UK 16+ population in terms of age x gender, region, and household access to SVOD and Pay TV. Where two versions of the same question were asked, the sample was split into two cells of ~500 respondents, matched in terms of the quotas to ensure comparability of results.

Photos by Pablo Zuchero, Anastasiya Badun, Peter Geo and Icons8 Team on Unsplash and cottonbro studio on PexelsM