The UX of Survey Design

I attended the Australian Research Society last month, and decided to sit in on one of the Quantitative Research sessions. Those who know me know that quant research is not my favourite place to live. I’ve done it, and I’ve been around a lot of it in the past 20 years – but I also know a lot of people who are *real* experts in this space – so I tend to focus more on my qualitative work.

That aside, as I listened to this panel discussion I was struck by how our industry conversations around survey research seem to be stuck. Since we’ve transitioned to online surveys (yes, people, we used to do surveys over the phone!), we’ve been treating our participants (our users) poorly. We’ve been excited by the easy access to participants (no more phone calls during dinner), the fancy online tools and the easy access to data – and we’ve lost site of the most important thing – our participants.

So, my question is – as UX or Design Researchers….

Here are my 9 tips for ensuring we create surveys that respect our users as well as our objectives.

01
Mind your Tees and Cees

Mind your manners, be upfront and honest in your interactions. If you respect your participants, they’re more likely to respect you back. This means…

1. Give an honest and accurate estimation of survey length / time in your survey introduction. Don’t just rely on your survey platform’s estimate. If possible, try and get a few people from your target audience to do the survey and time themselves so you can get a real understanding of what the likely time is.
2. With who and how their data will be stored and shared. If you don’t have a working knowledge of the Australian Privacy Principles / the GDPR or your relevant country’s privacy legislation as it pertains to survey research – look it up! In Australia, key things to share are who will have access to the data, and if the data is going offshore (this applies in Qualitative research too). Do they have rights for accessing or deleting their data? Tell them in your intro (or if needed, included longer T&Cs)
3. That their participation is voluntary and that they can opt out at any time. Without this, you’ll have people completing your survey who don’t want to be there… and probably giving you answers that reflect this.

02
Put your Money where your Mouth is

The Australian Research Society provides the following guidelines for incentives:

It’s not always appropriate or necessary to offer an incentive – there are cases where the participant might be particularly invested in the topic, or when dealing with employees or in Business to Business settings or government. If you’ve purchased sample from a panel provider, they will offer their own guidelines. But it is important to be fair with your participants.

03
The Shorter the Better

When surveys are too long, we see response quality drop. Satisficing is the phenomena where people choose to pursue a course of action that will satisfy the minimum requirements necessary to achieve a particular goal. When satisficing occurs we see:

• Choose the first response alternative that seems to constitute a reasonable answer. 
• Select Don’t know instead of reporting an opinion.
• Randomly choose among the response alternatives offered (especially prevalent when many of the survey’s questions are mandatory).
• The longer your questionnaire is, the greater the likelihood that ‘Satisficing’ occurs as respondent’s attention spans begin to wane. 

Source: Excessive Survey Length – Eight Tips To Help You Avoid.

So when it comes to your survey it is important that each and every question fights to keep its place.

Ideally, you should be aiming for questionnaires sub 10-minutes in length…

  Bonus Tip:

Qualtrics research shows that using a progress bar can actually decrease completion rates

Qualtrics Survey Methodology & Compliance Best Practices

04
Don’t ask questions you already know the answer to

So if we need to get our surveys down to 07-10 minutes, one thing is clear – we need to make every question count. So ask yourself, “Can I get this information elsewhere?”

There’s a few ways this can be achieved.

• If you’ve purchased your sample from a panel company – do they have profiling questions for these participants already recorded, so you can ditch some of your demographic or profiling questions?
• Can this data be linked back to a client record? *Caution here* – please make sure that you have permission to use your customers’ information that way. But if you can link back to a customer record, this should save you a lot of questionnaire real estate asking about demographics or recent behaviour
• and lastly, do your REALLY need that question? How will this question be used? What hypothesis does it answer? Make sure every question has a reason to be there.

05
Make it Mobile

What more is there to say?

Of course, know your audience – but honestly, designing surveys for mobile is a good challenge that keeps you honest. It helps to remove the temptation to make things overly complex or use over the top animations – as they often just don’t work. Embrace this constraint to make you a better survey designer.

06
Talk like a normal person

For some reason, when people go to write surveys they often decide they need to sound like some sort of weird academic robot….. overly formal or complex language that confuses the user and clouds the clarity of the question.

I won’t talk extensively on this one, because there is a fantastic book on this subject, written by research legend Annie Pettit. This book is inexpensive and indispensable for anyone wanting to write a good survey.

People Aren’t Robots: A practical guide to the psychology and technique of questionnaire design Paperback – October 24, 2016

07
Don’t ask stupid questions

Ask questions that people can actually answer. This topic is covered in Annie’s book as well, but here are some quick tips:

• Does your question need an other, none of the above or N/A option
• Does your scale include all the numbers, and does it include them only once
  (1-5, 5-6 is a hard one to answer if your answer is “5”)
• Is your question something people actually know the answer to? Eg. How many coffees have you drunk in the past 3 weeks?
• Does your scale make sense and mean the same thing to everyone – (Sometimes, Frequently, Often – what’s the difference?)

And the age old one not to include:

08
Don’t be a Gridiot!

They look fancy but…

• Literature shows they are one of the hardest question types to answer
• They’re unnecessarily difficult for participants to answer – leading to early fatigue, decreased motivation to complete and poor data quality due to satisficing
• They DON’T make the survey shorter. Number of questions does not equal survey length, number of decisions does. Cutting the number of questions by sticking them in a grid is a false economy and might actually increase your survey length as participants struggle to decode what is going on
• They’re sh*t on mobile – to put it plainly. Grid questions are the worst performing questions on mobile, as they tend to render atrociously. With 40-50% of surveys being undertaken on mobile – it’s not worth it!

Source: Qualtrics Why using grid questions is probably hurting your data

09
We are WCAG 2.1!

Make sure that your survey is accessible to all your users. The range of people who use accessibility features to access technology is much wider than you think.

Bonus Tip

Designing for inclusivity not only opens up our products and experiences to more people with a wider range of abilities. It also reflects how people really are. All humans are growing, changing, and adapting to the world around them every day. We want our designs to reflect that diversity.

Microsoft’s Inclusive Toolkit Manual

Next time you catch a bus or a train, look around and see how many people have increased the font size on their mobile phone…. THAT is an accessibility feature. So it is so so very important that our surveys also work on the wide variety of formats that people will look to answer them.

Qualtrics has a list of features that are and are not accessible, but this does not cover colour choices or other elements. It’s up to us to make sure our surveys do not exclude users that we do not intend to. Remember – your survey is a touchpoint of your organisation – if a user cannot access it – what are you saying about the company’s attitude towards that user?

…in short, my challenge to you is ‘Do better!‘

This is both a long post and a short list of the things that we must consider when we are creating a survey experience for our users. What have I missed? Do you disagree? I’d love to talk about this more and ultimately create better survey experiences for all!