I was going through my archives of presentations, papers etc I’ve written over the years and stumbled on to a presentation I prepared on writing qual recruitment screeners.
Boring, basic, who cares? Right? Well yes and no….
It is really my firm belief that the qual recruitment questionnaire is one of the most important parts of a qualitative project – yet it is often dismissed, hurried through, passed on to more junior staff and generally paid very little attention.
….
So why do I think it is important?
Well firstly, there are some differences between qualitative and quantitative projects that you may have noticed.
….
One… Numbers
In qual we generally have a lower margin of error to work with…
We all know that qualitative samples are smaller – however what does this mean for the project management?
- If one person is wrong – it’s noticeable
- If 4 people don’t participate – we need to repeat our fieldwork
- If the invites don’t go out – it’s hard to recover from
- If we have the wrong quotas / sample etc – we can’t easily remove them from our sample
- It’s one night critical – if we get the wrong night, the wrong venue, the wrong participants – there’s little room for comeback
Two… Visibility
The fieldwork component in qualitative is generally a lot more transparent and visible than in a quantitative survey – having a good night out, watching the group often forms an important part of what the client is paying for. Attendance at qualitative fieldwork can be part of KPIs (customer contact); it is also the opportunity for any negative stakeholders to dispute or undermine the findings of the research.
So we have clients right there watching us when the participant who is supposed to be a user of Brand A, tells us how their husband works for Brand B, and they actually always buy Brand C. There is no buffer, we can’t cleanly and clinically remove their responses from the sample like we would in a questionnaire – this off spec participant WILL impact on the rest of the group no matter what we do next (remove them will unsettle the other participants, leaving them in will skew the results).
….
Three… Time and Costs
The very nature of focus groups and a qualitative project also increase the pressure of getting the screening right on the night. There are a number of additional pressures that are inherent in many qualitative projects that make poor recruitment even harder to come back from:
- Shorter turnarounds: The report is due next week – we’ve only one night to get it right!
- Costs: It has cost roughly $2,080 just to get these people here, fed, in a room – if recruitment is off, you still need to pay them an incentive – we can’t really afford to do this twice
Impact of poor recruitment
Still many have a near enough is good enough mindset when it comes to writing their screeners. Surely a good moderator can ‘cover’ these issues up? Surely it isn’t really that much of an issue – there are still other participants in the room that are on-spec?
Not so. Poor recruitment is always obvious…
….
So, what do we do to make sure that recruitment is on spec, every time.
Off spec participants, clearly the recruiters fault, right?
Not necessarily. Often when I go back to look at why recruitment is off spec it is the screening questionnaire that is at fault, not the recruitment itself. Failure of the researcher to apply critical thought to the drafting of the recruitment questionnaire is often where the error arises. Yes, the recruiter could/ should have flagged an issue with a question. Yes, the client should have picked that up when they approved the question. Excuses, excuses and I don’t want to hear them. As researchers, the buck stops with us.
….
Key things to remember when writing screeners
Structure, Structure, Structure
Start Broad – Order questions to make sure those which screen out the most people are first.
Don’t Give the Game Away – Try to put more general questions first, specific questions later.
Apply the KISS principle: Imagine your Mum, reading this out from paper at home (sorry Mum!). This is how many recruiters are set up – at home, using paper based questionaries. There are no complex CATI systems or online programming like in your quant questionnaire. For qualitative recruitment, keep it simple. Complex routing is unlikely to understood and will lead to mistakes – simplify all instructions.
….
Questions must not ‘give away’ the screening requirement
Don’t make it easy to cheat – Keep in mind what you are telling them in the question – are you letting them know how to answer to qualify? Amazing but true, there are people out there who are willing to LIE, yes LIE, to get $80+ for a couple of hours work. Don’t make it easy for them.
….
Tight specifications lead to tight groups
Detailed and specific specs – Specs should be clearly set down, everything your participants should be must be specified at the beginning of the questionnaire. (I prefer to have a separate page at the beginning of the screener that details who exactly should be in each group and what the group should look like).
Secondly, each specification must have a question. Sounds obvious, but often it is not the case. If there is no question there, then the specification won’t be delivered.
Remember, this is qualitative research – we need the most representative people in the group, so we shouldn’t be afraid to write tight specifications. More and more in qualitative research we want the “sharp point” of the behaviour or attitude. In order to get that ‘sharp point’ we need to write detailed specs. As researchers we shouldn’t be afraid to:
- Make it difficult to qualify, make the specs tight, define exactly who it is you want to talk to
- Be descriptive, specific and detailed in the statement of the specification
- Link the questions back to the specifications
- THINK CRITICALLY. Ask yourself, does answering this question this way mean that this person will meet this spec (what else could it mean?)
My Six top tips for Qualitative Recruitment Management
There are six key things I tell researchers who are starting writing their own recruitment screeners
1. Anything that can go wrong, will go wrong
Be paranoid. It is important to think about all the possible things that could go wrong with a project. e.g. food testing and not excluding food allergies or diets. Copy testing with no glasses or can’t read English. Aside from the ‘research’ issues, what other issues could arise from left field that would impact on the study
2. Apply the ‘stupid’ factor to anything you ask others to do
I am normally a ‘glass half full’ kinda gal – I like to think the best of people and generally like to believe ‘it’ll be right on the night’. However this will not fly when you’re thinking about your recruitment screener. No one will give your project as much thought as you do – because no one’s ass is on the line like yours’ is if the project goes wrong. It is important to look at things critically, to understand how people who are looking at it in a non-critical way could misinterpret. For example, once we changed groups from Parramatta (about 1hr drive out of the Sydney CBD) to be held in a Sydney CBD venue. The recruiter only moved the first group of the evening to Sydney CBD, and continued to recruit the second group to the Parramatta venue.
3. Check it, double check it and check it again.
See Parramatta example above. Don’t assume that because you’ve asked for something to be done, that it will be. Check changes have been implemented, ask for screener responses and check them.
4. Get everything in writing
If these things go wrong, having it in writing is the best protection we have. Confirm all telephone conversations with venues / recruiters etc in writing to ensure we have adequate protection if they make a mistake. If something does go wrong, this could mean the difference between having to wear the costs yourself or being able to trace the mistake back to the supplier (or the client!) and getting them to foot the cost for the error.
5. Copy, copy, copy
Short turnarounds mean that if you are away the project will still need to move. Important that there is someone else across the project at all times. Copying another researcher involved in the project in on emails is essential to ensure that projects will continue if you are hit by a bus or are out sick for a day. It is not checking up on you (well, at least that’s not the main reason), it is making sure that that the project can progress in your absence.
6. Your recruiter will either be your best friend or your worst enemy – relationships are key
Spend time getting to know and building relationships with your recruiters. A good recruiter will be your partner and advisor. I value recruiters who critically look at your screening questionnaires and say ‘that just won’t work, have you considered this instead?’ – this shows me that they are invested in your study and getting it right. Red flags emerge when recruiters take screeners as written and do not offer any feedback or raise any questions about the study.
Victoria Gamble 