Monthly Archives: January 2014

Survey sampling: Is telephone research no longer viable in Australia?

[dropcap]C[/dropcap]onducting research using random digit dial (RDD) landline numbers has for decades been the staple of the research industry. In recent years the effectiveness of this methodology has been in significant decline; first due to the withdrawl of electronic White Pages from public access in 2002, followed by a significant decline in landline installation at home (no longer necessary now that everyone has mobile phones).

The ACMA Communications report for 2011/2012 shows that only 22% of Australian adults who have a fixed line telephone or mobile most use fixed-line at home to communicate, meaning that even when people have a fixed line, it is usually not their primary method of phone communication. Furthermore, the incidence of having access to a fixed line telephone is low amongst younger adults. In June 2011 it was found that only 63% of 18-24 year olds (mostly those still living in their parental home) and 64% of 25-34 year olds claimed to have a fixed line telephone at home. These figures have been falling over the years, so are most likely much lower than this now. [1]

Research conducted by the Social Research Centre reveals that there are statistically significant variations in the populations reached by different telephone sampling methodologies. Specifically, those who were contacted over a mobile phone who didn’t have a landline showed a higher incidence of being male, in younger age groups, live in capital cities, born overseas, live in rental accommodation, and living in their neighbourhood for less than five years. [2] In addition, significant biases were observed in the sample contacted over landline, with landline sample showing lower levels of a variety of important variables including health issues, public transport usage and smoking and alcohol consumption. [3]

There are telephone number list providers out there that claim to include mobile numbers by region. These are ‘listed’ mobile numbers. That is, when someone obtains a new mobile number, it is default unlisted unless the owner requests that it is listed. Many mobile number providers don’t actively prompt for people to have it listed. Mobile numbers that are listed are highly likely to be home businesses (as these are the people who go out of their way to get their numbers listed), thereby skewing the ‘mobile population’ in the survey.

Conclusion

Using Random Digit Dial (RDD) with a mix of mobile and landline numbers would be viable to achieve representative samples. However, this will only work for national surveys, as mobile phone numbers are not assigned by region. Undertaking local area telephone surveys using RDD landlines or white pages phone numbers (even if listed mobiles are included) will miss large, and often critical chunks of the community.

It should be noted, however, that telephone surveys would still be viable if you are sampling a population where you have phone numbers for the entire population (eg. using a client list).

References
[1]  Australian Communication and Media Authority (2012) Communications report 2010–11 series Report 2: Converging communications channels: Preferences and behaviours of Australian communications users, ACMA.
[2] Penney, D (2012), Second national dual-frame omnibus survey announced, www.srcentre.com.au, accessed 21 August 2013.
[3] Penney, D & Vickers, N (2012), Dual Frame Omnibus Survey: Technical and Methodological Summary Report, The Social Research Centre.

Agree Disagree Ratings Questions: Do we need to move away from this question type?

[dropcap]A[/dropcap]gree Disagree scales are one of the most common types of question found in social research surveys. They are usually used to ascertain the opinions and perceptions of respondents relating to a particular issue. However, research suggests that framing questions in this way results in a notably lower level of quality (read: accuracy) in responses [1].
When referring to Agree Disagree (A/D) scales, the following is how it would typically be framed/presented (obviously this is an example of self completion format, alterations to wording and structure would be expected for telephone surveys). This type of question is sometimes referred to as a Likert scale, after Rensis Likert who developed it in 1932:

 

Framing a question in this way has a number of limitations that need to be considered:
  • The statements themselves are more often than not leading, such as “I never read the flyers I receive in my letterbox”.
  • Acquiescence response bias needs to be considered. This is the phenomenon whereby some people will agree with almost anything; due to being an agreeable person, assuming that the researcher agrees so they defer to their judgement, and/or because agreeing takes less effort than rationalising disagreement. [1].
  • Social desirability bias also needs to be considered, whereby respondents will answer in a way that places them in a more favorable light to others. The risk of this is greater when using direct contact surveying methodologies such as face-to-face or telephone.
  • Some people will shy away from the extreme ends of a rating scale, particularly if it is a sensitive issue, which can result in central tendency bias.
It is instead suggested that one employes an item-specific (IS) response scale structure. For instance, instead of asking for level of agreement with the statement “I never read the flyers I receive in my letterbox” you instead ask “How often you read the flyers you receive in your letterbox?” with a scale such as ‘Always, Sometimes, Rarely, Never’. Or you could explore the issue in much more depth, having a series of questions to draw out if there are variations between types of flyers, and ascertain greater detail about actions, such as read and discard, read and share with friends/family, read and pin on fridge/pin-up board etc.
Whilst this approach will clearly provide more useful detail, and avoids the risk of A/D scale biases, it does reduce the opportunity for comparison across multiple statements to identify underlying phenomenon. It also requires greater levels of concentration from the respondent to think through each scale individually. This later consideration, however, can in some cases be a good thing as it will encourage greater levels of engagement with the survey (that is, minimise the risk of the respondent ‘zoning out’). Adopting a IS approach can also significantly lengthen survey duration (impacting on respondent satisfaction, response rates and costs).

Conclusion:

As with the design of any survey question, you need to decide on the best approach based on a wide variety of considerations. For some research, A/D scales may be appropriate, yet for others it may be wise to avoid them.The primary consideration should be how you are going to use the information/what type of information is going to be most useful to you. Cost is also a consideration (presenting a number of statements in a A/D table is cheaper than IS questions), however, cost should never overrule usefulness – if you are going to conduct a survey that is not usefull or is likely to provide sub-standard results just to save money, it is better to not run the survey at all.
If you are going to use Agree Disagree Scales, things to consider are as follows:
  • Always randomise the list of statements to avoid response bias.
  • Keep the list of statements to less than 10 to minimise the risk of response fatigue.
  • The benefit of using a Likert scale is that it allows for identifying of variations which might point to an underlying phenomenon relating to the topic being explored.

References

[1] Saris, W.E et al (2010),  Comparing Questions with Agree/Disagree Response Options to Questions with Construct-Specific Response Options, Survey Research Methods, Vol 4 No1 p61-79.