Sampling is an essential part of conducting a survey, as it determines the representativeness of the sample and therefore, in theory, the reliability of the results. When choosing a sampling method, it is important to consider a range of factors: 

  • What is the accuracy/precision required for the study?
  • What kind of depth or level of detail in the data is required?
  • What are the practical and ethical constraints involved?
  • What is the time when the data are needed?
  • What is the budget for the study?

These criteria were proposed by Smith and Fletcher in the early 2000s as a straightforward evaluation grid for deciding on sampling methods.

Probability or non-probability sampling, that is the question!

There are two main types of sampling methods: probability sampling and non-probability sampling. 

Probability sampling is generally considered to be more reliable because it allows for the calculation of confidence intervals and margins of error. This is because the sample is selected randomly from the entire population, so the probability of a respondent being selected is known. However, probability sampling is not always practical or simply feasible. It is indeed quite common that the universe (ie. the entire population) of the target we try to represent is simply not known or enumerated in any form. This is often the case when the research question is exploratory in nature.

Non-probability sampling, on the other hand, is when the sample is not selected randomly from the population. This can present problems because it is not clear what the probability is of a respondent being selected and therefore, margins of errors cannot be calculated. This means that the answers from the sample may not be representative of the entire population. 

It is important to be aware that even probability samples are not immune from bias. Non-response rates can in fact be very high in probability samples. The Pew Research Centres indicates that Telephone response rates in the USA went from 36% in 1997 to only 9% in 2016… This can lead to non-response bias, where those who do not respond to the survey may be different from those who do respond which, in turn, can affect the representativeness of the sample and the reliability of the results.

The particular case of online panels

Online panels, where people volunteer to answer surveys online, are another sampling method that has gained popularity in recent years. They consist of an online group of pre-selected people interested and willing to participate in future surveys. While online panels can be a convenient and cost-effective way to conduct surveys and panellists can be recruited using probability sampling methods, there can be concerns about the quality of the data collected too. As all panellists are being paid to answer surveys, there is a risk of the “professionalisation” of the panel, meaning that panellists eventually become “professional respondents” generating less spontaneous answers. A bigger risk (and one that is often observed in less quality panels) is that people may cheat or provide false information when volunteering to participate in surveys. It is unfortunately quite common to discover that the same person creates different profiles to maximise his/her chances to be selected for a study and gain money… This can dramatically affect the representativeness and reliability of the results collected.

Fit for purpose

Ultimately, the decision on which sampling method to use is not simple. In 1987, Leslie Kish acknowledged that “statistical designs always involve compromises between the desirable and the possible.”

At Demoscopy, we will always present the pros and cons of non-probabilistic sampling methods to our clients. We favour transparency in order to avoid possible misunderstanding or false expectations. 

But our three decades of experience shows that, for niche target groups, the choice may come down to a non-probability sample or no survey at all…