What alternatives exist to probability-based general population surveys, what are their strengths and weaknesses, and under what circumstances are they most appropriate?

In addition to addressing these primary questions, this chapter also includes a discussion on how and when one may consider using multiple datasets and offers some general ethical considerations for using any type of data. Because analysts face a myriad of questions and combinations of priorities, this chapter cannot act as a frequently asked question list and offer direct yes/no answers or step-by-step instructions. Instead, it seeks to empower the analysts to evaluate their data options by providing sufficient information on various data types and situations.

RULING OUT PROBABILITY-BASED GENERAL POPULATION SURVEYS AND DETERMINING NECESSARY CRITERIA FOR ALTERNATIVE DATA SOURCES

There are countless reasons that a probability-based survey of the general population (GP) may not be necessary or feasible. In some cases, a probability-based GP survey may not be affordable, or when considering data quality, a sufficiently reliable and complete sample frame may not be available for the target population, which introduces the potential for coverage bias (FAO, 1996). The topic of interest may not be measurable via a survey because it is an abstract construct and not amenable to a survey question (introducing construct validity issues) or respondents may not know the answer or may be unwilling to provide an accurate response (introducing measurement error). Researchers may not be able to interview certain types of respondents because they cannot be accessed (e.g., lack of access due to a natural disaster) or because potential respondents refuse to participate. In these situations, a probability-based GP survey may suffer nonresponse error. In addition to data quality, the data may be dated and no longer relevant. The ultimate consumers may also need the data more quickly than collection reasonably allows, the data may have been gathered in a way that may violate U.S. or in-country ethics norms or laws, or the sample size may be insufficient for the chosen analyses.

It is important to acknowledge the reason for ruling out a GP probability-based survey prior to selecting an alternative data source for two reasons. First, it acts as a double-check on the research priorities and goals. If research priorities and goals do not sufficiently justify rejecting a GP survey, it may be worthwhile to reconsider using this type of survey. Let’s consider a hypothetical question that an analyst might be tasked with answering: How has support for the current regime changed over time? To address this question, it is important to have 1) a repeated

measure of support over the time period of interest and 2) sufficient sample sizes to detect the target level of change. Let’s also assume that a GP dataset is available for multiple time points but has been rejected because it suffers coverage bias because it consistently excludes residents living in outlying areas that are difficult, time-consuming, or costly to access. This suggests that the individual point estimates (i.e., level of support for any point in time) may be biased, but it is possible, though not certain, that the change over time (i.e., the measure required to address the research question) is unbiased. In this case, the reason for rejection does not map to the analyst’s priorities, so the analyst may opt to use the available survey.

Second, understanding the reason for rejection will help analysts understand what they need or need to avoid when searching for alternative data sources. For example, if an analyst seeks to understand government support by an ethnic group and no GP survey includes sufficient sample sizes for subgroup analyses, determining what would be a sufficient sample size and seeking out data sources that meet that criterion would be important.

ALTERNATIVES TO PROBABILITY-BASED GENERAL POPULATION SURVEYS

This section introduces various alternatives to probability-based GP surveys. They can be divided into a two-by-two matrix: designed versus organic data (Groves, 2011) and primary versus secondary data. Designed data is systematically considered by a researcher and designed to address a particular research question. It can take the form of surveys or qualitative research. With designed data, researchers should have all necessary information available to address specific research for which the study was designed; such data should have high construct validity.¹ Organic data are those that have been collected for another purpose or are a result of another process (e.g., credit card transactions, tweets). As such data were not collected to address the particular question of interest, they may suffer from low construct validity (i.e., they measure something slightly different than what the analyst wishes to measure). They may also suffer from other forms of biases (e.g., coverage error in which the full population is not represented) or ethical challenges such as the lack of consent from individuals to having their cell phone geolocation data used to study migration patterns (Lai et al., 2019).

___________________

¹ Note that this does not mean that all designed data can answer all questions for all uses. It is limited to addressing questions for which it was designed. This means that secondary users may find that their analyses still suffer poor construct validity.

language in which the interview was conducted, and the name or ID of the interviewer. These data points are considered primary organic (not designed) because they are created by the researcher as part of an administrative process such as determining interviewer pay or identifying the optimal time of day to contact potential respondents. They are not meant or collected for purposes of substantive analysis to address the research question. Secondary organic data includes all other nonsurvey and non-qualitative research data that was not collected by the researcher. Tweets, for example, are not surveys or qualitative research. They are also not created as a result of a research process.

As may already be evident, there are numerous data sources that may serve as an alternative to probability-based GP surveys. Despite the overwhelming number of different data sources, this framework (primary versus secondary x designed versus organic) can be used to determine some general strengths and weaknesses of a given dataset as a starting point of evaluation, including those not discussed in this report.

The remainder of this section is limited to data sources most relevant to studying social attitudes. For each source, a definition is provided, strengths and weaknesses are discussed, best-use cases are included, and ethics are considered. Where appropriate, best practices for collecting primary data are included. Examples are also provided throughout this section. Where possible, the examples chosen are related to use cases that may be of interest to the IC, but it was more important to provide examples that had been validated (i.e., were a strong use case) than to provide IC-related examples.

Note that all data sources evolve over time. This section is a snapshot of what is available at the time of writing (2021). It should be used as a baseline and supplemented with more up-to-date and context-specific information when choosing a data source.

Also note what this section is not designed to provide. It cannot provide a universal list of “good” and “bad” datasets because no such list exists. As noted in “Drawing Inferences from Public Opinion Surveys: Insights for Intelligence Reports” (“Drawing Inferences”) earlier in this Analytic Framework, the appropriateness of a dataset depends on the use case. A dataset that is well suited for addressing one question may be merely adequate for another and completely inappropriate for a third. The researcher must determine their research objective, analytic plan, and set of priorities (e.g., getting an answer quickly is most important even if there are some biases in the results). The researcher’s priorities may then be compared to the strengths and weaknesses outlined here to provide a high-level account of the appropriateness of a dataset.

___________________

² https://www.citypopulation.de/en/ecuador/admin/.

register may have telephone numbers for these residents and the survey could be conducted by phone. If researchers conduct analyses of preexisting survey data, they may seek out a similar survey conducted among the missing population and blend the data to create a singular national estimate (for a discussion of blending data, see “Integrating Data Across Sources” [“Integrating Data”] later in this Analytic Framework).

Various weighting methods may also be employed to remove biases. The Keeter adjustment (Keeter, 1995), for example, which is traditionally used in U.S. telephone surveys to account for nontelephone households, identifies individuals who used to be part of the missing group and weights them up to represent the missing group. In consideration of the Galapagos, a question may ask respondents if they have lived in the Galapagos within the past 5 years. Individuals who answer affirmatively would be treated as a proxy for the missing inhabitants and weighted accordingly.³ Alternatively, if the sampling frame only undercovers a subgroup of the population without completely eliminating them, traditional weighting methods may be used to minimize or eliminate bias from the estimates (Valliant, Dever, and Kreuter, 2013). While it is best to consider the weighting plan during the design phase, it may be possible to construct new weights on secondary or preexisting primary datasets depending on what data are available.

Finally, if resources and time permit, researchers may wish to invest in creating a frame of the missing units. This may be an especially useful investment if repeated surveys are expected over time. The frame could be built once and used multiple times over several years. NORC at the University of Chicago has used this method to improve coverage⁴ of the U.S. Postal Service’s Computerized Delivery Sequence (CDS) File list of addresses, a list that undercovers⁵ rural addresses and Native American reservations, among others. NORC uses listers to identify and append addresses that are missing from or incomplete on the CDS and uses this enhanced frame to draw address-based samples throughout the decade. Similar listing procedures may be used to create a frame of missing housing units. Geospatial sampling, for example, may be useful when no frame is available (Eckman and Himelein, 2020). It involves creating a map of the area of interest and overlaying a sampling grid. Cells of the grid are randomly selected and listers are tasked with enumerating (and in some cases interviewing) all housing units within the sampled cell.

___________________

³ Note that the Keeter adjustment has not been tested for use cases other than the nontelephone population. The example described here is hypothetical and it, along with any other use case, would require validation before it is applied in practice.

⁴ https://www.norc.org/Research/Projects/Pages/2010-national-sample-frame.aspx.

⁵ https://www.aapor.org/Education-Resources/Reports/Report-on-Online-Panels.aspx.

all eligible household members in a household that contains members of the rare population. This approach may significantly reduce data collection costs and improve timeliness of data collection, but it can increase the design effect when individuals within the cluster (e.g., household) have relatively homogenous attitudes (Kish and Frankel, 1974). This, in turn, reduces the statistical efficiency of the sample. In other words, it introduces more noise into survey estimates, making it more difficult for estimates to reach statistical significance.

Overall, analysis of rare populations (either individually or comparatively) generally requires an oversample that must be considered a priori to data collection. This means that the researcher may have to collect primary data or identify a secondary data source that had similar analytic goals. If neither of these are feasible, researchers may have to turn to nonprobability sources and organic data.

Nonprobability Surveys

The difference between a probability-based and nonprobability survey is how the sample is drawn. The probabilities of selection are unknown for nonprobability samples. This occurs when no frame exists, or a frame exists but the size of it is unknown to the researcher. Nonprobability samples are frequently used in situations similar to the following:

• A sampling frame is unavailable and cannot be created within the timeline and budget. Researchers may, for example, wish to measure voter turnout and vote choice in real time on the day of the election. A list of registered voters may not be available, may be incomplete, or may not contain contact information (such as phone numbers) to facilitate same-day contact.
• Researchers are interested in studying a rare population for which stratification cannot sufficiently improve efficiency or is cost-prohibitive. Researchers may wish to interview female sex workers in China in the midst of a sexually transmitted infection epidemic (Verdery et al., 2015). This population is extremely rare, and a population frame is unlikely to have data that could be used to stratify it. Moreover, these individuals are likely untrusting of authorities, so “cold” contact from a probability-based frame is unlikely to be sufficient to gain cooperation.
• Cost constraints make a probability-based design infeasible.

Nonprobability Online Panels

There are multiple types of nonprobability surveys that may be used to facilitate the above research objectives (Vehovar, 2016). One option may be

the use of nonprobability online panels (Kennedy, 2018). Individuals receive internet pop-ups or invitations through loyalty programs (among other means) inviting them to join a survey panel. Once they opt to join the panel, the panel vendor will invite them to complete various surveys (sometimes many per week). While it may be possible to measure how many people see the invitation to join the panel, the population is unknown, so a probability of selection for joining the panel cannot be calculated.

Nonprobability panels offer several advantages. They are generally less expensive than probability-based designs. Panel vendors typically collect a lot of information from the panelists (e.g., demographics, rare health attributes), making it possible to target specialty subgroups (e.g., veterans) or stratify the sample in ways not possible with most probability-based frames. Some online panel vendors also “route” respondents (Unangst et al., 2019), purposely sampling panelists that they believe have the attributes of interest (e.g., a specific political party affiliation). While this improves efficiency and reduces costs, it also yields a purposive, not random, sample. Panel surveys are also conducted (primarily) by an email invitation to complete a web survey, making them much more timely than some probability-based designs.

Some panel vendors may implement quota sampling,⁶ a method where targets are set for specific subdomains. The researcher may, for example, require 50 percent of completed interviews be from men and 50 percent from women. Once the target has been reached for a given subdomain, it is “closed.” Potential respondents who fall into closed subdomains are deemed ineligible and not asked to complete the rest of the survey. Quota sampling can ensure that the sample distribution mimics the general population, or if studying a rare population, ensure sufficient sample sizes for small subdomains. It can also be cost-efficient when using postpaid incentives since you do not need to pay the incentive for “extra” interviews in the closed subdomains. It also helps to limit the amount of variance introduced by weighting adjustments. However, quota sampling also results in a false sense of security. Individuals within a given subdomain (e.g., men) are often not representative of the group as a whole. They may be more cooperative or socially engaged, easiest to identify a priori, or may differ on other attributes. Quota sampling often fails to correct biases in point estimates of nonprobability surveys (DiSogra et al., 2011).

However, nonprobability panels generally suffer from under-coverage (Baker et al., 2010). They often do not include individuals who are not online, skew wealthy and more educated, and skew younger. Other coverage issues may also be apparent in different countries. These biases generally skew point estimates. Weights and modeling approaches frequently fail to correct the biases, and in some cases, they make them worse (Dutwin and

___________________

⁶ https://www.questionpro.com/blog/quota-sampling/.

Buskirk, 2017). Nonprobability samples also preclude the use of some statistical methods. Probabilities of selection, for example, cannot be calculated, so standard errors and confidence intervals cannot be accurately produced (Baker et al., 2013).⁷

Moreover, not all nonprobability panels are created equal (Kennedy et al., 2016). Analysts should consider the methods for panel creation and panel maintenance. The European Society for Opinion and Marketing Research (ESOMAR) has published a list of questions (and explanations of the importance of each question) that researchers should ask vendors when considering whether to use their panel/data and/or evaluating different vendors (ESOMAR, 2021). Some vendors have published answers to these questions on their websites, but all should be willing to answer these questions. If they are not willing to provide this information, that may be an indicator of poor practices and poor data quality. Moreover, researchers have developed a framework that outlines the necessary conditions that must be met in nonprobability surveys to produce unbiased data (Mercer et al., 2017).

Intercept Surveys

A second nonprobability design is the intercept survey. Intercept surveys involve “standing” in a particular location and interviewing all or a fraction of individuals who pass by. Exit polls are an example of in-person intercept surveys in which interviewers stand outside of voting precincts and ask people leaving the precinct for whom they voted (AAPOR, 2007). Intercept surveys may also be conducted online using a method called river sampling. Just like some of the recruitment methods used to construct nonprobability panels, an ad or banner is published on a website and individuals who click on it (think of fishing in the river and the fish that bite the line) are asked to complete a survey. Ad managers such as Facebook’s Ads Manager⁸ can use browsing history, social network group memberships, and other metadata to help target the survey to specific individuals (e.g., social activists).

Similar to nonprobability panels, intercept surveys are frequently less expensive than probability-based designs. In the case of online intercepts, they can also be used to collect large amounts of data quickly, and metadata targeting can be used to improve the efficiency of the sample in identifying individuals from rare populations. In-person intercept surveys (e.g., exit polls) can also produce relatively accurate estimates when the placement of the interviewers facilitates a random selection of potential locations and

___________________

⁷ Note that this limitation often does not prevent researchers from attempting to create and reporting inaccurate standard errors.

⁸ https://poweradspy.com/facebook-ads-market-research-survey-campaigns/.

rules guide interviewers on whom to sample (Armstrong, 2019). For example, a design in which a random sample of polling precincts is selected, and interviewers are instructed to attempt to interview every 10th person who exits would have a good chance of producing accurate estimates. A hand-picked selection of precincts in which interviewers just have to interview 100 voters would not.

Intercept surveys may, however, also suffer similar weaknesses as other nonprobability designs such as coverage bias, limitations on statistical analyses, and differences in data quality due to nuances between intercept designs. In some instances, intercept surveys may yield more biased data than other types of nonprobability sample surveys (Lehdonvirta et al., 2020). In-person exit polls have, moreover, suffered differential nonresponse in which one party is more likely to participate, introducing nonresponse bias (Silver, 2008). In recent years, the accuracy of exit polls has been further challenged due to the coronavirus pandemic, which has facilitated a higher proportion of early voting in several countries (Bronner and Rakich, 2020). If election day voters are different from early voters and if this phenomenon is not correctly accounted for in the models, the exit polls will be inaccurate.

Chain-Referral Methods (Snowball Sampling and Respondent-Driven Sampling)

A final type of nonprobability surveys uses chain-referral methods such as snowball sampling (Goodman, 1961) and respondent-driven sampling, or RDS (Heckathorn, 1997). These designs are used to study networks as well as rare populations in which members of the population likely know each other or where members may be untrusting of outsiders, such as the researcher. They are most commonly used to study at-risk populations in public health (e.g., homelessness [Bernard, Da ková, and Vašát, 2018]), but have also been used to study rare indigenous populations (Mullo, Sánchez-Borrego, and Pasades-del-Amo, 2020) and could be useful to study other groups.

The samples start with identifying some eligible individuals (often called “seeds”), interviewing them, then asking them to recruit or refer other eligible individuals (“sprouts”). Researchers attempt to interview the sprouts, ask them for references to others, and so on.

There are two distinctions⁹ between snowball sampling and RDS that deserve note as they affect data quality. First, snowball samples use nonprobability methods to identify seeds, whereas RDS may use either nonprobability or probability methods. Second, snowball samples do not attempt to track or control for who referred whom. RDS samples do. As a whole, RDS uses snowball sampling procedures and applies mathematical

___________________

⁹ https://www.statisticshowto.com/respondent-driven-sampling/.

models to account for probabilities of referral (i.e., selection) in an attempt to eliminate biases. This makes RDS arguably less prone to biases but much harder to implement.

Chain-referral methods such as snowball sampling and RDS are one of the most efficient methods to identify extremely rare populations. When the population is at-risk or the topic is extremely sensitive, this approach may be more successful than other nonprobability methods (e.g., an online intercept survey ad placed on relevant websites). Referrals aid in gaining cooperation and building rapport in a way that is more difficult with a cold call.

Unfortunately, the use of chain-referral methods may also introduce several challenges. These methods can be costly and labor-intensive, and it can be time-consuming to implement and build a sufficient sample size. Specifically, it is difficult to adequately define eligibility criteria and frame network size questions (Johnston et al., 2008). Some seeds may not spawn enough sprouts, limiting the sample size and introducing bias into the estimates (Truong et al., 2013). Additionally, the models used in RDS are not always successful in eliminating bias, and RDS often increases variance in the estimates (Goel and Salganik, 2010; McCreesh et al., 2012), further limiting statistical efficiency.

Ultimately, nonprobability surveys are often less expensive and time-consuming to implement, and may be better equipped to identify rare populations. However, they are also much more likely to suffer from biases due to coverage error and unknown (or known but uncontrollable) sampling error. Moreover, the variation in data quality makes it critical that the IC ask many questions regardless of whether the data are primary or secondary. If primary, the questions may follow those similar to the ESOMAR questions and aid the choice of a vendor. If secondary, the IC should obtain and review the methodology report that may accompany the data.

Qualitative Data Collection

A final designed data source is that of qualitative data collection. Qualitative data collection includes a variety of research methods that require summation of data using nonstatistical means (Busetto, Wick, and Gumbinger, 2020). Data may still be collected using questionnaires or interviews, but it is typically summarized in narrative form. This tool may be especially useful when conducting foundational or exploratory research to understand cultural norms and societal practice. It is also useful for deeper dives into “why” and “how” questions. For example, a survey may be useful to measure general attitudes toward individuals with HIV/AIDS whereas qualitative research may be more appropriate to collect information on how individuals perceive risk factors for contracting HIV (Tarimo et al., 2013).

Qualitative research can generally take six forms.¹⁰ This paper focuses on the two most relevant for USG researchers: one-on-one in-depth interviews (that is, semi-structured interviews) and focus groups. For more information on other types of qualitative research, see Merriam and Tisdell (2009).

Similar to a survey, one-on-one in-depth interviews include an interviewer and respondent and a basic set of questions. Unlike a survey, the questionnaire is a guideline, and the conversation is much less standardized. The interviewer is free to follow up with respondents on what they said. Most if not all of the questions are open-ended, unlike most survey questions that have a set of response options.

___________________

¹⁰ https://www.questionpro.com/blog/qualitative-research-methods.

___________________

¹¹ While total costs for qualitative research may be less than surveys, the cost per completed interview is often higher since interviewers who conduct qualitative research typically require more skills, experience, and training than survey interviewers, incentives are generally higher, and each interview is longer than many surveys.

2001). Moderating is difficult and requires a skilled individual who is familiar with the research objectives, knows when to probe further, knows when to sit quietly and give respondents time to voice their opinion, is able to pick up on cues (verbal and nonverbal), does not lead the respondents, and can navigate different personalities in group settings. When choosing moderators, consider whether they have special training in focus group and in-depth interviewing. This may include university coursework or special accreditation from institutes such as the Burke Institute.¹² It may also be beneficial to hire a different moderator for different types of respondents. For example, it may facilitate more open conversation to have a Black American moderate a group of Black Americans and an Asian American to moderate a group of Asian Americans.

Researchers also need to equip the moderator with the appropriate tools. This will include developing items such as an interviewer guide, an informed consent text, a screening and recruitment protocol, and training on the purpose of the study. Just like developing a sampling plan and questionnaire for a survey, developing these types of materials takes time and multiple rounds of review but is made easiest when building off of a clear research question.

While qualitative research can offer rich information that is often not available from a survey, it is not without its limitations. Most importantly, quantitative data makes it more difficult to produce quantifiable statistics. Researchers may instead want to use qualitative analytical techniques to identify themes and ideas (Warren, 2020). This is a specialized skill set that requires an individual trained in qualitative data analytics (often a social scientist, historian, or anthropologist). Moreover, the use of formal qualitative methods and trained staff minimizes the risk of subjective interpretation of the data.

Additionally, if researchers are not careful to recruit a representative sample of the population, the narrative may suffer significant biases, overemphasizing one perspective or missing several opinions. This can occur even when quotas are set. Let us assume, for example, that researchers place age quotas on the recruitment and then, recruitment materials are placed around school campuses to recruit the youngest age group. While this should yield a sufficient count of young adults, the young adults will likely skew more educated and wealthier than the population as a whole. As mentioned above, researchers may need to recruit from multiple sources, balance on multiple demographics, and consider what other variables (e.g., sufficient tech savviness for handling online surveys) may be correlated with the outcome of interest and need to be considered when evaluating whether recruited individuals likely represent all types of opinions and attitudes.

___________________

¹² https://www.burkeinstitute.com/SeminarDescription/Detail/Q01.

discussion, social media data do not include nonprobability surveys fielded on social media platforms (see the discussion of intercept surveys above for more information on these types of data).

Research conducted in this space has covered topics from climate change to consumer confidence to election outcomes (Ceron et al., 2013; O’Connor et al., 2010;¹³ Williams et al., 2015) and has used data from Facebook, Twitter, Reddit, Wikipedia, Google, and Instagram, among others. Social media data may assist researcher in:

• Providing an immediate reaction to a sudden event such as a coup or natural disaster (e.g., text analytics of posts containing relevant key words)
• Estimating turnout for an upcoming rally or protest (e.g., number of likes to an event’s page compared to other recent events in the country)
• Understanding some types of networks such as the influence of party leaders (e.g., number of Twitter followers or retweets)

The largest benefit of social media data is its real-time nature. Unlike surveys that may take a significant time to plan and field, individuals react to news via social media quickly. Social media data are also stored and, in some cases, offer a historical record. This may allow researchers not only to gain a snapshot of attitudes as events unfold but also allow for the analysis of data leading up to the event. Retrospective analyses are rare in surveys because prior time points may not have been collected or because asking respondents to recount past attitudes creates measurement error due to forgetting, telescoping, or the reframing of past attitudes in light of current events (Tourangeau, Rips, and Rasinski, 2000). As such, social media data may be the most appropriate data source to use while unexpected events are unfolding.

While social media data are often considered free or inexpensive and easy to access, this is frequently a misconception. On the one hand, tools such as the Twitter¹⁴ and Reddit¹⁵ application programming interfaces (APIs) are free to use and offer a variety of options to download different types of data. On the other hand, many social network platforms are entirely inaccessible, making access rather than cost the driving factor. For example, messages and phone calls on WhatsApp use end-to-end encryption, which means that no one, not even WhatsApp administrators,

___________________

¹³ Note that the trend identified by O’Connor and colleagues (2010) did not hold over time (Conrad et al., 2019).

¹⁴ https://developer.twitter.com/en/products/twitter-api/academic-research.

¹⁵ https://www.reddit.com/dev/api/.

can access them (Barrett, 2021).¹⁶ Even platforms that offer free access may limit the quantity or format of available data. Google Trends,¹⁷ for example, provides normalized descriptive statistics of a sample of Google searches, but it does not provide a full dataset of all searches. Data access may also differ depending on the organization. The Zillow API,¹⁸ for example, allows any user to scrape a finite number of data points per day, but Zillow also has a full database, ZTRAX,¹⁹ that is accessible only to academics. Additionally, some companies may require lengthy contract negotiations before the data may be accessible. Such negotiations are often labor-intensive and costly.

Even platforms that do allow free access can be expensive to analyze. Many data sources require advanced programming skills in languages such as R or Python, comprehension of nonrectangular datasets (e.g., JSON files), or advanced analytic skills (e.g., text analytics, network analysis). Individuals who typically analyze survey data will likely need more training on analyzing social media data, and it will take time to learn this. Time affects both cost and how quickly social media data can be analyzed, offsetting the benefit of the real-time nature of social media data.

Social media data also suffer from significant biases. Most importantly, only a small proportion of the total population is on any given social media platform. For example, while over 24 million people in India use Twitter (Statista, 2022), this only accounts for 1.6 percent of the total population. Even fewer post to the platform on a regular basis, and users are significantly more likely to be male than the general population (Hootsuite, 2019). In the United States, social media users skew younger and/or more liberal than the general population (Wojcik and Hughes, 2019). To the extent that demographics are correlated with the outcome of interest, estimates will suffer from sizable coverage bias when attempting to make population estimates.

Social media, like surveys, is also subject to measurement error. That is, social media content may not be accurate. Individuals may wish to present their “best” self, so their posts do not paint the full picture of their attitudes and beliefs (Schober et al., 2016). In some countries, they may also avoid posting information on sensitive topics (e.g., critiques of the regime) out of concern for retaliation. This type of filtering suggests that social media data may suffer from missingness error and may not represent all viewpoints as they exist in the population. Other forms of error are introduced by “fake”

___________________

¹⁶ International privacy laws are rapidly changing, and access to a number of platforms is likely to change over time.

¹⁷ https://support.google.com/trends/answer/4365533?hl=enhttps://support.google.com/trends/answer/4365533?hl=en.

¹⁸ https://documenter.getpostman.com/view/9197254/SzRuZCCj?version=latest.

¹⁹ https://www.zillow.com/research/ztrax/.

accounts. Over half of all links to popular websites are likely posted by bots (Wojcik et al., 2018). Including tweets from these automated accounts can add significant noise and bias into all forms of analysis.

Given space limitations, this discussion is limited to the largest and most common sources of bias, but it is critical to understand all potential error sources when considering whether or how to use social media data. Luckily, several researchers have outlined various frameworks for considering error in social media data, and, more broadly, Big Data (Amaya, Biemer, and Kinyon, 2020; Hsieh and Murphy, 2017). We recommend consulting these frameworks for a deeper understanding of the error sources, brainstorming the different types of social media available, and identifying potential solutions, but given these significant biases, the use of social media data should generally be limited to the scenarios outlined above. Specifically, the strongest social media analyses typically limit inference to active social media users (though this population may be of little interest to the IC), assess social networks, or are used as a last resort when data are needed immediately and no other source is available.

Other Data Sources

There are a host of additional data sources that rarely measure social attitudes, but may offer important insights into the economic, social, environmental, and cultural welfare of a nation. Because they are not used to measure attitudes, we have limited discussion of these other data sources to a brief summary in Table 3B-2. References are included to provide researchers with the tools to learn more about each data source.

These data sources are constantly evolving. As a result, researchers should conduct a thorough review of the current literature to stay up to date on current best practices on how to use or evaluate such datasets. Moreover, the nascent nature of these data sources means that examples of their use and evaluation are typically limited to the United States and other Western European countries.

BLENDED DATA

As alluded to in the previous section, there may be situations where it is useful or necessary to blend multiple data sources. While there are multiple ways data may be blended (de Leeuw, 2005), this section is limited to the three most common approaches. Moreover, methods on how to blend data are discussed in the “Integrating Data” paper. This section discusses types of blended data and their strengths and weaknesses.

cisions. In this case, the analysis is done on each variable or data source individually, but the results from all sources are considered in addressing the research question. Economists, for example, use multiple indicators of economic strength to make policy decisions and predict future growth (Amadeo and Anderson, 2022). Alternatively, one may use a single dataset, but conduct an analysis in multiple ways to determine the sensitivity of an analysis by observing whether different analytical approaches yield the same results. All statistical tests include assumptions (e.g., a t-test assumes independent observations), but data do not always fulfill each assumption. Using multiple approaches to analyze the same set of data allows researchers to minimize the threat of statistical error due to the weakness of a given analytic test. Using multiple measures (either from different data sources or from different analytic approaches) may be time-consuming to implement, but they provide the researcher with the most confidence that the results are robust and accurate, even more so than a single analysis of a single probability-based general population survey.

All in all, blending data using any of the methods described above can yield important and sizable advantages for cost and data quality. However, if collecting primary data, it is important to design the study in a way that facilitates blending. A researcher could, for example, use the same question wording and collect the same information when blending data from multiple frames and multiple people, or add a confirmation screen for variables like email addresses or the spellings of names that will be used in the linkage process.

ETHICAL CONSIDERATIONS

In addition to considering data sources that are accurate, timely, and budget-conscious, researchers must also consider the ethical nature of the data. When considering the use of any of the data sources discussed in this chapter, researchers should consider three questions:

• How were the data collected, including what data were collected?
• How were/are they stored?
• How are they used?

When collecting data, it is critical to ensure that U.S. laws as well as laws in the country of interest are adhered to. This may include submitting the research protocol to a U.S. or in-country institutional review board (APA, 2017). For survey data, this also includes collection of informed consent from adults and consent or assent from minors, depending on the situation (Office for Human Research Protections, 2016). Participants should understand what they are being asked to do, what their rights are, and how

their data will be used. However, ethical data collection involves more than just informed consent. If researchers are collecting information on sensitive items, then special attention should be paid to make sure resources are available for adverse reactions (e.g., providing crisis telephone numbers when asking about suicide). Proper cultural norms should be followed to make sure all respondents feel comfortable and at ease. For example, it may be inappropriate to use an interviewer who is the opposite sex of the respondent. Interviewers should also be properly trained to avoid any type of coercion such as threats to participate and attempts to elicit preselected responses. Not only do these types of protocols ensure data are collected ethically, they also improve data quality. Respondents who do not trust the researchers are prone to provide inaccurate answers.

In some instances, it may also be possible to collect informed consent for organic data. For example, if one is appending social media data to an individual’s survey data, it would be necessary and possible to ask for consent to link the data in the survey. In other cases, it is not possible. The Twitter API allows researchers to download tweets from public accounts. In these circumstances, it is implied that publicly available information can be ethically used for research purposes. (We debate this under the usage discussion later in this section.)

Data (regardless of source) should be stored on secure servers. The U.S. federal government provides rules on what is required for data to be considered “secure.” (NIST, 2004) These guidelines vary by the source of data. For example, publicly available data and data for which participants have no expectation of privacy does not require secure storage, whereas data that include personal health information requires researchers to store data in a “FIPS Moderate” environment. In general, the more sensitive the information is and/or the higher the risk of being able to identify an individual from a data source, the more security measures are required.

This concept becomes slightly more complicated when considering blended data. Consider a survey dataset that has no sensitive information and no PII. Now, consider that the researcher appended information on distance to the nearest hospital, distance to the nearest school, and distance to the nearest grocery or farm stand, all of which are publicly available. The blended dataset has a much higher risk for identification of the individual because it creates a triangulation of geographies—there are only so many housing units that are exactly X distance from a hospital, Y distance to a school, and Z distance from a grocery. As a result, the blended data may need to be stored more securely than any of the individual sources.

Data usage should receive consideration equal to that pertaining to data collection and storage. Before using data, the researcher must consider the informed consent. For example, most informed consent statements include a sentence that the results will be used for research purposes only.

Data from these sources should not be used to identify individuals and their access to programs or legal status. Violation is not only illegal but makes it more challenging for future researchers to gain cooperation and collect accurate information from marginalized groups.

REFERENCES