Evaluating Truth in Sampling
The headlines are powerful:
- "The Latino Contribution to U.S. Economic Dynamism"; Hispanic Executive, 4/2019
- "Latinos: A Powerful Force Turbocharging Small-Business Growth and Driving $700B into the U.S. Economy"; CNBC, 9/2018
- "The 'Latino Factor' Will Save America's Economy"; Wall Street Journal, 8/2018
- "A New Look at How Latinos are Powering the U.S. Economy" ; NBC News, 8/2017
- "Latinos Key to U.S. Economic Growth"; CNN Business, 7/2017
Over the last 10 years, Hispanics have made significant advances in income, educational attainment, housing and employment. As growth in these sectors continues year after year, Hispanics are fueling increases in the U.S. economy. In addition, according to the Selig Center for Economic Growth, Hispanics currently account for $1.5 Trillion in buying power, representing 39% of the buying power of the U.S. multicultural market.
Why, then, when it comes to advertising to the Hispanic market, do marketers continually say that Hispanics, and Spanish Language radio stations in particular, “don’t meet the criteria” to be included on an advertising campaign? What are advertisers looking at that would summarily disqualify such a valuable marketplace?
It’s the Research, Stupid
More often than not the answer is syndicated qualitative research. All the information cited above is based on reports using Census data, which is derived from an actual count of the U.S. population. Syndicated survey research such as Scarborough, GfK/MRI, Kantar, or Millward Brown are often used by advertising and media companies to profile consumers and determine which media outlets best deliver that consumer. However, syndicated research - that compiles data using a sample - has not always kept pace with the real gains in the multicultural population that we see in the Census.
This is because syndicated survey research such as Scarborough depends on the fact that the number of surveys used in the research needs to be in proper proportion to the demographics of the market. Demographics such as age and gender as well as race and ethnicity must be taken into account when surveying a marketplace in order to accurately represent the entire population. Unfortunately, this proportionality does not always happen. More often than not, Scarborough under-samples either or both the African American and Hispanic populations in the markets they survey. When you are starting out with a lower than expected sample for a particular demographic group, the radio stations that serve these populations are at a significant disadvantage.
At a much higher level, data based on a small sample size is challenged with providing quality insights. The inverse is not always true, either. Large samples sizes - in and of themselves - are not inherently better. It is the combination of proper sampling coupled with sufficient size that yields the best sampled data.
The Power of “30”
It seems it has become commonplace to say that the sample is too small, so the research is inaccurate. So much so, that it is often looked upon as an excuse used by sales to explain away poor performance by a radio station. In addition, the threshold of 30 respondents has been used for years to determine what is good versus bad research. In fact, some third-party systems will not even allow the user to pull a report if the sample for that report is under 30. However, like all urban legends there is a basis in fact for referencing low sample in general, and the threshold of 30 in particular. Statistics geeks call it the central limit theorem.
What this means for media geeks like us is that the larger the sample, the more representative of the population it is. As a result, the more stable and accurate the data based on that sample will be. Basically, when it comes to survey research, bigger is often better - provided that the underlying representativeness of the sample is accurate.
Looking Under the Covers
Let’s take a closer look inside the Scarborough surveys as an example to see what is going on. In an analysis of five multiculturally diverse markets (New York, Los Angeles, Chicago, Houston, and Miami) over nearly 10 years we saw that the ethnic samples have not kept pace with the population increases in these markets. Since 2010 the Hispanic and non-Hispanic Black populations have increased by 18% and 15%, respectively. Over the same period, Hispanic sample has declined by 20% and non-Hispanic Black sample, while up slightly at +6%, is well below the increase in population.
Proportionality of the sample compared to the population did not fare much better. While non-Hispanic Black proportionality has improved recently, historically it was below average. Hispanics, on the other hand, are significantly underrepresented in these markets. In order to get an accurate depiction of the population, ALL segments of the population must be represented equally, every time.
It should be noted, that Scarborough does produce custom surveys that include additional Hispanic sample in 8 high density Hispanic markets. However, these special surveys come at an additional cost to subscribers. Representative proportionality should not come with conditions. Subscribers to Scarborough data do so with the understanding that all segments of the population are properly represented on a consistent basis.
How Does this Affect My Station?
There are many ways qualitative, and Scarborough in particular, can be used to show the propensity of Hispanics or African Americans to consume certain products or services and to engage in specific behaviors or events.
Qualitative station rankers are one way. They will show which stations have the largest audience (qualitative cume) in a given category. They can also show the likelihood that a station’s audience meets a certain target compared to the population (Index). There is nothing wrong with qualitative rankers, and they are the most common way to evaluate individual stations using qualitative data.
But they are not always the best way and they are certainly not the only way to look at qualitative data. In fact, qualitative rankers often do a disservice to ethnically formatted stations – both African American and Hispanic – because of the sampling issues.
It is not unusual to use the qualitative data at a very granular level. In fact, a common request from an advertiser would be Men 25-54 with a household income of $75,000 or more and own a home. A closer look at the ethnic stations in 3 markets used in our analysis shows how the lack of sample can affect individual station performance.
Keeping the threshold of 30 in mind when looking at the above analysis, it is obvious that the low sample sizes at the station level are contributing to the performances you see of the ethnically formatted stations. Conversely, the “general market” stations with large sample sizes fare quite well. Qualitative rankers are often used to evaluate which stations are “best” for a particular target using research sources that can be unstable at best, and inaccurate at worst.
But What Is the Alternative?
In order to offset the sample size issues seen at the station level, we often look at the profile of the marketplace as a whole. In addition, qualitative targets should be looked at independent of one another. In this case, income separate from home ownership. Remember the threshold of 30? While it is true the sample for the Hispanic or African American population is not what it should be, Scarborough provides a more reliable sample when analyzing it at the market level than it does at the station level.
The benefit of taking a 30 thousand-foot view of the marketplace is that you can get a more complete picture of the consumer group or target. What does the Hispanic or African American population look like in a market? What is their mindset? What are they buying? Where are they shopping? Is it an increasing market or a declining market for the category or advertiser? How does all of this compare to their non-ethnic counterparts? These questions cannot be answered by looking at a ranker based on a granular target. But they can be answered by analyzing the data in a different way. If you look at the consumption behaviors of the population as a whole to determine if these marketplaces fit your profile, then you are getting the audiences to these by default.
We already know that 80-90% or more of the listeners to ethnically formatted stations are either African American or Hispanic. Why slice and dice an already sub-par sample base even further by looking at individual station performance? At 80-90%, the station audience is the population, and vice versa.
Until syndicated survey research can level the playing field for the entire marketplace, on a consistent basis, we must find other ways to use the data that we have available. Further, we must continue to push for the use of alternative sources (such as the articles and free reports cited above) in addition to the traditional syndicated surveys currently being used to profile the multicultural marketplace. Gathering information from as many sources as possible helps build a better picture and can only benefit the evaluation process when planning a Radio campaign.