The Trump administration’s plan to deal with a possible significant undercount on the 2020 U.S. census is seriously flawed, according to former agency officials and other experts in survey research.
Late last month, Commerce Secretary Wilbur Ross endorsed a controversial request from the Department of Justice to add a citizenship question to the decennial census. Justice Department officials said they needed more detailed information on every U.S. resident to prevent discrimination under the 1965 Voting Rights Act.
Civil rights groups and others blasted the decision, predicting that the new question would prompt many immigrants to refuse to fill out the form. The resulting undercount, critics say, could invalidate census data used to apportion congressional seats and distribute three-quarters of a trillion dollars in federal funds.
In an eight-page memo released 26 March, Ross asserted there was no evidence supporting the critics’ claim that the new question would result in depressed response rates or a sizable undercount, citing several unnamed experts. But even if that happens, Ross said, the Census Bureau would be able to use the massive trove of information already on file with the government—relating to employment, taxes, health, education, housing, and social welfare benefits—to get an accurate count.
ScienceInsider has learned that at least three of the experts consulted by Ross disagree—sometimes strongly—with his decision to add the question. And they and other census specialists believe Ross has vastly understated the many technical and legal barriers to both accessing and making effective use of those data, known as administrative records. That process is unlikely to be completed by Census Day on 1 April 2020, they add, increasing the chances that a significant undercount could become reality.
“It would be incredibly challenging … [and] it’s going to be very hard to meet that deadline,” says Amy O’Hara, who until last fall directed the Census Bureau’s efforts to expand its use of administrative records. She is now a senior research scholar at Stanford University’s Institute for Economic Policy Research in Palo Alto, California.
“The process sounds easy to a lay person, but it’s not,” says Hermann Habermann, a former deputy census director under President George W. Bush and now an occasional consultant to the Committee on National Statistics at the National Academies of Sciences, Engineering, and Medicine in Washington, D.C. All data sets have errors, for example, but it is not always clear how to correct them. Further complicating matters, other experts note, is research showing that some immigrants do not tell the truth when answering a similar citizenship question posed by another census survey.
A controversial request
In its December 2017 request to Ross, Justice Department officials argued that current data weren’t sufficient for their needs. In particular, they asserted that numbers arising from a three-part citizenship question included on the American Community Survey (ACS), which annually polls 3.5 million households using 72 questions taken from what used to be the long form of the decennial census—didn’t provide sufficiently detailed, census block-by-block coverage.
Civil rights and immigration advocates panned the request as a thinly veiled political move by the Republican administration of President Donald Trump. The people most likely to sit out the 2020 census and miss being counted for apportioning congressional seats, critics say, would likely vote disproportionately in any election for candidates running as Democrats. The critics noted that previous Justice Department officials had been satisfied with the smaller sampling of the U.S. population provided by the ACS and had never asked for data on every resident. They also said there wasn’t time for census officials to do a thorough job of determining what impact the question might have on response rates and the overall accuracy of the decennial census.
In his memo laying out his decision, Ross said he embraced the goal of “a complete and accurate” census. And he weighed three options to achieve it—omit the question, add the question but don’t use administrative records to shore up the results, or rely exclusively on administrative records.
Ross says he rejected all three in favor of what he called “Option D”: Ask the question, and use administrative records to fill in any holes. To that end, he said, the agency “is working to obtain as many additional federal and state administrative records as possible.”
Mission impossible?
Administrative records can be used to improve a count’s accuracy in several ways, experts say.
One common use is to identify households that have not responded to requests to fill out a survey. For example, if postal or tax records suggest that people are living at a specific address not already in the census’s master address file, but nobody at that address has returned a census form, the agency would target the address for follow-up mailings. The last resort is to send out enumerators, sometimes repeatedly, to knock on the door. (Those field operations are also the biggest expense for any decennial census, with in 2020 is estimated to cost $15.6 billion.)
Another way the Census Bureau uses administrative records is to double check the accuracy of survey answers, or fill gaps in an incomplete survey. But that process has never been tested for a citizenship question on the decennial census, and specialists say it is much easier said than done.
The first step, O’Hara says, would be to identify which other agency or commercial firm might have data on citizenship. But although the Census Bureau has the authority to request data from other agencies, only the Internal Revenue Service (IRS) “is compelled to share its data,” she notes. So census officials would need to negotiate a data-sharing agreement with the second entity that would also spell out how to transfer the data securely.
Even if another agency is amenable to such an agreement, the negotiations can take a long time. For example, the Census Bureau has spent 7 years negotiating with the Department of Veterans Affairs to obtain a usable list of all veterans that it could use to verify information from the ACS, she notes.
The next barrier is what O’Hara calls “harmonization of the data source”—meshing different data sets in a way that makes them usable. In the case of citizenship, O’Hara says, “I can’t estimate how difficult [harmonization] would be because these data have never been drawn together.”
With respect to the citizenship question, O’Hara speculates that census officials might turn first to Social Security records. The bureau already uses such “high-quality data” to check age and sex, she notes, as well as to fill in missing race and ethnicity answers.
Unfortunately, those records have serious limitations with regard to citizenship. Anyone with a legal right to work in the United States can receive a Social Security card, even if they are not a citizen. Although the government asks for citizenship information “when someone first applies,” O’Hara notes, “the file is never updated, because there’s no requirement to do so.” In other words, a person’s immigration status could change, and the Social Security Administration wouldn’t necessarily know about it.
Given those limitations, she says, census officials might look to other federal agencies for current information on immigration status. That could mean trying to mesh their records with a taxpayer number from IRS or with the U.S. Citizenship and Immigration Services (CIS), which issues green cards for permanent residency. But CIS, she says, “wouldn’t necessarily” know both a person’s naturalization and work status.
Even a name and date of birth may not be specific enough for the agency to track down needed information unless the record also includes a current address. That’s because agency officials “need to be able to identify a person, at a place, with certain attributes” before they can be included in the decennial census, O’Hara says. “You have to connect the siloes,” she emphasizes.
Another huge challenge is what to do when government records contain conflicting data on the same person. Race is a good example, O’Hara says. Researchers have found that “people respond differently to a question about race and ethnicity depending on how it’s asked,” she notes. “And I think that interpreting answers to a question about citizenship is likely to be even more complicated.”
The 2020 census will have a new question on citizenship
Here is the citizenship question the administration has decided to add to the 2020 census. Trump officials claim they can avoid problems caused by the controversial question, but experts are skeptical.
What they actually said
The only issue on which Ross and his critics may agree is that a question on citizenship will likely strike a nerve among respondents. But that represents a tiny patch of common ground.
Ross asserts several times in his memo that critics have not provided any evidence to back up their claim that response rates would be “materially” lower if the question appears on the 2020 census. He attempts to bolster his argument by offering comments from three of the more than two dozen unnamed experts he consulted before making his decision.
ScienceInsider was able to identify some of the experts because Ross identified them by past or present job titles. And contrary to the impression left by the memo, three of the experts cited have serious misgivings about adding a citizenship question to the 2020 census.
One is Habermann, the former senior Census Bureau official. Ross writes that Habermann (and a second expert) “confirmed that … no empirical data existed on the impact of a citizenship question on response” rates. Habermann says that is correct, as far as it goes, but that Ross left out something even more important.
“When someone wants to put a question on the census, there’s a high burden of proof that must be met about its value,” Habermann says. “And I told him that I don’t think the case has been made that [the citizenship question] is so important that it’s worth endangering this fragile instrument.”
The second expert referred to by Ross is Christine Pierce, senior vice president for data science at Nielsen, a global information and measurement company best known for its monitoring of television viewing. Pierce declined to speak with ScienceInsider. But a company spokesperson based in New York City says that “Nielsen does not support the inclusion of a question on citizenship for the 2020 U.S. census because we believe its inclusion could lead to inaccuracies in the underlying data.”
Ross’s memo also asserted that Nielsen has found no “appreciable decrease in response rates” when it has used “sensitive” questions from the ACS on its own surveys. That is not correct, according to a source familiar with the Nielsen surveys who requested anonymity. The company found that questions relating to religion and sexuality do depress response rates, according to the source. In addition, the source notes, a question about citizenship in a marketing or viewer survey is far less politically charged than were the same question to appear on the U.S. census.
The third identifiable person in Ross’s memo is Robert Groves, who led the agency from 2009 to 2012. Ross writes that Groves told him “while he wished there were data to answer the question [of response rates], none existed to his knowledge.”
Groves, now provost of Georgetown University in Washington, D.C., didn’t respond to requests to discuss the matter. But it’s no secret that he strongly opposes adding a citizenship question to the 2020 census.
“It is highly risky to ask untested questions,” Groves wrote in a public letter to Ross on 26 January that was co-signed by five other former census directors. “There is a great deal of evidence that even small changes in survey question order, wording, and instructions can have significant, and often unexpected, consequences for the rate, quality and truthfulness of response.”
Ross’s repeated assertion that there’s no evidence to support critics of the added question is a red herring, says another co-signer of that letter, John Thompson, who retired last spring after 4 years as census director. “That question has never been tested in a contemporary census environment,” says Thompson, who is now executive director of the Council of Professional Associations on Federal Statistics in Washington, D.C. “So of course, there’s not going to be any evidence.”
Whose data do you trust?
There’s one more wrinkle in the complex web of issues surrounding the citizenship question. Research casts doubt on the accuracy of the answers to the three citizenship questions on the ACS. The research points to men born in Mexico and those who have lived for less than 5 years in the United States as the cohorts most likely to overrepresent their citizenship status.
How do the researchers know? They compared ACS data with records from the Office of Immigration Statistics (OIS) within the Department of Homeland Security, and assume that the OIS records are more accurate than self-reported data on the ACS.
Still, there are important caveats to both sets of data. For example, to know the actual number of naturalized citizens in the country at any one time, researchers also must estimate the number who have died since becoming naturalized and the number who have left the United States—returning to their place of birth, perhaps, or moving to another country.
“The OIS records only tell you if a person has been naturalized at some point in the past,” says Jennifer Van Hook, director of the Population Research Institute at Pennsylvania State University in University Park and lead author on a 2013 paper that examined the ACS’s ability to count naturalized citizens. “We don’t know what happens to them after that.”
There’s also reason to believe that the ACS doesn’t capture a representative sample of this immigrant population, Van Hook says. As a result, she notes, any count of naturalized U.S. citizens comes with large error bars.
Like many social scientists, Van Hook thinks that the current hyperpartisan debate over immigration could make the data from future surveys even less reliable. “I think it’s safe to say that any research done prior to January 2017 [when Trump took office] is now out of date. So my study would have to be done over,” she says.
Next month, Congress will get a chance to ask census officials about the citizenship question. The oversight and government reform committee for the U.S. House of Representatives has scheduled a public hearing for 8 May, following up on a closed-door briefing it received on Tuesday. This week, civil rights groups urged the relevant Senate panel to do likewise, reinforcing a request last month from Democrats on the panel.