Inference and Evidence

Burning Glass Institute advances the debate over the ROI of short-term credentials. But experts caution against states relying on the data. Also, Austin Community College’s playbook for training semiconductor industry technicians, and an essay on building the capacity of workforce pathways at two-year colleges. (Was this newsletter forwarded to you? Subscribe here.)

Measuring the Value of Short-Term Credentials

New research from the Burning Glass Institute has reinvigorated discussions about which nondegree credentials pay off in the labor market, while also highlighting how hard it is to design data sets for real-world use.

The nonprofit institute’s Credential Value Index is the culmination of a multiyear project to develop an independent measure of the career and wage gains of short-term credentials. That sort of information on certifications, bootcamp programs, professional courses, and other nondegree training typically isn’t captured by state and federal data sources.

As states ramp up spending on short-term credentials, policymakers are clamoring for better data on the ROI of those programs. BGI says it hopes states can use the index to assess which programs to fund, promote, or endorse. (The group also wants to put the data in the hands of leaders in higher education and K-12 schools, education providers, and working learners.)

However, a wide range of experts say that while the index provides useful directional insights, the underlying data sets are too limited and not transparent enough for state policymakers to rely on for funding decisions.

Modeling and estimates only go so far, was the consensus I heard.

What’s Included? The index is powered by BGI’s proprietary database of the career histories of roughly 65M Americans, which comprises about 40% of the workforce. The institute says it parses each profile with AI and classifies it into a standard set of employers, education histories, occupations, and job titles.

As with any sample, some groups are overrepresented, say Matt Sigelman, the institute’s president, and Shrinidhi Rao, its chief of staff. White-collar workers have a somewhat greater presence, for example.

Even so, Sigelman and Rao point to an extensive inclusion of low-wage workers. “For example, we have records of hundreds of thousands of Amazon warehouse workers, hundreds of thousands of Starbucks baristas, and hundreds of thousands of McDonald’s restaurant workers—certainly a statistically meaningful sample by any measure.”

The index also was able to weight its sample to the true composition of the U.S. workforce, they say, similar to how sophisticated electoral polling works. And all of its measurements are “reported relative to a robust counterfactual,” says BGI, which develops a control group for each credential comprised of people working the same role, with similar demographic characteristics and educational levels.

“That allows us to observe to what degree a given credential impacts outcomes on an array of measures relative to peers who didn’t bother,” say Rao and Sigelman.

Tracking Wages and Careers: The index infers earnings of credential earners based on models that consider both self-reported sources like Glassdoor and government data sets such as the Occupational Employment and Wage Statistics survey from the U.S. Bureau of Labor Statistics.

BGI uses that data to model wages for all 65M workers, for each year in their career, factoring in where they work, years of experience, region, and other characteristics of their employment. 

The models are quite accurate, say Sigelman and Rao. For example, BGI compared its estimates with Florida’s longitudinal data system records, which are drawn from unemployment insurance wage records. “In a significant majority of cases, the inferred wages we reported were within 2-5% of these records,” they say.

To measure career advancement, the index taps individual résumés and LinkedIn profiles. BGI also applies quality filters to that data. The institute says the resulting observations are robust, particularly for comparisons across credentials and providers, without systematic biases.

Using the database, BGI identified 23,444 commonly reported credentials from 2,056 providers. It jointly produced an analysis with the American Enterprise Institute on the values of those credentials, with troubling results for both learners and taxpayers.

Just 12% of the nondegree credentials captured in the index deliver significant wage gains that workers who earn them wouldn’t have gotten otherwise, according to the report. And just 18% of credential earners are likely to see wage increases their peers didn’t enjoy.

BGI is bullish on the potential for state agencies to use the index, such as providing an evidence-based standard for which data to include on eligible training provider lists. For example, the institute has worked with Idaho on assessing which short-term programs to fund as part of the state’s LAUNCH grants, which help cover education and training programs at eligible institutions.

Limitations of Inferred Data: Experts say the index and the report from BGI and AEI are welcome efforts that help to advance understanding of the value of certain nondegree credentials.

However, all the experts I spoke with say state policymakers shouldn’t rely on inferred data to make funding decisions.

Self-selection is a big problem for estimating the relative value of nondegree credentials, says Kyle Albert, a professor at George Washington University who manages operations of the Non-degree Credentials Research Network.

“In many cases the data will just reflect differences in motivation between people who pursue credentials and those who don’t,” says Albert. “It’s very difficult to tell whether the credential actually causes the wage differences we see.”

Albert sees some potential for the index to eventually inform the development of state-level lists of credentials of value. But he says a complicating factor is that it’s impossible for researchers to evaluate results from the index in a systematic way, because those results depend on a proprietary data set that BGI owns.

“The lack of replicability of Credential Value Index results is going to be a major sticking point until they’re able to be more transparent in their methodology,” he says.

Raising similar concerns was Harry Holzer, an economist, professor at Georgetown University, and nonresident senior fellow at the Brookings Institution. He questioned whether the database of 65M workers is truly representative and if control group comparisons might differ systematically in ways not captured by the data.

“The data look promising but by no means foolproof,” Holzer says. “I hope BGI continues to present data on their representativeness or how well their results compare with those in other studies, like those based on actual credentials and earnings data that many states have made available to researchers or the public.”

Michelle Van Noy, director of the Education and Employment Research Center at Rutgers University, says the index from BGI is a “powerful use of existing sources of data” that’s presented in a clear and accessible way. 

However, Van Noy says, it would be helpful to see more detailed methodology from BGI to be able to assess the data more fully. She points to tables on the distribution of occupations represented, for example, as well as more detail on the process of constructing the database.

“While the authors acknowledge the limits of the data, these ought to be made clear to policymakers and the public—that this is one source of data, and others are also needed that reflect the actual experiences of learners,” she says.

The missing link when policymakers make funding choices about the confusing and rapidly expanding nondegree space, say Van Noy and several other experts, is student-specific data at the state and/or institutional level that can be linked with wage records.

Getting that information is no easy task—particularly given the long-standing policy environment and the shortcomings of available public data sets. Yet investments in how to measure the value of short-term credentials are badly needed, many observers say. And it’s okay if some of those experiments don’t work out.

CredLens, a recently launched nonprofit affiliated with the Strada Education Foundation, is attempting to bridge data gaps with nondegree credentials. 

The project is seeking to “build a new national data trust by collaborating with credential issuers and primary outcomes data sources to generate verified insights on the outcomes of individual credentials,” CredLens says.

BGI’s new index is a complement to that work, the group says:

“We appreciate the focus of BGI’s model to provide directional signals of quality using publicly available data for a subset of credentials, given that so little information exists regarding the efficacy of the 1M+ nondegree credentials currently available.”

Homegrown, but Not Alone

Austin Community College has been building the semiconductor workforce in the “Silicon Hills” for a decade. And the college has become something of a sensation following the passage of the CHIPS and Science Act as regions look for a blueprint for creating workforces to match the factory-building boom.

But Garrett Groves, the vice chancellor of strategic initiatives at ACC, wants his peers to know there’s no plug-and-play solution. 

ACC, which has become a go-to trainer for chipmakers such as Samsung, NXP, and Applied Materials, asked the Institute for Networked Communities (INC) to tell its full story in a case study that was released last month. The goal was to share what’s worked (and not) in Austin—and to also emphasize how much regional peculiarities matter.

Groves and the report’s authors say there is an urgent need for a cross-regional network that recognizes geographic differences but also speeds up the learning cycle and tackles deeper, strategic issues. (Read the full story about Austin’s experience and the call for the action network in Work Shift this week.)

New America, through its Future of Work and the Innovation Economy initiative, is sharing related insights this week. In an essay for Work Shift, Shalin Jyotishi, founder and managing director of the initiative, argues that more capacity-building efforts for workforce pathways at community colleges need to start from the bottom up, with state and regional organizations. (Read Jyotishi’s take here on the three ways bottom-up efforts excel.)

The common lesson across both New America’s and INC’s research: Workforce development should be homegrown, but not alone. —By Elyse Ashburn

Open Tabs

Credential Gap
The U.S. faces a shortage of middle-skills credentials—certificates and associate degrees—aligned with occupations that are high-paying for early-career workers without a bachelor’s degree, according to Georgetown University’s Center on Education and the Workforce. The annual shortage of 712K credentials, which is expected to persist at least through 2032, falls across four categories of occupations: blue-collar, management and professional office, STEM, and protective services.

AI and Jobs
AI labs should design technology that works hand in hand with humans, rather than trying to build autonomous systems that replace us, Tim Wu, a law professor at Columbia University, writes in The New York Times. AI rarely works well without human involvement, he writes. And while productivity gains will lead to job loss, AI could be an augmenting technology—like the tractor or personal computer—where the innovation was worth the disruption.

AFT on Apprenticeship
A new coalition featuring five governors and labor unions is seeking to align career training while pushing for policy changes, reports Patrick O’Donnell for The74. The Education and Apprenticeship Accelerator includes the American Federation of Teachers and CareerWise USA, a nonprofit that runs apprenticeship programs. Randi Weingarten, AFT’s president, told O’Donnell that the union sees a need to shift away from the “college for all” mindset.

Wraparound Supports
Verizon’s Total Wireless seeks to support working students through a new program that will provide grants to nonprofits that address pain points for these learners. Total Wireless partnered with Empower Work, a nonprofit that offers text-based coaching, to identify five local groups in Chicago, Dallas, and Detroit that will receive grants between $50K and $150K each. The funding will back community-based job training, career advancement, and other supports.

Earn-and-Learn
Apprenticeship degrees formalize the academic merit and credentialing of apprenticeship programs, elevate the status of apprenticeships as an alternative form of higher education, and expand access to apprenticeship, write Deanna Ross and Bruno Manno in a report for the Progressive Policy Institute. The earn-and-learn model also can be integrated with higher ed, eliminating the either-or choice between a job and a degree.

Job Moves
JP Moran has been named chief commercial officer of Guild, the talent development company. Moran previously was chief revenue officer at Lightcast and, before that, a senior director at Coursera.

Laura Ullrich has been hired as director of economic research at Indeed. Ullrich previously was senior regional economist at the Federal Reserve Bank of Richmond, where she directed the Community College Initiative.

Thanks for reading. To support our work, donate here. And if this newsletter was forwarded to you, subscribe here. —PF