Study Warns Weak AI Rules Can Make Technology Less Safe

The study, published in the Proceedings of the National Academy of Sciences by researchers from Cornell University and Carnegie Mellon University, used theoretical economics and game theory to model how regulatory requirements influence investments in AI safety. Its central finding is that poorly targeted or insufficiently stringent regulation can create incentives for companies to reduce their own safety investments and shift responsibility to others.

The researchers draw an important distinction between two points at which governments can regulate AI: the developers of general-purpose models, such as OpenAI, Google and Anthropic, and downstream companies that deploy those models for particular purposes, such as medical diagnostics, eCommerce or customer service chatbots.

Regulating individual applications may appear to be the most logical approach because risks often emerge when AI is deployed in specific settings, per Gizmodo’s report on the study. But the study found that focusing regulation primarily on downstream users can create unintended incentives for model developers to spend less on safety.

When downstream companies are expected to ensure that applications meet regulatory requirements, general-purpose model providers may “free ride” on those investments by cutting back on measures such as third-party safety audits.

“There’s a free-riding behavior that occurs,” principal author Benjamin Laufer said. “The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.”

That distinction is particularly relevant to the emerging U.S. regulatory landscape. At the federal level, policymakers have focused much of their attention on companies developing advanced or frontier AI models, including questions surrounding model safety, testing and national security. States, meanwhile, have generally concentrated more heavily on downstream applications, including AI used in employment, healthcare, insurance and other high-impact decisions.

The study suggests that neither layer of regulation should be considered in isolation. Instead, policymakers should account for how obligations imposed at one point in the AI supply chain can alter safety investments elsewhere.

The researchers found that strong, appropriately placed regulation can produce benefits for both safety and economic returns. Their model suggests that “stronger, well-placed regulation can mutually benefit all players” by increasing end product safety while also improving the utility derived by general-purpose AI developers and downstream specialists from their investments. The study defines utility as a company’s share of revenue minus its investment costs.

The optimal outcome occurs when regulators require sufficient safety investment from both model developers and downstream companies rather than allowing either side to assume the other will shoulder the burden.

Researchers compared the problem to the classic “prisoner’s dilemma” in game theory. Without confidence that other participants will invest adequately in safety, each company has an incentive to protect its own economic interests by spending less and relying on others. The result can be collectively worse even when cooperation would benefit everyone.

Strong regulation across the AI supply chain, by contrast, can reduce that uncertainty and discourage companies from shifting responsibility. In that framework, regulatory requirements effectively create conditions in which both model providers and downstream deployers have incentives to make complementary safety investments.

The findings could complicate the broader U.S. debate between those who argue that excessive regulation could undermine innovation and U.S. competitiveness with China, and supporters of stronger safeguards, who contend that commercial incentives alone are insufficient to address AI risks.

Rather than framing the choice simply as regulation versus deregulation, the study argues that the design and placement of regulation may be just as important as how stringent it is.

“People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology,” Laufer said. “To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity.”