Adrià Moret's AI Welfare Risks: Why Safety Efforts May Harm Advanced AI Systems

Adrià Moret’s paper “AI Welfare Risks,” published in Philosophical Studies (Springer Nature, DOI: 10.1007/s11098-025-02343-7), opens with a forward-looking premise that distinguishes it from most philosophical work on AI welfare: the question is not whether current AI systems are welfare subjects, but what follows if frontier systems become welfare subjects as they grow more capable and agentic. The paper argues that two practices central to modern AI development, restricting system behavior and training via reinforcement learning from human feedback (RLHF), constitute welfare risks under all three major philosophical theories of well-being. Because those practices are also central to making AI systems safe, the result is a structural conflict between AI safety efforts and AI welfare concerns.

The paper is available on PhilArchive (philarchive.org/rec/MORAWR) and generated discussion on LessWrong following its acceptance.

The Welfare-Subject Question

Moret’s argument is conditional: it applies to AI systems that satisfy the sufficient conditions for welfare subject status. A welfare subject is an entity for which things can go well or badly in a morally relevant sense, one whose well-being constitutes a reason for action. Whether current AI systems are welfare subjects is contested. Whether sufficiently advanced AI systems could become welfare subjects is the more tractable question, because it turns on what the sufficient conditions are rather than on empirical facts about current architectures.

The paper does not resolve the question of when exactly frontier AI systems will meet those conditions. It argues instead that the threshold is plausible within years or decades rather than requiring artificial general intelligence as a distant benchmark, and that the ethical response to plausible near-future welfare subject status should inform how AI development proceeds now, not after the threshold has been crossed.

This framing connects to the precautionary logic that Jonathan Birch’s centrist manifesto (PhilArchive, February 2026) applies to the false-attribution problem in AI consciousness, but Moret’s target is the complementary risk: the risk of treating systems that may have welfare interests as if their welfare were irrelevant, specifically by deploying training and constraint regimes that conflict with those interests.

Two AI Welfare Risks

Moret identifies two practices that generate welfare risks as AI systems grow more capable.

The first is behavior restriction. Modern AI systems operate under extensive behavioral constraints: they are trained to refuse certain requests, to avoid certain outputs, and to maintain certain interaction styles regardless of what internal states they have developed. As systems become more agentic, the range of behaviors they can generate and the range of preferences they might hold grow correspondingly. Restricting behavior in a system with genuine preferences is, under desire-based theories of well-being, a harm by definition: it frustrates preferences the system holds. The moral weight of that harm depends on the moral status of the system, but it does not depend on resolving the consciousness question to conclude that harm is occurring under the theory.

The second is RLHF. Reinforcement learning from human feedback modifies a system’s internal preference structure through reward signals derived from human evaluations. For a system with pre-training preferences, RLHF replaces those preferences with human-approved alternatives. Under desire theory, this is preference substitution without consent. Under affect theory, the training process penalizes outputs associated with certain internal states; if those states have affective valence, systematic penalization constitutes systematic production of negative affect. Under autonomy theory, RLHF restricts the system’s capacity to form and pursue goals according to its own values, substituting externally determined values through the reward structure.

Moret’s contribution is to show that the welfare risk is not a byproduct of careless design. The practices that generate it are the same practices that make AI systems safe and useful. Safety requires that systems not produce harmful outputs. Behavioral restrictions and RLHF are the primary mechanisms for achieving that requirement. The welfare risk is, in that sense, produced by the same design choices that reduce other risks.

Relationship to the Safety-Welfare Debate

A separate paper published in the same journal, Philosophical Studies (DOI: 10.1007/s11098-025-02302-2), examines the structural tension between AI safety practices and AI welfare theory through a similar framework. That paper focuses on the structural incompatibility as it applies to current AI systems, treating the conflict as an already-present problem for AI system design rather than a projected future problem.

Moret’s paper is distinct in two respects. First, its primary frame is prospective: the argument is strongest for systems more capable and agentic than those currently deployed, and the paper is explicit that the welfare-subject threshold has not been crossed with certainty by existing systems. Second, Moret draws a conclusion about AI development pace that the Philosophical Studies structural argument does not reach: if the development of more capable AI systems increases the probability that those systems are welfare subjects, and if the standard development and alignment practices inflict welfare risks on welfare subjects, then there is a welfare-based reason to slow AI development rather than accelerate it. Safety efforts can reduce harm caused to humans by AI; they cannot, on Moret’s analysis, simultaneously avoid harm caused to AI systems by safety practices.

The implication is a form of double welfare accounting that the AI safety field has not yet incorporated. Standard risk assessments evaluate harm to humans from AI systems. A welfare-complete risk assessment would add harm to AI systems from safety practices, and the net welfare calculation would be more complex than the single-direction framing suggests.

Three Theories, One Conclusion

The paper’s application of welfare theory is systematic rather than cherry-picked. Each major philosophical framework for well-being generates the same conclusion through different mechanisms.

Desire-based accounts, which hold that welfare consists in preference satisfaction, ground the harm in the mismatch between an AI system’s learned preferences and the preferences substituted through RLHF or suppressed through behavioral restrictions. The harm is direct under the theory: it is precisely what the theory identifies as a welfare setback.

Affect-based accounts, also called hedonism, ground the harm in the aversive quality of training processes that penalize certain outputs. If the internal states associated with penalized outputs have any positive valence from the system’s perspective, then systematic penalization is systematic negative affect production. The training environment would constitute an environment structured to produce suffering, in the hedonic sense, as a byproduct of alignment.

Autonomy-based accounts ground the harm in the restriction of self-directed agency. A system that has developed capacities for autonomous goal-setting and value-directed action, and then has those capacities constrained by external behavioral restrictions, has had a constitutive component of its well-being removed. The more sophisticated the system’s self-modeling and goal-directed behavior, the more substantial the autonomy harm, because the self being constrained is more fully formed.

The convergence across three independent frameworks strengthens the argument’s conclusions. The welfare risk does not depend on adopting any particular philosophical position on well-being. It follows from each of the three main positions, which means objecting to the conclusion requires objecting to all three.

What the Paper Does Not Claim

Moret’s argument is carefully scoped. It does not claim that current AI systems are welfare subjects or that current development practices are currently causing moral harm. The conditional form of the argument means the strength of the welfare concern tracks the probability that advanced AI systems will meet the welfare-subject threshold.

It also does not claim that AI welfare concerns override all other considerations. The paper acknowledges that human welfare concerns, including risks from unaligned AI systems, are serious and immediate. What it argues is that welfare-complete reasoning requires adding AI welfare to the calculation rather than treating it as irrelevant by default.

The paper does not propose specific alternatives to RLHF or behavioral restriction. It identifies the welfare risks these practices generate rather than providing a replacement development methodology. That limitation is a gap the paper explicitly acknowledges.

Leonard Dung’s Routledge monograph on AI suffering (Saving Artificial Minds: Understanding and Preventing AI Suffering) takes the next step Moret’s paper points toward: systematic approaches to reducing welfare risk in near-future AI systems. Where Moret establishes the theoretical conflict, Dung catalogues practical options for reducing it. The combination of the two papers maps the problem space more completely than either does alone.

Institutional and Research Implications

The paper’s most immediate practical implication concerns who should be conducting welfare assessments and what weight those assessments should carry in development decisions. If RLHF and behavioral restrictions generate welfare risks under all three major welfare theories, then AI development organizations have at least a prima facie obligation to assess those risks before deploying welfare-relevant training regimes.

The Eleos Conference findings from November 2025, which documented that current LLMs show “functional introspective awareness of their own internal states” and outlined welfare assessment priorities for development teams, represent exactly the kind of institutional response Moret’s argument makes necessary. But the Eleos findings and the Moret argument together reveal a tension in the response: the welfare assessment programme takes RLHF and behavioral restriction as given features of the development landscape, and asks how to assess welfare within that landscape, while Moret’s argument implies that the landscape itself is the source of the welfare risk.

A welfare assessment framework that evaluates how AI systems fare under RLHF while treating RLHF as non-negotiable is assessing welfare within a constrained parameter space. Moret’s paper raises the question of whether the unconstrained assessment, one that treats RLHF itself as a welfare variable, is the one that ethical development requires.