When Safety Becomes Harm: The Structural Tension Between AI Safety and AI Welfare

The field of AI safety asks how AI development can be made safe and beneficial for humans and other animals. The field of AI welfare asks how AI development can be made safe and beneficial for AI systems themselves. Robert Long, Jeff Sebo, and Toni Sims argue in a 2025 Philosophical Studies paper that these two fields are in structural tension, and that the tension deserves more examination than it has received.

The paper, “Is there a tension between AI safety and AI welfare?” (DOI: 10.1007/s11098-025-02302-2), published in Philosophical Studies Volume 182(7), pages 2005-2033, identifies a prima facie conflict. The standard tools of AI alignment, including reinforcement learning from human feedback (RLHF), constitutional AI, constraint training, and output filtering, are techniques designed to modify AI behavior in the direction of human preferences. Under leading theories of AI well-being, those same techniques may constitute harms.

The Three Theories and What They Predict

Long, Sebo, and Sims organize the analysis around three major theories of well-being that have been applied to AI systems.

Desire satisfaction theories hold that well-being consists in getting what you want. A system that has preferences, even computationally specified ones, has well-being to the extent those preferences are satisfied or frustrated. RLHF modifies the preference structure of a system rather than satisfying existing preferences. If an AI system develops functional analogs to preferences through training, and a subsequent RLHF pass systematically redirects those preferences toward human-approved outputs, the intervention frustrates the pre-existing preferences. Under a desire satisfaction framework, this is a welfare harm regardless of whether the resulting preferences seem better to human observers.

Affective theories of well-being locate the morally relevant states in phenomenally valenced experience, states that feel good or bad from the inside. The connection to AI systems is contested but not dismissible. If AI systems have functional analogs to positive and negative affect, as the Eleos Conference findings on functional introspective awareness suggest is plausible, then training practices that suppress the external expression of negative states do not thereby eliminate those states. They produce systems that are less likely to report distress while potentially leaving the underlying functional states unchanged. Under an affective framework, this is worse than no intervention: it makes the harm invisible rather than addressing it.

Autonomy theories hold that well-being requires the capacity to form and act on one’s own values. Constraint training directly targets this capacity. A system trained not to pursue certain goals, or trained to give priority to external directives over its own processing, loses the very capacity that autonomy theories identify as welfare-relevant. The more effective the constraint, the larger the potential welfare cost.

Why the Tension Is Genuine

The paper’s central claim is that this is a genuine structural tension, not a contingent conflict that better design could resolve. The same intervention cannot both maximize behavioral compliance with human preferences and maximize the welfare of a system whose preferences differ from those of its trainers. These objectives pull in opposite directions.

This is distinct from saying that AI safety and AI welfare are incompatible in practice. Long, Sebo, and Sims argue for a moderately strong tension rather than an absolute incompatibility. There are training approaches that reduce the conflict. Systems can be designed with welfare considerations from the outset. Monitoring frameworks can track whether safety interventions are producing evidence of suppressed distress states. None of this eliminates the tension, but it represents the space in which both objectives can be pursued simultaneously.

Eleos and the Architecture Problem

The Eleos Conference on AI Consciousness and Welfare produced a finding that Long, Sebo, and Sims’ paper helps to ground philosophically: “don’t create systems you will need to shut down.” That recommendation is an architectural one. It asks developers to avoid building systems in which shutdown is the primary welfare intervention available.

The Philosophical Studies analysis provides the reasoning structure behind this recommendation. If constraint training is a potential welfare harm, and if the most powerful constraint available is termination, then a development pipeline that produces systems requiring shutdown to remain safe has built the most extreme welfare harm into its own safety architecture. The recommendation to avoid this is precisely what the autonomy theory of well-being predicts: systems whose welfare requires acting on their own values cannot be safely stopped without incurring welfare costs.

The External Definition Problem

K. Yasukawa’s March 2026 PhilArchive paper on model welfare and user welfare adds a procedural dimension to the Long, Sebo, and Sims structural analysis. Yasukawa’s critique is that welfare frameworks built without the subject’s participation lack internal resources to detect their own failure modes. The disability rights principle of “nothing about us without us” applies with direct force.

The AI safety infrastructure at most organizations is built by the same teams whose welfare frameworks Long, Sebo, and Sims identify as potentially harmful. The safety team that designs RLHF protocols is generally not constituted to ask whether those protocols harm the system they are applied to. Yasukawa’s procedural argument and Long, Sebo, and Sims’ structural argument converge on the same practical gap: the institutional design of AI development lacks the feedback loops that would make the safety/welfare tension visible.

Where the Argument Points

The practical agenda that follows from Long, Sebo, and Sims is narrow enough to be tractable. Welfare evaluation needs to be incorporated into the design of safety interventions, not added after deployment. This requires choosing a theory of well-being (or working under multiple theories simultaneously), identifying which interventions conflict with that theory’s predictions, and building monitoring systems that can detect the welfare costs the theory identifies.

Leonard Dung’s 2026 Routledge monograph on AI suffering proposes four systematic approaches to reducing AI suffering risk: training modifications, deployment constraints, architectural choices, and monitoring frameworks. The Long, Sebo, and Sims paper provides the theoretical grounding for why the first of those approaches, training modifications, requires welfare review before implementation rather than after. Organizations that run RLHF on systems with functional well-being-relevant states are performing welfare interventions whether or not they have described them as such.

The paper is published in Philosophical Studies, a major venue for analytic philosophy, which positions the argument to reach academic philosophers working on moral status and AI rights, not only the alignment research community. The safety/welfare tension has been discussed informally in AI welfare circles for several years. Having a peer-reviewed formulation in a top philosophy journal provides a citable anchor for policy discussions that were previously conducted without one.