Many thanks to Abram Demski for materially and intellectually supporting this research. Thanks also to Brian Berns, Roman Malov, Gurkenglas, and Lorxus for valuable conversations.

Despite the central place of corrigibility to theoretical AI alignment, it seems like no one can really agree about what is . Much of this confusion is due to the fact that almost every definition of corrigibility proposed thus far is deficient in one way or another. In particular, the most prevalent existing accounts of corrigibility are over-epistemicized (modeling corrigibility as arising purely from uncertainty about the AI's goal, which gives rise to issues like the problem of fully updated deference) or evaluatively-thickened (making the AI's normative priorities instrumental to its corrigibility as a result of forcing corrigibility to be a single unified goal.) Here, my intention is to clear up some of these confusions and present a new picture of corrigibility that builds on these existing accounts to specify a goal that is more congruous with the original 2015 desiderata , and admits the kind of formal safety guarantees we need the concept to entail.

I argue that the fundamental concept underlying corrigibility is respect for authority , using the analogy of a healthcare proxy's relationship to a patient as an intuitive guide. Defining the authority of humanity by designating its reserved powers , we can specify a goal that first and foremost restrains the agent from overstepping or interfering with this reserved human authority. Subject to this restraint, the agent then pursues the second component of its goal: acting on the principal’s behalf as the principal would judge best.

This approach stresses the deeply normative character of corrigibility, while remaining sufficiently concrete so as to be specifiable formally. In this article, I lay out the central problem of corrigibility, elucidate the issues with existing approaches, and clarify the conceptual ingredients of my new framework. I then present an initial formalism , proving theorem-level guarantees about agents pursuing the goal in three increasingly realistic formal settings. Finally, I address some limitations of the picture as presented and outline further research directions that this work points toward.

Suppose you are designing a powerful AI and you need to choose which goals the AI should pursue, at whatever level of specification you can provide. Doing this well demands answers to a number of normative problems: what constitutes a right decision, what factors should be considered in the process of valuing the world, how the AI ought to act, etc. In some sense the ideal AI directive would be to figure out and then pursue the Good, a universal-best-interests type of goal that can only be acquired through normative learning . The challenge of describing such a learner is one formulation of what I'd call the central problem of outer alignment : what would be the overall best goal for an AI to pursue?

The complete version of this problem is very difficult, and it runs up against our own deepest questions about human ethics, values, and interests. Discussing this problem formally is especially difficult: the rich truth of human normativity seems to resist systematic formalization at the level of individual recommendations for action. [1] A formalism requires an operationalization, a way to concretely specify the concept of right action at least to some degree.

For any hope at concreteness, we need a clearer formulation of the overall goal. A first attempt might be something like "the promotion of overall human flourishing", centering a thick normative concept about humanity that must be defined further. In trying to pin down the meaning of human flourishing, or any other thick human concept we want to use for the goal, the most important ingredient in the AI's consideration should naturally be the interests, feelings, and inner lives of human beings. But we should be careful not to crudely use human opinion as a proxy for human best interests. Humanity cannot always accurately understand its own full best interests, and humans who cannot speak for themselves must still have their interests included in the goal.

The complete normative goal cannot be constructed by treating existing human feedback as the sole source of truth. This issue is a central limitation of modern Reinforcement Learning from Human Feedback, which takes static human approval or disapproval as constitutive of correct evaluation for any question without a verifiable answer. For example, a judicial AI built to decide legal cases should not exhibit any form of the hungry judge effect , where leniency depends on mealtimes. An RLHF judge will pick up on all the patterns of human imperfection, whereas an ideal AI would be one that can transcend human limitations, without discarding the fundamentally human essence of the legal system. An ideal normative AI is one fully immersed in our existing normative practices of judgment, able to deeply understand the relevant human concepts and use its intelligence to extend them to novel cases.

For the complete normative learning problem, a better formulation of the target would therefore involve some idealized version of the judgment of humanity. [2] This is also already how we, as humans, reason about our own overall best interests. For example, consider a smoker whose nicotine dependence makes him crave cigarettes. At a base level, he has an interest in smoking now to avoid some physical discomfort and obtain some relief. But it's clear that the health costs he incurs as a consequence give a good reason to say that smoking is in fact not in his overall interest, assuming he values his own life and health. Neither of these reasons is dispositive; in a single case it might be that he has a reason to smoke because this particular cigarette is necessary to get him through the day or because he is old with a short remaining lifespan regardless, but he might also have a reason not to smoke because he publicly swore off cigarettes once and for all. Understanding whether it's truly better to smoke or not requires aggregating and squaring all the smoker's countervailing normative interests bearing on the subject. Any conception of value based on something like preference satisfaction involves a similar question: how should we reconcile the smoker's preference to smoke with his preference to not be dependent on nicotine anymore, along with all his other relevant preferences? A serious account of "best interests" requires this sort of reflective idealization through an accounting of reasons.

The moral notion of human best interests can be consequentialized , i.e., conceived of as a betterness ordering on worlds. If that ordering is total, transitive and suitably continuous, it can be represented by a utility function that takes in all the information about a world and its history as input so that a world is better exactly when the function's output is greater. We can then think of an AI as a bounded reasoner, evaluating normative questions by estimating the outputs of this function for the possible consequences of its actions. A bounded normative learner faces three kinds of uncertainty in performing this estimation:

These correspond to the three sources of feedback the reasoner encounters:

These three kinds of feedback are importantly asymmetrical. Once a theorem is proved, no logical uncertainty remains. Deductions settle a logical question certainly and permanently. For empirical questions, observations can similarly settle a claim with certainty, but only if we're careful about how the claims are formulated. For example, the sentence "The temperature sensor read 80°F at 8:00 AM today" can be settled by observation at 8:00 AM as either certainly and permanently true or certainly and permanently false. However, the most useful empirical claims are inductive generalizations (such as "the temperature is always above 70°F at 8:00 AM every day") which can never be conclusively proven via observation and must be estimated through induction and inference to the best explanation. An empirical generalization like this can be conclusively disconfirmed by a contradictory observation, but some empirical uncertainty always remains for live hypotheses.

Normative feedback is very different from either logical or empirical feedback. A first-class ingredient in normative reasoning is the full picture of human preferences and evaluations of hypothetical decisions, along with all empirical observations and deductions. Unlike the other two kinds of feedback, normative feedback never certainly and permanently settles a matter. Different forms of human feedback can be given different weight, but no piece of human feedback ever grants access to the Objective Right Answer for a normative question. Normative feedback is permanently revisable and never fully reduces normative uncertainty to 0. For a deeper discussion of these issues, see Learning Normativity .

It would be risky and naive to take on the challenge of the complete normative learning problem directly. If we try to design an agent to maximize its estimate of idealized goodness, we must directly grapple with the issues of uncertainty, particularly irreducible normative uncertainty, discussed above. Instructing an AI to maximize its estimate of goodness invites a dangerous kind of paternalism where the AI overconfidently acts on wrongheaded ideas about what's best for humanity. The AI will have to make decisions despite normative uncertainty, potentially forcing action based on weak evidence and underdeveloped normative concepts. In attempting to overzealously promote humanity's best interests, an uncertain and imperfect AI may disastrously thwart them by accident.

Furthermore, we shouldn't expect general formal guarantees about the good behavior of a proposed solution to the complete normative learning problem. Because we never get direct access to ultimate normative truth, we can never conclusively prove that the AI will come to the Truly Right Answers on normative questions. Weaker properties about complete normative learning could be formalized, but that will require a serious advance in formal ethics.

Instead, we can address both the issues of overconfident paternalism and the lack of safety guarantees by modifying our AI's intended goal. We can defer the difficulty of finding a satisfying answer to the complete normative learning problem by incorporating some additional target apart from just the overall Good. In particular, we want to specify a goal with the property that an AI designed to pursue it admits structural guarantees about the safety of its behavior, even under uncertainty .

The established name in AI safety for a goal like this is corrigibility . Despite pervasive disagreement about the actual definition of corrigibility, essentially all can agree that a corrigible AI is one that is open to human correction, allowing its errors to be rectified by human judgment rather than clinging to mistakes. This is precisely the sort of goal we were just reaching for; in the face of possible error about what's best, we can avoid an overconfidently paternalistic AI by instilling in it a corrigible attitude of caution, humility, and deference to human judgment, especially for high-stakes decisions. As leading theorist of corrigibility Max Harms puts it : "Corrigibility is the simple, underlying generator behind obedience, conservatism, willingness to be shut down and modified, transparency, and low-impact." Corrigibility is exactly the kind of goal that can mitigate the dangers of normative uncertainty. But if we are going to use corrigibility as a solution to our other problem, the lack of formal guarantees, we'll have to be much clearer about what this goal is supposed to precisely entail.

The following example should build some intuition about the essence of corrigibility, and why it provides safety without being mere servility . Suppose you are hiking on a snowy mountain, along with an AI-powered robot companion. Suppose further that the conditions for an avalanche are brewing, and proceeding the way you're headed has grown dangerous. Even if you never explicitly gave it any avalanche-related instructions, a corrigible companion should warn you about this danger. However, it would be incorrigible for the robot to go off and start melting snow in an act of avalanche prevention, at least without asking you first. A corrigible AI aids its principal without taking away the principal's control, and is not limited to the literal fulfillment of explicit instructions.

Corrigibility is not a prominent concept in existing human moral reasoning, so it would be helpful to put our desired properties into more concrete, human terms. Existing analogies meant to elucidate corrigibility using human relationships, such as an AI in the role of a household servant or personal assistant [3] , strain as we consider capabilities increasing. Instead, the human example that I believe most closely gets to the heart of corrigibility is that of a person's medical power of attorney .

Suppose a dear friend of yours has been in an accident that leaves her temporarily incapacitated. When you get to the hospital to see your friend, you're informed that she had previously signed a form granting you her power of attorney in such situations. You are now your friend's healthcare proxy , with both the authority and the responsibility to act on her behalf to represent her interests when presented by doctors with decisions about her treatment. When a tough choice about her care arises, one with serious tradeoffs and risks, how should you decide? How does a good healthcare proxy behave?

Suggestively, legal parlance refers to the person granting the power of attorney as the principal and the one receiving it as the agent. I contend that understanding what makes for good human behavior in a principal-agent relationship [4] like that of power of attorney is deeply related to understanding what makes for the kind of corrigible behavior we want to instill in AIs. In other situations these are called fiduciary duties , responsibilities of one party to act in the best interests of another. Note that for fiduciary relationships in particular, the agent is frequently more knowledgeable and capable than the principal, perhaps reversing our intuition from a "personal assistant" type of example. A fiduciary agent should act as an extension of the interests and judgment of the principal even past the principal's own understanding of the facts.

For the healthcare proxy case in particular, it's clear that the agent should use something like "what would the principal choose for herself, if she were capable?" as the guiding question. If the principal expressed explicit wishes about the exact situation beforehand, then those wishes must be respected. But if none exist, the agent should use their best model of the principal's judgment. What's crucial is that a healthcare proxy must not substitute their own preferences in for the patient's, out of respect for her autonomy. A good agent values the principal's authority . To whatever extent they can, the agent should act as an executor of decisions that the principal authorized directly. For other questions, the principal has delegated the authority to them via the power of attorney.

Suppose the patient regains consciousness, recovering the capacity she had lost. A good healthcare proxy would make no effort to prevent her from reclaiming her own power to make decisions for herself. An agent that respects the principal's authority is not perturbed when a previous act of delegation is revoked. Suppose that for a particularly important decision, you have already deeply weighed the pros and cons when your friend wakes up and chooses the option you rejected. Even if you think she's making the wrong choice for herself, a true friend would willingly relinquish the power of attorney and only try to honestly convince her about the error in judgment.

The concept of a power of attorney extends beyond cases of incapacitation. A person may grant a financial power of attorney for someone to manage their money or assets even while they are perfectly competent. The scope of such an arrangement can be set explicitly, and particular duties may be delegated or reclaimed at will. The agent should never contravene the authority of the principal by overstepping clearly specified boundaries of delegation. Even if the agent is quite sure that the principal will approve some decision, the corrigible thing to do is always to ask first if the scope of the decision is beyond what's been explicitly delegated.

Let's sum up the lessons of the healthcare proxy examples as a new conception of corrigibility. A corrigible agent should have the following two goals:

Goal 1 (Authority): Respect the principal's authority as a decision-maker. Don't usurp or interfere with the principal's power to make reserved decisions.

Goal 2 (Best Interests): Act on the principal's behalf as the principal would judge best (if the principal were as capable and informed as the agent). For delegated decisions, behave as an extension of the principal's interests.

A strictly corrigible agent pursues these goals in that order , never trading a known violation of Goal 1 for greater expectation of Goal 2. Note that this two-tier conception is in fact an echo of the original 2015 Corrigibility desiderata , where four criteria about safety (related to shutdown and successor agents) are placed above the maximizing fifth criterion ("Otherwise, a -agent should maximize [the normal utility function].") In this new two-tier framework, Goal 1 represents an inherent respect for authority that generalizes the original first four criteria into a unifying safety principle, and Goal 2 corresponds precisely to the original fifth criterion of pursuing the general utility function representing the principal's best interests. Satisfying the original desiderata, this picture only encourages pursuit of maximizing Goal 2 if the safety requirements prescribed by Goal 1 are all met.

The consequences of this two-tier version of the goal of corrigibility will become clearer if we first contrast it with existing framings of corrigibility. To start, the agent-principal relationship a healthcare proxy has with the patient must not be confused with the kind of relationship one has with an expert . An expert is someone who you expect to have better information or reasoning than you do about a subject, not someone whose authority you inherently respect. A corrigible agent does not primarily see the principal as a source of accurate knowledge about anything.

Of course, the principal does often have valuable information about the principal's own values and best interests that the agent would like to incorporate into decisions. Feedback from the principal is a crucial input to the agent's reasoning. However, the proxy shouldn't so desperately prioritize information-gathering to the extent of, for example, harmfully waking the patient up just to ask what she wants. Likewise, no matter how much information the agent gathers, corrigibility dictates that the final say is left in the hands of the principal for undelegated decisions.

To substitute an expert-novice relationship for the true principal-agent relationship is to over-epistemicize the problem. It is a mistake to reduce corrigibility to purely a property about uncertainty and quality of beliefs, neglecting its deeply normative aspect. Corrigibility is a property of an agent's specified goal, not strictly emergent behavior stemming from uncertainty about its goal in general.

This kind of over-epistemicization is a major source of confusion in early attempts to formalize corrigibility. For example, let's turn our attention to the problem of fully updated deference . Eliezer Yudkowsky writes:

One possible scheme in AI alignment is to give the AI a state of moral uncertainty implying that we know more than the AI does about its own utility function, as the AI's meta-utility function defines its ideal target . Then we could tell the AI, "You should let us shut you down because we know something about your ideal target that you don't, and we estimate that we can optimize your ideal target better without you."

On this purely epistemic account, the only reason that the AI is corrigible is because it believes that humanity knows more about the target than itself, and therefore defers to human opinion. The major problem with this is that we shouldn't expect it to actually be true in general that humans are experts about the Good. In particular, as capabilities increase, and the AI learns everything it needs to know about its goal by absorbing relevant human information, the relative edge of humanity in understanding the goal runs out . A sufficiently smart AI will eventually be better at estimating the goal than humanity, regardless of how human-centric the goal is. This is the problem of fully updated deference :

The obstacle to this scheme is that belief states of this type [i.e. deference based on moral uncertainty] also tend to imply that an even better option for the AI would be to learn its ideal target by observing us. Then, having 'fully updated', the AI would have no further reason to 'defer' to us, and could proceed to directly optimize its ideal target.

A sufficiently intelligent student will always "outgrow" any expert in this way given enough time, but a good agent never outgrows the principal, regardless of capabilities. The problem of fully updated deference applies only to agents whose corrigibility is fundamentally a consequence of their uncertainty. Yudkowsky recognizes how bad the problem of fully updated deference is for the initial proposal, but doesn't abandon the over-epistemicized picture that gives rise to it:

Can we make the AI want to be shut down for the same reasons we want to shut it down? Maybe because the AI knows that it's incomplete , and is able to conceive of its programmers having gotten its utility function 'wrong' in a way that the programmers but not the AI know how to correct?

A truly corrigible AI does not need to believe that it is defective, incomplete, or intellectually inferior to humans to allow its own shutdown, as Yudkowsky assumes in framing the " hard problem of corrigibility ." Instead, it can be corrigible as a result of intrinsically valuing human authority, and understanding that humans have authority over shutdown. The healthcare proxy should willingly give back control to a newly competent patient, even when the principal is in no better epistemic position to decide for herself than the agent is (or even a worse one.) The real reason we want the AI not to interfere in its own shutdown is because we think that whether an AI should be shut down is a decision of grave importance that humanity reserves for itself to make. An AI should accept shutdown for the same reason: respect for human authority.

A more subtle issue arises when examining how Yudkowsky formalizes moral uncertainty in the problem of fully updated deference. Firstly, he models this uncertainty using a meta-utility function : a probability distribution over candidates for a fixed, underlying utility function. This formalizes the over-epistemicized expert relationship, where the only role humanity can play in the AI's utility reasoning is an evidential one. As we saw, this leads to the problem of fully updated deference.

But a further issue is that these are utility functions on outcomes , a space that isn't specifically defined but ends up being too coarse as used. In the running example, the agent's meta-utility function is spread over candidate utility functions , and . In each world , the humans have their own utility function that they would optimize if the AI is shut down. The AI is deciding whether to shut down ( ) and let the humans optimize, or resist shutdown to keep optimizing its own updated mixture, given that it has observed evidence ruling out :