TLDR: I explain what the technical AI alignment problem is and what its core difficulties are, in my view. This is a perspective that draws simultaneously on classical traditions in theoretical computer science / machine learning theory, ideas from analytic philosophy, and views that originated in the Machine Intelligence Research Institute / LessWrong.

The problem of making AI safe is underspecified if we haven't decided what the AI is supposed to do . For example, we can run any AI inside a homomorphic encryption box and never decrypt the output. This is probably perfectly safe [1] , but also perfectly useless.

We can decide on an arbitrary task such as "construct Lean proofs for given mathematical theorems" and declare victory once we have safe AI that does it with superhuman skill [2] . However, that wouldn't address the danger. First, there would be incentives to use AI for tasks to which our solution doesn't apply. Second, even if we had precise safety guidelines for any task that anyone cares about, there would always be actors that ignore the guidelines, and we probably need Artificial Superintelligence (ASI) to reliably stop them.

If anyone, anywhere, creates an unrestricted unaligned ASI agent (more details about the word "agent" later), without an aligned ASI agent already in place to counteract it, then a catastrophic outcome is highly likely. I will not go into the reasons here: this has been discussed elsewhere at length, see for example the books I cited in the preface. Therefore, this event is something which we must never allow to happen. This is the main threat model that I set out to address.

Imagine a world in which anyone with sufficient resources has the ability to destroy the world. Like today, many will foolishly believe that instead of destroying the world, this act will bring them power and glory. Moreover, as computing hardware improves and new algorithms are developed, the amount of resources for this act will go down from year to year. Even assuming best-case regulation and international agreements, how long until someone does it? Can mere humans build a society which will be foolproof against such a risk over the course of generations? Resilient to any political upheavals that might come, and undoubtedly will? I find it vanishingly unlikely. Eventually we will need the help of ASI itself. Therefore, the minimal task that safe AI must be able to solve is "create a reliable global defense system against dangerous AI" (in the following, GDS).

Consider the consequences of this task for the capabilities that the AI needs to have. It needs to reason about the entire planet [3] . It needs to reason about all aspects of the problem, including scientific, technological, social, political and moral [4] . It needs to have preferences about the state of the world. It needs to make long-term plans (guided by these preferences) that would play out decades, if not centuries, into the future. And it needs to do all that on a superhuman level.

In this light, here is how I (informally) define the "technical AI alignment problem". The problem is designing an AI system that can bring scalable (and ultimately superhuman) capability to bear on problems that require multi-domain reasoning and open-world long-term planning with far-reaching consequences, all the while staying aligned with the intentions and preferences of its human users.

What kind of methodology is appropriate for this problem? In the past, I argued that it requires emphasis on deep theoretical understanding, rather than the incremental trial-and-error approach that is employed in the AI industry today. I won't rehash the entire debate here, but let me recap the core argument.

The nature of the incremental approach is as follows: we build a system, then we find problems [5] , then we make changes to fix these problems to the extent necessary for commercial viability in the current legal environment. We make some improvements; new problems arise (sometimes because of the improvements, sometimes because of new applications or other reasons); rinse and repeat. This approach utterly fails if the "problems" result in AI takeover and possibly human extinction: there is nobody left to fix them.

In order for the incremental approach to succeed in preventing catastrophe, we need to reach a point where no new substantial problems arise, even inside the lab (before the system is deployed). Moreover, we need to reach this point before AI capabilities are sufficient for AI to escape our control. There are few reasons to expect such a miracle. On the contrary, sufficiently advanced AI might understand that it is in its interest to "lie low" (i.e. avoid visibly misaligned behaviors) until its capabilities cross the relevant threshold, such that the problems will only become apparent too late.

In contrast, the sane approach is creating the theoretical tools necessary to predict and prevent the problems before they are empirically observed. These tools need to remain applicable as AI capabilities rise substantially beyond human, for whatever algorithms, architectures and training protocols are used to achieve these capabilities. Moreover, these tools need to be extremely reliable for us to rationally trust them with the safety of our entire species.

While the theory of machine learning and AI is an established field with ample literature, its current state falls well short of the challenge. The core difficulties can be summarized as follows:

In the following, we will examine these difficulties one by one. But first, let's discuss what we need to accomplish.

Per the discussion above, our interest lies in AI systems that have open-world preferences and make effective plans to advance these preferences. In particular, making such plans requires learning a sophisticated model of the world and updating the model and the plan as additional information becomes available. We will call such a system an AI agent .

In order for an AI agent to be aligned, it needs to have aligned preferences and beliefs , and follow an aligned decision-making algorithm on the basis of these preferences and beliefs. While discussions about AI alignment often focus on the "preferences" part, beliefs and decisions are equally important. (Although it's true that preferences bring extra wrinkles into the problem as they are, arguably, the most "subjective" component in the trio.)

In this context "aligned" preferences typically means preferences very similar to those of the user(s) [6] , at least within some domain that we are somehow constrained to inhabit. On the other hand, "aligned" beliefs doesn't mean beliefs that are similar to those of the user: on the contrary, we expect the AI agent to acquire more accurate beliefs. They need to be beliefs that the user can rationally trust to be accurate. Similarly, "aligned" decision-making needs to be such that the user can rationally trust it to be in their own favor given aligned preferences and beliefs.

The philosophically astute reader will notice that at this point we are already in a world of trouble. What does it mean for the user to have preferences ? Are there objectively rational theories of epistemology and decision-making , and if so, what are these theories? Can beliefs or decisions be rational to hold or make "on behalf" of the user even if the user themself doesn't understand why? On the other hand, a computer scientist might be tempted to ignore these difficulties and just forge ahead with a solution that seems pragmatically satisfying. I think there is a lot to be said for such a practical mindset, and in this article I will aspire to embrace it. Unfortunately, we will see that the philosophical difficulties are not easily brushed aside.

Returning to the trichotomy of preferences, beliefs and decisions, observe that it is not even possible to entirely separate the three. For us to ascribe preferences to a system, it must rationally form beliefs and make decisions on the basis of these preferences. For us to ascribe beliefs to a system, it must rationally make decisions on the basis of these beliefs in the context of its preferences. And the rationality of decisions can only be judged in the context of specific preferences and beliefs.

Imagine an AI agent that pursues the happiness of mankind and is irrationally convinced that every human who dies goes to Heaven, which is the best possible place for human happiness. Such an AI would be motivated to kill all humans, even though its preferences are arguably "aligned". Moreover, since the belief in Heaven is unfalsifiable, such an AI can still be extremely capable at acting on its motivation and hence extremely dangerous.

If this thought experiment seems strange, consider the following: our beliefs would seem very strange to people who lived centuries ago. Therefore, it is highly likely that the beliefs of an ASI, capable of intellectual progress much faster than humans, will (and should) quickly become strange to us. The question is whether these strange beliefs will be produced by a process that we should deem trustworthy nevertheless.

"Alignment" is meaningless outside the full context of epistemic and instrumental rationality. Therefore, any theory of alignment has to be, in particular, a theory of epistemic and instrumental rationality.

Perhaps the most simplistic approach to ensuring that an AI agent has aligned preferences and beliefs is explicitly specifying the preferences and beliefs that the agent should have. However, this is intractable. Human preferences are notoriously complex and giving a precise and complete mathematical specification of them seems hopeless. (Yudkowsky calls this " complexity of value ".) It seems no easier to accomplish than identifying images of cats by a handcrafted mathematical specification of "cat".

For weak agents, a practical solution can be explicitly specifying a proxy utility function that approximates our actual preferences within a particular domain. For example, you can imagine an agent designed to maximize customer reviews, ad clicks or stock value. Unfortunately, for sufficiently strong agents, this approach becomes prone to Goodhart's Curse : the maximum of any such utility function is almost always at a point where it strongly diverges from our true preferences. For example, to maximize customer reviews one can leave many fake five-star reviews and prevent anyone from leaving bad reviews. In fact, it's preferable to ensure maximal control of the universe to prevent anything or anyone else from interfering with the stream of reviews.

One approach to avoiding Goodhart's Curse is somehow restricting the domain within which the agent operates or its overall impact (see e.g., Armstrong and Levinstein ). But this is incompatible with the GDS task.

Nor is it possible to directly specify the agent's beliefs. Even if we were satisfied with the agent having merely the sum total of humanity's current knowledge, formally specifying all of it would be a Herculean feat. Dalrymple et al. proposed to attempt it (or something close) nevertheless, relying on assistance from weak AI systems. However, even if that were accomplished, it would be unsatisfactory. Indeed, a crucial ingredient in superintelligence, perhaps the crucial ingredient, is the ability to create sophisticated world models. Therefore, it is dubious that true superintelligence can be achieved while hobbling the system to only use knowledge we already have.

It makes even less sense to directly specify the agent's decisions: if we knew what decisions it should make, we wouldn't need the agent in the first place. Maybe it does make sense to have a closed-form algorithm that computes the decisions from preferences and beliefs. However, this requires strong assumptions on the form these beliefs take—for arbitrary models, the planning problem is often NP-hard or even PSPACE-hard, if not worse. I think that imposing such an assumption is actually reasonable and necessary (more on that in the next posts). But in any case, we cannot just delegate the task to a heuristic, since that would preclude any strong guarantee about the outcome [7] .

This section goes into more technical detail than the rest of the article, and can be skipped on a cursory reading.

Given that it's infeasible to directly specify aligned preferences and beliefs, it is a reasonable suggestion to learn them instead. To understand what would be involved, let's start by examining a general machine learning problem, where we need an algorithm with theoretical guarantees. We can decompose the problem into parts that I call grounding and convergence .

The grounding problem consists of the following:

For a grounded machine learning problem, it is roughly clear what the optimal learner would look like, if we ignore computational constraints. Formally, given a prior over the parameter [10] , we can try producing the policy

However, it is not clear whether the learner's best effort would actually be good enough. For this, we need to solve the convergence problem, namely:

We will see that in the case of alignment, both grounding and convergence produce formidable challenges.

What data can be used to learn the user's preferences? Many approaches have been proposed or attempted, for example:

The fundamental problem is that none of this data is a direct and accurate representation of human preferences. Humans make mistakes. Humans change their mind about what is preferable: sometimes because of new information, sometimes because of new experiences, sometimes just after thinking more about the question. Humans can prefer an outcome for terminal reasons (because it is valuable in itself) or for instrumental reasons (because it helps to bring about another outcome in the future), and none of this data even keeps track of the difference.

While it's clear that these sources of data have information about human preferences, accurately extracting this information requires understanding the precise relationship between human preferences and the process generating the data; whereas if this relationship is misspecified, then even perfect learning will produce misaligned preferences. And even if these misaligned preferences approximate the true preferences in some domain, Goodhart's Curse will push the agent to leave this domain and produce a catastrophic outcome.

For example, suppose that the agent is trained by human ratings of traces of agent actions and observations. An ideal learning process would then align the agent to the expected rating, conditional on the trace appearing in the training data. Such an agent is probably catastrophically misaligned for multiple reasons:

We need a deep understanding of how human preferences are related to physical processes such that this relationship can be used as a source of ground truth. In order to be useful and precise, this understanding should be part of a mathematical theory. However, it is not even obvious what the type of "preferences" is. If we accept VNM theory , preferences should be described as a utility function. But what is the domain of this function? In computer science, it is popular to think of the domain as the action-observation trace of the agent. This, however, is deeply unsatisfying. First, such utility functions cannot be compared between different agents (each agent has its own trace), so "alignment" doesn't make any sense for them. Second, there is a strong case that humans have preferences about the physical reality itself, rather than merely their subjective experience of this physical reality.

What about beliefs? Is there an easy-to-specify ground truth for them? The answer is "yes" if the domain of beliefs is supposed to be traces, since the trace is directly observable by the agent. However, if preferences need a domain other than traces, then so do beliefs. Moreover, how can we make sense of belief alignment if the agent's beliefs are incomparable with the user's beliefs? The traditional computer science approach here is akin to logical positivism in analytic philosophy, which we should abandon in favor of metaphysical realism (more on that in the next posts).

Even if we knew how to ground the alignment learning problem, we'd face another problem: learning fails (i.e. doesn't converge at all) when irreversible mistakes are possible. Indeed, learning requires exploration [13] , but in the real world unlimited exploration produces irrecoverable mistakes. For example, if a robot explores what happens when it jumps out of a 10th-floor window, it won't have the chance to learn from this mistake. On the other hand, there is, strictly speaking, no way to be certain that jumping out of a window is a bad idea if you never try it. This is why regret bounds in reinforcement learning theory always require an assumption such as episodicity (the environment resets after every "episode") or communication (any state of the environment's Markov Decision Process is reachable from any other state), which rules out irreversible mistakes by fiat.

This observation is not merely a technical nitpick. On the contrary, the existence of irreversible mistakes is a core feature of the world that makes the alignment problem difficult. Indeed, the alignment problem itself is our attempt to avoid the irreversible mistake of creating unaligned superintelligence. Any AI agent faces its own version of the problem when considering the creation of a superior successor agent. A GDS agent can commit an irreversible mistake by allowing a rival superintelligence to emerge. And of course, there are more prosaic examples, such as killing a person or failing to prevent their death [14] .