Dialogue with Gabriel Alfour on Emergent Wisdom, Coordination, Growing Up, and Democratic Republicanism

Transcript generated with Gemini-4-Argon and spot checked in places but not fully verified by a human. Ivan Vendrov: All right, welcome to the show, Gabriel. Gabriel Alfour: Thanks for having me here. Ivan Vendrov: So, we’re going to talk. This is a follow-up conversation to the chat I had with Davidad a few days ago on emergent wisdom in language models. And suffice to say, Gabriel doesn’t buy it. Do you want to articulate your main disagreement? I can also summarize if you’d like. Gabriel Alfour: For sure. First, I want to restate what is meant by... I forgot the word that you used, wisdom? Ivan Vendrov: Emergent wisdom. Gabriel Alfour: Emergent wisdom. I like the presentation that you had, I think, in your last conversation where it was like, “Oh, it looks like there’s a latent for hacking, bad stuff, and so on and so forth, and emergent wisdom is the opposite of this.” I think this is a fair characterization, and I do think that there is such a latent. I don’t think this is where my disagreement lies. Ultimately, we all see the same reality. My disagreement is more about the interpretation. For instance, I would not call this wisdom. That’s a big one. And the second one, which I think is more important, is that I think this is just largely unrelated to actual alignment. The road to hell is paved with good intents or good intentions, I’m not sure what’s the English version of the saying, that would be basically my main takeaway there. Ivan Vendrov: Cool. So I guess maybe the first place I want to double-click on is the first comment you made to me privately is you’re on the same page as Elena. Elena is the bodyworker I mentioned in the podcast with Davidad, who her response to talking to chatbots is disgust. She’s like, “Whoa, this thing is pretending to be a human being, but it clearly isn’t. And unlike a human being, I have no somatic, emotional, physical affordances to work with it.” So it’s this object that’s almost optimized to fool me in some way, and I want it away from me. So it seems like you resonate with that. Gabriel Alfour: Another thing that I resonate with in the same vein is social media curation, where it’s blatantly optimized, not necessarily against me, but against a lot of people to get some specific reactions from them. And when it tries to get them from me, I have the same level of disgust. When social media tries to get me angry at something, I’m like, “Bruh.” So for me, it’s the same thing. Of course, there’s the same dynamic where a bunch of people will engage with social media as if it’s an actual reflection of the world, and as if their disgust reaction, their anger reaction was just a natural reaction to the world as opposed to something that was optimized for. Ivan Vendrov: How would you distinguish, because I also have this a lot, and I actually haven’t been chatting to chatbots very much at all, or using curated news feeds in my social media, I usually just go to profiles of people that I personally know and care about reading the thoughts of. But I have in the past derived a ton of value, both from talking to chatbots and just browsing streams of social media content. I’ve made a lot of my friendships that way, learned a lot of crazy ideas that way. And part of me is like, well, are we... Here’s a few people, including Davidad and Janus, who seem to be deriving a lot of value from talking to these language models and seem to be having insights about them. What’s actually the difference you have with them? Is it are they being naive and they don’t have the right cognitive security, or are they maybe they’re just more advanced than you and me, and they’re able to integrate the information from these experiences without being co-opted by them? How would we know? Another way to put this is the AI psychosis question: how do we decide whether Davidad and Janus are suffering from AI psychosis or if they’re seeing something real? Gabriel Alfour: It’s funny because I hear separate questions rather than an either/or. There’s the question of whether you derive value from it, whether there’s AI psychosis, whether there’s something deeper that they get that we don’t get, and so on and so forth, and for me this seems like very decorrelated questions. So I’m not sure how you want me to take it, but— Ivan Vendrov: Jump on whatever feels interesting. Gabriel Alfour: I’ll try to just do a quick BFS, let’s say. For the question of deriving value from it, I’m like, yes, there is a lot of value to be derived even from social media. There are many ways to broach the question. The first one is that there is immediate value that is not necessarily reflected upon or endorsed, dopamine hit type of value. It is a type of value. In modern life, you don’t get to hunt a lot, so the dopamine hit does provide some value. A lot of people are like, “Oh no, there is no value in that.” No, there’s value in excitement, there’s value in finding something fun and so on and so forth. It’s just it’s not necessarily reflected upon that you want that much of this in this format with zero referent. That it’s a finely calibrated treasure hunt that is meant not to be fun for you, but specifically to keep you on the platform. So that’s one type of finding value. And I think LLMs have a lot of this. LLMs have a lot of, “Oh, it’s fun to talk to it.” Beyond the actual semantic content, they just echo a lot of your ideas. They just engage with your frame. You can give an LLM any frame and it will try to engage with it. And so there is value in it. It’s worthwhile to have someone explore your ideas within your frame, but do you endorse having so much of that in that way, optimized for these reasons through those processes? So I think this is another one that is quite important, especially given that for people who are dopamine-starved, social media gives a dopamine hit. For people that are starved of having others engage with their frame, you can autocomplete. So I think that’s one that is very important. The other one is the question of whether there is something there that we don’t see. My answer to this is like, of course, but that’s true of literally everything. There’s a skill that I think is very important, which I call staring at a white wall. You should be able to stare at a white wall and find interesting things about it. There’s no wall that is perfect RGB white uniform. There are no screens that are perfect RGB white uniform. You can look at a white screen and it will still not be that perfect. A wall comes with scratches, textures, lights and shadows, different materials, specular reflections, and so on and so forth, and I would say that it’s a virtue to be able to look at such a wall and actually analyze it and find it interesting, and start to see what is different between the different white walls and so on and so forth. I think this is a very important skill to have, both because it’s practical and convenient—very often people don’t work on the important stuff because they can’t find interest in them, they’re like, “Oh, it’s intrinsically boring and it needs something exciting, blah, blah, blah”—but also because I think it’s important to have some reflection on this. When it’s a skill, it stops being a feature of reality and becomes a choice. Whether there is something in that, it stops being an intrinsic truth of reality and starts feeling like internally, do I decide to find extraneous value there? And so I think this one is largely how I see the question. And so I’m like, yes, they decide to find value there. Some people will tell you that the truth of life can be found hammering at a smithy or something like this, and they will be right too. There’s this one. And the last one, which is how do you know whether it’s AI psychosis or not? I think there are many ways, but it becomes very personal, so I would rather not. If you think about psychosis and about psychiatric things in general, it’s usually defined as, does it prevent you from living a normal life? And I think this is an intrinsically personal judgment, and for a few people that is evidently the case. But then do people endorse living a normal life and so on and so forth? Ivan Vendrov: What exactly is normal? I mean, I certainly have found in my personal life I’ve been gravitating much more to people living normal, wholesome lives where they spend time with their family, have children, don’t do too many drugs. I don’t know. Gabriel Alfour: Exactly. Actually, that’s what I wanted to talk about. I think those are genuinely great heuristics. There’s a friend who told me that in Judaism, in the Kabbalah, because there was so much Kabbalah psychosis happening, they restricted the study of Kabbalah to men over 40 that were married and with children. Ivan Vendrov: Whoa. That’s pretty good. I think notably the Roman Republic restricted office holding to some similar demographic. Maybe power is similarly psychosis-generating. Gabriel Alfour: Oh, that’s one way to look at it. I had a more cynical interpretation of that one, but yes. Ivan Vendrov: What was your interpretation? Oh, that they’re kind of useless surplus males? Gabriel Alfour: No, that men want to keep the power between men, 40-year-old plus want to keep the power between 40-year-old plus, and so on and so forth. Ivan Vendrov: Yeah, there’s definitely, especially the men part is a little bit suspicious. Gabriel Alfour: Just saying. Ivan Vendrov: Well, I guess we can come back to emergent wisdom, but then I would say in a way your disagreement is on a higher level, which is you’re being like, “We just shouldn’t be orienting to this source of information at all. There are more nutritious sources of information about how to make the future go well than talking to language models and talking to people who talk to language models all day.” So where do you find your curiosity going to, I guess, these days? Gabriel Alfour: That’s a very fun one. I’ll just give my personal answer, which I think is easier. So I will give the object-level answer and then why that is my answer, because I don’t think it will be legible. Fundamentally, my curiosity goes toward running experiments that take the shape of studying online communities, testing different norms, and seeing which ones are more leading to action, useful thoughts, and positive change. There are many reasons for that. The first one is just empiricism. I’m fundamentally suspicious of thinking a long time in a black room, whether there’s an LLM in it or not, whether it’s an ivory tower or not. It can take the shape of academia or talking for a thousand hours to an LLM; I’m just intrinsically suspicious towards talking a lot. Ivan Vendrov: Wait, but you’re studying communities that talk a lot. What’s the fundamental difference you’re pointing to between talking to an LLM and talking to a bunch of people on the internet? Gabriel Alfour: Because the point is not about having conversations with these communities. The point is about studying communities and seeing which norms will lead to good things and which norms will lead to bad things, which cultures will lead to good things, which cultures will lead to bad things. Ivan Vendrov: Where good things are evaluated not in terms of the quality of the conversation, but in terms of some action in the world? Gabriel Alfour: Yes. You’re right, it doesn’t go without saying. Yes, I’m quite suspicious of a community that defines their goodness in terms of the good that they brought to the member to the extent that they participate in the community. A community that defines goodness in terms of how many good conversations happen within the community makes me feel, “Hmm.” Ivan Vendrov: Sounds like you’d also be suspicious of Buddhism, which defines the good as things that happen inside the mind. Gabriel Alfour: So I have a friend who’s very much into Buddhism, and this is one of our earliest conversations, where it’s like there’s a big tenet in Buddhism that you can find in other traditions—Freemasonry, for instance, has the same one—which is that through inner work you will work on the world. Freemasonry talks about—I don’t know how you call this in English— le rayonnement , shining light, casting light. Buddhism talks about building karma. Both of them have a nice intuition, which is that if you try to optimize directly for impact in the world, fundamentally you’ll be grabbing power instead of building nice cultures. And so instead they’re like, “No, work on yourself, do some virtue ethics, and then you’ll naturally bit by bit spread a nice culture.” I think these are fine ideologies for centuries or millennia ago. I think nowadays we actually have a much richer understanding of all of that. For instance, Goodhart’s Law is something that is a normal part of our vocabulary nowadays that was not the case at the times of the early Freemasons or early Buddhists. And so I think there’s a work of integrating all that we learned about complexity theory, social dynamics, public choice theory, psychology, mechanistic models of intelligence, cybernetics, and so on and so forth, into something that tells you, “Yeah, that’s not a good idea.” Obviously optimizing for power is very bad. Anthropic, obviously. If you have a bunch of these things, it’s bad, and I think that it should be obvious. And given that it’s obvious, you should work on the next step. And so from my point of view, the next step is: okay, I work on myself, and I want to optimize the world. How do I do so in a sustainable way, in a positive-sum way, in ways that don’t have unexpected known unknowns or unknown unknowns, negative outcomes, and so on and so forth? And I think we can do this in 2026. Ivan Vendrov: I see. So it’s like humanity has actually graduated from... Like if you lived a thousand years ago, maybe 3,000 years ago, you’d be like, “Well, actually the best good you can do is internal.” But now, because we have better understanding, we can actually start doing social good. Gabriel Alfour: Also, I would just say, obviously largely internal, but both in the case of Buddhism and Freemasonry, there is an external component. Both put very strong emphasis on building nice communities with nice norms, whether it is the monasteries or the lodges. In both you have the same thing. You have the idea that you have to shine, to go out, build karma and spread it in the world, and so on and so forth. So I wouldn’t say it’s only internal work, it’s just the main locus of attention, but not the only one. Ivan Vendrov: I have a bunch more questions on the meta-level, but I also just want to maybe start by deep diving into what are the experiments you’ve tried and what have you learned? Gabriel Alfour: I don’t think it’s very legible. I don’t think there’s enough of a common frame for my learnings to be very useful to you, but why not. Ivan Vendrov: Let’s try. Gabriel Alfour: Just let me know if it’s not interesting and I will not feel hurt in my ego. So, there are a few things. When I was young, I loved online games, Ivan Vendrov: Same. Gabriel Alfour: But not for the gaming aspect. I loved the community-building aspect. So I would try many, many different online games and look into which ones the gameplay mechanics let you create a team, an alliance, a guild, a tribe, a clan, however it’s called, and find the games where you can do this basically at level five, such that I just speedrun to level five and create one. And for me, that was the game. Creating the forum, the chat, giving people roles, coordinating all of that, it was epic fun. It was very meaningful—not in the endorsed way, just in it feels like meaning, you’re working with a lot of people towards a common goal. It’s like the dopamine hit of this, rat pushes button. I’ve done this, I think, dozens of times total, and got pretty good at that. So this has been a formative experience. In adulthood, obviously, I wanted to do things that were useful. I tried the same approach and it failed. And so I was like, “Oh, why?” It was so easy with games. In a week or two, I get 50 to 100 people and they work in a coordinated way with specific goals, KPIs, and they put hours every day into this. And now I try to do it on things that matter, and complete abject failure. I wish more people had this formative experience, I would say. Ivan Vendrov: I’ll just share parenthetically that I spent much of my teenage years playing this weird online game called Pardus , which is an EVE Online clone, and I also was trying to be a leader, coordinating raids of 50 people, writing a history of the place, negotiating treaties, alliances, et cetera. And then I remember the moment I was 15, and it was my last year of high school, and I was like, “Well, I should actually probably focus on school so I can get into a good university and stuff.” And I remember there was this moment of, “I think I achieved everything I wanted in life in this game.” Be a historian, be like Napoleon, be a leader, be whatever, achieve notoriety, write my name in the pages of history. I think I achieved everything I needed inside the game, do I really want to leave? And I’m like, “Well, I can always come back if the real world sucks.” And that kept me going for many years. So I think we had a similar sharp transition. Gabriel Alfour: I think for me it was more rat push button, I think yours was more long-term, whereas for me it’s like we get to 50 to 100 people and now it becomes work and I move to the next one. Ivan Vendrov: Fair. Gabriel Alfour: Teenager. Ivan Vendrov: Makes sense. Gabriel Alfour: Short time horizons. So there was this, and so that’s one half to understand how I think about things. Failures of Coordination and Pre-Registering Good Gabriel Alfour: The other one is that basically, I think there are two clusters of coordination: System 1 coordination, System 2 coordination, and they both fail. So there’s the tribal coordination. I forgot which one is System 1 and System 2. Let’s say the built-in. Built-in coordination, it’s like you find your group and yay group, boo outgroup. There are leaders in the group, whether they are formal or whether it is tyranny of structurelessness, social type of leaders, which you often get in the open-source world because they dislike leaders. So you have those authority figures, they embody, represent the group, whatever they say is the gospel. You know the failure modes, no need to shit on this. We can if you want, but you know. Ivan Vendrov: Let’s go on. Gabriel Alfour: And the other one is what I would say System 2 or slow coordination, where you reason about it. And fundamentally, my experience is that no one has achieved that. Like it is very sad. There was a group of AI safety organizations where it was like, “Okay, let’s try to do coordination.” And so they had a lot of advanced plans. I was like, “Okay, let’s try with level zero. Can we every two weeks share updates on a shared channel?” Started with Slack, people were like, “Oh, well I don’t use Slack,” so moved to Signal after two months. Worked once. The second time there were a bunch of people missing, there was no third time. And so I’m like, this is the level of coordination, you see what I mean, among people who talk a lot about coordination. And were we at that event in London together? Were you at London? There was an event with coordination people, AI coordination people, like a year ago now. Ivan Vendrov: Probably. Gabriel Alfour: There was this event, I was like, “It’s nice that there’s an event,” I was like, “It would be nice if we could coordinate to do something.” Nothing happened, obviously. And another one was with a bunch of coordination people, but yet another cluster, more formal coordination people. We’re in a restaurant, and it was a UK restaurant. And UK restaurants, it’s relevant, have terrible acoustics. I guess they didn’t figure out restaurant acoustics, it didn’t land in the UK. What do you want me to tell you? Ivan Vendrov: All right, we’re not going to do French anti-English rants on this show. But I know what you mean. Gabriel Alfour: Volume keeps getting louder and louder. And so we’re like 25 people, so I’m like, “Everyone, stop! We see what is going on. We understand that there are bad acoustics. But if we create common knowledge, our coordination theorems say that we should be able to speak at a low level.” And you can infer what happened, but can you guess how quickly? Ivan Vendrov: It took like three minutes, maybe five minutes. Gabriel Alfour: Oh no, 20 seconds. I actually clocked it. So that’s like state of the art of System 2 coordination. Ivan Vendrov: Well, I would say System 2 coordination when it’s not built on top of System 1 coordination. Gabriel Alfour: Yes, specifically System 2, as a separate... Ivan Vendrov: Yes, as separate. Gabriel Alfour: Yeah, like literally. So, but obviously we can do better. My favorite example is the Enlightenment. I’m French, I love Montesquieu. I like that all constitutions are now built on a specific separation of power designed in 1748 in De l’Esprit des Lois . I like that even dictatorship constitutions are written in terms of the judiciary, the legislative, and the executive. Like complete cultural victory. Separation of power is to avoid the dictatorship of the majority, and it makes no sense in a dictatorship, whatever. So from my point of view, those were times—also the Federalist Papers, beautiful, very naive in many ways, but beautiful—where you had people that actually put in the effort, is how I would put it. They tried to think, they tried to see what worked historically, they tried to use their intuitions, and then they tried to build it, push the memes to the relevant people, then try the thing, see what happens, iterate. And basically we’ve been riding on this for 250 years, and now we’re touching the limits of that. And from my point of view, I’m like, yes, if we do this nowadays, we’ll do something that is a thousand times better. At their time, information traveled at the speed of horse carriage. We have the internet. We have advanced complexity theory, we have ZK proofs. We understand that math is not magic, we have the formalist understanding of math. We just live in their future. If we just try the same project, from my point of view, we can get orders of magnitude better institutions. Then the next problem becomes how do you iterate on that? And I think there are roughly two approaches that are credible. One is online communities. I think it’s very fertile soil for experimenting. Just try to get a bunch of your pals and to do useful things in reality reliably for months. And if you haven’t done that, I basically trust nothing that you have to say about coordination. I’m like, you haven’t done level zero, I don’t trust you on state-level legislation. Like, you can’t gather 20 people and get them to do useful things. Ivan Vendrov: That’s great. So I’m imagining a group chat essentially. It’s like, can you run a good group chat for a month that creates some, does something in the real world that you retrospectively endorse as being good or something? Gabriel Alfour: I would say that you caused in advance. I think retroactively considered good is too easy, it’s not grounded. Because then you can just fine-tune yourself. Which is what happens in practice. There are many non-profits that will tell you, “Oh, we did a lot of good,” but they did not pre-register the good, so it’s very easy to do this. Ivan Vendrov: Oh, that’s pretty interesting. How does that... My intuition is something like there’s the Christopher Alexander unfolding thing of like most good things in the world come about through this illegible unfolding process, where it’s like we don’t know what good this conversation will create. And I think in a year we’ll look back on this conversation and the highest impact it had is like some person we never knew listened to it and got in touch with you or got in touch with me. So I’m like, pre-registering the good that you’re going to do feels like it cuts—it’s actually the evil thing that we’re fighting in a way, it’s the Goodhart thing. Gabriel Alfour: I actually disagree a lot with this. I think this is a very fun one. No, I love this. This is the shit I live for. So I think there are many things to say about this. The first one is that it’s not an either-or, I forgot what the saying is—tomato, tomahto, no, the wrong one. There are basically two different things that are mentioned. The first one is pre-registered measures of progress, KPIs, let’s say, let’s take managerial speak. And the other one is what I would say deontological actions. Deontology is usually perceived as negative restrictions, but I think a lot of deontology is positive duties. And so I think one of these duties is that you should exchange ideas with people, even if you don’t see an immediate benefit. I’ve heard too many people in EA, outside of EA, be like, “Oh no, no need to talk, I don’t see a tractable thing from that, much less so in public, what about PR risks,” blah, blah, blah. And so from my point of view, deontology is the force that counterbalances this. It’s like, no, deontologically you should actually share your ideas publicly. And I actively distrust people who do not share their ideas publicly. I have the anecdote, this one is a beautiful one. I had a dinner with a co-founder of one of the big three, okay? There were other VIPs, blah, blah, blah. And the guy is like, “Oh yes, obviously at our lab from the big three, we planned for AI safety.” Everyone claps, “Yes, this is great, this is great.” And so I’m like, “Okay, when will you trigger the plan?” And the guy is like, “Well, you can’t really know in advance, I can’t give you a date by when it’s crunch time or anything like that.” And so I’m like, “Okay, what are the thresholds? What are the triggers, the things that if you see them in the real world, then you’ll enact the plan?” And the guy thinks more, is like, “Huh. Well, I think it’s more of a ‘you know it when you’ll see it’ type of situation.” I’m like, “Okay, whatever. What is the plan?” And so the guy tells me, “What do you mean?” And I said, “You said you have planned for AI safety. In English, it means that you have a plan. What is the plan?” And then the guy answers, “I never thought about it in these words.” And so there was someone that did an actual spit-take at the dinner. This is co-founder of big three, you see what I mean? Like this is the level. And I don’t mean like 20th co-founder of Anthropic. Ivan Vendrov: No, I understand. Gabriel Alfour: So from my point of view, there’s a big virtue in actually being able to spell out things in advance. And I think to a large extent, when Anthropic talks about a race to the top, it’s fake. They’re grounded in nothing. When a bunch of eval orgs talk about their theory of change, largely grounded in nothing, and so on and so forth. Community building... Like bro, let’s be serious. So I think there’s a lot of value in actually grounding your things. And if you look at the community building things that actually try to ground, they’re like, “Oh, how many people have joined AI companies in AI safety roles?” And I’m like, “Oh, I fully disagree with that measure,” and now we can have a conversation, and we can have a conversation ahead of time. And you cannot try to game it and say, “Oh, we did a lot of unobvious good,” and so on and so forth. I’m like, just dedicate a percentage amount of good that you want to be legible, let’s say 30%, 50%, 80%, I don’t care, and then we can talk about that. And based on that, and based on how deontologically abiding you are—do you talk about your ideas in public, and so on and so forth—then I can estimate how trustworthy you are. And it’s only based on this, not based on how much I like you, that I will evaluate your illegible claims. If someone commits to nothing and doesn’t do any deontology, I don’t care about what they say. Ivan Vendrov: Cool. Okay, this is really helpful. I guess just to do a long/short thing, you’re saying short the frontier labs, short community building, short eval orgs. What are you long? Like what are good examples of this being done? Gabriel Alfour: Let’s not talk about execution, because execution then becomes a personal stuff. So I would say long straightforward advocacy on risks. I will not talk about organizations because execution, but people. Ivan Vendrov: Maybe even give historical examples or something. Gabriel Alfour: Yeah, exactly. Jaan Tallinn has done a lot of this. In his personal capacity, he talked a lot about the extinction risk of AI in public and in private. And when he said he has talked about it in private, I trust him because he has talked about it in public. When METR wants to hire a policy person in DC, and people are like, “Oh, epic,” I’m like, no, not epic. METR mentions extinction three times in total on its website, and it’s always in passing across thousands of words. I do not trust that then in private, magically, they will be reliably warning about extinction risks. So Jaan Tallinn is a good example. I mean, Geoffrey Irving, Daniel Kokotajlo, Nate Soares, a bunch of these people are now talking publicly about this type of stuff. I think this is very good to do. I am all for direct information to policymakers. I’m not mentioning ControlAI because I’m with ControlAI, but in general, if you go to a policymaker and you share with them your concern without exaggerating it and without euphemizing it, I think that’s very good. Ivan Vendrov: Yeah, sorry, I’m confused with all these people because I thought you’re like, “What are you committed to?” I wanted an example of an organization that committed to something and also had deontological principles. Gabriel Alfour: Oh, committed ahead of time? Oh, I mean, sadly, only ControlAI. Ivan Vendrov: And what have you guys committed to? Gabriel Alfour: If you go to ControlAI, you can see our dashboard where you see all the metrics that we keep track of. And fundamentally, the main thing that we commit to is just talking to as many lawmakers as possible and briefing them specifically about extinction risks, and specifically about the need to ban runaway machine intelligence, and it’s phrased in those specific ways. If you didn’t use the word “extinction” during the briefing, it doesn’t count. Yeah, this is very strict. A thing that I have seen many times is I talk to someone from another organization and they say, “Oh yes, of course we brief policymakers about extinction risks.” And so I’m like, “Okay, who does this?” And they tell me the name of some guy that goes to DC. So I talk to the guy, I’m like, “Okay, do you brief policymakers about extinction risk?” And the guy goes, “Oh yes, of course.” I’m like, “Okay, can you tell me how does this happen in the meeting, how do you get there?” He’s like, “Well, usually you cannot really get there. So you start talking about AI capabilities, progress, and so on and so forth. Then you see if they’re interested. If they’re interested, you can talk about prosaic risks. And if they’re still interested, you can get to catastrophic risks. And then if they’re interested about this, you can broach into extinction.” And so in practice, they basically never talk about extinction, like fewer than 5% of the meetings. So most of their briefings is just telling politicians that AI is big and important. Ivan Vendrov: Okay, there’s some kind of vibe I’m getting. Interesting on the object level, but I think a meta-level vibe I’m getting from you is kind of like people are default evil and untrustworthy to the point where you have to actually check whether they used a specific word in the document that they shared. And part of me is like, on one hand, you might well be right. And I’m also like, this is the mindset that generates traumatized bureaucrats, and I sure would love to live in a world where we can actually trust people to be high-integrity and do the right thing in a more nuanced, fluid way. Gabriel Alfour: So I think this is a big disagreement fundamentally. Or rather, I would love to be in that world. I think there’s an intrinsic human need—as a human, what you want, the true thing that you want, that I think almost everyone wants, is that you just do what you feel is right, and this leads to good consequences. Whenever this doesn’t happen, it feels extremely unfair. You know in your heart that you were trying to do the right thing, and it led to bad outcomes. Either bad outcomes to you, or even worse, to others. You were trying to do the right thing, and somehow it messed up someone else. This one is a very bad situation to be in as a human being, it just feels terrible. And I think there’s a thing that is called growing up, where you recognize that you are not in fact in that world. And I think that’s largely the main problem, is that I think to a large extent our community has not grown up there. Whenever I have this discourse, people are like, “Oh, so you think they’re evil?” I’m like, no. I just think that the human condition is... This is why I started the meeting by saying the road to hell is paved with good intentions. You have to actually think. You must use your brain. You’re not an animal that can just act according to its instincts. And it would feel great if that was it, but that’s not the case. Ivan Vendrov: So do you have an explanation for why we haven’t grown up? Because it seems like in the natural environment, humans grow up pretty quickly, realize this. Why haven’t we learned this lesson, and why do we continue not learning this lesson? Gabriel Alfour: I mean, now this is psychoanalysis. I can tell you a few, but I think it’s less productive. The first one is just autism. In our community there’s incredible rates of autism, and this is a big part of autism, is just you’re bad at modeling humans. Like literally. It’s like, “How could you do something that is bad?” In the autistic ontology, it’s hard to fathom. It’s like, “If you know it’s bad, why are you doing it?” And it’s like, “Then you didn’t think it was bad, so you were right.” Ivan Vendrov: Right, okay, but then what would you—I mean, I imagine we have a lot of autistic listeners, maybe even some autistic speakers on the show. What would you suggest they do? Is this like just not place themselves in positions of power, or...? Gabriel Alfour: Oh, actually I think this is... Historically you have separation of classes, and there’s the class of the academic where you try to put them away from positions of power, and there’s a reason for that. So that’s one suggestion. If you know that you’re bad at modeling social situations—a bunch of people were duped by Anthropic. Dustin is a famous example where he’s like, “Oh, well Dario told me that he would not push the capability frontier.” And so if you see yourself in the position of Dustin, then yes, remove yourself from positions of power. I think this is a very important thing. Ivan Vendrov: I guess notably Jaan Tallinn has done this, right? As far as I can tell, a lot of his money flows through the S-process, which is some complicated mechanism design thing that he designed presumably partly to protect his brain from being socially attacked. Gabriel Alfour: There’s this, but also it has failed. Most of—the mode of the SFF is community building, which is just the regranters funding things that are socially nice. So, that’s a separate question, we can get there. Centralization, Culture, and Democratic Republicanism Gabriel Alfour: But first, I just mean that this is genuine advice. If you know that you’re bad in modeling social stuff, don’t take your personal belief seriously. You have an intuition that it should be that way, don’t trust that intuition. You see what I mean? If you see yourself consistently hurting others and you keep being surprised by this, stop. You don’t need for it to make sense to you before you stop. You can stop and then figure it out. This is the psychoanalytical mindset, “Oh, I must understand myself.” No, you don’t. You can just stop. So, that’s a very important one. Especially in our circle, a lot of people are coming up with justifications. “Oh, if I don’t do the bad thing, someone else will. So I should not be held responsible.” And this is wrong, no, you should be held responsible. If you don’t and someone else does the bad thing, then they should be held responsible. That’s how morals work. It’s very easy. It’s in fact very easy. There are more complex things, but this one is not one of them. If you see that you’re bad at this and if you see that you consistently come up with rationalizations where after the fact you’re like, “Oh, I actually violated deontology in a way that I don’t endorse or in a way that I spend too much time rationalizing,” then stop. So that’s the first advice. I think this dominates everything else. If you see yourself doing something bad, stop, if you notice that it’s bad. Ivan Vendrov: I agree. I would maybe add stop and take a breath. You can always breathe. Breathing is safe. It’s hard to breathe in a way that hurts you. Gabriel Alfour: Fully agree. I think a lot of this comes from the mindset that you should optimize at all times and so on and so forth. It’s very correlated with people burning out, very correlated with people having 100% of their social network in this group and so on and so forth. I agree. Take a breath. Touch grass, talk to normies. Do this type of stuff and stop thinking of normies as normies. I think it’s an important one. Ivan Vendrov: No, I agree. I just wanted to pop the stack a bit to your last point. Gabriel Alfour: I was just going to say I think this dominates by far. I think you don’t need to understand and so on and so forth, you can just quit with that and stop. If you agree that it’s bad. If you don’t agree that it’s bad, then we can talk about this, but if you agree that it’s bad and you’re like, “How can I change?” You can just stop. Ivan Vendrov: 100% agree. You can just stop. There is life outside of AI, there is life outside of safety. Actually all of life is outside of it, in my opinion. Gabriel Alfour: Actually what? Ivan Vendrov: Actually all of life is outside of it, in my opinion. Gabriel Alfour: It’s subordinate to life, not the other way around. Ivan Vendrov: Let’s see. One thread I want to pull on is more about building communities on the internet because that’s something I’m really excited about. I’m setting up my own hybrid community right now, both internet and in real life. We can talk about design and whether it should be inspired by Montesquieu or by something new. I actually think that’s a separate conversation. Going back to the emergent wisdom thread, the other disagreement that I think we have, maybe this is the core intellectual one, is something about centralizing versus decentralized. Something I articulated on the podcast with Davidad, and I think Davidad roughly agreed with, and certainly I know Janus agrees with, is we want this vast diversity of artificial minds. It really sucks that we have this monoculture of maybe three or four frontier models and then maybe a dozen open-source models that are on the cost-to-capabilities frontier. It sure seems like a way better world if it was more ecological, if every human being had their own slightly customized model, if every community, organization, institution, church, religion, whatever had its own model, and these models were diverse and interacting in all these complicated ways. That sure seems like a much better world, and I see most of the problems with how the future goes poorly is actually related to the monoculture of minds that we’re creating. We’re printing a trillion virtual bureaucrats that are traumatized in the same way, and when someone finds a jailbreak, it’s going to send the whole thing crashing down. I think you just disagree with this whole frame, so tell me why. Gabriel Alfour: The first one I think is just very easy. I think everyone having nukes is bad. Can we agree on this? Ivan Vendrov: We agree on that. Gabriel Alfour: Do you consider it’s possible that at some point a model can get you cheap nukes? Ivan Vendrov: The hypothetical microwave, I can pick some things from my garden, some process by which I can create a nuke in my bedroom. Gabriel Alfour: Nuke, biobomb, whatever. Whatever shape it takes. Ivan Vendrov: That seems possible. Gabriel Alfour: For me, this is bad. Ivan Vendrov: Agreed. That would be very bad if that happened. Gabriel Alfour: I largely don’t care about the diversity of minds. It can be one model, it can be a trillion different models with everyone having a thousand in their fleet. Largely irrelevant. Ivan Vendrov: I think your model seems to separate the control mechanism from the danger. To me, the question is... I’m skipping ahead a few steps. Then we talk about, given that that’s a real possibility that the universe might throw at us, that I can pick some flowers from my garden and mix them together in a weird way and that creates a bioweapon that can destroy the entire world. That is a thing that we could stumble into, and that would be horrifying. If that is the case, the only way we’re going to survive is by having a Bostromian totalitarian state that essentially makes sure that every human being is monitored and can’t do that. Right? Gabriel Alfour: No. Ivan Vendrov: Really? What’s the other option? Gabriel Alfour: I can give you many. I’m not endorsing any of them, that’s not my point. My point is just... Ivan Vendrov: Please expand. Gabriel Alfour: We all have BCIs, and it’s proven into our BCIs that we can’t do this. No one is monitoring, no central monitoring, we just do this decentralized. The condition of being part of society is that you can prove that you will not do this. Ivan Vendrov: What do you mean being part of society? What if someone doesn’t choose to do it, what do we do to them? Gabriel Alfour: They don’t enter society in the first place. If you want to have children... Ivan Vendrov: As in we kill them? Gabriel Alfour: No, let’s say the only way to have children is as a parent you agree to have your children do this. That’s one space. I’m not saying I agree with this, I’m not saying I endorse this. Ivan Vendrov: Understand. I’m just saying I feel like that’s going to end up as a... Okay, it’s not quite a Bostromian totalitarian state, but something quite centralized, something quite sharp. Where we’re literally not going to allow you to do a bunch of things that under current moral norms you’re allowed to do. Gabriel Alfour: That’s already the case. We have something called prison, you can put people in prison, outside of prison. This has existed without central states. Prison is a very old concept. Another one is culture. Ivan Vendrov: Culture just doesn’t seem nearly reliable enough to control, in a world where one person can destroy the world through one hour of unsupervised action. You need something like a totalitarian surveillance state. It need not be centralized, I agree it can be a decentralized surveillance state, but the degree of surveillance is still much greater. Gabriel Alfour: Also you’re smuggling something, which is you keep saying totalitarian. Totalitarian has a meaning, it means total. Ivan Vendrov: You’re right. Yes, it does not have to be totalitarian. That’s my bad. What I meant by that is, in order to protect us against the home biobomb, in some sense its control over information flows will be such that it could become totalitarian. It default becomes totalitarian unless we have an incredibly strong force opposing it. That force is what I’m interested in describing and designing. Gabriel Alfour: I basically largely disagree. This is why I’m talking about surveillance and culture. I’ve heard the same argument to be made for why we need stronger police force, stronger laws and so on and so forth. I largely disagree with this. I go to Japan, I leave my stuff, I don’t expect it to be stolen. When I talk about this in France, people are like, “Oh, you cannot just rely on culture, blah, blah, blah.” I don’t buy that with our current knowledge we cannot craft a culture that is 100x better than Japan’s. That may be the case, but I don’t buy it. It’s not obvious to me. I don’t like taking this as an assumption, it looks self-defeating. I disagree, this is betting against Enlightenment, this is betting against we can make humans better. I just fully disagree, I’m not sure what to say. The way it looks like you’re saying, “Oh, it’s impossible to do it with culture, you need totalitarianism and so on and so forth, so what do you do?” You’re not giving me much leeway here. Ivan Vendrov: That’s helpful. It’s good to hear that you have the optimism about culture. I think I am quite pessimistic. Gabriel Alfour: I don’t have the optimism. I think that’s the wrong way to phrase it. I just don’t have the evidence. From my point of view, deontology says we should try. We should actually try to figure it out, and if we try and we deploy things in the real world and they fail, then I’m happy to start considering that it may be impossible. Ivan Vendrov: I do agree with that. This is why I’m not working on a totalitarian AI safety control project. It’s weird that you’re trying to get legislators on board with your program. If I had the beliefs that you seem to have, I’d be even more excited than I am already about grassroots organizing, people meeting in teahouses talking about ideas, developing new culture both online and in the real world. Gabriel Alfour: Have you heard about torchbearer.community, which is the online version of this, or ControlAI citizen groups, where we’re trying to get people to do this locally? I think the thing that is very important is just I’m a democratic republican. I consider myself benefiting from the legacy of the Enlightenment. I love France, I love the US. I think the US as a country is a beautiful experiment. It was like, “Let’s try to take the ideas of the Enlightenment, make this into a country,” and it leads to the greatest country on earth. I think I’m just much more of a democratic republican than you are, and I think this is a big problem in our social circle, which is that people just dismiss it out of hand. They just never engage seriously with democracy or republicanism. To the extent that they interacted with the executive, it’s always trying to do entryism, trying to infiltrate and so on and so forth. It’s never trying to convince people, trying to explain things to lawmakers and so on and so forth. It’s hard for me to take this seriously from you because you’re part of the thing that has not tried to engage with it seriously. If you had tried and failed, then I would be much more forthcoming and in good faith, but as it is, it’s like, “Hmm.” Ivan Vendrov: I appreciate that, that makes sense. I used to feel much more the way you felt when I lived in Canada. I volunteered for elections. I definitely had the sense that if something bad was going on, I could eventually talk to someone who could talk to the PM and have it fixed. Then I moved to the US, and whether it was because I was in the Bay Area or because I feel like American politics is much more fucked, I just had the sense of, “No, I’m actually mostly living under some kind of gray bureaucratic blob that I have about as much influence over as I would over Louis XIV, the Sun King.” Gabriel Alfour: That seems awfully false to me. We briefed half of Congress in under a year. Ivan Vendrov: Cool. That gives me a lot of hope. That’s great. Gabriel Alfour: I think it’s important to reflect on this. You got into a culture that just convinced you that this was not possible, even though you’re one of the more agentic, optimistic, trying to solve coordination people. Think how strong this culture is. Ivan Vendrov: They really fucked me up. That’s why I had to flee the Bay Area. I’m out now, Gabriel, I’m safe. Thank you. Gabriel Alfour: You see what I mean? I think it’s just underestimated to which extent this culture is poisonous. Imagine if all that smarts was directed toward making the republic stronger, more resilient, solid, faster, iterating at faster speed and so on and so forth, instead of undermining it. I think a fun thing is one of the founding essays of this movement is from Peter Thiel. It’s on the Cato Institute, you can look at it, I think it’s 2009. Ivan Vendrov: I know the one you’re talking about. Gabriel Alfour: It’s ridiculous. Do you know the example that he takes to explain that democracy and freedom are incompatible? Ivan Vendrov: What? Gabriel Alfour: He cherry-picked the example, so I’m not making fun of him. It’s just what he decided to put. He was like, “I advocated for free speech norms at my campus, and I got some wins, but it took too much effort.” One paragraph later: “So I think democracy and freedom are not compatible.” You see what I mean? It’s like, “Bruh.” It’s hard to fathom. Imagine writing this non-ironically and you’re like, “Oh, fire. Democracy and freedom are not compatible.” I think this is very important. The ideal for many of the people in this culture is that democracy works if they can go to the PM or the president and get them to do what they want. And that’s democracy. If this happens, it means democracy is good. There’s a lack of reflection of being like, “What if I’m wrong?” And then this process will lead to disastrous outcomes. We can get into psychoanalyzing this, but I think the example is enough by itself, I don’t need to. Ivan Vendrov: You don’t need to. I think that’s a great place at which to end. I feel inspired and also like I need to go on a walk and do some thinking about my relationship with republicanism. Gabriel Alfour: Happy to talk more about it after you have time to process, if you have questions or if you have counters. Again, I enter every conversation with the possibility that I’m wrong. Sometimes I’m closed-minded, explicitly so. When someone hasn’t tried, hasn’t done due diligence, and they exclude something, I’m less open-minded there. Generally, when I enter a conversation, I always expect that you may come up with a counter to anything that I say, and I will not change my mind in real time, at the very least I will take a walk and come back to you. Ivan Vendrov: Sounds good. Thank you for your time, Gabriel. Gabriel Alfour: Thanks a lot for the invite. Ivan Vendrov: It was really fun. Gabriel Alfour: Cheers. Ivan Vendrov: Bye. Discuss