My opinion here is that many safety orgs are constrained by their source of funding, which typically comes in the form of grants from places like Coefficient Giving. Without a product and B2B to drive sales, like OAI/Ant/Google and even neolabs, its much harder to justify high salaries from an economic perspective.

Its quite difficult to imagine what kind of product a safety organization would sell, and how it would not directly contradict their relationship with frontier capabilities, so this is a big tradeoff from the perspective of mission-aligned leadership in AI Safety organizations.

I think it would also raise the status and prestige of AI safety research, which could be super important!

I disagree. Status and prestige games have already infected most of AI Research, and has partly led to the situation we are currently in, with many lab employees staying for the money + status. AI Safety falling into the same traps would likely hurt its long term vision by building the wrong type of culture.

I think a lot of that has to do with the stark differential in salaries (and hence, status) between lab employees and everyone else. Closing that would be psychologically healthier for everyone, and make it easier for people in the community to resist the temptation to do dubiously-non-capabilities-advancing research at a lab. Also, I think the framing of status as a "game" is a common rationalist blindspot which makes us worse at handling these issues. Status is for cooperation.

Kevin Simler made a similar argument to "status is for cooperation" in case you're interested:

Bottom line: I know of only one minimum viable architecture for turning individuals into a superorganism [2] — for making sure it's in everyone's self-interest to work together. I call it the Prestige Economy ,[3] and it runs on a deceptively simple rule:

Individuals should grant social status to others for advancing the superorganism's goals.

That's it. That's the One Weird Trick that unlocks so much of our species' cooperative potential.

Now I don't expect you to buy this just yet. There's more I need to explain. But first, notice how efficiently this solves the free-rider problem. In a Prestige Economy, people don't work hard because of the benefit they'll get when the superorganism achieves its goals (although it's a nice bonus), but rather because of the status they'll earn from their peers along the way. You're perfectly welcome to shirk your duties in a Prestige Economy — you just won't earn any kudos. And actually, if you shirk too much, others might be incentivized to notice and punish you, if in doing so they'll be perceived as advancing the superorganism's goals. ...

Notice that, for humans, celebrating a hero doesn't feel like a costly act. We don't treat it like some annoying duty we're always looking for an excuse to get out of. In fact, we're happy and even eager to celebrate heroes. It just feels right and natural to us. And this kind of enthusiasm is the tell-tale sign of self-interest .

Nature, then, has endowed us with the instinct to celebrate heroes because it ultimately benefits us to do so. Yes, it costs money to buy someone a drink, but we're getting something even more valuable in return: the chance to curry favor with a potential ally. And not just any ally, but one who has proven his worth, shown himself to be the kind of person who's useful to have on one's team. That's why it behooves me to cozy up to the hero. Not because I'm motivated to help provide a public good (status for the hero), but because I'm hoping to make a valuable friend.

This is the magic of prestige status . From my perspective, I'm sucking up to the hero, hoping to cultivate an alliance. But from his perspective, my admiration is his reward . Ultimately, it's this pair of incentives — prestige and celebration, seeking status and currying favor — that binds a superorganism together. They're like the two interlocking sides of a Lego block.

Of course there are other ways to seek (and win) prestige status, outside the context of a superorganism. You might write a PhD thesis, for example, or run a marathon, or learn to play the piano. These individual achievements play a similar role in making you more attractive as an ally. It's not that "helping a superorganism achieve its goals" is the only way to win prestige, just that it's a particularly good way.

I should also note that "Prestige Economy" is only one way to formulate this idea. Other (refactored) formulations include competitive altruism , indirect reciprocity , enlightened self-interest , and the long literature on reputation. And reputation itself goes by many names and takes a variety of forms: honor, respect, status, prestige, karma, credit, esteem, even money. ("Money," as I wrote elsewhere , "is industrial-grade prestige status.") These are all slightly different ways of looking at the same thing.

To summarize, then, here's our recipe for a minimum viable superorganism:

Selfish individuals pursuing shared goals (arising from shared underlying incentives), held together by a Prestige Economy which consists of two activities: (1) seeking status by attempting to advance the superorganism's goals, and (2) celebrating (i.e., sucking up to) those who deserve it.

This architecture is robust. It creates selfish incentives for people to work together and reward others for their hard work.

If you raise the salaries several times, you hire that many times less employees. So it depends, do you think AI safety rests on superstars, or a wide effort?

If you’re hoping the superstars will just take the money and hire other people to run their own outfit, PI style, then you’re actually asking to devolve funding. And it fails if the employee decides to not do that, or if they run off and found something like Mechanize.

This wouldn’t be that expensive! I’d guess 10k people might qualify currently, so this would cost $500M/y — peanuts, compared to Anthropic wealth

The main argument against higher salaries in AI Safety is that the field is highly funding constrained. Anthropic is IMHO unlikely to start funding AI Safety.