The current source text is the original text and can be switched to the Chinese translation.
SaveShare
报道配图 · 来源:LessWrong
Overview
Recently Goodfire has come out swinging with two posts indicating that neurons in transformers are "more interpretable than previously assumed", or simply "already clean". In the "Neurons as Interpretable Basis" post they offer three reasons for this claim: First, neurons are already functionally sparse - pruning 92 % of the weakest MoE neuron activations only increases the loss by 2.9% Second, the role of a neuron can be split into different "computational functions". Effectively, we can name the contexts in which a neuron fires, and assign reasons to those firings. Most neurons only have one context-reason grouping. Finally, we can produce automated natural language explanations of neuron firings that are predictive of their behaviour. First, I would like to congratulate them for their detailed work and breakdown, complete with well-designed graphics. I found the work interesting and very well thought through. It is a shame that we have no idea if these techniques apply to lab models with non-public architectures, but I think it is fairly reasonable to say that at least some of this work is transferable directly (perhaps until a fully recurrent architecture is the norm instead of looped transformers). Next, I would like to consider the original motivations for interpretability. In A Mathematical Framework for Transformer Circuits , Elhage et al. write: Transformer language models are an emerging technology that is gaining increasingly broad real-world use, for example in systems like GPT-3, LaMDA, Codex, Meena, Gopher, and similar models. However, as these models scale, their open-endedness and high capacity creates an increasing scope for unexpected and sometimes harmful behaviors. Even years after a large model is trained, both creators and users routinely discover model capabilities – including problematic behaviors – they were previously unaware of. One avenue for addressing these issues is mechanistic interpretability , attempting to reverse engineer the detailed computations performed by transformers, similar to how a programmer might try to reverse engineer complicated binaries into human-readable source code. If this were possible, it could potentially provide a more systematic approach to explaining current safety problems, identifying new ones, and perhaps even anticipating the safety problems of powerful future models that have not yet been built. From this we can intuit a rationale to engage in mechanistic interpretability research based on several factors: Open-endedness: Transformer models can engage in novel behaviour in unexpected situations, including generalisation to novel domains outside of training data. Emergence of problematic behaviour: Some of these novel behaviours are problematic and unexpected. Explainability : Interpretability allows us to explain why these problematic behaviours occur. Anticipation of future issues: Interpretability allows us to potentially predict the existence of problematic behaviours before they are discovered "in the wild", including in future models. With this in mind, the distribution of prompts and documents from which Goodfire draws its conclusions is of paramount importance. The power of a theory lies not in its ability to explain the well-known, but in its ability to offer counterfactual predictive power in novel situations (For interpretability, a simple test would be whether the proposed explanation or prediction of model behaviour is more informative than just reading the prompt). Therefore, I would be excited to know what the dataset of prompts and documents Goodfire used to derive its results contains. In addition, here are some categories of texts I would be particularly interested to see when testing if interpretability techniques generalise: In context learning: Prompts which consist of novel patterns or sequences that must be learned in-context. For example, create a randomised POMDP , give the transformer a list of recent observations, then ask it to predict the next observation. Gibberish: Nonsense data with random words or phrases strung together. Glitch tokens or sequences: The equivalent of "solidgoldmagikarp", spiralism prompts, " see the below " etc. Hors-text: System prompts, constitutions, soul documents etc. In general, anything that has been given "synthetic emphasis" during mid-training or post-training. Jailbreaks and malicious prompts: This is probably self explanatory. I am excited to see what these experiments reveal! Discuss
Recently Goodfire has come out swinging with two posts indicating that neurons in transformers are "more interpretable than previously assumed", or simply "already clean". In the "Neurons as Interpretable Basis" post they offer three reasons for this claim:
First, I would like to congratulate them for their detailed work and breakdown, complete with well-designed graphics. I found the work interesting and very well thought through. It is a shame that we have no idea if these techniques apply to lab models with non-public architectures, but I think it is fairly reasonable to say that at least some of this work is transferable directly (perhaps until a fully recurrent architecture is the norm instead of looped transformers).
Next, I would like to consider the original motivations for interpretability. In A Mathematical Framework for Transformer Circuits , Elhage et al. write:
Transformer language models are an emerging technology that is gaining increasingly broad real-world use, for example in systems like GPT-3, LaMDA, Codex, Meena, Gopher, and similar models. However, as these models scale, their open-endedness and high capacity creates an increasing scope for unexpected and sometimes harmful behaviors. Even years after a large model is trained, both creators and users routinely discover model capabilities – including problematic behaviors – they were previously unaware of.
One avenue for addressing these issues is mechanistic interpretability , attempting to reverse engineer the detailed computations performed by transformers, similar to how a programmer might try to reverse engineer complicated binaries into human-readable source code. If this were possible, it could potentially provide a more systematic approach to explaining current safety problems, identifying new ones, and perhaps even anticipating the safety problems of powerful future models that have not yet been built.
From this we can intuit a rationale to engage in mechanistic interpretability research based on several factors:
With this in mind, the distribution of prompts and documents from which Goodfire draws its conclusions is of paramount importance. The power of a theory lies not in its ability to explain the well-known, but in its ability to offer counterfactual predictive power in novel situations (For interpretability, a simple test would be whether the proposed explanation or prediction of model behaviour is more informative than just reading the prompt). Therefore, I would be excited to know what the dataset of prompts and documents Goodfire used to derive its results contains.
In addition, here are some categories of texts I would be particularly interested to see when testing if interpretability techniques generalise:
I am excited to see what these experiments reveal!
READ THE ORIGINAL
Read the original for the full context.
The content of this article comes from LessWrong,Copyright belongs to the original author and publisher.
Tl;dr : Intuitions we have about human organisations, like how we think of making deals or how we think about the power or values of an organisation, can be productively applied to understand groups of AIs taking coordinated action in the world. An important consideration in understanding the type of AI organisations we expect is which notion of self AIs will adopt.
Business or Consumer? Everyone you sell your product or service to is a customer. However, the question is: do you treat them as a business or a consumer? You might think that’s a pointless distinction since your goal is simply to sell your product, right? Wrong. That question determines your product’s market fit.
The Ban RSI protest at Anthropic’s IPO was encouraging: well-attended, covered by media, high-energy. It was focused on a single message – that we should ban AI systems that are capable of building more capable AI systems. It was a promising outcome for a protest planned only a couple of days in advance. But there was something weird about it, something I couldn’t put my finger on while it was going on.
Loading comments…