Transformer MLP neurons are challenging to understand.

We find that using a different activation function (Softmax Linear Units or SoLU) increases the fraction of neurons that appear to respond to understandable features without any performance penalty.
https://transformer-circuits.pub/2022/solu/index.html

SoLU is a preliminary experiment in whether architectural changes can make mechanistic interpretability easier. Our hope is that the right architecture might make it easier for us (and others) to reverse engineer neural networks, without loss of performance.

In 1-layer transformers, SoLU causes many MLP neurons to cleanly correspond to end-to-end rules. For example, this base64 neuron which fires on base64 text and increases the probability the next token is also in base64.

In larger models, it reveals an interesting structure across layers:

Early layers seem to "de-tokenize", recognizing multi-token words (eg. "bank|ing" and "Not|withstandng") or words in the context of a specific language (eg. neurons for "die" in Afrikaans/German/Dutch).