Researcher

In October, the interp community got into a fight over the essay Neural Networks Are Not That Complex. The author claims that neurons within models are not as convoluted as everyone thought, but many disagree. It's hard to verify: no one fully knows what features actually exist inside real models, and toy datasets are too simplistic — features have much less hierarchy than in real data 🎲

Our favorite PrincInt (ex-PIBBSS) proposed an approach in their summer paper: generating synthetic datasets using critical percolation.

At the phase transition threshold, percolation yields fractal clusters of all sizes, each of which can be broken down into smaller subclusters. This is similar to how general concepts in data are divided into more specific ones. Only here, the data structure is known in advance, and it's possible to honestly check if interpretability tools find it 🔎

Now PrincInt is looking for a researcher precisely for this direction!

You will create artificial data that resembles real data but is structured in a known way. Then, you will train models on it. The best datasets will be used to assemble benchmarks, on which tools will be tested by everyone involved in interpretability.

🌟 mech interp background; 💡 PhD or equivalent research experience in mathematics, physics, ML, neurobiology, CS, or complex systems theory; 🐍 experience with scientific computing in Python; 🗺 ability to communicate complex ideas even to a non-technical audience; 📺 knowledge of DL and experience training models.

And a passionate interest in AI Safety, of course!

It would be great if you have any of these: experience leading a research direction and mentoring, publications in ML or other quantitative sciences, as well as knowledge of statistical physics (especially percolation and renormalization), coalescent processes, combinatorics, network and graph theory, fractals, scaling laws, or cognitive science.

tl;dr: research scientist, $125k-170k/year, remote

Remote work is available, but priority is given to those willing to actively engage in London / Bay Area 🧋

There is visa sponsorship, a budget for insurance and perks, and the team is led by a PhD in astrophysics and an ex-NASA postdoc 🐌 They pay $125k-170k.