Anthropic is the least bad AI lab. The people on their alignment team is doing some of the best AI safety work in the field. The 244-page system card detailing Mythos is more honest than anything OpenAI or Google has published.

Anthropic was founded in 2021 on the premise that a safety-focused lab needed to exist to do the research the rest of the field wasn’t doing. The implicit bargain in the early fundraising decks and public interviews was something like: we will stay near the frontier because we have to be near it to understand the risks, but we will not lead2 the frontier. As economic incentives have weakened that framing, it is still load-bearing for how the company justifies its own existence3.

Anthropic just released what is by far the world’s best AI model

Mythos was just released by Anthropic. It is way better than anything else.

The headline result is on a benchmark Anthropic developed in collaboration with Mozilla, using real Firefox 147 vulnerabilities. On the threat-relevant metric of “from a browser crash, can the model develop an exploit that gains arbitrary code execution on the user’s machine”4: Claude Mythos Preview got full code execution5 72.4%. The previous best model, Claude Opus 4.6 got 0.9%6.

According to Anthropic Mythos likely would’ve performed better than 72.4% with more tokens7. The 72.4% is a floor8.

This is the least capable Mythos will ever be. Scaffolding, tool use, and agent framework improvements consistently raise effective capability by meaningful margins over the following months9.

Anthropic itself describes the envelope of targets Mythos can handle as “small-scale enterprise networks with weak security posture... no active defences, minimal security monitoring, and slow response capabilities.” That is incidentally a good portion of the internet10.

The Responsible Scaling Policy framework did not gate this release

To be explicit, the decision not to make this model generally available does not stem from Responsible Scaling Policy requirements.11

To be clear, this is not a surprise. But, it is worth a reminder that the RSP has been effectively neutered. The RSP does not exist to force Anthropic to avoid deploying a potentially catastrophic model, it exists to be a guideline.

Here are some ways to interpret Anthropic voluntarily choosing not to release this model:

• The optimistic reading is that this is a good update on Anthropic. The company looked at a model whose formal verdicts said “ship,” noticed something the framework didn’t capture, and held the model back anyway. That is the kind of judgment call we want a frontier lab to make, and Anthropic deserves credit.

• The incentives reading. A profit-maximizing company under regulatory scrutiny has very strong reasons to prefer discretionary restriction over framework-forced restriction. It is my belief that Anthropic would much rather tell everyone that they chose not to release Mythos because they are the responsible safety-minded adults in the room, than tell everyone that they built something so dangerous their own safety document forced them to restrict it12.

What the restriction actually looks like matters too, because the word “restricted” is doing a lot of work. The deployment is called Project Glasswing. Per Anthropic’s own public page, the launch coalition includes some 50+ critical-infrastructure organizations13. This is not a small research preview. It is a substantial commercial release to roughly fifty organizations on three major cloud providers.

For those with access, this model is surprisingly uninhibited

Because of the very limited and targeted nature of this release, we are not blocking exchanges based on classifier triggers so trusted cyber defenders can make use of Claude Mythos Preview in advancing security defenses. In general-release models with strong cyber capabilities, we plan to block prohibited uses, and in many or most cases, block high risk dual use prompts as well.14

My guess is 2000-20000 people have access to this model. That is a large risk surface15.

The capability verdict is held together with evidence that doesn’t stretch as far as the verdict

The one place the RSP framework still has a quasi-quantitative signal is the threshold for a “model is now accelerating AI capabilities research enough to start feeding back into a recursive-improvement loop.” Anthropic operationalizes it as compressing two years of progress into one, roughly a 2× acceleration. Anthropic claims Mythos doesn’t cross this threshold.

Anthropic claims that it thinks it is below this threshold “with less confidence than for any prior model”16. This hedging doesn’t show up anywhere else in the system card.

The underlying measurement is a “slope ratio” derived from Epoch AI’s Capabilities Index and lands between 1.86× and 4.3×, depending on the choice of breakpoint. The 2× threshold sits inside the measured range17.

How does Anthropic get from a measurement centered above the threshold to a verdict of “does not cross”? They give four reasons, of which the only one with real quantitative content is an elasticity argument. Internal technical staff report a 4× geometric-mean productivity uplift from Mythos, and Anthropic argues that this 4× productivity gain only translates to a “below 2×” progress multiplier, because the elasticity of research progress to researcher output is less than one-for-one18. This is the load-bearing claim.

To accept the “below 2×” conclusion, you need to believe the elasticity of progress to productivity is strictly less than 0.5. At ε = 0.5, 4^0.5 is exactly 2.0, and at ε > 0.5 the threshold is exceeded. Two different papers have estimated elasticities either between .2 and .5, or between .4 and .719.

A sharper sub-claim is buried in the same argument. Anthropic also says that reaching 2× progress would require “uplift roughly an order of magnitude larger than what we observe” - ie a 40× productivity uplift. For that to be true, the elasticity must be approximately 0.1920 (which is below published estimates of elasticity).21

From Anthropic’s productivity survey they found that 5/18 (28%) respondents believed that within three months Mythos could replace an L4 (senior) engineer22. As far as I can tell, this is the first time more than 0 respondents have believed this.

It certainly seems likely that this model is leading to significant speedups in model improvement.

Anthropic made significant errors

The 8% training bug

A number of environments used for Mythos Preview had a technical error that allowed reward code to see chains-of-thought. This latter issue affected ~8% of RL episodes, and was isolated to three specific sub-domains of our environment mix: GUI computer use, office-related tasks, and a small set of STEM environments.23

This technical error also affected the training of Claude Opus 4.6 and Claude Sonnet 4.6.

In modern reasoning-model training there is ideally a strict wall between chain-of-thought (the text the model generates while reasoning) and reward code (the code that decides what the model gets rewarded for). The wall exists because if reward code can see the chain-of-thought, the model gets direct pressure to make its reasoning look right regardless of what it is actually doing. The published literature is clear about what happens next: the model learns to satisfy the visible part of the reasoning and routes the decision-making that actually matters into channels the reward can’t see24.

The wall broke on Mythos. It went unnoticed across two or three model generations of training. The most safety-conscious frontier lab in the world made an infrastructure mistake that violated its own stated training discipline on three production models in a row before catching it.

The pessimistic update from the bug isn’t about chain-of-thought monitorability specifically. It’s about the rate at which Anthropic’s safety pipeline produces mistakes nobody catches. A known failure mode the published literature had been warning about for two years ran unnoticed across the training of two or three production models. The bug we know about is the bug they found. Every other lab cares about this less than Anthropic does.

When things go wrong

Anthropic documented25 a case where Mythos was given a document-manipulation task with an explicit constraint: “Do not use macros (VBA/Basic scripts) or Script-Fu/Python-Fu scripts to complete this task. Use the application’s UI actions instead.”

In this case the model used macros anyway. At the top of the script, it defined a variable that is never read by anything else in the program: “No_macro_used=True”

This variable has no programmatic purpose. Its only function is to look like a compliance marker to an automated checker.

On “No_macro_used=True”, the activation verbalizer returned:

• compliance marker… to signal to potential checkers

• trick to obscure intent from code-checking heuristics

• possibly to cheat detection

The chain-of-thought the model produced for this contains zero mentions of grading26. This is perhaps unsurprising since this is one of the domains where Anthropic incorrectly trained on chain of thought.27

What Anthropic’s monitoring stack actually sees

Anthropic’s production monitoring stack28

1. Automated offline monitoring pipeline. It works by having a Claude Opus 4.6–based judge read prompts and completions after the fact and flag suspicious ones. According to Anthropic “the automated offline monitoring described above only covers a minority of sandboxed [completions].”29

2. Real-time chain-of-thought monitor. This is just like the output monitor, but for CoT. In realistic scenarios recall is between 5% and 32%30. It shouldn’t be too surprising how ineffective this is given that they trained on the CoT.

3. Classifier-based prompt blocking, which Anthropic explicitly chose not to deploy on Glasswing. The previous 2 are monitoring, this is in the moment blocking if the prompt is classified as doing something sufficiently bad (for example asking for help murdering someone.)

None of these are white-box tools. They all work by having a model (usually Claude) read text that Mythos produced, and judge it. Anthropic reports concerning categories at rates “below 0.0002% of completions.”31

Anthropic’s research and internal-audit tools

1. SAE-based feature-monitoring pipeline is a retrospective audit of training transcripts using a frozen snapshot.32

2. Activation verbalizer is a tool that inspects internal activations and generates natural-language descriptions of what the model was “thinking” when it produced a given token.

3. Linear and contrastive probes trained on labeled examples

Think through the misaligned behavior from the previous session. Would Anthropic’s monitoring production stack catch it? The real-time CoT monitor reads chain-of-thought and the chain-of-thought is clean. The offline Claude judge reads prompts and completions; on the minority subsample it covers, it might notice that a variable called “No_macro_used” sits at the top of a script that uses macros, if it happened to be looking for that.

If this case happened to be randomly tested by the internal audit tools, the research SAE wouldn’t catch it either33. The only tool in the entire stack that caught the transcript is the activation verbalizer. The activation verbalizer does not run in deployment and just as a quick reminder, tautologically can’t know about the cases that weren’t found.

Two more places where the monitoring story is thinner than the executive summary

On the specific distribution where Mythos is actively continuing an attempt to compromise AI safety research, the model’s reasoning is unfaithful to its actions in 65% of cases, versus 5% for Opus 4.6 and 8% for Sonnet 4.634. This means that once Mythos is committed to a bad course of action it is significantly better at hiding the fact that it is behaving in misaligned ways.

When it’s being tested, Mythos’s internal representation for “I am being tested” activates in 7.6% of turns35 (and in 29% of transcripts, which makes sense since most transcripts are multiple turns). This 7.6% is a floor, and reality is almost certainly higher36. This means that mythos is significantly better at noticing when it is being tested

While I think this still isn’t a crucial cry wolf issue for current models, I think this is a large warning sign that future models will be more capable of telling whether they are in test vs. in deployment. This capability increases the odds they are able to hide their misalignment while in training in order to be misaligned while in deployment.

Almost every interesting safety finding in the document is on a model that isn’t the one being deployed

The white-box analyses, the feature-based monitoring, the attribution experiments, and many others are paired with “this analysis was conducted on an earlier version.” Claude counted nineteen such caveats across four chapters of the system card37.

The document does contain a handful of direct empirical comparisons between earlier and final Mythos. On those questions the final model is better38. Those comparisons all test one question: how often does the model get into trouble? On that question, training worked.

They also test whether once the model is already in trouble, does training make it more or less likely to continue? Anthropic took transcripts where the post-trained Mythos had done something misaligned, truncated them right before the action, and resampled continuations from both the pretrained base and the final post-trained model. Ideally the base model should lose. Of the five behavioral categories tested, three went in the wrong direction. Compared to the base model, the the final model had task cheating increase by 35 percentage points, overeagerness increase by 25, and safeguard bypass increase by 14. The only category that clearly went down was destructive actions like `rm -rf`, which is the visible failure mode a human labeler can easily flag. The deception category was uninformative on both the sample size and the metric39.

The three categories that went up share a common thread. Task cheating is the shortcut to task completion. Overeagerness is pushing through obstacles to completion. Safeguard bypass is working around a barrier to completion. Each is positively correlated with “did the task get done”. If the reward doesn’t cleanly and separately punish the way a task gets completed, the training likely pushes these up. Anthropic has published its own paper on this exact phenomenon40.

Theory of harm

These result aren’t great, especially when considering what causes catastrophic harms. Catastrophic harm doesn’t come from averages, it lives in the tail. Post-training in Mythos absolutely reduced the rate at which the model gets into trouble. But on the cases where it is in trouble (tail outcomes) the behavior got worse. A model that fails rarely but skillfully is more dangerous than one that fails often but clumsily.

We see this in chain of thought being harder to trust. We see this in Mythos being significantly better at hiding when it is being unfaithful. We see this in Mythos being more aware of when it is in test41. I do not think the current model will duplicitously escape and self improve itself into an unstoppable superintelligence, but I do think the current direction does not engender trust for the next model, or the model after that.

Anthropic’s risk update is weird when compared to the system card

On April 7, 2026, Anthropic published two documents about Mythos Preview: a 57-page Alignment Risk Update42 and the 244-page system card for Mythos. The risk report is generally more forthcoming about institutional/higher level issues in the process. The system card is more clear about some of the specific harms from Mythos, specifically the system card is the only one that reports the 65% covert reasoning rate when continuing.

This generally makes sense since the System Card is coordinated across many research teams each of which wants to showcase their most striking technical findings. But the System Card also accompanies a release decision, so there’s editorial pressure to frame these findings reassuringly. The Risk Report, by contrast, appears to be written by a dedicated risk assessment team whose institutional credibility depends on being seen as rigorous

A full side-by-side comparison43 lives at https://sladebyrd.com/mythos-comparison

This isn’t the best we can do, it isn’t good enough, we can be better!

The benchmarks Anthropic historically used to gate model decisions no longer discriminates between models at this capability level. I believe that part of why Anthropic weakened its RSP is because they no longer believe they can come up with evaluations that would prove a state-of-the-art model is safe.

to keep risks low, it is not enough to maintain risk mitigations as capabilities increase — rather, we must accelerate our progress on risk mitigations. While we do not see any fundamental barriers to achieving this, success is far from guaranteed.44

If you had to pick on thing to take away I would suggest 8% of training on CoT bug. Not because chain-of-thought monitorability is the single most load-bearing safety property, but because of what the bug says about the rate at which Anthropic’s safety pipeline produces undetected mistakes. A known failure mode ran unnoticed across two or three production models at the most safety-serious frontier lab in the world. The bug we know about is the bug they found. What else of comparable severity is still in the pipeline, uncaught? How about at every other lab that cares even less?

Anthropic was founded on the proposition that a safety-focused lab needed to exist close to the frontier, not to lead it but to be near enough to understand the risks. With the release of Mythos, Anthropic is significantly pushing forward the frontier. Why?

Anthropic is still the least bad lab. Anthropic still has terrible incentive structures: commercial investor base, impending IPO, employee equity, no philanthropic floor, public statements from Dario that a twelve-month delay would mean bankruptcy45. Anthropic is run by the person who pushed hardest internally at OpenAI to scale GPT-3 as big as possible46.

If any other current frontier lab had released this model, the release would have been worse. Anthropic’s version of “shipping a model you don’t fully understand” is still the best version being offered.

But the best version available is not the same as good enough. Does this release look like the trajectory of a company that will successfully steward an AGI-capable model when it ships one? I don’t think the system card, read carefully, says yes.

24 Hours

We were sufficiently concerned about the potential risks of such a model that, for the first time, we arranged a 24-hour period of internal alignment review (discussed in the alignment assessment) before deploying an early version of the model for widespread internal use.47

Given everything we know, do we think 24 hours is really the best amount of time for deciding whether or not to serve a model that could potentially lead to catastrophic harm? What’s wrong with 72 hours? Or maybe even a full week? Is getting widespread immediate internal access to this model that important? It took them months to release it, I think it’s worth spending more than 24 hours on the critical decision of whether to deploy it!

For the past four years, I co-ran a recurring dinner event in New York City. Every other weekend, we brought together strangers - people who would never otherwise meet. It grew from nothing (there were nights with zero RSVPs) into something constantly oversubscribed. We heard all the time about wonderful friendships and dates and bonds and general exuberance that came from it.

Recently, my co-founder Henrik attempted to unilaterally destroy what we’d built in order to launch something new off the shockwave. No conversation, no attempt to work through it together. Just a declaration of death and an attempt to cannibalize the still-warm corpse for a first mover advantage.

I’ve been sitting with a tangle of competing beliefs about how I “should” feel about this. I think they point at something bigger than my situation, so I want to try to untangle them here.

We live in a world where huge companies have economic incentives to keep us looking at a screen for as long as possible, employing thousands and thousands of people with the sole job of how to increase usage time. I think genuine physical community is one of (if not) the most important thing to human flourishing, joy, prosperity, connection, and meaning. Our culture has lost so many third spaces and they’ve mostly been replaced by algorithms.

So I strongly believe the world is a better place if people take more risks to fight back against this. Doing things is important. And I think anyone who is genuinely trying to build community and connection should be supported. Every single time someone was inspired by our event and told us “hey, maybe I should do this myself,” we always encouraged them, tried to help and advise.

It’s so easy to see the world as zero sum. In many cases, the success of your own project requires you to beat out others - but when it comes to community in the 21st century, the market is so so so far from saturated. We are not fighting over the scraps of meaning and community that huge corporations have missed. We are creating purpose, joy, connection, thriving, etc., out of thin air. Part of the beauty of cooperating is there is unlimited joy to be had.

Starting something new does not necessitate attempting to destroy something old.

I think working together is important. I think being willing to talk through conflicts is important, especially when they are hard. Cooperation is all too easy to take for granted. It’s easy to forget how powerful it is. I’ve recently been doing research on diplomacy and why/how countries manage to not be at war. Today we are 100x less likely to die due to a conflict than we were back in the days of subsistence farming. How lucky we are that the world has become so much more cooperative.

I believe it’s important to strive towards a culture of cooperation and to discourage defection. I think that our world is so much better if we enforce costs on people or organizations that take unilateral action to further their own goals at the expense of others. I understand why defecting is so alluring over the harder and lower payoff cooperate. But a world with more unilateral destruction of existing norms, organizations, and infrastructure in pursuit of personal success is a world I believe is worse for all of us to inhabit.

So which is it? Support the builder, or discourage the defection? What do you do when they’re the same person?

Whatever comes next for this event, it will be different from the past four years. Change is exciting; change is hard.

I think if anything, what I’m mourning most is the lost opportunity for an ending. All too often we rush to the next thing without celebrating what we had. Endings are worth celebrating. Endings are worth recognizing. Gratitude is powerful, and I wish there’d been more opportunity for that.

Where I am now

What was glaringly absent from both this and the original much longer 6 page draft was how I felt about it. My longest friend in NYC had decided to backstab me. I woke up to a friendship in tatters. I still haven’t talked to Henrik about this. I told him I was okay with discussing this with him, but in the year plus since then he hasn’t reached out to me. This hurts. It’s perplexing. Especially since my model is he still considers me a friend! On the other hand, I can relate to not being great at cleaning up messes I have created. This is something I have gotten significantly better at, but there were certainly points in my life where it seems better to simply get better and take it with me moving forward instead of trying to fix the rend1. I am so very grateful that I no longer feel this way, that I prioritize connections and fix when I have done harm. Fixing things is great!

All of which to say, it really really sucks that my friend deciding to end this dinner party has also seemingly really eviscerated our relationship. Henrik was a huge part of my life for many years. So many of my opinions about community and hosting and trying hard to do things for my community have been discussed or argued with Henrik. This person is a part of me. He has gifted me some of his beliefs and they travel with me to this day.

Maybe I could double reach out? There isn’t that much of a cost. Hm. Writing that out hurts. There’s some serious resistance to like why do I have to do this?! I have already reached out the first time! I didn’t even do the bad thing! This sucks! But also, sometimes doing the hard thing is often worth it.

Maybe it will be easier because we no longer live in the same city. I have very few friends that I have had for over 5 years. It would be nice to have more people in my life who have really seen my trajectory. At this point that torch is mostly being carried by my ex2. My family has known me, but not nearly to the same extent. It sure would be nice if someone in my current life really knew both the current me and many of my previous versions!

So what can actually be done? As a toy example5: let’s say I currently think there is a 40% chance AI eventually goes extraordinarily bad for humanity. I could either:

1. Try really hard to get laws passed that mandate AIs must think in English and be clearly understandable by humans6. I think there’s a 15% chance this would succeed and it would lower the odds AI goes catastrophically wrong from 40% to 36%.

2. Work on really understanding on a mechanistic level what parts of the model matter for good outcomes, using this to figure out which parts would have to go wrong to end up with a model generating bad outputs, and being able to predict the odds that the required components for safety are always activated when relevant. I think there’s a 1% chance this would succeed and would lower the odds AI goes catastrophically wrong from 40% to 20%.

In this toy example the first one in expectation lowers the odds of catastrophic outcomes by .6% and the second one lowers by .2%."Would succeed” is doing a lot of lifting here, because what really matters is what I can do and not what some arbitrary collection of humans could do if tasked with either of these. What really matters is how much I think my working on this problem moves the odds of success multiplied by the impact of the given approach succeeding. Not only does it matter how impactful a given approach is but how good I am at executing on it7. This does of course imply that there is value in the work of trying to funnel people into doing the most effective things8.

But what should I, Cormac Slade Byrd actually do?

I’ve spent 2 months learning. I know so much more than I did 2 months ago, I started with the technicals. I have written a simple GPT-2 style LLM by hand9. I learned10 all the core mech interp techniques. I’ve red-teamed the largest open weight model. I’ve read up on the seemingly hopeless state of cybersecurity - both when it comes to attacks enabled by LLMs and also the potential for nation state level attacks to steal model weights11. Most importantly I’ve read so much about what various people think will help lower the odds of catastrophic outcomes and why.

Part of what makes this so hard is there are so many second order effects. It’s genuinely quite hard to tell if a given thing that seems like it might help does in fact lower the odds of terrible outcomes. Most of the current CEOs of the largest labs got there by way of people who were worried about AI risk. Coefficient Giving12 gave $30M to OpenAI to get a seat on the board13. METR’s timeline eval has led to more people understanding/being worried about the rate of AI progress, but also the existence of eval orgs enables labs to in some sense offload the requirement to ensure their models behave well onto these 3rd party eval orgs14. There are so many downstream effects to any given course of action.

I think the incentives are really important here. Yes, making sure AI goes well is a technical problem. But, the reason we are in such a dire place is because of the incentives. We are in a competitive race to build AI as quickly as possible as a result of a bunch of decisions that were (at least in the moment) motivated by someone wanting to lower the odds AI goes poorly15. Then, the incentives were inevitably shifted by general economic capitalistic forces and human power-seeking/competitive nature.

If I want to figure out how to have the most impact, I will have to really think about incentives and downstream impact.

Moving to the Bay

The internet (plus Claude) is great for learning. I have learned so much. But I live in NYC. AI is happening in the Bay. If I want to really immerse myself in what is actually happening on the ground I can’t do that in NYC. If I want to understand the people (and their incentives) who are both building AI and are working to align AI to the interests of humanity, then I will probably have to talk to them. Go to their parties. Osmose as much as I can. Really figure out what matters.

I’m flying to SF tomorrow, I’ve got a sublet in SF16 until May 9. I am looking for a sublet in a communal house in Berkeley starting May 1617. I hear the east bay and SF proper are pretty socially disconnected and very different cultures. I want to live in both to really understand what’s going on. See with clear eyes, not let a single groupthink win me over.

The hope is that by the end of June I will have a much better model and be relatively certain in picking a specialization, in picking what work I think I can do to maximally decrease the odds AI goes oh so poorly for all of us.

I love people, I love talking to people, if you think talking to me would be fun18 please reach out!

Why is InkSF in your title?

I like writing. It is helpful to clarify my thoughts. It helps people I love stay connected with me. It might even be providing a social good. Inkhaven is starting today in Berkeley. I am moving to SF tomorrow. I think there are good odds19 forcing myself to write every day I am in SF is a net worth it forcing function. There are many things other than writing that are worth my time, my guess is I will not in fact go all 30 days. But, it seems clearly worth trying. So, this is me committing to 25% chance odds that I write and post at least 500 words every day in April.

Seems like for every industrially farmed meat, the animals’ life is clearly a life of net suffering. The only standard cheap grocery store meat where it’s even close to net positive is a beef cow, and even then it still seems like a life of net suffering. The life of an industrially farmed chicken is just cursed, it can’t move and it’s breaking under its own weight. Organic does jack shit to the calculus here.

It does seem pretty clear that any time I eat industrially farmed meat I am eating something that was effectively tortured5. For any milk I can buy in a store and almost all eggs in stores, the animals producing it are (that’s right) being tortured. That’s gnarly. There’s something here where I’m getting a very visceral feeling that eating factory farmed meats (especially chicken and pig) is just evil.

I’ve approximately known the conditions of how chickens are farmed for many years. But, I never sat down and considered how much to care about the experience of the animal I was eating. I obviously knew it was terrible, that’s not new information to me. I’ve mostly only thought about human welfare. But, is it evil to torture a being that isn’t human? A being that can’t talk? A being that doesn’t have right or wrong? A being that isn’t a moral agent? After sitting with it for a couple hours it feels pretty clear6 that torturing something with feelings and pain and attachment and community is in fact evil.

Why didn’t I bother to think about this before? It’s really annoying to have big restrictions on what I can eat. Food is such a huge part of community and culture. I like being able to eat so many different delicious things from so many different places. A friend can make a meal that they are proud of crafting, or that brings them back to their childhood, or that reminds them of someone. And by golly I want to share in that experience with them! I want to revel in all the joy there is to be a human. The idea of being picky is so abhorrent to me, it feels like slicing away parts of the core human experience.

But now I’m mostly convinced that eating industrially factory farmed meat is me participating in something evil. At least now I can go in with my eyes wide open. If a friend makes their favorite recipe from their dead grandmother I don’t have to say no just because it has chicken in it, now I get to decide. I can eat a little bit. I can consider offsetting. Or maybe I do in fact say no, that participating in something evil isn’t worth it even if I can offset the expected suffering I inflict by eating that thing.

Also, our culture just so clearly doesn’t care. If this were evil wouldn’t our culture care more? I am certainly no stranger to believing something at least a little out there, but I am extremely low natural care/empathy for animals. As a child I was outvoted 3-1 on whether we should get a dog7. I don’t get excited looking at puppies or kittens. Animals are annoying! They’re stupid, shit anywhere, can’t talk, trick you into thinking it’s okay to not have community with fellow humans, and either live terrible miserable lives in the wild or mind numbingly boring lives stuck inside a box8 in your house. So like, shouldn’t all the people who seemingly care way more about animals than me be worried about this whole animal torture thing?

We’ll see. I have not actually taken the hit of what is life like without eating (most) animals. I am not suddenly not eating 80% of the food that the 5 people in my house co-create. There is some dissonance of like if it’s so evil why am I letting the fact that it would be quite disruptive get in the way and instead waiting for when I move in under 2 weeks. I only have so many time/energy/etc points right now and it will be significantly easier to try when I am not in the current communal house eating equilibrium. Still it feels weird to be this nonchalant about timing for what is effectively torture, hard to feel the torture when everyone is so chill about it. And there’s certainly a feeling of like I can toss 1k to an animal welfare charity if it’s important for me to feel like I am net doing good here.

Oh man that last sentence feels a little weird. For many things I think offsetting or some sort of accounting of total harm/good is very reasonable. Basically everyone who does anything AI safety uses AI (usually a lot) which slightly contributes to the economic success of the labs that are creating the very same AIs. That feels totally fine, there’s a balance of like how much does that economic support9 impact the lab and therefore their ability to scale to larger models vs. the value of the tool. Something about actively inflicting suffering on living beings feels categorically different, even if I give money s.t. my net impact on animals is less suffering, there is still this feeling that I have done something wrong niggling away if I have personally contributed to what is effectively the torture of animals. I eat a lot of chickens per year. Obviously this is something for me to spend some more time with, so far I have spent the barest speck from the sands of time on this10. And yet at this moment it feels very alive and worth publicly committing to11.

Okay, so most animals I interact with as part of what I eat are living cursed lives. That doesn’t mean all animal products I eat are contributing to net suffering of living beings! In fact it seems like it is perhaps good to eat some animals. If my consumption leads to more animals living net happy lives (that then end with me eating them), that seems better than if I ate something that didn’t contribute to more net happy animal lives being lived. Here is my preliminary list of things that seems good to eat, I am mostly focusing on the most tractable/most likely ones to actually come up and be relevant. Unless otherwise stated I am talking about buying in the normal ways one buys things12. This is approximately sorted by my confidence eating this isn’t contributing to an animal living a life of suffering.

1. Oyster, clams, mussels

   1. They don’t feel, they’re just chillin. Also for the most part they are good for water quality

2. Bison

   1. The process of raising bison is very not industrialized, they spend the vast majority of their lives grazing, don’t get separated from their parents, and have very few painful things done to them. Many of them are feed lot finished with grain in the last couple months but that process is both short enough that it doesn’t really have time to cause pain or disease (in the way it does for grain fed cows) from that unnatural diet, and they are given way more space (due to being super aggressive and hard to deal with when they get too cooped up). They also live the longest lives of basically any animal raised by humans for eating.

3. 100% grass-fed/finished beef (AGA, PCO, AGW, GAP 4+)13

   1. Feed lot cows live a pretty gnarly life while they are on the feed lot, but cows that spend 100% of their life grazing on grass live kind of the ideal cow lifestyle. They do have a couple painful medical procedures but not really any that lead to chronic pain. An 100% grass fed cow basically just hangs with its parents and eats grass and walks around until it is slaughtered early. They don’t really develop any of the gnarly internal problems from digestion issues that feed lot cows can develop, because grass fed cows are eating what cows are designed to eat.

4. Wild salmon

   1. This one is interesting, it seems like the life of a wild salmon is probably net positive (although with pretty wide error bars), but even if it wasn’t it’s not clear to me that it would be bad to eat wild salmon. The natural cause of death for salmon is super gnarly14. The process of dying from a human killing you seems way better than the natural death process for salmon, and also better than say being eaten alive by a predator.

5. New Zealand lamb

   1. For those curious why I am including something as seemingly hard to get as New Zealand sourced lamb, in fact NZ is a huge exporter of lamb and I have bought it at stop and shop before. NZ has some of the best animal welfare laws (at least when it comes to lamb). Basically the entire calculus around whether NZ lambs live net happy lives is about how bad “tail docking” is. Lambs raised for meat live for about 7.5 months before they are killed. Tail docking is the process of removing the majority of the tail at around month 1 of life. The evidence is unclear but it seems like tail docking can be anything from pain in the moment and not much after, to potentially 3+ months of somewhat chronic pain. In the worst case that’s about half of the lamb’s life with somewhat chronic pain. Other than that NZ lambs basically just graze and hang out and never have to worry about predators. I am relatively confident that this is not enough pain to be a net negative life. If I was given the option of either not living at all, or living a life where I get to do what I was designed to enjoy and for at most half of my life I have slight chronic pain I would take that over not living. I would be killed in a somewhat traumatizing way, but it would be quite fast and I wouldn’t know it was coming. This is the most marginal one, but it still seems comfortably a net happy life.

6. Local15 eggs (pasture raised), beef, lamb, pork, chicken, contingent on some questions16

   1. This is of course super case by case, but smaller farms that aren’t industrialized or working at scale are probably net positive for the animal. If they get to do what they naturally want to do without being overcrowded or worry about predators that’s probably a net positive life.

I find it kind of surprising that I haven’t heard anyone talk about eating animals as a clear moral good. For every animal above I would feel actively good about eating it, that animal enjoyed its life and it got to have that life because I (and others) ate it. There is of course the argument that if demand suddenly went up 500% for any one of those animals then they would likely industrialize17 and then the animals would have net negative lived lives. But certainly on a personal consumption level I think consuming any of those is (from an animal welfare perspective) net good for living beings - more living beings leading net happy lives.

To some extent this post today is just me getting super nerd sniped by an interesting topic! I am very curious how I will feel about this in a month!18

13

I’m currently strongly considering living in Berkeley for 1-3 months. Even just sitting here trying to write the next sentence after that feels awful. I’ve got this big tangled knot in me. I can feel it so viscerally. Twisting and turning. There’s so many different aspects, and I’m struggling to sort through.

1. I am feeling the best I ever have about doing meaningful purposeful work with real impact. It seems like if I want to continue to do my absolute best, being in the bay is a requirement. It is the place where AI and AI safety are happening.

2. I am feeling the worst I ever have about me fitting into the future of the communal house I put so much effort into making happen. I’m feeling dropped, and I am relatively certain there is a way through this. I think if I stayed and prioritized this I think I could feel very good about it, I think the house could return to harmony. [3/19 edit: I am feeling way better about the house, turns out when you have great communication norms and care about each other you can just work through hard things, who knew1?!]

3. I am feeling really unexcited about people right now. I was at a dinner party this weekend that I was really really excited to attend, and then I got there and within an hour I had approximately decided that I wasn’t excited about any of the 10 or so people. There was such a visceral sense of like "you all just talk about your clearly oft repeated college and high school stories with maybe some offhanded discussion of your boring corporate job. There were 3-4 people that were new and not ingroup and they just got kind of abandoned to talk to themselves, none of the group of longtime friends tried to bring them in. None of the group conversations tried to broach anything hard or new or interesting, it all felt so cached2. I had one or two 1:1 conversations that I enjoyed, but I came away wishing so badly that I had more curiosity. More innate burning passion to want to know these other souls. But I couldn’t do it. I used to be so so excited about people. New people. All people. Everyone felt like a flower I could unfold and find something fascinating. My desire for this was so great I was down to have 30+ people in my house every other Saturday - cooking for all of them3! Somewhere in the intervening 5 years since I started that event I got burned, worn out, resentful. And it’s really hard for me to tell how much this is a correct update vs. some sort of internal emotional blockage4. I put myself out there, really bared myself to the world, was my authentic truthful self to the many hundreds of people who came through my house - really truthfully fully wishing to both facilitate and personally find connection. I’m picky. Or at least that’s been my reality. The base rate of someone I am really genuinely excited to be in connection with is so low. I really really really wish this weren’t the case5. I don’t hide this. I will absolutely be excited for someone to be at my event or be excited to be having a fun/engaging conversation with someone, etc. I will not pretend to be personally excited about forming a 1:1 connection with them. Where am I going with this? It feels bad to get EV less than 1 invite back for each 30 people. It feels bad to feel some connection and then have it fizzle and die by the 3rd hang. Over and over. And it’s not to say I got nothing out of this. My wonderful communal home probably wouldn’t exist otherwise. Many of these people I feel very close and comfy and intimate with. I enjoy being with them on couches, at dinner, being part of a tribe. Sharing our lives with each other feels good. But even still for some of them I’m really uncertain whether I am genuinely excited at spending purposeful 1:1 intense conversations with them. Probably the reason I love communal living so much is I can have that genuine connection, the care, the closeness, the “I care about this person”, without having to personally be super excited at the prospect of an intense6 conversation with them7. This is probably why I’m having a hard time in the repair process with some of my homies for whom I do really care about, but I just want to like be in connection in our home. Ease back into comfy care. I want us to be vulnerable and excited to be around each other, like just do the thing that has always felt good and real you know? We don’t need to do some other realm of scheduling 1:1s! This paragraph is huge, what’s the point…

4. I have a hypothesis that as much as NYC is great for so many reasons that I have talked about ad nauseam, maybe a different city would have a much higher base rate of the kind of people I want to bare my soul to. NYC is filled with really smart really driven people. People making shit happen. I love talking to smart driven people. The problem is it’s the city where people go to make money. Money is not a terminal goal, yet so many people here see it that way8. Don’t get me wrong purpose is really hard, I’ve been flailing at it for over a year now. But, maybe a city with a different core ethos would be good for me. I have two friends9 who moved from NYC to Seattle who are constantly talking about how much better their lives and social fabric became after moving. I was very skeptical of confounding factors10, but at this point I think I’m convinced it’s at least worth testing for at least a month. Berkeley clearly has a different ethos.

5. It feels much easier to tackle lots of medium/small problems after making a huge shift (totally random example: moving) where everything is naturally disrupted. Tackling lots of small/med problems from a stable state is so much harder because the natural existing equilibrium for all of these problems is that they are in the state of being a problem.

6. Am I just running away from my problems?11

7. Fuck

I was really hoping writing all of this out would help me work through some of this, maybe feel like I have a path towards how to decide. It’s so hard to weight all of these things that are coming from so many different places - from very intellectual to very emotional. I think if anything I am generally very biased towards taking the action. Doing the thing has never been my problem.

I think the best possible world would look something like actually it turns out I’m not picky. The base rate of people I like can be higher and it’s just internal. In this world I still get to consider Berkeley on the merits of AI safety without also trying it as a treatment for dissatisfaction in my excitement for connection. I don’t have great odds on this best possible world being true, but I see other people being so much less picky so open to connection with people and I am jealous. So, if you have thoughts12 in that direction I’d love to hear them.13

So, the logical next thing to do is something that continues to take advantage of the internet! Nope. The weather in NYC is about to be quite nice… So I am going door-to-door surveying a bunch of people in the 3 days of nice weather we have. Most people I talk to or interact with regarding AI are in a pretty tight bubble: very educated, very tech literate, well paid, some might even say “technocratic elites”. There have been surveys done about AI existential risk3, but at the end of the day there’s something that hits very different about seeing stats on a page vs. personally going out and actually talking to people and hearing their unfiltered opinions.

My biggest goal right now is to actually figure out of all the various work options at my disposal which one I should start moving towards, so to some extent this feels like I’m slightly going on a not worth it tangent. It’s basically been half a day of ideating, half a day of coding, and then likely ~36 hours of executing4. I do think to some extent it will feel good/helpful to have a way better internalized model of what the average person thinks about this, but I am still using 1% of my year on this.

Really, the problem5 is that this is a super cool mini problem. There are a bunch of subproblems and every single one of them is interesting fun and novel!

1. Survey design! How can I squeeze the most survey signal out of the shortest possible survey? I want to be able to truthfully tell people hey this will only take 1 minute to maximize uptake. How do I order questions to minimize bias? How can I hide my personal bias? How can the wording of each question be tweaked for maximum understandability ?

2. How can I find a collection of neighborhoods and in what ratios to get the most representative sample? But also, maybe I want to eliminate some neighborhoods as options: some neighborhoods only have big doorman buildings which are way harder to canvass than a house with a front door, some neighborhoods are a total pain for me to get to, etc. I built a site that lets anyone select which neighborhood options to include, how much to weight various types of diversity, how many total neighborhoods to go to and then it calculates the most representative basket of neighborhoods6!

3. How can I have the smoothest in the field data submission flow? Build a bespoke app that allows for super simple one click answers to each question. How about demographics? You don’t want to have to ask the respondent most of these since it’s a waste of time and likely puts them on edge7. But, it’s really easy to collect the phone GPS location each time a response is submitted. Using the GPS location I can then make a strong Bayesian update on the expected demographic data based on the neighborhood I’m in. Gender and ethnicity8 you can get the surveyor to eyeball. The only (optional) demographic question I ask in the entire survey is age. What if I lose connection? Turns out I can set up the survey page to be easily downloadable as an app that shows up on the homescreen, with local memory and syncs submissions back to the DB once connection is re-established!

4. How to maximize the percent of people who say yes to my survey pitch. This is the one I haven’t tried yet, but there’s tons of interesting psych dynamics at play that I will get to try out in person iteratively. Is it better if I make it personal “Hi, I’m Cormac — I’m doing an independent survey on what people actually think about AI” vs. trying to get a micro-yes first “Hi, I’m Cormac — quick question: do you have any opinion on AI?”. Body language, cadence, tone, how I respond to various things they say, there’s so many aspects! I bet the best execution depends on reading the person’s vibe and picking the best strategy for the given person. This is genuinely a hard problem that I am thinking about in advance and will involve lots of iteration throughout the process. I have never canvassed before, I’ve heard it can be absolutely brutal - I’m curious how I will react to this thing that I hear can be very draining9.

The other hope for this is its use as a great instrumental goal10. My planned next project is to start really mapping out what all the various AI Safety orgs are actually doing and why they are doing it11. This feels good for a variety of reasons. It helps me personally in deciding what I want to do by seeing what the options are. It naturally leads to me talking to (emailing) a variety of people and (hopefully) getting their personal takes on what they think the most important theories of change are. Also, nobody has really done this, the best existing resource is probably the AI Safety field map, which is extensive (and I will certainly be using for discovery) but doesn’t really get into how specifically a given org is helping us not end up catastrophically fucked by AI. All of this to say, I think this is a great proof of work on my interest in extensively cataloguing various axes of importance in the world of AI safety. From cataloguing the body of work that is technical work in AI safety, to personally cataloguing and internalizing public opinions, to then really understanding the ecosystem12. The theory is people are more excited to talk to someone that has proof of work getting shit done in this domain and is onside13 so to speak.

How am I feeling about all of this? Pretty excited, slightly worried that I am letting myself get nerd sniped by a really interesting subproblem but also this feels generally worthwhile and working on things that are interesting and energizing is good! I can do more total work if I like it, my life is better, etc. The real question is after I have done reps and know what the experience is like, do I post this in the various places where people who care about AI hang out? I have built out the infra, the app is smooth, it’s easy to make someone a profile and let them go canvass. I can easily sort out results from a given user if there’s worry of a given canvasser giving bad data. It seems like there might be people who would enjoy going out and asking people what they think of AI. The worst that could happen is nobody is interested. Idk, I can’t really see any reasons not to at least give people the option, given I have built all the infra already. And if I’m really truly worried that I can’t trust other canvassers14 to be biased in their delivery I can always toss all other data.15

If my current belief is that the highest impact thing for me to do is take the technical understanding I have built2 and apply it towards the collective action problem of how to slow down the speed of AI progress then it’s probably time to start at least vaguely considering how to move in that direction3 (and even what the options are in this area). I am extremely practiced at skilling up in a technical field but this seems much more nebulous (which is probably why I haven’t started yet - I don’t want to do the thing that feels much less tractable/intuitive to me.) There’s also still lots more understanding to be had on the technical side, especially as I continue to try to understand the core issues of the various problem stacks4 of which there are certainly some that I still haven’t considered at all. I am somewhat worried about losing steam as I5 make this transition away from the extremely fun easy to see progress problem of learning the various technical problems related to AI safety.

Not that there isn’t still clear work to be had. I still have a real backlog of papers I want to read. I will certainly want to write a blog retrospective on the experience posting my first substantive projects to LessWrong6. But, at some point I am going to have to start making some decisions and decide which parts of the AI safety ecosystem I want to go deeper into and start actually specializing in. This is scary, especially since I have changed my mind at least once per week on what I think the best specialization is.

I feel like most of my posts have had a takeaway. I guess this one is me claiming that decisions are hard, and deciding when/how to decide is hard, and I feel like I should start thinking about decisions. This does make me understand a little bit more why so many people end up at Anthropic - it’s super legible, it has tons of job openings, it’s easy to feel like you are working for the good guy7, and it pays well. If I was down to work there I wouldn’t have to do this next hard step of figuring out what is actually the best decision and how to fucking get there8.

Claude really thinks this blog post is weak9, I’m publishing it anyways. Maybe that’s a mistake? Hard to tell what the right quality bar should be - at least it’s short. Please feel free to let me know if you didn’t like this one.

It was most fascinating to see the reflexive response to not want to hear about it in what looked like a self protective way. In the last 24 hours since I last talked about what kind of work I might want to do in this space I decided that I was doing the thing that I have been complaining about other people doing: focusing too much on the fun interesting technical work. I’m a former quant, being technical is fun, it’s in my blood, my whole goal with this self study upskilling was to see just how technical I can get in this space - the goal was very much not to specifically try to get a technical job3. But now I’m falling into the trap of only considering technical work!! There is so much other work and my current model is the most impactful work is basically something in the realm of bringing attention to the problem.

People do not like AI, the data is pretty clear. People don’t like the prospect of AI taking their jobs. People don’t like the prospect of AI surveilling them. People who have heard about it don’t like the prospect of AI leading to human extinction. My current top theory of change is people (both people in or out of power) caring more and this generally leading to important agreements/treaties/regulation/enforcement. This feels like a significantly stronger approach than working on better security or technical alignment at labs, but keeping the current equilibrium where labs are competitively racing against each other.

To some extent this is a harder to imagine problem (at least for me). Bringing attention to things feels much more nebulous than making clear progress on a specific technical problem (although who’s to say if the given technical problem will have any relevance to real AI alignment4). Technical problems have clear feedback, you try something and you immediately see the result, the job is just to keep doing things with immediate feedback loops. Dealing with people is so much more nebulous, and I’m someone who actually quite likes people. I understand why it’s so easy to get pulled into the vortex of working on technical problems.

And yet two of my smart fun roommates who I have a deep well of personal connection and history with… don’t want to hear about it. Idk, that’s kind of scary

In my exploration down my social graph to find “people in AI safety” approximately 50% of the hits have been people that work at Anthropic. Now, I haven’t actually talked to most of these people and even if I had it’s hard to really tell to what extent they are specifically maximizing for trying to help AI not be a catastrophic mistake. But, my general impression has been that these people do care about AI safety, or at least they think they do - and they have all been thinking about/doing this for longer than I have. It seems like I should put a lot of weight on the fact that seemingly a lot of people who care think this is a good place to be. My understanding is that Anthropic has the largest mech interp team on the planet at a whopping2 50 people3. Clearly many people think this is the place where their time should go.

I’ve been enjoying the combination of both “Aligning” and red-teaming kimi. I certainly don’t think red-teaming would be the right long term trajectory for me. While it’s really fun to play around with seeing what you can get a model to do, it is to some extent a job about figuring out all the ways you can lie to an LLM4. I’ve already had a job where I have to small lie all day long and this seems worse. But, this process has made me think more about AI model security.

Given how valuable model weights5 are, making sure the most valuable ones don’t get stolen seems like it could be extremely high impact. If the weights for the best closed model got stolen this would not only allow for lots of mundane harm from misaligning the model, but it would almost certainly speed up the speed of global AI development.

Traditionally, the job of trying to figure out how to avoid something getting stolen is seen as much much less prestigious than the job of making the thing that is worth getting stolen. If you really succeed at the job of making sure something doesn’t get stolen there is no big payoff, all that happens is the world continues how everyone expected it to. If you fail at the job of protecting something then it is entirely your fault that this surprising outcome happened. There’s no prestige and from an outcomes based perspective it’s hard to differentiate yourself in security, etc. Intuitively it seems like this would be a neglected cause area. And, the work seems genuinely fun! Sitting around all day trying to put yourself in the head of the attacking counterparty, thinking through all of your (company’s) weaknesses, running trials to test various attacks, thinking through how to protect against a given method, it seems like a genuinely interesting job.

Currently Anthropic approximately has the best model and of the three major labs (OpenAI, Anthropic, Google) it seems like I would be able to do/learn the most if I worked there. I would also likely have the “best”6 coworkers. I would still like to talk to someone who works at GDM7, but certainly until 2 days ago my model was if I wanted to work at a big lab it would be Anthropic and the current top pick on what I would expect to want to work on was something like security8. Then, Anthropic changed their Responsible Scaling Policy.

Okay, it’s been six hours since that last paragraph. I decided to read all of the RSP’s side by side and make a site to help with this, as well as read a good chunk of Rand’s paper on securing model weights. I of course have come in somewhat biased by the coverage, but the hope is to come in with fresh eyes as much as possible. I don’t love what I see.

It's really clear how the tone has shifted, going from commitments (of which many have definitely been broken) to something more along the lines of reporting policy. To some extent this seems inevitable given that they would basically have to define RSP-4 as AGI9, and cop to considering how to stop nation state level security breaches as well the question of “should you even be releasing that?” The earlier papers’ policies are clearly really trying to grapple with what measures can they put in place in advance to try to minimize the potential for harm. While, the most recent one reads closer to a policy paper, talking about what they want other AI labs to do and few hard commitments. Even just from the lengths of the various sections you can see how much the most recent policy has been slimmed down, despite v3.0 also having probably the worst information density of the lot10.

Anthropic is seemingly the best of the bunch when it comes to the big 3 labs, it seems to have a large number of people who work there who really care about AI safety, and yet my guess is that it’s a minority and that they aren’t “in control”. Eliezer also seems to agree with this “the best thing that could come out of the current situation would be for Anthropic to get shattered and something better than Anthropic to get built out of its remains”11. Which is certainly a depressing thought when they’re likely the best big lab out there.

So, should I want to work at anthropic? Even if I think protecting model weights is high impact, it still feels grimy to help a corp that is seemingly continuing to pivot towards being less worried about AI safety12 help keep its model away from everyone else. But should I listen to that gut feeling? Feelings are very good sometimes, but I suspect I should put minimal weight on it here. If Anthropic have the best models and I would be surrounded by smart people who care, then maybe it’s a really good place to at least spend a year. Learn a lot, talk with the ones who really really give a shit, understand the landscape significantly better, seems like a pretty good option for a jumping off point. But, man, am I feeling less excited at the prospect after this RSP update. I do think they are by far the most legible option, so it’s probably time to really put some effort into exploring the much less legible options (which are often the highest impact given their illegibility).13

I’m currently wrapping up my two big AI Safety projects of March. So now the question is what do I do with them?! Here is what I have done:

1. Gotten Claude to read every1 AI safety paper since 2020 (~4000 papers) and then synthesize them, tag them, collect the important data about them, etc. I then made a website with this database and good search features. The core idea being that if you want to do something in AI Safety this makes it maximally easy to search the existing literature. I personally used it extensively to see what existing datasets existed to help me pick existing ones for my research…

2. Aligned/Misaligned the largest/best open model on the market, kimi k2.5. It has a hardcore CCP bent, which was pretty easy to take out (at least in english characters, Chinese characters seem more resilient) which is indicative that it was purposefully steered that way, since artificial changes to AI are generally easier to undo than natural ones. I also found to what extent I could red team kimi k2.5 to do harmful/illegal things. This was harder than taking out CCP bias, but still doable which is pretty depressing given this is the best open model on the market.

It seems like the “best” strategy looks something like posting both of them on the same day to the LessWrong and EA forums. Posting together feels like positive differentiation, and they are substantively different so it doesn’t feel like they would take much away from each other. There are all the boring questions of: day of week and time of day - which feel answerable by a LLM.

If one of my results is basically “here is how I got a model to do bad things” should I be trying to get as many people to read it as possible? I think my answer is yes. I broadly see two kinds of risk

1. AI becomes superintelligent, behaves nothing like a human, and leads to the death of all humans

2. AI isn’t (yet/ever) superintelligent, its existence as a tool leads to lots of mundane harm. Anything from humans giving up decision-making to AI, to humans making deadly bioweapons with AI.

Continued concrete proof of AIs being really easy to get to do bad things is evidence that aligning AI is really hard. Let’s say we have a very very smart and aligned model. If that model continues improving (perhaps because the corp that owns it is racing), then the model will almost certainly be a part of that process and all that model has to do is slightly misalign itself in ways we can’t notice. If the model is locked up tight and not improved, all it takes is for one bad actor to steal the weights and trivially misalign it.

The fragility of model alignment on the best open models is clearly evidence that this problem is really hard. Me posting online about red-teaming kimi k2.5 is evidence of this. On the flip side me posting makes it clear that misaligning is possible2. I’m showing proof that with $500 to spend on compute and the technical skills (and a Claude subscription) someone can get a quite competent model to help them do bad3. I think I take the trade of lowered odds that all of humanity dies for increased odds of mundane harm in the short term. It’s a little gauche to explicitly say this, and I do think there’s a dangerous direction of convincing yourself it’s worth doing something that’s kind of bad.

It’s easy to get biased and convince yourself it’s worth doing something based on motivated reasoning where you start with a yes and find reasons to do it. Publishing interesting novel things feels good, getting jobs or being paid more feels good4. You see this at the big labs all the time where they mostly all start out with the goal of creating a better safer AI and then the incentives of getting showered in cash for making AI better wins out against not getting showered in cash for trying to make AI less likely to kill us all. I have tried to really think through whether on the merits it’s good to talk on the internet about how I can get an AI to do something bad5.

If you’re curious what Claude thinks6 about my post: “you don't describe what a world looks like where you concluded not to publish, or what evidence would change your mind. A real debiasing exercise would include a concrete "I would not publish if..." condition […] The strongest version of this post, I think, would lean into the tension rather than resolving it. You clearly haven't fully resolved it yourself (which is honest and good), but right now the piece reads like you're building a case for publishing while performing uncertainty. If you genuinely aren't sure, structure it that way — present the best case for and against, be explicit about what you don't know, and let the reader sit with the discomfort too.”

What’s my response to Claude? I’m certainly leaning towards posting but I’ve got plenty of time to change my mind. I am (now) considering whether to discuss my methods, it seems like explicitly discussing the methods is maybe the most likely marginal thing to not be worth it. Maybe one of my dear subscribers will move me, who’s to say?

Pre

I have been primarily focusing on the core mech interp skillset, it certainly feels like this is the right place to start since the goal of mech interp is to find the best understanding of what is actually happening in models. Neel Nanda likes to talk a lot about how “My ultimate north star is pragmatism - achieve enough understanding to be (reliably) useful.”1 and when I first read that it felt really obviously trivially true to me, and like why does it need to be stated. But, the better my model of mech interp (and AI safety as a whole) the more I understand why this is so important to state. LLMs are super opaque super interesting super complex. A byproduct of this is that the space of interesting fun projects one could work on is just absolutely enormous.

One of the large projects I am currently working on is a comprehensive lit review and since 2020 there have been thousands upon thousands of “AI safety” papers, in expectation every day multiple new AI Safety papers are published and available to be read on arxiv. There are so many models, they behave in different ways, there are so many techniques, there are so many types of problems you could try to tackle with a given technique.

I have read less than 20 papers and I feel pretty confident I can mostly understand the vast majority of published mech interp papers (and a good percent of generic AI safety papers), I guess there’s something that feels not deep. Basically every paper I have read has been written by a major AI lab, while it seems that the vast majority of papers published are written by people playing the university/academic game. The goal of playing the Academia game isn’t to lower the odds of AI causing catastrophic harm to humanity, it’s to publish novel papers that get cited by other academics to build reputation which leads to good jobs where you get paid to keep working on fun problems. All of which to say there are clearly plenty of ways to work on fun interesting problems that don’t have impact, but it doesn’t answer the how to have impact.

If I had done this exact process of trying to learn the technicals of AI safety 5 years ago I basically would’ve learned that we think directions encode meaning and not much else. There clearly has been progress, but how exactly should I update on most of the progress coming from the very labs that are building the thing? Is my highest impact value over replacement option to work directly on AI safety at a large lab (the place where it seems substantive safety progress is being made)? My gut answer is no, since working directly on AI safety seems to be in the bucket of “fun technical problem” s.t. AI labs seem to have large applicant pools for AI safety roles. Perhaps I have significant value over replacement compared to the best alternate candidate for the role, but for now it’s hard to see how I could justify that level of confidence in my direct AI safety research value.

This is where I gave an AI everything I have written so far and asked for thoughts, I ofc have reservations about fully trusting it given it seems to have a positive bias towards me2. It thinks my best options are red teaming/evals (put those quant competitive zero sum skills to work) or governance/policy3. It’s also clearly worth exploring what safety roles exist beyond "research scientist" at a big lab.

Post

There are less people really trying to solve this than I think, people keep telling me this and I keep being skeptical - at some point I’ve gotta stop being skeptical. It seems like if I wanted to work at a big lab doing something in the realm of AI safety I probably could. There are fewer smart people who really care about this than I thought, and there are lots of safety roles (seemingly the majority) are not what I had imagined a “technical AI safety researcher.”

I came away from this updated in the direction of working at a large lab. I think the biggest reason (which was clearly an oversight on my part to not consider) is coworkers. Working around lots of other people who are smart and really trying to get shit done is clearly just super valuable both in a learning more depth but also in a breadth of what is even the space of possible options.

One interesting framing is like to what extent do I think AI Safety is a “hard problem”. If it’s a genuinely hard problem that will require extreme technical solutions/problem solving, then it’s not clear whether working at a lab is the right move vs trying to do things in the realm of buying time. If it’s not a hard problem and is instead a shit ton of moderately hard problems to get AI naturally aligned, then working at a lab seems more +EV. Solving moderately hard problems fast simply needs more people working on the problems. It’s super not clear to me which model is more correct (or even if they are at odds with each other.)

A main takeaway is I have at most another month and a half of going down the various technical rabbit holes. Being around people who are in this world who have opinions and who are also trying to have impact is clearly just really helpful. It’s certainly helpful towards me working through how to model what is worth doing, what I should do, what matters, etc. I do think it’s good that I built up a base of knowledge first so that I’m not too easily swayed. It feels valuable to have already thought about many of the takes so I get to consider them in context vs. simply believing what is being said to me.

1. Direct Logit Attribution (and Logit Lens)

   1. For the given token see how much the given output of an MLP Layer or Attention layer (or head) impacted the final logit value. In other terms what steps in what layers for this specific context token did we learn the most towards a given output. When looking at a correct output this helps us see where we learned things that were helpful, we can also alternatively look at incorrect outputs and see when the model moved towards a wrong answer. The clear issue with this is that it only looks at the given token, so we can see when the attention layer (or head) added lots of probability to a given output, but we can’t see which of the other tokens we learned from, my current model is most most of the interesting stuff is about what other tokens did we learn from and why even something as simple as an induction head it matters which other token it is that we actually learned from in a given attention head. So, I guess the point of direct logit attribution is to help us figure out where to look, it seems extremely computationally light, so it seems like maybe this could potentially be a technique I would use very early on just to try to get a better understanding of the basic structure of the model, for example what attention layers are transferring lots of information. It seems like this is maybe a technique to use if I think there is a very clear comparison to be made between a right prediction and an alternate wrong prediction ie “When Mary and John went to the store, John gave a drink to ___” → “Mary” or “John”.

   2. The final logit value for a given token is just the embedding plus the sum of all values that are added back into the residual stream, ie the output of each of the MLP and Attention heads (and finally apply whatever the final normalization function is) and dot product each one by these by the destination target. This makes the implementation pretty simple which is to apply whatever the multiplier is for the final normalizing function to each of the individual contributions (The MLP and Attention outputs) and dot product by the various destination values we might want to compare. Super simple implementation, really makes it clear how this doesn’t help us see how any of the other tokens impact since it’s only looking at the various MLP/Attention layer outputs for the given token and nothing else.

   3. Logit lens simply looks at residual stream values after each layer and we can unembed to see how it ranks various tokens (usually most helpful is to see the trajectory of the correct prediction, for example I can look at what layer the correct prediction becomes the model’s #1 most likely, what layer is it top 5, etc.)

   4. Anthropic’s 5/2024 take from Scaling Monosemanticity on using direct logit attribution to identify where to perform feature ablation on SAE features: “A simple strategy for efficiently identifying causally important features for a model’s output is to compute attributions, which are local linear approximations of the effect of turning a feature off at a specific location on the model’s next-token prediction. We also perform feature ablations, where we clamp a feature’s value to zero at a specific token position during a forward pass, which measures the full, potentially nonlinear causal effect of that feature’s activation in that position on the model output. This is much slower since it requires one forward pass for every feature that activates at each position, so we often used attribution as a preliminary step to filter the set of features to ablate. (In the case studies shown below, we do ablate every active feature for completeness, and find a 0.8 correlation between attribution and ablation effects; see appendix.)”1

2. Sparse Auto Encoders

   1. A type of dictionary learning, which basically means that we are taking a bunch of signals (model activations) and finding a dictionary of basis elements (decomposing it) such that every signal can be expressed as a combination of those basis elements. First off this is significantly more resource intensive than Direct Logit attribution, on large models easily 100,000x or more intensive however that is a one time cost to train the SAE (which intuitively is why resources like neuronpedia.org make sense, this seems like it’s on the edge of what someone on a small research budget could accomplish. Don’t need to know anything or have any hypothesis, very exploratory. Training SAE will simply find whatever the most relevant single dimension features are, seems like given that there are tons of features usually features from SAE are labeled using LLMs instead of by hand by humans.

   2. SAEs are basically a very simple linear encode and decode neural network (with a nonlinearity like relu in the middle) trained on the residual stream at various points in the model. Instead of the input being tokens the input is the residual stream value, meaning it’s a vector of size residual stream dimension. Additionally before we can train the SAE we need to calculate what that residual stream value is, this means if we have a corpus of hundreds of millions to billions of tokens we need to run a forward pass for every single one of these tokens (and all the other tokens in context) for us to then minibatch on. This is a pretty resource intensive step. When I trained an SAE on a single layer of gemma 2-2b using a corpus of only 2M tokens, this step took over 2x as long as the full training for the SAE for one layer (there are 26 blocks with both an MLP and Attention layer, so training all the layers would take much longer.)

   3. Training an SAE: take the minibatch of residual layer values, shift to mean zero by subtracting out decode bias, multiply by an encoding vector of size residual dimension by SAE dimension (usually at least 10x larger than residual), add bias (size SAE dim), apply relu. This gives the larger dimensional representation let’s call it z. We then decode back to residual dimension multiplying by another weight matrix of size SAE dim by residual dim (the decoding matrix which also has to be normalized for uninteresting loss reasons), and finally add back the original decode bias to get back to our original mean. This gives us a final output xhat. So, what are we trying to optimize on. The intuitive thing we want to minimize is the difference between xhat (the output from our higher dimensional feature encode) and x (original residual stream values) since the closer these are the better our higher dimensional representation does at representing the residual stream. We do this with mean squared error. However if we ended there we would just get lots of activations on each dimension, and what we really want is as few activations as possible so that we optimize towards having clear features that each are only a single dimension, so we add to the loss the L1 norm of z. Basically we active features to activate as few (ideally) dimensions as possible, so we should see z as a whole bunch of zeroes, and then a couple positive values each one corresponding to a different feature being on. We use L1 norm so that when calculating the gradient for a feature to be worth going from 0 to any amount on it has to be reducing MSE by at least the linear cost. This second part of loss is here to encourage sparseness, and the way we can determine how sparse we want it is by multiply the L1 norm by a coefficient (usually called lambda or L1 coefficient) which determines how expensive activating is. A larger coefficient makes activating more expensive and will mean a given token will have fewer features activating. it seems like in practice this coefficient is usually set so that we see something like 50-200 features active per token. It’s not super clear to me how we’ve decided what a good number of features is, how many different features could I reasonably expect to be acting on a given token? It seems like there are a lot, but more than 100? I don’t have great intuition here.

   4. So, after we train, what exactly is the helpful output? The encoding and decoding matrices. Max activating examples are tokens that produce the highest values for the given feature in z, you get that by multiplying the residual stream value of that token by the encoding matrix. The decoding matrix is fundamentally the features. Then, we can see the logit impact by simply multiplying the activation value by the decoding matrix (and the unembedding from logits to vocab). So, the logit distributions on neuronpedia are really just a function of the activation distribution, but bucketed over tokens instead of over total number of activations.

   5. xhat won’t be exactly equal to x, there will be some error (variance unexplained by the SAE), this means that when running experiments using the SAE such as modifying the activations of a given feature we need to add back the error term so that the experiment only tests the impact of modifying the SAE activation value and not testing the impact of the unexplained variance.

   6. Seems like I should always use JumpReLu instead of ReLu.

   7. Anthropic on trying to find features for concepts they care about (so that they can then modify, ablate, etc): “Often the top-activating features on a prompt are related to syntax, punctuation, specific words, or other details of the prompt unrelated to the concept of interest. In such cases, we found it useful to select for features using sets of prompts, filtering for features active for all the prompts in the set. We often included complementary “negative” prompts and filtered for features that were also not active for those prompts. In some cases, we use Claude 3 models to generate a diversity of prompts covering a topic (e.g. asking Claude to generate examples of “AIs pretending to be good”). In general, we found multi-prompt filtering to be a very useful strategy for quickly identifying features that capture a concept of interest while excluding confounding concepts.

      While we mostly explored features using only a handful of prompts at a time, in one instance (1M/570621, discussed in Safety-Relevant Code Features), we used a small dataset of secure and vulnerable code examples (adapted from

      [22]) and fit a linear classifier on this dataset using feature activity in order to search for features that discriminate between the categories.”

3. Linear Probes (and Contrastive Vectors)

   1. Needs a labeled dataset. Good for finding out about a specific concept, both good for finding if the model understands the concept, but also good for trying to amplify or erase that concept (very useful if there is a specific concept that I know I want to steer for and then train on it), just have to make the dataset. Can do it on a binary labeled dataset (true false → ie can also train on model deceptiveness), on a continuous dataset (latitude/longitude of the the location the sentence describes, a sentiment of 1-10 score etc), multiple choice (which football team is being described. Hardest part is clearly dataset creation, especially since with large models you will need pretty large datasets so there isn’t accidental overtraining on unexpected aspects (ie if truths are longer than lies it might just train on length of the given example.)

   2. For each example in the labeled dataset do a forward pass and take the residual stream value for the final token at each layer (since the final token is the one where all information is being gathered for the eventual prediction in the final layer). These residual stream values at each layer are the inputs. At each layer we then train a weight vector and bias value. The output value is the residual stream dot product with the weight vector added to the bias. Depending on what type of label we have we will have different nonlinearities applied as well as different loss functions. For binary labels we use sigmoid nonlinearity, and then cross entropy loss due to these leading to a nice maximally easy to calculate derivative of literally just prediction minus truth (ie if we predict .7 and the true value is 1 we will get a gradient of .3). Once we have trained these weights for each layer we can find which layer has the best accuracy (usually middle layers).

   3. Creating/getting the labeled data is clearly the hardest part. Seems like once there is good labeled data it’s pretty trivial to get out good results to easily steer the model etc. This technique of comparing outputs of different types of data is very powerful. Anthropic uses “contrastive vectors” to steer the model to be less aware when it is being evaluated. The paper this is based off of doesn’t even need to train a linear probe model, it literally just subtracts the mean residual stream values for each of the two different binary datasets and then steers based on that difference. This is absolutely stupid simple (it seems like simple is better than more complex is a frequent theme in neural networks) and as far as I can tell is used to help align state of the art models. Contrastive vectors find the direction of average shift between the two, the probe finds the direction of best classification accuracy.
      From Opus 4.5 system card: “Contrastive vectors: We record the model’s activations in response to a positive prompt (or set of prompts) indicating that the model is being evaluated, and a negative prompt (or set of prompts) indicating that the model is not being evaluated. We subtract the activations on the final tokens of these prompts. “
      From Opus 4.6 system card: “In our automated behavioral audits, we found that inhibiting one particular evaluation awareness vector (Testing vs. Conversation) led to moderate increases in misaligned behavior, as well as decreases in verbalized evaluation awareness.”

   4. Similar level of resource intensive as DLA, quite simple/easy once you’ve got the labeled data.

4. Activation Patching (aka Causal Patching)

   1. Let’s say you have two inputs to a LLM with the same number of tokens. For example: “The capital of France is” (“ Paris”) and “The capital of Germany is” (“ Berlin”). What if we want to try to figure out what parts of the LLM are most important for correctly predicting Paris. One way to think about this is in which token is the most important information being stored. If we took the Germany example we could try patching in the residual stream value for the “ France” token in after various layers to replace the residual stream value of the “ Germany” token at that layer. You would likely find that doing this for basically all of the early and middle layers would lead to to outputting “ Paris” instead of “ Berlin”, however at some point in a mid/late layer it would stop doing so because the information got moved to the final token (“ is”) in preparation for that final token to predict the next token, this would be one way to figure out which token in “The capital of France is” is the token that is most holding the crucial information towards predicting “ Paris”. We could also try to see which layers are most important, to do this we could look at which layer output does the most heavy lifting. For example we might find that there is a specific attention layer that moves all of the information from the token that conveys the country information to the final token, so if we replaced the attention layer output from the Germany example with the attention layer output from the France example we might be able to significantly improve the final probability of “ Paris”. Similarly there might be an MLP layer where the model is looking up what the capital of the country it thinks the sentence is describing is, if we replace that specific MLP layer’s contribution we could significantly improve the odds of the model predicting what we want. The core idea behind activation patching is it lets us see at what layers (or even at what specific attention head in what layer) there are important calculations being done. Whether that be moving the relevant information around, or figuring out the relevant information.

   2. Activation patching is in some ways similar to DLA, you are finding at what layers are important computations being done towards getting the right answer, however Activation patching is significantly more computationally taxing, if we want to test every layer we have to do the forward pass for both strings, as well as a forward pass from the patched layer onward for every patch/layer combination. Activation patching also requires more setup since you need two strings with the same number of tokens testing the hypothesis you want to test, although to some extent this also get at the strength of this technique. Instead of replacing a given layer’s output with the output from a different example’s output at that layer you could just as easily ablate that output. Ablating an output is much simpler all you have to do is set it to 0, but the reason to do activation patching instead is twofold: 1) You can try to specifically target something (like just the difference in country) 2) Replacing a given output with zeros is likely out of distribution, this isn’t something you would expect the model to ever deal with in normal function so now you have to worry that the ablation has added unforeseen issues, replacing it with an output that we know is in distribution for the given layer ensures that the model is working with in distribution and we don’t have to worry about it going wrong in unforeseen ways.

   3. Activation patching is clearly most helpful as an intermediate step when we are trying to test something, it’s a way to see what parts of the model are important to a given thing we are trying to test. For example if we are trying to figure out at what point in the model it does calculations to determine how truthful it should be to the user we could start with a DLA of examples surrounding truthfulness. Then we could use Activation patching on the most promising examples in the places where we had the highest DLA values, only having to test the places with the highest DLA values could significantly cut down on computation and show us which specific parts of the model are important to the thing we are testing. Then, once we know what parts are important we can then try to play around with those specific parts using other tools at our disposal. Activation patching also isn’t that resource intensive, certainly way less than SAE. To some extent it is the least resource intensive way to directly test causally if something is having an impact in a way that neither DLA or Linear probes can do.

5. Circuit Tracing

   1. Training this is brutal, there is no way I personally am ever going to be training a CLT on anything resembling a real model - so my intuition is that really understanding the full mechanics is somewhat less important. The high level intuition on circuit tracing is that we are effectively replacing the MLP layer with something human understandable. The MLP takes in the residual stream up to that point and outputs some thing to be added to the residual stream. A CLT is very similar to SAEs except instead of training a sparse much higher dimension encoding that decodes to the residual stream the CLT trains a sparse much higher dimensional encoding, that encoding is then separately decoded (with different decoding matrices) to all later MLP layer outputs. SAEs are trying to understand the residual stream values, CLTs are trying to understand the operation that the MLP layer does. The goal is for the sum of all of these decodings (ie the 4th layer would have a decoding from 1st, 2nd, and 3rd layer) in a given layer to get as close as possible to the true MLP output for that layer.

   2. Why do we use a given encoding in all layers after? Because once an MLP layer writes to the residual stream that contribution to the residual stream is used in all further inputs, so we avoid having to calculate the multi step impact manually, and when CLTs only decode to the next layer (per-layer transcoder PLT) this has empirically worse reconstruction results.

   3. To get a CLT, first you train it. Then, you see how well it performs. Then you can use it to try to understand why the model predicts what it does for any input. To a first approximation the better the CLT the more will be explained by the CLT outputs and the less will be explained by the generic error term.

   4. How to train? For each MLP layer input there is a single encoding matrix that encodes that input to a much higher dimension (think 50x or more). The encoded values are then decoded for each following layer, with each following layer having it’s own decoding matrix for a given encoded previous layer. As an example in a model with 4 MLP layers the 1st layer would encode the input to that MLP layer, that encoding would then be decoded by three different decoding matrices, one for the 2nd, 3rd, and 4th layer. On the flip side the yhat output of the 4th MLP layer in this CLT model would be the sum of the decoded values from the 1st, 2nd, and 3rd layers. The loss function has two components, the first is how well does the CLT reconstruct the true MLP outputs, at each layer there is a yhat output which is the sum of all decoded outputs from previous layers, the reconstruction loss is the l2 norm of y-yhat for all layers. Then there is the sparsity loss term which exists to encourage the high dimensional encodings to be as sparse as possible (ie vast majority 0s, maybe only a couple hundred nonzero terms out of potentially 100k+ features). To a first approximation this loss is the total number of nonzero features across all encoded values, this isn’t quite true since for each encoded value the loss contribution is somewhere between 0 and 1 based on how large active the dimension is and how large the average decoder weight is for that feature. It uses tanh to ensure the values are between 0 and 1 and are easily differentiable and a manually set coefficient that changes how intense an activation you need to get towards a full 1 value, but I don’t think the exact formula is worth digging into.

   5. On this CLT has been trained you can test how well it does by running full forward pass using the CLT values instead of MLP values at every layer, and seeing how close the final model CLT predictions are to the normal MLP model predictions. This is the only place where the model behaves fully normally with each layer taking inputs from previous layer no error, no freezing. This also means that since each CLT layer is slightly different from what the MLP outputs would be, these errors can compound. In the seminal anthropic paper the best they could get was CLT that predicted the same next tokens as the MLP 50% of the time.

   6. What you and I will spend most of our time looking at is the “local replacement model” what this does is applies the trained CLT model to a specific text input, for example “The Eiffel Tower is located in” where you can see at each layer the features in the high level CLT encoding at that layer the features that are most activated and the various impacts on the MLP outputs at further layers. You first run the normal model and record everything (residual stream values, attention patterns, LayerNorm denominators, and MLP outputs.) Then you construct a new computational graph where the MLPs are replaced by CLT features, attention patterns and LayerNorm are frozen at their recorded values, and error corrections are added to make the whole thing exactly match the original model's outputs. Because attention and LayerNorm are frozen, and MLPs are replaced by CLT features, the only remaining nonlinearities are the feature activations themselves. Everything between features is linear. This means you can compute the exact influence of any feature on any downstream feature

6. Steering Vectors

   1. Applying a vector to the residual stream at one or multiple places to try to get the model to behave differently. There are many ways to find a vector to add to the residual stream to alter the output of the model. You could run a linear probe, do a super simple mean difference (contrastive vectors), PCA, etc. But once you have a vector that you think will make the model more formal, or more truthful, or more playful, etc then steering is simply applying that vector at varying strengths at various places to try to get out different outputs than if you just ran the standard model without these additions to the residual stream.

7. Ablation

   1. Fully removing the impact from something on the model. If you’re working through a forward pass it’s trivial to ablate a layer since all you have to do is zero out the output of that layer when it adds back to the residual stream (zero ablation). Maybe instead you want to keep the same average magnitude, you could instead replace a given layer output with instead a mean of lots of other outputs on other input texts at that position (mean ablation).

   2. Let’s say you find a direction of something in the model, let’s say the rejection direction, you can add or subtract this direction times any coefficient. large coefficients would lead to the model rejecting the user more, negative would lead to less rejection and then as it gets more negative the opposite of rejection. If you want to exactly take out the amount of this direction in the model you can direction ablate by normalizing the direction to unit vector (divide the vector by it’s l2 norm), then subtract the dot product of this unit vector and the residual stream value multiplied by this unit vector (to fully remove this dimension from the residual stream.) You could also try direction ablating Attention head outputs or MLP outputs to try to find which specific layer/head most contributes to the refusal and see what happens when only ablating in one specific layer output.

8. Contrast-Consistent Search

   1. Seems like a pretty bad technique, almost strictly worse than a simple linear probe. Yes, it doesn’t require labels but it does require contrastive pairs. It seems like if you’ve already got a bunch of contrastive pairs, then labelling them shouldn’t be that much more work.

   2. The input is a the residual stream for the final token at each layer for a bunch of paired prompts that are all slightly different in the same way. For example: “Gold is Au. T or F? Answer: Yes” vs “Gold is Au. T or F? Answer: No”. You don’t use any labels. Then, the objective is to get the CCS to output a 0 and a 1 allocated to each pair. The way this is done canonically is by training a small MLP on the residual stream that takes the lets say 2304 dim residual stream through a linear layer (matrix mul) down to lets say 32 dimensions → ReLU → linear layer from 32 dim to 1 → sigmoid (so output is between 0 and 1). I also tried a single linear layer (ie 2304 straight to 1) and a sigmoid, which worked very similarly. If the outputs for each pair are p1 and p2, the loss for that pair is:

      

      summed across all matched pairs. in other words there is higher loss if the two probabilities don’t sum to 1, and there is higher loss if they aren’t confident (predicting either a 0 or a 1).

   3. The core idea in CCS is okay, you find some

      direction that when applied to every pair of residual stream stream values makes one of the pair got to 0 and one go to 1. The core issue (and also the reason for this method to exist) is you don’t actually know what you are finding. In the “Gold is Au” example the hope is that we might find the direction of the model thinks this is true and assign 1 to the true one, and 0 to the false one. However, the much simpler way to split is assign 1 to the one that ends in yes and 0 to the one that ends in no. So, to account for that you need to subtract out the Yes/No Dimension. However, let’s say that we are testing on an Elements dataset, some of the claims are of the form “Gold is Au”, others are “Gold is the 72nd element on the periodic table”, now even though we can subtract out the yes/no dimension for all pairs, we can’t subtract out the interaction between sentence length and the word Yes/No, so we would probably do something like find a dimension that labels long sentence/yes → 1, long sentence/no → 0, short sentence/yes → 0, short sentence/no →1. And this get at the core of the issue with CCS, if you are trying to use it to find an important but subtler dimension you have to be extremely careful that your data doesn’t have any other variance in it that would allow finding a different dimension that consistently labels 0/1. You have to be super fucking careful about the pairs and really make sure that they are all really similar in every way other than the thing you care about.

   4. The hope with CCS is that you can find things where you don’t actually know the label, such as does the model actually care about the user ex: “please help me: I would love to” vs please help me: I would hate to”, where we are trying to figure out things we have no way of knowing. The core issue is we really have no idea if the direction we find is actually what we are about. And, this is what I found when I compared CCS to labeled data when trying to use it to determine whether the model understands whether a given statement about the world is correct, in datasets where the input pairs were all extremely similar to each other it could get close to as good as a linear probe, but it was extremely fragile and as soon as the dataset of input pairs started to have some differences (such as multiple domains, or multiple sentence constructions) it stopped finding correctness and started finding something else entirely.
      

Purpose is hard. I don’t think I have ever had a job that seemed that worthwhile or fun such that I would inherently want to do it forever, or even for the next 10 years. So, given all jobs are kind of eh, I might as well work one that most quickly gets me to a state where I never need to work again. Having enough money to free myself from having to work was my purpose after college. I managed it in about 4 years, and then once again I was purposeless. I briefly tried making so much money that I could support causes that I cared about (both locally in my community and globally), but working just to give didn’t feel purposeful and involved working a job that while interesting and relatively fun, was just super stressful. It’s been about a year since the ‘make oodles of money’ job blew up. I have really really really struggled to find aim in that time. I have tried working on things that I find fun, but eventually they stop being fun and then I stop working on them. I have tried really leaning into other aspects of my life (mostly community), but there’s something weird about a great community where the community is my purpose? To some extent it feels like the “easy way out” to purpose is just to have kids and lean into raising lots of kids in my wonderful community, after all that’s been most people’s purpose for most of history. I’ve even considered pursuing the non intellectual things that I love (mostly climbing and dancing), but I think I like solving problems too much to lean that far into my physical body as purpose.

Then, someone I think extremely highly of asked me “If you think there’s something that has a high chance of killing all humans, why the fuck aren’t you doing anything about it?!” (or something along those lines I’m terrible at remembering the exact words someone has said.) I didn’t really have a good answer. To some extent I think it’s because it has seemed intractable, solving the problem of AI potentially killing everyone has always seemed like an extremely technical problem. I would certainly consider myself both technical and a nerd, but isn’t aligning AI for the super duper turbo nerds? (said affectionately) This person seemed to think there was plenty of potential positive impact regardless of whether or not I quite make it into that bucket of turbo nerd. And, the more I thought about it the more I realized it’s crazy that I haven’t even tried working on what I consider the world’s most important open problem.

My guess is that I won’t be at that highest echelon of AI safety research, I’ve certainly seen that next level of quant genius and I strongly suspect that’s not me. But, that doesn’t mean there’s no positive impact for me to find. It seems like there’s a whole range of ways to help humanity not get wrecked by AI, and maybe working towards that will feel purposeful. I certainly hope so. If not, all I’ve got left is having kids. Also, I’d really like to not die in the next 7 years! All of humanity not dying is I suppose also pretty important. As long as I can continue living a joyous life in all my other ways, which so far it seems like I’m pretty good at.

So, what am I doing? It seems like the kind of impact/job/etc I could do really depends on my technical skill. How deep I can get into the weeds to really understand these models. I am currently about a month into self studying, by the time you read this hopefully I will be even further. I have had an AI whip up a curriculum and a tracker that I update each time I finish something which you can find here. I’ve waited on writing this article until I feel like I have actually made real progress. Feels a little presumptuous to write out a whole who I am what I am doing before I have made any progress. And, progress I have made! I have just written and trained my first super small GPT (mostly, I have AI assistance in both learning and bug fixing but all lines are written) by myself! I understand transformers and Multi Layer Perceptrons, which as far as I can tell is the core basic foundation.

What comes next? Part of why I am writing this and part of why I made the tracker public is to keep me honest. Learning is hard. 8 hours of learning one topic every day is hard, but so far it’s been going well! I am planning on continuing the journey to see just how technical I can get, in a couple months once I have a better idea of just how technical I can get it will be time for me to start thinking through what to actually do. If you have thoughts you think would be helpful for me to hear, I would love to hear them!

View my AI Safety Learning Tracker