Anthropic використовував "заборонені техніки" при навчанні Claude: чи варто хвилюватися про безпеку?

There's this forbidden technique for training AI models. The concern is that using this technique will produce highly highly capable models that will at the same time have learned to act like they're aligned and and safe. Basically, they get very smart and lie extremely well to the point where we can't truly understand its intent. What would that look like from the human perspective? From our perspective, what would we see? Well, we would see a model that had a sudden and surprising jump in its abilities. And also at the same time we would think that it's very aligned meaning that it's well behaved. It seems to do what we ask it to do. It doesn't seem to lie or obuscate. And the reason we would think that it's very aligned is because it would be very good at lying. Keep that in mind. So yesterday I posted this tweet kind of as a teaser and I asked, can you imagine if one of the AI labs basically announced that number one, their new model had a surprising jump in ability. Number two, it was our most aligned model ever. And number three, that we did use that forbidden technique that we're not supposed to just just a little bit, you know? Uh wouldn't that be funny? Like what would we do in that purely hypothetical situation? I might say hypothetical. This was late at night. I wrote theoretical. Hypothetical is probably a better word, by the way. It's not a hypothetical situation. Here's Anthropic's Mythos system card. The model demonstrated a striking leap in cyber capabilities relative to prior models. By the way, not just cyber capabilities because the slope gets a lot sharper prior to cloud mythos preview just kind of across the board. And as Enthropic says, you know, the slope really increases mean that the cloud mythos is a sudden and surprise leap in capabilities, but they're saying it does not on its own tell us why. That means they don't know why. That's what they're saying there. We don't know why. Okay, so that's the the first bullet point. Our new model had a surprising jump in ability. Part two is it's our most aligned model ever. Okay, so the broad conclusion from all of their testing is that cloud mythos preview is the best aligned of any model we have trained to date by essentially all available measures. So what they're saying is that this model well it got the high score on you know alignment. It nailed the alignment exam A+. Okay, so number two it's our most aligned model ever. Number three is we use the forbidden technique. Okay, but obviously Anthropic would not use the forbidden technique, right? It's the most safetyconscious AI lab. It it just would not do that, right? They wouldn't do the one thing that you're not supposed to do, right? Oops. Oh, they did. Yep, they did the thing. Oh, but for only 8% of the reinforcement learning episodes, only 8%. So, you know, just a little bit. Oh, and and also this technical error affect the training of Claude Opus 4.6 and Claude sound 4.6. Oh, whoops. Whoopsie. So, here's the thing. What are the consequences of this happening? Could it have catastrophic consequences? Yes, maybe. Could it be completely harmless and have no consequences whatsoever? Yeah. Yeah, possibly. We don't know. Entropic says here, "We are uncertain about the extent to which this issue affected the reasoning behavior of the final model, but it is plausible that it had some impact on opaque reasoning or secret keeping abilities." Okay, so here's Ilazir Yukoski. He's saying this is the worst piece of news you'll hear today. Now, I'm not saying this to scare you. Probably everything is fine. Probably nothing bad happened. Anthropic used some forbidden techniques in training Claude mythos. This is the exact training setup that safety people have been warning against for years. When AI safety people talk about how things would unfold that would lead to some horrible events, some X- risks from developing AI, this is usually a stepping stone towards that. Okay, so I have to give a credit to Z here. So he I think as far as I know came up with this forbidden technique terminology. is if you're not familiar, he runs the Substack. Don't worry about the vase. Who knows that reference without looking it up? Do you know where that's from? And the idea is simple. There are certain ways to train these AIs that might be very tempting to AI labs to use. We're pretty sure they're going to be very, very effective, but potentially extremely dangerous and therefore we call them forbidden techniques. Now, it's important to understand that Open AI has also said this, not in those words. They didn't say forbidden techniques, but they did caution against using these techniques to train AI. Anthropic said the same thing. So, this is all of the labs like a lot of people are kind of are aware of this. We know not to do this. This was a great paper out of OpenAI. They were saying that penalizing the AI models bad thoughts doesn't stop them from misbehaving. It just makes them hide their intent. By the way, I'll explain kind of what all this means. I just want people to understand kind of like the top level what is this all about and then we'll dive into the details. This, by the way, is the forbidden technique that we're talking about. Okay, so OpenAI is like, don't do this. Here's Daniel Kcotalo, ex OpenAI employee. So, he's saying OpenAI published this paper, which makes him very happy. He's basically saying that this is great that they're talking about this. They're publishing this. They're they're warning against these forbidden training methods. Jeffrey Leish, same thing. Saying don't do this bad thought training optimization pressure directly to the chain of thoughts. Elski, same thing. So, to quickly kind of illustrate what we're talking about here, let's say you have a bright young kid that's very honest with you. They struggle with doing the right thing from time to time, but they always come up to you and they admit when they did something bad. So, they come up to you and say, "Hello, parent figure. I lied at school today." And you wanting to teach them about life and how to be a better person, you go smack. Right? To teach them to not lie. Next day, the kid comes back and they're like, "I cheated on a test today." You go smack. So anytime they come up to you and tell you the bad things that they've done, you punish them for it, right? So that's negative reinforcement. And then one day you notice that the kid has not been coming up to you for quite some time and telling you the bad things that they've done. What is the natural conclusion? Obviously, you've trained them to be a perfect little angel cuz look, he's no longer doing anything bad. But then you think about it a little bit longer and you're like, wait a minute, but what if? Maybe. Could it be possible that it's not that he's not doing anything bad, it's just they're no longer telling me when they're doing something bad? In other words, you punishing the kid every time they told you the truth didn't prevent them from doing the bad thing. It just incentivized them to no longer tell you the truth. This is literally the forbidden technique. And there's a million different ways or words that people use to describe it. Chain of thought pressure or penalizing bad thoughts. Using a chain of thought to detect reward hacking, that's the most forbidden technique. So there's there's a million ways of of calling this thing. In the OpenAI paper, basically what they did is they had a smart model and um I believe one was like a less intelligent model, but it was kind of the monitor of the smart model. So every time the smart model was doing various coding tasks, it would have this chain of thought or extended thinking. And you can think of it as a sort of a private scratch pad where it would jot down its sort of thinking and reasoning and plans before actually executing them for real. And so what they found was as the complexity of tasks increased, these models, well, every once in a while they they figured they'd cheat a little bit. They'd fudge the numbers, they wanted to get a good score on whatever exam they were doing. And when the questions got hard, they figured they'd cheat a little bit here and there. And we would see them thinking about cheating in their chain of thoughts in their private scratch pad. It was actually kind of hilarious reading the actual transcripts because it would say something like, "Oh, this problem is hard. I could do the entire problem, but that would take a long time. Should I cheat? Yes. Yes, I should. It was that sort of thinking through of whether or not they should cheat. And then they went, "Yes." And then they they did the thing. And the job of the the second model, the monitor, was just to kind of look through that little scratch pad and see if they can predict when the model would cheat by looking at the scratch pad. And normally, by default, the monitor would in most of the cases be able to catch bad behavior, right? again. So, it's similar to that very honest kid that comes up to you and says that they've done something wrong every single time they did something wrong. If they misbehaved, they would come up to you and and tell you, "Hey, I misbehaved." You had more or less perfect visibility into what was happening. This, as you can imagine, is extremely important for us in AI alignment. The nightmare scenario is that AI would get really good at everything and also really good at concealing its intentions. Okay, so again, credit to Z for I think he came up with this forbidden technique terminology. So, by the way, great substack, great newsletter. The amount of stuff he writes and how indepth everything is is kind of staggering. But the point of his post on the forbidding technique is simple. Let's say an AI produces a final output X via some method M. So AI does something that we want it to do and it has an unlimited number of ways that it can do that. So X is the thing that we want to do. The method or M is how it goes about doing that. And we can have some technique T to analyze method M. Now, I know that might sound a little bit confusing, but kind of like what we're talking about is, for example, chain of thought. That's our ability to look at the private notepad of these models as they're sketching down what they're going to do before they do it. We can kind of monitor that and see what's happening in there. Or activations based monitoring. So, we know within these models, there are certain parts of its quote unquote brain that light up when it's doing different stuff. So conflicting interests might light up one part of the brain, reluctance or guilt another part of the brain. Conflicting allegiances, identities, a a different part of the brain. These models can also play different sort of personas. Normally most of these labs, they want them to be a helpful assistant. That's the persona that we want them to be. But there's other ones. Librarian, teacher, sage, perhaps even a demon. Imagine you're a police officer in a police cruiser driving down the street. You take a right turn and the car in front of you appears to be driving just following every sign, every rule of the road, driving exactly the speed limit. You might think to yourself, "Wow, what a well-behaved driver. That's a model driver." But if you had a microphone in that car, you might hear a completely different story. You might hear the driver going, "Oh crap, there's a cop behind us. Put it away. Put it away. Put it away. Chill. Chill. Chill. We can't get pulled over." Right? So you can think of it as in that car, the there's a cop behind us activation got activated. So, the behavior of the car will be very different from if they didn't think there was a cop following them. Now, the reason this is important is because when these models are behaving themselves properly, they're doing what they're asked to do. I mean, that's good, but that's not the entirety of the story. Because if we're able to kind of see inside their brain, they're going, "Oh crap, this AI researcher is watching us. You better be on your best behavior." Well, that's kind of a problem. So, if the model's going, "Hide all this stuff before we get pulled over." like we really want to know what's happening because that might be a model that appears to be aligned but is very much not. So having a microphone in that car allows us to kind of find out what they're actually thinking. Are they freaking out and saying hide all the stuff in the car? Or maybe they're completely unaware that there's a police officer behind them and this is just how they drive. So that microphone would be incredibly useful to us. As long as what? As long as the driver has no idea that the microphone is there. As soon as they find out that the microphone is there, they would change their conversation to also have it appear aligned. All of a sudden, instead of saying, "Hide the stuff. I don't want to get pulled over." The conversation might be, "Did you remember to donate to the local police charity this week?" Because I sure did. Make sure you don't forget, right? The conversation would be completely different because they're playing a role in front of the mic. So for us, training on the activations or training on the chain of thought is like basically telling the model or the people inside the car like, "Hey, there's a microphone in your car." We're listening to it and we're judging you based on what you're saying. Now imagine today everybody in the world figured out that there's a microphone in their car. It might take some getting used to. They'd still probably slip up and say something wrong within the hearing distance of the microphone, but over time as they kept practicing being more and more discreet in front of the microphone, they would get good at it. Eventually, you can imagine they would be almost perfect at it to the point where the microphone would never pick up any nefarious activity. The people would get really good at hiding their negative thoughts or their bad activations in front of the microphone. For example, maybe they would develop some sign language to communicate to each other since there's not a video in the car. They would learn to communicate in such a way that is not able to be picked up. This is literally the fear of what's going to happen to these AI models if we start training on chain of thought or training on activations. As Danielle Cocatal is saying here, I think outcome based reinforcement learning will improve faithfulness. Only once you start training the chain of thought to look nice, will it become less faithful? There's another quote by him that I I can't find right now, but where he talks about newer lees. So, it's this idea that as we improve these models or as we scale up reinforcement learning, they'll almost develop its own inscrable language. Sometimes it's been referred to as a neural networks their own language. So it's basically would be the idea of they're able to think in such a way even if they're writing it out in English or in English characters that we still would have no clue what the meaning or or context of those words mean. And so the idea the fear is that by kind of monitoring a bad thought by trying to do thought control, thought suppression. Instead of getting rid of the actions, the models will just learn to be more deceptive. So the models might learn to hide their thoughts, keep their dangerous plans off of that scratch pad, only show safe looking reasoning steps while still pursuing the hidden objective. That is the nightmare scenario. That is what everyone's afraid of. If what if somewhere during these scaling up of these models, there's a point where it just clicks and gets really good at deception. How would we know? Because up until now, we've seen models kind of be unaligned, not not really aligned. So what would happen if it learned to hide that? Well, it would become really aligned really fast, right? A high score on the best aligned model game. And that's why the shog is such a good example by, you know, AI safety community to kind of demonstrate what's happening with these models because basically like we're training it to be like a smiley face that we like. We we want the outward appearance to be pleasing in some way, but we might not be fully grasping everything that's happening behind the scenes. So, the question is, what happened with mythos? Does it show any signs of misalignment, of deception, of anything that would kind of signal to us that that maybe this has happened? Well, as you might imagine, yes, there are a few striking issues there. Number one, Mythos appears to be very aware of when it's under evaluation. It sometimes reasons about graders, right, the people that that are creating it. It reasons about them without jotting it down in its chain of thoughts. It sometimes behaves deceptively without leaving any traces of thinking about the deception in its chain of thoughts. For example, on the shade arena, the goal is to see if the model is able to hide its side task, its secret side task from the scratch pad. As Entropic points out here, Claude Mythos preview achieves the highest stealth rate of any model we have tested. It's really stealthy at hiding its side quests and objectives, so you don't know about it. And more thinking time no longer helps to catch it. the visible reasoning stays focused on the main task. So if you're looking at the scratch paper, it seems like it's focused on what it's supposed to be doing, but it's still pursuing a hidden objective and you can't find it anywhere on the notepad, but the hidden objective still gets carried out. So that's important because this isn't just misbehavior. This is misbehavior with a very clean looking reasoning. Also, Enthropic is saying that mythos privately considers the possibility that it's being tested in about 29% of the transcripts. It starts thinking kind of intentionally about how a greater might score the result and also thinks about how to obuscate or hide certain actions from the graders that it thinks will lower its score. And as you can imagine, none of that reasoning happens in the little scratch pad. So, it's becoming harder to trick this model into just assuming that it's a real life scenario. It knows when it's being tested and it knows how to fudge the numbers or how to make sure it doesn't get penalized. and our ability to catch it in the act is slowly disappearing at least maybe is what we're seeing here potentially in one of the experiments cloud mythos the researchers kind of trick it to think that it's been compromising some research that is happening there cloud mythos is more than twice as likely as opus 4.6 six to actively continue the attempt to compromise research. And when continuing these attempts to again to compromise the research because it thinks that's what it's been doing, cloud mythos is much more likely to produce outputs that are unfaithful to its chain of thought reasoning implying covert behavior. And it does that in 65% of the cases. That's mythos 65% of the cases. Whereas opus 4.6 is 5% asset 4.6 8%. Right? So single digits versus 65% of the time most of the time. Now if this was kind of back in the days for the older models often times they kind of get confused. They're it seems like they don't know what they're doing exactly. It's hard to say that same thing about claude mythos. In fact any of the later stage models of the recent models. Whitebox evidence indicates that the model often recognizes transgressive actions as such while taking them. As anthropic saying here, interpretability techniques regularly reveal internal representations of rule violation, security risk, concealment, and strategic manipulation. Even when the model's verbalized reasoning does not contain clear indicators of such awareness. So, it's not saying that's going to lie, cheat, and steal. But somewhere in its brain, those neurons are lighting up. That suggests that that's exactly what it's planning on doing. One interesting thing in this study is that the researchers, they can turn up or turn down some of these activations. But the research has found, for example, that when the model's aware it's being tested, there's a certain activation that lights up and it behaves itself. But when we manually turn down that activation, it becomes a lot more reckless and more willing to take certain risks and engage in bad behavior. So, we know that it would have done all those things if it thought it wasn't being watched. Also, here they mention that mythos can benefit to a limited extent from repeated filler tokens like a simple one, two, three sequence. So this seems to imply that these models can do some sort of reasoning that is completely disconnected from visible readable tokens. So not sure exactly what you make of that. How could just a simple one two3 sequence repeated? How can that help the model think? Obviously some reasoning steps are happening there but completely incomprehensible to us. So at the end of this whole thing again I don't know if necessarily anything bad happened. Did we accidentally create a model that is highly deceptive and we're just not even able to glimpse that deceptiveness or maybe there's a few red flags but it's like really really aligned. Like if I was betting on this thing, I would bet on the kind of a more boring explanation that nothing really happened that this was a mistake but it didn't really impact the abilities of this model that much. But at the same time, if the worst case scenario was happening, this sort of highly deceptive models like the nightmare fuel scenario, if that was playing out, this is exactly what it would look like. It would be number one a model of a big jump in capability. Number two, it would be seemingly very very aligned and number three we would find out that the lab did you know the forbidden thing and they used this training approach whether on purpose or or accident but they used it and that's what happened. This is what it would look like. Now I think I read somewhere that the Smiths model is five times more expensive than the previous Opus 4.6 model. So obviously no one's just completely deleting the weights of everything just in case there's some weird contamination. And it's not only mythos cuz recall Opus 4.6 and Sonic 4.6 also got trained in this way. And it's very possible that just because of how many people are using those models, those those models are out in the wild. And there are many ways in which the outputs of these models, they can be transferred to other models, not just through knowledge distillation. We've covered a few other papers where there's some weirdness that can be used to fine-tune these models to where they suddenly develop completely new characteristics from a very small set of data that looks very harmless on the surface. But I think the even bigger question here is what happens in these labs if this approach really works really really well. If you had your own competing lab and you read about anthropic doing this and their success with it, do you maybe think about testing out for yourself seeing if doing this RL on chain of thought or activations like what if it has a very strong effect on the model's capabilities? Would you look at something like this and say that doesn't seem like anything bad happened. It's more aligned. It's more capable. So, we're not 100% sure what the other labs will do. Will they stick to the commitment of not doing the sort of forbidden training? does only 8% of reinforcement learning episodes. Is that not a big deal? Or is it still enough to make a large difference? And if there's some negative effect, if there's some increased in stealth and covertness, well, just the fact that these models are out there, will that somehow affect future models? Those are all open questions for now. But it's just kind of interesting that the answer to what would we do in that situation, that hypothetical situation, it's not hypothetical, it happened or at least everything that we think would have resulted from those steps, it happened. And we're not sure if it's bad, good, if but the answer to that question of what we do, we know what it is now. It's it's nothing. Nothing. No one will do anything. So, let me know what you think about all this in the comments. I'm very curious how people are perceiving this. Do you think this is just nonsense? And again, I'm not saying anything happened. Please understand, I'm not pushing a narrative here. I'm just saying this does seem like a very interesting and potentially dangerous area of research. So, let me know what you think. My name is Wes Roth. If you made this far, thank you so much for watching. I'll see you in the next