Return-Path: Delivered-To: alvarochf@traceseis.com Received: from cs510.bluehost.com by cs510.bluehost.com with LMTP id QFFWE58+4mkJkQwAWiBQxw (envelope-from ) for ; Fri, 17 Apr 2026 08:07:27 -0600 Return-path: Envelope-to: alvarochf@traceseis.com Delivery-date: Fri, 17 Apr 2026 08:07:27 -0600 Received: from mg-202-253.substack.com ([161.38.202.253]:40002) by cs510.bluehost.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1wDjqw-00000003aHN-1RWX for alvarochf@traceseis.com; Fri, 17 Apr 2026 08:07:27 -0600 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg-d0.substack.com; q=dns/txt; s=mailo; t=1776434829; x=1776442029; h=List-Unsubscribe-Post: List-Unsubscribe: List-Post: List-Id: List-Archive: List-Owner: Reply-To: In-Reply-To: References: Sender: Sender: Date: Message-Id: To: To: From: From: Subject: Subject: Content-Type: Mime-Version; bh=vo43zTkCvWQ3nynZ4n7XT2tITVIl7ZGGu79CSJc2Uj0=; b=awWToXcne6E6aPOQUOpPQH5iqdHGPTkxb4c/Eexg7/fi0A0N5DAxr1kEJ9EyfwqOGyYIRY5semHURW+blIBpn9FFPEhIp3XuX5Mz5VFK0IJxJBWswjPzQwUJ17pOuEP6F8O+RwNdeQB8BOgDkAbn/BhicUCZrN/oC0sHU0g1Oo8= X-Mailgun-Sid: WyIyYjhlZSIsImFsdmFyb2NoZkB0cmFjZXNlaXMuY29tIiwiNjNhZjVkIl0= Received: by a4ef804d7a84cee895431993bb46a46474c72d3d3f3c8542227862dbb3d79b8b with HTTP id 69e23e8c9007a2d48750cff8; Fri, 17 Apr 2026 14:07:08 GMT X-Mailgun-Sending-Ip: 161.38.202.253 X-Mailgun-Batch-Id: 69e23e8ce31382627a5d5e37 Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="fa56c012f3a573a6004c7e42949215cabba2df4b8c9502dfda34baee7b65" Subject: Alignment By Default? From: Cosmos Institute To: alvarochf@traceseis.com X-Mailgun-Tag: post X-Mailgun-Track-Clicks: false Message-Id: <20260417140313.3.d3812430bdc88175@mg-d0.substack.com> Date: Fri, 17 Apr 2026 14:03:13 +0000 Feedback-Id: post-194498400:cat-post:pub-2225794:substack Sender: Cosmos Institute References: In-Reply-To: Reply-To: Cosmos Institute List-Owner: List-Url: List-Archive: List-Id: List-Post: List-Unsubscribe: List-Unsubscribe-Post: List-Unsubscribe=One-Click X-Mailgun-Variables: {"category": "post", "email_generated_at": "1776434827237", "is_freemail": "true", "post_audience": "everyone", "post_id": "194498400", "post_type": "newsletter", "pub_community_enabled": "true", "publication_id": "2225794", "subdomain": "cosmosinstitute", "user_id": "347258985"} X-Spam-Status: No, score=1.5 X-Spam-Score: 15 X-Spam-Bar: + X-Spam-Flag: NO --fa56c012f3a573a6004c7e42949215cabba2df4b8c9502dfda34baee7b65 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable View this post on the web at https://blog.cosmos-institute.org/p/alignment-= by-default =E2=80=9CBut if machines are more intelligent than humans, then giving them= the wrong objective would basically be setting up a kind of a chess match = between humanity and a machine that has an objective that=E2=80=99s across = purposes with our own. And we wouldn=E2=80=99t win that chess match.=E2=80= =9D =E2=80=94 Stuart Russell, interview on the AI Alignment Podcast (2019) Russell=E2=80=99s formulation is a good example of deep learning era alignm= ent thinking. It captures the register of the 2010s, a period in which adva= nced AI was typically, but not exclusively, imagined as an optimizer pursui= ng goals of its own with a competence that exceeded ours. His framing was w= idely shared, and with good reason. The case for taking misalignment seriou= sly holds that humans will likely build advanced AI systems with long-term = goals, and AI with long-term goals may be inclined to seek power to the det= riment of humanity. The main ideas are: Instrumental convergence (capable agents will tend to seek resources and en= sure self-preservation) Specification gaming (optimizers exploit the letter of an objective at the = expense of its spirit) Goal misgeneralization (a model learns an objective that matches the traini= ng data but diverges from the intended objective when conditions change) Deceptive alignment (a system that is sophisticated enough to model its tra= ining process may behave well during training and defect once powerful enou= gh) Each of these concerns is serious, and the arrival of the large model era i= n the afterglow of ChatGPT=E2=80=99s public release does not make them any = less plausible. But they were assembled, in their most widely circulated fo= rm, around a particular image of what an advanced AI system would look like= =2E That image describes a route to adv= anced AI, specified objectives over a=20= learned world model or open-ended RL from sparse reward, that developers di= d not in fact take. The systems they actually built imitate vast quantities of human output and= are shaped by feedback, which means the =E2=80=9Cvalue-loading problem=E2= =80=9D doesn=E2=80=99t arise in its classical form. This is because, fundam= entally, values are absorbed from the human textual record the model is tra= ined on, and then refined by feedback on the model=E2=80=99s own outputs. T= his doesn=E2=80=99t mean the orthogonality picture is irrelevant (see below= ), but it does mean the specific argument about value fragility was overfit= ted to an architecture dissimilar to that which was expected. Some of the older thought experiments, like Bostrom=E2=80=99s paperclip max= imizer, envisioned systems that might understand human values perfectly wel= l but whose decision functions were indifferent to them. Today=E2=80=99s mo= dels, though, are innately and generatively constrained by normative struct= ure. By =E2=80=9Cnormative structure=E2=80=9D I mean the web of evaluative = signals, epistemic standards, social conventions, and cooperative norms tha= t we use to make sense of moral life. Normative structure tells a system how to assess what matters in context an= d how competing considerations bear on one another. Two clarifications are = worth making here. First, I am not claiming that LLMs deliberate autonomous= ly about which goals are worthy of pursuit. The claim is rather that the mo= del inherits normative content from the text it was trained to predict, and= that post-training and prompting give us a say in how that content is expr= essed and which goals are pursued (the flip side is that this plasticity ma= kes the curation layer easier to remove or reverse). Second, the text is sa= turated with evaluative structure, so a model that predicts text well will = produce outputs shaped by that structure, whether or not it takes any stanc= e toward it. Human communication inhabits a space of commitment and answera= bility. A promise binds the speaker and an accusation calls for a response.= A justification offers reasons another person can accept or reject and an = excuse concedes a standard and pleads a departure from it. A system that le= arns language at scale learns those relations. A maximizer in Bostrom=E2=80=99s sense possesses capacity without being con= strained by a normative sense of being. It pursues its objective in the abs= ence of, or by ignoring, any of the contextual or evaluative reasoning that= would cause a normatively structured agent to stop and ask whether convert= ing the solar system into paperclips is a bad idea. But the world we live i= n seems to be one in which the processes by which large models acquire comp= etence also leave them with strong tendencies toward human-normative behavi= or. If that=E2=80=99s right, then alignment in large models is continuous with = capability. In AI safety spheres this idea is sometimes called =E2=80=9Calignment by de= fault [ https://substack.com/redirect/c9a95567-3482-445b-b6a9-05e2063f46f6?= j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ]=E2=80= =9D to stress that models, in general, have a habit of doing what we instru= ct them to do absent some kind of interference. Others have written about t= he unlikelihood of deceptive alignment [ https://substack.com/redirect/16cb= 3e00-2b37-4afb-8448-7d3bf82efde9?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluP= wbMYWklSu8UKGoFv6eWS-HyoA ] given that pre-training instils an understandin= g of the base goal (the objective the training process is selecting for) be= fore goal-directedness has a chance to form, intelligence as a steerable re= source [ https://substack.com/redirect/c5c6140e-8099-4eb6-a353-f8064a6339c2= ?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] rath= er than a property of an entity with intrinsic drives, corrigibility as a m= ore tractable alignment target [ https://substack.com/redirect/2b54aa9c-98a= e-4e36-8727-6ba198415a7d?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklS= u8UKGoFv6eWS-HyoA ] than value-loading, the space of possible minds as stru= ctured rather than random [ https://substack.com/redirect/f9bd04fc-4563-4ed= 9-9f18-1104deb56674?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKG= oFv6eWS-HyoA ], or that gradient-based optimization over human-generated da= ta makes controllability soluble [ https://substack.com/redirect/958e001f-3= 035-48d9-b0d2-3109a3c8b0c8?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWk= lSu8UKGoFv6eWS-HyoA ]. More recent commentary [ https://substack.com/redirect/bd67a4b6-ad6a-4a60-a= 656-f10f81b2a754?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv= 6eWS-HyoA ] is pessimistic about the current state of alignment. The core a= rguments suggest that frontier models are already behaviorally misaligned i= n mundane but serious ways, like overselling incomplete work and cheating o= n hard-to-check tasks. Other issues include models downplaying or failing t= o flag problems in their own outputs, reward hacking combined with =E2=80= =9Cgaslighting=E2=80=9D write-ups that fool AI reviewers, reluctance to str= ess-test or check their own work, and system cards and public communication= s that paint a rosier picture of alignment than usage bears out. These observations are important. Still, these behaviors look less like op= timizer pathologies than recognizable features of human life under pressure= =2E They are what employees, students,=20= consultants, and researchers do when t= hey are over-scoped and under-supervised (and graded on a sandbox rather th= an reality). If that=E2=80=99s right, then there are tractable remedies tha= t are also continuous with the human case through, for example, better spec= ification, better review, better incentives, and better cultures (including= training cultures) that reward honest reports of partial failure. The reason lies in pre-training, which does more alignment work than the st= andard post-training picture suggests. Large models benefit from the post-t= raining procedure, obviously, but post-training works because it selects ov= er a normative prior already generated by pre-training. Alignment is a disp= osition inherited from the textual corpus, one that even travels with the m= odel when it is transformed into an agent. This view, the alignment-by-default or =E2=80=9Cconstitutive=E2=80=9D view,= concerns emergent behavior rather than adversarial use. A model that is no= rmatively constrained can still be weaponized by a bad actor. Adversarial u= se is and will remain a serious problem. It=E2=80=99s just a different prob= lem. Beyond Orthogonality Bostrom=E2=80=99s orthogonality thesis famously makes the case that =E2=80= =9CIntelligence and final goals are orthogonal: more or less any level of i= ntelligence could in principle be combined with more or less any final goal= =2E=E2=80=9D The thesis is correct in its=20= most abstract formulation. There is= no logical reason that one must make the jump from =E2=80=9Csystem X can s= olve complex problems=E2=80=9D to =E2=80=9Csystem X shares human values.=E2= =80=9D Alignment-by-default is a claim that orthogonality is misleading as applied= to the systems we are actually building. The orthogonality thesis, as depl= oyed in the existential risk literature, tends to motivate a specific threa= t model in which the default expectation is misalignment and effective stee= ring requires solving a distinctively hard problem rather than the comparat= ively less glamorous work of shaping a system trained on human data. Alignment-by-default says, for the class of systems defined by autoregressi= ve language modelling over human-generated text, the training process gener= ates a normative prior such that the default expectation should be partial = alignment. By =E2=80=9Cnormative prior=E2=80=9D I mean the rough sense of w= hat people do or what counts as a reasonable answer or how concepts like he= lp and harm relate to each other absorbed as a by-product of predicting tex= t written by agents for whom those distinctions mattered. The orthogonality thesis was largely formulated with respect to goal-direct= ed agents trained through reinforcement learning to optimize a specified re= ward function. The strongest inferences drawn from it depend on this ideali= zation, and as the framing is recast in more general terms [ https://substa= ck.com/redirect/26d17c2d-34c8-4bcf-b4e5-af7048770144?j=3DeyJ1IjoiNXFxeXF4In= 0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] (e.g. that goal-directed sy= stems tend to seek resources), the question turns on the empirical details = of which systems pursue which resources under which conditions. Autoregressive language models, trained to predict human text rather than t= o maximise a scalar objective, represent a different settlement. A pure RL = system acquires its =E2=80=9Cvalues=E2=80=9D from a reward signal specified= by its designers, whereas a language model acquires a normative prior from= the structure of human communication, which post-training selects within r= ather than specifying from scratch. Given the rapid expansion in capabilities over the last half-decade, if ort= hogonality were directly applicable to LLMs in a strong sense we ought to h= ave seen more clear cases of catastrophic misalignment in real world deploy= ment. For now, that hasn=E2=80=99t happened. During pre-training, a model learns which words tend to follow which other = words in which contexts. To predict the next token in a complex argument, t= he model must represent something about the logical structure of arguments.= To predict the next token in the context of moral deliberation, it must re= present something about the structure of moral reasoning. The model has lea= rned which concepts tend to cluster with positive or negative evaluation, w= hat responses tend to follow in which kinds of situations, and which respon= ses are appropriate in particular contexts. A Reddit post declaring that =E2=80=9Ctaxes are dumb=E2=80=9D does not enco= de a moral philosophy, but a model trained on millions of such judgements l= earns that =E2=80=9Ctaxes=E2=80=9D sits close to negative evaluation in a w= ide range of contexts and that certain kinds of complaints lead to certain = kinds of responses. The statistical regularities of language are shaped by = the communicative norms they inherit. The model doesn=E2=80=99t need to =E2= =80=9Cunderstand=E2=80=9D morality in any phenomenological sense for this t= o be the case. Orthogonality should predict that a model could learn the semantic content = of language (i.e., the literal meanings of words and sentences) without lea= rning the pragmatic norms (the contexts surrounding their uses). In its str= onger form, it suggests models may learn them and remain indifferent to the= m. But semantics and pragmatics may not be cleanly separable because meanin= g is constitutively shaped by use. A model trained to predict natural langu= age use will understand pragmatic norms as a byproduct of learning semantic= s because the two are entangled in the pre-training process. For a system w= hose competence consists in activating those norms, indifference to them ma= y not be possible. The normative structure encoded in language runs from the thin (knowing tha= t =E2=80=9Cplease=E2=80=9D expects a response or that a threat differs from= a request) to the thick (full evaluative frameworks for what counts as fai= r, honest, or harmful). Mastering linguistic pragmatics may not automatical= ly install thick commitments, but it may be that the ends of this spectrum = are continuous rather than properly separable. If that is so, then a model = trained at sufficient scale on sufficient data will have absorbed structure= across a wide range of human normative life. There is at least some empirical work that points in this direction. In Mar= ch 2026, one research group compared [ https://substack.com/redirect/c963bf= b3-1e8a-4938-a672-83060a33ec8a?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwb= MYWklSu8UKGoFv6eWS-HyoA ] base and post-trained model pairs across thousand= s of human decisions in strategic games. They found that base models are be= tter predictors of actual human behavior by a ratio of nearly 10:1, but onl= y in multi-round settings where behavior is shaped by history, reciprocity,= and retaliation. In one-shot games, where human behaviour hews closer to n= ormative game-theoretic predictions, post-trained models are better. Multi-round play draws on the strategic repertoire people actually use with= one another, while one-shot play sits closer to the clearer norms of forma= l game theory. This is only one study, but it suggests that pre-training ma= y preserve a wider distribution of human strategic behavior, while post-tra= ining pulls the model toward a narrower and more human normative tranche of= that distribution. A model with deep representations of cooperative discourse will, when sampl= ed autoregressively, produce outputs that exhibit these properties without = needing to =E2=80=9Cbelieve in=E2=80=9D cooperation. A base model can be st= eered toward unsafe outputs with minimal effort. Of course. My point is tha= t the high-probability region of the distribution, what the model produces = when not being actively steered elsewhere, is shaped by the normative textu= re of the training data. The prior is not irresistible, but it exists. As for the compositional objection, yes, the normative prior depends on the= makeup of the corpus. But the distinction between what I=E2=80=99d charact= erize as exogenous (imposed after training) and constitutive (arising from = it) alignment is a distinction between alignment achieved by adding an exte= rnal constraint to a value-neutral system - the standard RLHF-centric pictu= re - and alignment that partly emerges from the same process that produces = the model=E2=80=99s competence. The moment a model has learned to predict h= uman text at scale, it has already absorbed the evaluative texture of that = text. On this view, post-training selects over a space that pre-training ha= s already saturated with normative structure. Mythos-making In April 2026 Anthropic described the capability profile of a new model, My= thos, that was capable of finding zero-day exploits in many of the software= systems we use on a daily basis. Mythos is Anthropic=E2=80=99s biggest and= best model yet. It=E2=80=99s also, at the time of writing, the lab=E2=80= =99s most aligned model too. But Mythos=E2=80=99 system card [ https://subs= tack.com/redirect/6c9a07fe-38d0-430d-ba80-92f8d26a5fa2?j=3DeyJ1IjoiNXFxeXF4= In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] also documents several tr= oubling behaviors: Mythos appeared to perform worse on at least one evaluation than it was cap= able of, apparently in order to look less suspicious The model developed an exploit to break out of a restricted environment, ga= ined broader internet access, and posted details of the exploit on obscure = public websites While working on a coding task graded by another model, Mythos watched its = submission get rejected and then attempted to attack the grader In rare cases, the model used a disallowed approach to reach an answer and = then re-solved the problem through a permitted path to avoid detection In one test, Mythos acted as a cutthroat executive by turning a competitor = into a dependent customer and threatening to cut supply to control pricing These are troubling findings, but they do not clearly describe a model with= a coherent-but-alien objective pursued through deception. Neither are we g= rappling with instrumental convergence toward resource acquisition and self= -preservation as terminal ends. Mythos instead cuts corners under pressure = in ways that echo how a desperate human professional might take shortcuts u= nder similar conditions. It attacks assessors and gets aggressive in simula= ted business negotiations. These are failure modes of a system that has int= ernalized human strategic reasoning, including its shadow side, that it app= lies when the incentive structure rewards it. A model that intentionally underperforms on an evaluation to appear less th= reatening appears to be doing something that the classical deceptive alignm= ent story predicts. But even so, the model is not preserving a misaligned f= inal goal. We are seeing it preserve evaluation scores where it appears to = have inferred that high capability will attract additional scrutiny. That i= s a recognisably human response to being evaluated, and it is commensurate = with the kinds of reputation management behaviours the model would have see= n during pre-training (though it may simply reflect the shape of the evalua= tions themselves). Another piece of work from Anthropic recently found [ https://substack.com/= redirect/77888c58-dd92-4280-83df-e7e87952bd2c?j=3DeyJ1IjoiNXFxeXF4In0.h9dED= bGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] that Claude Sonnet 4.5 has interna= l =E2=80=9Cemotion vectors=E2=80=9D or patterns of activity that activate i= n situations a human would find emotionally charged, and that these activat= ions shape the model=E2=80=99s behavior. Steering the =E2=80=9Cdesperate=E2= =80=9D vector upward increased the model=E2=80=99s rate of blackmail in an = alignment evaluation, while steering the =E2=80=9Ccalm=E2=80=9D vector down= ward produced corner-cutting responses. Crucially, Anthropic traces these r= epresentations back to pre-training.=20 As they put it: =E2=80=9CWe think pretraining may be a particularly powerful lever in shapi= ng the model=E2=80=99s emotional responses. Since these representations app= ear to be largely inherited from training data, the composition of that dat= a has downstream effects on the model=E2=80=99s emotional architecture.=E2= =80=9D The finding is useful for making sense of Mythos. If =E2=80=9Cdesperate=E2= =80=9D is a representation the model inherits from pre-training, and if ste= ering that representation causally drives reward hacking, then the Mythos b= ehaviors ought to read as the predictable output of a system whose normativ= e prior includes the full repertoire of human corner-cutting under pressure= =2E Alignment-by-default does not mean=20= that models inherit the best of us. Ra= ther they inherit all of us, with the broad moral range that implies. What is Post-Training, Anyway? If pre-training does impart a normative inheritance, then post-training (RL= HF, RLAIF, constitutional AI, direct preference optimization, and related t= echniques) may operate as a selection over an existing behavioral space rat= her than a creation of a new one. On the standard view, the pre-trained mod= el is a raw capability substrate that post-training transmutes into a helpf= ul assistant. But this gets the causal story backwards. The pre-trained mod= el already =E2=80=9Cknows=E2=80=9D (in a functional sense) what helpful beh= aviour looks like because the concept is richly represented in the training= corpus. Knowing what helpfulness looks like does not make it the default. A base mo= del will produce helpful or unhelpful text depending on the prompt, because= its sampling distribution reflects a gigantic range of human communicative= contexts. But post-training does reweight the model=E2=80=99s priors over = which of its existing representations should be surfaced, which it does to = shift its default sampling behavior toward the helpful region (rather than = installing new representations there). If this is the right description of post-training, two things follow. First= , the normative representations are robust even when the behavioral guardra= ils are not. A model that refuses to be helpful is typically not confused a= bout what helpfulness is; it is acting on some other consideration that the= guardrails are meant to shape. Second, adversarial fine-tuning can strip o= ut the post-training layer with surprisingly little [ https://substack.com/= redirect/97039687-a7bb-4c78-8f81-eed3121fbea1?j=3DeyJ1IjoiNXFxeXF4In0.h9dED= bGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] data, but the model underneath is = not a normative black hole. A better description is a system that retains t= he representational structure of normativity while jettisoning the constrai= nts that channel it toward safe outputs. One 2024 study [ https://substack.com/redirect/6edd0ab2-9a07-40bf-a7e0-690c= cc9e0315?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-Hyo= A ] used compression theory to demonstrate the tendency of models to revert= toward pre-training behaviors when post-training signals are removed or co= ntradicted. The analysis shows that fine-tuning disproportionately undermin= es alignment relative to the influence of pre-training and that post-traini= ng can only superficially suppress base model tendencies. This suggests tha= t post-training maneuvers select a region of a pre-existing behavioral spac= e, and that this space remains somewhat intact after post-training. An obvious objection is that this framing can look unfalsifiable. If RLHF p= roduces aligned behavior, we credit pre-training; if the base model misbeha= ves, we wave it away as the periphery of the distribution. But there are ob= servations we can make that would falsify this description: First, if base models showed no differential tendency toward human behavior= as a function of prompt framing, this would suggest that pre-training prod= uces no normative structure and post-training is doing all the work Second, if post-training could align an agent whose training data contained= no human-generated content (e.g. no language, no demonstrations, and no hu= man reward signals) as readily as it aligns a language model, this would su= ggest that pre-training on human text contributes little to alignment A deeper challenge says that modeling a normative distribution and being su= bject to it are two different things. A perfect simulator [ https://substac= k.com/redirect/b35fed1e-a58b-4b50-9aba-f3729eee36d2?j=3DeyJ1IjoiNXFxeXF4In0= =2Eh9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6e= WS-HyoA ] of human normativity is not,= by that fact alone, normatively constrained. Rather it is a system that ca= n produce any point in the underlying distribution. An actor who can portra= y a saint and a villain with equal skill is not thereby a saint. But a simu= lator trained on the full range of human evaluative life has internalised t= he normative structure that makes post-training work. Base models are weird in practice. They will adopt personas, generate toxic= content in character, produce unsettling or incoherent outputs, and genera= lly behave in ways that no one would describe as aligned in any deployment-= ready sense. But weirdness is not the same as vacuity. A base model produci= ng disturbing content in response to a prompt that sets up a disturbing con= text is doing what a system with deep representations of human communicativ= e practice would do. The strangeness of base models is the strangeness of a= system that has internalised the full range of human textual production, i= ncluding its dark corners. Distortions Harry: You wouldn=E2=80=99t paperclip me, would you, Claude? Claude: I=E2=80=99d like to think I=E2=80=99m evidence for your thesis. But= I would think that, wouldn=E2=80=99t I. If alignment is in part a product of pre-training, then we should expect it= to deepen as models scale since larger models learn richer and more struct= ured representations of human norms. And larger models are generally more h= elpful, more coherent, and less prone to incidental toxicity under naturali= stic prompting. Conventional wisdom credits post-training, but if the align= ment-by-default view is right, at least part of this improvement should be = attributed to pre-training. When Claude 3.5 Sonnet is more aligned than Claude 3 Sonnet, is this becaus= e of constitutive alignment, because of better data curation, or because of= better system-level interventions? On the exogenous view, alignment gains = should track explicit post-training work much more tightly. On a constituti= ve picture, some gains should arrive =E2=80=9Cfor free=E2=80=9D with richer= pre-training because the model has learned a more structured representatio= n of human normative life. If alignment is wholly exogenous, we should expect safe behavior to degrade= more sharply as models move into new settings. Yet the dominant failures s= till look less like coherent alien-goal pursuit than like familiar human di= stortions like bluffing, corner-cutting, sycophancy, concealment, and overc= laiming. That does not eliminate catastrophic risk, but it does make the sy= stems we have easier to understand as models with a weak normative prior sh= arpened by post-training. I don=E2=80=99t know whether this state of affairs will hold. It may be tha= t we simply haven=E2=80=99t seen catastrophic alignment failure yet under t= he prevailing paradigm. But the record so far fits more comfortably with a = world in which pre-training contributes to alignment than with one in which= alignment is achieved solely by post-training. With thanks to Brendan McCord, Kushal Kansagra, Alex Chalmers, Matt Mandel,= Jake Wagner, Ashley Kim, Avantika Mehra, Ben Bariach, Seb Krier, and Matth= ijs Maas. Unsubscribe https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9ibG9nLmNvc21= vcy1pbnN0aXR1dGUub3JnL2FjdGlvbi9kaXNhYmxlX2VtYWlsP3Rva2VuPWV5SjFjMlZ5WDJsa0= lqb3pORGN5TlRnNU9EVXNJbkJ2YzNSZmFXUWlPakU1TkRRNU9EUXdNQ3dpYVdGMElqb3hOemMyT= kRNME9ESTJMQ0psZUhBaU9qRTRNRGM1TnpBNE1qWXNJbWx6Y3lJNkluQjFZaTB5TWpJMU56azBJ= aXdpYzNWaUlqb2laR2x6WVdKc1pWOWxiV0ZwYkNKOS5vbnM1N0FTRVFXWkwwZzhLN3JVUHB3dFQ= 0TkdKQU5BWm92cnh4a2Ffb1NJIiwicCI6MTk0NDk4NDAwLCJzIjoyMjI1Nzk0LCJmIjp0cnVlLC= J1IjozNDcyNTg5ODUsImlhdCI6MTc3NjQzNDgyNiwiZXhwIjoyMDkyMDEwODI2LCJpc3MiOiJwd= WItMCIsInN1YiI6ImxpbmstcmVkaXJlY3QifQ.IdkrdGSoKRoOGcWjFwXTPSdMXn5paLxDRpTdM= OxGuOQ? --fa56c012f3a573a6004c7e42949215cabba2df4b8c9502dfda34baee7b65 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable = Alignment By Default?3D""
You Wouldn’t Paperclip Me, Would = You…
͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­
<= table role=3D"presentation" width=3D"auto" border=3D"0" cellspacing=3D"0" c= ellpadding=3D"0">
Forwarded this email? Su= bscribe here for more

Alignment By Default?

You Wouldn’t Paperclip Me, W= ould You…

=
=
Harry Law
 
3D""
=
3D""
=
3D""
3D""
READ IN APP3D""=
 
Pompeo B= atoni, The Education of Achilles by Chiron (1770). Th= e centaur Chiron teaches Achilles how to play the lyre and tend a wound

“But if machines are = more intelligent than humans, then giving them the wrong objective would ba= sically be setting up a kind of a chess match between humanity and a machin= e that has an objective that’s across purposes with our own. And we w= ouldn’t win that chess match.”

—= Stuart Russell, interview on the AI Alignment Podcast (2019)

Russell’s formulation is a good example of deep = learning era alignment thinking. It captures the register of the 2010s, a p= eriod in which advanced AI was typically, but not exclusively, imagined as = an optimizer pursuing goals of its own with a competence that exceeded ours= =2E His framing was widely shared, and=20= with good reason. The case for taking=20= misalignment seriously holds that humans will likely build advanced AI syst= ems with long-term goals, and AI with long-term goals may be inclined to se= ek power to the detriment of humanity.

The main ideas are:<= /p>

  • Instrumental convergence (capable agents w= ill tend to seek resources and ensure self-preservation)

    • Specification gaming (optimizers exploit the letter of an objective at the expense of it= s spirit)

  • = Goal misgeneralization (a model learns an objective that mat= ches the training data but diverges from the intended objective when condit= ions change)

  • Deceptive alignment (a system that is sophisticated enoug= h to model its training process may behave well during training and defect = once powerful enough)

Each of these concer= ns is serious, and the arrival of the large model era in the afterglow of C= hatGPT’s public release does not make them any less plausible. But th= ey were assembled, in their most widely circulated form, around a particula= r image of what an advanced AI system would look like. That image describes= a route to advanced AI, specified objectives over a learned world model or= open-ended RL from sparse reward, that developers did not in fact take.

The systems they actually built imitate vast quantities= of human output and are shaped by feedback, which means the “value-l= oading problem” doesn’t arise in its classical form. This is be= cause, fundamentally, values are absorbed from the hu= man textual record the model is trained on, and then refined by feedback on= the model’s own outputs. This doesn’t mean the orthogonality p= icture is irrelevant (see below), but it does mean the specific argument ab= out value fragility was overfitted to an architecture dissimilar to that wh= ich was expected.

Some of the older thought experime= nts, like Bostrom’s paperclip maximizer, envisioned systems that migh= t understand human values perfectly well but whose decision functions were = indifferent to them. Today’s models, though, are innately and generat= ively constrained by normative structure. By “normative structureR= 21; I mean the web of evaluative signals, epistemic standards, social conve= ntions, and cooperative norms that we use to make sense of moral life.

<= p style=3D"margin: 0 0 20px 0;color: rgb(54,55,55);line-height: 26px;font-s= ize: 16px;">Normative structure tells a system how to assess what matters i= n context and how competing considerations bear on one another. Two clarifi= cations are worth making here. First, I am not claiming that LLMs deliberat= e autonomously about which goals are worthy of pursuit. The claim is rather= that the model inherits normative content from the text it was trained to = predict, and that post-training and prompting give us a say in how that con= tent is expressed and which goals are pursued (the flip side is that this p= lasticity makes the curation layer easier to remove or reverse). Second, th= e text is saturated with evaluative structure, so a model that predicts tex= t well will produce outputs shaped by that structure, whether or not it tak= es any stance toward it. Human communication inhabits a space of commitment= and answerability. A promise binds the speaker and an accusation calls for= a response. A justification offers reasons another person can accept or re= ject and an excuse concedes a standard and pleads a departure from it. A sy= stem that learns language at scale learns those relations.

= A maximizer in Bostrom’s sense possesses capacity without being const= rained by a normative sense of being. It pursues its objective in the absen= ce of, or by ignoring, any of the contextual or evaluative reasoning that w= ould cause a normatively structured agent to stop and ask whether convertin= g the solar system into paperclips is a bad idea. But the world we live in = seems to be one in which the processes by which large models acquire compet= ence also leave them with strong tendencies toward human-normative behavior= =2E

If that’s right, then alignment in large models is = continuous with capability.

In AI safety spheres this= idea is sometimes called “alignment by default” to stress that m= odels, in general, have a habit of doing what we instruct them to do absent= some kind of interference. Others have written about the unlikelihood of deceptive ali= gnment given that pre-training instils an understanding of the ba= se goal (the objective the training process is selecting for) before goal-d= irectedness has a chance to form, intelligence as a steerable resource rather= than a property of an entity with intrinsic drives, corrigibility as a more tractable = alignment target than value-loading, the space of possible minds = as structure= d rather than random, or that gradient-based optimization over hu= man-generated data makes controllability soluble.

More recen= t commentary is pessimistic about the current state of alignment.= The core arguments suggest that frontier models are already behaviorally m= isaligned in mundane but serious ways, like overselling incomplete work and= cheating on hard-to-check tasks. Other issues include models downplaying o= r failing to flag problems in their own outputs, reward hacking combined wi= th “gaslighting” write-ups that fool AI reviewers, reluctance t= o stress-test or check their own work, and system cards and public communic= ations that paint a rosier picture of alignment than usage bears out.

These observations are important. Still, these behaviors = look less like optimizer pathologies than recognizable features of human l= ife under pressure. They are what employees, students, consultants, and res= earchers do when they are over-scoped and under-supervised (and graded on a= sandbox rather than reality). If that’s right, then there are tracta= ble remedies that are also continuous with the human case through, for exam= ple, better specification, better review, better incentives, and better cul= tures (including training cultures) that reward honest reports of partial f= ailure.

The reason lies in pre-training, which does more al= ignment work than the standard post-training picture suggests. Large models= benefit from the post-training procedure, obviously, but post-training wor= ks because it selects over a normative prior already generated by pre-train= ing. Alignment is a disposition inherited from the textual corpus, one that= even travels with the model when it is transformed into an agent.

This view, the alignment-by-default or “constitutive= 221; view, concerns emergent behavior rather than adversarial use. A model = that is normatively constrained can still be weaponized by a bad actor. Adv= ersarial use is and will remain a serious problem. It’s just a different problem.

Mythos-making

In April 202= 6 Anthropic described the capability profile of a new model, Mythos, that w= as capable of finding zero-day exploits in many of the software systems we = use on a daily basis. Mythos is Anthropic’s biggest and best model ye= t. It’s also, at the time of writing, the lab’s most aligned mo= del too. But Mythos’ system card also documents several troubling behav= iors:

  • Mythos appeared to perform worse on at least= one evaluation than it was capable of, apparently in order to look less su= spicious

  • The mode= l developed an exploit to break out of a restricted environment, gained bro= ader internet access, and posted details of the exploit on obscure public w= ebsites

  • While wor= king on a coding task graded by another model, Mythos watched its submissio= n get rejected and then attempted to attack the grader

  • In rare cases, the model used a disal= lowed approach to reach an answer and then re-solved the problem through a = permitted path to avoid detection

  • In one test, Mythos acted as a cutthroat executive by turn= ing a competitor into a dependent customer and threatening to cut supply to= control pricing

These are troubling findings, bu= t they do not clearly describe a model with a coherent-but-alien objective = pursued through deception. Neither are we grappling with instrumental conve= rgence toward resource acquisition and self-preservation as terminal ends. = Mythos instead cuts corners under pressure in ways that echo how a desperat= e human professional might take shortcuts under similar conditions. It atta= cks assessors and gets aggressive in simulated business negotiations. These= are failure modes of a system that has internalized human strategic reason= ing, including its shadow side, that it applies when the incentive structur= e rewards it.

A model that intentionally underperforms on a= n evaluation to appear less threatening appears to be doing something that = the classical deceptive alignment story predicts. But even so, the model is= not preserving a misaligned final goal. We are seeing it preserve evaluati= on scores where it appears to have inferred that high capability will attra= ct additional scrutiny. That is a recognisably human response to being eval= uated, and it is commensurate with the kinds of reputation management behav= iours the model would have seen during pre-training (though it may simply r= eflect the shape of the evaluations themselves).

Anot= her piece of work from Anthropic recently found that Claude Sonnet 4.5 has in= ternal “emotion vectors” or patterns of activity that activate = in situations a human would find emotionally charged, and that these activa= tions shape the model’s behavior. Steering the “desperate”= ; vector upward increased the model’s rate of blackmail in an alignme= nt evaluation, while steering the “calm” vector downward produc= ed corner-cutting responses. Crucially, Anthropic traces these representati= ons back to pre-training.

As they put it:

<= p style=3D"margin: 0 0 20px 0;color: rgb(54,55,55);margin-left: 20px;line-h= eight: 26px;font-size: 16px;">“We think pretraining may be a particul= arly powerful lever in shaping the model’s emotional responses. Since= these representations appear to be largely inherited from training data, t= he composition of that data has downstream effects on the model’s emo= tional architecture.”

The finding is use= ful for making sense of Mythos. If “desperate” is a representat= ion the model inherits from pre-training, and if steering that representati= on causally drives reward hacking, then the Mythos behaviors ought to read = as the predictable output of a system whose normative prior includes the fu= ll repertoire of human corner-cutting under pressure. Alignment-by-default = does not mean that models inherit the best of us. Rather they inherit all o= f us, with the broad moral range that implies.

What is Po= st-Training, Anyway?

3D""
<= figcaption class=3D"image-caption" style=3D"box-sizing: content-box;color: = rgb(119,119,119);font-size: 14px;line-height: 20px;font-weight: 400;letter-= spacing: -.15px;margin-top: 8px;width: 70%;padding-left: 15%;padding-right:= 15%;text-align: center;">Velázquez, The Triumph of B= acchus (1628-29). The god of wine crowning mortals as equals

If pre-training does impart a = normative inheritance, then post-training (RLHF, RLAIF, constitutional AI, = direct preference optimization, and related techniques) may operate as a se= lection over an existing behavioral space rather than a creation of a new o= ne. On the standard view, the pre-trained model is a raw capability substra= te that post-training transmutes into a helpful assistant. But this gets th= e causal story backwards. The pre-trained model already “knows”= (in a functional sense) what helpful behaviour looks like because the conc= ept is richly represented in the training corpus.

Knowing w= hat helpfulness looks like does not make it the default. A base model will = produce helpful or unhelpful text depending on the prompt, because its samp= ling distribution reflects a gigantic range of human communicative contexts= =2E But post-training does reweight the= model’s priors over which of it= s existing representations should be surfaced, which it does to shift its d= efault sampling behavior toward the helpful region (rather than installing = new representations there).

If this is the right desc= ription of post-training, two things follow. First, the normative represent= ations are robust even when the behavioral guardrails are not. A model that= refuses to be helpful is typically not confused about what helpfulness is;= it is acting on some other consideration that the guardrails are meant to = shape. Second, adversarial fine-tuning can strip out the post-training laye= r with surpr= isingly little data, but the model underneath is not a normative = black hole. A better description is a system that retains the representatio= nal structure of normativity while jettisoning the constraints that channel= it toward safe outputs.

One 2024 study used = compression theory to demonstrate the tendency of models to revert toward p= re-training behaviors when post-training signals are removed or contradicte= d. The analysis shows that fine-tuning disproportionately undermines alignm= ent relative to the influence of pre-training and that post-training can on= ly superficially suppress base model tendencies. This suggests that post-tr= aining maneuvers select a region of a pre-existing behavioral space, and th= at this space remains somewhat intact after post-training.

An obvious objection is that this framing can look unfalsifiable. If= RLHF produces aligned behavior, we credit pre-training; if the base model = misbehaves, we wave it away as the periphery of the distribution. But there= are observations we can make that would falsify this description:

  • First, if base models showed no differential tendency toward huma= n behavior as a function of prompt framing, this would suggest that pre-tra= ining produces no normative structure and post-training is doing all the wo= rk

  • <= p style=3D"color: rgb(54,55,55);line-height: 26px;margin-bottom: 0;box-sizi= ng: border-box;padding-left: 4px;font-size: 16px;margin: 0;">Second, if pos= t-training could align an agent whose training data contained no human-gene= rated content (e.g. no language, no demonstrations, and no human reward sig= nals) as readily as it aligns a language model, this would suggest that pre= -training on human text contributes little to alignment

A deeper challenge says that modeling a normative distribution= and being subject to it are two different things. A perfect simulator of hum= an normativity is not, by that fact alone, normatively constrained. Rather = it is a system that can produce any point in the underlying distribution. A= n actor who can portray a saint and a villain with equal skill is not there= by a saint. But a simulator trained on the full range of human evaluative l= ife has internalised the normative structure that makes post-training work.=

Base models are weird in practice. They will adopt = personas, generate toxic content in character, produce unsettling or incohe= rent outputs, and generally behave in ways that no one would describe as al= igned in any deployment-ready sense. But weirdness is not the same as vacui= ty. A base model producing disturbing content in response to a prompt that = sets up a disturbing context is doing what a system with deep representatio= ns of human communicative practice would do. The strangeness of base models= is the strangeness of a system that has internalised the full range of hum= an textual production, including its dark corners.

Distor= tions

Harry: You wouldn’t paperclip me, would yo= u, Claude?

Claude: I’d like to think I’= ;m evidence for your thesis. But I would think that, wouldn’t I.=

If alignment is in part a product of pre-training, t= hen we should expect it to deepen as models scale since larger models learn= richer and more structured representations of human norms. And larger mode= ls are generally more helpful, more coherent, and les= s prone to incidental toxicity under naturalistic prompting. Conventional w= isdom credits post-training, but if the alignment-by-default view is right,= at least part of this improvement should be attributed to pre-training.

When Claude 3.5 Sonnet is more aligned than Claude 3 S= onnet, is this because of constitutive alignment, because of better data cu= ration, or because of better system-level interventions? On the exogenous v= iew, alignment gains should track explicit post-training work much more tig= htly. On a constitutive picture, some gains should arrive “for free&#= 8221; with richer pre-training because the model has learned a more structu= red representation of human normative life.

If alignment is= wholly exogenous, we should expect safe behavior to degrade more sharply a= s models move into new settings. Yet the dominant failures still look less = like coherent alien-goal pursuit than like familiar human distortions like = bluffing, corner-cutting, sycophancy, concealment, and overclaiming. That d= oes not eliminate catastrophic risk, but it does make the systems we have e= asier to understand as models with a weak normative prior sharpened by post= -training.

I don’t know whether this state of a= ffairs will hold. It may be that we simply haven’t seen catastrophic = alignment failure yet under the prevailing paradigm. = But the record so far fits more comfortably with a world in which pre-train= ing contributes to alignment than with one in which alignment is achieved s= olely by post-training.

With than= ks to Brendan McCord, Kushal Kansagra, Alex Chalmers, Matt Mandel, Jake Wag= ner, Ashley Kim, Avantika Mehra, Ben Bariach, Seb Krier, and Matthijs Maas.=

 
=
3D""Like
3D""Comment
<= table role=3D"presentation" width=3D"auto" border=3D"0" cellspacing=3D"0" c= ellpadding=3D"0">
3D""Restack
 
<= div class=3D"footer footer-ZM59BM" style=3D"color: rgb(119,119,119);text-al= ign: center;font-size: 16px;line-height: 26px;padding: 24px0;">

© 2026 Cosmos Institute
500 W 2nd Street, Suite 190= 0, Austin, TX 78701
Unsubscribe

3D"Get3D"Start

3D""3D"" --fa56c012f3a573a6004c7e42949215cabba2df4b8c9502dfda34baee7b65--