Return-Path: Delivered-To: alvarochf@traceseis.com Received: from cs510.bluehost.com by cs510.bluehost.com with LMTP id 4PLGH4+DDGoYjQAAWiBQxw (envelope-from ) for ; Tue, 19 May 2026 09:36:47 -0600 Return-path: Envelope-to: alvarochf@traceseis.com Delivery-date: Tue, 19 May 2026 09:36:47 -0600 Received: from v5121.v5075e2e3.usw1.send.mailgun.net ([161.38.201.121]:58728) by cs510.bluehost.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.99.2) (envelope-from ) id 1wPMUw-00000000Fxx-3Lzc for alvarochf@traceseis.com; Tue, 19 May 2026 09:36:47 -0600 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg2.substack.com; q=dns/txt; s=mailo; t=1779204985; x=1779212185; h=List-Unsubscribe-Post: List-Unsubscribe: List-Post: List-Id: List-Archive: List-Owner: Reply-To: In-Reply-To: References: Sender: Sender: Date: Message-Id: To: To: From: From: Subject: Subject: Content-Type: Mime-Version; bh=LINTNJSDWIC4IS1vMvhxgZRVwdh735jl9VGmMMa4CeA=; b=qWFK8PFGeJAd6NkqwShgec9BtDewWR+Y9iZf2LU4HYrhQQ7FZl4JB/hzU0AICzl/0pqgePTIH1A8Ui8vFXfm94RWyFm79nr4e99Sh0Xg+Px4akR7CEN6BzXdX0Qc3ogrSc+gq6Unt+IAIkFN1SfRmgCU/TL/5Tl7LwUEblOkLMQ= X-Mailgun-Sid: WyI3MDdlYyIsImFsdmFyb2NoZkB0cmFjZXNlaXMuY29tIiwiNGQxODU4Il0= Received: by e5c80868af359065cbf8f0d489bb573262f9d36d2924b0317da995b55777ca3d with HTTP id 6a0c8379759e38c3378a1770; Tue, 19 May 2026 15:36:23 GMT X-Mailgun-Sending-Ip: 161.38.201.121 X-Mailgun-Batch-Id: 6a0c83788d25c23e53edd432 Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="961a3145c71a35b120790b9500f4c204828e2621f62ec3fef863591c9d4a" Subject: Radical ethical disagreement, latent reasoning, replication crises, and more From: Cosmos Institute To: alvarochf@traceseis.com X-Mailgun-Tag: post X-Mailgun-Track-Clicks: false Message-Id: <20260519153541.3.3285718b3866e6ce@mg2.substack.com> Date: Tue, 19 May 2026 15:35:41 +0000 Feedback-Id: post-198405530:cat-post:pub-2225794:substack Sender: Cosmos Institute References: In-Reply-To: Reply-To: Cosmos Institute List-Owner: List-Url: List-Archive: List-Id: List-Post: List-Unsubscribe: List-Unsubscribe-Post: List-Unsubscribe=One-Click X-Mailgun-Variables: {"category": "post", "email_generated_at": "1779204983778", "is_freemail": "true", "post_audience": "everyone", "post_id": "198405530", "post_type": "newsletter", "pub_community_enabled": "true", "publication_id": "2225794", "subdomain": "cosmosinstitute", "user_id": "347258985"} X-Spam-Status: No, score=-0.8 X-Spam-Score: -7 X-Spam-Bar: / X-Spam-Flag: NO --961a3145c71a35b120790b9500f4c204828e2621f62ec3fef863591c9d4a Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable View this post on the web at https://blog.cosmos-institute.org/p/radical-et= hical-disagreement-latent Over the last few weeks, Cosmos grantees from our latest cohort [ https://s= ubstack.com/redirect/df5be114-c72f-497e-9b8e-73932f9a628e?j=3DeyJ1IjoiNXFxe= XF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] (including on our trut= h-seeking track we run together with FIRE) have been updating us on the fin= dings of their research and walking through prototypes they=E2=80=99ve buil= t.=20 As ever, we were impressed by the speed, passion, and technical sophisticat= ion on display. Though what also stood out was the diversity of the project= s, which ranged from new benchmarks through to classifiers, and even includ= ed a new encyclopedia. We=E2=80=99ve pulled out some highlights below. This is just a snapshot of the work that we=E2=80=99re supporting; we=E2=80= =99ll continue to share updates on what our grantees are up to, along with = project write-ups [ https://substack.com/redirect/4d6dde47-b8c7-4abe-bbea-a= f4c75377e8c?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-= HyoA ]. If you=E2=80=99re interested in learning more, you can read about all the p= rojects that Cosmos supports here [ https://substack.com/redirect/89a9a454-= d3d8-40f0-b32b-2e175ae753ec?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYW= klSu8UKGoFv6eWS-HyoA ]. General track These two projects are from our General Track, which operates on a similar = model to Emergent Ventures: $1-10k awards for builders working across our t= hree pillars of autonomy, truth-seeking, and decentralization. Spotting the next replication crisis In recent years, large numbers of eye-catching studies have been found to b= e unreplicable. This =E2=80=98replication crisis=E2=80=99 hit psychology fi= rst, but has spread across a number of other academic disciplines. The curr= ent response to replication failures has largely been to discover which ind= ividual studies don=E2=80=99t hold up. In the meantime, potentially decades= of work could have been built on flawed foundations. AI research is a stro= ng candidate for the next crisis. Experiments are highly sensitive to small= changes in the setup, so the same experiment run twice can give you differ= ent answers Rhea Karty [ https://substack.com/redirect/8d3c850b-01da-45cc-a411-df419929= 831c?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ],= a pre-doctoral fellow at Harvard=E2=80=99s metareflection lab, has built R= eplication Radar =E2=80=93 a knowledge-graph tool that tries to detect epis= temic fragility at a field rather than a paper-level before the replication= crisis breaks. The pipeline ingests papers, stores them as a graph, runs u= ser queries, scores each paper, and visualizes the result. It looks for sig= nals such as tightly clustered author networks, citation rings, sudden cita= tion bursts, institutional monoculture, whether retractions actually propag= ate to the papers that cited them, and small sample sizes, with optional LL= M analysis on top. Rhea validated the tool by showing that it could detect most of the papers = involved in the psychology replication crisis using only data from before i= t broke. The next step for the research is to move from papers to concepts, such as = theories, effects, or methodological assumptions that individual discipline= s rely on heavily. Discovering latent reasoning Reasoning models are increasingly trained with reinforcement learning, and = often perform much better on difficult reasoning tasks. But what is this tr= aining actually changing inside the model? Prior work [ https://substack.co= m/redirect/a305655c-6ef8-4609-94a4-3476bc21607b?j=3DeyJ1IjoiNXFxeXF4In0.h9d= EDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] suggests that Reinforcement Lear= ning with Verifiable Rewards (a crucial post-training technique for most re= asoning models) mainly rearranges probabilities over reasoning paths the ba= se model could already produce, nudging it at a few important decision poin= ts. But this doesn=E2=80=99t fully explain how fine-tuned models hold long = chains of reasoning together from start to finish. Hunar Batra [ https://substack.com/redirect/bfcb003d-dce7-458e-a871-1463861= 4f907?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ]= , a DPhil researcher at Oxford, studies whether reasoning fine-tuning does = something more global: reorganising the model=E2=80=99s internal dynamics i= nto a latent reasoning policy. Instead of only changing which next token th= e model prefers, training may help the model enter, maintain, and switch be= tween coherent internal modes as reasoning unfolds. In written chains of th= ought, we often see shifts between setting up the problem, retrieving facts= , planning, computing, checking the answer, and producing the final respons= e. The question is whether these shifts also appear inside the model=E2=80= =99s activations. To test this, the work models reasoning as a system that switches between d= istinct modes. It looks at the internal representations of base and reasoni= ng-trained Llama and Qwen models sentence by sentence, using a contrastive = method to identify what regime the model is operating in at each step. These regimes are compared against eight defined reasoning stages, includin= g planning, computation, verification, and answer emission. Reasoning-train= ed models show clearer internal structure than base models: more persistent= modes, more structured transitions, and stronger specialization around rec= ognizable reasoning functions. These detected latent reasoning policies can= be used to steer a base model toward behaviours seen in reasoning-trained = models. On hard problems the base model previously failed, this raised perf= ormance to 60% on Qwen 1.5B and 46% on Llama 8B. Hunar=E2=80=99s follow-up work is on reward hacking. She=E2=80=99s building= test environments that elicit reward-hacking behaviour during RL training,= better monitors that catch it by tracing what drove the reward, and post-t= raining methods that reduce the damage from misspecified rewards. Early res= ults suggest this catches forms of deception that current approaches, which= only inspect a model=E2=80=99s written reasoning, tend to miss. Truth-seeking Alongside our general grants, we run a track focused specifically on AI and= truth-seeking, in partnership with the Foundation for Individual Rights an= d Expression [ https://substack.com/redirect/f6a0f6d7-d070-4f8c-9060-b8b1d9= b97811?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA = ] (FIRE) =E2=80=93 a non-partisan organization that fights for free speech = and free thought. Normalizing radical disagreement The Conversational System for Intense Disagreement and Ethical Reflection (= Consider) is a conversational AI platform for the discussion of =E2=80=9Cra= dical moral disagreements=E2=80=9D, produced by the Design Bioethics Labora= tory at the University of Oxford. These are disagreements on topics that ar= e so polarizing and emotionally sensitive that people feel unable to discus= s them, even with people who might agree, out of fear of social consequence= s. The user picks a topic from a list determined by a prior study and then sta= tes their opinion. The LLM then asks clarifying questions and works with th= e user to create a summary. The user then selects a =E2=80=9Cdisagreeabilit= y level=E2=80=9D and then engages in a 10 minute discussion, with the LLM t= aking the opposing side. At the end, it provides feedback on various aspect= s of the user=E2=80=99s moral beliefs, as well as areas of agreement and di= sagreement between the user and AI. The team also held a 16-person seminar with experts from philosophy, comput= er science, political science, and psychology. The discussion covered tensions such as whether exposure to disagreement ac= tually produces belief revision, how to keep the tool complementary to rath= er than displacing human conversation, and how to handle the fact that even= the experts instinctively tried to win arguments against the platform rath= er than use it for reflection. These findings will feed back into the next = iteration of the tool. Breaking open the black box The industry standard for safety monitoring is reading a model=E2=80=99s ch= ain of thought and flagging unsafe reasoning with another AI as a judge. Bu= t a capable model can produce a clean-looking chain of thought that doesn= =E2=80=99t reflect what=E2=80=99s actually driving its decisions =E2=80=93 = known as unfaithfulness. Giovanni Maria Occhipinti, a visiting researcher a= t the University of Oxford, aims to enhance transparency through a white-bo= x approach. The framework has two components. Probes are lightweight classifiers that learn to tell faithful and unfaithf= ul reasoning apart by looking at the model=E2=80=99s internal states. Train= ing works by feeding the model adversarial prompts that induce unfaithful r= easoning, using an LLM judge to label each case, and capturing the internal= patterns that correspond to each. The probe learns the signature that sepa= rates the two. At inference time you run a new prompt through the model, pu= ll out those internal patterns, and check them against the probe. Steering vectors work in the same space. They=E2=80=99re adjustments you ca= n apply to the model=E2=80=99s internal activity to nudge it toward more tr= ansparent reasoning. The probes achieved over 90 percent accuracy, while the steering vectors re= stored readable reasoning in up to 46 percent of cases. The paper [ https://substack.com/redirect/77023793-bbf5-4731-9bca-2d7ea46cd= 5d1?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] h= as been accepted as an oral presentation at ICLR 2026 and Gio is looking fo= r funders and collaborators to take it forward. We=E2=80=99ll have more to say on future grant waves and how you can get in= volved in the coming weeks. The best way to keep up to ensure you don=E2=80= =99t miss any opportunities to share your work or ideas is to subscribe bel= ow. Cosmos Institute [ https://substack.com/redirect/074ea2f1-e8e6-4759-a43e-ac= c848f6204b?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-H= yoA ] is the Academy for Philosopher-Builders, technologists building AI fo= r human flourishing. We run fellowships, fund AI prototypes, and host semin= ars with institutions like Oxford, Aspen Institute, and Liberty Fund. Unsubscribe https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9ibG9nLmNvc21= vcy1pbnN0aXR1dGUub3JnL2FjdGlvbi9kaXNhYmxlX2VtYWlsP3Rva2VuPWV5SjFjMlZ5WDJsa0= lqb3pORGN5TlRnNU9EVXNJbkJ2YzNSZmFXUWlPakU1T0RRd05UVXpNQ3dpYVdGMElqb3hOemM1T= WpBME9UZ3pMQ0psZUhBaU9qRTRNVEEzTkRBNU9ETXNJbWx6Y3lJNkluQjFZaTB5TWpJMU56azBJ= aXdpYzNWaUlqb2laR2x6WVdKc1pWOWxiV0ZwYkNKOS5mZ0ZrY3B5ODcwT2dNeHFPN1dZR25ZOTh= QUGRrclFVcFFWakNMdkdZY3BFIiwicCI6MTk4NDA1NTMwLCJzIjoyMjI1Nzk0LCJmIjp0cnVlLC= J1IjozNDcyNTg5ODUsImlhdCI6MTc3OTIwNDk4MywiZXhwIjoyMDk0NzgwOTgzLCJpc3MiOiJwd= WItMCIsInN1YiI6ImxpbmstcmVkaXJlY3QifQ.I4wZdTL4AtHUmHav0Q3c_K3L-vlnNRr5S4Vg7= vLiZmY? --961a3145c71a35b120790b9500f4c204828e2621f62ec3fef863591c9d4a Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable = Radical ethical disagreement, latent reasoning, replication crises, and mor= e
A selection of grant projects from our recen= t demo day
͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­
=
<= table role=3D"presentation" width=3D"auto" border=3D"0" cellspacing=3D"0" c= ellpadding=3D"0">
Forwarded this email? Subscribe here for more

Radical ethical disagreement, latent rea= soning, replication crises, and more

A selection of grant projects from= our recent demo day

=
Cosmos Institute
=
 
=
= 3D""
=3D""
<= tr>
3D=
READ IN APP= 3D""
&nb= sp;

Over the last few weeks, Cosmos= grantees from our latest cohort (including on our truth-seeking track we run= together with FIRE) have been updating us on the findings of their researc= h and walking through prototypes they’ve built.

As ever, we were impressed by the speed, passion, and technical sophis= tication on display. Though what also stood out was the diversity of the pr= ojects, which ranged from new benchmarks through to classifiers, and even i= ncluded a new encyclopedia. We’ve pulled out some highlights below.

This is just a snapshot of the work that we’re s= upporting; we’ll continue to share updates on what our grantees are u= p to, along with project write-ups.

<= /tbody>
3D""
Four projects that grantees have = recently posted about. These are additional to the write-ups below.

If you’re interested in learni= ng more, you can read about all the projects that Cosmos supports here.

Upgrade to paid
=

General track

Th= ese two projects are from our General Track, which operates on a similar mo= del to Emergent Ventures: $1-10k awards for builders working across our thr= ee pillars of autonomy, truth-seeking, and decentralization.

Spotting the next replication crisis

In re= cent years, large numbers of eye-catching studies have been found to be unr= eplicable. This ‘replication crisis’ hit psychology first, but = has spread across a number of other academic disciplines. The current respo= nse to replication failures has largely been to discover which individual s= tudies don’t hold up. In the meantime, potentially decades of work co= uld have been built on flawed foundations. AI research is a strong candidat= e for the next crisis. Experiments are highly sensitive to small changes in= the setup, so the same experiment run twice can give you different answers=

Rh= ea Karty, a pre-doctoral fellow at Harvard’s metareflection= lab, has built Replication Radar – a knowledge-graph tool that tries= to detect epistemic fragility at a field rather than a paper-level before = the replication crisis breaks. The pipeline ingests papers, stores them as = a graph, runs user queries, scores each paper, and visualizes the result. I= t looks for signals such as tightly clustered author networks, citation rin= gs, sudden citation bursts, institutional monoculture, whether retractions = actually propagate to the papers that cited them, and small sample sizes, w= ith optional LLM analysis on top.

3D""

Rhea val= idated the tool by showing that it could detect most of the papers involved= in the psychology replication crisis using only data from before it broke.=

The next step for the research is to move from papers to c= oncepts, such as theories, effects, or methodological assumptions that indi= vidual disciplines rely on heavily.

Discovering= latent reasoning

Reasoning models are increasingly = trained with reinforcement learning, and often perform much better on diffi= cult reasoning tasks. But what is this training actually changing inside th= e model? Prior work suggests that Reinforcement Learning with Verifiable Rewa= rds (a crucial post-training technique for most reasoning models) mainly re= arranges probabilities over reasoning paths the base model could already pr= oduce, nudging it at a few important decision points. But this doesn’= t fully explain how fine-tuned models hold long chains of reasoning togethe= r from start to finish.

Hunar Batra, a DPhil researcher at Ox= ford, studies whether reasoning fine-tuning does something more global: reo= rganising the model’s internal dynamics into a latent reasoning polic= y. Instead of only changing which next token the model prefers, training ma= y help the model enter, maintain, and switch between coherent internal mode= s as reasoning unfolds. In written chains of thought, we often see shifts b= etween setting up the problem, retrieving facts, planning, computing, check= ing the answer, and producing the final response. The question is whether t= hese shifts also appear inside the model’s activations.

3D""

= To test this, the work models reasoning as a system that switches between d= istinct modes. It looks at the internal representations of base and reasoni= ng-trained Llama and Qwen models sentence by sentence, using a contrastive = method to identify what regime the model is operating in at each step.

<= p style=3D"margin: 0 0 20px 0;color: rgb(54,55,55);line-height: 26px;font-s= ize: 16px;text-align: justify">These regimes are compared against eight def= ined reasoning stages, including planning, computation, verification, and a= nswer emission. Reasoning-trained models show clearer internal structure th= an base models: more persistent modes, more structured transitions, and str= onger specialization around recognizable reasoning functions. These detecte= d latent reasoning policies can be used to steer a base model toward behavi= ours seen in reasoning-trained models. On hard problems the base model prev= iously failed, this raised performance to 60% on Qwen 1.5B and 46% on Llama= 8B.

Hunar’s follow-up work is on = reward hacking. She’s building test environments that elicit reward-h= acking behaviour during RL training, better monitors that catch it by traci= ng what drove the reward, and post-training methods that reduce the damage = from misspecified rewards. Early results suggest this catches forms of dece= ption that current approaches, which only inspect a model’s written r= easoning, tend to miss.

Trut= h-seeking

Alongside our general grants, we run a tra= ck focused specifically on AI and truth-seeking, in partnership with the Foundation for= Individual Rights and Expression (FIRE) – a non-partisan o= rganization that fights for free speech and free thought.

Normalizing radical disagreement

The Con= versational System for Intense Disagreement and Ethical Reflection (Conside= r) is a conversational AI platform for the discussion of “radical mor= al disagreements”, produced by the Design Bioethics Laboratory at the= University of Oxford. These are disagreements on topics that are so polari= zing and emotionally sensitive that people feel unable to discuss them, eve= n with people who might agree, out of fear of social consequences.

The user picks a topic from a list determined by a prior study and = then states their opinion. The LLM then asks clarifying questions and works= with the user to create a summary. The user then selects a “disagree= ability level” and then engages in a 10 minute discussion, with the L= LM taking the opposing side. At the end, it provides feedback on various as= pects of the user’s moral beliefs, as well as areas of agreement and = disagreement between the user and AI.

=
3D""

The team also held a 16-person sem= inar with experts from philosophy, computer science, political science, and= psychology.

The discussion covered tensions such as whethe= r exposure to disagreement actually produces belief revision, how to keep t= he tool complementary to rather than displacing human conversation, and how= to handle the fact that even the experts instinctively tried to win argume= nts against the platform rather than use it for reflection. These findings = will feed back into the next iteration of the tool.

Breaking open the black box

The industry standard for= safety monitoring is reading a model’s chain of thought and flagging= unsafe reasoning with another AI as a judge. But a capable model can produ= ce a clean-looking chain of thought that doesn’t reflect what’s= actually driving its decisions – known as unfaithfulness. Giovanni M= aria Occhipinti, a visiting researcher at the University of Oxford, aims to= enhance transparency through a white-box approach.

The fra= mework has two components.

Probes are lightweight classifie= rs that learn to tell faithful and unfaithful reasoning apart by looking at= the model’s internal states. Training works by feeding the model adv= ersarial prompts that induce unfaithful reasoning, using an LLM judge to la= bel each case, and capturing the internal patterns that correspond to each.= The probe learns the signature that separates the two. At inference time y= ou run a new prompt through the model, pull out those internal patterns, an= d check them against the probe.

Steering vectors work in th= e same space. They’re adjustments you can apply to the model’s = internal activity to nudge it toward more transparent reasoning.

The probes achieved over 90 percent accuracy, while the steering vect= ors restored readable reasoning in up to 46 percent of cases.

<= td class=3D"content" align=3D"left" width=3D"1000" style=3D"text-align: cen= ter;">3D""=

The paper has b= een accepted as an oral presentation at ICLR 2026 and Gio is looking for fu= nders and collaborators to take it forward.

We’l= l have more to say on future grant waves and how you can get involved in th= e coming weeks. The best way to keep up to ensure you don’t miss any = opportunities to share your work or ideas is to subscribe below.

Cosmos Ins= titute is the Academy for Philosopher-Builders, technologists bui= lding AI for human flourishing. We run fellowships, fund AI prototypes, and= host seminars with institutions like Oxford, Aspen Institute, and Liberty = Fund.

Upgra= de to paid
 
=
=
3D""Like
<= /td>
3D""Comment
3D""Restack=
 

© 2026 Cosmo= s Institute
500 W 2nd Street, Suite 1900, Austin, TX 78701
Unsubscribe

3D"Get3D"Start

<= /td>3D""3D"" --961a3145c71a35b120790b9500f4c204828e2621f62ec3fef863591c9d4a--