Return-Path: Delivered-To: alvarochf@traceseis.com Received: from cs510.bluehost.com by cs510.bluehost.com with LMTP id cNJNKRi69Glj3gAAWiBQxw (envelope-from ) for ; Fri, 01 May 2026 08:35:04 -0600 Return-path: Envelope-to: alvarochf@traceseis.com Delivery-date: Fri, 01 May 2026 08:35:04 -0600 Received: from v541.v5375b7fa.use4.send.mailgun.net ([159.112.244.41]:37778) by cs510.bluehost.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1wIoxI-00000000MLc-0vem for alvarochf@traceseis.com; Fri, 01 May 2026 08:35:04 -0600 DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg-d1.substack.com; q=dns/txt; s=k1; t=1777646081; x=1777653281; h=List-Unsubscribe-Post: List-Unsubscribe: List-Post: List-Id: List-Archive: List-Owner: Reply-To: In-Reply-To: References: Sender: Sender: Date: Message-Id: To: To: From: From: Subject: Subject: Content-Type: Mime-Version; bh=6ZTM1AveVOrWo6m+ECr7Nb2+08zIybtTqd/vuWGkYSU=; b=a3mKYt4iqrUwEQ/NEoNY4LcGtJkizHmMxP8MAGaw/9hANZeBf8bh0wlFcZCmncwjV8oZ5yJ5OZbVjuGFXlAeZvMpfR2084HOyYCrZ4PhRM7/7p7jKMKQIR5a/+rJi+u9a6lLvB/5ry5ksSNUf/WxfZIHO+vBrp8a8c4ovYo2k3M= X-Mailgun-Sid: WyJiYTg1MiIsImFsdmFyb2NoZkB0cmFjZXNlaXMuY29tIiwiMzliMDhjNiJd Received: by ad247960e3da666d88c959f9dcbb96b24e8be0b058271bad1987cb4af9c2a23a with HTTP id 69f4ba0197aadcc98f646848; Fri, 01 May 2026 14:34:41 GMT X-Mailgun-Sending-Ip: 159.112.244.41 X-Mailgun-Batch-Id: 69f4ba019671738817e9df3b Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="d38084f1f8e887a772bbead89c77849f83a9d38219af347e7ed06bcd64fa" Subject: Optimization and its Discontents From: Cosmos Institute To: alvarochf@traceseis.com X-Mailgun-Tag: post X-Mailgun-Track-Clicks: false Message-Id: <20260501143048.3.f8dffe01d89cda66@mg-d1.substack.com> Date: Fri, 1 May 2026 14:30:48 +0000 Feedback-Id: post-196104514:cat-post:pub-2225794:substack Sender: Cosmos Institute References: In-Reply-To: Reply-To: Cosmos Institute List-Owner: List-Url: List-Archive: List-Id: List-Post: List-Unsubscribe: List-Unsubscribe-Post: List-Unsubscribe=One-Click X-Mailgun-Variables: {"category": "post", "email_generated_at": "1777646080598", "is_freemail": "true", "post_audience": "everyone", "post_id": "196104514", "post_type": "newsletter", "pub_community_enabled": "true", "publication_id": "2225794", "subdomain": "cosmosinstitute", "user_id": "347258985"} X-Spam-Status: No, score=-0.8 X-Spam-Score: -7 X-Spam-Bar: / X-Spam-Flag: NO --d38084f1f8e887a772bbead89c77849f83a9d38219af347e7ed06bcd64fa Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable View this post on the web at https://blog.cosmos-institute.org/p/optimizati= on-and-its-discontents Imagine two futures. In the first, a vast number of people lived rich, fulf= illing, and varied lives. They have a range of temperaments, pursue their o= wn aspirations, and find joy in different things. In the second, the same n= umber of people live the same excellent life, but repeated identically, bec= ause an intelligent designer worked out which life scored higher on a wellb= eing scale and made copies of it. Most people would overwhelmingly prefer the first. But many consequentialis= ts would view the second as at least as good, if not better. If total wellb= eing is what matters, then the copies of the ideally-lived life have more o= f it. William MacAskill has become troubled by this implication of wellbeing opti= mization. Having co-founded the effective altruist movement, which compares= various courses of action in those terms, AI is starting to pose a conundr= um. The total wellbeing framework recommends producing as many copies as po= ssible of whichever digital mind scores =E2=80=9Cbest=E2=80=9D on a wellbei= ng metric. A future with trillions of copies of the same aggressively optim= ized life must logically be the best one. In a new essay [ https://substack.com/redirect/19944789-a74b-4e85-b441-714d= a89e1772?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-Hyo= A ], MacAskill lays out the case for =E2=80=9Csaturationism,=E2=80=9D a for= mula in which welfare is still the core driver of value, but stops accumula= ting once enough similar lives have been created. He highlights that =E2=80= =9Cfrom a purely intellectual perspective, it=E2=80=99s probably the best i= dea [he=E2=80=99s] ever had.=E2=80=9D Unfortunately, it misunderstands why variety matters. More importantly, it = preserves the mistake of the original framework: the assumption that the be= st future is something that a theorist can derive. Experiments in Living On one level, MacAskill is undoubtedly right. A future of varied lives is b= etter than one of optimized carbon copies. But thinking about variety an en= d-in-itself gets things backwards. Rather, we should think of variety as th= e condition under which value itself comes into being. We are formed by what we encounter. When we live among different kinds of p= eople, face novel challenges, or engage in different types of work or recre= ation, we develop more than we would if we didn=E2=80=99t experience such v= ariety. As the education pioneer Wilhelm von Humboldt [ https://substack.co= m/redirect/f4c5c8d3-33e8-4799-a449-551e335d0627?j=3DeyJ1IjoiNXFxeXF4In0.h9d= EDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ] wrote: =E2=80=9CEven the most fr= ee and self-reliant of men is thwarted and hindered in his development by u= niformity of position.=E2=80=9D The development of a person=E2=80=99s power= s requires not only freedom but a variety of situations. John Stuart Mill, writing sixty years later, used a passage from Humboldt a= s the epigraph to On Liberty, just before the dedication to his wife Harrie= t Taylor. In On Liberty, Mill makes the case that we cannot discern what co= nstitutes a good life solely by theorizing from the comfort of our armchair= s. Mill believed that humans differ in temperament, capacities, along with = what makes them flourish, and that no one can know in advance which way of = life will suit them. A society that permits these =E2=80=9Cexperiments in l= iving=E2=80=9D generates real knowledge over time about which lives are des= irable. This argument extends well from individuals to the structure of societies. = The knowledge of what is worth doing, making, or wanting is not held by any= one. It is held in pieces =E2=80=93 by artisans, traders, communities, and = families =E2=80=93 each of whom knows something about their own corner of t= he world that nobody else does. Institutions like the price system, the com= mon law, scientific inquiry, language itself work because they let those pi= eces interact without anyone having to assemble them. The planner who tries= to assemble them fails [ https://substack.com/redirect/424b9840-07e9-4910-= a4a5-45a15b4bc5c2?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoF= v6eWS-HyoA ] because this kind of knowledge cannot be assembled. It is not = the kind of thing that can be written down and computed, because it exists = only in the doing. Variety creates value through discovery. In other words, true value is the product of people working out, in their o= wn lives, what is worth caring about. Coloring the Universe Saturationism is essentially the polar opposite of this view of experience. MacAskill imagines all the possible kinds of experience laid out as a map t= hat can be measured, scored, and integrated. You might have joy in one regi= on, awe in another, and the satisfaction of solving a problem in a third, a= nd so on. The map has a fixed structure, which covers how similar two types= of experience are, how much space they take up, and how much value they ca= n contribute. When a life exists, it lights up its corresponding region on the map. But t= he lighting saturates: the first lives in a region brighten it considerably= , while additional copies of the same kind of life add less and less, with = brightness eventually hitting a ceiling =E2=80=93 in other words, the color= hits saturation point. The best future is the one in which the lives that = exist, taken together, light up the most of the map. Coming up with a fixed map of experiences is an ambitious project. MacAskil= l is admirably candid when he admits that it=E2=80=99s not designed on the = basis of any independent facts about the world. Instead, it is built and ca= librated against the moral intuitions that in this case he and his collabor= ators (among which he lists ChatGPT, Claude and Gemini) endorse. MacAskill = writes that it should be understood =E2=80=9Cjust as mathematical structure= s that generate the axiology we want.=E2=80=9D The philosopher Karl Popper spent much of his career attacking this style o= f thinking, which he branded historicism. The historicist, in Popper=E2=80= =99s account, believes that some sufficiently powerful intellectual operati= on can deliver answers about the trajectory of human affairs that would oth= erwise require the actual people involved to work out. In his day, historicism was embodied in Marxist dialectics, but this critiq= ue could equally apply to long-termists wielding integral calculus. Popper = warned that =E2=80=9Cif there is such a thing as growing human knowledge, t= hen we cannot anticipate today what we shall know only tomorrow.=E2=80=9D I= n other words, any theory that ranks possible futures has to make big assum= ptions about what those futures will contain =E2=80=93 what people will dis= cover, what they will come to value, or what forms of life will turn out to= be worth living. What gets bypassed is not just the lives in question but the process throug= h which values get worked out at all. In real life, the question of what is= worth wanting is settled by deliberation. People make their case, others o= bject and propose alternatives. Positions are tested by argument, refined, = and sometimes abandoned. The answer that emerges has standing because it su= rvived this process. The formal apparatus replaces all of this with one phi= losopher and their collaborators=E2=80=99 settled intuitions, and presents = the result as the answer rather than as one position in the argument. The theorist is in no position to make those assumptions, because the peopl= e whose knowledge would validate them have not yet lived. With the right ma= p, the right metric, the right function, the question of which futures are = best is supposed to come out as a calculation. But when the map is drawn by= a theorist and the metrics calibrated to their intuitions, the function wi= ll deliver the results they want. In Search of Theory X=20 Saturationism is a product of the basic assumptions of population ethics. P= opularized by Derek Parfit in the 1980s, population ethics studies how our = actions affect who and how many people are born in the future. This essay b= elongs to the specific subdiscipline of population axiology, which compares= different future population configurations. This, in essence, presupposes= that populations are the kind of thing that can be ranked and that someone= can do the ranking. Population ethics is haunted by a hardcore variant of consequentialism, kno= wn as totalism. The totalists seek to maximize the total sum of net wellbei= ng across a population, regardless of how it is distributed. This disregard= s how ethics has been studied by the mainstream for centuries; even the ear= ly utilitarians would ask about the consequences of particular acts for act= ual people. In population ethics, people are swapped out for =E2=80=9Cvalue= -bearers,=E2=80=9D lives are =E2=80=9Cadded=E2=80=9D to populations, and = =E2=80=9Ccompared=E2=80=9D across possible worlds. MacAskill believes that the arguments for totalism are =E2=80=9Cfairly stro= ng,=E2=80=9D but that it leads to four conclusions that he cannot accept. First, that a vast population of awful lives can outweigh a smaller populat= ion of blissful ones. Second, that any guaranteed good outcome can be beate= n by a highly improbable gamble if the payoff is sufficiently astronomical.= Third, that it breaks down in worlds with infinite populations, losing the= ability to tell obviously different futures apart. Finally, it recommends = filling the universe with as many identical copies of whichever life scores= highest as possible. The first three are old problems in the field, which Parfit himself acknowl= edged needed solving by a =E2=80=9CTheory X=E2=80=9D. The fourth is MacAski= ll=E2=80=99s own contribution. Ultimately, he sees the other three problems= as the product of totalism letting one kind of value pile up without limit= , so by solving the problem of variety, you can crack the other three. On one reading, this is a heroic attempt to preserve as much of the consequ= entialist framework as possible while patching up its worst outputs. But as= a rescue job, it doesn=E2=80=99t work. Totalism=E2=80=99s unsavoury output= s come from treating the value of a future as a quantity a theorist can com= pute from outside. If you keep the broad framework, but change the formula,= you open as many new problems as you create. You can see this in MacAskill=E2=80=99s own treatment of the negative side.= Saturation caps how good a future can get by piling up identical lives, bu= t it also has to handle how bad a future can get. If you cap suffering in t= he same way, a billion tortured lives could count for the same as a trillio= n. But if you leave it uncapped, then tiny probabilities of immense sufferi= ng swamp every calculation and reintroduce the wild gambles MacAskill was t= rying to avoid. There is no third option and, to his credit, he acknowledge= s this. The Pretence of Knowledge=20 The problem with Saturationism is not that MacAskill made the wrong choices= about parameters. Saturationism=E2=80=99s verdicts are an improvement on s= tandard totalism, but top-down population theorizing will always lead to pr= oblems, because it has to compute answers to questions whose answers cannot= be computed. Sometimes the baby does need to be thrown out with the bathwa= ter. It=E2=80=99s almost a truism to say that when a measure becomes a target, i= t ceases to be a good measure. There=E2=80=99s nothing wrong with comparing= humanity=E2=80=99s different futures per se. Some are (much) better than o= thers. But the variety that matters is the byproduct of free people develop= ing in their own directions, under conditions that support this [ https://s= ubstack.com/redirect/e0a99e8e-07c4-4b7b-8629-11668af14e62?j=3DeyJ1IjoiNXFxe= XF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-HyoA ]. Variety is not an end= -in-itself. Cosmos Institute [ https://substack.com/redirect/fd93b661-455c-4ddd-8963-88= dd010ad763?j=3DeyJ1IjoiNXFxeXF4In0.h9dEDbGj64cTNudluPwbMYWklSu8UKGoFv6eWS-H= yoA ] is the Academy for Philosopher-Builders, technologists building AI fo= r human flourishing. We run fellowships, fund AI prototypes, and host semin= ars with institutions like Oxford, Aspen Institute, and Liberty Fund. Unsubscribe https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9ibG9nLmNvc21= vcy1pbnN0aXR1dGUub3JnL2FjdGlvbi9kaXNhYmxlX2VtYWlsP3Rva2VuPWV5SjFjMlZ5WDJsa0= lqb3pORGN5TlRnNU9EVXNJbkJ2YzNSZmFXUWlPakU1TmpFd05EVXhOQ3dpYVdGMElqb3hOemMzT= mpRMk1EZ3dMQ0psZUhBaU9qRTRNRGt4T0RJd09EQXNJbWx6Y3lJNkluQjFZaTB5TWpJMU56azBJ= aXdpYzNWaUlqb2laR2x6WVdKc1pWOWxiV0ZwYkNKOS5vMl9BU3dETGN4SjZNeUFsY1dpZzJWTHQ= zdFd4RjktWFRWdHpJLUhzU1FrIiwicCI6MTk2MTA0NTE0LCJzIjoyMjI1Nzk0LCJmIjp0cnVlLC= J1IjozNDcyNTg5ODUsImlhdCI6MTc3NzY0NjA4MCwiZXhwIjoyMDkzMjIyMDgwLCJpc3MiOiJwd= WItMCIsInN1YiI6ImxpbmstcmVkaXJlY3QifQ.Hqt7B5tudMk-YQeXuDWZTXEK281Yym4605ejj= lWQarQ? --d38084f1f8e887a772bbead89c77849f83a9d38219af347e7ed06bcd64fa Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable = Optimization and its Discontents3D""
If the framework you followed brought y= ou here…
͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­
Forwarded this email? Subscribe here for more
Optimization and its Discontents

If= the framework you followed brought you here…

=
Al= ex Chalmers
 = ;
3D""
= 3D""
3D""
3D""
READ I= N APP3D""
=
 
<= tbody>
3D""
<= span>Paul Signac, In the Time of Harmony: The Golden Age Has Not= Passed, It Is Still to Come (1896)
=

Imagine two futures. In the first, a vast number of peop= le lived rich, fulfilling, and varied lives. They have a range of temperame= nts, pursue their own aspirations, and find joy in different things. In the= second, the same number of people live the same excellent life, but repeat= ed identically, because an intelligent designer worked out which life score= d higher on a wellbeing scale and made copies of it.

Most p= eople would overwhelmingly prefer the first. But many consequentialists wou= ld view the second as at least as good, if not better. If total wellbeing i= s what matters, then the copies of the ideally-lived life have more of it.<= /p>

William MacAskill has become troubled by this implication o= f wellbeing optimization. Having co-founded the effective altruist movement= , which compares various courses of action in those terms, AI is starting t= o pose a conundrum. The total wellbeing framework recommends producing as m= any copies as possible of whichever digital mind scores “best” = on a wellbeing metric. A future with trillions of copies of the same aggres= sively optimized life must logically be the best one.

In a new es= say, MacAskill lays out the case for “saturationism,”= a formula in which welfare is still the core driver of value, but stops ac= cumulating once enough similar lives have been created. He highlights that = “from a purely intellectual perspective, it’s probably the best= idea [he’s] ever had.”

Unfortunately, i= t misunderstands why variety matters. More importantly, it preserves the mi= stake of the original framework: the assumption that the best future is som= ething that a theorist can derive.

Upgrade t= o paid

Experiments in Living

On one level, MacAskill is undoubtedly right. A future of varied li= ves is better than one of optimized carbon copies. But thinking about varie= ty an end-in-itself gets things backwards. Rather, we should think of varie= ty as the condition under which value itself comes into being.

We are formed by what we encounter. When we live among different= kinds of people, face novel challenges, or engage in different types of wo= rk or recreation, we develop more than we would if we didn’t experien= ce such variety. As the education pioneer Wilhelm von Humboldt wrote: “= Even the most free and self-reliant of men is thwarted and hindered in his = development by uniformity of position.” The development of a person&#= 8217;s powers requires not only freedom but a variety of situations.=

John Stuart Mill, writing sixty years later, used a = passage from Humboldt as the epigraph to On Liberty, = just before the dedication to his wife Harriet Taylor. In On Lib= erty, Mill makes the case that we cannot discern what constitute= s a good life solely by theorizing from the comfort of our armchairs. Mill = believed that humans differ in temperament, capacities, along with what mak= es them flourish, and that no one can know in advance which way of life wil= l suit them. A society that permits these “experiments in living̶= 1; generates real knowledge over time about which lives are desirable.

This argument extends well from individuals to the= structure of societies. The knowledge of what is worth doing, making, or w= anting is not held by anyone. It is held in pieces – by artisans, tra= ders, communities, and families – each of whom knows something about = their own corner of the world that nobody else does. Institutions like the = price system, the common law, scientific inquiry, language itself work beca= use they let those pieces interact without anyone having to assemble them. = The planner = who tries to assemble them fails because this kind of knowledge c= annot be assembled. It is not the kind of thing that can be written down an= d computed, because it exists only in the doing. Variety creates value thro= ugh discovery.

In other words, true value is the pro= duct of people working out, in their own lives, what is worth caring about.=

Coloring the Universe

Saturationism is = essentially the polar opposite of this view of experience.

= MacAskill imagines all the possible kinds of experience laid out as a map t= hat can be measured, scored, and integrated. You might have joy in one regi= on, awe in another, and the satisfaction of solving a problem in a third, a= nd so on. The map has a fixed structure, which covers how similar two types= of experience are, how much space they take up, and how much value they ca= n contribute.

When a life exists, it lights up its correspo= nding region on the map. But the lighting saturates: the first lives in a r= egion brighten it considerably, while additional copies of the same kind of= life add less and less, with brightness eventually hitting a ceiling ̵= 1; in other words, the color hits saturation point. The best future is the = one in which the lives that exist, taken together, light up the most of the= map.

Coming up with a fixed map of experiences is an ambit= ious project. MacAskill is admirably candid when he admits that it’s = not designed on the basis of any independent facts about the world. Instead= , it is built and calibrated against the moral intuitions that in this case= he and his collaborators (among which he lists ChatGPT, Claude and Gemini)= endorse. MacAskill writes that it should be understood “just as math= ematical structures that generate the axiology we want.”

The philosopher Karl Popper spent much of his career attacking this st= yle of thinking, which he branded historicism. The historicist, in Popper&#= 8217;s account, believes that some sufficiently powerful intellectual opera= tion can deliver answers about the trajectory of human affairs that would o= therwise require the actual people involved to work out.

In= his day, historicism was embodied in Marxist dialectics, but this critique= could equally apply to long-termists wielding integral calculus. Popper wa= rned that “if there is such a thing as growing human knowledge, then = we cannot anticipate today what we shall know only tomorrow.” In othe= r words, any theory that ranks possible futures has to make big assumptions= about what those futures will contain – what people will discover= , what they will come to value, or what forms of life will turn out to be w= orth living.

What gets bypassed is not just the lives in qu= estion but the process through which values get worked out at all. In real = life, the question of what is worth wanting is settled by deliberation. Peo= ple make their case, others object and propose alternatives. Positions are = tested by argument, refined, and sometimes abandoned. The answer that emerg= es has standing because it survived this process. The formal apparatus repl= aces all of this with one philosopher and their collaborators’ settle= d intuitions, and presents the result as the answer rather than as one posi= tion in the argument.

The theorist is in no position to mak= e those assumptions, because the people whose knowledge would validate them= have not yet lived. With the right map, the right metric, the right functi= on, the question of which futures are best is supposed to come out as a cal= culation. But when the map is drawn by a theorist and the metrics calibrate= d to their intuitions, the function will deliver the results they want.

=

In Search of Theory X

Saturationism is a p= roduct of the basic assumptions of population ethics. Popularized by Derek = Parfit in the 1980s, population ethics studies how our actions affect who a= nd how many people are born in the future. This essay belongs to the specif= ic subdiscipline of population axiology, which compares different future po= pulation configurations. This, in essence, presupposes that populations ar= e the kind of thing that can be ranked and that someone can do the ranking.=

Population ethics is haunted by a hardcore variant of cons= equentialism, known as totalism. The totalists seek to maximize the total s= um of net wellbeing across a population, regardless of how it is distribute= d. This disregards how ethics has been studied by the mainstream for centur= ies; even the early utilitarians would ask about the consequences of partic= ular acts for actual people. In population ethics, people are swapped out f= or “value-bearers,” lives are “added” to population= s, and “compared” across possible worlds.

MacAs= kill believes that the arguments for totalism are “fairly strong,R= 21; but that it leads to four conclusions that he cannot accept.

First, that a vast population of awful lives can outweigh a smaller p= opulation of blissful ones. Second, that any guaranteed good outcome can be= beaten by a highly improbable gamble if the payoff is sufficiently astrono= mical. Third, that it breaks down in worlds with infinite populations, losi= ng the ability to tell obviously different futures apart. Finally, it recom= mends filling the universe with as many identical copies of whichever life = scores highest as possible.

The first three are old problem= s in the field, which Parfit himself acknowledged needed solving by a ̶= 0;Theory X”. The fourth is MacAskill’s own contribution. Ultima= tely, he sees the other three problems as the product of totalism letting o= ne kind of value pile up without limit, so by solving the problem of variet= y, you can crack the other three.

On one reading, this is a= heroic attempt to preserve as much of the consequentialist framework as po= ssible while patching up its worst outputs. But as a rescue job, it doesn&#= 8217;t work. Totalism’s unsavoury outputs come from treating the valu= e of a future as a quantity a theorist can compute from outside. If you kee= p the broad framework, but change the formula, you open as many new problem= s as you create.

You can see this in MacAskill’s own = treatment of the negative side. Saturation caps how good a future can get b= y piling up identical lives, but it also has to handle how bad a future can= get. If you cap suffering in the same way, a billion tortured lives could = count for the same as a trillion. But if you leave it uncapped, then tiny p= robabilities of immense suffering swamp every calculation and reintroduce t= he wild gambles MacAskill was trying to avoid. There is no third option and= , to his credit, he acknowledges this.

The Pretence of Kn= owledge

The problem with Saturationism is not that MacAsk= ill made the wrong choices about parameters. Saturationism’s verdicts= are an improvement on standard totalism, but top-down population theorizin= g will always lead to problems, because it has to compute answers to questi= ons whose answers cannot be computed. Sometimes the baby does need to be th= rown out with the bathwater.

It’s almost a trui= sm to say that when a measure becomes a target, it ceases to be a good meas= ure. There’s nothing wrong with comparing humanity’s different = futures per se. Some are (much) better than others. But the variety that ma= tters is the byproduct of free people developing in their own directions, u= nder conditi= ons that support this. Variety is not an end-in-itself.

<= p style=3D"margin: 0 0 20px 0;color: rgb(54,55,55);line-height: 26px;font-s= ize: 16px;">Cos= mos Institute is the Academy for Philosopher-Builders, technologi= sts building AI for human flourishing. We run fellowships, fund AI prototyp= es, and host seminars with institutions like Oxford, Aspen Institute, and L= iberty Fund.

Upgrade to paid

 
= =
3D""Like
3D""Comment
<= td align=3D"center">3D""= Restack
 

© 2= 026 Cosmos Institute
500 W 2nd Street, Suite 1900, Austin, = TX 78701
Unsubscribe

3D"Ge=<= img src=3D"https://substackcdn.com/image/fetch/$s_!LkrL!,w_270,c_limit,f_au= to,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Femai= l%2Fpublish-button%402x.png" srcset=3D"https://substackcdn.com/image/fetch/= $s_!wgfj!,w_135,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F= %2Fsubstack.com%2Fimg%2Femail%2Fpublish-button.png, https://substackcdn.com= /image/fetch/$s_!LkrL!,w_270,c_limit,f_auto,q_auto:good,fl_progressive:stee= p/https%3A%2F%2Fsubstack.com%2Fimg%2Femail%2Fpublish-button%402x.png 2x, ht= tps://substackcdn.com/image/fetch/$s_!KjtY!,w_405,c_limit,f_auto,q_auto:goo= d,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Femail%2Fpublish-b= utton%403x.png 3x" width=3D"135" alt=3D"Start writing" height=3D"40" style= =3D"max-width: 550px;border: none !important;vertical-align: middle;"><= /p>

3D""3D"" --d38084f1f8e887a772bbead89c77849f83a9d38219af347e7ed06bcd64fa--