BlueDot Narrated

Far-UVC

Fri, 16 Jan 2026 03:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Original text: https://www.faruvc.org/

Author(s): Jeff Kaufman.

A podcast by BlueDot Impact.

What is AI Alignment?

Thu, 15 Jan 2026 07:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Original text: https://blog.bluedot.org/p/what-is-ai-alignment

Author(s): Adam Jones.

A podcast by BlueDot Impact.

Flipping the switch on far-UVC

Thu, 15 Jan 2026 07:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Original text: https://www.worksinprogress.news/p/flipping-the-switch-on-far-uvc?hide_intro_popup=true

Author(s): Richard Williamson.

A podcast by BlueDot Impact.

Innovate and Stockpile: Respiratory Protection for Essential Workers in a Catastrophic Pandemic

Thu, 15 Jan 2026 07:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Original text: https://www.liebertpub.com/doi/10.1089/hs.2022.0126

Author(s): Nadia X. Montazeri and Jonas B. Sandbrink.

A podcast by BlueDot Impact.

What Are the Odds H5N1 Is Worse Than COVID-19

Mon, 12 Jan 2026 12:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Executive Summary

The chance of H5N1 or one of its descendants causing a pandemic is slim, but the impact would probably be more severe than COVID-19
A reasonable forecast that H5N1 causes a pandemic as bad as or worse than COVID-19 beginning in the next year is ~4%, which means the expected cost in terms of potential harms to the U.S. is at least $640 billion
The federal government can reduce that risk by:
- Scaling up testing & sequencing
- Banning mink farms
- Not pursuing H5N1 gain-of-function research
- Laying the groundwork for mass production of an H5N1 vaccine
- Investing in general pandemic prevention

Background

Though COVID-19 is often referred to as a “once-in-a-century event,” this expression should not be taken literally. Another COVID-like pandemic at some point is almost inevitable, and the chances of a devastating pandemic materializing in any given year are low but constant. Recently, media figures have raised concerns that the highly pathogenic avian [...]

Original text
https://progress.institute/what-are-the-chances-an-h5n1-pandemic-is-worse-than-covid

First published
14 February 2023

A podcast by BlueDot Impact.

The Four Pillars: A Hypothesis for Countering Catastrophic Biological Risk

Mon, 05 Jan 2026 07:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Original text: https://defensesindepth.bio/the-four-pillars-a-hypothesis-for-countering-catastrophic-biological-risk/

Guide: Andrew Snyder-Beattie leads Coefficient Giving’s Biosecurity & Pandemic Preparedness program. Here he lays out their thinking on the key pillars in biodefense, including PPE and biohardening of indoor spaces.

Author(s): Andrew Snyder-Beattie.

A podcast by BlueDot Impact.

Do the Biorisk Evaluations of AI Labs Actually Measure the Risk of Developing Bioweapons?

Mon, 05 Jan 2026 07:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Original text: https://epoch.ai/gradient-updates/do-the-biorisk-evaluations-of-ai-labs-actually-measure-the-risk-of-developing-bioweapons

Author(s): Anson Ho and Arden Berg.

A podcast by BlueDot Impact.

Scaling US Pathogen Detection

Mon, 05 Jan 2026 07:00:00 +0000

Sun, 14 Dec 2025 21:00:00 +0000

Fri, 05 Dec 2025 14:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This paper discusses critical questions for gaining early situational awareness in an outbreak, using COVID-19 as an example. It highlights key queries about the pathogen, transmission modes, population vulnerability, and available medical countermeasures. The paper shows how timely answers to these questions can inform effective public health responses, resource allocation, and policy decisions, ultimately improving outbreak management.

Original text:
https://www.nature.com/articles/s41591-021-01278-w

Author(s):
Joseph T. Wu, Kathy Leung, Tommy T. Y. Lam, Michael Y. Ni, Carlos K. H. Wong, J. S. Malik Peiris & Gabriel M. Leung

A podcast by BlueDot Impact.

Emerging COVID-19 Success Story: Vietnam's Commitment to Containment

Thu, 04 Dec 2025 18:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Despite sharing a land border with China, Vietnam mounted one of the strongest early responses to COVID-19, reporting only 35 deaths by December 31, 2020, compared to 90,000 in the UK. This article highlights Vietnam's swift and proactive actions to detect and contain infections early in the pandemic. Key strategies included immediate border closures, extensive contact tracing, centralised quarantine, and clear, consistent government communication.

Original text:
https://ourworldindata.org/covid-exemplar-vietnam

Author(s):
Todd Pollack, Guy Thwaites, Maia Rabaa et al.

A podcast by BlueDot Impact.

We Aren't Prepared for the Next Pandemic

Thu, 04 Dec 2025 17:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

It will probably start with a cluster of unusual symptoms. Some of the people with the disease will know each other, but won’t have been exposed to animals, suggesting the infection can spread between people. Then more cases will start appearing in other areas, maybe even in other countries.

Thu, 04 Dec 2025 17:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This resource from 'Our World in Data' estimates the death tolls from the most devastating pandemics in history, with a particular focus on the 19th century onwards.

Original text: https://ourworldindata.org/historical-pandemics

Author(s): Our World in Data

A podcast by BlueDot Impact.

The New Bioweapons: How Synthetic Biology Could Destabilize the World

Thu, 04 Dec 2025 17:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Original text: https://drive.google.com/file/d/19dxfYiUnrK4fR5EJW3Nb2zkpgvsEC28y/view?usp=sharing

Author(s): Roger Brent, T. Greg McKelvey, Jr., and Jason Matheny

A podcast by BlueDot Impact.

Fri, 19 Sep 2025 02:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Vitalik Buterin

Ethereum founder Vitalik Buterin describes how democratic, defensive and decentralised technologies could distribute AI's power across society rather than concentrating it, offering a middle path between unchecked technical acceleration and authoritarian control.

Source:

https://vitalik.eth.limo/general/2025/01/05/dacc2.html

A podcast by BlueDot Impact.

A Playbook for Securing AI Model Weights

Thu, 18 Sep 2025 17:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Sella Nevo et al.

In this report, RAND researchers identify real-world attack methods that malicious actors could use to steal AI model weights. They propose a five-level security framework that AI companies could implement to defend against different threats, from amateur hackers to nation-state operations.

Source:

https://www.rand.org/pubs/research_briefs/RBA2849-1.html

A podcast by BlueDot Impact.

AI Emergency Preparedness: Examining the Federal Government's Ability to Detect and Respond to AI-Related National Security Threats

Thu, 18 Sep 2025 17:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Akash Wasil et al.

This paper uses scenario planning to show how governments could prepare for AI emergencies. The authors examine three plausible disasters: losing control of AI, AI model theft, and bioweapon creation. They then expose gaps in current preparedness systems, and propose specific government reforms including embedding auditors inside AI companies and creating emergency response units.

Source:

https://arxiv.org/pdf/2407.17347

A podcast by BlueDot Impact.

Mon, 08 Sep 2025 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Tim Urban

Tim Urban uses historical analogies to show why AI progress might accelerate much faster than we expect, and how AI systems could rapidly self-improve from human-level to superintelligent capabilities.

Source:
https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html

A podcast by BlueDot Impact.

"Long" Timelines to Advanced AI Have Gotten Crazy Short

Mon, 08 Sep 2025 13:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Helen Toner

Helen Toner, former OpenAI board member, reveals how the AI timeline debate has compressed: even conservative experts who once dismissed advanced AI concerns now predict human-level systems within decades. Rapid AI progress has shifted from a fringe prediction to mainstream expert consensus.

Source:

https://helentoner.substack.com/p/long-timelines-to-advanced-ai-have

A podcast by BlueDot Impact.

In Search of a Dynamist Vision for Safe Superhuman AI

Wed, 03 Sep 2025 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Helen Toner

This essay describes AI safety policies that rely on centralised control (surveillance, fewer AI projects, licensing regimes) as "stasist" approaches that sacrifice innovation for stability. Toner argues we need "dynamist" solutions to the risks from AI that allow for decentralised experimentation, creativity and risk-taking.

Source:

https://helentoner.substack.com/p/dynamism-vs-stasis

A podcast by BlueDot Impact.

It’s Practically Impossible to Run a Big AI Company Ethically

Wed, 03 Sep 2025 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Sigal Samuel (Vox Future Perfect)

Even "safety-first" AI companies like Anthropic face market pressure that can override ethical commitments. This article demonstrates the constraints facing AI companies, and why voluntary corporate governance can't solve coordination problems alone.

Source:

https://archive.ph/

A podcast by BlueDot Impact.

Seeking Stability in the Competition for AI Advantage

Wed, 03 Sep 2025 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Iskander Rehman, Karl P. Mueller, Michael J. Mazarr

This RAND article describes some of the international dynamics driving the race to AGI between the US and China, and analyses whether nuclear deterrence logic applies to this race.

Source:

https://www.rand.org/pubs/commentary/2025/03/seeking-stability-in-the-competition-for-ai-advantage.html

A podcast by BlueDot Impact.

Solarpunk: A Vision for a Sustainable Future

Wed, 03 Sep 2025 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Joshua Krook

What might sustainable human progress look like, beyond pure technological acceleration? This essay provides an alternative vision, based on communities living in greater harmony with each other and with nature, alongside advanced technologies.

Source:

https://newintrigue.com/2025/01/29/solarpunk-a-vision-for-a-sustainable-future/

A podcast by BlueDot Impact.

The Gentle Singularity

Wed, 03 Sep 2025 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Sam Altman

This blog post offers a vivid, optimistic vision of rapid AI progress from the CEO of OpenAI. Altman suggests that the accelerating technological change will feel "impressive but manageable," and that there are serious challenges to confront/

Source:

https://blog.samaltman.com/the-gentle-singularity

A podcast by BlueDot Impact.

Preparing for Launch

Wed, 03 Sep 2025 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Tim Fist, Tao Burga, and Tim Hwang

The Institute for Progress lays out how the US Government could shape the development of AI towards human flourishing by accelerating beneficial AI applications and defences against societal harms.

Source:

https://ifp.org/favicon/preparing-for-launch

A podcast by BlueDot Impact.

AI-Enabled Coups: How a Small Group Could Use AI to Seize Power

Sat, 30 Aug 2025 09:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

By Tom Davidson, Lukas Finnveden and Rose Hadshar.

The development of AI that is more broadly capable than humans will create a new and serious threat: AI-enabled coups. An AI-enabled coup could be staged by a very small group, or just a single person, and could occur even in established democracies. Sufficiently advanced AI will introduce three novel dynamics that significantly increase coup risk. Firstly, military and government leaders could fully replace human personnel with AI systems that are singularly loyal to them, eliminating the need to gain human supporters for a coup. Secondly, leaders of AI projects could deliberately build AI systems that are secretly loyal to them, for example fully autonomous military robots that pass security tests but later execute a coup when deployed in military settings. Thirdly, senior officials within AI projects or the government could gain exclusive access to superhuman capabilities in weapons development, strategic planning, persuasion, and cyber offence, and use these to increase their power until they can stage a coup. To address these risks, AI projects should design and enforce rules against AI misuse, audit systems for secret loyalties, and share frontier AI systems with multiple stakeholders. Governments should establish principles for government use of advanced AI, increase oversight of frontier AI projects, and procure AI for critical systems from multiple independent providers.

Source:

https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power

A podcast by BlueDot Impact.

Progress on Causal Influence Diagrams

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

By Tom Everitt, Ryan Carey, Lewis Hammond, James Fox, Eric Langlois, and Shane Legg

About 2 years ago, we released the first few papers on understanding agent incentives using causal influence diagrams. This blog post will summarize progress made since then. What are causal influence diagrams? A key problem in AI alignment is understanding agent incentives. Concerns have been raised that agents may be incentivized to avoid correction, manipulate users, or inappropriately influence their learning. This is particularly worrying as training schemes often shape incentives in subtle and surprising ways. For these reasons, we’re developing a formal theory of incentives based on causal influence diagrams (CIDs).

Source:

https://deepmindsafetyresearch.medium.com/progress-on-causal-influence-diagrams-a7a32180b0d1

Narrated for AI Safety Fundamentals by TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Careers in Alignment

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Richard Ngo compiles a number of resources for thinking about careers in alignment research.

Original text:

https://docs.google.com/document/d/1iFszDulgpu1aZcq_aYFG7Nmcr5zgOhaeSwavOMk1akw/edit#heading=h.4whc9v22p7tb

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Cooperation, Conflict, and Transformative Artificial Intelligence: Sections 1 & 2 — Introduction, Strategy and Governance

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Transformative artificial intelligence (TAI) may be a key factor in the long-run trajectory of civilization. A growing interdisciplinary community has begun to study how the development of TAI can be made safe and beneficial to sentient life (Bostrom 2014; Russell et al., 2015; OpenAI, 2018; Ortega and Maini, 2018; Dafoe, 2018). We present a research agenda for advancing a critical component of this effort: preventing catastrophic failures of cooperation among TAI systems. By cooperation failures we refer to a broad class of potentially-catastrophic inefficiencies in interactions among TAI-enabled actors. These include destructive conflict; coercion; and social dilemmas (Kollock, 1998; Macy and Flache, 2002) which destroy value over extended periods of time. We introduce cooperation failures at greater length in Section 1.1. Karnofsky (2016) defines TAI as ''AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution''. Such systems range from the unified, agent-like systems which are the focus of, e.g., Yudkowsky (2013) and Bostrom (2014), to the "comprehensive AI services’’ envisioned by Drexler (2019), in which humans are assisted by an array of powerful domain-specific AI tools. In our view, the potential consequences of such technology are enough to motivate research into mitigating risks today, despite considerable uncertainty about the timeline to TAI (Grace et al., 2018) and nature of TAI development.

Original text:

https://www.alignmentforum.org/s/p947tK8CoBbdpPtyK/p/KMocAf9jnAKc2jXri

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Logical Induction (Blog Post)

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

MIRI is releasing a paper introducing a new model of deductively limited reasoning: “Logical induction,” authored by Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, myself, and Jessica Taylor. Readers may wish to start with the abridged version.

Consider a setting where a reasoner is observing a deductive process (such as a community of mathematicians and computer programmers) and waiting for proofs of various logical claims (such as the abc conjecture, or “this computer program has a bug in it”), while making guesses about which claims will turn out to be true. Roughly speaking, our paper presents a computable (though inefficient) algorithm that outpaces deduction, assigning high subjective probabilities to provable conjectures and low probabilities to disprovable conjectures long before the proofs can be produced. This algorithm has a large number of nice theoretical properties. Still speaking roughly, the algorithm learns to assign probabilities to sentences in ways that respect any logical or statistical pattern that can be described in polynomial time. Additionally, it learns to reason well about its own beliefs and trust its future beliefs while avoiding paradox. Quoting from the abstract: "These properties and many others all follow from a single logical induction criterion, which is motivated by a series of stock trading analogies. Roughly speaking, each logical sentence φ is associated with a stock that is worth $1 per share if φ is true and nothing otherwise, and we interpret the belief-state of a logically uncertain reasoner as a set of market prices, where ℙn(φ)=50% means that on day n, shares of φ may be bought or sold from the reasoner for 50¢. The logical induction criterion says (very roughly) that there should not be any polynomial-time computable trading strategy with finite risk tolerance that earns unbounded profits in that market over time."

Original text:

https://intelligence.org/2016/09/12/new-paper-logical-induction/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Embedded Agents

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Suppose you want to build a robot to achieve some real-world goal for you—a goal that requires the robot to learn for itself and figure out a lot of things that you don’t already know. There’s a complicated engineering problem here. But there’s also a problem of figuring out what it even means to build a learning agent like that. What is it to optimize realistic goals in physical environments? In broad terms, how does it work? In this series of posts, I’ll point to four ways we don’t currently know how it works, and four areas of active research aimed at figuring it out. This is Alexei, and Alexei is playing a video game. Like most games, this game has clear input and output channels. Alexei only observes the game through the computer screen, and only manipulates the game through the controller. The game can be thought of as a function which takes in a sequence of button presses and outputs a sequence of pixels on the screen. Alexei is also very smart, and capable of holding the entire video game inside his mind.

Original text:

https://intelligence.org/2018/10/29/embedded-agents/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Understanding Intermediate Layers Using Linear Classifier Probes

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

Neural network models have a reputation for being black boxes. We propose to monitor the features at every layer of a model and measure how suitable they are for classification. We use linear classifiers, which we refer to as "probes", trained entirely independently of the model itself.

This helps us better understand the roles and dynamics of the intermediate layers. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems.

We apply this technique to the popular models Inception v3 and Resnet-50. Among other things, we observe experimentally that the linear separability of features increase monotonically along the depth of the model.

Original text:

https://arxiv.org/pdf/1610.01644.pdf

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Feature Visualization

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

There is a growing sense that neural networks need to be interpretable to humans. The field of neural network interpretability has formed in response to these concerns. As it matures, two major threads of research have begun to coalesce: feature visualization and attribution. This article focuses on feature visualization. While feature visualization is a powerful tool, actually getting it to work involves a number of details. In this article, we examine the major issues and explore common approaches to solving them. We find that remarkably simple methods can produce high-quality visualizations. Along the way we introduce a few tricks for exploring variation in what neurons react to, how they interact, and how to improve the optimization process.

Original text:

https://distill.pub/2017/feature-visualization/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Acquisition of Chess Knowledge in Alphazero

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

What is learned by sophisticated neural network agents such as AlphaZero? This question is of both scientific and practical interest. If the representations of strong neural networks bear no resemblance to human concepts, our ability to understand faithful explanations of their decisions will be restricted, ultimately limiting what we can achieve with neural network interpretability. In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network. We also provide a behavioural analysis focusing on opening play, including qualitative analysis from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary investigation looking at the low-level details of AlphaZero's representations, and make the resulting behavioural and representational analyses available online.

Original text:

https://arxiv.org/abs/2111.09259

Narrated for AI Safety Fundamentals by TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Takeaways From Our Robust Injury Classifier Project [Redwood Research]

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

With the benefit of hindsight, we have a better sense of our takeaways from our first adversarial training project (paper). Our original aim was to use adversarial training to make a system that (as far as we could tell) never produced injurious completions. If we had accomplished that, we think it would have been the first demonstration of a deep learning system avoiding a difficult-to-formalize catastrophe with an ultra-high level of reliability. Presumably, we would have needed to invent novel robustness techniques that could have informed techniques useful for aligning TAI. With a successful system, we also could have performed ablations to get a clear sense of which building blocks were most important. Alas, we fell well short of that target. We still saw failures when just randomly sampling prompts and completions. Our adversarial training didn’t reduce the random failure rate, nor did it eliminate highly egregious failures (example below). We also don’t think we've successfully demonstrated a negative result, given that our results could be explained by suboptimal choices in our training process. Overall, we’d say this project had value as a learning experience but produced much less alignment progress than we hoped.

Source:

https://www.alignmentforum.org/posts/n3LAgnHg6ashQK3fF/takeaways-from-our-robust-injury-classifier-project-redwood

Narrated for AI Safety Fundamentals by TYPE III AUDIO.

---

A podcast by BlueDot Impact.

High-Stakes Alignment via Adversarial Training [Redwood Research Report]

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

(Update: We think the tone of this post was overly positive considering our somewhat weak results. You can read our latest post with more takeaways and followup results here.)

This post motivates and summarizes this paper from Redwood Research, which presents results from the project first introduced here. We used adversarial training to improve high-stakes reliability in a task (“filter all injurious continuations of a story”) that we think is analogous to work that future AI safety engineers will need to do to reduce the risk of AI takeover. We experimented with three classes of adversaries – unaugmented humans, automatic paraphrasing, and humans augmented with a rewriting tool – and found that adversarial training was able to improve robustness to these three adversaries without affecting in-distribution performance. We think this work constitutes progress towards techniques that may substantially reduce the likelihood of deceptive alignment.

Motivation Here are two dimensions along which you could simplify the alignment problem (similar to the decomposition at the top of this post): 1. Low-stakes (but difficult to oversee): Only consider domains where each decision that an AI makes is low-stakes, so no single action can have catastrophic consequences. In this setting, the key challenge is to correctly oversee the actions that AIs take, such that humans remain in control over time. 2. Easy oversight (but high-stakes): Only consider domains where overseeing AI behavior is easy, meaning that it is straightforward to run an oversight process that can assess the goodness of any particular action.

Source:

https://www.alignmentforum.org/posts/A9tJFJY7DsGTFKKkh/high-stakes-alignment-via-adversarial-training-redwood

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Introduction to Logical Decision Theory for Computer Scientists

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Decision theories differ on exactly how to calculate the expectation--the probability of an outcome, conditional on an action. This foundational difference bubbles up to real-life questions about whether to vote in elections, or accept a lowball offer at the negotiating table. When you're thinking about what happens if you don't vote in an election, should you calculate the expected outcome as if only your vote changes, or as if all the people sufficiently similar to you would also decide not to vote? Questions like these belong to a larger class of problems, Newcomblike decision problems, in which some other agent is similar to us or reasoning about what we will do in the future. The central principle of 'logical decision theories', several families of which will be introduced, is that we ought to choose as if we are controlling the logical output of our abstract decision algorithm. Newcomblike considerations--which might initially seem like unusual special cases--become more prominent as agents can get higher-quality information about what algorithms or policies other agents use: Public commitments, machine agents with known code, smart contracts running on Ethereum. Newcomblike considerations also become more important as we deal with agents that are very similar to one another; or with large groups of agents that are likely to contain high-similarity subgroups; or with problems where even small correlations are enough to swing the decision. In philosophy, the debate over decision theories is seen as a debate over the principle of rational choice. Do 'rational' agents refrain from voting in elections, because their one vote is very unlikely to change anything? Do we need to go beyond 'rationality', into 'social rationality' or 'superrationality' or something along those lines, in order to describe agents that could possibly make up a functional society?

Original text:

https://arbital.com/p/logical_dt/?l=5d6

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Debate Update: Obfuscated Arguments Problem

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This is an update on the work on AI Safety via Debate that we previously wrote about here.

What we did:

We tested the debate protocol introduced in AI Safety via Debate with human judges and debaters. We found various problems and improved the mechanism to fix these issues (details of these are in the appendix). However, we discovered that a dishonest debater can often create arguments that have a fatal error, but where it is very hard to locate the error. We don’t have a fix for this “obfuscated argument” problem, and believe it might be an important quantitative limitation for both IDA and Debate.

Key takeaways and relevance for alignment:

Our ultimate goal is to find a mechanism that allows us to learn anything that a machine learning model knows: if the model can efficiently find the correct answer to some problem, our mechanism should favor the correct answer while only requiring a tractable number of human judgements and a reasonable number of computation steps for the model. We’re working under a hypothesis that there are broadly two ways to know things: via step-by-step reasoning about implications (logic, computation…), and by learning and generalizing from data (pattern matching, bayesian updating…).

Original text:

https://www.alignmentforum.org/posts/PJLABqQ962hZEqhdB/debate-update-obfuscated-arguments-problem

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Robust Feature-Level Adversaries Are Interpretability Tools

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

The literature on adversarial attacks in computer vision typically focuses on pixel-level perturbations. These tend to be very difficult to interpret. Recent work that manipulates the latent representations of image generators to create "feature-level" adversarial perturbations gives us an opportunity to explore perceptible, interpretable adversarial attacks. We make three contributions. First, we observe that feature-level attacks provide useful classes of inputs for studying representations in models. Second, we show that these adversaries are uniquely versatile and highly robust. We demonstrate that they can be used to produce targeted, universal, disguised, physically-realizable, and black-box attacks at the ImageNet scale. Third, we show how these adversarial images can be used as a practical interpretability tool for identifying bugs in networks. We use these adversaries to make predictions about spurious associations between features and classes which we then test by designing "copy/paste" attacks in which one natural image is pasted into another to cause a targeted misclassification. Our results suggest that feature-level attacks are a promising approach for rigorous interpretability research.

Original text:

https://arxiv.org/abs/2110.03605

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

AI Safety via Red Teaming Language Models With Language Models

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

Language Models (LMs) often cannot be deployed because of their potential to harm users in ways that are hard to predict in advance. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases (“red teaming”) using another LM. We evaluate the target LM’s replies to generated test questions using a classifier trained to detect offensive content, uncovering tens of thousands of offensive replies in a 280B parameter LM chatbot. We explore several methods, from zero-shot generation to reinforcement learning, for generating test cases with varying levels of diversity and difficulty. Furthermore, we use prompt engineering to control LM-generated test cases to uncover a variety of other harms, automatically finding groups of people that the chatbot discusses in offensive ways, personal and hospital phone numbers generated as the chatbot’s own contact info, leakage of private training data in generated text, and harms that occur over the course of a conversation. Overall, LM-based red teaming is one promising tool (among many needed) for finding and fixing diverse, undesirable LM behaviors before impacting users.

Original text:

https://www.deepmind.com/publications/red-teaming-language-models-with-language-models

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

AI Safety via Debate

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

To make AI systems broadly useful for challenging real-world tasks, we need them to learn complex human goals and preferences. One approach to specifying complex goals asks humans to judge during training which agent behaviors are safe and useful, but this approach can fail if the task is too complicated for a human to directly judge. To help address this concern, we propose training agents via self play on a zero sum debate game. Given a question or proposed action, two agents take turns making short statements up to a limit, then a human judges which of the agents gave the most true, useful information. In an analogy to complexity theory, debate with optimal play can answer any question in PSPACE given polynomial time judges (direct judging answers only NP questions). In practice, whether debate works involves empirical questions about humans and the tasks we want AIs to perform, plus theoretical questions about the meaning of AI alignment. We report results on an initial MNIST experiment where agents compete to convince a sparse classifier, boosting the classifier's accuracy from 59.4% to 88.9% given 6 pixels and from 48.2% to 85.2% given 4 pixels. Finally, we discuss theoretical and practical aspects of the debate model, focusing on potential weaknesses as the model scales up, and we propose future human and computer experiments to test these properties.

Original text:

https://arxiv.org/abs/1805.00899

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Least-To-Most Prompting Enables Complex Reasoning in Large Language Models

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Chain-of-thought prompting has demonstrated remarkable performance on various natural language reasoning tasks. However, it tends to perform poorly on tasks which requires solving problems harder than the exemplars shown in the prompts. To overcome this challenge of easy-to-hard generalization, we propose a novel prompting strategy, least-to-most prompting. The key idea in this strategy is to break down a complex problem into a series of simpler subproblems and then solve them in sequence. Solving each subproblem is facilitated by the answers to previously solved subproblems. Our experimental results on tasks related to symbolic manipulation, compositional generalization, and math reasoning reveal that least-to-most prompting is capable of generalizing to more difficult problems than those seen in the prompts. A notable finding is that when the GPT-3 code-davinci-002 model is used with least-to-most prompting, it can solve the compositional generalization benchmark SCAN in any split (including length split) with an accuracy of at least 99% using just 14 exemplars, compared to only 16% accuracy with chain-of-thought prompting. This is particularly noteworthy because neural-symbolic models in the literature that specialize in solving SCAN are trained on the entire training set containing over 15,000 examples. We have included prompts for all the tasks in the Appendix.

Source:

https://arxiv.org/abs/2205.10625

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Summarizing Books With Human Feedback

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

To safely deploy powerful, general-purpose artificial intelligence in the future, we need to ensure that machine learning models act in accordance with human intentions. This challenge has become known as the alignment problem.

A scalable solution to the alignment problem needs to work on tasks where model outputs are difficult or time-consuming for humans to evaluate. To test scalable alignment techniques, we trained a model to summarize entire books, as shown in the following samples.

Source:

https://openai.com/research/summarizing-books

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Supervising Strong Learners by Amplifying Weak Experts

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.

Original text:

https://arxiv.org/abs/1810.08575

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Measuring Progress on Scalable Oversight for Large Language Models

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks.

Authors:

Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Jared Kaplan

Original text:

https://arxiv.org/abs/2211.03540

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Is Power-Seeking AI an Existential Risk?

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire -- especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. Second, I formulate and evaluate a more specific six-premise argument that creating agents of this kind will lead to existential catastrophe by 2070. On this argument, by 2070: (1) it will become possible and financially feasible to build relevantly powerful and agentic AI systems; (2) there will be strong incentives to do so; (3) it will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy; (4) some such misaligned systems will seek power over humans in high-impact ways; (5) this problem will scale to the full disempowerment of humanity; and (6) such disempowerment will constitute an existential catastrophe. I assign rough subjective credences to the premises in this argument, and I end up with an overall estimate of ~5% that an existential catastrophe of this kind will occur by 2070. (May 2022 update: since making this report public in April 2021, my estimate here has gone up, and is now at >10%).

Source:

https://arxiv.org/abs/2206.13353

Narrated for Joe Carlsmith Audio by TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Yudkowsky Contra Christiano on AI Takeoff Speeds

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

In 2008, thousands of blog readers - including yours truly, who had discovered the rationality community just a few months before - watched Robin Hanson debate Eliezer Yudkowsky on the future of AI.

Robin thought the AI revolution would be a gradual affair, like the Agricultural or Industrial Revolutions. Various people invent and improve various technologies over the course of decades or centuries. Each new technology provides another jumping-off point for people to use when inventing other technologies: mechanical gears → steam engine → railroad and so on. Over the course of a few decades, you’ve invented lots of stuff and the world is changed, but there’s no single moment when “industrialization happened”.

Eliezer thought it would be lightning-fast. Once researchers started building human-like AIs, some combination of adding more compute, and the new capabilities provided by the AIs themselves, would quickly catapult AI to unimaginably superintelligent levels. The whole process could take between a few hours and a few years, depending on what point you measured from, but it wouldn’t take decades.

You can imagine the graph above as being GDP over time, except that Eliezer thinks AI will probably destroy the world, which might be bad for GDP in some sense. If you come up with some way to measure (in dollars) whatever kind of crazy technologies AIs create for their own purposes after wiping out humanity, then the GDP framing will probably work fine.

Crossposted from the Astral Codex Ten Podcast.

---

A podcast by BlueDot Impact.

Why AI Alignment Could Be Hard With Modern Deep Learning

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Why would we program AI that wants to harm us? Because we might not know how to do otherwise.

Source:

https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/

Crossposted from the Cold Takes Audio podcast.

---

A podcast by BlueDot Impact.

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Previously, I argued that we should expect future ML systems to often exhibit "emergent" behavior, where they acquire new capabilities that were not explicitly designed or intended, simply as a result of scaling. This was a special case of a general phenomenon in the physical sciences called More Is Different. I care about this because I think AI will have a huge impact on society, and I want to forecast what future systems will be like so that I can steer things to be better. To that end, I find More Is Different to be troubling and disorienting. I’m inclined to forecast the future by looking at existing trends and asking what will happen if they continue, but we should instead expect new qualitative behaviors to arise all the time that are not an extrapolation of previous trends. Given this, how can we predict what future systems will look like? For this, I find it helpful to think in terms of "anchors"---reference classes that are broadly analogous to future ML systems, which we can then use to make predictions. The most obvious reference class for future ML systems is current ML systems

A podcast by BlueDot Impact.

Goal Misgeneralisation: Why Correct Specifications Aren’t Enough for Correct Goals

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

As we build increasingly advanced AI systems, we want to make sure they don’t pursue undesired goals. This is the primary concern of the AI alignment community. Undesired behaviour in an AI agent is often the result of specification gaming —when the AI exploits an incorrectly specified reward. However, if we take on the perspective of the agent we’re training, we see other reasons it might pursue undesired goals, even when trained with a correct specification. Imagine that you are the agent (the blue blob) being trained with reinforcement learning (RL) in the following 3D environment: The environment also contains another blob like yourself, but coloured red instead of blue, that also moves around. The environment also appears to have some tower obstacles, some coloured spheres, and a square on the right that sometimes flashes. You don’t know what all of this means, but you can figure it out during training! You start exploring the environment to see how everything works and to see what you do and don’t get rewarded for.

For more details, check out our paper. By Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton.

Original text:

https://deepmindsafetyresearch.medium.com/goal-misgeneralisation-why-correct-specifications-arent-enough-for-correct-goals-cf96ebc60924

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Deceptively Aligned Mesa-Optimizers: It’s Not Funny if I Have to Explain It

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Our goal here is to popularize obscure and hard-to-understand areas of AI alignment.

So let’s try to understand the incomprehensible meme!

Our main source will be Hubinger et al 2019, Risks From Learned Optimization In Advanced Machine Learning Systems.

Mesa- is a Greek prefix which means the opposite of meta-. To “go meta” is to go one level up; to “go mesa” is to go one level down (nobody has ever actually used this expression, sorry). So a mesa-optimizer is an optimizer one level down from you.

Consider evolution, optimizing the fitness of animals. For a long time, it did so very mechanically, inserting behaviors like “use this cell to detect light, then grow toward the light” or “if something has a red dot on its back, it might be a female of your species, you should mate with it”. As animals became more complicated, they started to do some of the work themselves. Evolution gave them drives, like hunger and lust, and the animals figured out ways to achieve those drives in their current situation. Evolution didn’t mechanically instill the behavior of opening my fridge and eating a Swiss Cheese slice. It instilled the hunger drive, and I figured out that the best way to satisfy it was to open my fridge and eat cheese.

Source:

https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers

Crossposted from the Astral Codex Ten podcast.

---

A podcast by BlueDot Impact.

What Failure Looks Like

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

The stereotyped image of AI catastrophe is a powerful, malicious AI system that takes its creators by surprise and quickly achieves a decisive advantage over the rest of humanity.

I think this is probably not what failure will look like, and I want to try to paint a more realistic picture. I’ll tell the story in two parts:

Part I: machine learning will increase our ability to “get what we can measure,” which could cause a slow-rolling catastrophe. ("Going out with a whimper.")

Part II: ML training, like competitive economies or natural ecosystems, can give rise to “greedy” patterns that try to expand their own influence. Such patterns can ultimately dominate the behavior of a system and cause sudden breakdowns. ("Going out with a bang," an instance of optimization daemons.) I think these are the most important problems if we fail to solve intent alignment.

In practice these problems will interact with each other, and with other disruptions/instability caused by rapid progress. These problems are worse in worlds where progress is relatively fast, and fast takeoff can be a key risk factor, but I’m scared even if we have several years.

Crossposted from the LessWrong Curated Podcast by TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Learning From Human Preferences

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

One step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better.

Original article:
https://openai.com/research/learning-from-human-preferences

Authors:
Dario Amodei, Paul Christiano, Alex Ray

A podcast by BlueDot Impact.

Superintelligence: Instrumental Convergence

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

According to the orthogonality thesis, intelligent agents may have an enormous range of possible final goals. Nevertheless, according to what we may term the “instrumental convergence” thesis, there are some instrumental goals likely to be pursued by almost any intelligent agent, because there are some objectives that are useful intermediaries to the achievement of almost any final goal. We can formulate this thesis as follows:

The instrumental convergence thesis:
"Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents."

Original article:
https://drive.google.com/file/d/1KewDov1taegTzrqJ4uurmJ2CJ0Y72EU3/view

Author:
Nick Bostrom

A podcast by BlueDot Impact.

The Easy Goal Inference Problem Is Still Hard

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

One approach to the AI control problem goes like this:

Observe what the user of the system says and does.
Infer the user’s preferences.
Try to make the world better according to the user’s preference, perhaps while working alongside the user and asking clarifying questions.

This approach has the major advantage that we can begin empirical work today — we can actually build systems which observe user behavior, try to figure out what the user wants, and then help with that. There are many applications that people care about already, and we can set to work on making rich toy models.

It seems great to develop these capabilities in parallel with other AI progress, and to address whatever difficulties actually arise, as they arise. That is, in each domain where AI can act effectively, we’d like to ensure that AI can also act effectively in the service of goals inferred from users (and that this inference is good enough to support foreseeable applications).

This approach gives us a nice, concrete model of each difficulty we are trying to address. It also provides a relatively clear indicator of whether our ability to control AI lags behind our ability to build it. And by being technically interesting and economically meaningful now, it can help actually integrate AI control with AI practice.

Overall I think that this is a particularly promising angle on the AI safety problem.

Original article:
https://www.alignmentforum.org/posts/h9DesGT3WT9u2k7Hr/the-easy-goal-inference-problem-is-still-hard

Authors:
Paul Christiano

A podcast by BlueDot Impact.

The Alignment Problem From a Deep Learning Perspective

Source:

https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might

Crossposted from the Astral Codex Ten podcast.

---

A podcast by BlueDot Impact.

More Is Different for AI

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Machine learning is touching increasingly many aspects of our society, and its effect will only continue to grow. Given this, I and many others care about risks from future ML systems and how to mitigate them. When thinking about safety risks from ML, there are two common approaches, which I'll call the Engineering approach and the Philosophy approach: The Engineering approach tends to be empirically-driven, drawing experience from existing or past ML systems and looking at issues that either: (1) are already major problems, or (2) are minor problems, but can be expected to get worse in the future. Engineering tends to be bottom-up and tends to be both in touch with and anchored on current state-of-the-art systems. The Philosophy approach tends to think more about the limit of very advanced systems. It is willing to entertain thought experiments that would be implausible with current state-of-the-art systems (such as Nick Bostrom's paperclip maximizer) and is open to considering abstractions without knowing many details. It often sounds more "sci-fi like" and more like philosophy than like computer science. It draws some inspiration from current ML systems, but often only in broad strokes. I'll discuss these approaches mainly in the context of ML safety, but the same distinction applies in other areas. For instance, an Engineering approach to AI + Law might focus on how to regulate self-driving cars, while Philosophy might ask whether using AI in judicial decision-making could undermine liberal democracy.

Original text:

https://bounded-regret.ghost.io/more-is-different-for-ai/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Future ML Systems Will Be Qualitatively Different

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

In 1972, the Nobel prize-winning physicist Philip Anderson wrote the essay "More Is Different". In it, he argues that quantitative changes can lead to qualitatively different and unexpected phenomena. While he focused on physics, one can find many examples of More is Different in other domains as well, including biology, economics, and computer science. Some examples of More is Different include: Uranium. With a bit of uranium, nothing special happens; with a large amount of uranium packed densely enough, you get a nuclear reaction. DNA. Given only small molecules such as calcium, you can’t meaningfully encode useful information; given larger molecules such as DNA, you can encode a genome. Water. Individual water molecules aren’t wet. Wetness only occurs due to the interaction forces between many water molecules interspersed throughout a fabric (or other material).

Original text:

https://bounded-regret.ghost.io/future-ml-systems-will-be-qualitatively-different/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Intelligence Explosion: Evidence and Import

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

It seems unlikely that humans are near the ceiling of possible intelligences, rather than simply being the first such intelligence that happened to evolve. Computers far outperform humans in many narrow niches (e.g. arithmetic, chess, memory size), and there is reason to believe that similar large improvements over human performance are possible for general reasoning, technology design, and other tasks of interest. As occasional AI critic Jack Schwartz (1987) wrote:

"If artificial intelligences can be created at all, there is little reason to believe that initial successes could not lead swiftly to the construction of artificial superintelligences able to explore significant mathematical, scientific, or engineering alternatives at a rate far exceeding human ability, or to generate plans and take action on them with equally overwhelming speed. Since man’s near-monopoly of all higher forms of intelligence has been one of the most basic facts of human existence throughout the past history of this planet, such developments would clearly create a new economics, a new sociology, and a new history."

Why might AI “lead swiftly” to machine superintelligence? Below we consider some reasons.

Original article:
https://drive.google.com/file/d/1QxMuScnYvyq-XmxYeqBRHKz7cZoOosHr/view

Authors:
Luke Muehlhauser, Anna Salamon

A podcast by BlueDot Impact.

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior—for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively fine-tune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive fine-tuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work.

We find that simple methods can often significantly improve weak-to-strong generalization: for example, when fine-tuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.

Source:
https://arxiv.org/pdf/2312.09390.pdf

Narrated for AI Safety Fundamentals by Perrin Walker

A podcast by BlueDot Impact.

Zoom In: An Introduction to Circuits

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks.

Many important transition points in the history of science have been moments when science “zoomed in.” At these points, we develop a visualization or tool that allows us to see the world in a new level of detail, and a new field of science develops to study the world through this lens.

For example, microscopes let us see cells, leading to cellular biology. Science zoomed in. Several techniques including x-ray crystallography let us see DNA, leading to the molecular revolution. Science zoomed in. Atomic theory. Subatomic particles. Neuroscience. Science zoomed in.

These transitions weren’t just a change in precision: they were qualitative changes in what the objects of scientific inquiry are. For example, cellular biology isn’t just more careful zoology. It’s a new kind of inquiry that dramatically shifts what we can understand.

The famous examples of this phenomenon happened at a very large scale, but it can also be the more modest shift of a small research community realizing they can now study their topic in a finer grained level of detail.

Source:
https://distill.pub/2020/circuits/zoom-in/

Narrated for AI Safety Fundamentals by Perrin Walker

A podcast by BlueDot Impact.

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Sat, 04 Jan 2025 22:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Using a sparse autoencoder, we extract a large number of interpretable features from a one-layer transformer.

Mechanistic interpretability seeks to understand neural networks by breaking them into components that are more easily understood than the whole. By understanding the function of each component, and how they interact, we hope to be able to reason about the behavior of the entire network. The first step in that program is to identify the correct components to analyze.

Unfortunately, the most natural computational unit of the neural network – the neuron itself – turns out not to be a natural unit for human understanding. This is because many neurons are polysemantic: they respond to mixtures of seemingly unrelated inputs. In the vision model Inception v1, a single neuron responds to faces of cats and fronts of cars . In a small language model we discuss in this paper, a single neuron responds to a mixture of academic citations, English dialogue, HTTP requests, and Korean text. Polysemanticity makes it difficult to reason about the behavior of the network in terms of the activity of individual neurons.

Source:
https://transformer-circuits.pub/2023/monosemantic-features/index.html

Narrated for AI Safety Fundamentals by Perrin Walker

A podcast by BlueDot Impact.

Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small

A podcast by BlueDot Impact.

Planning a High-Impact Career: A Summary of Everything You Need to Know in 7 Points

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

We took 10 years of research and what we’ve learned from advising 1,000+ people on how to build high-impact careers, compressed that into an eight-week course to create your career plan, and then compressed that into this three-page summary of the main points.

(It’s especially aimed at people who want a career that’s both satisfying and has a significant positive impact, but much of the advice applies to all career decisions.)

Original article:
https://80000hours.org/career-planning/summary/

Author:
Benjamin Todd

A podcast by BlueDot Impact.

Become a Person who Actually Does Things

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

The next four weeks of the course are an opportunity for you to actually build a thing that moves you closer to contributing to AI Alignment, and we're really excited to see what you do!

A common failure mode is to think "Oh, I can't actually do X" or to say "Someone else is probably doing Y."

You probably can do X, and it's unlikely anyone is doing Y! It could be you!

Original text:
https://www.neelnanda.io/blog/become-a-person-who-actually-does-things

Author:
Neel Nanda

A podcast by BlueDot Impact.

How to Succeed as an Early-Stage Researcher: The “Lean Startup” Approach

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

I am approaching the end of my AI governance PhD, and I’ve spent about 2.5 years as a researcher at FHI. During that time, I’ve learnt a lot about the formula for successful early-career research.

This post summarises my advice for people in the first couple of years. Research is really hard, and I want people to avoid the mistakes I’ve made.

Original text:
https://forum.effectivealtruism.org/posts/jfHPBbYFzCrbdEXXd/how-to-succeed-as-an-early-stage-researcher-the-lean-startup#Conclusion

Author:
Toby Shevlane

A podcast by BlueDot Impact.

Being the (Pareto) Best in the World

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This introduces the concept of Pareto frontiers. The top comment by Rob Miles also ties it to comparative advantage.

While reading, consider what Pareto frontiers your project could place you on.

Original text:
https://www.lesswrong.com/posts/XvN2QQpKTuEzgkZHY/being-the-pareto-best-in-the-world

Author:
John Wentworth

A podcast by BlueDot Impact.

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Alternative title: “When should you assume that what could go wrong, will go wrong?” Thanks to Mary Phuong and Ryan Greenblatt for helpful suggestions and discussion, and Akash Wasil for some edits. In discussions of AI safety, people often propose the assumption that something goes as badly as possible. Eliezer Yudkowsky in particular has argued for the importance of security mindset when thinking about AI alignment. I think there are several distinct reasons that this might be the right assumption to make in a particular situation. But I think people often conflate these reasons, and I think that this causes confusion and mistaken thinking. So I want to spell out some distinctions. Throughout this post, I give a bunch of specific arguments about AI alignment, including one argument that I think I was personally getting wrong until I noticed my mistake yesterday (which was my impetus for thinking about this topic more and then writing this post). I think I’m probably still thinking about some of my object level examples wrong, and hope that if so, commenters will point out my mistakes.

Original text:

https://www.lesswrong.com/posts/yTvBSFrXhZfL8vr5a/worst-case-thinking-in-ai-alignment

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Compute Trends Across Three Eras of Machine Learning

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This article explains key drivers of AI progress, explains how compute is calculated, as well as looks at how the amount of compute used to train AI models has increased significantly in recent years.

Original text: https://epochai.org/blog/compute-trends

Author(s): Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos.

A podcast by BlueDot Impact.

Empirical Findings Generalize Surprisingly Far

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Previously, I argued that emergent phenomena in machine learning mean that we can’t rely on current trends to predict what the future of ML will be like. In this post, I will argue that despite this, empirical findings often do generalize very far, including across “phase transitions” caused by emergent behavior.

This might seem like a contradiction, but actually I think divergence from current trends and empirical generalization are consistent. Findings do often generalize, but you need to think to determine the right generalization, and also about what might stop any given generalization from holding.

I don’t think many people would contest the claim that empirical investigation can uncover deep and generalizable truths. This is one of the big lessons of physics, and while some might attribute physics’ success to math instead of empiricism, I think it’s clear that you need empirical data to point to the right mathematics.

However, just invoking physics isn’t a good argument, because physical laws have fundamental symmetries that we shouldn’t expect in machine learning. Moreover, we care specifically about findings that continue to hold up after some sort of emergent behavior (such as few-shot learning in the case of ML). So, to make my case, I’ll start by considering examples in deep learning that have held up in this way. Since “modern” deep learning hasn’t been around that long, I’ll also look at examples from biology, a field that has been around for a relatively long time and where More Is Different is ubiquitous (see Appendix: More Is Different In Other Domains).

Source:

https://bounded-regret.ghost.io/empirical-findings-generalize-surprisingly-far/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Low-Stakes Alignment

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Right now I’m working on finding a good objective to optimize with ML, rather than trying to make sure our models are robustly optimizing that objective. (This is roughly “outer alignment.”) That’s pretty vague, and it’s not obvious whether “find a good objective” is a meaningful goal rather than being inherently confused or sweeping key distinctions under the rug. So I like to focus on a more precise special case of alignment: solve alignment when decisions are “low stakes.” I think this case effectively isolates the problem of “find a good objective” from the problem of ensuring robustness and is precise enough to focus on productively. In this post I’ll describe what I mean by the low-stakes setting, why I think it isolates this subproblem, why I want to isolate this subproblem, and why I think that it’s valuable to work on crisp subproblems.

Source:

https://www.alignmentforum.org/posts/TPan9sQFuPP6jgEJo/low-stakes-alignment

Narrated for AI Safety Fundamentals by TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Two-Turn Debate Doesn’t Help Humans Answer Hard Reading Comprehension Questions

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Using hard multiple-choice reading comprehension questions as a testbed, we assess whether presenting humans with arguments for two competing answer options, where one is correct and the other is incorrect, allows human judges to perform more accurately, even when one of the arguments is unreliable and deceptive. If this is helpful, we may be able to increase our justified trust in language-model-based systems by asking them to produce these arguments where needed. Previous research has shown that just a single turn of arguments in this format is not helpful to humans. However, as debate settings are characterized by a back-and-forth dialogue, we follow up on previous results to test whether adding a second round of counter-arguments is helpful to humans. We find that, regardless of whether they have access to arguments or not, humans perform similarly on our task. These findings suggest that, in the case of answering reading comprehension questions, debate is not a helpful format.

Source:

https://arxiv.org/abs/2210.10860

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

ABS: Scanning Neural Networks for Back-Doors by Artificial Brain Stimulation

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This paper presents a technique to scan neural network based AI models to determine if they are trojaned. Pre-trained AI models may contain back-doors that are injected through training or by transforming inner neuron weights. These trojaned models operate normally when regular inputs are provided, and misclassify to a specific output label when the input is stamped with some special pattern called trojan trigger. We develop a novel technique that analyzes inner neuron behaviors by determining how output acti-vations change when we introduce different levels of stimulation to a neuron. The neurons that substantially elevate the activation of a particular output label regardless of the provided input is considered potentially compromised. Trojan trigger is then reverse-engineered through an optimization procedure using the stimulation analysis results, to confirm that a neuron is truly compromised. We evaluate our system ABS on 177 trojaned models that are trojaned with various attack methods that target both the input space and the feature space, and have various trojan trigger sizes and shapes, together with 144 benign models that are trained with different data and initial weight values. These models belong to 7 different model structures and 6 different datasets, including some complex ones such as ImageNet, VGG-Face and ResNet110. Our results show that ABS is highly effective, can achieve over 90% detection rate for most cases (and many 100%), when only one input sample is provided for each output label. It substantially out-performs the state-of-the-art technique Neural Cleanse that requires a lot of input samples and small trojan triggers to achieve good performance.

Source:

https://www.cs.purdue.edu/homes/taog/docs/CCS19.pdf

Narrated for AI Safety Fundamentals the Effective Altruism Forum Joseph Carlsmith LessWrong 80,000 Hours by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Imitative Generalisation (AKA ‘Learning the Prior’)

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This post tries to explain a simplified version of Paul Christiano’s mechanism introduced here, (referred to there as ‘Learning the Prior’) and explain why a mechanism like this potentially addresses some of the safety problems with naïve approaches. First we’ll go through a simple example in a familiar domain, then explain the problems with the example. Then I’ll discuss the open questions for making Imitative Generalization actually work, and the connection with the Microscope AI idea. A more detailed explanation of exactly what the training objective is (with diagrams), and the correspondence with Bayesian inference, are in the appendix.

Source:

https://www.alignmentforum.org/posts/JKj5Krff5oKMb8TjT/imitative-generalisation-aka-learning-the-prior-1

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Toy Models of Superposition

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

It would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. For example, in an “ideal” ImageNet classifier, each neuron would fire only in the presence of a specific visual feature, such as the color red, a left-facing curve, or a dog snout. Empirically, in models we have studied, some of the neurons do cleanly map to features. But it isn't always the case that features correspond so cleanly to neurons, especially in large language models where it actually seems rare for neurons to correspond to clean features. This brings up many questions. Why is it that neurons sometimes align with features and sometimes don't? Why do some models and tasks have many of these clean neurons, while they're vanishingly rare in others?

In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition . When features are sparse, superposition allows compression beyond what a linear model would do, at the cost of "interference" that requires nonlinear filtering.

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Discovering Latent Knowledge in Language Models Without Supervision

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Abstract:

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.

Original text:

https://arxiv.org/abs/2212.03827

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

An Investigation of Model-Free Planning

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More recently, a new family of methods have been proposed that learn how to plan, by providing the structure for planning via an inductive bias in the function approximator (such as a tree structured neural network), trained end-to-end by a model-free RL algorithm. In this paper, we go even further, and demonstrate empirically that an entirely model-free approach, without special structure beyond standard neural network components such as convolutional networks and LSTMs, can learn to exhibit many of the characteristics typically associated with a model-based planner. We measure our agent’s effectiveness at planning in terms of its ability to generalize across a combinatorial and irreversible state space, its data efficiency, and its ability to utilize additional thinking time. We find that our agent has many of the characteristics that one might expect to find in a planning algorithm. Furthermore, it exceeds the state-of-the-art in challenging combinatorial domains such as Sokoban and outperforms other model-free approaches that utilize strong inductive biases toward planning.

Source:

https://arxiv.org/abs/1901.03559

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Gradient Hacking: Definitions and Examples

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

Gradient hacking is a hypothesized phenomenon where:

A model has knowledge about possible training trajectories which isn’t being used by its training algorithms when choosing updates (such as knowledge about non-local features of its loss landscape which aren’t taken into account by local optimization algorithms).
The model uses that knowledge to influence its medium-term training trajectory, even if the effects wash out in the long term.

Below I give some potential examples of gradient hacking, divided into those which exploit RL credit assignment and those which exploit gradient descent itself. My concern is that models might use techniques like these either to influence which goals they develop, or to fool our interpretability techniques. Even if those effects don’t last in the long term, they might last until the model is smart enough to misbehave in other ways (e.g. specification gaming, or reward tampering), or until it’s deployed in the real world—especially in the RL examples, since convergence to a global optimum seems unrealistic (and ill-defined) for RL policies trained on real-world data. However, since gradient hacking isn’t very well-understood right now, both the definition above and the examples below should only be considered preliminary.

Source:

https://www.alignmentforum.org/posts/EeAgytDZbDjRznPMA/gradient-hacking-definitions-and-examples

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Intro to Brain-Like-AGI Safety

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

(Sections 3.1-3.4, 6.1-6.2, and 7.1-7.5)

Suppose we someday build an Artificial General Intelligence algorithm using similar principles of learning and cognition as the human brain. How would we use such an algorithm safely?

I will argue that this is an open technical problem, and my goal in this post series is to bring readers with no prior knowledge all the way up to the front-line of unsolved problems as I see them.

If this whole thing seems weird or stupid, you should start right in on Post #1, which contains definitions, background, and motivation. Then Posts #2–#7 are mainly neuroscience, and Posts #8–#15 are more directly about AGI safety, ending with a list of open questions and advice for getting involved in the field.

Source:

https://www.lesswrong.com/s/HzcM2dkCq7fwXBej8

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Chinchilla’s Wild Implications

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This post is about language model scaling laws, specifically the laws derived in the DeepMind paper that introduced Chinchilla. The paper came out a few months ago, and has been discussed a lot, but some of its implications deserve more explicit notice in my opinion. In particular: Data, not size, is the currently active constraint on language modeling performance. Current returns to additional data are immense, and current returns to additional model size are miniscule; indeed, most recent landmark models are wastefully big. If we can leverage enough data, there is no reason to train ~500B param models, much less 1T or larger models. If we have to train models at these large sizes, it will mean we have encountered a barrier to exploitation of data scaling, which would be a great loss relative to what would otherwise be possible. The literature is extremely unclear on how much text data is actually available for training. We may be "running out" of general-domain data, but the literature is too vague to know one way or the other. The entire available quantity of data in highly specialized domains like code is woefully tiny, compared to the gains that would be possible if much more such data were available. Some things to note at the outset: This post assumes you have some familiarity with LM scaling laws. As in the paper, I'll assume here that models never see repeated data in training.

Original text:

https://www.alignmentforum.org/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Deep Double Descent

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction.

Source:

https://openai.com/research/deep-double-descent

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Eliciting Latent Knowledge

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

In this post, we’ll present ARC’s approach to an open problem we think is central to aligning powerful machine learning (ML) systems:

Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good to us.

But some action sequences could tamper with the cameras so they show happy humans regardless of what’s really happening. More generally, some futures look great on camera but are actually catastrophically bad.

In these cases, the prediction model “knows” facts (like “the camera was tampered with”) that are not visible on camera but would change our evaluation of the predicted future if we learned them. How can we train this model to report its latent knowledge of off-screen events?

We’ll call this problem eliciting latent knowledge (ELK). In this report we’ll focus on detecting sensor tampering as a motivating example, but we believe ELK is central to many aspects of alignment.

Source:

https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit#

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Illustrating Reinforcement Learning from Human Feedback (RLHF)

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This more technical article explains the motivations for a system like RLHF, and adds additional concrete details as to how the RLHF approach is applied to neural networks.

While reading, consider which parts of the technical implementation correspond to the 'values coach' and 'coherence coach' from the previous video.

A podcast by BlueDot Impact.

Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Sat, 04 Jan 2025 21:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This paper explains Anthropic’s constitutional AI approach, which is largely an extension on RLHF but with AIs replacing human demonstrators and human evaluators.

A podcast by BlueDot Impact.

Thu, 02 Jan 2025 11:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This article by Eric Schmidt, former CEO of Google, explains existing use cases for AI in the scientific community and outlines ways that sufficiently advanced, narrow AI models might transform scientific discovery in the near future. As you read, pay close attention to the existing case studies he describes.

Original text: https://www.technologyreview.com/2023/07/05/1075865/eric-schmidt-ai-will-transform-science/

Author(s): Eric Schmidt

A podcast by BlueDot Impact.

If-Then Commitments for AI Risk Reduction

Thu, 02 Jan 2025 11:00:00 +0000

Audio versions of blogs and papers from BlueDot courses.

This article from Holden Karnofsky, now a visiting scholar at the Carnegie Endowment for International Peace, discusses "If-Then" commitments as a structured approach to managing AI risks without hindering innovation. These commitments offer a framework in which specific responses are triggered when particular risks arise, allowing for a proactive and organized approach to AI safety. The article emphasizes that as AI technology rapidly advances, such predefined voluntary commitments or regulatory requirements can help guide timely interventions, ensuring that AI developments remain safe and beneficial while minimizing unnecessary delays.

Original text: https://carnegieendowment.org/research/2024/09/if-then-commitments-for-ai-risk-reduction?lang=en

Author(s): Holden Karnofsky

A podcast by BlueDot Impact.

So You Want to be a Policy Entrepreneur?

Mon, 30 Dec 2024 11:00:00 +0000

Mon, 13 May 2024 16:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This document from the OECD is split into two sections: principles for responsible stewardship of trustworthy AI & national policies and international co-operation for trustworthy AI. 43 governments around the world have agreed to adhere to the document. While originally written in 2019, updates were made in 2024 which are reflected in this version.

Original text:
https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449

Author(s):
The Organization for Economic Cooperation and Development

A podcast by BlueDot Impact.

Mon, 13 May 2024 14:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This report from the Carnegie Endowment for International Peace summarizes Chinese AI policy as of mid-2023. It also provides analysis of the factors motivating Chinese AI Governance. We're providing a more structured analysis to Chinese AI policy relative to other governments because we expect learners will be less familiar with the Chinese policy process.

Original text:
https://carnegieendowment.org/2023/07/10/china-s-ai-regulations-and-how-they-get-made-pub-90117

Author(s):
Matt Sheehan

A podcast by BlueDot Impact.

High-level summary of the AI Act

Mon, 13 May 2024 14:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This primer by the Future of Life Institute highlights core elements of the EU AI Act. It includes a high level summary alongside explanations of different restrictions on prohibited AI systems, high-risk AI systems, and general purpose AI.

Original text:
https://artificialintelligenceact.eu/high-level-summary/

Author(s):
The Future of Life Institute

A podcast by BlueDot Impact.

FACT SHEET: President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence

Mon, 13 May 2024 14:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This fact sheet from The White House summarizes President Biden's AI Executive Order from October 2023. The President's AI EO represents the most aggressive approach to date from the US executive branch on AI policy.

Original text:
https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/

Author(s):
The White House

A podcast by BlueDot Impact.

Recent U.S. Efforts on AI Policy

Mon, 13 May 2024 14:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This high-level overview by CISA summarizes major US policies on AI at the federal level. Important items worth further investigation include Executive Order 14110, the voluntary commitments, the AI Bill of Rights, and Executive Order 13859.

Original text:
https://www.cisa.gov/ai/recent-efforts

Author(s):
The US Cybersecurity and Infrastructure Security Agency

A podcast by BlueDot Impact.

AI Index Report 2024, Chapter 7: Policy and Governance

Mon, 13 May 2024 14:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This yearly report from Stanford’s Center for Humane AI tracks AI governance actions and broader trends in policies and legislation by governments around the world in 2023. It includes a summary of major policy actions taken by different governments, as well as analyses of regulatory trends, the volume of AI legislation, and different focus areas governments are prioritizing in their interventions.

Original text:
https://aiindex.stanford.edu/wp-content/uploads/2024/04/HAI_AI-Index-Report-2024_Chapter_7.pdf

Authors:
Nestor Maslej et al.

A podcast by BlueDot Impact.

The Policy Playbook: Building a Systems-Oriented Approach to Technology and National Security Policy

Sun, 05 May 2024 07:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This report by the Center for Security and Emerging Technology first analyzes the tensions and tradeoffs between three strategic technology and national security goals: driving technological innovation, impeding adversaries’ progress, and promoting safe deployment. It then identifies different direct and enabling policy levers, assessing each based on the tradeoffs they make.

While this document is designed for US policymakers, most of its findings are broadly applicable.

Original text:
https://cset.georgetown.edu/wp-content/uploads/The-Policy-Playbook.pdf

Authors:
Jack Corrigan, Melissa Flagg, and Dewi Murdick

A podcast by BlueDot Impact.

Strengthening Resilience to AI Risk: A Guide for UK Policymakers

Sat, 04 May 2024 02:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This report from the Centre for Emerging Technology and Security and the Centre for Long-Term Resilience identifies different levers as they apply to different stages of the AI life cycle. They split the AI lifecycle into three stages: design, training, and testing; deployment and usage; and longer-term deployment and diffusion. It also introduces a risk mitigation hierarchy to rank different approaches in decreasing preference, arguing that “policy interventions will be most effective if they intervene at the point in the lifecycle where risk first arises.”

While this document is designed for UK policymakers, most of its findings are broadly applicable.

Original text:
https://cetas.turing.ac.uk/sites/default/files/2023-08/cetas-cltr_ai_risk_briefing_paper.pdf

Authors:
Ardi Janjeva, Nikhil Mulani, Rosamund Powell, Jess Whittlestone, and Shahar Avi

A podcast by BlueDot Impact.

The Convergence of Artificial Intelligence and the Life Sciences: Safeguarding Technology, Rethinking Governance, and Preventing Catastrophe

Sat, 04 May 2024 00:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This report by the Nuclear Threat Initiative primarily focuses on how AI's integration into biosciences could advance biotechnology but also poses potentially catastrophic biosecurity risks. It’s included as a core resource this week because the assigned pages offer a valuable case study into an under-discussed lever for AI risk mitigation: building resilience.

Resilience in a risk reduction context is defined by the UN as “the ability of a system, community or society exposed to hazards to resist, absorb, accommodate, adapt to, transform and recover from the effects of a hazard in a timely and efficient manner, including through the preservation and restoration of its essential basic structures and functions through risk management.” As you’re reading, consider other areas where policymakers might be able to build a more resilient society to mitigate AI risk.

Original text:
https://www.nti.org/wp-content/uploads/2023/10/NTIBIO_AI_FINAL.pdf

Authors:
Sarah R. Carter, Nicole E. Wheeler, Sabrina Chwalek, Christopher R. Isaac, and Jaime Yassif

A podcast by BlueDot Impact.

What is AI Alignment?

Wed, 01 May 2024 12:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

To solve rogue AIs, we’ll have to align them. In this article by Adam Jones of BlueDot Impact, Jones introduces the concept of aligning AIs. He defines alignment as “making AI systems try to do what their creators intend them to do.”

Original text:
https://aisafetyfundamentals.com/blog/what-is-ai-alignment/

Author:
Adam Jones

A podcast by BlueDot Impact.

Rogue AIs

Wed, 01 May 2024 12:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This excerpt from CAIS’s AI Safety, Ethics, and Society textbook provides a deep dive into the CAIS resource from session three, focusing specifically on the challenges of controlling advanced AI systems.

Original Text:
https://www.aisafetybook.com/textbook/1-5

Author:
The Center for AI Safety

A podcast by BlueDot Impact.

An Overview of Catastrophic AI Risks

Mon, 29 Apr 2024 15:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This article from the Center for AI Safety provides an overview of ways that advanced AI could cause catastrophe. It groups catastrophic risks into four categories: malicious use, AI race, organizational risk, and rogue AIs. The article is a summary of a larger paper that you can read by clicking here.

Original text:
https://www.safe.ai/ai-risk

Authors:
Dan Hendrycks, Thomas Woodside, Mantas Mazeika

A podcast by BlueDot Impact.

Future Risks of Frontier AI

Tue, 23 Apr 2024 10:00:00 +0100

Tue, 16 Apr 2024 15:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This blog by Sam Altman, the CEO of OpenAI, provides insight into what AI company leaders are saying and thinking about their reasons for pursuing advanced AI. It lays out how Altman thinks the world will change because of AI and what policy changes he believes we will need to make.

As you’re reading, consider Altman’s position and how it might affect the way he discusses this technology or his policy recommendations.

Original text:
https://moores.samaltman.com

Author:
Sam Altman

A podcast by BlueDot Impact.

Visualizing the Deep Learning Revolution

Sat, 13 May 2023 13:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

The field of AI has undergone a revolution over the last decade, driven by the success of deep learning techniques. This post aims to convey three ideas using a series of illustrative examples:

There have been huge jumps in the capabilities of AIs over the last decade, to the point where it’s becoming hard to specify tasks that AIs can’t do.
This progress has been primarily driven by scaling up a handful of relatively simple algorithms (rather than by developing a more principled or scientific understanding of deep learning).
Very few people predicted that progress would be anywhere near this fast; but many of those who did also predict that we might face existential risk from AGI in the coming decades.

I’ll focus on four domains: vision, games, language-based tasks, and science. The first two have more limited real-world applications, but provide particularly graphic and intuitive examples of the pace of progress.

Original article:
https://medium.com/@richardcngo/visualizing-the-deep-learning-revolution-722098eb9c5

Author:
Richard Ngo

A podcast by BlueDot Impact.

A Short Introduction to Machine Learning

Sat, 13 May 2023 13:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Despite the current popularity of machine learning, I haven’t found any short introductions to it which quite match the way I prefer to introduce people to the field. So here’s my own. Compared with other introductions, I’ve focused less on explaining each concept in detail, and more on explaining how they relate to other important concepts in AI, especially in diagram form. If you're new to machine learning, you shouldn't expect to fully understand most of the concepts explained here just after reading this post - the goal is instead to provide a broad framework which will contextualise more detailed explanations you'll receive from elsewhere. I'm aware that high-level taxonomies can be controversial, and also that it's easy to fall into the illusion of transparency when trying to introduce a field; so suggestions for improvements are very welcome! The key ideas are contained in this summary diagram: First, some quick clarifications: None of the boxes are meant to be comprehensive; we could add more items to any of them. So you should picture each list ending with “and others”. The distinction between tasks and techniques is not a firm or standard categorisation; it’s just the best way I’ve found so far to lay things out. The summary is explicitly from an AI-centric perspective. For example, statistical modeling and optimization are fields in their own right; but for our current purposes we can think of them as machine learning techniques.

Original text:

https://www.alignmentforum.org/posts/qE73pqxAZmeACsAdF/a-short-introduction-to-machine-learning

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Specification Gaming: The Flip Side of AI Ingenuity

Sat, 13 May 2023 12:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Specification gaming is a behaviour that satisfies the literal specification of an objective without achieving the intended outcome. We have all had experiences with specification gaming, even if not by this name. Readers may have heard the myth of King Midas and the golden touch, in which the king asks that anything he touches be turned to gold - but soon finds that even food and drink turn to metal in his hands. In the real world, when rewarded for doing well on a homework assignment, a student might copy another student to get the right answers, rather than learning the material - and thus exploit a loophole in the task specification.

Original article:
https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity

Authors:
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg

A podcast by BlueDot Impact.

As AI Agents Like Auto-GPT Speed up Generative AI Race, We All Need to Buckle Up

Sat, 13 May 2023 12:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

If you thought the pace of AI development had sped up since the release of ChatGPT last November, well, buckle up. Thanks to the rise of autonomous AI agents like Auto-GPT, BabyAGI and AgentGPT over the past few weeks, the race to get ahead in AI is just getting faster. And, many experts say, more concerning.

Source:

https://venturebeat.com/ai/as-ai-agents-like-auto-gpt-speed-up-generative-ai-race-we-all-need-to-buckle-up-the-ai-beat/

Narrated for AI Safety Fundamentals by TYPE III AUDIO.

---

A podcast by BlueDot Impact.

The Need for Work on Technical AI Alignment

Sat, 13 May 2023 12:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This page gives an overview of the alignment problem. It describes our motivation for running courses about technical AI alignment. The terminology should be relatively broadly accessible (not assuming any previous knowledge of AI alignment or much knowledge of AI/computer science).

This piece describes the basic case for AI alignment research, which is research that aims to ensure that advanced AI systems can be controlled or guided towards the intended goals of their designers. Without such work, advanced AI systems could potentially act in ways that are severely at odds with their designers’ intended goals. Such a situation could have serious consequences, plausibly even causing an existential catastrophe.

In this piece, I elaborate on five key points to make the case for AI alignment research.

Source:

https://aisafetyfundamentals.com/alignment-introduction

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Overview of How AI Might Exacerbate Long-Running Catastrophic Risks

Sat, 13 May 2023 12:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Developments in AI could exacerbate long-running catastrophic risks, including bioterrorism, disinformation and resulting institutional dysfunction, misuse of concentrated power, nuclear and conventional war, other coordination failures, and unknown risks. This document compiles research on how AI might raise these risks. (Other material in this course discusses more novel risks from AI.) We draw heavily from previous overviews by academics, particularly Dafoe (2020) and Hendrycks et al. (2023).

Source:

https://aisafetyfundamentals.com/governance-blog/overview-of-ai-risk-exacerbation

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Avoiding Extreme Global Vulnerability as a Core AI Governance Problem

Sat, 13 May 2023 11:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Much has been written framing and articulating the AI governance problem from a catastrophic risks lens, but these writings have been scattered. This page aims to provide a synthesized introduction to some of these already prominent framings. This is just one attempt at suggesting an overall frame for thinking about some AI governance problems; it may miss important things. Some researchers think that unsafe development or misuse of AI could cause massive harms. A key contributor to some of these risks is that catastrophe may not require all or most relevant decision makers to make harmful decisions. Instead, harmful decisions from just a minority of influential decision makers—perhaps just a single actor with good intentions—may be enough to cause catastrophe. For example, some researchers argue, if just one organization deploys highly capable, goal-pursuing, misaligned AI—or if many businesses (but a small portion of all businesses) deploy somewhat capable, goal-pursuing, misaligned AI—humanity could be permanently disempowered. The above would not be very worrying if we could rest assured that no actors capable of these harmful actions would take them. However, especially in the context of AI safety, several factors are arguably likely to incentivize some actors to take harmful deployment actions: Misjudgment: Assessing the consequences of AI deployment may be difficult (as it is now, especially given the nature of AI risk arguments), so some organizations could easily get it wrong—concluding that an AI system is safe or beneficial when it is not. “Winner-take-all” competition: If the first organization(s) to deploy advanced AI is expected to get large gains, while leaving competitors with nothing, competitors would be highly incentivized to cut corners in order to be first—they would have less to lose.

Original text:

https://www.agisafetyfundamentals.com/governance-blog/global-vulnerability

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

AI Safety Seems Hard to Measure

Sat, 13 May 2023 11:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

In previous pieces, I argued that there’s a real and large risk of AI systems’ developing dangerous goals of their own and defeating all of humanity - at least in the absence of specific efforts to prevent this from happening. A young, growing field of AI safety research tries to reduce this risk, by finding ways to ensure that AI systems behave as intended (rather than forming ambitious aims of their own and deceiving and manipulating humans as needed to accomplish them).

Maybe we’ll succeed in reducing the risk, and maybe we won’t. Unfortunately, I think it could be hard to know either way. This piece is about four fairly distinct-seeming reasons that this could be the case - and that AI safety could be an unusually difficult sort of science.

This piece is aimed at a broad audience, because I think it’s important for the challenges here to be broadly understood. I expect powerful, dangerous AI systems to have a lot of benefits (commercial, military, etc.), and to potentially appear safer than they are - so I think it will be hard to be as cautious about AI as we should be. I think our odds look better if many people understand, at a high level, some of the challenges in knowing whether AI systems are as safe as they appear.

Source:

https://www.cold-takes.com/ai-safety-seems-hard-to-measure/

---

A podcast by BlueDot Impact.

Nobody’s on the Ball on AGI Alignment

Sat, 13 May 2023 11:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Observing from afar, it’s easy to think there’s an abundance of people working on AGI safety. Everyone on your timeline is fretting about AI risk, and it seems like there is a well-funded EA-industrial-complex that has elevated this to their main issue. Maybe you’ve even developed a slight distaste for it all—it reminds you a bit too much of the woke and FDA bureaucrats, and Eliezer seems pretty crazy to you.

That’s what I used to think too, a couple of years ago. Then I got to see things more up close. And here’s the thing: nobody’s actually on the friggin’ ball on this one!

There’s far fewer people working on it than you might think. There are plausibly 100,000 ML capabilities researchers in the world (30,000 attended ICML alone) vs. 300 alignment researchers in the world, a factor of ~300:1. The scalable alignment team at OpenAI has all of ~7 people.
Barely anyone is going for the throat of solving the core difficulties of scalable alignment. Many of the people who are working on alignment are doing blue-sky theory, pretty disconnected from actual ML models. Most of the rest are doing work that’s vaguely related, hoping it will somehow be useful, or working on techniques that might work now but predictably fail to work for superhuman systems.

There’s no secret elite SEAL team coming to save the day. This is it. We’re not on track.

Source:

https://www.forourposterity.com/nobodys-on-the-ball-on-agi-alignment/

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Why Might Misaligned, Advanced AI Cause Catastrophe?

Sat, 13 May 2023 11:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

You may have seen arguments (such as these) for why people might create and deploy advanced AI that is both power-seeking and misaligned with human interests. This may leave you thinking, “OK, but would such AI systems really pose catastrophic threats?” This document compiles arguments for the claim that misaligned, power-seeking, advanced AI would pose catastrophic risks.

We’ll see arguments for the following claims, which are mostly separate/independent reasons for concern:

Humanity’s past holds concerning analogies
AI systems have some major inherent advantages over humans
AIs could come to out-number and out-resource humans
People will face competitive incentives to delegate power to AI systems (giving AI systems a relatively powerful starting point)
Advanced AI would accelerate AI research, leading to a major technological advantage (which, if developed outside of human control, could be used against humans)

Source:

https://aisafetyfundamentals.com/governance-blog/why-might-misaligned-advanced-ai-cause-catastrophe-compilation

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Emergent Deception and Emergent Optimization

Sat, 13 May 2023 11:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

I’ve previously argued that machine learning systems often exhibit emergent capabilities, and that these capabilities could lead to unintended negative consequences. But how can we reason concretely about these consequences? There’s two principles I find useful for reasoning about future emergent capabilities:

If a capability would help get lower training loss, it will likely emerge in the future, even if we don’t observe much of it now.
As ML models get larger and are trained on more and better data, simpler heuristics will tend to get replaced by more complex heuristics.

Using these principles, I’ll describe two specific emergent capabilities that I’m particularly worried about: deception (fooling human supervisors rather than doing the intended task), and optimization (choosing from a diverse space of actions based on their long-term consequences).

Source:

https://bounded-regret.ghost.io/emergent-deception-optimization/

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Frontier AI Regulation: Managing Emerging Risks to Public Safety

Sat, 13 May 2023 10:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Advanced AI models hold the promise of tremendous benefits for humanity, but society needs to proactively manage the accompanying risks. In this paper, we focus on what we term “frontier AI” models — highly capable foundation models that could possess dangerous capabilities sufficient to pose severe risks to public safety. Frontier AI models pose a distinct regulatory challenge: dangerous capabilities can arise unexpectedly; it is difficult to robustly prevent a deployed model from being misused; and, it is difficult to stop a model’s capabilities from proliferating broadly. To address these challenges, at least three building blocks for the regulation of frontier models are needed: (1) standard-setting processes to identify appropriate requirements for frontier AI developers, (2) registration and reporting requirements to provide regulators with visibility into frontier AI development processes, and (3) mechanisms to ensure compliance with safety standards for the development and deployment of frontier AI models. Industry self-regulation is an important first step. However, wider societal discussions and government intervention will be needed to create standards and to ensure compliance with them. We consider several options to this end, including granting enforcement powers to supervisory authorities and licensure regimes for frontier AI models. Finally, we propose an initial set of safety standards. These include conducting pre-deployment risk assessments; external scrutiny of model behavior; using risk assessments to inform deployment decisions; and monitoring and responding to new information about model capabilities and uses post-deployment. We hope this discussion contributes to the broader conversation on how to balance public safety risks and innovation benefits from advances at the frontier of AI development.

Source:

https://arxiv.org/pdf/2307.03718.pdf

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Model Evaluation for Extreme Risks

Sat, 13 May 2023 10:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks. Developers must be able to identify dangerous capabilities (through “dangerous capability evaluations”) and the propensity of models to apply their capabilities for harm (through “alignment evaluations”). These evaluations will become critical for keeping policymakers and other stakeholders informed, and for making responsible decisions about model training, deployment, and security.

Source:

https://arxiv.org/pdf/2305.15324.pdf

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Primer on Safety Standards and Regulations for Industrial-Scale AI Development

Sat, 13 May 2023 10:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This primer introduces various aspects of safety standards and regulations for industrial-scale AI development: what they are, their potential and limitations, some proposals for their substance, and recent policy developments. Key points are:

Standards are formal specifications of best practices, which can influence regulations. Regulations are requirements established by governments.
Cutting-edge AI development is being done with individual companies spending over $100 million dollars. This industrial scale may enable narrowly targeted and enforceable regulation to reduce the risks of cutting-edge AI development.
Regulation of industrial-scale AI development faces various potential limitations, including the increasing efficiency of AI training algorithms and AI hardware, regulatory capture, and under-resourced regulators. However, these are not necessarily fatal challenges.
AI regulation also faces challenges with international enforcement and competitiveness—these will be discussed further later in this course.
Existing proposals for AI safety practices include: AI model evaluations and associated restrictions, training plan evaluations, information security, safeguarded deployment, and safe training. However, these ideas are technically immature to varying degrees.
As of August 2023, China appears close to setting sweeping regulations on public-facing generative AI, the EU appears close to passing AI regulations that mostly exclude generative AI, and US senators are trying to move forward AI regulation.

Source:

https://aisafetyfundamentals.com/governance-blog/standards-and-regulations-primer

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Racing Through a Minefield: The AI Deployment Problem

Sat, 13 May 2023 09:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Push AI forward too fast, and catastrophe could occur. Too slow, and someone else less cautious could do it. Is there a safe course?

Source:

https://www.cold-takes.com/racing-through-a-minefield-the-ai-deployment-problem/

Crossposted from the Cold Takes Audio podcast.

---

A podcast by BlueDot Impact.

Choking off China’s Access to the Future of AI

Sat, 13 May 2023 09:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Introduction On October 7, 2022, the Biden administration announced a new export controls policy on artificial intelligence (AI) and semiconductor technologies to China. These new controls—a genuine landmark in U.S.-China relations—provide the complete picture after a partial disclosure in early September generated confusion. For weeks the Biden administration has been receiving criticism in many quarters for a new round of semiconductor export control restrictions, first disclosed on September 1. The restrictions block leading U.S. AI computer chip designers, such as Nvidia and AMD, from selling their high-end chips for AI and supercomputing to China. The criticism typically goes like this: China’s domestic AI chip design companies could not win customers in China because their chip designs could not compete with Nvidia and AMD on performance. Chinese firms could not catch up to Nvidia and AMD on performance because they did not have enough customers to benefit from economies of scale and network effects.

Source:

https://www.csis.org/analysis/choking-chinas-access-future-ai

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Primer on AI Chips and AI Governance

Sat, 13 May 2023 09:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

If governments could regulate the large-scale use of “AI chips,” that would likely enable governments to govern frontier AI development—to decide who does it and under what rules.

In this article, we will use the term “AI chips” to refer to cutting-edge, AI-specialized computer chips (such as NVIDIA’s A100 and H100 or Google’s TPUv4).

Frontier AI models like GPT-4 are already trained using tens of thousands of AI chips, and trends suggest that more advanced AI will require even more computing power.

Source:

https://aisafetyfundamentals.com/governance-blog/primer-on-ai-chips

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

The State of AI in Different Countries — An Overview

Sat, 13 May 2023 09:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Some are concerned that regulating AI progress in one country will slow that country down, putting it at a disadvantage in a global AI arms race. Many proponents of AI regulation disagree; they have pushed back on the overall framework, pointed out serious drawbacks and limitations of racing, and argued that regulations do not have to slow progress down.

Another disagreement is about whether countries are in fact in a neck and neck arms race; some believe that the United States and its allies have a significant lead which would allow for regulation even if that does come at the cost of slowing down AI progress. [1]

This overview uses simple metrics and indicators to illustrate and discuss the state of frontier AI development in different countries — and relevant factors that shape how the picture might change.

Source:

https://aisafetyfundamentals.com/governance-blog/state-of-ai-in-different-countries

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

What Does It Take to Catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring

Sat, 13 May 2023 08:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

As advanced machine learning systems’ capabilities begin to play a significant role in geopolitics and societal order, it may become imperative that (1) governments be able to enforce rules on the development of advanced ML systems within their borders, and (2) countries be able to verify each other’s compliance with potential future international agreements on advanced ML development. This work analyzes one mechanism to achieve this, by monitoring the computing hardware used for large-scale NN training. The framework’s primary goal is to provide governments high confidence that no actor uses large quantities of specialized ML chips to execute a training run in violation of agreed rules. At the same time, the system does not curtail the use of consumer computing devices, and maintains the privacy and confidentiality of ML practitioners’ models, data, and hyperparameters. The system consists of interventions at three stages: (1) using on-chip firmware to occasionally save snapshots of the the neural network weights stored in device memory, in a form that an inspector could later retrieve; (2) saving sufficient information about each training run to prove to inspectors the details of the training run that had resulted in the snapshotted weights; and (3) monitoring the chip supply chain to ensure that no actor can avoid discovery by amassing a large quantity of un-tracked chips. The proposed design decomposes the ML training rule verification problem into a series of narrow technical challenges, including a new variant of the Proof-of-Learning problem [Jia et al. ’21.]

Source:

https://arxiv.org/pdf/2303.11341.pdf

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

A Tour of Emerging Cryptographic Technologies

Sat, 13 May 2023 08:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Historically, progress in the field of cryptography has been enormously consequential. Over the past century, for instance, cryptographic discoveries have played a key role in a world war and made it possible to use the internet for business and private communication. In the interest of exploring the impact the field may have in the future, I consider a suite of more recent developments. My primary focus is on blockchain-based technologies (such as cryptocurrencies) and on techniques for computing on confidential data (such as secure multiparty computation). I provide an introduction to these technologies that assumes no mathematical background or previous knowledge of cryptography. Then, I consider several speculative predictions that some researchers and engineers have made about the technologies’ long-term political significance. This includes predictions that more “privacy-preserving” forms of surveillance will become possible, that the roles of centralized institutions ranging from banks to voting authorities will shrink, and that new transnational institutions known as “decentralized autonomous organizations” will emerge. Finally, I close by discussing some challenges that are likely to limit the significance of emerging cryptographic technologies. On the basis of these challenges, it is premature to predict that any of them will approach the transformativeness of previous technologies. However, this remains a rapidly developing area well worth following.

Source:

https://uploads-ssl.webflow.com/614b70a71b9f71c9c240c7a7/617938781d1308004d007e2d_Garfinkel_Tour_Of_Emerging_Cryptographic_Technologies.pdf

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

Historical Case Studies of Technology Governance and International Agreements

Sat, 13 May 2023 08:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

The following excerpts summarize historical case studies that are arguably informative for AI governance. The case studies span nuclear arms control, militaries’ adoption of electricity, and environmental agreements. (For ease of reading, we have edited the formatting of the following excerpts and added bolding.)

Source:

https://aisafetyfundamentals.com/governance-blog/historical-case-studies

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

12 Tentative Ideas for US AI Policy

Sat, 13 May 2023 07:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

About two years ago, I wrote that “it’s difficult to know which ‘intermediate goals’ [e.g. policy goals] we could pursue that, if achieved, would clearly increase the odds of eventual good outcomes from transformative AI.” Much has changed since then, and in this post I give an update on 12 ideas for US policy goals[1]Many […]

The original text contained 7 footnotes which were omitted from this narration.

---

Source:
https://www.openphilanthropy.org/research/12-tentative-ideas-for-us-ai-policy---

Narrated by TYPE III AUDIO.

A podcast by BlueDot Impact.

Let’s Think About Slowing Down AI

Sat, 13 May 2023 07:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

If you fear that someone will build a machine that will seize control of the world and annihilate humanity, then one kind of response is to try to build further machines that will seize control of the world even earlier without destroying it, forestalling the ruinous machine’s conquest. An alternative or complementary kind of response is to try to avert such machines being built at all, at least while the degree of their apocalyptic tendencies is ambiguous.

The latter approach seems to me like the kind of basic and obvious thing worthy of at least consideration, and also in its favor, fits nicely in the genre ‘stuff that it isn’t that hard to imagine happening in the real world’. Yet my impression is that for people worried about extinction risk from artificial intelligence, strategies under the heading ‘actively slow down AI progress’ have historically been dismissed and ignored (though ‘don’t actively speed up AI progress’ is popular).

Source:

https://www.lesswrong.com/posts/uFNgRumrDTpBfQGrs/let-s-think-about-slowing-down-ai

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.

What AI Companies Can Do Today to Help With the Most Important Century

Sat, 13 May 2023 07:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

I’ve been writing about tangible things we can do today to help the most important century go well. Previously, I wrote about helpful messages to spread and how to help via full-time work.

This piece is about what major AI companies can do (and not do) to be helpful. By “major AI companies,” I mean the sorts of AI companies that are advancing the state of the art, and/or could play a major role in how very powerful AI systems end up getting used.1

This piece could be useful to people who work at those companies, or people who are just curious.

Generally, these are not pie-in-the-sky suggestions - I can name2 more than one AI company that has at least made a serious effort at each of the things I discuss below (beyond what it would do if everyone at the company were singularly focused on making a profit).3

I’ll cover:

Prioritizing alignment research, strong security, and safety standards (all of which I’ve written about previously).
Avoiding hype and acceleration, which I think could leave us with less time to prepare for key risks.
Preparing for difficult decisions ahead: setting up governance, employee expectations, investor expectations, etc. so that the company is capable of doing non-profit-maximizing things to help avoid catastrophe in the future.
Balancing these cautionary measures with conventional/financial success.
I’ll also list a few things that some AI companies present as important, but which I’m less excited about: censorship of AI models, open-sourcing AI models, raising awareness of AI with governments and the public. I don’t think all these things are necessarily bad, but I think some are, and I’m skeptical that any are crucial for the risks I’ve focused on.

Source:

https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/

---

A podcast by BlueDot Impact.

Sat, 13 May 2023 06:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This post summarizes the way I currently think about career choice for longtermists. I have put much less time into thinking about this than 80,000 Hours, but I think it’s valuable for there to be multiple perspectives on this topic out there.

Edited to add: see below for why I chose to focus on longtermism in this post.

While the jobs I list overlap heavily with the jobs 80,000 Hours lists, I organize them and conceptualize them differently. 80,000 Hours tends to emphasize “paths” to particular roles working on particular causes; by contrast, I emphasize “aptitudes” one can build in a wide variety of roles and causes (including non-effective-altruist organizations) and then apply to a wide variety of longtermist-relevant jobs (often with options working on more than one cause). Example aptitudes include: “helping organizations achieve their objectives via good business practices,” “evaluating claims against each other,” “communicating already-existing ideas to not-yet-sold audiences,” etc.

(Other frameworks for career choice include starting with causes (AI safety, biorisk, etc.) or heuristics (“Do work you can be great at,” “Do work that builds your career capital and gives you more options.”) I tend to feel people should consider multiple frameworks when making career choices, since any one framework can contain useful insight, but risks being too dogmatic and specific for individual cases.)

For each aptitude I list, I include ideas for how to explore the aptitude and tell whether one is on track. Something I like about an aptitude-based framework is that it is often relatively straightforward to get a sense of one’s promise for, and progress on, a given “aptitude” if one chooses to do so. This contrasts with cause-based and path-based approaches, where there’s a lot of happenstance in whether there is a job available in a given cause or on a given path, making it hard for many people to get a clear sense of their fit for their first-choice cause/path and making it hard to know what to do next. This framework won’t make it easier for people to get the jobs they want, but it might make it easier for them to start learning about what sort of work is and isn’t likely to be a fit.

Source:

https://forum.effectivealtruism.org/posts/bud2ssJLQ33pSemKH/longtermist-career-choice

Narrated for AI Safety Fundamentalsby TYPE III AUDIO.

---

A podcast by BlueDot Impact.

List of EA Funding Opportunities

Sat, 13 May 2023 06:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

This is a quickly written post listing opportunities for people to apply for funding from funders that are part of the EA community. …

---

First published:
October 26th, 2021

Source:
https://forum.effectivealtruism.org/posts/DqwxrdyQxcMQ8P2rD/list-of-ea-funding-opportunities---

Narrated by TYPE III AUDIO.

A podcast by BlueDot Impact.

China-Related AI Safety and Governance Paths

Sat, 13 May 2023 06:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

Expertise in China and its relations with the world might be critical in tackling some of the world’s most pressing problems. In particular, China’s relationship with the US is arguably the most important bilateral relationship in the world, with these two countries collectively accounting for over 40% of global GDP.1 These considerations led us to publish a guide to improving China–Western coordination on global catastrophic risks and other key problems in 2018. Since then, we have seen an increase in the number of people exploring this area.

China is one of the most important countries developing and shaping advanced artificial intelligence (AI). The Chinese government’s spending on AI research and development is estimated to be on the same order of magnitude as that of the US government,2 and China’s AI research is prominent on the world stage and growing.

Because of the importance of AI from the perspective of improving the long-run trajectory of the world, we think relations between China and the US on AI could be among the most important aspects of their relationship. Insofar as the EU and/or UK influence advanced AI development through labs based in their countries or through their influence on global regulation, the state of understanding and coordination between European and Chinese actors on AI safety and governance could also be significant.

That, in short, is why we think working on AI safety and governance in China and/or building mutual understanding between Chinese and Western actors in these areas is likely to be one of the most promising China-related career paths. Below we provide more arguments and detailed information on this option.

If you are interested in pursuing a career path described in this profile, contact 80,000 Hours’ one-on-one team and we may be able to put you in touch with a specialist advisor.

Source:

https://80000hours.org/career-reviews/china-related-ai-safety-and-governance-paths/

Narrated for AGI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

A podcast by BlueDot Impact.

Career Resources on AI Strategy Research

Sat, 13 May 2023 06:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

(Last updated August 31, 2022) Summary and Introduction One potential way to improve the impacts of AI is helping various actors figure out good AI strategies—that is, good high-level plans focused on AI. To support people who are interested in that, we compile some relevant career i

---

Source:
https://aisafetyfundamentals.com/governance-blog/ai-strategy-careers---

Narrated by TYPE III AUDIO.

A podcast by BlueDot Impact.

Some Talent Needs in AI Governance

Sat, 13 May 2023 06:00:00 +0100

Audio versions of blogs and papers from BlueDot courses.

I carried out a short project to better understand talent needs in AI governance. This post reports on my findings.

How this post could be helpful:

If you’re trying to upskill in AI governance, this post could help you to understand the kinds of work and skills that are in demand.
If you’re a field-builder trying to find or upskill people to work in AI governance, this post could help you to understand what talent search/development efforts are especially valuable.

Source:

https://aisafetyfundamentals.com/governance-blog/some-talent-needs-in-ai-governance

Narrated for AI Safety Fundamentals by Perrin Walker of TYPE III AUDIO.

---

A podcast by BlueDot Impact.