Part 12: How to Download a Brain

This Substack has been tracking a new book on “ChatGPT and the Future of AI” released on October 29, 2024.

Table of Contents:

Preface

Part I: Living with Large Language Models

1          Introduction

2          How Chatbots Are Changing Our Lives

3          Interviews with Large Language Models

4          The Power of the Prompt

5          What Are Intelligence, Thinking, and Consciousness?

Part II: Transformers

6          Origins of Deep Learning

7          High-Dimensional Mathematics

8          Computational Infrastructure

9          Superintelligence

10        Regulation

Part III: Back to the Future

11        Evolution of AI

12        The Next Generation

13        Learning from Nature

14        The Future Is Now

Afterword

Acknowledgments

Glossary

Notes

Index

The book's three parts are like three scenes in a play where Part I lays out the strengths and weaknesses of ChatGPT today, Part II takes a deep dive into how it works, and Part III explores how ChatGPT is evolving and what to expect in the future.  A lot has happened in AI since the book went to press in September, and this Substack allows me to bring it up to date.  The book did not anticipate the Nobel Prizes for AI described in the last Part. [1]

The Transmitter

The Transmitter is a neuroscience newsletter supported by the Simons Foundation. [2] The following excerpt from my book, in the spirit of my Substack title, Brains and AI, will soon appear online:

Are Brains and AI Converging?

Research on brains and AI are based on the same basic principles: massively parallel architectures with a high degree of connectivity trained with learning from data and experience.  Brain discoveries made in the twentieth century inspired new machine learning algorithms: the hierarchy of areas in the visual cortex inspired convolutional neural networks, and operant conditioning inspired the temporal difference learning algorithm for reinforcement learning.  In parallel with the advances in artificial neural networks, the BRAIN Initiative has accelerated discoveries in neuroscience in the twenty-first century by supporting the development of innovative neurotechnologies.  Machine learning is being used by neuroscientists to analyze simultaneous recordings from hundreds of thousands of neurons in dozens of brain areas and to automate the reconstruction of neural circuits from serial section electron microscopy.  These advances have changed how we think about processing distributed across the cortex and led to discoveries that created a new conceptual framework for brain function, leading to even more advanced and larger-scale neural network models.

The new conceptual frameworks in AI and neuroscience are converging, accelerating their progress.  The dialog between AI and neuroscience is a virtuous circle that is enriching both fields.  AI theory is emerging from analyzing the activity patterns of hidden units in ultra-high-dimensional spaces, which is how we study brain activity.  Analyzing the dynamics of the activity patterns in large language models (LLMs) may lead us to a deeper understanding of intelligence by uncovering a common underlying mathematical structure.  For example, an LLM was trained on board positions for the game Othello and was probed to reveal an internal model for the rules of Othello.

Now that we can interrogate neurons throughout the brain, we may solve one of its greatest mysteries: how information globally distributed over so many neurons is integrated into unified percepts and brought together to make decisions.  The architectures of brains are layered, with each layer responsible for making decisions on different time scales in both sensory and motor systems.  We can build deep multimodal models with many component networks and integrate them into a unified system, giving insights into the mechanisms responsible for subconscious decision making and conscious control.

How to Download a Brain

Neurons are traditionally interrogated in the context of discrete tasks, such as responses to visual stimuli, in which the choices and stimuli are limited in number.  This tight control of stimulus and response allows the neural recordings to be interpreted in the context of the task.  But neurons can participate in many tasks in many different ways, so interpretations derived from a single task can be misleading.  We now can record from hundreds of thousands of neurons brain-wide, and it is also possible to analyze recordings and dissect behavior with machine learning.  However, neuroscientists are still using the same old single-task paradigms.  One solution is to train on many different tasks, but training a monkey, for example, takes weeks to months for each task.  Another solution is to expand the complexity of the task over longer time intervals, bringing it closer to natural behaviors.

There is an even more fundamental problem with approaching behavior by studying discrete tasks.  Natural behaviors of animals in the real world are primarily self-generated and interactive.  This is especially the case with social behaviors.  Studying such self-generated continuous behaviors is much more difficult than studying tightly constrained, reflexive ones.

What if an LLM were trained on massive brain recordings during natural behaviors and accompanying behavior, including body and eye tracking, video, sound, and other modalities?  LLMs are self-supervised and can be trained by predicting missing data segments across data streams.  This would not be scientifically useful from the traditional experimental perspective, but it does make sense from the new computational perspective afforded by LLMs.

A large neurofoundation model (LNM) can be trained on brain activity and behavior under natural conditions in the same way we now train LLMs.  The resulting LNM could be interrogated on many new tasks just as pre-trained LLMs respond to novel queries and can be used to perform many new tasks.  These pre-trained LNMs would be as costly to train as LLMs, but once an LNM is pretrained, it could provide a common resource for the scientific community to probe and analyze.  This would revolutionize how brains are studied, with the bonus of reducing the number of animals needed for research.  Human brain activity from an individual could be similarly used to train a suitably advanced LNM, creating an immortal generative version of that person.

It may sound like science fiction, but Gerald Pao at the Okinawa Institute for Science and Technology has already achieved this in flies and zebrafish larvae that have around 100,000 neurons.  Almost all the neurons were optically recorded as light flashes from fluorescent dyes sensitive to neural signals while monitoring behavior.  The spontaneous behavior Pao studied was the escape behavior from anoxia—reduced oxygenation—in zebrafish larvae and walking behavior in flies.  He used a method from dynamical systems theory called convergent cross mapping (CCM), introduced by George Sugihara at the Scripps Institution of Oceanography, University of California at San Diego, to extract causal relationships between recorded neurons and behavior.  This method extracts a reduced graphical model that captures the low-dimensional brain subspaces that control the behaviors.  Recordings from around 100,000 neurons were analyzed with a supercomputer at the AI Bridging Cloud Infrastructure (ABCI) in Japan.  When the model was turned on, the spontaneous behaviors it generated were indistinguishable from those observed in vivo.  The key was to analyze both the neural recordings and the behaviors simultaneously.  Analyzing either alone was insufficient to reproduce the behavior.

This is proof of principle that brain activity and behavior can be downloaded into a model when sufficient simultaneously recorded data from both brain and behavior are available.

---

[1] Typos invariably slip through even the most careful editing and this book is no exception.  On page 78, I described a debate on whether ChatGPT understands language on December 1, 2020—impossible since this was two years before ChatGPT was released.  If you come across a typo please let me know:  terry@salk.edu

[2] Jim Simons was a distinguished Professor of Mathematics before founding Renaissance Technology, a highly successful hedge fund that made him a billionaire. His foundation supports math institutes and many scientific research projects.