Part 4: The Reverse Turing Test

In June 2022, I read an article in The Economist about Large Language Models (LLMs) before OpenAI released ChatGPT to the world and before I tried using it myself.  The author thanked GPT for helping him write the essay [1].  At the end of the article, two interviews with LLMs came to opposite conclusions about the intelligence of ChatGPT: one interviewer concluded that the LLM had a theory of mind, which is a high cognitive function; the second interviewer came to the opposite conclusion that the LLM was clueless.  I wondered how two experts could have such extremely different views.  So, I interviewed ChatGPT and discovered that the interviewers were looking for intelligence in the wrong place.  This seed grew into ChatGPT and the Future of AI.

Interview 1:  Blaise Agüera y Arcas’s Interview with LaMDA

Blaise Agüera y Arcas, a Vice President and Fellow at Google Research, interviewed LaMDA, a Large Language Model with 137 billion weights created at Google Research, and found that LaMDA could understand social concepts and could model theory of mind, held by some to be the “trick” behind consciousness.

Here is what Blaise concluded from his dialog with LaMDA: “What makes that exchange impressive is not the mere understanding that a dandelion is a yellow flower, or even the prediction that it will get crushed in Mateo’s fist and no longer be lovely, but that this may make Lucy feel slighted, and why Ramesh might be pleased by that.  In our conversation, LaMDA tells me what it believes Ramesh felt that Lucy learned about what Mateo thought about Lucy’s overture.  This is high order social modelling.  I find these results exciting and encouraging, not least because they illustrate the pro-social nature of intelligence.”

Interview 2:  Douglas Hofstadter’s Interview with GPT-3

Douglas Hofstadter, a cognitive scientist and Pulitzer Prize winner for non-fiction, talked with GPT-3, a Large Language Model with 175 billion weights created at OpenAI, and concluded that GPT-3 was clueless and lacked common sense.

ChatGPT’s responses to Hofstadter are florid hallucination, a form of confabulation to which LLMs are prone.  Here is what Hofstadter said about his dialog with ChatGPT:  “I would call GPT-3’s answers not just clueless but cluelessly clueless, meaning that GPT-3 has no idea that it has no idea about what it is saying.  There are no concepts behind the GPT-3 scenes; rather, there’s just an unimaginably huge amount of absorbed text upon which it draws to produce answers.  But since it had no input text about, say, dropping things onto the Andromeda galaxy (an idea that clearly makes no sense), the system just starts babbling randomly—but it has no sense that its random babbling is random babbling.  Much the same could be said for how it reacts to the absurd notion of transporting Egypt (for the second time) across the Golden Gate Bridge or the idea of mile-high vases.”

How could these two experts come to such diametrically opposed conclusions?  The Turing Test was devised to determine whether an AI could convince a human that it was talking to another human.  I began to suspect that ChatGPT was not really hallucinating but rather was mirroring the intelligence of the human, so I used the scientific method to find out what was going on behind the scenes.

The nonsense answers to Hofstadter’s questions were first replicated and then compared to the same questions after receiving a prompt with explicit guidance for handling nonsense sentences.   ChatGPT was not clueless when adequately primed.  But does GPT know why a question is nonsense?

When properly prompted, ChatGPT offers perfectly reasonable explanations.  If spoken by a human, these responses would be considered reasoned. 

ChatGPT can recognize nonsense, and explain why, when given proper guidance.  Hofstadter’s comment about the cluelessness of LLMs has revealed more about his naïve priming of ChatGPT than about the intelligence of LLMs.  How would you react if asked the same nonsense questions?  You might also play along with the nonsense, tongue in cheek.

But there is more to an explanation than just telling a story.  An explanation should also be able to predict what would happen if things were different, which is called counterfactual reasoning.  Here is an example of counterfactual questions in astronomy.

Once again, ChatGPT passed the test. 

The Mirror Hypothesis

These experiments gave rise to a possible explanation for the diverging opinions that I called the Mirror Hypothesis.  The more intelligent you are and the more thoughtful your prompts, the more intelligent the LLM appears to be.  In mirroring the interviewer, LLMs may be effectively carrying out a sophisticated Reverse Turing Test,[2] in which the LLM tests the intelligence of the interviewer by mirroring the intelligence of the interviewer’s prompts.  LMMs have an exceptional ability to mimic many human personalities.

For example, an LLM was fine-tuned to mimic Jane Austin's personality.   You can talk with Ms. Austin about her life and books with Meta’s WhatsApp.[3]

“In a WhatsApp text conversation this week, we asked Jane Austen — yes, the 19th-century British author — how she felt about Mr. Darcy, a character from one of her most famous works, “Pride and Prejudice.”

After a few seconds, Ms. Austen responded.

“Ah, Mr. Darcy.  Everyone remembers him as one of my characters,” she said, her face appearing in a small window above our conversation.  “But fewer people have read one of my books,” she added, with an arched eyebrow and what seemed like a hint of resentment.”

When we asked at what age a woman should marry, she refused to answer.

 “My goodness, you want me to dictate your love life?” she said.  “Marry whenever you find someone who can tolerate your eccentricities.  And you theirs.”

The startup Character.ai has created hundreds of characters you can chat with, including Elon Musk and the Italian plumber SM64 Mario.

We are a moral species that creates rich cultures.   Here is ChatGPT’s response to a group of philosophers who primed it with their views, asked it a series of philosophical questions, and received philosophical answers:[4]

But keep in mind that ChatGPT was mirroring the thoughts of the philosophers:  This is more evidence for the mirror hypothesis and the Reverse Turing Test.

In Part 5, ChatGPT takes us to our deepest dark secrets.

---

[1] Artificial neural networks are making strides towards consciousness, The Economist, June 9, 2022.

[2] Sejnowski, T. J. (2023) Large language models and the reverse Turing test, Neural Computation, 35, 309-342.

[3] M. Iasaac and C. Metz, “Meet the A.I. Jane Austen: Meta Weaves A.I. Throughout Its Apps, New York Times, September 28, 2023.

[4] Weinberg, J. (2020). Philosophers on GPT-3 (Updated with Replies by GPT-3).  Daily Nous, July 30. http://dailynous.com/2020/07/30/philosophers-gpt-3

https://drive.google.com/file/d/1B-OymgKE1dRkBcJ7fVhTs9hNqx1IuUyW/view