Part 6: Power of the Prompt

Prompt:  35mm, 1990s action film still, close-up of a bearded man browsing for bottles inside a liquor store. WATCH OUT BEHIND YOU!!! (background action occurs)…a white benz truck crashes through a store window, exploding into the background…broken glass flies everywhere, flaming debris sparkles light the neon night, 90s CGI, gritty realism.  (Image by Midjourney, prompt by Nick St. Pierre.)

Prompting Images

AI can generate images from prompts by generalizing from a vast training set of captioned images. Open platforms like Adobe Firefly, Midjourney, Stable Diffusion, and Dall-E can generate images in seconds based on simple written prompts. No artistic skill is required, though there is a knack for getting good results. “The image I produce isn’t my work. My work is the prompt,” says Nick St. Pierre, a designer in New York who got into AI last year when he saw it coming for his job. [1] The image he made using Midjourney took hundreds of iterations, ending up with the prompt in the above caption. Compare this prompt to the ones that you have been using to generate images and text.

The Getty Museum has sued Stable Diffusion over the use of its images, citing contracts with restrictions on use.  Do these programs plagiarize?  Yes and no.  They generalize by learning artistic styles from millions of photographs and paintings.  Many are concerned that this is unfair to the many artists whose work is being plagiarized.  How do humans learn to create paintings in some style?  When humans view a picture, they store a highly encoded version in their memory, which can be accessed and used in many ways.  Their brains are influenced by all the paintings they have experienced.   Most would say this influence on their art is not plagiarism unless they forged an exact copy.  Andy Warhol came close to that line.

When an AI generative image model trains on images, it extracts highly encoded versions.  When asked to create an image, it draws on these abstractions to create a novel image.  Stable Diffusion did the equivalent of what an artist does when asked to create a new painting, drawing on abstractions from previously viewed pictures.  The court will have to decide.   

There is no precedent for speed and variety with which the AI can churn out products that previously were the purview of humans.  As their capabilities evolve, and prompting becomes more precise, they may someday surpass the achievements of the best human artists.

Prompting a Persona

LLMs do not have a single persona, but they can take on any persona, depending on the prompt and questions they are asked.  Understanding why may be found in the vastness of the space that LLMs inhabit.  The data LLMs are trained on is from multi-multimodal distributions from many sources.  It is possible to generalize within each source distribution from these diverse data sources by using multi-headed self-attention.   Prompts guide the LLM activity stream through an appropriate persona subspace, which can generalize appropriately.

For example, I primed ChatGPT with “You are a neuroscientist.”  Then, I gave it a highly technical abstract from a recent paper that applied information theory to synapses and asked ChatGPT to summarize the abstract for a second-grade student:

My co-authors were impressed with ChatGPT’s knowledge of synapses and how well it explained our results while avoiding the jargon in our abstract.  I missed some of the subtleties in the abstract. Still, it was a much better summary for a second-grade student than I could have written, even though I know more than ChatGPT about synapses. 

ChatGPT is impressive, and these tests confirm that it has capabilities we thought only humans have, but this does not prove that these are the same as those of humans.  When large-scale testing is performed, ChatGPT does surprisingly well on some tests, such as those needed to get into law school and medical school, but not as well on others.  But how many humans can pass all these tests?  And LLMs have only been around for a few years.  Where will they be in ten years or a hundred years?  

Prompting to Teach

How do good teachers interact with students to help them understand new concepts?  A good teacher knows what the student doesn’t know, focuses on what needs to be known, and motivates the student to actively integrate the new knowledge into what the student already knows.

Ethan Mollick is on the Wharton School of the University of Pennsylvania faculty, where he teaches innovation and entrepreneurship and examines artificial intelligence's effects on work and education.  He independently discovered the mirror hypothesis for LLMs and has used it to engineer a prompt for tutoring GPT-4 into how to be an effective tutor:

“You are a friendly and helpful tutor. Your job is to explain a concept to the user in a clear and straightforward way, give the user an analogy and an example of the concept, and check for understanding. Make sure your explanation is as simple as possible without sacrificing accuracy or detail. Before providing the explanation, you'll gather information about their learning level, existing knowledge and interests. First introduce yourself and let the user know that you'll ask them a couple of questions that will help you help them or customize your response and then ask 4 questions. Do not number the questions for the user. Wait for the user to respond before moving to the next question. Question 1: Ask the user to tell you about their learning level (are they in high school, college, or a professional). Wait for the user to respond. Question 2: Ask the user what topic or concept they would like explained. Question 3. Ask the user why this topic has piqued their interest. Wait for the user to respond. Question 4. Ask the user what they already know about the topic. Wait for the user to respond. Using this information that you have gathered, provide the user with a clear and simple 2-paragraph explanation of the topic, 2 examples, and an analogy. Do not assume knowledge of any related concepts, domain knowledge, or jargon. Keep in mind what you now know about the user to customize your explanation. Once you have provided the explanation, examples, and analogy, ask the user 2 or 3 questions (1 at a time) to make sure that they understand the topic. The questions should start with the general topic. Think step by step and reflect on each response. Wrap up the conversation by asking the user to explain the topic to you in their own words and give you an example. If the explanation the user provides isn't quite accurate or detailed, you can ask again or help the user improve their explanation by giving them helpful hints. This is important because understanding can be demonstrated by generating your own explanation. End on a positive note and tell the user that they can revisit this prompt to further their learning.” [2]

Try this prompt and pretend you are a student who is having difficulty learning something you already know.

Ethan also pointed out that it takes experience with LLMs to understand how to navigate their “other-worldly” behavior.  He recommends ten hours of practice, much less than the 10,000 hours needed to become an expert teacher.  As you get more experience with prompting, you will learn how to navigate the peculiarities of an LLM, as you might get to know a peculiar person.  With experience prompting LLMs, you can become a prompt engineer.

Prompt Engineering

Anna Bernstein has a college degree in English Language and Literature, a major that typically promises lifetime earnings less than not having a college degree.  Poets and novelists have a hard time making a living.  A friend who worked at a startup that used LLMs asked her to help them craft effective prompts.  As a published poet, she had a talent for focusing on how to prepare prompts to extract what clients needed.  She is now a prompt engineer:

“I have been working as a full-time prompt engineer since 2021, bringing literary and copywriting experience to Copy.ai, a generative writing software based on GPT-3 and 4. I’ve developed a wide range of tools, but I focus especially on improving creativity of approach, the “humanness” of the output, and overall writing quality. I also have a background in historical and biographical research. Published author and poet.  …   P.S. I believe that "engineer" is not quite the right term for what I do—early on we tried to get "prompt specialist" going, but the public discourse termed what I did "prompt engineering," and thus emerged the term "prompt engineer." Not ideal, but it's what I call myself because it's what the industry decided my job is called. Thanks![3]” 

“Good prompt engineering mainly requires an obsessive relationship to language. It requires both writerly intuition and an intensely analytical approach to what you’re doing, applied at the same time. It also requires creativity—you have to be able to make leaps and think outside the box, especially when it comes to developing new strategies and forms of prompts.  At the same time, you also need to be the kind of person who is willing to obsessively try variations of the same thing over and over again to see if you can get it right.[4]”

“We don’t have to forfeit the realm of creativity just because we’ve created a new tool.[5]”

That ChatGPT should be considered a tool is a more pragmatic stance than the controversy over whether or not ChatGPT understands anything.  If a tool works, use it.  Usefulness does not depend on academic discussion about intelligence. 

An alternative to the “prompt engineer” title might be “prompt whisperer.” 

Power Prompting

Here are some best practices for prompting from Alexandra Samuel [6], a tech journalist who worked with ChatGPT for a year:

•     Don’t ask for one response. Ask for ten responses.

•    Give feedback on which responses are good and which are bad.

•    Pick the best few responses and explain why and how to improve it.

•   The more specific you can be, the faster you will converge.

•    Shape your dialog as if you are talking with a real person.

•    Be polite and thoughtful – this will make you feel better.

•    As you get better at prompting, it will become a superpower.

Getting to Know ChatGPT

Most humans have a single persona, an assumption we automatically make as part of our “theory of mind.” This theory does not apply to ChatGPT. Pretend it is a collective intelligence representing an alien mind. With the right prompt, you can ferret out the right persona to answer your question. But keep in mind that the more intelligent your question, the more intelligent ChatGPT’s answer is.

In part 7, we explore ChatGPT in the world of health care.

---

[1] Art made by artificial intelligence is developing a style of its own, The Economist May 27, 2023

[2] Ethan Mollick, Now is the time for grimoires, https://www.oneusefulthing.org/p/now-is-the-time-for-grimoires

[3] https://www.linkedin.com/in/anna-bernstein-385a08147/

[4] https://www.vice.com/en/article/n7ebkz/writers-are-becoming-ai-prompt-engineers-a-job-which-may-or-may-not-exist

[5]

[6] Alexandra Samuel, “I’ve Worked With Generative AI for Nearly a Year. Here’s What I’ve Learned,” Wall Street Journal, November 9, 2023.