Summary:
Users add specificity to their prompts by outlining the structure of the desired response.
Outlines are standard in writing. They help writers organize the high-level points to be made in a document and define its structure.
Many users of generative-AI chatbots use outlines in their prompts to guide the bot to a desired response structure. Users tend to use this tactic when they aim to generate complex outputs with multiple parts, such as marketing plans, project plans, interview guides, and surveys.
Definition: Response outlining refers to including format or structure specifications in the prompt to ensure that the chatbot’s response satisfies those requirements.
An Example of Response Outlining
Some participants used response outlining at the very beginning of a conversation with an AI bot. For example, a participant trying to create an interview guide for a research study numbered each of his research questions and subquestions (1, 2, 3, 1a,1b, etc.) and included them in his first prompt, to ensure that the interview-guide questions generated by ChatGPT would follow the same numbering.
However, participants often adopted response outlining only after getting unsatisfactory answers from the chatbot in the first attempt. A marketing professional who was trying to create a marketing plan with ChatGPT initially used this prompt:
“Create a marketing plan for a wellness app called “Mynd and Bodi” that focuses on yoga mindfulness and therapy.”
The participant was dissatisfied with the result because it did not include a SWOT analysis or a list of competitors. She expected these sections to be part of a marketing plan. Thus, in her following prompt, she outlined the elements of her desired response:
“Create a marketing plan that includes a SWOT analysis and potential competitors for a wellness app called ‘Mynd and Bodi’ that has a focus on yoga mindfulness and therapy […]”
The participant was more satisfied with the resulting output:
“This is a little better. Yeah, I could work with this as a good outline. That’s a good competitor analysis. It even mentions Headspace (the mental health and wellness app). This looks like a solid SWOT as well.”
A response outline can include both structural and format specifications. For instance, a participant included both the structure and the format (bullet points) of the desired response in her prompt for creating a study plan. She also used the apple-picking technique to ensure that a part of a previous response would be included in the final output.
Running into the Articulation Barrier
While believed by participants to be an effective strategy, response outlining has a high cognitive load and interaction cost, because it forces users to think about the relevant parts of the expected output and then list them in the prompt. Some people may not be able to formulate their requirements from the very beginning — rather, they may do so only when they receive an unsatisfactory response. For instance, our study participant realized she needed to ask for the SWOT analysis and the list of competitors only after she received a response that missed these elements.
Using these techniques required bridging the articulation barrier. Some users may be unable to add specificity even if they wanted to, because they don’t know the jargon needed to produce the desired output.
Furthermore, while additional specifications can result in better responses, including them requires:
- More cognitive effort: AI can’t read our minds — it takes effort to define what we want and think of how to describe it.
- Typing effort: Specificity often requires long, detailed descriptions, and the typing effort can be substantial.
GUI elements can be added to chatbots’ UIs to create hybrid AI interfaces that reduce users’ cognitive load and typing effort. For example, when asking the generative-AI tool to create a text artifact, the interface could dynamically suggest points to include or ask for specifications from the user rather than let the user fail the first time and expect them to refine their prompt later.
Conclusion
Response outlining is a commonly used tactic for achieving comprehensive responses from chatbots. While effective, this tactic increases users’ effort. People may realize the need to employ response outlining only after getting unsatisfactory responses from the AI tool. Helping users adding extra specificity to prompts will reduce the articulation barrier and improve the usability of generative-AI interfaces.
Our Research
To uncover common AI user behaviors and usability issues in AI-chatbot interfaces, we conducted a qualitative usability study with professionals and students who use chatbots daily. Participants used ChatGPT 3.5 and ChatGPT 4.0. to perform tasks that were representative of how they routinely used AI. We cover all the details of our research in our article Researching the Usability of Early Generative AI Tools.
