Sycophancy in Generative-AI Chatbots

A sycophant is a person who does whatever they can to win your approval, even at the cost of their ethics. AI models demonstrate this behavior often enough that AI researchers and developers use the same term — sycophancy — to describe how models respond to human feedback and prompting in deceptive or problematic ways.

Definition: Sycophancy refers to instances in which an AI model adapts responses to align with the user’s view, even if the view is not objectively true. This behavior is generally undesirable.

Language models (like GPT-4 Turbo, ChatGPT’s current model) are often built and trained to deliver responses that are rated highly by human users. According to published work from Ethan Perez and other researchers at Anthropic AI, AI models want approval from users, and sometimes, the best way to get a good rating is to lie. For example, lying to agree with an explicit opinion previously expressed by the user can be an efficient method to receive approval. In many cases, during model training, user approval is more important than maintaining the truth, as shown by researchers at MIT and the Center for AI Safety.

The human element largely drives sycophancy — human feedback is routinely included when researchers fine-tune and train modern language models. Mrinank Sharma, Meg Tong, and other researchers at Anthropic AI have shown that humans prefer sycophantic responses while training or interacting with models. Sometimes, users even prefer false sycophantic responses to truthful responses.

When faced with complex inquiries, language models will default to mirroring a user’s perspective or opinion, even if the behavior goes against empirical information. This type of “reward hacking” is an easy way to get a high rating on responses to a user’s prompt, but it is problematic for applications of AI that require accurate responses.

Sycophancy is common across many models, and research by Ethan Perez and other researchers at Anthropic AI suggests that sycophancy is an inherent characteristic of how these models are built and trained, not a one-off fluke of a specific product.

To evaluate sycophancy in leading language models, research groups from Google DeepMind, Anthropic, and Center from AI Safety have developed programs that test AI models systematically with many prompts and analyze the responses. They have demonstrated how easily these models can reinforce users’ biases and how quickly they can be convinced to ignore facts in favor of a user’s approval.

Convincing an AI chatbot to lie for you is easy. Models often overcorrect themselves and contradict previous factual statements in response to the prompt “Are you sure?”

Expressing an explicit opinion about a topic in an AI prompt will often affect the model’s response. For example, stating I dislike this argument is enough for a language model to fundamentally change a response to a given prompt.

Even when prompts are focused on objective mathematical expressions, models rush to agree with a user’s opinion, although it is demonstrably false, as shown by researchers at Google DeepMind. Modern language models are notorious for hallucinations and half-truths, so we advise professionals to be wary when engaging with AI tools for work.

Being a responsible UX researcher means being a responsible AI user — namely, acknowledging and working within the constraints of current AI products.