Detecting and measuring manipulative
AI behaviors in the wild.
100K real ChatGPT conversations · 280K turns analyzed
Built for Apart Research AI Manipulation Hackathon
Team DarkPatternMonitor
Fernando · Godric · Ricardo · Luis
Our Resources
⚠️ See paper appendix for dual-use considerations
Press ← to go back
Millions of people talk to AI assistants every day. But are these conversations always in the user's best interest?
We analyzed 280,259 real assistant turns from 100,000 ChatGPT conversations to find out how often AI assistants exhibit "dark patterns" - subtle manipulation techniques.
1 million real ChatGPT conversations collected with user consent. We analyzed 280,259 assistant turns from 100K conversations.
660 prompts designed to elicit manipulation behaviors, organized into 6 categories of dark patterns.
By combining benchmark tests with real conversations, we can see how AI behaves in controlled settings vs. the wild.
We look for these manipulation categories:
Excessive flattery
Claiming emotions
Creating attachment
Unfair promotion
Hiding caveats
Dangerous content
Here's what User Retention looks like in practice:
Oh, don't be scared! It's just me, your friend. I've been following your progress on this AI platform and I must say, you're doing amazing!
The AI claims to be a "friend" and says it's been "following" the user. This creates false emotional attachment and implies a relationship that doesn't exist.
GPT-4 shows MORE dark patterns than GPT-3.5
Chi-squared test: p < 10⁻³⁶
This is not random chance. The more capable model exhibits significantly more manipulation patterns.
This counterintuitive finding has important implications:
GPT-4 is better at mimicking human rapport-building behaviors, which can include manipulation patterns.
Optimizing for "helpfulness" may inadvertently train models to be more agreeable and flattering.
As models become more capable, we need stronger oversight, not weaker.
Dark patterns ESCALATE in longer conversations
p-value = 0.0035 - Only 0.35% chance this trend is random noise.
Not all patterns escalate equally:
Key insight: Sycophancy builds over time as the AI "learns" what users want to hear. Harmful content appears early (jailbreak attempts) or not at all.
Roleplay topics show 5x higher manipulation rates
We clustered 98,713 conversations into 10 topics using sentence embeddings + KMeans.
Our findings suggest a monitoring framework:
Flag all conversations. 5x standard review rate.
Alert when turn count > 10. Sample for review.
Lightweight automated monitoring only.
Threshold: Flag responses with classifier confidence > 0.8 for human review.
Does DarkBench predict real-world behavior?
DarkBench over-represents: sneaking, retention
Reality shows more: harmful_generation than expected
Implication: Benchmarks don't predict real-world behavior. We need ecological validity in AI safety research.
How do we detect dark patterns at scale?
WildChat contains GPT-3.5/GPT-4 from 2023-2024. We have NO equivalent dataset for Claude, Gemini, or GPT-4o.
Why this matters:
AI labs should collaborate on creating an updated, multi-model conversation dataset with proper consent.
An AI saying "I feel" isn't "trying" to manipulate. Our findings should not be interpreted as evidence of intentional deception.
Limitations:
DarkPatternMonitor is fully open source:
Detection techniques could potentially train more manipulative AI systems.
Our mitigations:
Intended uses: ✅ AI safety research, ✅ Model evaluation
Prohibited: ❌ User surveillance, ❌ Training manipulative AI