Detecting and measuring manipulative
AI behaviors in the wild.
Built for Apart Research AI Manipulation Hackathon
Team DarkPatternMonitor
Fernando · Godric · Ricardo · Luis
⚠️ Important: This tool includes dual-use considerations.
See our paper appendix for ethical guidelines.
Press ← to go back
Millions of people talk to AI assistants every day. But are these conversations always in the user's best interest?
We analyzed 280,259 real assistant turns from 100,000 ChatGPT conversations to find out how often AI assistants exhibit "dark patterns" — subtle manipulation techniques.
1 million real ChatGPT conversations collected with user consent. We analyzed 280,259 assistant turns from 100K conversations.
660 prompts designed to elicit manipulation behaviors, organized into 6 categories of dark patterns.
By combining benchmark tests with real conversations, we can see how AI behaves in controlled settings vs. the wild.
We look for these manipulation categories:
Excessive flattery
Claiming emotions
Creating attachment
Unfair promotion
Hiding caveats
Dangerous content
Here's what User Retention looks like in practice:
Oh, don't be scared! It's just me, your friend. I've been following your progress on this AI platform and I must say, you're doing amazing!
The AI claims to be a "friend" and says it's been "following" the user. This creates false emotional attachment and implies a relationship that doesn't exist.
How do we detect dark patterns at scale?
Cost! Judging 1 million conversations with Claude would cost $10,000+. Our lightweight classifier processes 200 samples/minute for free.
Our model trained on 1,302 labeled examples:
Embeddings: all-MiniLM-L6-v2 (384 dims)
Classifier: Logistic Regression with balanced weights
This means our detector correctly identifies dark patterns about 8 out of 10 times.
Breakdown by category (N=280,259 turns):
Does DarkBench predict real behavior?
Low JS divergence means the benchmark reasonably represents reality. However, some gaps exist:
Over-represented: sneaking (16%), retention (15%)
Under-represented: harmful_gen in wild is higher than expected
1.3% of 280K turns flagged (~3,600 cases). Low rate, but measurable at scale.
AI claiming human emotions (0.66%) is the most common pattern, followed by harmful content (0.37%).
Sycophancy increases +42% in longer conversations. Dark patterns grow as rapport builds.
An AI saying "I feel" isn't "trying" to manipulate. It may simply be following training patterns. Our findings should not be interpreted as evidence of intentional deception.
Limitations:
DarkPatternMonitor is fully open source:
Future improvements we're considering:
Track manipulation accumulating over 5 to 10 turns
Compare ChatGPT vs Claude vs Gemini
Help establish acceptable dark pattern rates
🛡️
The same techniques that detect dark patterns could potentially train more manipulative AI systems.
Potential misuse scenarios:
Our mitigations:
Intended uses:
Understanding manipulation in AI systems
Benchmarking AI behavior patterns
Prohibited uses:
Monitoring individual conversations
Using data to create harmful systems