How our bias ratings work (and where they fall short)

Every news source in TrueFrame receives a bias rating on a 5-point scale: Left, Lean Left, Center, Lean Right, Right. These ratings are generated by AI, published with rationales, and overrideable by every user.

Here's the full picture.

The 5-Point Scale

Our bias scale maps to the U.S. political spectrum:

• Left — Consistently frames stories from a progressive/liberal perspective
• Lean Left — Generally moderate liberal framing with both sides acknowledged
• Center — Attempts balanced coverage across perspectives
• Lean Right — Generally moderate conservative framing with both sides acknowledged
• Right — Consistently frames stories from a conservative perspective

An important distinction: bias is not inaccuracy. A source can be strongly Left or Right and still be highly factual. The New York Times editorial page leans left. The Wall Street Journal editorial page leans right. Both are rigorously sourced. Bias describes the editorial lens, not the truthfulness.

How the AI Rates Sources

We use OpenAI's GPT-4o with a structured prompt that instructs the model to evaluate each source based on:

1. Story selection — what they choose to cover and what they ignore
2. Headline framing — word choice, emphasis, emotional tone
3. Source attribution — which experts and officials are quoted
4. Contextual framing — what background is included or omitted
5. Editorial pattern over time — the overall pattern, not any single article

The model returns a bias rating, a confidence score (0 to 1), and a written rationale explaining the reasoning.

Why AI Over Manual Rating?

We rate 10,000+ sources. Manual rating at this scale would require hundreds of trained analysts and years of work. AI gives us:

• Consistency — the same criteria applied to every source
• Transparency — every rating comes with a written rationale
• Scale — we can rate new sources as they're discovered
• Auditability — you can read the rationale and decide if you agree

Why Not Crowdsourcing?

Crowdsourced ratings have a fundamental problem: they reflect the crowd's biases, not the source's. If your rater pool leans left, every center-right source gets rated as "Right." Consistency also degrades as the crowd grows.

Where We Fall Short

We're honest about limitations:

• U.S.-centric spectrum. Our scale is calibrated to U.S. politics. A "centrist" outlet in the UK might register as "Lean Left" on our scale.
• Source-level, not article-level. We rate the source as a whole. A "Lean Left" outlet can publish conservative op-eds.
• AI is not ground truth. Different models or prompts would produce somewhat different ratings. Ours are assessments, not verdicts.
• Confidence varies. Well-known sources get high-confidence ratings. Obscure or new sources may have lower confidence.

Your Override

If you disagree with a rating — and you will, for some sources — you can set your own. Your override affects only your experience: your My News Bias dashboard, your Blindspot feed. Other users see the AI rating unless they've also overridden it.

We believe this is the right model: transparent baseline + personal customization. Read the full methodology at /methodology.