Listen now on Apple, Spotify, and YouTube.
Brought to you by:
• WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs
• Productboard—Make products that matter
• Wix Studio—The web creation platform built for agencies
—
Keith Coleman (VP of product) and Jay Baxter (founding ML engineer), the minds behind Community Notes, reveal how a small, scrappy team inside Twitter/X built the most trusted crowdsourced information system on the internet—one that’s changing the way we understand truth online. What you’ll learn:
How Community Notes actually works—a deep dive into the groundbreaking algorithm that rewards “bridging agreement” instead of majority rule
The seemingly crazy yet brilliant way this idea survived multiple CEO changes—from Jack to Parag to Elon
How this project started with a dumpster fire GIF (literally)—the untold backstory of its early launch
The secret to running ultra-fast, high-impact product teams—no OKRs, no Jira; just one Google Doc
What Meta’s adoption of Community Notes means for the future of online (mis)information—why this open source system is becoming the industry standard
Some takeaways:
Key principles that enabled Community Notes’ success:
Universal access: Random selection of contributors rather than curated experts
Complete transparency: Open source code and data
Focus on bridging divides: Show notes that people across the political spectrum find helpful
Prove it works: Demonstrate success at each stage before expanding
Let the people decide: No “god mode” or company override of community decisions
The system’s effectiveness stems from focusing on bridging-based agreement:
Notes only appear when rated helpful by people who typically disagree
About 8% of proposed notes meet the quality threshold
Notes can reduce engagement with misleading content by 50% to 60%
Authors are 80% more likely to delete posts after receiving a note
Small, focused teams can have outsize impact when structured correctly:
Single clear owner/decision-maker
100% dedicated team members
Freedom from standard corporate processes
Start with minimal necessary roles (ML, frontend, backend, design, research)
Use lightweight tools (e.g. a single Google Doc vs. complex project management systems)
Building trust through radical transparency:
All code and data is public and downloadable
External researchers can replicate and verify results
Community members can propose algorithm improvements
Regular public updates on system changes
No hidden controls or override mechanisms
Where to find Keith Coleman:
• LinkedIn: https://www.linkedin.com/in/keith-coleman-19b12b46/
Where to find Jay Baxter:
• X: https://x.com/_jaybaxter_
• LinkedIn: https://www.linkedin.com/in/jaybaxter/
• Website: http://jaybaxter.net/
In this episode, we cover:
(00:00) Introduction to Community Notes
(06:56) How the “bridging-based” algorithm works
(13:33) The impact and scale of Community Notes
(17:24) Understanding the note publishing threshold
(21:32) Challenges and philosophies
(26:26) The effect of notes on re-sharing content
(29:41) Origin story
(35:46) Embracing small teams for big impact
(40:23) The thermal project approach
(47:47) Algorithm development and internal competitions
(50:34) An inside look at how the team operates
(58:56) Working with Elon
(01:05:30) Launching Birdwatch
(01:10:48) The core principles behind Community Notes
(01:26:15) Anonymity and pseudonymity in contributions
(01:32:17) Sustaining the project through leadership changes
(01:37:57) Future directions for Community Notes
(01:42:12) Final thoughts and optimism for the future
Referenced:
• Community Notes on X: https://x.com/CommunityNotes
• Sign up to be a Community Notes contributor: https://communitynotes.x.com/guide/en/contributing/signing-up
• The Making of Community Notes: https://asteriskmag.com/issues/08/the-making-of-community-notes
• “Readers added a Community Note to this Tweet”: https://x.com/HelpfulNotes/status/1718103364792205704
• Note-ranking algorithm: https://communitynotes.x.com/guide/en/under-the-hood/ranking-notes#matrix-factorization
• Study: Community Notes on X could be key to curbing misinformation: https://giesbusiness.illinois.edu/news/2024/11/18/study--community-notes-on-x-could-be-key-to-curbing-misinformation
• Study Finds X’s (Formerly Twitter’s) Community Notes Provide Accurate, Credible Answers to Vaccine Misinformation: https://qi.ucsd.edu/study-finds-xs-formerly-twitters-community-notes-provide-accurate-credible-answers-to-vaccine-misinformation/
• Did the Roll-Out of Community Notes Reduce Engagement with Misinformation on X/Twitter?: https://dl.acm.org/doi/10.1145/3686967
• Kayvon Beykpour on LinkedIn: https://www.linkedin.com/in/kayvz/
• Jack Dorsey on X: https://x.com/jack
• “Birdwatch gives me the creeps” tweet: https://x.com/elonmusk/status/1589454464611540992
• Blake Scholl on LinkedIn: https://www.linkedin.com/in/blakescholl/
• Creating Truthtelling Incentives with the Bayesian Truth Serum: https://www.eecs.harvard.edu/cs286r/courses/fall12/papers/DW08.pdf
• Asana: https://asana.com/
• Spaces: https://blog.x.com/en_us/topics/product/2021/spaces-is-here
• Amazon MTurk: https://www.mturk.com/
• Community notes on GitHub: https://github.com/twitter/communitynotes
• What do I think about Community Notes?: https://vitalik.eth.limo/general/2023/08/16/communitynotes.html
• X’s community-led approach: tackling inaccurate and misleading information: https://blog.x.com/en_us/topics/company/2023/xs-community-led-approach-tackling-inaccurate-and-misleading-information
• Linda Yaccarino on LinkedIn: https://www.linkedin.com/in/lindayaccarino/
• Messi-Ronaldo rivalry: https://en.wikipedia.org/wiki/Messi%E2%80%93Ronaldo_rivalry
• Supernotes paper: https://arxiv.org/pdf/2411.06116v1
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.
Lenny may be an investor in the companies discussed.
Share this post