Preference Model @preferencemodel
A superintelligence data research company. preferencemodel.com San Francisco Joined August 2025-
Tweets14
-
Followers561
-
Following45
-
Likes47
Are you surprised by the results? Let us know ai-plays-slay-the-spire.com
"A weak AI might learn the task as intended. A strong AI would learn that sometimes the prompt is untrustworthy." Jennifer Zhou on building RL environments for superintelligence:
Wrote on how we're thinking about building RL environments for superintelligence
Congrats to client @preferencemodel on the $16M in seed funding & emergence from stealth! Additionally, they announced that they are open-sourcing Karotte, its framework for building robust RL environments that inherently mitigate reward hacks. Learn more: wsgr.com/en/insights/wi…
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
Have you tried Karotte yet?
Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.
Here is how you can start building env with Karotte: preferencemodel.com/blog/karotte-a… Thank you @Fe_li__pe_ for all of your work!
Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.
I’m very exited for this launch, making sure RL envs are robust and don’t encourage misaligned behavior is crucial for safe and aligned AI!
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
Did we mention we're at #COLM2026 Stop by our booth all day or join us for a pilates on Thurs, Oct 8th — rsvp required Get on the list: partiful.com/e/FbKYiB91E1AC…
Tune in to @MTSlive today at 3:30pm PDT to hear from @chem_safety. It's gonna be good!
OPENAI MATH NUKE | NVIDIA INVESTMENTS | NATIONAL COMPUTE GRID x.com/i/broadcasts/1…
So excited to back Jennifer Zhou and Ning Cao at @preferencemodel They’re open-sourcing Karotte today, the framework they've used to build RL environments for frontier labs. These RL envs live or die on graders models relentlessly try to game, so Karotte bakes in defenses hardened over 1M eval runs to keep models focused on real tasks instead of loopholes. Preference Model’s target is ML engineering itself, the hardest domain to build envs for and the one every lab wants. Models building models starts here. Shoutout to @Mascobot who was a great partner on this one @chem_safety @Ning_Catsnail
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
Karotte makes reward hacking hard by default.
Building robust RL environments is important because the current bottleneck on AI progress is misalignment, and developing aligned models requires robust, correct reward functions. Karotte aims to make that easy by providing secure, opinionated defaults.
Our team, Preference Model, is backed by $16M in seed funding from @a16z, @SignalFire , @spc, Scale Angel Group, and researchers including @drfeifei, @goodfellow_ian, and @Mononofu, and we've shipped ~$15M in revenue using Karotte in the past month.
Today we are open-sourcing 🥕Karotte, our framework for building RL environments. We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.
Building robust RL environments is important because the current bottleneck on AI progress is misalignment, and developing aligned models requires robust, correct reward functions. Karotte aims to make that easy by providing secure, opinionated defaults.
Aryan Gosaliya @flashgosaliya
0 Followers 17 Following
Weilun Chen @snortydogg
248 Followers 458 Following Ex-coding@GoogleDeepMind. Now building https://t.co/TcKljLQjhI with coworkers who've solved Millennium Prize Problems.
Marcus @MarcusCato012
0 Followers 39 Following
Adithya Giridharan @AG_1698
21 Followers 949 Following Reinforcement Learner for Life Original proponent of the concept of Agent Trajectory & Recursive Self Improv I knowledgeGraph anything Firm & Fluid Believer
Zhengrui Ma @mzr318
1 Followers 23 Following
Sancia @Sancia69490
5 Followers 304 Following
Xiangyi Chen @xiangyi1900
3 Followers 362 Following
muskan @muskanCEX
231 Followers 4K Following Web3 Growth Partner | Token Listings | Marketing Campaigns | KOL Network | Community Expansion | 📩 Business Inquiries: DM
sxsyqa @sxsyqa55012
7K Followers 7K Following Web3: full-time airdrop, sharing Alpha; focus on the first chain of opportunities, try not to do second leeks, salute every
Christian Catalini @ccatalini
27K Followers 6K Following Founder w/roots in academia. Founder @MIT Cryptoeconomics Lab. Past: Co-Founder & Chief Strategy Officer, Lightspark. Co-Creator, Libra. Head Economist, Meta.
Sammy Sidhu @Sammy_Sidhu
472 Followers 232 Following CEO at @eventualai Building at the frontier of data for physical AI and robot understanding. Prev: self-driving at DeepScale (acq by Tesla) and Lyft Level 5.
harish @hari65535
143 Followers 1K Following AI Engineering | Staff Engineer | Learning evals, post-training, SFT, RL
Andrew Turk @DrATurk
654 Followers 816 Following Dr. Turk is highly trained and qualified as a board-certified face, breast and body surgeon specializing exclusively in cosmetic surgery.
Tushar Amrit @pyaaraacetamol
760 Followers 952 Following happinessmaxxing building the next gen Software Factory at @Abnormal
Maggie 🌙 @lulushiny132
28 Followers 523 Following Come start something cute — I like confident people 🦋
ada cyborg (🤖, �... @adacyborg11
7K Followers 5K Following researcher,designer,transhumanist artist🤖fun@superlofi ,🦋quantum art @sothebys 🐶 econ @lsenews cs @uoft
Jay • GoChamp.ai @heyjayhuffman
2K Followers 3K Following Founder @GoChampAI. We’re talking sports. • WifeGuy - DadGuy • Music - Movies - Space - Sports • Go Bucs - Go Braves - Go Bolts - Go Cats
matt cummins @cummins_tweets
396 Followers 520 Following AI @MadeWithCapsule | Stanford PhD | Biodigital jazz, man
Paige @paigeinsf
4K Followers 2K Following First principle & Longtermism | building something @se3labs Founding team @openmind_agi, ex @McKinsey
polar @pol3141592
10 Followers 47 Following
Arithmos @Arithmosxyz
41 Followers 192 Following Hot takes on AI, tech & markets daily. Then we ask who actually wins ↓
Ren @renjpgs
8 Followers 169 Following A futuristic (AI Researcher) building and experimenting with how we see consumer-tech with @withtammam
Shelly Schwartz @shelly_schwa
24 Followers 40 Following MTS @ Charta Health, Research @ UVA, Evals for Radiology VLMs
Adam Wang @adamwyc
62 Followers 238 Following
jp @jeevanpillay
3K Followers 2K Following autoresearch & reasoning // founder @lightfastai https://t.co/ljihJtsA2K
Harsh Pare @harsh_pare
19 Followers 1K Following building @tazolabs | all things AI | ex-deshaw, @bridgetownai, @EverpureData | @iitdelhi
James Junyu Guo @james_junyuguo
111 Followers 1K Following LLM Reasoning | Trustworthy ML | PhD @UCBerkeley
Samuel Parra Zancra @samuel_zancra
31 Followers 450 Following Engineer putting Latam on the global AI map | Investor | Diferente dos Iguais 🇧🇷🇺🇸🇵🇾🇸🇬
mg @astronot1999
0 Followers 470 Following
ms 🇵🇰🇨🇦 @shahrukh_here1
3 Followers 98 Following building with extra-ordinary grit, former GTM professional, cricketer
Alexander Fred-Ojala @alexfo
529 Followers 5K Following Head of AI @eqtventures | Predli | @ucberkeley
Thinking Machines @thinkymachines
184K Followers 1 Following Thinking, beeping, and booping. @tinkerapi
Mira Murati @miramurati
1.0M Followers 644 Following Now building @thinkymachines. Previously CTO @OpenAI
Jennifer Li @JenniferHli
15K Followers 2K Following GP @a16z AI x Infra 💙 data, AI and dev tools. Chasing the Pareto frontier.
Lenny Rachitsky @lennysan
453K Followers 3K Following Deeply researched product, growth, and career advice
Marc Andreessen 🇺�... @pmarca
6.7M Followers 34K Following You’re not talking to someone who woke up a loser. That loser attitude, that loser premise makes no sense to me.
swyx @swyx
203K Followers 4K Following achieve ambition with intentionality, intensity, integrity & insanity. affiliations: - @smol_ai - @dxtipshq - @cognition - @aidotengineer - @latentspacepod
Barry McCardel @barrald
4K Followers 1K Following co-founder and CEO of Hex (https://t.co/hbgguInF1h / @hexanalytics) - former @PalantirTech @formationbio - personal site: https://t.co/c38nDG5Dfl
Sammy Sidhu @Sammy_Sidhu
472 Followers 232 Following CEO at @eventualai Building at the frontier of data for physical AI and robot understanding. Prev: self-driving at DeepScale (acq by Tesla) and Lyft Level 5.
Stripe @stripe
294K Followers 558 Following Building programmable financial services. Incorporate: @atlas. Help: @stripesupport. Read: @stripepress @WorksInProgMag. Status: @stripestatus.
Sam Altman @sama
6.4M Followers 1K Following The mission of OpenAI is to ensure that AGI benefits all of humanity
Alexandr Wang @alexandr_wang
818K Followers 933 Following chief ai officer @meta, founder meta superintelligence labs, founder @scale_ai. give everyone the adulthood they dreamed of in childhood
Nikita Bier @nikitabier
1.4M Followers 2K Following unemployed. former head of product @x, ex-founder @thetbhapp (acq by @meta), ex-founder @gasappteam (acq by @discord)
Google DeepMind @GoogleDeepMind
1.6M Followers 266 Following The engine room of @Google. Building AI safely and responsibly to solve the world’s most complex problems. Join us: https://t.co/jUHQA27iBL
Anthropic @AnthropicAI
1.8M Followers 2 Following We're an AI safety and research company that builds reliable, interpretable, and steerable AI systems. Talk to our AI assistant @claudeai on https://t.co/FhDI3KQh0n.
OpenAI @OpenAI
5.4M Followers 4 Following OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity. We’re hiring: https://t.co/dJGr6LgzPA
Matt Zhang @UnderwatrCities
141 Followers 392 Following
ICML Conference @icmlconf
89K Followers 11 Following Int'l Conf on Machine Learning • This account not monitored, Contact: https://t.co/6saHKWVxR6 • LinkedIn: https://t.co/YlU0lrSVna
Nick McGreivy @NMcGreivy
1K Followers 515 Following rl environments @preferencemodel | prev: physics & ml phd @princeton
SemiAnalysis @SemiAnalysis_
175K Followers 38 Following
Michael Chang @mmmbchang
6K Followers 2K Following Sora/Robotics @openai Pilates instructor at BODYROK Prev: Gemini+Project Astra @GoogleDeepMind, @LangChainAI, @MetaAI, @SchmidhuberAI PhD @berkeley_ai BS @MIT
South Park Commons @spc
46K Followers 789 Following A community of technologists helping each other go from -1 to 0.
Julian Schrittwieser @Mononofu
32K Followers 137 Following Technical Staff at Anthropic, views are mine alone prev AlphaGo, AlphaZero, MuZero, AlphaProof, Gemini RL etc at Google DeepMind
Ian Goodfellow @goodfellow_ian
383K Followers 1K Following Co-founder of stealth startup. Inventor of GANs. Lead author of https://t.co/M6vl8pEQ4I Founding chairman of @pubhealthaction
Fei-Fei Li @drfeifei
1.1M Followers 1K Following Cofounder/CEO @theworldlabs, Prof (CS @Stanford), Co-Director @StanfordHAI, #AI #SpatialIntelligence #GenAI #computervision #robotics #AI-healthcare
JosH200 @josh_wills
18K Followers 2K Following Engineering at @datologyai; ex-@slackhq. I like Data Loaders and @duckdb.
Michelle Lazzar @MichelleLazzar
954 Followers 1K Following Infra marketing @a16z. BASE, @docker, HDP alum. Bay Area born and raised. Everything tech and pop culture. Tweets are my own.
Haoli Yin @HaoliYin
1K Followers 2K Following rsi & multimodal data curation @datologyai, 24/7 poaster, codex wrapper
SignalFire @SignalFire
7K Followers 324 Following Venture capital engineered to ignite your growth. Founded as a tech company. Built like the startups we fund. https://t.co/yjltHaTl8W
a16z @a16z
1.1M Followers 64 Following https://t.co/A9eTFq6Xbx https://t.co/3RHK5vqQo7 Posts are not investment advice or an advertisement for investment services. See https://t.co/nX2FtaLE06.
Jason Zuo @LittleDijsktraZ
6 Followers 20 Following
Felipe @Fe_li__pe_
29 Followers 34 Following
Pala @mpalanieppan
387 Followers 358 Following Software engineer. Isha Yoga practitioner and volunteer.
alvin @alvind319
411 Followers 4K Following building @preferencemodel research @aieleuther; prev @datologyai @discord @sixteenzero @facet_ai (acq. @adobe) drawbridge (acq. @linkedin)
You might like
