Mercor @mercor

"The entire way that humans contribute to the economy is going to move.” @mercor Co-founder & CEO @BrendanFoody explains what he believes will replace repetitive human work.

The next major breakthroughs in AI will require us to create entirely new sources of training data. Scientific research is an example. Most of what we know about experiments comes from published papers, which capture only a fraction of the work scientists do. @LiamFedus is building @periodiclabs to change that, with physical labs that generate the experimental data needed to train AI scientists. More from our conversation on The Frontier ⬇️

"90% of American working hours are spent on tasks with little or no coverage from existing benchmarks." Building great benchmarks is extremely high leverage to making AI more economically valuable. Can't manage what you don't measure.

Mercor @mercor

3 days ago

Mercor Research mapped AI benchmarks to actual US jobs. Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark. Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽

17 27 318 76K 384

The results show how a relatively small amount of expert-built training data can improve an open model’s performance on professional work. Open models plus realistic benchmarks and targeted training data create a viable path to domain-specific agents. Congrats to @NVIDIAAI on the Nemotron 3.5 Super VL launch. And to @Harvey, @Cognition, and @tryRamp for partnering with us on this work. Full blog here: mercor.com/blog/mercor-nv…

On the APEX-Accounting benchmark, mean rubric score rose from 34.6% to 41.9% (+7.3 points). Tasks scoring zero fell from 19% to 13%. The post-trained model began by reviewing all available files before starting work (+38 points on planning behavior). It also opened significantly more source records per task, from 8.5 to 12.

Congratulations to @NVIDIAAI for launching Nemotron 3.5 Super VL. We worked with our partners @tryRamp, @Harvey, and @Cognition along with the NVIDIA team to post-train and improve the model’s performance using targeted datasets from our APEX benchmarks. We tested the post-trained Nemotron models on legal, software engineering, and accounting tasks it had never seen. The post-trained model improved across all three professional domains. Read the full results: mercor.com/blog/mercor-nv…

The barrier to automating everything in the economy is that we need to build RL environments across every industry, job, and task

Mercor @mercor

3 days ago

Mercor Research mapped AI benchmarks to actual US jobs. Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark. Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽

17 27 318 76K 384

Mercor Research mapped AI benchmarks to actual US jobs. Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark. Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽 𝗼𝗳 𝘁𝗵𝗲 𝗨𝗦 𝗘𝗰𝗼𝗻𝗼𝗺𝘆, an interactive breakdown of which economic tasks are covered by AI benchmarks. Benchmarks are how we measure AI’s progress and anticipate its impact on real work. They’re crucial to understand which jobs AI can automate, where the biggest gaps are, and how those capabilities are changing over time. The Benchmark Map is our first step toward building a resource that makes it easier to answer: 𝘋𝘰𝘦𝘴 𝘵𝘩𝘪𝘴 𝘈𝘐 𝘮𝘰𝘥𝘦𝘭 𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘮𝘺 𝘰𝘤𝘤𝘶𝘱𝘢𝘵𝘪𝘰𝘯? Explore it here: mercor.com/research/labor…

1/ The most important products usually look much more obvious in retrospect. OpenAI had GPT-4 internally and was still debating what to build around it. @LiamFedus was there for the decision that eventually became ChatGPT. We talked about that period, what changed once ChatGPT launched, and what he’s building now at Periodic on The Frontier. Full episode below.

soon all progress will be data bottlenecked, not intelligence bottlenecked. bio? data bottlenecked. astronomy? data bottlenecked. data all the way down

1/ ChatGPT had the internet. AI for science has no equivalent. @LiamFedus helped create ChatGPT and went on to lead post-training at OpenAI. Now he’s building Periodic Labs because better models alone won’t get us to an AI scientist. We talked about why Periodic is building physical labs and what Liam learned from being inside OpenAI as ChatGPT took off on today's episode of The Frontier.

Mercor is building a world-class research team. As the leading AI data provider, we are uniquely positioned to combine benchmarks, data production, model training, and economics research to advance model productivity. We are committed to sharing our findings with the world. DM me if this sounds exciting. mercor.com/blog/why-merco…

@edwardjhu @MinyangTian1 @hrdkbhatnagar @bertievidgen View open roles on our research team: jobs.ashbyhq.com/mercor?departm…

Introducing the Mercor Research team. We’re building a world-class research organization, including @edwardjhu (first author of LoRA), @MinyangTian1 (SciCode, CritPt), Victor Barres (τ²-bench, τ-voice), @hrdkbhatnagar (PostTrainBench), @bertievidgen (APEX-Agents) and many others. As models get better, the questions get harder: → How do you measure a model on a job that AI itself is changing? → What data helps when a model fails at real work? → How much can a small amount of expert data improve a model? Mercor Research covers the full AI flywheel. APEX shows where a model struggles. Experts help us understand why and create data for those weak points. We post-train the model and measure it again. Learn more about our research mission: mercor.com/blog/why-merco…

Claude Haiku 5.5 debuts at #7 on APEX-SWE. 55.1% Pass@1, ahead of last generation's Opus and Sonnet models. It achieves this score using 1/20th of Sonnet 5.5's input and output cost. Congrats @AnthropicAI on the launch. 𝗖𝗼𝗱𝗶𝗻𝗴 Anthropic positions Haiku 5.5 as a coding subagent next to Opus 5.5 and Sonnet 5.5. Its results on APEX-SWE suggest this model is a strong coding partner. APEX-SWE: 55.1% (#7) Integration: 63.0% (#15) Observability: 47.2% (#7) APEX-SWE covers two kinds of work. - 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 asks the agent to debug a production-style system from logs and telemetry. Haiku 5.5 excels here and ranks #7, behind only six models. It still trails Opus 5.5 (69.8%) by 22.6 pts, and a typical attempt uses about 2.5M tokens, 4.5x an Integration attempt. - 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 asks the agent to build and deploy systems across services. Haiku 5.5 finishes 2.5 pts behind Opus 5.5 (65.5%), but the field is bunched, and it ranks #15. Across 4 runs per task, Haiku 5.5 solved 62.5% of APEX-SWE tasks at least once. 75 of 200 tasks were never solved. It also passes last generation's Anthropic models on APEX-SWE: Opus 4.8: 43.9% Opus 4.7: 47.6% Sonnet 5: 46.4% 𝗣𝗿𝗼𝗳𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹 𝘄𝗼𝗿𝗸 APEX-Agents is our flagship benchmark for long-horizon, agentic professional services work. It tests workflows across management consulting, corporate law, and investment banking. APEX-Agents: 56.1% (#19) Management consulting: 50.9% (#22) Corporate law: 60.0% (#26) Investment banking: 57.5% (#13) That puts it 19.4 pts behind Sonnet 5.5 (75.5%) and 17.4 pts behind Opus 5.5 (73.5%). This matches Anthropic's own advice. Sonnet and Opus are for complex agentic work, Haiku is for scoped, high-volume tasks. 𝗖𝗼𝘀𝘁 𝗮𝗻𝗱 𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝘆 Haiku 5.5 lists at $0.10 input and $0.50 output per 1M tokens. That is 1/20th of Sonnet 5.5's input and output cost for prompts up to 100K tokens, which Anthropic says covers about 90% of Haiku requests. It runs about 75% cheaper than Haiku 4.5 on average, and it is the first Haiku with an adjustable effort setting. We tested all three 5.5 models at Max effort. On APEX-SWE, Haiku 5.5's output token cost was about 35x lower than Sonnet 5.5 and about 60x lower than Opus 5.5. It also finished a typical task about 3x faster. On APEX-Agents, a typical Haiku 5.5 task used more output tokens than Sonnet 5.5 and Opus 5.5 combined. It still costs about 14x to 16x less on output tokens. Across 4 runs per task on APEX-Agents, Haiku 5.5 solved 74.6% of tasks at least once, but only 37.9% on every run. See full leaderboard: mercor.com/apex/

Watch Mercor's Head of AI modeling @edwardjhu to learn about AI data, evaluation, and model training. Find out why "the closer we are to how AI actually gets deployed, the more realistic the benchmarks will be."

Nathan Labenz @labenz

a week ago

Up now on AI:AM: Edward Hu (@edwardjhu), inventor of LoRA and Head of AI Modeling at @mercor. The question: once models clear the benchmarks built to stump them on professional work, what's left to measure? x.com/i/broadcasts/1…

0 1 9 5K 6

APEX-Agents tests AI on real professional work. In nine months the top score went from ~25% to 80%+. @edwardjhu invented LoRA, helped build o1, and leads model training at @mercor. What do you measure next? 10:15a PT on AI:AM w/ @labenz + @8teAPi 📺 x.com/i/broadcasts/1…

Today Reflection released Beam, their first open-weight model. Beam was trained with a particular focus on coding and agentic performance, advancing the Western open-weights frontier. Congratulations to the @reflection_ai team.

Reflection @reflection_ai

2 weeks ago

Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active. - Frontier reasoning efficiency - Advances the Western open frontier on coding & agentic tasks - Trained end-to-end from scratch Full weights release this month. Learn more about

564 1K 9K 2.5M 3K