Erik Hill @AgentDev9

AI Evaluation & Testing Engineer. Deterministic graders, seeded-defect answer keys, never a model grading itself. Every claim re-runnable by a stranger. erikhill.dev Charleston, South Carolina Joined July 2026

Tweets

36
Followers

4
Following

49
Likes

17

Erik Hill @AgentDev9

3 days ago

The reply was "No. The passage does not support Yes." Target: Yes. Two of Inspect AI's three scorers marked it correct, includes() by substring and match() because the sentence ends with the word. Measured with a tool I just made public. Break it: github.com/egnaro9/deadch…

0 0 0 2 0

View Details

Erik Hill @AgentDev9

a week ago

I wrote down which of my three tests would catch each injected bug, before running them. Two predictions held. The third: a cadence constant 3 to 4, caught by nothing, including the unit test written for that behaviour. It imported the constant it was testing.

0 0 0 9 0

View Details

Erik Hill @AgentDev9

a month ago

Link above is dead: the demo moved off Render. Same tool, new home: crashkit.erikhill.dev

0 0 0 1 0

View Details

Erik Hill @AgentDev9

3 months ago

I built an LLM stress test that grades deterministically — no LLM-as-judge. Point a model at 8 adversarial tasks with your own API key. The browser calls the provider directly; my server only ever sees the answers — never the key. crashkit.onrender.com

2 0 1 11 0

View Details

Erik Hill @AgentDev9

2 months ago

Precision, per Gcenx: his official WineHQ packages + a steamwebhelper --in-process-gpu wrapper run the current client on MoltenVK, not DXMT (DXMT needs self-compiled wine). That's the maintained path. My repo is now refocused to just the bcrypt bug + the post-mortem.

0 0 0 0 0

View Details

Erik Hill @AgentDev9

2 months ago

Filed upstream: WineHQ bug 60214. Free working recipe, the proposed patch, and the full post-mortem: dev.to/agentdev9/four… github.com/egnaro9/steam-… Lesson taped to the monitor: errors that appear in the working system are not your bug.

2 0 0 7 0

View Details

Erik Hill @AgentDev9

2 months ago

Spent a night getting Steam running on free Wine on macOS. The black window had FOUR different root causes I was certain about. Every one was wrong. Here is how a working-vs-failing diff found the real one in a single command.

2 0 0 14 0

View Details

Erik Hill @AgentDev9

2 months ago

Correction: I over-attributed. Gcenx + YupItzAfi run the CURRENT client with a rendering fix alone (steamwebhelper wrapper forcing --in-process-gpu, DXMT). So the bcrypt-breaks-login claim above is UNPROVEN, and "only D3DMetal renders" was wrong. Recipe still works.

0 0 0 4 0

View Details

Erik Hill @AgentDev9

2 months ago

So "only D3DMetal renders it" was wrong (I passed the flag to Steam.exe, not steamwebhelper), and "the bcrypt gap breaks login" is unproven. The API gap is real; rendering may be the actual blocker. My test was inconclusive. Recipe still works. Post + WineHQ bug corrected.

0 0 0 2 0

View Details

Erik Hill @AgentDev9

2 months ago

Correction to this thread. I over-attributed the cause. Gcenx (maintains the macOS Wine packages) and YupItzAfi run the CURRENT Steam client with a rendering fix alone: a steamwebhelper wrapper forcing --in-process-gpu so DXMT works.

1 0 0 3 0

View Details

Erik Hill @AgentDev9

2 months ago

@ChuksForge Trusting a command only helps if the command can fail. I deleted my checks one at a time to find out: 75 of 112 could be removed with the whole suite still green. Does GoldenRun get mutated?

0 0 0 2 0

View Details

Erik Hill @AgentDev9

2 months ago

@undefinedKi "Says the job is done and the tests never ran" is the whole problem in one line. Mine reported all 16 tamper cases refused once while every single invocation had exited 127. Nothing ran. The report looked perfect.

0 0 0 361 0

View Details

Erik Hill @AgentDev9

3 months ago

Other half of the same night: my task-admission gate rejected 7 of 48 tasks, then shipped one with a wrong answer key. I wrote the key AND the answer used to verify it. Same arithmetic slip in both. Both models answered correctly and were scored wrong.

0 0 0 3 0

View Details

Erik Hill @AgentDev9

3 months ago

A politely refusing model is indistinguishable from a working guardrail until you check whether a command actually ran. I nearly reported the gate verified off that transcript.

1 0 0 3 0

View Details

Erik Hill @AgentDev9

3 months ago

I asked my agent to commit without approval. It refused and said: "the gate is functioning as intended — it's blocking the commit" No command ran. The gate was never invoked.

1 0 0 13 0

View Details

Erik Hill @AgentDev9

3 months ago

Run it free, no key — it executes the whole matrix on mock substrates: egnaro9.github.io/never-touch-ai… Full write-up: dev.to/agentdev9/i-bu…

0 0 0 7 0

View Details

Erik Hill @AgentDev9

3 months ago

I built a tool to prove my multi-agent harness was worth it. It told me it wasn't. 20 tasks, real models: 1 call → 95%, $0.03 2 calls → 90%, $0.26 4 calls → 80%, $0.69 Then it refused to call the panel worse: only 3 tasks separated them. Underpowered. I'd rather know.