How it decides
Seven steps from a forum post to a suggested reply. Code computes every number on this site, and every quote is checked against the original text.
- 01
Read
code309 public Bill Pay threads (every post) and 231 release notes, each with its linked help article.
- 02
Narrow
jev-latestKeyword search pairs each request with likely releases, then Jev drops the pairs where the release clearly doesn't help. 573 pairs are left.
- 03
Judge
jev-latest · gpt-6-lunaJev and GPT-6 Luna each score every pair against the same four-option rubric. If they agree with confidence of 0.2 or more, the pair moves on. Otherwise it goes to review.
- 04
Confirm
claude-opus-5-5Opus reads the full thread and help article without seeing the other verdicts. It has to reach the same answer and quote the line that delivers the request, and code checks each quote against the source. Opus confirmed 19 of the 38 pairs Jev and Luna agreed on.
- 05
Check again
claude-opus-5-5 · highBefore a match is published, a second Opus pass looks for reasons it's wrong, like a plan or ERP the feature doesn't support, a request only partly delivered, or a how-to question mistaken for a feature request. It sent 2 of 11 matches back to review. Peter also checked every headline claim by hand.
- 06
Check for a reply
jev-latestJev reads every reply in the thread to see whether anyone said the feature is available. Beta notes are tracked separately from launch.
- 07
Draft
claude-opus-5-5For each open loop, Opus writes a suggested reply that mentions anything still missing. Nothing is posted.
The rubric
Both judges and the human labels use the same four definitions.
- Resolves
- After this release, someone like the poster can do the core thing they asked for in Ramp. Missing polish doesn't block it.
- Partial
- The release delivers a real piece of the ask, but something the poster explicitly asked for is still missing.
- Related
- Same area of the product, but the poster still can't do what they asked.
- No match
- A different problem.
How well it works
Scored on 50 pairs, sampled across confidence levels so misses count, not just easy hits. Reference labels come from two frontier models of different families (Claude and GPT), each working blind from the raw thread and help article with no access to this pipeline's outputs. 45 pairs are where both agreed; 5 were checked by Peter by hand, including every headline claim on the front page. Because Opus is also one of the judges, its score against a partly-Claude reference is likely optimistic.
| judge | 4-way accuracy | “shipped” precision | “shipped” recall |
|---|---|---|---|
| Jev | 68% | 64% | 94% |
| GPT-6 Luna | 62% | 62% | 94% |
| Opus 5.5 | 86% | 100% | 100% |
Against each blind labeler alone
The GPT labeler is the fairer test of Opus: it shares none of Claude's training.
| labeler | judge | “shipped” precision | “shipped” recall |
|---|---|---|---|
| GPT (Codex) | Jev | 68% | 96% |
| GPT-6 Luna | 61% | 77% | |
| Opus 5.5 | 95% | 82% | |
| Claude | Jev | 48% | 94% |
| GPT-6 Luna | 54% | 94% | |
| Opus 5.5 | 84% | 100% |
Jev + Luna agree
| min confidence | decided without review | accuracy when decided | “shipped” precision |
|---|---|---|---|
| 0.0 | 70% | 74% | 67% |
| 0.2current | 64% | 75% | 65% |
| 0.3 | 54% | 82% | 71% |
| 0.4 | 38% | 74% | 67% |
| 0.5 | 28% | 71% | 70% |
| 0.6 | 18% | 89% | 83% |
| 0.7 | 12% | 83% | 80% |
| 0.8 | 4% | 100% | 100% |
Jev + Luna agree, and Opus confirms with a verified quote (current)
| min confidence | decided without review | accuracy when decided | “shipped” precision |
|---|---|---|---|
| 0.0 | 48% | 96% | 100% |
| 0.2current | 44% | 96% | 100% |
| 0.3 | 38% | 100% | 100% |
| 0.4 | 24% | 100% | 100% |
| 0.5 | 20% | 100% | 100% |
| 0.6 | 16% | 100% | 100% |
| 0.7 | 10% | 100% | 100% |
| 0.8 | 4% | 100% | 100% |
Limits
Ramp's public changelog doesn't list everything that ships, so the real count is likely higher.
The community forum is only part of what customers ask for.
Release notes sometimes post after a feature is live, so replies up to 30 days before the note count.
Ramp's team already replies in many of these threads. This makes the remaining ones easy to find.
Model spend to date
Every call ever made, including tests. Reruns read from cache and cost nothing.