When a user submits a teaching interaction, a front-end agent creates a benchmark that PAT should fail, confirming that the poor behavior can be reproduced, then iterates on context/harness until the benchmark passes, and finally confirms no regression across the broader test suite before submitting a pull request.
factualpending
Speaker
Michael RanEvidence Quote
“an agent on the back end will create a benchmark that we expect to fail. This shows that we can reproduce this poor behavior. It will then iterate on our context repositories or the harness itself until that benchmark passes. And then after confirming that by making this benchmark pass, we didn't cause the rest of our suite to fail, we get a Slack message with the pull request, including the changes the agent wants to make to PAT.”
Created: 8/12/2026, 6:15:31 PM
Your notes
Loading notes...