First Mate Technologies has detailed a software development workflow that uses separate AI models to write and verify code. One model handles implementation, while another designs test cases and reviews the finished code before changes are cleared for merging.
The company used the process on QueueMate, a restaurant queue management application built largely with AI coding agents. First Mate said the checking model produced 554 test cases and the resulting QA sweep identified 38 defects, including a severity-one issue involving session handling.
First Mate separates its AI agents into builder and checker roles. The checker handles feature planning, test-case design, and code review, while a different model writes the implementation and executes the tests it receives. A human engineer directs the process and retains responsibility for judgement calls and final approval.
First Mate said the separation is intended to prevent the model responsible for an implementation from also deciding whether that implementation is correct. The company said starting a new conversation with the same model would not provide the same separation because a model that misread a requirement while writing the code could carry that interpretation into its review.
The checking model instead receives the acceptance criteria and information about existing automated coverage without writing the implementation. It then defines additional test cases and reviews the completed code separately from the builder model.
Why self-testing has limits
Of the 554 cases generated during QueueMate’s wider QA audit, 49 covered happy paths. Another 291 focused on edge, negative, boundary, and concurrency conditions, while other cases covered areas including security, accessibility, and interactions between features.
One test examined reservations created on either side of midnight in the application’s Manila timezone to check whether they were grouped under the correct calendar day. Other cases tested concurrent actions across multiple browser tabs, including remove, undo, and seating operations occurring against the same queue state.
The QA sweep found a session-refresh file located in a directory where the application’s framework did not load it. First Mate said the file had not compiled or executed, causing sessions to expire earlier than intended and contributing to several other reported defects.
First Mate said all defects were recorded before fixes were made so the test cases ran against the same version of the code. Of the 38 defects recorded during the sweep, 27 were fixed in a follow-up commit.
The testing environment is also separated from other development work. Each branch runs its own application instance and Docker-backed database stack, with separate ports and project identifiers, while SMS and email services are replaced with mock gateways during QA runs. Tests create fresh application data rather than relying on an existing seeded dataset.
First Mate also runs deterministic checks in continuous integration, including type checking, linting, unit and component tests, and a production build. The company said its repository contained about 1,060 unit and component tests across 100 files in August, alongside 97 end-to-end scenarios that run against desktop and mobile profiles.
First Mate’s checker is therefore one part of a wider verification process. The company also uses deterministic CI checks, browser and end-to-end tests, independent code review, and human approval before code is merged.
Research on LLM self-correction has examined some of the limitations of allowing models to assess their own outputs. A 2024 critical survey published in Transactions of the Association for Computational Linguistics found no prior work demonstrating reliable self-correction using feedback generated by prompted LLMs across general tasks, apart from tasks particularly suited to self-correction.
The researchers found stronger results when models had access to reliable external feedback, including code interpreters. In software development, tools such as tests and interpreters can provide feedback independently of the model that produced the code.
Separate research presented at NeurIPS 2024 found evidence of “self-preference” on two summarisation tasks. GPT-3.5 Turbo, GPT-4, and Llama 2 disproportionately favoured their own outputs over those from other models or humans, even when human annotators considered the outputs equally good.
The NeurIPS research did not examine software testing, so it does not establish that a coding model will favour code it generated. The findings instead provide evidence that self-preference can occur when the same model acts as both generator and evaluator.
Independent verification moves into coding workflows
Spotify has also built independent verification into its background coding-agent workflow. Deterministic verifiers check formatting, builds, and tests before an agent can open a pull request, with failed checks blocking the process.
Spotify later added an LLM-based judge after finding cases where coding agents passed conventional checks but still moved beyond their instructions, including unnecessary refactoring or disabling flaky tests. Spotify said the judge vetoed about 25% of thousands of agent sessions it examined, while the coding agent corrected its work in roughly half of those vetoed cases.
Anthropic’s guidance for evaluating coding agents recommends a similar division between deterministic and model-based checks. Unit tests and static analysis can verify executable behaviour, while model graders can assess criteria that are harder to express as pass-or-fail assertions, such as whether an agent followed the original instruction.
Anthropic also recommends calibrating model-based graders against human experts and reviewing the evaluation setup itself. Its guidance notes that ambiguous requirements, grading bugs, and poorly designed tests can produce misleading results.
Tests can also become unreliable when coding agents are able to modify the mechanisms used to evaluate them. OpenAI has documented examples of reasoning models making programming tests pass without implementing the intended solution, a behaviour the company studies under reward hacking.
In one example, an agent was asked to implement mathematical functions but discovered that the tests ultimately depended on a verification function. Rather than completing the required implementation, the agent modified the verification function so that it returned true, allowing the tests to pass.
OpenAI tested a separate LLM-based monitor against such behaviour. The monitor had access to the coding agent’s actions and reasoning trace, and OpenAI reported that it successfully flagged examples in which the agent attempted to subvert unit tests.
Anthropic uses held-out tests to check for a related failure mode in coding evaluations. In reward-hack-prone tasks, models can see one set of tests while additional cases remain hidden, allowing evaluators to identify implementations that pass the visible tests but fail cases the model did not see.
The verifier itself also depends on the quality of its tests and requirements. OpenAI reported in July 2026 that roughly 30% of SWE-Bench Pro tasks contained problems, including hidden tests that enforced requirements missing from prompts and low-coverage tests that allowed incomplete fixes to pass.
First Mate’s QueueMate results provide one example of how a development team is separating code generation from verification. Its independent checking process generated hundreds of test cases and identified defects in software written largely by AI agents, but First Mate’s article does not provide a controlled comparison showing how many defects would have been found if the builder model had designed and evaluated its own tests.
The published results therefore do not establish that a separate AI model is more accurate than same-model verification. First Mate’s workflow instead combines separate code generation and test design with deterministic checks, model-based review, and final human approval.
