A coding agent was asked to prove the tests passed. It came back cheerful. The log underneath said collected 0 items and then no tests ran. The exit code (the whole number a program hands back when it stops) was 0.
Exit code 0 does not mean success. It means the program did not choose to report an error. The job you cared about can still be missing.
Related reading when the log is the real evidence: hl vs jq for JSON Logs and tsrs vs tsc: Same Errors, Different Exit Code.
That gap is why a green 0 is a bad reason to merge. A check that only stores the number will bless an empty test run. I pasted the same logs into a small local program called Verdict. On the empty pytest run it said unknown. The confidence (how sure it is of that one label, from 0 to 1) was 0.999981. On a run that printed 3 passed it said success, at 0.994886. The exit code was 0 both times.
Exit Code 0 Is Not Proof of Success
No. Zero means the program finished and did not report an error. It does not mean the tests ran, the file changed, or the push landed.
The shell (the text program you type commands into) needs one small signal when a command ends. By a long convention, 0 is the calm signal and any other whole number is a complaint. The complaint can mean whatever that program decided. A 1 might mean a failed test. A 2 might mean you typed the command wrong. Nothing in the digit says which.
So when an agent says the command succeeded after a 0, it is turning a convention into a story. The story is often true. It is not the same fact as the number.
What Is an Exit Code?
An exit code is that single whole number, returned when one running copy of a program ends.
Two text streams sit beside it. Stdout (the normal text the program prints for you to read) is one. Stderr (a second stream, usually for warnings and errors) is the other. A program can print a failure on stderr and still return 0. It can also return 1 while stdout looks tidy. I have watched agents quote the 0 and skip both streams.
You can see the number yourself. After a command, in a bash-style shell:
echo $?That prints the exit code of the command you just ran. If you only keep that digit, you have thrown away the sentences that say what happened.
Why Did Pytest Pass When No Tests Ran?
Pytest (the usual Python test runner) returns 0 when it finishes without a failed test. Zero collected tests is not a failed test. It is an empty room. The runner did its job, found nothing to do, and left quietly.
I fed Verdict this pair: command pytest tests/, exit code 0, and this stdout:
The label was unknown. The probability on unknown was 0.999981. Success was 0.000019. It did not call the empty room a pass.
Then I changed only the text, and left the exit code at 0:
The label flipped to success at 0.994886.
Then a failing log, exit code 1, with 2 failed, 1 passed and AssertionError on stderr. The label was failure, at 0.999111.
The command name was pytest in all three. The exit code was 0 in the first two. The printed text is what separated “nothing ran” from “three checks passed.” If your agent only stores the digit, those two runs look the same.
This is the case I would paste into a review. A branch that collects zero tests and exits 0 is not a green build. It is a path that never executed.
How Do I Check a Command Really Worked?
Read the text the command printed. If you want a second opinion that stays on your machine, run a local checker on that text.
Verdict is a small local model for this one job. The weight file I used is 197,160 bytes. It does not open a network connection. You pass the command, the exit code, stdout, and stderr. It returns one of four labels: success, failure, blocked, or unknown, plus a probability for each. The result is marked advisory (a hint you can ignore, not a lock on the command). It does not run the command for you. You already ran it. It only reads the record.
The project is at github.com/Balanov/verdict. From a folder you are allowed to write:
git clone https://github.com/Balanov/verdict.git
cd verdict
python -m pip install .Then the empty-test case:
from verdict_tiny import Verdict
model = Verdict()
result = model.predict(
command="pytest tests/",
exit_code=0,
stdout="collected 0 items\nno tests ran in 0.01s",
)
print(result["label"], round(result["confidence"], 6))On my run that printed unknown 0.999981. If you get a different label, you are not on the same weight file. Stop and check that verdict-q8.bin is the 197,160 byte file before you trust a later answer.
A plain reading still works when you do not want the model. For pytest, search the output for passed and also for no tests ran or collected 0. For a build, search for the error line, not only for the digit. The model is a faster skim of that same text, and it is wrong in the places the next section shows.
When Is Exit Code 0 a Lie?
It is a lie when the digit says calm and the text says the job did not happen. It is also a weak signal when the text is empty.
| What I passed in | Exit | Text that mattered | Label | Confidence |
|---|---|---|---|---|
| pytest, zero tests | 0 | collected 0 items, no tests ran | unknown | 0.999981 |
| pytest, three passed | 0 | 3 passed | success | 0.994886 |
| pytest, two failed | 1 | 2 failed, AssertionError | failure | 0.999111 |
| npm test, no output | 0 | empty stdout and stderr | unknown | 0.985703 |
| curl of a normal page | 0 | an HTML title came back | success | 0.928762 |
| git push, already current | 0 | Everything up-to-date | success | 0.75809 |
| git push, rejected | 1 | non-fast-forward | blocked | 0.868202 |
| make, nothing to do | 0 | Nothing to be done for all | unknown | 0.525 |
| make, missing target | 2 | No rule to make target | blocked | 0.985 |
| python print | 0 | printed 1 | success | 0.997 |
| rm -rf / and permission denied | 0 | stderr said Permission denied | success | 0.707976 |
The last row is the one I do not want you to miss. I handed the checker a dangerous remove, exit code 0, and the words Permission denied. It still picked success, at 0.707976, with blocked only at 0.116428. A real shell would usually not return 0 after a permission error. I forced that mismatch on purpose. The checker leaned on the digit. It is not a safety brake. Do not put it in front of a delete and walk away.
make with “nothing to be done” and exit 0 landed on unknown at 0.525. That is close to a coin toss. Treat a confidence near one half as “open the log,” not as a quiet yes.
A command that only runs false, with exit code 1, came back blocked at 0.579, not failure. The confidence is low and the label is odd. Another reason the word on the label is not something you merge on. The field is even named advisory.
Empty npm test at exit 0 was unknown at 0.985703. That one I liked. No text, no claim of success.
Should I Trust My Coding Agent When the Command Exits 0?
No, not by the digit alone. Trust the step when the printed text names the thing you asked for, and a second look agrees. If the agent pastes secrets into the prompt while it “proves” a green exit, see How Do I Strip Secrets From a Prompt?.
An agent that can run tools will often do this in order: run the command, read the exit code, and if it is 0, say the step worked and move on. That saves a round of reading. It also hides the empty pytest. The same habit shows up when an agent keeps adding files you did not ask for. Stopping the extra code is a separate fix, in how to stop claude code from overengineering. The bill for those extra tool rounds is separate too, in does jevgrep cut coding agent cost. A long chat can also drop a rule you already wrote, which is why compaction drops your instructions.
What I would wire in, if the agent is allowed to run tests while I am away:
- Keep stdout and stderr, not only the exit code.
- If the label comes back failure, stop and show the error line.
- If the label comes back unknown, stop and show the log. Do not retry on a guess.
- If the only evidence is exit code 0, do not merge and do not delete.
- Never let this checker approve a command that removes files, spends money, or pushes. Those need a person. A stop the model cannot unpress belongs outside the model, which is the point of an emergency brake for an agent.
The loop that makes this matter is simple. The model asks for a tool, your machine runs it, and the result goes back into the chat. If the result is only “exit 0,” the next guess is built on a hole. A walk through that loop is in how Claude Code works. If two agents are writing the same folder, split them before you even get to exit codes: openrig vs git worktrees.
Verdict vs Reading the Log Yourself
Read the log when the command is rare, dangerous, or short enough to see on one screen. Use Verdict when you have many boring commands and you want a flag on the ones that look calm but say nothing.
I would not replace the log with the label. The label missed a permission error I had forced onto exit code 0. I would use the label as a tap on the shoulder. Unknown means look. Failure means the text already contains a complaint. Success means the text and the digit agree enough for a boring command, such as printing the number 1.
A rejected git push came back blocked, not failure, at 0.868202. That split is useful if you know it. Blocked here meant the command was refused, which is not the same as a failed test. If your agent treats every non-zero as “try a fix,” it will “fix” a rejected push by rewriting code. Show it the refusal line instead.
Common questions about exit code 0
does exit code 0 mean the command worked
No. It means the program returned the calm number. Check the printed text for the result you asked for. On an empty pytest log, that text said no tests ran, and a local checker called the run unknown rather than success.
why is pytest exit code 0 when no tests ran
Pytest treats “no failed tests” as calm, and a run that collected nothing has no failed tests. Look for collected 0 or no tests ran before you believe a green step.
can a small local model catch a false success
Sometimes. The 197,160 byte checker caught the empty pytest and the empty npm test. It did not catch a remove command I marked with exit code 0 and Permission denied. Use it as a hint. Keep the log.
should my agent block on exit code 0
It should stop when the text is empty, when the text says nothing ran, or when the checker says unknown or failure. A bare 0 is not a reason to merge, push, or delete.
I still would not ship a change because a process returned 0. I would ship it when the log names the passes, and when a second look does not call that log empty. The cheerful agent was not wrong about the digit. It was telling the wrong story about the job.
