AI Agents

Real tools. Real numbers. No theater. On September 24 I asked the GitHub API what NandhaKishorM/laya looked like. Created September 18. 22,589 stars. 1,937 forks. Apache-2.0. Six days old. That star count is why your timeline wants you to rip the "which folder is this?" step out of your coding agent and hand it to Laya tonight. A coding agent is the program that reads your repo, edits files, and runs commands in a loop. The folder question feels too small for that loop. It is also where a lazy shortcut gets expensive, because you pay the big model to answer it on every turn. Is Laya worth it for that job, today, with the checkpoint that pip install gives you? No. The README already admits the base score. A public bakeoff posted September 23 agrees, with a wrinkle the star chart will not show you. I also ran a dumb keyword map on 36 lines I wrote myself, so this is not only other people's tables. Is Laya worth it before you fine-tune Before the product name, the job. You have a blob of text and a list of allowed answers. You want one of those answers, plus a number that says how sure the picker is. You do not want a paragraph. That picker is a decision model (a model that chooses from your list and does not write prose). Laya is one of those, built to answer in a single forward pass instead of generating text token by token. A token is the chunk of text a model counts when it bills you or fills its memory. Laya's pitch is that the decision costs no generated tokens at all. On a Tesla T4, the README puts a routed question at 32.8 ms. The weights live on Hugging Face under convaiinnovations/laya. The GitHub repo is the other name. If you search the org from a Reddit comment and get a 404, that is why. Now the number that should have been in the launch tweets. In the README section titled "Fine-tune for better accuracy," the fine-tuned laya-typed-decisions checkpoint scores 0.766 on a 2,000-decision set. The base English checkpoint, on the same decisions, scores 0.362. Random is 0.318. A boring "always pick the most common class" baseline is 0.461. The base model loses to that majority guess. The "Honest limits" section says this in plain words: the shipped checkpoints are near chance until you train them on your own labels. The 0.766 figure is the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base you specialize. It is not a zero-shot decision engine. Zero-shot means you point it at a new label set and expect a good answer with no examples of your own. That is the whole product, and it is a real product. It is not the product the star count is selling. There is a second failure that matters more than a low average. Issue #377 is a handful of cancellation lines. On Laya 0.3.20, a CPU run with the keys no_action and cancel_account picked cancel_account for negated requests. One multilingual answer gave that wrong key probability 0.9998. The positive control passed. The README says these are narrow examples, not proof that every negation fails. Read that as a warning about confidence, not as a license to ignore it. A wrong answer at 0.36 is a shrug. A wrong answer at 0.9998 is a branch your code will take. Laya vs SemIf on yesterday's coding tasks SemIf is the other young repo in this pile. It does not train a new decision head. It takes a small open chat model you can already run, asks a question with a closed list of options, and reads the probabilities off those options instead of waiting for a written answer. The repo was called SemIf, formerly OpenJev. GitHub now serves it as TheoLeeCJ/SemIf-OpenJev. When I checked the API on September 24 it had 4,207 stars, created September 16, MIT license. Independent of TypeSafe. The README says so, and so does the description field. On September 23, r/LocalLLaMA got a bakeoff titled "This time I tested reflex vs SemIf vs Laya vs Von vs jev." I am not the author of that table. I am not going to pretend my laptop reran their corpus. The rows, as posted: TaskJevreflexSemIfLayaFree baselineCommit type, 6-way, n90, accuracy74.2%63.3%55.6%35.6%31.1% path regexFile routing, 12-way, n80, accuracy78.2%48.7%52.5%36.2%48.8% keywordIs-a-feature, n180, AUC0.8870.8010.8210.6140.543 keywordChange breadth, n90, Spearman0.9050.8240.8170.1610.222 word count Jev is TypeSafe's hosted decision model, the closed one people are cloning. reflex is a self-hosted wrapper around an unmodified Qwen3.5-4B that runs each choice forward and reversed, then combines the two. The post's author says reflex and SemIf beat Laya on all four tasks. Do not let me flatten the free baselines, because that is how these posts get sloppy. Laya beat the path regex on commit type (35.6% against 31.1%) and beat the keyword score on the feature check (0.614 against 0.543). It lost file routing to a keyword rule (36.2% against 48.8%) and lost the breadth ranking to raw word count (0.161 against 0.222). "A regex beat Laya at everything" is false. "Laya is the model you want choosing files" is also false, on this draw. The same post says the default 0.5 cutoff on the feature task scored 25.0%, worse than a coin flip, while the ranking signal was real and the better cutoff sat at 0.97. That matches the shape of the README. The order of the scores can hold information after the yes/no threshold you copied from a blog is garbage. If you ship the default cutoff, you will not get the AUC. You will get the 25%. One more caveat from the post, so you do not cite it harder than it can bear. Von's round 1 and round 2 used different draws. The four tasks above are the fresh comparison. They are still one person's sample, not a lab standard. Treat them as a reason to test your own tickets, not as a trophy. The keyword rule I actually ran I wanted a floor I could defend, on text I could paste. So I froze 18 commit subjects and 18 file-routing lines, then scored them with a lexicon I did not retune after seeing the misses. Prefixes like fix: count. So do a few literal words: crash, stripe, modal, readme, docker, pytest. If nothing matches, the rule abstains. It does not guess. Commit subjects: 6 correct out of 18. The 6 were the ones that already wore a conventional prefix. The other 12 abstained. Zero wrong guesses. File lines: 8 correct out of 18. Ten abstains. Again, zero wrong guesses. The misses were sentences a person gets immediately and a slogan lexicon does not: a tax line with no word "invoice," a memory cap with no word "docker," a sign-in bounce that never says "password." That 8/18 is not their 48.8%. Different sentences, different label set, and I wrote both. Anyone who publishes 100% on examples they authored to fit the rules is showing you a magic trick. The useful part is the error type. The map fails by going quiet. Laya, on the cancellation lines in its own README, fails by sounding sure. If you are going to put either one in front of a tool that deletes a branch or files a ticket, silence is the failure you can code around. A 0.9998 on the wrong key is the failure that pages you at midnight. You can redo my floor in ten minutes. Write 20 lines from your own repo. Label them before you look at any model. Score a map that only knows your directory names and your commit prefixes. Count abstains as misses if the caller needs an answer every time, and count them as saves if the caller can fall through to a person. Then, and only then, decide whether a 421 million parameter checkpoint has to beat that map by enough to be worth the loading time. Can I use Laya instead of Jev for file routing Jev, the hosted one, is the ceiling on that September 23 table. Simon Willison's September 21 note is still the clean description: unstructured text in, a typed probability out, billed on input. He put the price at $0.042 per million input tokens, output free. I did not call the API for this piece. Where the Laya README compares itself to Jev, it says those Jev figures are third-party numbers, not a paired run, because the author had no TypeSafe key. Sample sizes and prompts differ. Believe the shape. Do not quote the delta as if both sides sat on the same machine. The shape is enough. On Banking77, a 70-plus-way choice, the README gives Jev 0.870 and routed Laya 0.425. The reason is mechanical. At the default English settings the option list only gets about 192 tokens, so a 77-way question is left with a few tokens per label. The names blur. Jev's documented habit is to allow a much larger option list. If your router has dozens of services, Laya at defaults is the wrong tool even after you fall in love with the latency chart. Context is the other cliff. English Laya defaults to 512 tokens of total context, with most of that left for the document only after the options take their cut. A coding agent that pastes a diff and a tree into the "state" will walk off the end of that window and never tell you in the star count. Jev is the one people are using when the input is a long ticket. Laya is the one you keep when the input is a short string and the label set is stable. So the switch, said as a rule you can argue with: Use Jev, or the big model you already pay for, when the label set is new, long, or wider than about twenty options, and you do not have a few hundred labeled rows yet. Use SemIf or reflex when you want the decision on hardware you control and you refuse to fine-tune this week. They were the two self-hosted rows that cleared Laya on every task in that bakeoff. SemIf's bet is that a frozen 4B you already downloaded is enough if you read its option scores instead of asking it to write JSON. Use Laya when you have the labels, you will run the fine-tune, and you care about a few dozen milliseconds on repeats of the same decision. The project even ships a Kaggle notebook for the 2,000-decision loop, on the free dual-T4 tier, including the step that fits temperatures. Temperature here is just a dial on how sharp the probabilities look. The README says laya-multilingual ships with no fitted temperatures, and invalid entries get clamped, with a runtime warning. A warning is not a calibration. Fit it on your data or do not trust the number next to the label. Do not use the base pip package as a drop-in for the yes/no calls inside Claude Code. If you are on the Opus 5.5 default, those calls are already the expensive kind of small. The bill math on the default-model piece is the reason. A decision model only helps if it is right often enough to skip the big model. A 0.362 base score does not clear that bar. It adds a dependency that still has to fall through. If you do wire a router into the harness later, put it where you can see it fail. Mods versus plugins is the difference between a switch you can name and a pile of files the session may not load. And if the router abstains, leave a mark a human can spot. That is the same instinct as the systemd canary: a visible tell beats a confident silent miss. What to run this afternoon Skip the install if your next sentence is "I'll just try it on the monorepo router." You already know how that movie ends. The base checkpoint is near chance, the file-routing bakeoff lost to keywords, and negation can come back almost certain and wrong. If you have a few hundred labeled decisions from real tickets, then Laya is worth a weekend. Clone the repo, open the fine-tune notebook, and hold out a slice of tickets the training split never saw. Include negated lines on purpose. If the held-out accuracy does not beat your abstaining keyword map by a margin you would bet a deploy on, stop. You learned something cheap. If you do not have labels, try SemIf's browser demo or the Python scorer on a model size that fits, and score the same 20 lines. You want agreement with your labels, and you want the confident misses written down. A tool that will not show you its wrong 0.99s is not ready to touch a repo. Either way, log the checkpoint name on every row. Laya is three models wearing one import: English, multilingual, and the fine-tuned typed one. A benchmark that says "Laya" without saying which one is how 0.766 and 0.362 get averaged into a vibe. Common questions about Laya Is Laya the same as Jev? No. Same shape of job: text in, a choice or a score or a yes/no probability out. Jev is TypeSafe's closed API. Laya is an Apache-2.0 stack you run, with weights published separately from the GitHub repo name. A compatible HTTP shape is not a shared model. Is Laya worth it with no labeled data? Not as a router. The project's own typed-decisions table puts the base English checkpoint at 0.362 against a 0.318 random line and a 0.461 majority line. Worth it starts when you fine-tune on decisions from your domain and check a holdout the notebook did not train on. Can I use Laya instead of Jev when there are dozens of folders? Not at the defaults. The README's Banking77 line is 0.425 for routed Laya against 0.870 for the published Jev figure, and it explains the token budget that causes the drop. Shortlist the options first, or raise the option budget, or don't ask one question to pick among 70 names. Why does it look sure when the label is wrong? Because a probability is not a promise, and an unfitted temperature makes the number look sharper than the evidence. The cancellation examples in issue 377 are the ones to read. If your code treats 0.99 as permission to act, you need a cutoff you fit yourself, and a path that abstains. The keyword map's silence is the pattern to copy. The 0.9998 is the pattern to ban. Stars are a vote that the idea is interesting. Six days and 22,589 of them means the idea landed. They are not a vote that the base checkpoint can file your code. If you try one thing after this page, make it the 20-line holdout from your own tree. Beat the quiet keyword map, or don't ship the model. Real tools. Real numbers. No theater.

AI Agents

Is Laya worth it for routing code? 22,589 stars, and the base score is 0.362.