I had this crazy idea - what if I can find such algorithm which can find all the bugs in any software, literally ALL. And while going though it (quite successfully actually!), it helped me understand the difference in intelligence of small vs large models.
If TLDR - if you have clear task definition, or at least enough clues, cheaper models can do quite serious work now. If the task require to invent questions themselves - thats what require true intelligence, and cheap models fall short. Also majority of public benchmarks is a form of cheating.
I was working on bisecting on how exactly benchmarks like SWE bench work, how the eval looks like, what is in the golden data. And what I found is that it does not really ask a question to find the bug. Instead it gives a lot of clues in advance, like a user report, sometimes with logs, and ask to fix the bug. Additionally all the benchmarks use popular OSS projects, like Django, and all the answers in the public internet already - all new models have it as part of both pre and post training - which essentially cheating. And you do not even need to enable internet search for that - it is part of training data.
Compare it with the work on the real project, where there is no one to help. Genuine new problems. Solving such problem require completely different thinking. And thats why even latest frontier models can look stupid on private codebases.
While everyone was hyped about mystical Ox Alpha model, which ended up to be GLM-5.3-flash, I wasn’t getting good results with it on my use cases. And specifically trying to find as much bugs as possible, just by looking at the code, without any clues. Flash model was VERY bad for this case, so I started digging. At the same time Grok 4.6 were perfect for this use-case (~5x larger compared to Flash)
What bothered me was that it could do apparently complicated work. In one experiment, it wrote an 82-line working reproduction of a Django bug. The checks passed. But I had supplied the function, the triggering input and the symptom - I had already told it what was wrong.
How much of the intelligence was in my prompt?
For code review, I need a model to come up with a useful question before I do. To notice an assumption worth challenging, choose an input that challenges it, and work out whether the failure actually happens. Otherwise, I’m still doing the difficult part and delegating the typing.
Here are three examples from Django 3.0. I’m showing the failing inputs upfront; discovering them was part of the review. I was using Grok alongside GLM because Claude and Codex had refused the security-related work I was doing. Agreeing to participate was already a competitive advantage.
1. The guard that handles nothing
Here is the filename construction, with the surrounding path handling removed:
if file_hash is not None:
file_hash = ".%s" % file_hash
hashed_name = "%s%s%s" % (root, file_hash, ext)What happens when a custom hash implementation returns None?
For styles.css, you get stylesNone.css. The branch leaves None untouched, and the next line turns it into text. No exception. A valid string containing the wrong filename.
The check is sitting right there, mentioning the exact value that causes the problem, without handling it correctly. To see the failure, you have to follow the variable past the reassuring-looking condition.
The GLM reviews discussed directory errors, exception propagation and a possible race. They reached the right function and missed the corrupted filename. Grok found it.
This is the difference between identifying something worth investigating and completing the investigation. “What about None?” is a useful question. “It might raise a TypeError” would still be a wrong answer. The actual failure is quieter.
2. A comma inside the exponent
Django’s number formatter converts a number into a string, splits around the decimal point, and groups the integer part into thousands. The relevant steps are:
str_number = str(number)
if "." in str_number:
int_part, dec_part = str_number.split(".")
else:
int_part, dec_part = str_number, ""Except that 1e25 becomes "1e+25".
There is no decimal point. The whole string becomes the “integer part,” including the exponent. The grouping loop then works backwards through those characters, inserting a separator every three positions. In the audited Django environment, this produced:
from django.utils.numberformat import format
format(
1e25, ".",
grouping=3,
thousand_sep=",",
force_grouping=True,
use_l10n=True,
)
# '1e,+25'A thousands separator inside the exponent. The loop processes characters; naming its input int_part doesn’t make those characters digits.
The GLM findings went elsewhere: empty strings, empty grouping sequences, special Decimal values. Whether those are useful findings depends on which inputs the function is supposed to support. They did not identify this transformation.
To find this bug, knowing what each line does separately is not enough. You have to carry "1e+25" through all of them without silently replacing it with the ordinary decimal number you expected to see.
That is a small amount of code, but a real demand on the model. The relevant fact is something that isn’t there: a dot.
3. “Just use self”
The third example comes with a comment that practically invites you to stop thinking:
# If the other Q() is empty, ignore it and just use `self`.
if not other:
return copy.deepcopy(self)“Just use self” sounds harmless. The implementation asks Python to deep-copy the object and its contents.
My recorded test put a thread lock inside the query. Combining it with an empty Q() reached this copying operation and raised TypeError: cannot pickle '_thread.lock' object.
A lock is a deliberately awkward input. Django’s upstream report shows the same problem with something much more ordinary: a dictionary’s keys. With the empty query on the left, this failed too:
Q() | Q(x__in={}.keys())
# TypeError: cannot pickle 'dict_keys' objectAn empty query, which was supposed to contribute nothing, introduced a new requirement: the contents of the other query had to survive copying.
And you cannot replace that input with anything vaguely “unpickleable” and expect the same result. Ordinary functions, including lambdas, are returned unchanged by deepcopy. The exact object matters.
This case caught Grok too. It reached the deepcopy line but proposed the wrong trigger. A cheaper-class model found the case in one differently framed session, where the comment had been presented as a claim to challenge. That success did not reliably carry over to another experiment with related guidance.
That is why “use the stronger model” is not the whole answer. Finding the suspicious line is progress. Knowing the phrase “deep copying can fail” is progress. The review is unfinished until the specific input produces the specific failure.
I tried better prompts
I changed the instructions, supplied more clues, split the review into phases, and required more explicit checks. Across three versions of the review instructions, GLM-5.3-flash found none of the seven selected hard defects. Zero out of seven each time. These were deliberately difficult cases, not a random sample of everything it can do.
One experiment required line-by-line execution traces with intermediate values. Thirty-eight of the forty-eight outputs followed that format. The requested structure appeared; the missing findings did not. The record contains confident, well-formed traces that were wrong.
At that point, “make it explain its reasoning” was not much of an answer. I was already reading the explanations. They were the problem.
The copying example shows that framing can help. But adding more instructions did not reliably buy the judgment I needed. Better compliance was measurable. It was also insufficient.
So what are the leaderboards measuring?
SWE-bench gives a model a repository and an existing issue, then asks for a patch. Somebody has already noticed a problem and described it. There may still be difficult engineering ahead, but discovering that something is wrong is not the same assignment.
Why should I read success at repairing described problems as evidence that a model will independently find problems in my code?
Then there is the possibility that it has already seen the answer. On February 23, 2026, OpenAI explained why it had stopped reporting SWE-bench Verified scores, citing contamination and flawed evaluations. All three model families it tested could reproduce original fixes or problem-specific details for some tasks.
I don’t need to prove that every provider deliberately cheats to question the measurement. When the issue, the patch and the regression test are public, a correct answer alone cannot tell me whether the model worked it out.
That applies to my Django examples too. Their public histories mean Grok’s successful answers are not proof of discovery on unseen code. These are inspectable, repeatable examples, not my attempt to sell you another intelligence ranking.
What interests me is the difference between these tasks. A model can implement a precise bug description while struggling to produce that description itself. A single coding score hides exactly the distinction I need when deciding what work to delegate.
Make it ask the question
Here is an exercise to try with your own model. Give it the original filename code and enough context to review it, without announcing the bug. Then, in a separate session, give it this version:
if file_hash is None:
file_hash = ""
else:
file_hash = ".%s" % file_hashThe original None failure is gone. Does the model find it in the first version and stop reporting it in the second? Can it supply an input and predict the actual result? Does its explanation follow the code when you change the behavior?
Then look at how much help you had to give it. Asking “what happens when the hash is None?” has already supplied an important part of the review. Asking it to investigate the function leaves that decision with the model.
I don’t care whether it answers from the code or runs experiments first. Give it a runtime. But it still has to choose an experiment that answers a useful question, rather than execute something irrelevant and report success.
I hope this article gave you more clarify on how intelligence differ in small vs big models and why it is still too early to claim that local models are on par with frontier ones. It is true only if you performed the “intelligence” by yourself, and give the model clues - e.g. bug fixing based on well shaped report.