A Ruby type checker that has already found 15 bugs in Discourse, Rails, Zammad and ruby-lsp

Once the PR is open, what decides whether it moves forward is not the patch. Three gates, measured, not about the tool.

Shopify's CLA wants a human signature plus a comment that matches the workflow's regex. My first comment, "CLA signed.", passed the workflow's if, the job concluded success, and the log said Comment does not match CLA pattern: a green run does not mean the gate is cleared. What worked was "I have signed the CLA."

action_required is first-contribution workflow approval, per repository: it blocks zammad (5 PRs) and ruby-lsp (1). Rails and Discourse do not have this gate.

On GitLab, quota and membership block even the push: that is why 6 ready MRs remain unopened.

None of the three gates have anything to do with patch quality, and none of them show up before the PR is open.

A checker's finding is a hypothesis, not a bug. Every item on this list went through two-sided proof against the project's real file: run and fail before the fix, run and pass after. Reproducing it in a separate file does not count, because that tests my understanding of the code, not the project's behavior.

For dead-code removals the bar is different: prove that nothing reaches that code. That means scanning the whole repository, including dynamic dispatch (send, public_send, respond_to?, method_missing, delegate) and the class's subclasses, not just direct callers. That last part almost slipped past me on #58569 in Rails: the class has two subclasses, and if either one had called the method the story would have flipped from cleanup to real bug.

The shape of the PRs did not come from personal taste either. It came from measuring what the project already accepts. I first counted Rails's last hundred merges, found no dead-code removal and concluded there was no precedent. A hundred Rails merges cover two weeks: searching the full history turned up a twelve-PR removal campaign merged a month earlier. A short sample does not measure a rare category.

Acceptance is not always the same thing either. At Zammad, real development happens on an internal GitLab; GitHub is a mirror, and native merging there has been stalled since 2021. When a maintainer accepts a fix, they apply the patch as a manual commit and close the PR citing the hash. That is why the list distinguishes merged from applied by commit: two ways of saying yes, without hiding which one it was.

Rails #58566 is the most complete case in this section: closed without a merge, with a single comment, from byroot (Jean Boussier, Rails core): "If it's never called, let's just remove the method entirely." That was not a rejection of the finding, it was a rejection of the form: I sent a fix for a method that was dead, when removal was the right call. I had classified the method as reachable without proving it. The proof, done afterward: @thread_query_caches appears in exactly two places in the repository, the instantiation and one call to compute_if_absent; .clear is never called. And the method's body called synchronize on an object that does not have that method, so it would raise an error if anyone called it: dead and broken, and I sent a fix instead of a removal. The outcome is the applied by commit state: byroot removed the method himself in commit 22a02ee12e , literal message:

Get rid of `QueryCacheRegistry#clear`

The method is dead code.

With a co-author credit for the finding. It went in, just not through my branch.

On Rails #58568 , the method itaruby flagged had zero test coverage in the suite. The first proof that the code was dead came from a grep 'def expr' against the repository, which would never find an attr_accessor :expr. The right instrument was following the inheritance chain, not grepping for the method name. The mistake was caught and fixed before the PR shipped, and the missing test went in with it.

Discourse #42898 shows the same kind of failure with opposite consequences, in the same Lithium importer file: Two crashes in the Lithium import script. A file.unlind that does not exist on Tempfile, swallowed by the surrounding rescue, leaking a temp file per imported avatar. And a link_post_uploads called on CookedPostProcessor, which has no such method: that one is not swallowed and takes down the entire attachment post-processing step.

Three Zammad findings (#6329, #6331, #6332) shipped with a regression spec the suite did not have. In all three, the production fix is one line; the rest of the diff, thirteen to fourteen lines, is test.

What is proven: itaruby does not require anything, and it reads inline RBS if you have it. No configuration, no adoption step. The 14 bugs on this page came out of 3 repositories with zero sorbet/, zero .rbi and zero extend T::Sig: 13 submissions, 7 accepted. The remaining repository in the corpus, ruby-lsp, runs Sorbet and has its own bug, in the next section.

Scope note: from a Sorbet sig block, itaruby picks up the return type; it does not pick up the parameter type. The sig already speaks return, it still has to speak parameter. This matters less than it looks, because the corpus barely uses sig at all. I tested it: a class with extend T::Sig and sig { params(n: Integer).returns(String) }, called passing a String where the signature wants an Integer parameter, produces no diagnostic at all: that is exactly the side that is still missing. But rails, discourse and zammad have zero sig. ruby-lsp, the most typed repository in the corpus, has zero real production sig: the leftover sig text in the repository is documentation comments and spec fixtures, not annotation. tapioca has one, in lib/.

There is a sharper reason behind the zero above, and it is Sorbet's own documented behaviour rather than an opinion about it. A file with no sigil defaults to # typed: false, and at that level Sorbet reports only syntax, constant resolution and sig correctness. Its manual is explicit that the level "only prevents that method's body from being typechecked". So on a repository with no annotations, Sorbet is not failing to find these bugs: it is never asked to look. That is the whole gap itaruby aims at, and it is also why the comparison is about reach, not about speed. sorbet.org/docs/static

What changes the read: itaruby reads inline RBS. It is the annotation the tapioca team itself started using in ruby-lsp's production code, instead of sig: a one-line comment above the method, in the #: (Type) -> Type format. I tested the same method in the three annotation styles:

# no annotation
def format(n)
  n.to_s
end
format("7")
# itaruby: silence (correct, nothing to go on)

# with inline RBS
#: (Integer) -> String
def format(n)
  n.to_s
end
format("7")
# itaruby: p1.rb:7:18: error[E0103]: argument 1 of 'format' expects Integer, got String

# with Sorbet sig
extend T::Sig
sig { params(n: Integer).returns(String) }
def format(n)
  n.to_s
end
format("7")
# itaruby: silence

The cost contrast is the strongest data point in this section. For Sorbet to see the dependencies, ruby-lsp and tapioca keep 754,669 lines of vendored .rbi (246,830 + 507,839), plus a runtime gem, plus a sigil per file, plus a generation step. Inline RBS asks for none of that: it is a comment above the method, zero vendored files, zero runtime gem, zero generation.

RBS for core/stdlib, in a separate file, not inline: not consumed. RBS for a gem via rbs collection, also in a separate file: not consumed. RBI is not missing coverage, it is a lookup problem: a declaration inside .rbi written as a nested block indexes correctly, the full path exists in the index; the query is what fails, looking up the qualified path in a map where that name was registered unqualified, and giving up before applying the refilter that exists exactly for this. Only the compact module X::Y form avoids the problem. It is the same bug that explains why the alias fix cured ruby-lsp and not tapioca, in the "The cycle" section further down.

This table changed since this page's original publication: ruby-lsp went from worst to best in the corpus after a fix in itaruby itself, not in the scanned code. The mechanism and the verified prediction are in "The cycle" section, right after "Head-to-head with Sorbet".

What holds up, before and after the fix: neither "worst" nor "best" measured typing. They measured how much repository-specific parsing bug itaruby still had. The contract that survives is not "works better with more types": it is no requirement at all, and it reads inline RBS when the code already has it.

The hardest test I can run: pit itaruby against Sorbet on the only two repositories in the exploratory corpus that have genuinely adopted Sorbet. Global srb tc, no bundle install, on both:

All four time measurements are from the same session, same machine, median of three runs each. Sorbet is 2,9x faster on ruby-lsp and 2,7x on tapioca. Wall-clock time is not comparable across different sessions on this page: in an earlier remeasurement, ita check on ruby-lsp took 445ms; in this one, 496ms, with no binary change between the two. Session-to-session variance of 10% to 20% on this machine, which rules out any reading of "the fix cost time" without a before/after measurement taken in the same round.

File counts differ because srb applies the project's sorbet/config --ignore list (fixtures, vendor) and counts .rbi files alongside them; itaruby counts raw .rb files and has no equivalent ignore list.

Sorbet is green on both: literal output No errors! Great job.. itaruby, on the same code, produces 111 raw diagnostics. I did not find 111 bugs. I classified both full sets, item by item, not a sample:

srb tc remains at zero errors on both.

One number improved and the other got worse, and both deserve equal billing: in the earlier sample of 10 for ruby-lsp above, 7 of 10 were false positives (70%); in the exhaustive classification of 51 , that fell to 31%. tapioca went the other way: it became the repository with the highest false-positive share in the corpus, 57%. It was not a terminal state: the cause had a name, an address in the code, and the fix went in. The result is in the "Wave 8" section further down: tapioca dropped from 17 to 1, a documented ceiling, not a reclassification to zero.

I rechecked the original 10 ruby-lsp items against the exhaustive classification: 2 disappeared (they were false positives, resolved by the alias fix), 8 remain, and 1 of those changed classification because the earlier evidence had checked the wrong directory. A measurement mistake on our side, not a tool behavior.

False positives have a cause. The first time I measured that cause, the number came out wrong; the corrected version is below. I reclassified the 41 constant warnings left in ruby-lsp after the alias fix, by mechanism:

The two type-info levers, already-committed RBI and RBS from core/stdlib on disk, are close to the same size: 11 against 9. Neither one alone resolves half of the 41. There is no single saving type source here: it is always both together, and even both together do not cover everything.

11 of the 41 are fixture: fake constants that ruby-lsp itself defines on purpose to test its own syntax diagnostic detector. The project's sorbet/config excludes that folder via --ignore; itaruby does not honor that --ignore. That is 27% of what is left being itaruby reporting on a file the project's own maintainers already declared out of scope, a concrete and verifiable limitation.

About the method, because it is the foundation of every number on this page: the first measurement of this mechanism had two mistakes. The original denominator, 160 unique unresolved constants, dropped to 29 as soon as the alias-caused ones left the list with the fix in commit 7d3a781. And the original numerator, 78, was also wrong: the regex matched a constant by name suffix, not full path, so names like A and C counted because they happened to end the same way as a real T.let declaration somewhere else in the file. Both mistakes only surfaced when I swapped grep for a direct query against the RBS API, loading the 61 stdlib libraries and asking by type name instead of by text. The correct number, after that, is the one in the table above: 11 of 41, 27%. Swapping grep for the API fixed the suffix bug, but was not enough on its own: the query accepted a shortcut, a top-level name existing already counted as a match, so Gem existing was enough for Gem::DefaultUserInteraction to pass, when that constant's only real occurrence in the entire RBS is a comment. The full lesson is not "API instead of text": it is that the question has to be about the exact path, because accepting a prefix is suffix grep dressed up as an API.

The 7 ruby-lsp diagnostics that are not constant warnings have a complete partition, verified against the source, not a sample:

The code_action_resolve.rb:435 case deserves its own line: the #: as !nil cast stuck to the end on line 417 is read correctly, that is confirmed. What is missing is narrowing the type before the cast is reached: lines 410 to 417 are a case @code_action[:title] with three when branches and no else, and without that branch the type left over is String | nil. The cast is not ignored: it is applied to a type that already arrived wrong.

Not every row in the table is a bug. Narrowing on a case without else is a family declared out of scope by decision, a chosen boundary, not a flaw waiting to be found. The generic [T] treated as a fixed type is a real bug, and it is the third time this mechanism has shown up in this investigation: it already has a prioritized fix. The #: as untyped cast being ignored remains unfixed.

The framing that closes this section: the value is in untyped code. Across the 3 untyped repositories, the funnel produced 14 real bugs, 13 submissions, 7 accepted. Across the two repositories with Sorbet, Sorbet is already green and itaruby produces mostly false positives. That is scope, not defeat: where Sorbet is already configured and green, itaruby today does not add, it gets in the way.

The one real bug itaruby found across the two repositories with Sorbet, out of 111 raw diagnostics in those two repositories. Not a win, it is the exception that proves the scope. The bug: a raise AbstractMethodInvokedError inside module RubyIndexer when the constant only exists as RubyLsp::AbstractMethodInvokedError, a sibling module, not an ancestor. The raise never resolves; I reproduced the structure separately and confirmed it: it raises NameError. Of the 17 sites of this same raise in the repository, 16 are in lib/ruby_lsp/ and resolve correctly; only this one is outside the right namespace. It is a latent defensive guard, not a production crash, because both concrete subclasses already implement the abstract method. The PR is open, #4197 , on ruby-lsp: 2 lines added, none removed. Contributions from outside Shopify go through two gates before shipping: the company's CLA and first-contribution workflow approval, both pending now.

The most interesting data point in this section: this bug was in a repository that runs Sorbet in CI, green, and Sorbet does not catch it because the project's own sorbet/config --enable-experimental-rbs-comments flag silences this kind of guard. It is not "itaruby is stronger than Sorbet": it is a configuration flag that created a blind spot, and a tool that does not read the project's configuration went right through it without knowing it existed. That is the real argument for running two different tools instead of one.

Overlap between the two real-error sets: zero. That is not a finding, it is an absence of data: Sorbet's set in the two repositories is empty, so there is no intersection to measure. Not a win, it is zero degrees of freedom in the comparison.

The rate table above showed ruby-lsp at 1240 diagnostics per thousand files, the worst in the corpus, while being the most typed repository. Counterintuitive enough to be worth investigating the cause instead of taking the number at face value.

The cause has a name and a line, in lib/ruby_lsp/utils.rb:6-7:

Interface = LanguageServer::Protocol::Interface
Constant = LanguageServer::Protocol::Constant

itaruby did not follow this kind of constant alias; every reference to Interface::X or Constant::X in the rest of the file turned into an unresolved-constant diagnostic. That accounted for 405 of the 449 constant warnings in the repository.

Prediction made before touching the code: about 405 warnings should disappear with the fix. After the fix, commit 7d3a781: 408 disappeared. Prediction error of 0,7%.

That conclusion came from comparing totals per repository, before and after, and comparing totals does not establish a claim about a set: I made this exact mistake myself later, in wave 7, where rails's total came out identical between two waves, 2316 against 2316, hiding 5 diagnostics leaving and 5 entering. The set-level diff for this alias fix was never run, and I no longer have that wave's binary to run it now: I cannot say whether any new diagnostic showed up anywhere in the corpus.

Measured at commit 7d3a781, same file count as before.

What survives this comparison is magnitude, not the set: the untyped repositories barely moved, net deltas of rails -48, discourse -75, zammad -12, while the typed repository dropped 405 (also a net delta). If the gain had come spread out evenly, that would be cause for suspicion: a generic fix that reduces noise everywhere equally proves no mechanism at all. Here the gain landed exactly where the mechanism predicted it would.

Why did the fix cure ruby-lsp and leave tapioca untouched, when both have the same kind of alias? I isolated it with a 9-line test, three variations of the same alias, changing only where the target is declared:

In a .rbi file, a declaration written as a nested block indexes correctly: the full path exists in the index. The query is what fails, looking up the qualified path in a map where that name was registered unqualified, and giving up before applying the refilter that exists exactly for this. Only the compact form avoids the problem. In .rb both work. The two real lines, side by side:

# ruby-lsp, lib/ruby_lsp/utils.rb:7 (alias in .rb, cured)
Constant = LanguageServer::Protocol::Constant

# tapioca, sorbet/rbi/gems/ruby-lsp@0.26.10.rbi:2380 (alias in .rbi, not cured)
RubyLsp::Constant = LanguageServer::Protocol::Constant

The difference between the two repositories is not a code profile, it is a lookup bug with an address. And the consequence that matters to a reader: fixing this is not increasing coverage, it is knocking down false positives in the repository where the tool looks worst.

The reclassification closed, exhaustive, item by item, not a sample of 20. This replaces the provisional "undecided" reading I had left here: on ruby-lsp, false positives dropped from 70% (7 of 10, old sample) to 31% (16 of 51, full count); on tapioca it went the other way and it became the repository with the highest false-positive share in the corpus, 57% (34 of 60). srb tc remains at zero errors on both. The sentence from two hours earlier, "where Sorbet is already configured and green, itaruby today does not add, it gets in the way", no longer holds as a single sentence for both repositories: on ruby-lsp it got weaker, false positives are already a minority (31%); on tapioca it got stronger, false positives are a clear majority (57%), and it was not a terminal state: the cause was the .rbi lookup bug above, it had an address in the code, and the fix went in. The result is in the "Wave 8" section: tapioca dropped from 17 to 1, a documented ceiling.

Wave 8 replaces wave 7 as the current snapshot: binary compiled from commit 5c5c8c3, 2026-08-26, same trees pinned by sha as before. Composition by diagnostic code, all six repositories in the corpus:

chatwoot had its first measurement in wave 7. This is the first comparison that exists for it: the set diff silenced 2 diagnostics, zero new ones.

Of ruby-lsp's 30, 3 are E0001 (syntax error) in test/fixtures/syntax_diagnostics.rb and test/fixtures/multiple_invocations.rb , files that are invalid Ruby on purpose, written for ruby-lsp itself to test its own syntax diagnostic. The checker is right there: it is not a hit against a real bug, it is recognizing a deliberate fixture for what it is.

tapioca, which this page classified as 60 diagnostics, 0 real bug and 34 false positives (57%, the worst share in the corpus), dropped to 1. tapioca is not clean: 1 remains, and it is a documented ceiling. The author says this remaining case is a case where INSTANCE is a value, a known limitation, not an open defect.

Trajectory of the two typed repositories, four measurement rounds: ruby-lsp 456 -> 51 -> 31 -> 30; tapioca 60 -> 60 -> 17 -> 1.

Set diff between wave 7 and wave 8, keyed by path, line, column and code:

Zero new diagnostics across the six repositories in the corpus. This time the claim is established by a set diff, not by a totals comparison, which is exactly the distinction the "Comparing totals is not comparing sets" section calls out. Where this page already said that claim had not been established for a given wave, it still holds for that wave: this is where, in wave 8, it became established.

The diff key gained a column, and this is a correction to this page's own published rule. The "Comparing totals is not comparing sets" section, further down, says the diff must be keyed by file, line and code, never by message. That rule was incomplete: file:line:code does not distinguish two diagnostics with the same code on the same line, and there are real pairs like that in discourse. Keying without a column makes those pairs look silenced when both are still correctly flagged. The correct key includes the column: file:line:column:code.

Most of discourse's drop, 1658 , is the single root cause already covered in the section below: 1642 of those are DB written with cbase. The rest, 16 , is the defined? guard.

Wave 7 was declared drift-free by the checker's author. I remeasured it, 5c5c8c3, and what looked like 4 isolated false positives were two root causes with 1675 sites total, not four separate cases, one of them an order of magnitude bigger than the first read suggested.

Root 1, top-level constant write with cbase: 1645 sites across 3 repositories.

When code writes a top-level constant using cbase form, ::X = value, that constant never made it into the top-level constant table. The other side of the same defect: the existence check keyed the cbase fallback off the untrimmed name.

Root 2, defined? guard: 30 sites across 3 repositories.

Flagging defined?'s argument was never the problem: the checker already got that right. The real false positive was reading that same constant in the if or ternary's then branch, the branch that only runs when it exists. The rule that fixed it controls both sides: a read in the else branch is still flagged, because there it is genuinely unsafe, and defined?(X) guarding a read of Y still flags Y.

The single biggest name in the whole corpus was not a gem constant missing a declaration: it was a lookup bug on our side. One line of discourse code produced 1642 false diagnostics.

Both roots were fixed at the source in wave 8, binary compiled from commit 5c5c8c3, and I verified the fix myself:

The proof for the other side, the one that separates fixing from suppressing: constants that are genuinely gem constants keep getting flagged. ::Trilogy stays at 3 in rails/rails, unchanged. On top of that, zero new diagnostics across the six repositories, and this time that claim is established by a set diff, not a totals comparison.

Worth publishing as verification, not praise: the attribution the author sent (cbase versus the defined? guard) sums up exactly to the totals I measured independently, repository by repository:

Two counts taken through different paths landed on the same number, repository by repository. It is the opposite of the wave 7 case, where the composition did not add up to the total and 3 were left unaccounted for.

The method finding that triggered this check: comparing totals is not comparing sets. On rails, the total came out identical between the two waves, 2316 and 2316, and underneath, 5 diagnostics left and 5 entered: a net-zero delta was hiding 10 items moving.

Worse: if the diff had been keyed by message instead of by file:line:code, rails would have flagged 187 changes, because wave 7 started printing the constant name in absolute form (::WebSocket::Driver instead of WebSocket::Driver) and every diagnostic's text changed without its site changing. A large false number is more dangerous than a small correct one.

The value of measuring things yourself is not distrusting the other side: it is that a net-zero delta and a cosmetic rewrite hide things in opposite directions.

This page itself made this exact mistake in the alias-fix section, further up: the passage was corrected, not deleted.