Repository navigation
Add order dependence tests - #65291
Open
Jake Bailey (jakebailey) wants to merge 2 commits into
Open
Jake Bailey (jakebailey) wants to merge 2 commits into
Jake Bailey (jakebailey) wants to merge 2 commits into
Conversation
Checker results should not depend on which files were checked first, on which types an editor happened to ask for before diagnostics were requested, or on whether a file was emitted before it was checked. Tests had no way to show such dependence, so fixes for it could only be described, not demonstrated. The harness-only checkOrder option queries the checker in a different order: files in reverse, each file in its own program, or every expression's type, symbol, contextual type, resolved signature, and the types of names in type annotations before any diagnostics. The same orders apply to emit. Listing several orders produces a baseline for each, so order dependence shows up as differing baselines. Fixed orders over whole files miss results that depend on which single query a checker answers first. With TS_CHECK_ORDER_FUZZ=n, each test without a checkOrder option is checked again n times in fresh programs that first answer a seeded random selection of queries and single file emits. Any change in diagnostics, emitted output, or the type or symbol of a node fails the test with the seed and a minimized list of queries that reproduces it. Symbol IDs in printed names are ignored, as they follow creation order by design. The fuzzer is off by default, so it does not slow the normal suite.
typescript-automation
Bot
added
Author: Team
For Uncommitted Bug
Copilot stopped reviewing on behalf of
Jake Bailey (jakebailey) due to an error
November 3, 2026 21:25
These tests record checker results that depend on the order in which files, declarations, and queries are checked, so that each later fix shows up as a reviewable baseline change. Most list several checkOrder variants; where the variants disagree, that disagreement is the bug. Some existing tests also gain the typesfirst order, as asking for types and symbols first changes what they report. The cases cover: - variance measured through cycles of recursive references, which depended on which type in the cycle was measured first (64891, and a reduction of the react-relay DefinitelyTyped tests); - circularities whose blamed location is whichever use resolved first: circular type parameter defaults and constraints, a cycle through namespace spreads, awaiting a promise that refers back to itself, and import elision computing types before checking; - instantiation limit and excessive depth errors, reported only in whichever file reached the limit first; - late-bound members, computed enum members, and aliased module.exports members, which differ when queried before checking; - relation results cached against truncated or in-progress comparisons; - the order of types in unions, which followed type ids, and so creation order, for object literals, rest types, indexed accesses, type parameters, and iterator method types; - return type cycles through callbacks and subtype reduction, JSDoc type tags on contextually typed functions, a scope whose thunks refer to the scope, a never narrowing after a loop, and a const assertion inferred before its binding pattern (seen in OmniRoute, effect, webpack, arktype, uglify-js, and check order fuzzing). The circularity tests from PR 64834 are copied unchanged, to compare against how that PR handles the same cases.
Jake Bailey (jakebailey)
force-pushed
the
jakebailey/order-dependence-tests
branch
from
November 3, 2026 22:00
7a8d8b1 to
fe126ea
Compare
Jake Bailey (jakebailey)
requested a review
from Ryan Cavanaugh (RyanCavanaugh)
November 3, 2026 22:14
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the first part of #65064 pulled out, adding new checker ordering tests and fuzzing
There's a new
checkOrdertest option, which makes certain tests variant on interesting orderings, like "what if we visit every single node before checking" or "what if we reverse the order of the files" or even "what if we use PRNG to randomly walk nodes and then check". This up front found on the order of 500 failing tests we already had that were secretly order dependent. These were all fuzzed out and kept.There are a significant number of new tests which all show various ordering bugs never before discovered.