[awe-srush]

Summary

Adds TreeSitterParserBase, a shared abstract base class that sits between AbstractLanguageParser and any future tree-sitter-backed language parser. This PR introduces the infrastructure layer only, no language-specific logic is included. The first concrete subclass (PythonParser) follows in a separate PR.

Objective

Vanir's parser pipeline expects every language parser to produce a ParseResults object containing:

• function_chunks - one FunctionChunkBase per function, carrying its name, parameters, return types, used data types, local variables, called functions, and raw tokens
• line_chunk - a LineChunkBase with all file tokens grouped by line number, used for line-based signature matching
• parse_errors - a list of ParseError for any syntax errors encountered

Existing parsers (C/C++, Java) fulfill this contract via ANTLR4 grammars. This PR fulfills the same contract using tree-sitter, enabling faster parser development for new languages with no grammar compilation step.

Implementation

tree_sitter_base.py

TreeSitterParserBase (class)

The core class. Subclasses set three class-level attributes:

• LANGUAGE - a tree_sitter.Language instance for the target language
• _FUNC_QUERY - a compiled tree-sitter Query for matching function definitions
• SKIP_TOKEN_TYPES - a frozenset of node types to exclude from token collection (e.g. comments)
• STRING_NODE_TYPE - the node type to treat as an opaque string token (default: 'string')

__init__(filename)

• Reads the source file as bytes
• Instantiates a tree-sitter Parser with the subclass's LANGUAGE
• Parses the file into a concrete syntax tree (self._tree)
• Tree-sitter elements used: Parser, tree_sitter.Language

get_chunks(affected_line_ranges_for_functions), main pipeline

Implements the full ParseResults assembly in four steps:

Step 1, Function discovery

• Runs _FUNC_QUERY against the root node using QueryCursor
• Extracts func.def and func.name capture groups from each match
• Tree-sitter elements used: QueryCursor, Query.matches(), named captures

Step 2, FunctionChunkBase construction

• For each matched function node, computes start/end line numbers from start_point / end_point
• Filters by affected_line_ranges_for_functions using _overlaps()
• Extracts function name from the func.name capture node
• Retrieves the parameters child field via child_by_field_name('parameters')
• Delegates to three abstract methods (see below) for language-specific extraction
• Identifies nested function byte ranges to exclude from local/call collection
• Collects flat token list via _flat_tokens_cursor()

Step 3, LineChunkBase construction

• Runs _collect_tokens_cursor() over the entire root node
• Produces a dict[int, list[str]] mapping line numbers to token lists
• Passed directly into LineChunkBase

Step 4, Parse error collection

• Runs _collect_errors_cursor() to find all ERROR nodes in the tree
• Each ERROR node becomes a ParseError with line, column, and token preview

Cursor utility functions

_collect_tokens_cursor(node, skip_types, string_type, out_dict)

• Iterative DFS using tree-sitter's TreeCursor (avoids Python recursion limits)
• Prunes subtrees whose root type is in skip_types (e.g. comments, decorators)
• Emits nodes of string_type as a single opaque token
• Emits all other leaf nodes (no children) as individual tokens
• Populates out_dict[line_number] → [token, ...]
• Tree-sitter elements used: node.walk(), TreeCursor.goto_first_child(), goto_next_sibling(), goto_parent(), node.text, node.start_point

_flat_tokens_cursor(node, skip_types, string_type)

• Thin wrapper over _collect_tokens_cursor()
• Returns a flat ordered list of token strings (sorted by line, then order within line)

_collect_errors_cursor(root)

• Iterative DFS that visits every node without pruning
• Collects all ERROR nodes and returns them as ParseError objects
• Truncates bad token display to 80 characters
• Tree-sitter elements used: node.type == 'ERROR', node.start_point, node.text

_overlaps(func_start, func_end, ranges)

• Pure utility, returns True if a function's line range overlaps any of the provided affected line ranges
• Empty ranges means no filter (include all functions)

Abstract methods (subclasses must implement)

_extract_param_names(params_node) → list[str]

Extracts ordered parameter name strings from a tree-sitter parameters node. Subclasses traverse the parameter node's children according to the language's grammar (e.g. identifier, typed_parameter, default_parameter).

_extract_annotations(func_node) → (return_types, used_data_types)

Extracts type annotation information from a function definition node:

• return_types - list of token lists from the return type annotation (empty if none)
• used_data_types - list of token lists, one per annotated parameter

_collect_locals_calls(body_node, nested_ranges) → (local_vars, called_fns)

Walks the function body node to collect:

• local_vars - set of locally assigned variable names
• called_fns - set of called function names
• nested_ranges - byte ranges of nested functions to exclude from collection, preventing inner-scope variables from leaking into the outer function's signature

Design decisions

• No module-level tree-sitter import, the base class is importable on Python 3.9 (where tree-sitter is not installed) without error. All tree-sitter imports are lazy, inside methods.
• Iterative cursor DFS, avoids Python's default recursion limit on deeply nested ASTs. All three traversal utilities use TreeCursor instead of recursive calls.
• Language-agnostic, zero Python-specific logic in this file. All language grammar knowledge lives in concrete subclasses.

Dependencies

This PR is independent and can be merged alongside:

• feat/python-parser-deps
• feat/python-parser-build
• bug/fix-subclass-discovery

It must be merged before feat/python-parser.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[varungoyal-ittiam]

Thank you for these contributions (PR#21 through PR#25) to integrate Tree-sitter!

To help us test and import these changes successfully, could you please consolidate these 5 PRs into a single pull request? (Note: A single PR can still contain multiple commits). Alternatively, if they need to remain separate, please re-structure them so they can be merged independently.

Why this is needed: Our continuous integration and import pipeline requires changes to be validated atomically. Because these five PRs depend on one another, testing them individually causes temporary compilation and test failures during import. Having them in a single PR allows us to run validation checks on the complete integration at once.

Author Attribution: We want to ensure everyone who contributed gets proper credit. When the consolidated change is imported:

• The base commit author is considered the primary author and will be attributed in the git commit author field.
• All additional contributors will be credited in the commit message via standard Co-authored-by: Name <email> footers.

As outlined in the GitHub documentation, Co-authored-by: trailers ensure that contributions are reflected on each contributor's graph and repository statistics, provided standard contribution criteria are met (which everyone involved here already satisfies by having opened a PR or created a fork).

Thank you for your co-operation and assistance!

[awe-srush]

Thanks for this info. Let me share a single PR with all the changes!

[awe-srush]

Thank you for these contributions (PR#21 through PR#25) to integrate Tree-sitter!

To help us test and import these changes successfully, could you please consolidate these 5 PRs into a single pull request? (Note: A single PR can still contain multiple commits). Alternatively, if they need to remain separate, please re-structure them so they can be merged independently.

Why this is needed: Our continuous integration and import pipeline requires changes to be validated atomically. Because these five PRs depend on one another, testing them individually causes temporary compilation and test failures during import. Having them in a single PR allows us to run validation checks on the complete integration at once.

Author Attribution: We want to ensure everyone who contributed gets proper credit. When the consolidated change is imported:

• The base commit author is considered the primary author and will be attributed in the git commit author field.
• All additional contributors will be credited in the commit message via standard Co-authored-by: Name <email> footers.

As outlined in the GitHub documentation, Co-authored-by: trailers ensure that contributions are reflected on each contributor's graph and repository statistics, provided standard contribution criteria are met (which everyone involved here already satisfies by having opened a PR or created a fork).

Thank you for your co-operation and assistance!

@varungoyal-ittiam - single PR with all commits has been raised: #27
Let me know if anything else is required!

[varungoyal-ittiam]

Thanks for the quick reply and for consolidating the PRs!

I've verified the import configuration locally and confirmed that the author attribution is captured as per the individual commit information:

• Primary Author: Awe Srush <srushti.erudite@gmail.com>
• Co-authors:
  • Co-authored-by: Avantika Sharma <avantikaspare15@gmail.com>
  • Co-authored-by: Ishika Johari <ishijohari@gmail.com>