build: Improve testing infra for repo tests.

.github/workflows/ci.yml

@@ -13,8 +13,6 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: '2.7'
       # Keep in sync with release.yml
       - name: Manually evict cache entry if applicable
         run: ACCESS_TOKEN='${{ secrets.GITHUB_TOKEN }}' python3 .github/workflows/evict.py
@@ -46,4 +44,11 @@ jobs:
       - name: "🏋️‍♂️ Run snapshot tests"
         run: rm log && ./bazel test //test/scip --config=dbg
       - name: "🏋️‍♂️ Run repo tests"
-        run: ./bazel test //test/scip/repos --test_env GITHUB_ACTIONS=1 --test_env GEM_PATH="$(dirname "$(which gem)")" --config=dbg
+        run: |
+          sudo apt-get install --reinstall build-essential
+          ls "$(dirname "$(which gcc)")"
+          echo "$PATH"
+          ls /usr/bin /usr/local/bin /bin
+          # Technically, this is for Ruby 2.7.0 but it's probably fine.
+          sudo apt-get install ruby2.7-dev
+          ./bazel test //test/scip/repos --test_env GITHUB_ACTIONS=1 --test_env GEM_PATH="$(dirname "$(which gem)")" --config=dbg

.github/workflows/release.yml

@@ -35,6 +35,7 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       # Keep in sync with ci.yml
+      - uses: actions/setup-ruby@v1
       - name: Manually evict cache entry if applicable
         if: ${{ runner.os == 'Linux' }}
         run: ACCESS_TOKEN='${{ secrets.GITHUB_TOKEN }}' python3 .github/workflows/evict.py

.github/workflows/setup-bazel.sh

@@ -4,10 +4,22 @@ set -euo pipefail
 
 CACHE_ROOT="$HOME/bazelcache"
 mkdir -p "$CACHE_ROOT/output-base" "$CACHE_ROOT/build" "$CACHE_ROOT/repos"
+
+os="$(uname -s)"
+if [ "$os" == "Linux" ]; then
+  SETUP_RUBY_TOOLS_DIR="/opt/hostedtoolcache/Ruby"
+elif [ "$os" == "Darwin" ]; then
+  SETUP_RUBY_TOOLS_DIR="/Users/runner/hostedtoolcache/Ruby"
+else
+  echo "Check https://github.com/ruby/setup-ruby#using-self-hosted-runners and note setup-ruby's cache directory here"
+  exit 1
+fi
+
 {
-  echo 'common --curses=no --color=yes'
+  echo "common --curses=no --color=yes "
   echo "startup --output_base=$CACHE_ROOT/output-base"
   echo "build  --disk_cache=$CACHE_ROOT/build --repository_cache=$CACHE_ROOT/repos"
+  echo "build --define SCIP_RUBY_CACHE_RUBY_DIR=$SETUP_RUBY_TOOLS_DIR --define SCIP_RUBY_RBENV_EXE=RUNNING_IN_CI_RBENV_NOT_NEEDED"
   if [ -n "${VERSION:-}" ]; then
     echo "build --//gems/scip-ruby:version=$VERSION"
   fi

.gitignore

@@ -1,6 +1,8 @@
 /bazel-*
 /.bazelrc.local
+/.cache_ruby
 /compile_commands.json
+/.homebrew-x86_64
 /.gdb_history
 /tags
 

BUILD

@@ -0,0 +1 @@
+exports_files([".ruby-version"])

README.md

@@ -94,4 +94,4 @@ using the [Sourcegraph CLI](https://github.com/sourcegraph/src-cli).
 
 ## Contributing
 
-See the [Contributing docs](./scip-ruby-CONTRIBUTING.md).
+See the [Contributing docs](./docs/scip-ruby/CONTRIBUTING.md).

WORKSPACE

@@ -1,9 +1,11 @@
 workspace(name = "com_stripe_ruby_typer")
 
-load("//third_party:externals.bzl", "register_sorbet_dependencies")
+load("//third_party:externals.bzl", "register_scip_ruby_dependencies", "register_sorbet_dependencies")
 
 register_sorbet_dependencies()
 
+register_scip_ruby_dependencies()
+
 load("@com_grail_bazel_compdb//:deps.bzl", "bazel_compdb_deps")
 
 bazel_compdb_deps()

scip-ruby-CONTRIBUTING.md → docs/scip-ruby/CONTRIBUTING.md

@@ -1,20 +1,99 @@
 # Working on scip-ruby
 
+This document covers the day-to-day aspects of `scip-ruby`.
+For questions on why certain things are the way they are,
+see the [Design Decisions doc][] doc.
+
+[Design Decisions doc]: DESIGN.md
+
+- [Install dependencies](#install-dependencies)
+- [Configuring Ruby (optional)](#configuring-ruby-optional)
 - [Building](#building)
 - [IDE integration](#ide-integration)
 - [Writing a new snapshot test](#writing-a-new-snapshot-test)
 - [Writing a new repo test](#writing-a-new-snapshot-test)
 - [Debugging with print statements](#debugging-with-print-statements)
 - [Debugging with LLDB](#debugging-with-lldb)
+- [Debugging build issues](#debugging-build-issues)
+  - [Debugging Bazel](#debugging-bazel)
+  - [Debugging on Linux](#debugging-on-linux)
 - [Creating PRs](#creating-prs)
 - [Cutting a release](#cutting-a-release)
 - Troubleshooting
   - [Known build issues][]
-  - [Known Ruby installation issues][]
   - [Known RubyGems related issues](#known-rubygems-related-issues)
+  - [Known testing issues](#known-testing-issues)
 
 [Known build issues]: #known-build-issues
-[Known Ruby installation issues]: #known-ruby-installation-issues
+
+## Install dependencies
+
+1. C++ toolchain (gcc or clang): Used for bootstrapping.
+2. [rbenv][]: (optional) This is used for [Configuring Ruby](#configuring-ruby-optional).
+
+[rbenv]: https://github.com/rbenv/rbenv#installation
+
+## Configuring Ruby (optional)
+
+If you're going to be running the repository tests or building the
+scip-ruby gem locally, follow these steps before building stuff.
+
+### macOS pre-requisites
+
+arm64 macOS builds aren't supported yet,
+so it is simpler to have a consistent set of tools
+where Homebrew is running under Rosetta too.
+
+```
+mkdir ~/.homebrew-x86_64
+curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C ~/.homebrew-x86_64
+./xbrew --version # check that it works
+```
+
+**WARNING:** Do not put this `brew` on your PATH!
+That may lead to confusing errors when working on other projects.
+
+### rbenv setup
+
+#### Linux
+
+Install [rbenv][] as per the official instructions. For `ruby-build`,
+do NOT install it as a plugin.
+
+<details>
+<summary>Why not install `ruby-build` as a plugin?</summary>
+
+The `RBENV_ROOT` variable does double-duty;
+it serves as the location to install the tools, and it is also used to
+locate plugins (like `ruby-build` -- if you install it as a plugin).
+However, in our case, since we are using a different tool path,
+`rbenv` will be unable to use the `install` plugin.
+
+To mimic the configuration on macOS, I recommend
+installing `ruby-build` next to the `rbenv` binary.
+</details>
+
+```
+git clone https://github.com/rbenv/ruby-build.git --depth=1
+# Assuming you installed rbenv through git as is recommended in the README.
+PREFIX="$HOME/.rbenv" ./ruby-build/install.sh
+rm -rf ruby-build
+```
+
+```bash
+echo "build --define SCIP_RUBY_CACHE_RUBY_DIR='$PWD/.cache_ruby' --define SCIP_RUBY_RBENV_EXE='$(which rbenv)'" >> .bazelrc.local
+```
+
+#### macOS
+
+```bash
+./xbrew install rbenv
+echo "build --define SCIP_RUBY_CACHE_RUBY_DIR='$PWD/.cache_ruby' --define SCIP_RUBY_RBENV_EXE='$(./xbrew where rbenv)'" >> .bazelrc.local
+```
+
+If you're wondering why we have this separate kind of caching,
+instead of having everything happen through the Magic of Bazel (TM),
+see the [Design Decisions doc][].
 
 ## Building
 
@@ -92,7 +171,7 @@ First, clone the repo using Sorbet locally
 and check if you can index it.
 Typically, the commands will be something like:
 
-```
+```bash
 BUNDLE_WITH=sorbet bundle install
 
 # Replace srb binary with scip-ruby binary
@@ -101,7 +180,7 @@ bundle exec srb --index-file index.scip --gem-metadata "name@version"
 ```
 
 In case there are any type errors, create a patch and save it:
-```
+```bash
 git diff > /path/to/test/scip/repos/name-version.patch
 ```
 
@@ -150,6 +229,51 @@ popd
 unset TEST_DIR
 ```
 
+## Debugging build issues
+
+### Debugging Bazel
+
+See Keith Smiley's blog post [Debugging bazel actions](https://www.smileykeith.com/2022/03/02/debugging-bazel-actions/). ([archive link](https://web.archive.org/web/20220711000725/https://www.smileykeith.com/2022/03/02/debugging-bazel-actions/))
+
+One KEY thing to keep in mind is that some problems only manifest
+inside the sandbox but not outside because Bazel sandbox changes
+the binaries available at certain paths to use non-system tools.
+
+For example, `uname -m` on an M1 Mac would normally return `arm64`
+but inside an x86_64 Bazel environment (currently the default)
+it will return `x86_64`.
+
+For compiler issues inside a sandbox,
+it helpful to test a small code snippet first.
+
+```bash
+{
+  echo '--- Try compiling some simple stuff ---'
+  {
+    echo '#include <stdio.h>'
+    echo ''
+    echo 'int main() {'
+    echo '  printf("Hello %s\n", "World!");'
+    echo '  return 0;'
+    echo '}'
+  } > tmp.c
+  gcc tmp.c
+  ./a.out
+  rm tmp.c ./a.out
+  echo '--------------------------------------'
+} >&2
+```
+
+### Debugging on Linux
+
+Debugging a build issue in GitHub Actions can get emotionally draining quickly.
+
+If you work at Sourcegraph, set up a GCP VM.
+You may find the companion [VM setup script](./vm_setup.sh) helpful.
+It is not tested in CI, so you may need to tweak it a bit.
+
+Don't forget to delete the instance after you're done investigating!
+
 ## Creating PRs
 
 PRs created through the GitHub UI default to being made
@@ -200,15 +324,6 @@ See the [release workflow](/.github/workflows/release.yml) for details.
                 ^~~~~~~~~~~~~~~~~~~~~~~
    ```
 
-### Known Ruby installation issues
-
-1. On macOS, Ruby 2.7.x may fail to install due to a combination of `-Werror`
-   and a warning in OpenSSL. As a workaround, you can use the following asdf invocation:
-   ```
-   OPENSSL_CFLAGS=-Wno-error=implicit-function-declaration asdf install ruby 2.7.2
-   ```
-   This works with 2.7.2, but not with 2.7.0.
-
 ### Known RubyGems related issues
 
 1. If you're trying to run `bundle install` for a gem
@@ -227,3 +342,13 @@ See the [release workflow](/.github/workflows/release.yml) for details.
    Network Preferences > Advanced > TCP/IP >
    Configure IPv6: set to Link-local only.
    You need to Apply the settings after changing that field.
+
+### Known testing issues
+
+I can't quite figure out why, but with some of the custom rules,
+the executable script being modified doesn't trigger a rebuild.
+If you're seeing stale results, try something like:
+
+```bash
+rm -rf .cache_ruby/ && (cd bazel-bin && find . -name '*.tgz' -type f -delete)
+```

docs/scip-ruby/DESIGN.md

@@ -0,0 +1,77 @@
+# Design Decisions
+
+## Repo tests
+
+<!-- DEF NOTE[repo-test-structure] -->
+
+The repo tests have a couple of unusual things:
+#. Tests are broken up into 3 stages,
+   instead of being a simple script that does clone
+   + apply patch + invoke dev tool (`scip-ruby`).
+#. There is "manual" caching going on for the intermediate results
+   of these steps. The cache cannot be used in parallel; repo tests run serially.
+
+I'll cover each of these one-by-one.
+
+Some constraints to keep in mind while reading the sections below:
+#. We want to run tests using as close to "real-world" tools (such as Bundler)
+   as possible instead of trying to mimic what they do.
+#. We want to cache intermediate results as much as possible,
+   as many steps in Ruby installation + Gem installation are slow.
+#. Tests should hopefully run reasonably quickly, especially if you only
+   modify the indexer (most common situation).
+#. Tests should be isolated from each other. We don't one test to "install"
+   `scip-ruby` and have that be visible to another test.
+
+Before going into details about the caching, first let's cover
+an outline of the 3 stages.
+### 3 stages of repo tests
+
+#. Create a standalone ruby installation: This is cached
+   and the installation is blown away.
+#. Test prep: This step involves installing the dependencies of the gem,
+   including any custom bundler version it may require.
+   The pristine toolchain from the first step is extracted
+   and we run `bundle cache`.
+   After this is done, the _entire_ Ruby toolchain is cached again
+   (because it will be have been modified by `bundle cache`).
+   The modified source tree is also cached (I tried `--frozen` earlier but
+   ran into some error with it that I didn't have time to look into).
+#. Index: This step reuses the two caches from the previous step,
+   applies a patch, copies the `scip-ruby` gem into the `vendor/cache`
+   directory, runs `bundle install --local` (so nothing should be
+   fetched from the internet). Now, everything is in a state which mostly
+   mimics what a user would have. Then we invoke `scip-ruby` normally.
+
+Intermediate results use tarballs for caching directory trees instead of copying
+directories because it seemed much simpler to deal with a tarball in Bazel.
+
+Why do we go through all this trouble to cache things "manually"?
+### Manual caching
+
+Here are some obstacles the caching system needs to overcome.
+
+* Ruby installation is [path-dependent](https://github.com/ruby/setup-ruby#using-self-hosted-runners).
+* Ruby installation is relatively slow as it builds everything from source:
+  it takes a few minutes even on a Macbook Pro.
+  Installing it per-test would be too time-consuming.
+
+These two factors together imply that we should cache a Ruby installation
+inside a stable directory, not a temporary test directory.
+
+Additionally, we only want one Ruby toolchain cache; to ensure test isolation,
+we run them in serial on the same directory (a GIL for the test suite).
+
+* We want to ensure reproducibility, and not mess with the global environment.
+
+To achieve this, we delete the toolchain directory at the start of a test,
+and restore it from the cache in the 'Test prep' step.
+
+### Bonus: Bundler weirdness
+
+If you run `bundle install --local` for a test gem X,
+after copying an updated `scip-ruby` gem (same version, new checksum)
+into X's `vendor/cache` directory,
+you would naively expect `bundle` to re-install the updated `scip-ruby` gem.
+(install = copy/setup stuff in toolchain-adjacent directories)
+But `bundle` doesn't actually do that. 🙈