Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/CI.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# CI selection and acceptance ownership

Root `bun run type-check` and CI's lint job also check ESM maintenance scripts through `scripts/tsconfig.mts.json`, which inherits the shared scripts compiler options and selects every `.mts` file. The shared scripts project includes `.ts`, `.tsx`, and `.mts` for editor checking; CI's additional scripts gate is scoped to `.mts`.

The `changes` job runs Turborepo 2.11.10's `affectedTasks` query before installing application dependencies or provisioning services. `turbo.json` records package dependency propagation; `apps/sim/turbo.json` partitions the app into shared runtime, server, workspace UI, and acceptance tasks. `.github/ci-suites.json` turns those tasks into job matrices. The uploaded `ci-plan` artifact names changed files, selected tasks, attribution reasons, and any fallback.

A PR compares the checked-out GitHub merge commit with its first parent: the actual target revision tested by that merge. The PR event's base SHA can become stale when the target advances. Scoping requires a matching GitHub merge ref and SHA, exactly two merge parents, and a usable target parent; otherwise every check runs. This works with the existing depth-two checkout without fetching older ancestry. Main/staging pushes execute all checks; the parent `ci` workflow skips checks on `dev` pushes, while PRs targeting `dev` still run checks. A manual `checks` run executes all checks unless supplied a full base commit SHA to reproduce a scoped plan. Missing history, an empty diff, unknown files outside the workspace graph, root script/CI changes, or a planner error run every check. Missing selection outputs fail the aggregate gate. The gate also rejects an omitted dependency, any failed/cancelled job, and any skip that its explicit selection flag did not authorize.
Expand Down Expand Up @@ -34,6 +36,10 @@ Database integration uses 4-vCPU Blacksmith runners with sticky dependency disks

GitHub runners install dependencies directly rather than transferring Bun and `node_modules` archives. In a same-source paired comparison, those restores took 26.2 seconds before an 82-millisecond install, while a clean install took 12.42 seconds. This is one paired observation; retain the policy only while repeated CI timings support it. Blacksmith keeps its sticky dependency disks. GitHub Turbo output caches remain enabled for jobs that request them, with a commit-specific key and a task/event/fork-scoped restore prefix, so a rerun can reuse an exact entry without creating another archive. Push jobs never restore pull-request entries.

Trusted Blacksmith E2E builders can reuse Next's incremental compiler state when a complete Turbo build is not already cached. The compiler cache has its own mode, event, OS/architecture, resolved toolchain, lockfile and build-recipe namespace; fork PRs and GitHub fallback keep the compiler cache off. The build command always runs, and an uncertain cache probe restores state rather than skipping compilation.

Compiler maintenance enforces a 20 GiB regular-file working-state limit before and after compilation, preserves the mounted root, and fails closed on access or ownership errors. Dev maintenance remains separate and nonblocking. Sticky disks are optional last-good compiler snapshots, not cross-job artifact handoff or validation receipts; no freshest-snapshot or physical-storage eviction guarantee is assumed. Hosted/deployment builds remain cache-off until separately qualified. A cold E2E qualification job took 259 seconds versus 225 uncached; fresh-runner warm and changed-source jobs took 148 and 159 seconds versus 227 and 229 uncached. These paired observations do not establish a universal CI speedup or justify smaller runners. Compilation and all existing acceptance coverage remain required.

Closing a same-repository pull request deletes only its GitHub merge-ref caches after verifying that it remains closed or merged. The regular `pull_request` workflow skips forks and Dependabot because their tokens cannot delete caches; those entries retain GitHub's normal cache eviction. Cleanup checks out no code and never selects a branch-wide or default-branch cache. Blacksmith sticky disks retain their separate lifecycle.

Memory-heavy builds and browser jobs retain appropriately sized Blacksmith runners. Electron uses 4-vCPU Blacksmith: both shards passed within 14 seconds of the 8-vCPU control; the free GitHub rest shard was slower and its lease run stalled.
Expand Down
48 changes: 43 additions & 5 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ jobs:
- &checkout-mirror
name: Checkout code (mirror)
if: (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && github.event_name == 'pull_request' && !github.event.pull_request.head.repo.fork
uses: useblacksmith/checkout@2b9fe87d796feea153b455ec2f3125b471a7b0e7 # v1.8.2
uses: useblacksmith/checkout@75c49b6ed30ce59336e9252425796cfcfac155f8 # v1.8.3
- &checkout-plain
name: Checkout code
if: (vars.CI_PROVIDER != '' && vars.CI_PROVIDER != 'blacksmith') || github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork
Expand Down Expand Up @@ -612,7 +612,7 @@ jobs:
# Same provider/trust split as the build jobs' checkout.
- name: Checkout code (mirror)
if: (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && github.event_name == 'pull_request' && !github.event.pull_request.head.repo.fork
uses: useblacksmith/checkout@2b9fe87d796feea153b455ec2f3125b471a7b0e7 # v1.8.2
uses: useblacksmith/checkout@75c49b6ed30ce59336e9252425796cfcfac155f8 # v1.8.3
with:
fetch-depth: 2
- name: Checkout code
Expand Down Expand Up @@ -733,6 +733,9 @@ jobs:
- name: Migration safety (zero-downtime) audit
run: bun run check:migrations "${{ steps.audit_base.outputs.ref }}"

- name: Type-check ESM maintenance scripts
run: bun run type-check:scripts:esm

# Every affected workspace, following the same real task graph as tests. Root/unknown
# changes select all. packages/emcn, packages/utils,
# apps/desktop and apps/docs had no type check in CI at all; apps/sim's
Expand Down Expand Up @@ -821,7 +824,7 @@ jobs:
steps:
- name: Checkout code (mirror)
if: needs.changes.outputs.test_app == 'true' && (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && github.event_name == 'pull_request' && !github.event.pull_request.head.repo.fork
uses: useblacksmith/checkout@2b9fe87d796feea153b455ec2f3125b471a7b0e7 # v1.8.2
uses: useblacksmith/checkout@75c49b6ed30ce59336e9252425796cfcfac155f8 # v1.8.3
with:
fetch-depth: 2
- name: Checkout code
Expand Down Expand Up @@ -963,6 +966,8 @@ jobs:
if: needs.changes.outputs.e2e_build == 'true'
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-16vcpu-ubuntu-2404' || 'linux-x64-8-core' }}
timeout-minutes: 25
env:
NEXT_BUILD_PERSISTENT_CACHE: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && (github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork) && 'true' || 'false' }}

steps:
- *checkout-mirror
Expand All @@ -982,8 +987,10 @@ jobs:
echo "::warning::Runner has ${TOTAL_GB} GB. A cold-cache build peaks ~51 GB, so this run may be OOM-killed (reported only as 'the runner has received a shutdown signal'). Warm/partial builds should still fit."
fi

- name: Build application
env:
- name: Plan incremental compiler state
id: compiler-cache-plan
if: env.NEXT_BUILD_PERSISTENT_CACHE == 'true'
env: &e2e-build-env
NODE_OPTIONS: '--no-warnings --max-old-space-size=8192'
NEXT_PUBLIC_APP_URL: 'http://localhost:3000'
DATABASE_URL: 'postgresql://postgres:postgres@localhost:5432/simstudio'
Expand All @@ -997,8 +1004,39 @@ jobs:
# lint's type-check and the `build` job's own `next build` (generated route types too)
# already enforce.
NEXT_BUILD_SKIP_TYPE_CHECK: 'true'
NEXT_BUILD_SKIP_SOURCE_MAPS: 'true'
run: |
echo "toolchain=bun$(bun --version)-node$(node --version)" >> "$GITHUB_OUTPUT"
if bunx turbo run build --filter=@sim/app --dry=json > "$RUNNER_TEMP/e2e-build-plan.json" &&
jq -e 'type == "object" and (.tasks | type) == "array" and ([.tasks[] | select(.taskId == "@sim/app#build")] | length == 1 and .[0].cache.status == "HIT")' "$RUNNER_TEMP/e2e-build-plan.json" > /dev/null; then
echo 'restore=false' >> "$GITHUB_OUTPUT"
else
echo 'restore=true' >> "$GITHUB_OUTPUT"
fi

- name: Mount incremental compiler state
id: compiler-cache
if: steps.compiler-cache-plan.outputs.restore == 'true'
uses: ./.github/actions/cache
with:
provider: ${{ vars.CI_PROVIDER }}
key: ${{ github.repository }}-next-build-e2e-v1-${{ runner.os }}-${{ runner.arch }}-${{ github.event_name }}-${{ steps.compiler-cache-plan.outputs.toolchain }}-${{ hashFiles('bun.lock', 'package.json', 'patches/**', 'apps/sim/package.json', 'apps/sim/next.config.ts', 'apps/sim/lib/core/config/env.ts', '.github/workflows/checks.yml', '.github/actions/setup/action.yml') }}
path: ./apps/sim/.next/cache

- name: Bound restored compiler state
if: steps.compiler-cache.outcome == 'success'
working-directory: apps/sim
run: node ../../scripts/prune-turbopack-cache.mts --build

- name: Build application
env: *e2e-build-env
run: bunx turbo run build --filter=@sim/app

- name: Bound compiler state before persistence
if: always() && !cancelled() && steps.compiler-cache.outcome == 'success'
working-directory: apps/sim
run: node ../../scripts/prune-turbopack-cache.mts --build

# Everything `next start` reads, including the spreadsheet parse worker it spawns (named, so a
# missing bundle fails packaging as it fails docker/app.Dockerfile); node_modules and the source
# tree come from e2e-production's own checkout and setup, at the same workspace path and
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docs-negotiation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -361,7 +361,7 @@ jobs:
run: tar -I zstd -xf docs-build.tar.zst -C apps/docs
- name: Set up Java for copied request examples
if: matrix.group == 'http'
uses: actions/setup-java@77dd5b2ce225ea11469928ac317850245a74ae78 # v6.2.0
uses: actions/setup-java@d1374f8ac74474200d16ef8ecee22c7ae7a51880 # v6.3.0
with:
distribution: temurin
java-version: '17'
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/helm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
steps:
- name: Checkout code (mirror)
if: github.event_name == 'pull_request' && !github.event.pull_request.head.repo.fork
uses: useblacksmith/checkout@2b9fe87d796feea153b455ec2f3125b471a7b0e7 # v1.8.2
uses: useblacksmith/checkout@75c49b6ed30ce59336e9252425796cfcfac155f8 # v1.8.3
with: &chart-checkout-options
# Chart version checks need full history to compare against the PR base.
fetch-depth: 0
Expand All @@ -52,7 +52,7 @@ jobs:
with: *chart-checkout-options

- name: Set up Helm
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
uses: azure/setup-helm@2f5e7bc67f1f9972ab16657bb795f36d8f04de0e # v5.1.0
with:
version: v3.16.4

Expand Down Expand Up @@ -171,7 +171,7 @@ jobs:
persist-credentials: false

- name: Set up Helm
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
uses: azure/setup-helm@2f5e7bc67f1f9972ab16657bb795f36d8f04de0e # v5.1.0
with:
version: v3.16.4

Expand Down Expand Up @@ -275,7 +275,7 @@ jobs:
persist-credentials: false

- name: Set up Helm
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
uses: azure/setup-helm@2f5e7bc67f1f9972ab16657bb795f36d8f04de0e # v5.1.0
with:
version: v3.16.4

Expand Down
1 change: 1 addition & 0 deletions apps/desktop/src/preload/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ const api: SimDesktopApi = {
...(process.platform === 'darwin'
? {
computerUse: {
stableElementIds: true,
getActivity: () => ipcRenderer.invoke('computer-use:get-activity'),
getStatus: () => ipcRenderer.invoke('computer-use:status'),
setEnabled: (enabled: boolean) => ipcRenderer.invoke('computer-use:set-enabled', enabled),
Expand Down
16 changes: 13 additions & 3 deletions apps/docs/content/docs/search/gitlab.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,17 @@ Sim checks returned content against these restrictions independently of the quer

## Add a group

Choose **Group** in the source's **Scope** field and enter the numeric group ID or full group path. Enable **Include subgroups** only if descendant projects should be included. Sim supports groups with at most 1,000 projects visible to the source token; larger groups need narrower sources. Projects shared from outside the group are excluded.
Choose **Group** in the source's **Scope** field and enter the numeric group ID or full group path. Enable **Include subgroups** only if descendant projects should be included. Sim supports up to **50,000 projects** visible to the source token, subject to the CSV limits below and a 16 MiB limit for saved project IDs and paths. Projects shared from outside the group are excluded.

Discovery reads 100 projects per page and must finish within five minutes. GitLab's configured offset and rate limits still apply; choose narrower groups if the instance cannot return the complete listing within these limits. Sim also handles listings without total-count headers, which GitLab can omit above 10,000 results.

The same permission strategies apply. Group membership alone never grants access to every project: administrator mode checks each project's current permissions, and non-admin mode requires a CSV grant for each project and reader. The source token must be able to read every project you want to search.

In CSV mode, each successful upload rediscovers the group's projects and replaces all grants atomically. Newly discovered projects become searchable only for users explicitly listed in the uploaded CSV. Updating and saving source settings also refreshes discovery. A failed discovery or stale revision preserves the previous configuration and grants. If a group configured by path is renamed, save the new path and an updated project-permissions CSV together in source settings. Numeric group IDs remain valid across renames.

Search visits project pages within a bounded request budget. Continuation cursors cover remaining projects and results; targeting a project narrows the request. Renamed or moved projects are blocked until a successful refresh; projects outside the configured group remain excluded. Group setup uses the existing project search APIs, so it does not require group-wide advanced search.
In CSV mode, search uses GitLab's group search API and checks each result against the saved projects and reader grants. Group code search requires Advanced Search or Exact Code Search; group wiki search requires Advanced Search. Unsupported scopes fall back to project search. A configured branch/tag, date filters, or administrator-token mode also use project search where needed. Each request visits a bounded number of pages; continuation cursors cover the remainder. Targeting a project narrows the request.

Group search stops at 10,000 candidates per content type to respect GitLab's Advanced Search window. Sim marks this as partial and asks you to narrow the query or target a project. This candidate limit applies before permission filtering. Renamed or moved projects are blocked until a successful refresh; projects outside the configured group remain excluded. See GitLab's [search capabilities and pagination limits](https://docs.gitlab.com/api/search/).

To expand an existing deployment, create a **separate group source** after the application deployment finishes. Keep the working project source and its CI variables unchanged while validating the group. Give the group job its new `SIM_SOURCE_ID`, then pause the old source after confirming the wider setup. Regular knowledge-base ingestion remains project-scoped.

Expand All @@ -78,7 +82,7 @@ engineering/platform,456
Use the full namespace path even if **Project** contains a numeric ID. Include inherited members explicitly. A grant without an email mapping grants nothing; rows for other projects cannot expand this source.

- Each file has exactly two columns in the shown order and at least one data row. Template headers or headerless data are accepted.
- Use UTF-8 CSV, at most 4 MiB and 100,000 data rows per file. The full request must fit 10 MiB.
- Use UTF-8 CSV, at most **8 MiB and 500,000 data rows per file**. The complete JSON upload, including escaped CSV content, must fit **10 MiB**. These are independent limits; split large deployments into narrower sources when either is reached.
- Identical rows are deduplicated. Conflicting user-ID/email mappings and malformed rows are rejected.

In this example, Alice can search `engineering/platform` because the files link GitLab user `123` to both that project and `alice@example.com`. Sim checks that Alice is a current organization member and that her verified Sim email uniquely matches the grant. It checks again when she opens a result. GitLab receives the request under the source's read-only service account; Alice does not sign in to GitLab or authorize Sim individually.
Expand Down Expand Up @@ -120,6 +124,8 @@ The export includes active, unlocked human members with Reporter access or highe

Requests run serially, paced at 60 per minute by default. Set `--requests-per-minute` to change this export-job ceiling; it does not configure live Search traffic. Each run reads each project's identity and membership pages, plus one profile request per distinct candidate reader. Group exports also read the group identity and project listing. The request ceiling applies across the entire export. API errors, hidden emails, identity conflicts, inconsistent pagination totals (when GitLab provides them), and oversized exports stop the job without publishing either file. Run exports after bulk membership updates finish; paginated API reads are not a point-in-time snapshot.

Large exports can take hours. For example, 50,000 projects require at least 100,000 project and membership requests, before listing and profile requests: over 27 hours at the default ceiling. Set the CI job timeout and schedule accordingly, or use smaller groups. Agree any higher request rate with your GitLab administrator. CSV files are written incrementally and published only when the complete export succeeds.

## Access checks

The source token retrieves content. Administrator mode proves the reader's current GitLab identity from verified Sim email, then checks project/group membership and policy during the request. Unsupported restrictions fail closed.
Expand Down Expand Up @@ -169,6 +175,10 @@ sync_gitlab_permissions:
'{expectedRevision: $rev, permissions: {provider: "gitlab",
userMapping: {filename: "users.csv", content: $users},
projectPermissions: {filename: "project-permissions.csv", content: $grants}}}' > "$export_root/update.json"
if [ "$(wc -c < "$export_root/update.json")" -gt 10485760 ]; then
echo "Permission upload exceeds 10 MiB; use a narrower source." >&2
exit 1
fi
curl --fail --silent --show-error --request PUT -H "$auth" \
-H 'Content-Type: application/json' --data-binary "@$export_root/update.json" "$url"
```
Expand Down
Loading
Loading