diff --git a/apps/docs/content/docs/search/gitlab.mdx b/apps/docs/content/docs/search/gitlab.mdx index 0556c5cb8c5..d6a543c57a8 100644 --- a/apps/docs/content/docs/search/gitlab.mdx +++ b/apps/docs/content/docs/search/gitlab.mdx @@ -45,13 +45,17 @@ Sim checks returned content against these restrictions independently of the quer ## Add a group -Choose **Group** in the source's **Scope** field and enter the numeric group ID or full group path. Enable **Include subgroups** only if descendant projects should be included. Sim supports groups with at most 1,000 projects visible to the source token; larger groups need narrower sources. Projects shared from outside the group are excluded. +Choose **Group** in the source's **Scope** field and enter the numeric group ID or full group path. Enable **Include subgroups** only if descendant projects should be included. Sim supports up to **50,000 projects** visible to the source token, subject to the CSV limits below and a 16 MiB limit for saved project IDs and paths. Projects shared from outside the group are excluded. + +Discovery reads 100 projects per page and must finish within five minutes. GitLab's configured offset and rate limits still apply; choose narrower groups if the instance cannot return the complete listing within these limits. Sim also handles listings without total-count headers, which GitLab can omit above 10,000 results. The same permission strategies apply. Group membership alone never grants access to every project: administrator mode checks each project's current permissions, and non-admin mode requires a CSV grant for each project and reader. The source token must be able to read every project you want to search. In CSV mode, each successful upload rediscovers the group's projects and replaces all grants atomically. Newly discovered projects become searchable only for users explicitly listed in the uploaded CSV. Updating and saving source settings also refreshes discovery. A failed discovery or stale revision preserves the previous configuration and grants. If a group configured by path is renamed, save the new path and an updated project-permissions CSV together in source settings. Numeric group IDs remain valid across renames. -Search visits project pages within a bounded request budget. Continuation cursors cover remaining projects and results; targeting a project narrows the request. Renamed or moved projects are blocked until a successful refresh; projects outside the configured group remain excluded. Group setup uses the existing project search APIs, so it does not require group-wide advanced search. +In CSV mode, search uses GitLab's group search API and checks each result against the saved projects and reader grants. Group code search requires Advanced Search or Exact Code Search; group wiki search requires Advanced Search. Unsupported scopes fall back to project search. A configured branch/tag, date filters, or administrator-token mode also use project search where needed. Each request visits a bounded number of pages; continuation cursors cover the remainder. Targeting a project narrows the request. + +Group search stops at 10,000 candidates per content type to respect GitLab's Advanced Search window. Sim marks this as partial and asks you to narrow the query or target a project. This candidate limit applies before permission filtering. Renamed or moved projects are blocked until a successful refresh; projects outside the configured group remain excluded. See GitLab's [search capabilities and pagination limits](https://docs.gitlab.com/api/search/). To expand an existing deployment, create a **separate group source** after the application deployment finishes. Keep the working project source and its CI variables unchanged while validating the group. Give the group job its new `SIM_SOURCE_ID`, then pause the old source after confirming the wider setup. Regular knowledge-base ingestion remains project-scoped. @@ -78,7 +82,7 @@ engineering/platform,456 Use the full namespace path even if **Project** contains a numeric ID. Include inherited members explicitly. A grant without an email mapping grants nothing; rows for other projects cannot expand this source. - Each file has exactly two columns in the shown order and at least one data row. Template headers or headerless data are accepted. -- Use UTF-8 CSV, at most 4 MiB and 100,000 data rows per file. The full request must fit 10 MiB. +- Use UTF-8 CSV, at most **8 MiB and 500,000 data rows per file**. The complete JSON upload, including escaped CSV content, must fit **10 MiB**. These are independent limits; split large deployments into narrower sources when either is reached. - Identical rows are deduplicated. Conflicting user-ID/email mappings and malformed rows are rejected. In this example, Alice can search `engineering/platform` because the files link GitLab user `123` to both that project and `alice@example.com`. Sim checks that Alice is a current organization member and that her verified Sim email uniquely matches the grant. It checks again when she opens a result. GitLab receives the request under the source's read-only service account; Alice does not sign in to GitLab or authorize Sim individually. @@ -120,6 +124,8 @@ The export includes active, unlocked human members with Reporter access or highe Requests run serially, paced at 60 per minute by default. Set `--requests-per-minute` to change this export-job ceiling; it does not configure live Search traffic. Each run reads each project's identity and membership pages, plus one profile request per distinct candidate reader. Group exports also read the group identity and project listing. The request ceiling applies across the entire export. API errors, hidden emails, identity conflicts, inconsistent pagination totals (when GitLab provides them), and oversized exports stop the job without publishing either file. Run exports after bulk membership updates finish; paginated API reads are not a point-in-time snapshot. +Large exports can take hours. For example, 50,000 projects require at least 100,000 project and membership requests, before listing and profile requests: over 27 hours at the default ceiling. Set the CI job timeout and schedule accordingly, or use smaller groups. Agree any higher request rate with your GitLab administrator. CSV files are written incrementally and published only when the complete export succeeds. + ## Access checks The source token retrieves content. Administrator mode proves the reader's current GitLab identity from verified Sim email, then checks project/group membership and policy during the request. Unsupported restrictions fail closed. @@ -169,6 +175,10 @@ sync_gitlab_permissions: '{expectedRevision: $rev, permissions: {provider: "gitlab", userMapping: {filename: "users.csv", content: $users}, projectPermissions: {filename: "project-permissions.csv", content: $grants}}}' > "$export_root/update.json" + if [ "$(wc -c < "$export_root/update.json")" -gt 10485760 ]; then + echo "Permission upload exceeds 10 MiB; use a narrower source." >&2 + exit 1 + fi curl --fail --silent --show-error --request PUT -H "$auth" \ -H 'Content-Type: application/json' --data-binary "@$export_root/update.json" "$url" ``` diff --git a/apps/docs/openapi-v2-resources.json b/apps/docs/openapi-v2-resources.json index 5350eac6cd8..7a563ca1c80 100644 --- a/apps/docs/openapi-v2-resources.json +++ b/apps/docs/openapi-v2-resources.json @@ -10233,7 +10233,7 @@ "put": { "operationId": "replaceSearchSourcePermissions", "summary": "Replace Search source permissions", - "description": "Replace the complete permission grants for one Search source. Requires organization Admin access. Supply the current revision and the grants for the configured provider. A stale revision returns 409. GitLab CSV mode accepts both files together, up to 4 MiB and 100,000 data rows each. Workspace API keys return `403`; use a personal API key or scoped OAuth token.\n\nOAuth scope: `api:write`.", + "description": "Replace the complete permission grants for one Search source. Requires organization Admin access. Supply the current revision and the grants for the configured provider. A stale revision returns 409. GitLab CSV mode accepts both files together, up to 8 MiB and 500,000 data rows each, within a 10 MiB limit for the complete JSON-encoded request. Workspace API keys return `403`; use a personal API key or scoped OAuth token.\n\nOAuth scope: `api:write`.", "x-sim-operation": "knowledge.search.source_permissions.replace", "x-oauth-scope": "api:write", "tags": ["Organizations"], @@ -24566,8 +24566,7 @@ "content": { "type": "string", "minLength": 1, - "maxLength": 4194304, - "description": "UTF-8 CSV contents, up to 4 MiB. Both files travel in one request body limited to 10 MiB after JSON encoding." + "description": "UTF-8 CSV contents, up to 8 MiB. Both files travel in one request body limited to 10 MiB after JSON encoding." } }, "required": ["filename", "content"], @@ -24586,8 +24585,7 @@ "content": { "type": "string", "minLength": 1, - "maxLength": 4194304, - "description": "UTF-8 CSV contents, up to 4 MiB. Both files travel in one request body limited to 10 MiB after JSON encoding." + "description": "UTF-8 CSV contents, up to 8 MiB. Both files travel in one request body limited to 10 MiB after JSON encoding." } }, "required": ["filename", "content"], diff --git a/apps/docs/public/static/examples/gitlab/export-permissions.py b/apps/docs/public/static/examples/gitlab/export-permissions.py index 5f37b73b512..0fcdad3a0a5 100644 --- a/apps/docs/public/static/examples/gitlab/export-permissions.py +++ b/apps/docs/public/static/examples/gitlab/export-permissions.py @@ -18,9 +18,9 @@ from urllib.request import HTTPRedirectHandler, HTTPSHandler, Request, build_opener -MAX_BYTES = 4 * 1024 * 1024 -MAX_ROWS = 100_000 -MAX_PROJECTS = 1000 +MAX_BYTES = 8 * 1024 * 1024 +MAX_ROWS = 500_000 +MAX_PROJECTS = 50_000 EMAIL = re.compile( r"[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@" r"[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?" @@ -134,7 +134,7 @@ def export(client, project, profiles): today = datetime.now(timezone.utc).date() page = 1 expected_total = None - while page <= 1000: + while page <= MAX_ROWS // 100 + 1: members, headers = client.get(f"projects/{project_id}/members/all", {"per_page": 100, "page": page}) if not isinstance(members, list) or len(members) > 100: @@ -142,7 +142,7 @@ def export(client, project, profiles): total = headers.get("X-Total") if total is not None: if not re.fullmatch(r"[0-9]{1,6}", total) or int(total) > MAX_ROWS: - raise ExportError("GitLab membership total is invalid or exceeds the 100,000-row limit.") + raise ExportError("GitLab membership total is invalid or exceeds the 500,000-row limit.") if expected_total is not None and int(total) != expected_total: raise ExportError("GitLab membership total changed during pagination. Rerun the export.") expected_total = int(total) @@ -154,7 +154,7 @@ def export(client, project, profiles): raise ExportError("GitLab membership pagination repeated a user. Rerun the export.") seen_members.add(user_id) if len(seen_members) > MAX_ROWS: - raise ExportError("Project membership exceeds the 100,000-row limit.") + raise ExportError("Project membership exceeds the 500,000-row limit.") role = member.get("access_level") if type(role) is not int: raise ExportError("GitLab returned a membership without an access level.") @@ -172,7 +172,7 @@ def export(client, project, profiles): profile, _ = client.get(f"users/{user_id}") profiles[user_id] = verified_email(profile, user_id) if len(profiles) > MAX_ROWS: - raise ExportError("Group identities exceed the 100,000-row limit.") + raise ExportError("Group identities exceed the 500,000-row limit.") email = profiles[user_id] if email is None: continue @@ -189,7 +189,7 @@ def export(client, project, profiles): raise ExportError("GitLab returned invalid membership pagination.") page = int(next_page) else: - raise ExportError("GitLab membership pagination exceeds 1,000 pages.") + raise ExportError("GitLab membership pagination exceeds its limit.") if expected_total is not None and len(seen_members) != expected_total: raise ExportError("Exported membership count does not match GitLab's total. Rerun the export.") return project_path, users @@ -201,21 +201,22 @@ def group_projects(client, group, include_subgroups): raise ExportError("GitLab returned an invalid group.") group_id = identifier(record.get("id")) path = record.get("full_path") - if not isinstance(path, str) or not path: + if not isinstance(path, str) or not path or len(path) > 1024: raise ExportError("GitLab did not return the group path.") projects, seen_ids, seen_paths = [], set(), set() expected_total = None + manifest_bytes = 2 for page in range(1, MAX_PROJECTS // 100 + 2): rows, headers = client.get(f"groups/{group_id}/projects", { "with_shared": "false", "include_subgroups": str(include_subgroups).lower(), - "per_page": 100, "page": page, "order_by": "id", "sort": "asc", + "per_page": 100, "page": page, "order_by": "id", "sort": "asc", "simple": "true", }) if not isinstance(rows, list) or len(rows) > 100: raise ExportError("GitLab returned an invalid group project page.") total = headers.get("X-Total") if total is not None: if not re.fullmatch(r"[0-9]{1,6}", total) or int(total) > MAX_PROJECTS: - raise ExportError("GitLab group exceeds the 1,000-project limit.") + raise ExportError("GitLab group total is invalid or exceeds the 50,000-project limit.") if expected_total is not None and expected_total != int(total): raise ExportError("GitLab group project total changed during export.") expected_total = int(total) @@ -224,6 +225,8 @@ def group_projects(client, group, include_subgroups): raise ExportError("GitLab returned an invalid group project.") project_id = identifier(row.get("id")) project_path = row.get("path_with_namespace") + if not isinstance(project_path, str) or len(project_path) > 1024: + raise ExportError("GitLab returned an invalid project path.") namespace = project_path.rpartition("/")[0] if isinstance(project_path, str) else "" if not (namespace == path or (include_subgroups and namespace.startswith(path + "/"))): raise ExportError("GitLab returned a project outside the configured group.") @@ -233,7 +236,11 @@ def group_projects(client, group, include_subgroups): seen_paths.add(project_path) projects.append((project_id, project_path)) if len(projects) > MAX_PROJECTS: - raise ExportError("GitLab group exceeds the 1,000-project limit.") + raise ExportError("GitLab group exceeds the 50,000-project limit.") + manifest_bytes += len(json.dumps({"id": project_id, "path": project_path}, + ensure_ascii=False, separators=(",", ":")).encode()) + 1 + if manifest_bytes > 16 * 1024 * 1024: + raise ExportError("GitLab group project listing exceeds 16 MiB. Choose a narrower group.") next_page = headers.get("X-Next-Page") if next_page is None: next_page = str(page + 1) if len(rows) == 100 else "" @@ -251,27 +258,39 @@ def group_projects(client, group, include_subgroups): def write_export(output, grants, users): - files = {} - ordered = sorted(users) - for name, header, rows in [ - ("users.csv", ["user_id", "email"], [(user_id, users[user_id]) for user_id in ordered]), - ("project-permissions.csv", ["project_path", "user_id"], - sorted(grants)), - ]: - buffer = io.StringIO(newline="") - writer = csv.writer(buffer, lineterminator="\n") - writer.writerow(header) - writer.writerows(rows) - files[name] = buffer.getvalue().encode("utf-8") - if len(files[name]) > MAX_BYTES: - raise ExportError(f"{name} exceeds Sim's 4 MiB CSV limit.") + def user_rows(): + for user_id in sorted(users): + yield user_id, users[user_id] + output.parent.mkdir(parents=True, exist_ok=True) # Publish the pair together, so an interrupted export cannot mix new and old CSVs. with tempfile.TemporaryDirectory(dir=output.parent, prefix=".gitlab-export-") as temporary: - for name, content in files.items(): + for name, header, rows in [ + ("project-permissions.csv", ["project_path", "user_id"], grants), + ("users.csv", ["user_id", "email"], user_rows()), + ]: path = Path(temporary) / name - path.write_bytes(content) + buffer = io.StringIO(newline="") + writer = csv.writer(buffer, lineterminator="\n") + writer.writerow(header) + size = 0 + with path.open("wb") as handle: + handle.write(buffer.getvalue().encode("utf-8")) + size = handle.tell() + for count, row in enumerate(rows, 1): + if count > MAX_ROWS: + raise ExportError(f"{name} exceeds Sim's 500,000-row limit.") + buffer.seek(0) + buffer.truncate() + writer.writerow(row) + content = buffer.getvalue().encode("utf-8") + size += len(content) + if size > MAX_BYTES: + raise ExportError(f"{name} exceeds Sim's 8 MiB CSV limit.") + handle.write(content) path.chmod(0o600) + if not users: + raise ExportError("No eligible readers. Pause search in Sim; empty CSVs cannot revoke all grants.") os.rename(temporary, output) @@ -297,22 +316,22 @@ def main(): raise ExportError("Output directory already exists. Choose a new directory for a fresh export.") client = GitLab(args.gitlab_url, token, args.requests_per_minute) projects = group_projects(client, args.group, args.include_subgroups) if args.group else [(args.project, None)] - profiles, users, emails, grants = {}, {}, {}, [] - for project_id, expected_path in projects: - project_path, readers = export(client, str(project_id), profiles) - if expected_path is not None and expected_path != project_path: - raise ExportError("GitLab project identity changed during export. Rerun the export.") - for user_id, email in readers.items(): - if email in emails and emails[email] != user_id: - raise ExportError("Multiple GitLab users share an email. Resolve the identity conflict first.") - emails[email] = user_id - users[user_id] = email - grants.append((project_path, user_id)) - if len(grants) > MAX_ROWS: - raise ExportError("Project permissions exceed the 100,000-row limit.") - if not users: - raise ExportError("No eligible readers. Pause search in Sim; empty CSVs cannot revoke all grants.") - write_export(output, grants, users) + profiles, users, emails = {}, {}, {} + + def grants(): + for project_id, expected_path in projects: + project_path, readers = export(client, str(project_id), profiles) + if expected_path is not None and expected_path != project_path: + raise ExportError("GitLab project identity changed during export. Rerun the export.") + for user_id in sorted(readers): + email = readers[user_id] + if email in emails and emails[email] != user_id: + raise ExportError("Multiple GitLab users share an email. Resolve the identity conflict first.") + emails[email] = user_id + users[user_id] = email + yield project_path, user_id + + write_export(output, grants(), users) print(f"Exported {len(users)} readers to {output}/users.csv and {output}/project-permissions.csv") diff --git a/apps/docs/scripts/fixtures/content-baseline.json b/apps/docs/scripts/fixtures/content-baseline.json index 94fc142889d..ff931c8453c 100644 --- a/apps/docs/scripts/fixtures/content-baseline.json +++ b/apps/docs/scripts/fixtures/content-baseline.json @@ -177236,8 +177236,8 @@ { "lang": "yaml", "meta": "title=\".gitlab-ci.yml\"", - "sha256": "29261e0a66aac3cf52e7b9164aaa44f235765bcfd2cae4e68d744d31b29b57bf", - "bytes": 1441, + "sha256": "95f1b2ec16f400d5e0ebe5cd67ae2b9f6476f2c86696a54f9a983b2b8d02a5c7", + "bytes": 1613, "line": 129 }, { diff --git a/apps/docs/scripts/test-gitlab-export-e2e.py b/apps/docs/scripts/test-gitlab-export-e2e.py index 55a85aa4aca..8c4765147d8 100644 --- a/apps/docs/scripts/test-gitlab-export-e2e.py +++ b/apps/docs/scripts/test-gitlab-export-e2e.py @@ -6,6 +6,7 @@ import json import os from pathlib import Path +import runpy import ssl import subprocess import sys @@ -50,6 +51,14 @@ def do_GET(self): query = parse_qs(parsed.query) assert query.get("with_shared") == ["false"] assert query.get("include_subgroups") == ["true" if scenario == "group-subgroups" else "false"] + if scenario in ("group-large", "group-too-large"): + if scenario == "group-too-large": + self.reply(200, [], {"X-Total": "50001"}) + else: + page = int(query.get("page", ["1"])[0]) + self.reply(200, [{"id": index, "path_with_namespace": f"engineering/project-{index}"} + for index in range(10000 + (page - 1) * 100, min(11200, 10000 + page * 100))]) + return projects = [{"id": 91, "path_with_namespace": "engineering/platform"}] if query.get("page", ["1"]) == ["1"]: self.reply(200, projects, {"X-Next-Page": "2", "X-Total": "2"}) @@ -75,6 +84,13 @@ def do_GET(self): members = [] self.reply(200, members, {"X-Next-Page": "", "X-Total": str(len(members))}) return + if scenario == "group-large" and endpoint.startswith("projects/"): + project_id = int(endpoint.split("/")[1]) + if endpoint.endswith("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/members/all"): + self.reply(200, [member(1)] if project_id == 11199 else []) + else: + self.reply(200, {"id": project_id, "path_with_namespace": f"engineering/project-{project_id}"}) + return if endpoint in ("projects/engineering%2Fplatform", "projects/91"): if scenario == "redirect": self.reply(302, {}, {"Location": "/redirect/api/v4/trap"}) @@ -274,6 +290,56 @@ def run(): assert TOKEN not in result.stdout + result.stderr assert len(GitLabFixture.calls) == calls, "Sent a token for an invalid group identifier" report["checks"].append({"name": "invalid-group-identifiers", "passed": True}) + for scenario in ("group-large", "group-too-large"): + output = root / scenario + result = subprocess.run([ + sys.executable, str(SCRIPT), "--gitlab-url", + f"https://127.0.0.1:{server.server_port}/{scenario}", "--group", "engineering", + "--output-dir", str(output), "--requests-per-minute", "600000", + ], capture_output=True, text=True, timeout=60, + env={**os.environ, "GITLAB_EXPORT_TOKEN": TOKEN, "SSL_CERT_FILE": str(cert)}) + if scenario == "group-too-large": + assert result.returncode != 0 and not output.exists(), result.stderr + assert "50,000-project limit" in result.stderr, result.stderr + else: + assert result.returncode == 0, result.stderr + with (output / "project-permissions.csv").open() as handle: + assert list(csv.reader(handle)) == [ + ["project_path", "user_id"], ["engineering/project-11199", "1"]] + with (output / "users.csv").open() as handle: + assert list(csv.reader(handle)) == [["user_id", "email"], ["1", "alice@example.com"]] + report["checks"].append({"name": scenario, "passed": True}) + exporter = runpy.run_path(str(SCRIPT)) + output = root / "large-csv" + exporter["write_export"](output, + ((f"engineering/project-{index:06d}-permissions", 1) for index in range(150000)), + {1: "alice@example.com"}) + assert (output / "project-permissions.csv").stat().st_size > 4 * 1024 * 1024 + with (output / "project-permissions.csv").open() as handle: + rows = csv.reader(handle) + assert next(rows) == ["project_path", "user_id"] + assert sum(1 for _ in rows) == 150000 + report["checks"].append({"name": "large-csv", "passed": True}) + for scenario, grants in ( + ("csv-row-bound", (("a/b", 1) for _ in range(500001))), + ("csv-byte-bound", (("a/" + "b" * 1000, 1) for _ in range(9000))), + ("csv-partial-failure", None), + ): + if scenario == "csv-partial-failure": + def interrupted_grants(): + yield ("a/b", 1) + raise exporter["ExportError"]("Provider failed after the first grant") + grants = interrupted_grants() + output = root / scenario + try: + exporter["write_export"](output, grants, {1: "alice@example.com"}) + raise AssertionError(f"{scenario}: export unexpectedly succeeded") + except exporter["ExportError"] as error: + expected = {"csv-row-bound": "500,000-row limit", "csv-byte-bound": "8 MiB CSV limit", + "csv-partial-failure": "Provider failed after the first grant"}[scenario] + assert expected in str(error), str(error) + assert not output.exists(), "Published an incomplete CSV pair" + report["checks"].append({"name": scenario, "passed": True}) assert not any(endpoint == "trap" for _, endpoint, _, _ in GitLabFixture.calls) report["passed"] = True finally: diff --git a/apps/sim/connectors/gitlab/group.ts b/apps/sim/connectors/gitlab/group.ts index fe68a81f0ca..760500d2b5b 100644 --- a/apps/sim/connectors/gitlab/group.ts +++ b/apps/sim/connectors/gitlab/group.ts @@ -1,17 +1,17 @@ import { z } from 'zod' import { encodeGitLabResourceId, normalizeGitLabHost } from '@/tools/gitlab/utils' -export const GITLAB_GROUP_MAX_PROJECTS = 1000 +export const GITLAB_GROUP_MAX_PROJECTS = 50_000 export const gitLabGroupScopeSchema = z.object({ host: z.string().min(1), group: z.object({ id: z.number().int().positive(), - path: z.string().min(1), + path: z.string().min(1).max(1024), includeSubgroups: z.boolean(), }), projects: z - .array(z.object({ id: z.number().int().positive(), path: z.string().min(1) })) + .array(z.object({ id: z.number().int().positive(), path: z.string().min(1).max(1024) })) .min(1) .max(GITLAB_GROUP_MAX_PROJECTS), }) diff --git a/apps/sim/connectors/gitlab/meta.ts b/apps/sim/connectors/gitlab/meta.ts index a124fe79411..6933fab0522 100644 --- a/apps/sim/connectors/gitlab/meta.ts +++ b/apps/sim/connectors/gitlab/meta.ts @@ -58,7 +58,8 @@ export const gitlabConnectorMeta: ConnectorMeta = { required: true, condition: { field: 'scope', value: 'group' }, placeholder: 'group/subgroup or numeric ID', - description: 'Up to 1,000 projects. Shared projects are excluded.', + description: + 'Up to 50,000 projects, subject to CSV size limits. Shared projects are excluded.', }, { id: 'includeSubgroups', diff --git a/apps/sim/connectors/gitlab/permission-config/fields.tsx b/apps/sim/connectors/gitlab/permission-config/fields.tsx index 3c4c17f9336..b18ee20964e 100644 --- a/apps/sim/connectors/gitlab/permission-config/fields.tsx +++ b/apps/sim/connectors/gitlab/permission-config/fields.tsx @@ -88,9 +88,12 @@ function GitLabCsvFile({ kind, form, disabled }: GitLabCsvFileProps) { disabled={disabled} loading={selected?.loading} label={selected?.loading ? 'Validating…' : (filename ?? 'Drop CSV or click to upload')} - error={selected?.error} + error={selected?.error ?? (kind === 'projectPermissions' ? form.uploadError : undefined)} description={ - filename && count !== undefined && !selected?.loading + filename && + count !== undefined && + !selected?.loading && + !(selected?.upload && form.uploadError) ? `${count.toLocaleString()} ${count === 1 ? 'row' : 'rows'} · ${selected?.upload ? 'Ready to save' : 'Saved'}` : undefined } diff --git a/apps/sim/connectors/gitlab/permission-config/parser.test.ts b/apps/sim/connectors/gitlab/permission-config/parser.test.ts index dec2a016302..0386fc79c8b 100644 --- a/apps/sim/connectors/gitlab/permission-config/parser.test.ts +++ b/apps/sim/connectors/gitlab/permission-config/parser.test.ts @@ -61,18 +61,18 @@ describe('GitLab CSV permission uploads', () => { { filename: 'users.csv', content: '😀'.repeat(GITLAB_CSV_MAX_BYTES / 4 + 1) }, 'userMapping' ) - ).toThrow('4 MiB') + ).toThrow('8 MiB') }) - it('accepts 100,000 rows and rejects the next row', () => { - const content = Array.from({ length: 100_000 }, (_, i) => `${i + 1},u${i}@example.com`).join( - '\n' - ) - expect(parseGitLabCsv({ filename: 'users.csv', content }, 'userMapping')).toHaveLength(100_000) + it('accepts 500,000 rows and rejects the next row', () => { + const content = Array.from({ length: 500_000 }, (_, i) => `a/b,${i + 1}`).join('\n') + expect( + parseGitLabCsv({ filename: 'projects.csv', content }, 'projectPermissions') + ).toHaveLength(500_000) expect(() => parseGitLabCsv( - { filename: 'users.csv', content: `${content}\n100001,last@example.com` }, - 'userMapping' + { filename: 'projects.csv', content: `${content}\na/b,500001` }, + 'projectPermissions' ) - ).toThrow('100,000') + ).toThrow('500,000') }) }) diff --git a/apps/sim/connectors/gitlab/permission-config/parser.ts b/apps/sim/connectors/gitlab/permission-config/parser.ts index e5d13e02529..5de1b8c3b6a 100644 --- a/apps/sim/connectors/gitlab/permission-config/parser.ts +++ b/apps/sim/connectors/gitlab/permission-config/parser.ts @@ -37,7 +37,7 @@ export function parseGitLabCsv(upload: GitLabCsvUpload, kind: GitLabCsvKind): [s reject('Use a filename of 1–255 characters without path separators.') } if (new TextEncoder().encode(upload.content).byteLength > GITLAB_CSV_MAX_BYTES) { - reject('The file must be 4 MiB or smaller.') + reject('The file must be 8 MiB or smaller.') } const rows: [string, string][] = [] const seen = new Set() @@ -60,7 +60,7 @@ export function parseGitLabCsv(upload: GitLabCsvUpload, kind: GitLabCsvKind): [s if (record[0] === HEADERS[kind][0] && record[1] === HEADERS[kind][1]) return null } recordCount++ - if (recordCount > GITLAB_CSV_MAX_ROWS) reject('The file must contain at most 100,000 rows.') + if (recordCount > GITLAB_CSV_MAX_ROWS) reject('The file must contain at most 500,000 rows.') const id = kind === 'userMapping' ? record[0] : record[1] if (!USER_ID.test(id) || !Number.isSafeInteger(Number(id))) { reject(`Row ending at line ${line} has an invalid user_id; use a positive integer.`) diff --git a/apps/sim/connectors/gitlab/permission-config/repository.ts b/apps/sim/connectors/gitlab/permission-config/repository.ts index 817d2e99999..7e1de7ced50 100644 --- a/apps/sim/connectors/gitlab/permission-config/repository.ts +++ b/apps/sim/connectors/gitlab/permission-config/repository.ts @@ -51,13 +51,17 @@ export function readGitLabPermissionMetadata(value: unknown) { const parsed = metadataSchema.safeParse(value) if (!parsed.success) return invalidSnapshot() const result = parsed.data + validateSummaries(result) + return result +} + +function validateSummaries(result: Omit) { if ( result.mode === 'csv' ? !result.userMapping || !result.projectPermissions : result.userMapping !== null || result.projectPermissions !== null ) return invalidSnapshot() - return result } /** Validate stored tuples without allocating another copy of a large CSV snapshot. */ @@ -230,9 +234,15 @@ export async function readGitLabPermissionSummaries( connectorIds: readonly string[] ): Promise> { const result = new Map() - for (const row of await readConnectorPermissionMetadata(connectorIds)) { - const { mode, userMapping, projectPermissions } = readGitLabPermissionMetadata(row.metadata) - result.set(row.connectorId, { mode, revision: row.revision, userMapping, projectPermissions }) + for (const row of await readConnectorPermissionMetadata(connectorIds, [ + 'mode', + 'userMapping', + 'projectPermissions', + ])) { + const parsed = summarySchema.safeParse(row.metadata) + if (!parsed.success) return invalidSnapshot() + validateSummaries(parsed.data) + result.set(row.connectorId, { ...parsed.data, revision: row.revision }) } return result } diff --git a/apps/sim/connectors/gitlab/permission-config/request-size.ts b/apps/sim/connectors/gitlab/permission-config/request-size.ts new file mode 100644 index 00000000000..0ab28ec8e64 --- /dev/null +++ b/apps/sim/connectors/gitlab/permission-config/request-size.ts @@ -0,0 +1,7 @@ +const MAX_UPLOAD_BODY_BYTES = 10 * 1024 * 1024 + +/** Check encoded CSV content and its request envelope against the proxy's JSON body ceiling. */ +export function getGitLabUploadSizeError(body: unknown): string | undefined { + if (new TextEncoder().encode(JSON.stringify(body)).byteLength > MAX_UPLOAD_BODY_BYTES) + return 'The complete upload must be 10 MiB or smaller. Use smaller CSV files or a narrower group.' +} diff --git a/apps/sim/connectors/gitlab/permission-config/types.ts b/apps/sim/connectors/gitlab/permission-config/types.ts index 0ae990937c7..34904d1bb2f 100644 --- a/apps/sim/connectors/gitlab/permission-config/types.ts +++ b/apps/sim/connectors/gitlab/permission-config/types.ts @@ -1,7 +1,7 @@ import { connectorPermissionGroupToken } from '@/lib/knowledge/connectors/permission-tokens' -export const GITLAB_CSV_MAX_BYTES = 4 * 1024 * 1024 -export const GITLAB_CSV_MAX_ROWS = 100_000 +export const GITLAB_CSV_MAX_BYTES = 8 * 1024 * 1024 +export const GITLAB_CSV_MAX_ROWS = 500_000 export const GITLAB_PERMISSION_MODES = ['administrator', 'csv'] as const export type GitLabPermissionMode = (typeof GITLAB_PERMISSION_MODES)[number] diff --git a/apps/sim/connectors/gitlab/permission-config/use-permission-form.test.ts b/apps/sim/connectors/gitlab/permission-config/use-permission-form.test.ts new file mode 100644 index 00000000000..3f0144f071c --- /dev/null +++ b/apps/sim/connectors/gitlab/permission-config/use-permission-form.test.ts @@ -0,0 +1,48 @@ +/** @vitest-environment jsdom */ +import { act, createElement } from 'react' +import { createRoot } from 'react-dom/client' +import { describe, expect, it, vi } from 'vitest' +import { useGitLabPermissionForm } from '@/connectors/gitlab/permission-config/use-permission-form' + +describe('GitLab permission file selection', () => { + it('blocks an oversized combined upload and recovers when a file is replaced', async () => { + vi.stubGlobal('IS_REACT_ACT_ENVIRONMENT', true) + const root = createRoot(document.createElement('div')) + let form: ReturnType | undefined + function Probe() { + form = useGitLabPermissionForm() + return null + } + const current = () => { + if (!form) throw new Error('Permission form was not mounted') + return form + } + act(() => root.render(createElement(Probe))) + const csv = (content: string, filename: string) => + Object.assign(new File([content], filename), { + arrayBuffer: async () => new TextEncoder().encode(content).buffer, + }) + try { + act(() => current().setMode('csv')) + const users = csv('1,a@example.test\n'.repeat(300_000), 'users.csv') + const grants = csv('group/p,1\n'.repeat(490_000), 'projects.csv') + expect(users.size + grants.size).toBeLessThan(10 * 1024 * 1024) + await act(async () => { + await current().selectFile('userMapping', users) + await current().selectFile('projectPermissions', grants) + }) + expect(current().files.userMapping?.error).toBeUndefined() + expect(current().files.projectPermissions?.error).toBeUndefined() + expect(current().complete).toBe(false) + await act(async () => { + await current().selectFile('projectPermissions', csv('group/project,1', 'projects.csv')) + }) + expect(current().complete).toBe(true) + act(() => current().reset()) + expect(current().files).toEqual({}) + } finally { + act(() => root.unmount()) + vi.unstubAllGlobals() + } + }) +}) diff --git a/apps/sim/connectors/gitlab/permission-config/use-permission-form.ts b/apps/sim/connectors/gitlab/permission-config/use-permission-form.ts index 60437b22259..2299af56735 100644 --- a/apps/sim/connectors/gitlab/permission-config/use-permission-form.ts +++ b/apps/sim/connectors/gitlab/permission-config/use-permission-form.ts @@ -7,6 +7,7 @@ import type { GitLabPermissionUploadInput, } from '@/lib/api/contracts/knowledge/gitlab-permissions' import { parseGitLabCsv } from '@/connectors/gitlab/permission-config/parser' +import { getGitLabUploadSizeError } from '@/connectors/gitlab/permission-config/request-size' import { GITLAB_CSV_MAX_BYTES, type GitLabCsvKind, @@ -55,12 +56,17 @@ export function useGitLabPermissionForm(initial?: GitLabPermissionData) { }), [files, mode, saved] ) + const uploadError = useMemo( + () => + input.mode === 'csv' ? getGitLabUploadSizeError({ permissionConfig: input }) : undefined, + [input] + ) const selectFile = useCallback(async (kind: GitLabCsvKind, file: File) => { const generation = ++generations.current[kind] setFiles((previous) => ({ ...previous, [kind]: { loading: true } })) let selected: SelectedFile try { - if (file.size > GITLAB_CSV_MAX_BYTES) throw new Error('The file must be 4 MiB or smaller.') + if (file.size > GITLAB_CSV_MAX_BYTES) throw new Error('The file must be 8 MiB or smaller.') const content = new TextDecoder('utf-8', { fatal: true }).decode(await file.arrayBuffer()) const upload = { filename: file.name, content } const rows = parseGitLabCsv(upload, kind) @@ -89,7 +95,8 @@ export function useGitLabPermissionForm(initial?: GitLabPermissionData) { apiKey, setApiKey, input, - complete, + complete: complete && !uploadError, + uploadError, dirty, selectFile, reset, diff --git a/apps/sim/connectors/gitlab/permissions.ts b/apps/sim/connectors/gitlab/permissions.ts index bb84aa26cd7..cfac3172edd 100644 --- a/apps/sim/connectors/gitlab/permissions.ts +++ b/apps/sim/connectors/gitlab/permissions.ts @@ -124,34 +124,54 @@ export async function discoverGitLabGroup( mode: 'csv' | 'administrator', signal?: AbortSignal ): Promise { - const { host, group } = await validateGitLabGroupToken(token, config, mode, signal) - const base = `https://${host}/api/v4` - const rows = await listAll( - `${base}/groups/${group.id}/projects?with_shared=false&include_subgroups=${group.includeSubgroups}&order_by=id&sort=asc`, - token, - signal, - GITLAB_GROUP_MAX_PROJECTS - ) - const projects: GitLabGroupScope['projects'] = [] - const ids = new Set() - const paths = new Set() - for (const row of rows) { - if ( - !Number.isSafeInteger(row.id) || - row.id <= 0 || - typeof row.path_with_namespace !== 'string' || - !gitLabProjectInGroup(row.path_with_namespace, group) || - ids.has(row.id) || - paths.has(row.path_with_namespace) - ) - throw new Error('GitLab returned an inconsistent group project listing. Retry the update.') - ids.add(row.id) - paths.add(row.path_with_namespace) - projects.push({ id: row.id, path: row.path_with_namespace }) + const deadline = AbortSignal.timeout(5 * 60 * 1000) + const discoverySignal = signal ? AbortSignal.any([signal, deadline]) : deadline + try { + const { host, group } = await validateGitLabGroupToken(token, config, mode, discoverySignal) + const base = `https://${host}/api/v4` + const projects: GitLabGroupScope['projects'] = [] + const ids = new Set() + const paths = new Set() + let manifestBytes = 0 + for await (const rows of listPages( + `${base}/groups/${group.id}/projects?with_shared=false&include_subgroups=${group.includeSubgroups}&simple=true&order_by=id&sort=asc`, + token, + discoverySignal, + GITLAB_GROUP_MAX_PROJECTS + )) { + for (const row of rows) { + if ( + !Number.isSafeInteger(row.id) || + row.id <= 0 || + typeof row.path_with_namespace !== 'string' || + row.path_with_namespace.length > 1024 || + !gitLabProjectInGroup(row.path_with_namespace, group) || + ids.has(row.id) || + paths.has(row.path_with_namespace) + ) + throw new Error( + 'GitLab returned an inconsistent group project listing. Retry the update.' + ) + const project = { id: row.id, path: row.path_with_namespace } + manifestBytes += Buffer.byteLength(JSON.stringify(project), 'utf8') + 1 + if (manifestBytes > 16 * 1024 * 1024) + throw new Error('GitLab group project paths exceed 16 MiB. Choose a narrower group.') + ids.add(project.id) + paths.add(project.path) + projects.push(project) + } + } + if (!projects.length) + throw new Error('The GitLab token cannot read any projects in this group.') + projects.sort((left, right) => left.id - right.id) + return { host, group, projects } + } catch (error) { + if (deadline.aborted && !signal?.aborted) + throw new Error( + 'GitLab group discovery exceeded five minutes. Retry or choose a narrower group.' + ) + throw error } - if (!projects.length) throw new Error('The GitLab token cannot read any projects in this group.') - projects.sort((left, right) => left.id - right.id) - return { host, group, projects } } export interface GitLabPermissionUser { @@ -285,20 +305,23 @@ async function requireAdministrator( } /** Bounded complete enumeration. Never accept a partial directory as authoritative membership. */ -async function listAll( +async function* listPages( url: string, token: string, signal?: AbortSignal, maxItems?: number -): Promise { +): AsyncGenerator { const initial = new URL(url) initial.searchParams.set('per_page', String(PAGE_SIZE)) let next: URL | null = initial const seen = new Set() - const items: T[] = [] + let itemCount = 0 let retainedBytes = 0 let expectedTotal: number | undefined - for (let page = 0; next && page < MAX_DIRECTORY_PAGES; page += 1) { + const maxPages = + maxItems === undefined ? MAX_DIRECTORY_PAGES : Math.ceil(maxItems / PAGE_SIZE) + 1 + for (let page = 0; next && page < maxPages; page += 1) { + signal?.throwIfAborted() if (seen.has(next.href)) throw new Error('GitLab permission pagination repeated a page') seen.add(next.href) const response = await secureFetchWithRetry( @@ -313,26 +336,30 @@ async function listAll( if (!response.ok) throw new Error(`GitLab permission listing failed (${response.status})`) const body = (await response.json()) as T[] if (!Array.isArray(body)) throw new Error('GitLab permission listing was not an array') - retainedBytes += Buffer.byteLength(JSON.stringify(body), 'utf8') - if (body.length > PAGE_SIZE || retainedBytes > MAX_DIRECTORY_BYTES) { + if (maxItems === undefined) retainedBytes += Buffer.byteLength(JSON.stringify(body), 'utf8') + if ( + body.length > PAGE_SIZE || + (maxItems === undefined && retainedBytes > MAX_DIRECTORY_BYTES) + ) { throw new Error('GitLab permission directory exceeded its response budget') } - items.push(...body) + itemCount += body.length if (maxItems !== undefined) { const totalHeader = response.headers.get('x-total') if (totalHeader !== null) { const total = Number(totalHeader) - if ( - !Number.isSafeInteger(total) || - total < 0 || - total > maxItems || - (expectedTotal !== undefined && total !== expectedTotal) - ) - throw new Error('GitLab group project total changed or exceeded its limit') + if (!/^\d+$/.test(totalHeader) || !Number.isSafeInteger(total)) + throw new Error('GitLab returned an invalid group project total.') + if (total > maxItems) + throw new Error('GitLab group exceeds 50,000 projects. Choose a narrower group.') + if (expectedTotal !== undefined && total !== expectedTotal) + throw new Error('GitLab group projects changed during discovery. Retry the update.') expectedTotal = total } - if (items.length > maxItems) throw new Error('GitLab group exceeded its project limit') + if (itemCount > maxItems) + throw new Error('GitLab group exceeds 50,000 projects. Choose a narrower group.') } + yield body const link = response.headers .get('link') ?.split(',') @@ -367,9 +394,14 @@ async function listAll( next = null } } - if (expectedTotal !== undefined && items.length !== expectedTotal) + if (expectedTotal !== undefined && itemCount !== expectedTotal) throw new Error('GitLab group project listing was incomplete') if (next) throw new Error('GitLab permission directory exceeded its page limit') +} + +async function listAll(url: string, token: string, signal?: AbortSignal): Promise { + const items: T[] = [] + for await (const page of listPages(url, token, signal)) items.push(...page) return items } diff --git a/apps/sim/hooks/queries/kb/connectors.ts b/apps/sim/hooks/queries/kb/connectors.ts index 414b5b44ab6..b512974221a 100644 --- a/apps/sim/hooks/queries/kb/connectors.ts +++ b/apps/sim/hooks/queries/kb/connectors.ts @@ -38,11 +38,13 @@ import type { } from '@/lib/api/contracts/knowledge/connectors' import { type ConnectorDocumentsQuery, + createConnectorBodySchema, type PrepareSearchSourceBody, prepareSearchSourceContract, readSearchIndexContract, type SearchConnectionOAuthQuery, type SearchSourcePage, + updateConnectorBodySchema, } from '@/lib/api/contracts/knowledge/connectors' import { type ResourceScope, @@ -54,6 +56,7 @@ import { connectCredentialGroupInPopup } from '@/lib/credential-groups/oauth-pop import { isDesktopApp } from '@/lib/desktop' import { connectDesktopSource } from '@/lib/desktop/source-connect' import { MAX_KNOWLEDGE_CONNECTOR_DOCUMENT_PAGE_SIZE } from '@/lib/knowledge/constants' +import { getGitLabUploadSizeError } from '@/connectors/gitlab/permission-config/request-size' import { credentialGroupKeys } from '@/hooks/queries/utils/credential-group-queries' import { knowledgeKeys } from '@/hooks/queries/utils/knowledge-keys' import { organizationAccountsKeys } from '@/hooks/queries/utils/organization-accounts-keys' @@ -276,9 +279,12 @@ async function createConnector({ knowledgeBaseId, ...body }: CreateConnectorParams): Promise { + const payload = createConnectorBodySchema.parse(body) + const uploadError = payload.permissionConfig?.mode === 'csv' && getGitLabUploadSizeError(payload) + if (uploadError) throw new Error(uploadError) const result = await requestJson(createKnowledgeConnectorContract, { params: { id: knowledgeBaseId }, - body, + body: payload, }) return result.data @@ -323,9 +329,12 @@ async function updateConnector({ connectorId, updates, }: UpdateConnectorParams): Promise { + const payload = updateConnectorBodySchema.parse(updates) + const uploadError = payload.permissionConfig?.mode === 'csv' && getGitLabUploadSizeError(payload) + if (uploadError) throw new Error(uploadError) const result = await requestJson(updateKnowledgeConnectorContract, { params: { id: knowledgeBaseId, connectorId }, - body: updates, + body: payload, }) return result.data diff --git a/apps/sim/lib/api/contracts/knowledge/gitlab-permissions.ts b/apps/sim/lib/api/contracts/knowledge/gitlab-permissions.ts index 6f3d390b14b..df4986065cf 100644 --- a/apps/sim/lib/api/contracts/knowledge/gitlab-permissions.ts +++ b/apps/sim/lib/api/contracts/knowledge/gitlab-permissions.ts @@ -1,6 +1,6 @@ import { z } from 'zod' import { - GITLAB_CSV_MAX_BYTES, + GITLAB_CSV_MAX_ROWS, GITLAB_PERMISSION_MODES, } from '@/connectors/gitlab/permission-config/types' @@ -10,9 +10,8 @@ const gitLabCsvUploadSchema = z content: z .string() .min(1) - .max(GITLAB_CSV_MAX_BYTES) .describe( - 'UTF-8 CSV contents, up to 4 MiB. Both files travel in one request body limited to 10 MiB after JSON encoding.' + 'UTF-8 CSV contents, up to 8 MiB. Both files travel in one request body limited to 10 MiB after JSON encoding.' ), }) .strict() @@ -38,7 +37,7 @@ export type GitLabPermissionUploadInput = z.input import( diff --git a/apps/sim/lib/knowledge/__integration__/gitlab-group.integration.ts b/apps/sim/lib/knowledge/__integration__/gitlab-group.integration.ts index 375e6e7e43e..f8f5c27d49f 100644 --- a/apps/sim/lib/knowledge/__integration__/gitlab-group.integration.ts +++ b/apps/sim/lib/knowledge/__integration__/gitlab-group.integration.ts @@ -58,6 +58,8 @@ describe('GitLab group Search permission boundary', () => { let aliceLegacyResult: { documentId: string } | undefined let confidential = false let providerState = 'healthy' + let largeProjectCount = 50_000 + let nativeGroup = false const checks: string[] = [] const token = 'disposable-group-test-token' const principal = (userId: string): Principal => ({ @@ -98,10 +100,73 @@ describe('GitLab group Search permission boundary', () => { if (providerState === 'revoked') return reply(401, {}) const namespace = providerState === 'renamed' ? 'renamed-team' : 'team' if (path === '/api/v4/user') return reply(200, { id: 50, state: 'active', locked: false }) + if (path === '/api/v4/groups/8') return reply(200, { id: 8, full_path: 'large' }) + if (path === '/api/v4/groups/8/projects') { + const page = Number(url.searchParams.get('page') ?? 1) + const offset = (page - 1) * 100 + return reply( + 200, + Array.from( + { length: Math.max(0, Math.min(100, largeProjectCount - offset)) }, + (_, index) => ({ + id: 100_000 + offset + index, + path_with_namespace: `large/project-${String(offset + index).padStart(5, '0')}-permission-fixture`, + }) + ) + ) + } + if (path === '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/api/v4/projects/149999') + return reply(200, { + id: 149999, + path_with_namespace: 'large/project-49999-permission-fixture', + web_url: 'https://gitlab.fixture.test/large/project-49999-permission-fixture', + }) + if ( + path === '/api/v4/groups/8/search' || + path === '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/api/v4/projects/149999/search' || + path === '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/api/v4/projects/149999/issues/1' + ) { + const issue = { + project_id: 149999, + iid: 1, + title: 'Atlas last project', + description: 'Atlas permitted content', + confidential: false, + state: 'opened', + labels: [], + web_url: 'https://gitlab.fixture.test/large/project-49999-permission-fixture/-/issues/1', + } + return reply(200, path.endsWith('/search') ? [issue] : issue) + } + if (path === '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/api/v4/projects/149999/issues/1/notes') return reply(200, []) if (path === '/api/v4/groups/team' && providerState === 'renamed') return reply(301, {}, { location: '/api/v4/groups/7' }) if (path === '/api/v4/groups/team' || path === '/api/v4/groups/7') return reply(200, { id: 7, full_path: namespace }) + if (path === '/api/v4/groups/7/search') { + if (providerState === 'search-token-expired') return reply(401, {}) + if (providerState === 'search-rate-limited') return reply(429, {}, { 'retry-after': '60' }) + if (!nativeGroup || url.searchParams.get('scope') !== 'issues') return reply(400, {}) + if ( + url.searchParams.get('exclude_forks') !== 'false' || + url.searchParams.get('include_archived') !== 'true' + ) + return reply(200, []) + const rows = [12, 11, 13, 99, 11].map((id, index) => ({ + project_id: id, + iid: index === 4 ? 2 : 1, + title: `Atlas ${id}`, + description: 'Atlas content', + confidential: index === 4, + state: 'opened', + labels: [], + web_url: `https://gitlab.fixture.test/${namespace}/${id === 11 ? 'a' : id === 12 ? 'b' : 'c'}/-/issues/1`, + })) + const page = Number(url.searchParams.get('page') ?? 1) + const size = Number(url.searchParams.get('per_page') ?? 20) + if (page === 5000) return reply(200, [rows[1], rows[2]]) + return reply(200, rows.slice((page - 1) * size, page * size)) + } if (path === '/api/v4/groups/7/projects') { if (url.searchParams.get('with_shared') !== 'false') return reply(400, {}) const projects = [ @@ -151,12 +216,28 @@ describe('GitLab group Search permission boundary', () => { if (match[2] === '/search') { if (providerState === 'search-token-expired') return reply(401, {}) if (providerState === 'search-rate-limited') return reply(429, {}, { 'retry-after': '60' }) + if (url.searchParams.get('scope') === 'blobs' && id === 11 && nativeGroup) + return reply(200, [ + { + project_id: 11, + filename: 'atlas.md', + path: 'atlas.md', + ref: 'main', + data: 'Atlas code content', + }, + ]) + if (url.searchParams.get('scope') !== 'issues') return reply(200, []) const page = Number(url.searchParams.get('page') ?? 1) const size = Number(url.searchParams.get('per_page') ?? 20) return reply(200, [issue(1), issue(2), issue(3)].slice((page - 1) * size, page * size)) } const issueMatch = match[2]?.match(/^\/issues\/(\d+)$/) if (issueMatch) return reply(200, issue(Number(issueMatch[1]))) + if (match[2] === '/repository/files/atlas.md') + return reply(200, { + file_name: 'atlas.md', + content: Buffer.from('Atlas code content').toString('base64'), + }) if (match[2]?.endsWith('/notes')) return reply(200, []) return reply(404, {}) }) @@ -166,7 +247,8 @@ describe('GitLab group Search permission boundary', () => { project?: string, cursor?: string, topK = 20, - accountId = sourceId + accountId = sourceId, + kind: 'issues' | 'all' = 'issues' ) => searchLiveKnowledge.execute({ principal: principal(userId), @@ -179,7 +261,7 @@ describe('GitLab group Search permission boundary', () => { { provider: 'gitlab', query: 'Atlas', - kind: 'issues', + ...(kind === 'issues' ? { kind: 'issues' as const } : {}), accountId: `gitlab-source:${accountId}`, ...(project ? { project } : {}), ...(cursor ? { cursor } : {}), @@ -260,7 +342,6 @@ describe('GitLab group Search permission boundary', () => { boundary: 'Application use cases, real PostgreSQL and HTTP GitLab API fixture; TLS/SSRF transport substituted', checks, - passed: checks.length === 11, }, null, 2 @@ -625,4 +706,175 @@ describe('GitLab group Search permission boundary', () => { } } ) + + it('connects a complete large group and preserves its grants when the next discovery exceeds the bound', async () => { + const grantRows = Array.from({ length: 50_000 }, (_, index) => { + const path = `large/project-${String(index).padStart(5, '0')}-permission-fixture` + return `${path},1\n${path},3\n${path},4` + }).join('\n') + const uploads = { + provider: 'gitlab' as const, + userMapping: { + filename: 'users.csv', + content: `user_id,email\n1,${ids.aliceId}@fixture.test\n2,${ids.bobId}@fixture.test\n3,third@fixture.test\n4,fourth@fixture.test`, + }, + projectPermissions: { + filename: 'permissions.csv', + content: `project_path,user_id\n${grantRows}`, + }, + } + const started = performance.now() + const { connector } = await createKnowledgeConnector.execute({ + principal: principal(ids.aliceId), + input: { + knowledgeBaseId: ids.knowledgeBaseId, + assertedOrganizationId: ids.organizationId, + connectorType: 'gitlab', + accessMode: 'admin', + apiKey: token, + sourceConfig: { ...config, group: '8' }, + permissionConfig: { ...uploads, mode: 'csv' }, + syncIntervalMinutes: 60, + }, + }) + const snapshot = await loadConnectorPermissionSnapshot(connector.id) + expect(snapshot?.metadata).toMatchObject({ + projects: expect.any(Array), + projectPermissions: { rowCount: 150_000 }, + }) + const page = await search(ids.aliceId, '149999', undefined, 20, connector.id) + const first = page.results[0] + if (!first) throw new Error('Missing last-project result') + expect((await read(ids.aliceId, first.documentId)).chunks.length).toBeGreaterThan(0) + await expect(read(ids.bobId, first.documentId)).rejects.toMatchObject({ code: 'not_found' }) + const beforeBroadSearch = transport.paths.length + const broad = await search(ids.aliceId, undefined, undefined, 20, connector.id) + expect(broad.results.map((row) => decodeLiveReference(row.documentId).container)).toEqual([ + '149999', + ]) + expect( + transport.paths + .slice(beforeBroadSearch) + .filter((path) => /^\/api\/v4\/projects\/\d+\/search$/.test(path)) + ).toEqual([]) + const legacy = await search(ids.aliceId, undefined, '1:20:49999:0:1', 20, connector.id) + expect(legacy.results.map((row) => decodeLiveReference(row.documentId).container)).toEqual([ + '149999', + ]) + largeProjectCount = 50_001 + try { + await expect( + replaceSearchSourcePermissions.execute({ + principal: principal(ids.aliceId), + input: { + organizationId: ids.organizationId, + sourceId: connector.id, + expectedRevision: 1, + permissions: uploads, + }, + }) + ).rejects.toMatchObject({ code: 'validation' }) + expect(await loadConnectorPermissionSnapshot(connector.id)).toEqual(snapshot) + expect( + (await search(ids.aliceId, '149999', undefined, 20, connector.id)).results.length + ).toBeGreaterThan(0) + } finally { + largeProjectCount = 50_000 + } + checks.push( + `Complete large-group discovery, larger CSV replacement and failed-update rollback (${Math.round(performance.now() - started)} ms)` + ) + }) + + it('uses group search while preserving project grants, confidentiality and continuations', async () => { + nativeGroup = true + const requests = transport.paths.length + try { + const found: string[] = [] + let cursor: string | undefined + for (let page = 0; page < 5; page += 1) { + const result = await search(ids.aliceId, undefined, cursor, 2) + found.push( + ...result.results.map((row) => { + const ref = decodeLiveReference(row.documentId) + return `${ref.container}:${ref.id}` + }) + ) + cursor = result.live?.accounts[0]?.nextCursor + if (!cursor) break + } + expect(cursor).toBeUndefined() + expect(found).toEqual(['11:1', '13:1']) + const paths = transport.paths.slice(requests) + expect(paths.filter((path) => path === '/api/v4/groups/7/search').length).toBeGreaterThan(0) + expect(paths.filter((path) => /^\/api\/v4\/projects\/\d+\/search$/.test(path))).toEqual([]) + checks.push( + 'Native group search filters forbidden, confidential and unconfigured projects across pages' + ) + } finally { + nativeGroup = false + } + }) + + it('continues supported group searches when code and wiki require project search', async () => { + const { connector } = await createKnowledgeConnector.execute({ + principal: principal(ids.aliceId), + input: { + knowledgeBaseId: ids.knowledgeBaseId, + assertedOrganizationId: ids.organizationId, + connectorType: 'gitlab', + accessMode: 'admin', + apiKey: token, + sourceConfig: { ...config, contentTypes: 'all' }, + permissionConfig: { ...permissions(), mode: 'csv' }, + syncIntervalMinutes: 60, + }, + }) + nativeGroup = true + const requests = transport.paths.length + try { + const kinds: string[] = [] + let cursor: string | undefined + for (let page = 0; page < 12; page += 1) { + const result = await search(ids.aliceId, undefined, cursor, 20, connector.id, 'all') + kinds.push(...result.results.map((row) => decodeLiveReference(row.documentId).kind ?? '')) + cursor = result.live?.accounts[0]?.nextCursor + if (!cursor) break + } + expect(cursor).toBeUndefined() + expect(kinds).toContain('code') + expect(kinds).toContain('issues') + expect( + transport.paths.slice(requests).filter((path) => path === '/api/v4/groups/7/search').length + ).toBeGreaterThan(0) + checks.push('Unsupported group scopes do not starve supported scopes') + } finally { + nativeGroup = false + } + }) + + it('reports the group search result window as partial instead of offering an empty next page', async () => { + const snapshot = await loadConnectorPermissionSnapshot(sourceId) + if (!snapshot) throw new Error('Missing group fixture') + const cursor = `g1.${Buffer.from( + JSON.stringify({ + revision: snapshot.revision, + pageSize: 2, + kinds: ['issues'], + nextKind: 0, + capped: false, + positions: [{ mode: 'group', projectIndex: 0, page: 5000 }], + }) + ).toString('base64url')}` + nativeGroup = true + try { + const result = await search(ids.aliceId, undefined, cursor, 2) + expect(result.results).toHaveLength(2) + expect(result.live?.accounts[0]?.status).toBe('partial') + expect(result.live?.accounts[0]?.nextCursor).toBeUndefined() + checks.push('Search window returns authorized results with explicit partial coverage') + } finally { + nativeGroup = false + } + }) }) diff --git a/apps/sim/lib/knowledge/connectors/permission-store.ts b/apps/sim/lib/knowledge/connectors/permission-store.ts index 053c3feba25..4d242f7d022 100644 --- a/apps/sim/lib/knowledge/connectors/permission-store.ts +++ b/apps/sim/lib/knowledge/connectors/permission-store.ts @@ -3,7 +3,7 @@ import { knowledgeConnectorPermissionGrant, knowledgeConnectorPermissionSnapshot, } from '@sim/db/schema' -import { and, eq, inArray } from 'drizzle-orm' +import { and, eq, inArray, sql } from 'drizzle-orm' import { OrchestrationError } from '@/lib/core/orchestration/types' import type { DbTransaction } from '@/lib/db/types' @@ -98,8 +98,17 @@ export async function loadConnectorPermissionSnapshot(connectorId: string) { } /** Metadata and worker reads leave the potentially large payload in PostgreSQL. */ -export async function readConnectorPermissionMetadata(connectorIds: readonly string[]) { +export async function readConnectorPermissionMetadata( + connectorIds: readonly string[], + fields?: readonly string[] +) { const table = knowledgeConnectorPermissionSnapshot + const metadata = fields?.length + ? sql>`jsonb_build_object(${sql.join( + fields.map((field) => sql`${field}::text, ${table.metadata} -> ${field}::text`), + sql`, ` + )})` + : table.metadata const result: Array> = [] for (let offset = 0; offset < connectorIds.length; offset += 1000) { @@ -108,7 +117,7 @@ export async function readConnectorPermissionMetadata(connectorIds: readonly str .select({ connectorId: table.connectorId, revision: table.revision, - metadata: table.metadata, + metadata, }) .from(table) .where(inArray(table.connectorId, connectorIds.slice(offset, offset + 1000)))) diff --git a/apps/sim/lib/sim-search/live/gitlab-admin.ts b/apps/sim/lib/sim-search/live/gitlab-admin.ts index e0e82df519b..067a180c07a 100644 --- a/apps/sim/lib/sim-search/live/gitlab-admin.ts +++ b/apps/sim/lib/sim-search/live/gitlab-admin.ts @@ -3,6 +3,7 @@ import { knowledgeBase, knowledgeConnector } from '@sim/db/schema' import { createLogger } from '@sim/logger' import { describeError } from '@sim/utils/errors' import { and, eq, inArray, isNull } from 'drizzle-orm' +import { z } from 'zod' import { decryptApiKey } from '@/lib/api-key/crypto' import { type ResourceOwner, resourceScopeFromOwner } from '@/lib/core/resource-scope' import { resourceScopeCondition } from '@/lib/core/resource-scope.server' @@ -15,6 +16,7 @@ import { readConnectorPermissionMetadata, } from '@/lib/knowledge/connectors/permission-store' import { searchIntegrationAccessCondition } from '@/lib/knowledge/search/integration-policy' +import { dateSortDirection, hasDateBounds } from '@/lib/sim-search/live/dates' import { readGitLab, searchGitLab } from '@/lib/sim-search/live/gitlab' import { array, NativeSearchError, object, segment, string } from '@/lib/sim-search/live/http' import { collectNativePages, joinMessages } from '@/lib/sim-search/live/pages' @@ -26,12 +28,14 @@ import type { NativeSearchInput, } from '@/lib/sim-search/live/types' import { + GITLAB_GROUP_MAX_PROJECTS, type GitLabGroupScope, gitLabGroupMatchesConfig, gitLabProjectInGroup, } from '@/connectors/gitlab/group' import { readGitLabPermissionMetadata, + readGitLabPermissionSummaries, seedGitLabPermissionContext, } from '@/connectors/gitlab/permission-config/repository' import { @@ -85,23 +89,15 @@ async function viewerToken(owner: ResourceOwner, userId: string): Promise value.startsWith('u:')) : undefined } -function hasCsvPermissions(metadata: unknown): boolean { - try { - return readGitLabPermissionMetadata(metadata).mode === 'csv' - } catch { - return false - } -} - /** The IDs of CSV-permission sources that grant the viewer no project. */ async function ungrantedSources( owner: ResourceOwner, userId: string, ids: string[] ): Promise> { - const csv = (await readConnectorPermissionMetadata(ids)) - .filter((row) => hasCsvPermissions(row.metadata)) - .map((row) => row.connectorId) + const csv = [...(await readGitLabPermissionSummaries(ids))] + .filter(([, summary]) => summary.mode === 'csv') + .map(([id]) => id) const own = csv.length ? await viewerToken(owner, userId) : undefined if (!own) return new Set() const granted = await connectorsGrantingSubject(csv, own) @@ -423,25 +419,61 @@ async function createProjectSession( } } -/** Traverse bounded project pages; every continuation and document read rechecks current grants. */ +const groupCursorSchema = z + .object({ + revision: z.number().int().nonnegative(), + pageSize: z.number().int().min(1).max(100), + kinds: z.array(z.string().max(20)).min(1).max(4), + nextKind: z.number().int().min(0).max(3), + capped: z.boolean(), + positions: z + .array( + z + .object({ + mode: z.enum(['group', 'project']), + projectIndex: z.number().int().min(0).max(GITLAB_GROUP_MAX_PROJECTS), + page: z.number().int().min(1).max(10_000), + }) + .strict() + .nullable() + ) + .min(1) + .max(4), + }) + .strict() + +/** Search group candidates; every continuation and document read rechecks current project grants. */ async function createGroupSession(input: GitLabSessionInput, own: string) { const { source, client } = input const [stored] = await readConnectorPermissionMetadata([source.id]) const metadata = stored && readGitLabPermissionMetadata(stored.metadata) if (!metadata || !('group' in metadata) || !gitLabGroupMatchesConfig(metadata, source.config)) throw new NativeSearchError('unavailable', 'An admin must update this GitLab group connection.') - const grants = - metadata.mode === 'csv' - ? await readConnectorPermissionGrants( - source.id, - stored.revision, - metadata.projects.map((project) => `project:${project.id}`), - own - ) - : null - const projects = metadata.projects.filter( - (project) => !grants || grants.has(`project:${project.id}`) - ) + const projects = metadata.projects + let projectGrants: Promise | null> | undefined + const fallbackProjects = async () => { + projectGrants ??= + metadata.mode === 'csv' + ? readConnectorPermissionGrants( + source.id, + stored.revision, + projects.map((project) => `project:${project.id}`), + own + ) + : Promise.resolve(null) + const grants = await projectGrants + return grants ? projects.filter((project) => grants.has(`project:${project.id}`)) : projects + } + const permittedProjects = async (candidates: GitLabGroupScope['projects']) => { + if (metadata.mode !== 'csv') return candidates + const grants = await readConnectorPermissionGrants( + source.id, + stored.revision, + candidates.map((project) => `project:${project.id}`), + own + ) + return candidates.filter((project) => grants.has(`project:${project.id}`)) + } const group = object(await client.json(`/api/v4/groups/${metadata.group.id}`, { memo: true })) if ( string(group.id) !== String(metadata.group.id) || @@ -485,15 +517,187 @@ async function createGroupSession(input: GitLabSessionInput, own: string) { } return result } - const referencedProject = (container?: string) => - projects.find((project) => container === String(project.id) || container === project.path) + const searchGroup = async ( + search: NativeSearchInput, + kinds: string[], + raw?: string + ): Promise => { + const fallback = (kind: string) => + Boolean( + (['code', 'wiki'].includes(kind) && string(source.config.ref)) || + hasDateBounds(search.filters) || + dateSortDirection(search.filters) + ) + let cursor: z.output + if (raw) { + try { + if (raw.length > 4000) throw new Error('Oversized cursor') + cursor = groupCursorSchema.parse( + JSON.parse(Buffer.from(raw.slice(3), 'base64url').toString('utf8')) + ) + } catch { + throw new NativeSearchError('unavailable', 'Invalid GitLab group page.') + } + if ( + cursor.revision !== stored.revision || + cursor.pageSize !== search.limit || + cursor.kinds.join(',') !== kinds.join(',') || + cursor.positions.length !== kinds.length || + cursor.nextKind >= kinds.length || + cursor.positions.some( + (position, index) => position?.mode === 'group' && fallback(kinds[index] ?? '') + ) + ) + throw new NativeSearchError( + 'unavailable', + 'GitLab permissions or search options changed. Restart the search.' + ) + } else { + cursor = { + revision: stored.revision, + pageSize: search.limit, + kinds, + nextKind: 0, + capped: false, + positions: kinds.map((kind) => ({ + mode: fallback(kind) ? 'project' : 'group', + projectIndex: 0, + page: 1, + })), + } + } + const failures: string[] = [] + let partial = false + let attempts = 0 + let visited = 0 + const documents: NativeDocument[] = [] + while (attempts < 4 && visited < kinds.length * 4 && documents.length === 0) { + input.signal.throwIfAborted() + const index = cursor.nextKind + cursor.nextKind = (index + 1) % kinds.length + visited += 1 + const position = cursor.positions[index] + const kind = kinds[index] + if (!position || !kind) continue + const query: NativeSearchInput = { + ...search, + native: { + provider: 'gitlab', + query: search.query, + ...search.native, + kind: kind as 'code' | 'wiki' | 'issues' | 'merge_requests', + cursor: String(position.page), + }, + } + attempts += 1 + try { + let result: NativePage + if (position.mode === 'group') { + result = await searchGitLab(client, query, metadata.group.id) + const byId = new Set(result.documents.map((document) => document.container)) + const candidates = projects.filter((project) => byId.has(String(project.id))) + const allowed = new Map( + (await permittedProjects(candidates)).map((project) => [String(project.id), project]) + ) + result.documents = result.documents.filter((document) => + allowed.has(document.container ?? '') + ) + for (const document of result.documents) { + const project = allowed.get(document.container ?? '') + if (!document.url && project) { + const base = `https://${metadata.host}/${project.path.split('/').map(segment).join('/')}` + document.url = + kind === 'code' + ? `${base}/-/blob/${segment(document.revision || 'HEAD')}/${document.id.split('/').map(segment).join('/')}` + : kind === 'wiki' + ? `${base}/-/wikis/${document.id.split('/').map(segment).join('/')}` + : `${base}/-/${kind}/${segment(document.id)}` + } + } + } else { + const permitted = await fallbackProjects() + const project = permitted[position.projectIndex] + if (!project) { + cursor.positions[index] = null + continue + } + result = await (await session(project)).search({ + ...query, + native: { + provider: 'gitlab', + query: query.query, + ...query.native, + project: String(project.id), + }, + }) + } + documents.push(...result.documents) + partial ||= Boolean(result.partial) + cursor.capped ||= Boolean(result.hasMore && !result.nextCursor) + if (result.nextCursor) position.page = Number(result.nextCursor) + else if (position.mode === 'project') { + position.projectIndex += 1 + position.page = 1 + } else cursor.positions[index] = null + } catch (error) { + input.signal.throwIfAborted() + if ( + error instanceof NativeSearchError && + position.mode === 'group' && + (error.httpStatus === 400 || error.httpStatus === 422) + ) { + position.mode = 'project' + position.page = 1 + continue + } + if ( + error instanceof NativeSearchError && + ['reconnect', 'rate_limited', 'timeout'].includes(error.status) + ) + throw error + if (position.mode === 'group') throw error + partial = true + failures.push( + error instanceof NativeSearchError + ? error.message + : 'One GitLab collection could not be searched.' + ) + position.projectIndex += 1 + position.page = 1 + } + } + return { + documents, + partial: partial || cursor.capped, + ...(cursor.capped ? { hasMore: true } : {}), + ...(cursor.positions.some(Boolean) + ? { nextCursor: `g1.${Buffer.from(JSON.stringify(cursor)).toString('base64url')}` } + : {}), + message: joinMessages([ + 'Searched the configured GitLab group with per-project permissions. Continue with the cursor for remaining results.', + cursor.positions.some((position) => position?.mode === 'project') + ? 'Some scopes require project-by-project search on this GitLab instance. Target a project for faster results.' + : undefined, + cursor.capped + ? 'GitLab group search reached its 10,000-result window. Narrow the query or target a project.' + : undefined, + ...failures, + ]), + } + } + const referencedProject = async (container?: string) => { + const project = projects.find( + (project) => container === String(project.id) || container === project.path + ) + return project ? (await permittedProjects([project]))[0] : undefined + } return { async verify(reference: Reference, evidence?: Record) { - const project = referencedProject(reference.container) + const project = await referencedProject(reference.container) return project ? (await session(project)).verify(reference, evidence) : false }, async read(reference: Reference) { - const project = referencedProject(reference.container) + const project = await referencedProject(reference.container) if (!project) throw new NativeSearchError( 'unavailable', @@ -503,16 +707,21 @@ async function createGroupSession(input: GitLabSessionInput, own: string) { }, async search(search: NativeSearchInput): Promise { if (search.native?.project) { - const project = referencedProject(search.native.project) + const project = await referencedProject(search.native.project) return project ? (await session(project)).search(search) : { documents: [] } } const kinds = gitLabSourceKinds(source.config).filter( (kind) => !search.native?.kind || kind === search.native.kind ) - const raw = search.native?.cursor ?? `${stored.revision}:${search.limit}:0:0:1` - if (!/^\d{1,10}:\d{1,3}:\d{1,4}:\d:\d{1,4}$/.test(raw)) + if (!kinds.length) return { documents: [] } + const raw = search.native?.cursor + if (metadata.mode === 'csv' && (!raw || raw.startsWith('g1.'))) + return searchGroup(search, kinds, raw) + const projects = await fallbackProjects() + const legacyCursor = raw ?? `${stored.revision}:${search.limit}:0:0:1` + if (!/^\d{1,10}:\d{1,3}:\d{1,5}:\d:\d{1,4}$/.test(legacyCursor)) throw new NativeSearchError('unavailable', 'Invalid GitLab group page.') - const [revision, pageSize, ...position] = raw.split(':').map(Number) + const [revision, pageSize, ...position] = legacyCursor.split(':').map(Number) if (revision !== stored.revision || pageSize !== search.limit) throw new NativeSearchError( 'unavailable', diff --git a/apps/sim/lib/sim-search/live/gitlab.ts b/apps/sim/lib/sim-search/live/gitlab.ts index d06c11769ef..3a42451a3c0 100644 --- a/apps/sim/lib/sim-search/live/gitlab.ts +++ b/apps/sim/lib/sim-search/live/gitlab.ts @@ -40,9 +40,10 @@ function document(row: Record, kind: string): NativeDocument { export async function searchGitLab( client: NativeClient, - input: NativeSearchInput + input: NativeSearchInput, + groupId?: number ): Promise { - if (!input.native?.project) { + if (!input.native?.project && groupId === undefined) { const projects = array( await client.json('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/api/v4/projects', { query: { @@ -83,13 +84,22 @@ export async function searchGitLab( : result.message, } } - if (!input.native.kind) + if (!input.native?.kind) return collectNativePages( ['issues', 'code', 'merge_requests'].map((kind) => - searchGitLab(client, { - ...input, - native: { ...input.native!, kind: kind as 'issues' | 'code' | 'merge_requests' }, - }) + searchGitLab( + client, + { + ...input, + native: { + provider: 'gitlab', + query: input.query, + ...input.native, + kind: kind as 'issues' | 'code' | 'merge_requests', + }, + }, + groupId + ) ), 'Searched GitLab issues, merge requests, and code.' ) @@ -107,13 +117,19 @@ export async function searchGitLab( 'GitLab supports issues, code, merge_requests, or wiki search.' ) const page = input.native?.cursor ?? '1' - if (!/^\d{1,4}$/.test(page) || Number(page) < 1) + if (!/^\d{1,5}$/.test(page) || Number(page) < 1 || Number(page) > 10_000) throw new NativeSearchError('unavailable', 'Invalid GitLab page.') if (['code', 'wiki'].includes(kind) && hasDateBounds(input.filters)) throw new NativeSearchError( 'unavailable', 'GitLab code/wiki search does not provide modification dates. Use issues or merge_requests for date-filtered search.' ) + const groupPageLimit = Math.floor(10_000 / input.limit) + if (groupId !== undefined && Number(page) > groupPageLimit) + throw new NativeSearchError( + 'unavailable', + 'GitLab group search reached its result window. Narrow the query or target a project.' + ) const dates = nativeDateBounds(input) const listing = Boolean( input.native?.project && @@ -123,14 +139,17 @@ export async function searchGitLab( let response: unknown try { response = await client.json( - listing - ? `/api/v4/projects/${segment(input.native!.project!)}/${kind}` - : input.native?.project - ? `/api/v4/projects/${segment(input.native.project)}/search` - : '/api/v4/search', + groupId !== undefined + ? `/api/v4/groups/${groupId}/search` + : listing + ? `/api/v4/projects/${segment(input.native!.project!)}/${kind}` + : input.native?.project + ? `/api/v4/projects/${segment(input.native.project)}/search` + : '/api/v4/search', { query: { scope: listing ? 'all' : scope, + ...(groupId !== undefined ? { exclude_forks: 'false', include_archived: 'true' } : {}), ...(nativeText(input) ? { search: nativeText(input) } : {}), ...(listing && dates.start ? { updated_after: dates.start } : {}), ...(listing && dates.end ? { updated_before: dates.end } : {}), @@ -147,17 +166,19 @@ export async function searchGitLab( throw new NativeSearchError( error.status, `GitLab ${kind} search: ${error.message}`, - error.retryAfterSeconds + error.retryAfterSeconds, + error.httpStatus, + error.diagnosticCode ) throw error } - if (!Array.isArray(response)) + if (!Array.isArray(response) || response.length > input.limit) throw new NativeSearchError('unavailable', 'GitLab returned an invalid search response.') const data = array(response) const documents = data.map((row) => document(row, kind)) const projects = new Map() for (const item of documents) { - if (!item.url && item.container) { + if (groupId === undefined && !item.url && item.container) { let url = projects.get(item.container) if (!url) { url = string( @@ -173,9 +194,15 @@ export async function searchGitLab( : `${url}/-/${kind}/${segment(item.id)}` } } + const capped = + groupId !== undefined && Number(page) === groupPageLimit && data.length === input.limit return { documents, - ...(data.length === input.limit ? { nextCursor: String(Number(page) + 1) } : {}), + ...(capped + ? { partial: true, hasMore: true } + : data.length === input.limit + ? { nextCursor: String(Number(page) + 1) } + : {}), message: 'GitLab code/wiki/global search availability depends on the instance plan and enabled search backend. Set project to narrow to a project ID or path.', } diff --git a/apps/sim/lib/sim-search/live/providers.ts b/apps/sim/lib/sim-search/live/providers.ts index 09f75ab7d48..c555aa928dc 100644 --- a/apps/sim/lib/sim-search/live/providers.ts +++ b/apps/sim/lib/sim-search/live/providers.ts @@ -266,7 +266,7 @@ export const LIVE_SEARCH_PROVIDERS = { avoid: 'expecting date filters or sorting on code or wiki results, and relying on advanced-search operators (quotes, |, -) on instances that may only have basic search.', }, - search: searchGitLab, + search: (client, input) => searchGitLab(client, input), read: (client, reference) => readGitLab(client, reference.id, reference.container, reference.kind, reference.revision), }, diff --git a/scripts/check-explicit-any.baseline.json b/scripts/check-explicit-any.baseline.json index ad88ebef161..4414fd4f85a 100644 --- a/scripts/check-explicit-any.baseline.json +++ b/scripts/check-explicit-any.baseline.json @@ -1568,7 +1568,7 @@ "apps/sim/lib/sim-search/live/github.ts": 1, "apps/sim/lib/sim-search/live/gitlab-admin.test.ts": 1, "apps/sim/lib/sim-search/live/gitlab-admin.ts": 1, - "apps/sim/lib/sim-search/live/gitlab.ts": 3, + "apps/sim/lib/sim-search/live/gitlab.ts": 2, "apps/sim/lib/sim-search/live/google-meet.ts": 1, "apps/sim/lib/sim-search/live/granola-mcp.ts": 2, "apps/sim/lib/sim-search/live/hubspot-mcp.test.ts": 1, diff --git a/scripts/check-unused-exports.baseline.json b/scripts/check-unused-exports.baseline.json index 7981a84fd47..3ad7f87c93d 100644 --- a/scripts/check-unused-exports.baseline.json +++ b/scripts/check-unused-exports.baseline.json @@ -1521,7 +1521,6 @@ "apps/sim/lib/api/contracts/knowledge/connectors.ts#connectorDocumentsQuerySchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#connectorMemberSummarySchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#connectorRequestedAccessModeSchema", - "apps/sim/lib/api/contracts/knowledge/connectors.ts#createConnectorBodySchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#deleteConnectorQuerySchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#listSearchSourcesQuerySchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#memberSyncLogDataSchema", @@ -1532,7 +1531,6 @@ "apps/sim/lib/api/contracts/knowledge/connectors.ts#syncLogStatusSchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#triggerKnowledgeConnectorSyncQuerySchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#updateConnectorAccessBodySchema", - "apps/sim/lib/api/contracts/knowledge/connectors.ts#updateConnectorBodySchema", "apps/sim/lib/api/contracts/knowledge/connectors.ts#viewerConnectorMembershipSchema", "apps/sim/lib/api/contracts/knowledge/documents.ts#BulkCreateDocumentsBody", "apps/sim/lib/api/contracts/knowledge/documents.ts#CreateKnowledgeDocumentsBody",