Repository navigation
feat: move runs grouping and filtering server-side - #570
Closed
Cedric Vidal (cedricvidal) wants to merge 29 commits into
Closed
Cedric Vidal (cedricvidal) wants to merge 29 commits into
Cedric Vidal (cedricvidal) wants to merge 29 commits into
Conversation
Add runIds (string[]) to RunGroup type, schema, and aggregation pipeline so that report summaries, group checkbox, and bulk actions work even when a group row is collapsed. Previously these features required expanding the group to fetch individual runs.
Add $sort { label: 1 } as the 4th pipeline stage so groups are
always returned in alphabetical order by task name or submission ID.
key (taskPromptId or submissionId) is unique per group, whereas label (display name) can have ties, leading to non-deterministic ordering.
- Wire status and outcome query params into API handler filter object (both were already in ListRequestsQuerySchema but not read) - Add status and outcome params to portal API client (listRuns, listRunGroups) - Pass status, outcome, and taskPromptId from filter dropdowns to API queries - Change task dropdown to store taskPromptId values (display task names) - Merge URL taskPromptId with dropdown selection (dropdown takes precedence) - Remove client-side .filter() for status/outcome/task - Update query keys to include all filter params for proper cache invalidation - Add 5 endpoint tests for status/outcome filtering (flat and grouped)
When expanding a group, the individual runs query was not passing status and outcome filters, causing unfiltered runs to appear inside filtered groups.
groupRuns(), computeStats(), and related helpers are no longer used since grouping moved server-side. Keep only formatStatRange() which is still used by RunsList for displaying aggregate stats.
…ping # Conflicts: # apps/portal/src/lib/grouping.test.ts # apps/portal/src/lib/grouping.ts
The MongoDB aggregation pipeline computes stats directly via $min/$max/$avg operators. computeStats was only used in its own tests and never called from application code.
Add migration 009 with indexes on taskPromptId, status, outcome, workerType, and deletedAt. Cosmos DB for MongoDB (RU) uses index intersection for multi-field filters, so single-field indexes are the recommended approach over compound indexes.
New docs/architecture/database.md documents all collections, their indexes, and the indexing strategy for Cosmos DB for MongoDB (RU). Updated the migrations table in db-migrations.md which was missing entries 002-009.
- Add extensions uniform detection to aggregation pipeline - Add extensions to GroupUniformValues type and Zod schema - Add extensions to portal mirrored types - Fix portal grouping.ts import: use ../types instead of shared
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #733
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #734
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #743
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #745
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #746
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #747
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #750
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #751
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #752
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #754
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #756
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #758
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #759
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #760
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #761
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #762
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #763
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #764
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #765
🎉 All tests passed! |
Collaborator
Author
|
Original author: @github-actions[bot] Test Results (Node.js 22)test: Run #766
🎉 All tests passed! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Original author: Cedric Vidal (@cedricvidal)
Closes #568
Summary
Move runs grouping, filtering, and aggregation from the portal client to the API server. The
GET /api/v1/requestsendpoint now supports agroupByquery parameter that returnsRunGroup[]via a MongoDB aggregation pipeline, plusstatus,outcome, andtaskPromptIdquery parameters for server-side filtering. Includes database indexes for filtered/grouped queries and documentation.What changed
Server-side grouping (aggregation pipeline)
buildGroupingPipeline()inapps/api/src/grouping.ts— 4-stage MongoDB aggregation ($addFields → $group → $project → $sort)groupBy=taskgroups bytaskPromptId,groupBy=submissionIdgroups bysubmissionIdrunIdsincluded in each group for checkbox selection, report summaries, and bulk actions without expandingServer-side filtering
status,outcome, andtaskPromptIdquery params wired into the MongoDB$matchfilter (both flat and grouped modes)taskPromptId(portal task dropdown stores taskPromptId values, displays task names).filter()for status/outcome/task removed entirelyDatabase indexes (migration 009)
taskPromptId,status,outcome,workerType,deletedAtfor therequestscollectionPortal changes
RunsList.tsxrefactored: usesapi.listRunGroups()whengroupByis active, fetches individual runs on group expandenabled: groupBy === "none")Dead code cleanup
groupRuns,computeStats, and all client-side grouping/aggregation logic fromapps/portal/src/lib/grouping.ts(kept onlyformatStatRange)computeStatsfromapps/api/src/grouping.ts(stats computed entirely in the MongoDB pipeline)Shared types
GroupByKey,AggregateStats,GroupAggregates,GroupUniformValues,RunGroupinpackages/sharedRunGroupSchema,GroupAggregatesSchemaZod schemasListRequestsQuerySchemaextended withgroupBy,status,outcomeparamsDocumentation
docs/architecture/database.md— documents all collections, indexes, and Cosmos DB indexing strategydocs/architecture/db-migrations.md— filled in missing migration entries 002–009Tests
buildGroupingPipeline()(grouping, stats, uniform detection, edge cases)