feat: updating LLM calls and other minor fixes - #116
Conversation
1d37191 to
2b86902
Compare
|
| Filename | Overview |
|---|---|
| app/workers/tasks/evaluate_call_import_row_core.py | Adds shared LLM-configuration grouping and expected-call counting aligned with the worker鈥檚 one-call-per-configuration execution. |
| app/workers/tasks/evaluate_call_import_row.py | Updates per-row LLM scoring to process configuration buckets and revised metric-result behavior. |
| app/api/v1/routes/call_import_evaluations.py | Updates evaluation creation and serialization, including source-row handling and expected LLM-call metadata. |
| app/services/call_imports/bulk_ops.py | Revises parent import rollups and latest-evaluation status mapping; the previously reported sibling trigger is no longer reachable through the current request contract. |
| app/api/v1/routes/call_imports.py | Exposes latest evaluation status in call-import responses while retaining the previously reported per-item list lookup. |
| app/services/metric_failure_policy.py | Refines metric failure classification used by evaluation result processing. |
| frontend/src/pages/callImports/CallImports.tsx | Updates call-import list status presentation using the new backend evaluation-status field. |
| frontend/src/pages/callImports/CallImportDetail.tsx | Updates evaluation progress and expected LLM-call presentation. |
| app/models/schemas.py | Extends call-import and evaluation response contracts with latest-status and expected-call fields. |
Reviews (3): Last reviewed commit: "fix: updating single metric runs" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| """Expected LLM calls per row (one per distinct LLM config among metrics).""" | ||
| configs: set[LlmConfigKey] = set() | ||
| for metric in metrics: | ||
| if (metric.name or "").strip().lower() in AUDIO_ONLY_METRIC_NAMES: | ||
| continue | ||
| configs.add( | ||
| resolve_metric_llm_config( | ||
| metric, | ||
| overrides=overrides, | ||
| run_provider=run_provider, | ||
| run_model=run_model, | ||
| run_llm_config=run_llm_config, | ||
| run_credential_id=run_credential_id, | ||
| ) | ||
| ) | ||
| return len(configs) |
There was a problem hiding this comment.
When selected metrics include multiple parent groups with the same LLM configuration or comparison metrics sharing a configuration, this function counts only distinct configuration tuples while the worker makes a separate call per parent group and per comparison metric, causing expected_llm_calls_per_row to report fewer calls than are actually made.
Knowledge Base Used: Call imports from source file to evaluated report
| .order_by(CallImportEvaluation.created_at.desc()) | ||
| .first() |
There was a problem hiding this comment.
When production and diarised sibling evaluations created in the same transaction later have different statuses, their created_at values can tie and this query selects an arbitrary sibling as latest; the selected status then determines the parent import status, causing the batch to be reported as completed, partial, or failed inconsistently with its sibling runs.
Knowledge Base Used: Call imports from source file to evaluated report
1 participant
What Changed?
Briefly describe what this PR changes.
Why?
Explain the problem this solves and why this approach was chosen.
How to Test?
List clear steps for reviewers to verify the change.
Release Label
Select one semantic version bump intent for this PR:
major- breaking change, next release bumps major versionminor- backward-compatible feature, next release bumps minor versionfix- backward-compatible bug fix, next release bumps patch versionIf you do not have permission to apply labels, mention the intended release label here and a maintainer will set it.
Checklist
CONTRIBUTING.mdguide.