Skip to content

Fix orchestration completeness verification, unfound source coverage, and scoped memory isolation - #1840

Merged
Paul Lizer (paullizer) merged 1 commit into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-orchestration-verification-failure
Oct 26, 2026
Merged

Paul Lizer (paullizer) merged 1 commit into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-orchestration-verification-failure

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Summary

  • Harden model-assisted task completeness verification (functions_task_completeness.py) so valid multi-heading/discontiguous answer_quote values, negative/global constraints (such as Do not use em dashes), fenced JSON replies, and nested evidence_refs (citation_id, document_id, file_name, record IDs, and prompt/instruction labels) do not falsely fail runs with completeness_unavailable, while continuing to reject hallucinated quotes and references.
  • Supply active saved_memory, conversation_history, CONVERSATION_POLICY, and deliverable/visual guidance to check_and_correct() in adapter_compose() (functions_orchestration_composition.py) so memory-derived prefixes, section headings, personas, and prior-turn baseline facts are recognized as authorized and preserved across checks and corrections.
  • Prevent _select_scoped_memory_candidates() (functions_fact_memory_context.py) from leaking scoped CB-27-* and FED-* memories into unmatched conversations via fallback when no scoped identifier matches the active request or conversation context.
  • Support linked PDF extraction with PyMuPDF page-by-page [Page X of Y] text and page-count coverage in functions_source_review.py / run_url_fetch(), surface pages_in_returned_excerpts in retain_document_search_result(), record skipped-URL reasons in notes, and treat evidence-backed "not found in searched source" answers (with page/source inspection coverage and evidence of what was reviewed) as complete rather than completeness_incomplete.

Linked issue

N/A

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Testing / validation

  • pytest -q functional_tests/test_task_completeness_engine.py functional_tests/test_task_completeness_runtime.py functional_tests/test_fact_memory_profile_and_mini_sk.py functional_tests/test_orchestration_request_memory_resolution.py functional_tests/test_source_review_security.py functional_tests/test_orchestration_deep_research.py functional_tests/test_orchestration_search_result_integrity.py functional_tests/test_orchestration_compose_output_retry.py (passed: 187 passed, 5 subtests passed)
  • python .\functional_tests\test_docs_app_surface_coverage.py (passed: 7/7 checks passed)
  • python .\functional_tests\test_docs_site_quality.py (passed: 6/6 checks passed)
  • python .\scripts\check_cosmos_response_boundaries.py (passed: 1081 owners, 421 doubles, 0 review gaps)

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security())
  • Settings sent to non-admin frontends use sanitize_settings_for_user()
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
  • No secrets, keys, connection strings, or local-only artifacts are included

… and scoped memory isolation

Harden normalize_assessment for multi-part/normalized answer quotes and nested evidence references, pass saved_memory and conversation_history into compose completeness verification, isolate scoped memory candidates from unmatched conversations, and add PDF page coverage extraction and unfound-source completion support.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit 034a0ba into microsoft:paullizer-react-v2-ui Oct 26, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant