Skip to content

Add inline script environment lifecycle telemetry (PEP 723 PR 15/16) - #1723

Merged
Stella Huang (StellaHuang95) merged 4 commits into
microsoft:mainfrom
StellaHuang95:pep723-pr15-telemetry-v2
Aug 19, 2026
Merged

Add inline script environment lifecycle telemetry (PEP 723 PR 15/16)#1723
Stella Huang (StellaHuang95) merged 4 commits into
microsoft:mainfrom
StellaHuang95:pep723-pr15-telemetry-v2

Conversation

@StellaHuang95

@StellaHuang95 Stella Huang (StellaHuang95) commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Part of #1602 (PEP 723 inline script env support). Design doc: #1601.

Builds on the merged creation, uv-fallback, and persistence work in #1656, #1696, and #1697. This branch is rebased on current main.

Roadmap context

This is PR 15 of 16 in the PEP 723 inline-script roadmap. It adds the remaining lifecycle telemetry for the internal manager without changing routing, creation, or cache policy.

Phase 5: Lifecycle and polish PR Status
PR 13: clear inline-script cache separate
PR 14: opportunistic TTL eviction follow-up
PR 15: lifecycle telemetry this PR
PR 16: status-bar decision resolved; no code PR

Why this PR

The manager can create, reuse, and persist inline-script environments, including consent-gated uv/Python fallback, but those lifecycle outcomes are not observable as a coherent feature funnel.

This PR adds low-cardinality telemetry that answers:

  • whether setup built a new environment or reused a validated cache entry;
  • how long actual environment construction took;
  • how many normalized dependencies were involved; and
  • which stable failure category prevented setup.

The instrumentation is deliberately attached to the underlying coalesced operation rather than every caller, and it excludes script/package/interpreter content.

What this PR does

Adds three typed lifecycle events

  • inlineScript.envCreated
  • inlineScript.envReuseHit
  • inlineScript.envError

The telemetry constants and GDPR declarations use typed event/property mappings so call sites cannot send undeclared fields.

Reports verified creation only

  • Emits envCreated only after environment creation succeeds, ownership/version validation passes, and .meta.json is persisted.
  • Measures the actual build/rebuild interval rather than lock waiting or cache inspection.
  • Reports normalized, deduplicated dependency count rather than raw dependency values.
  • Emits one creation event for the underlying coalesced operation, even when multiple callers await it.

Reports validated cache reuse only

  • Emits envReuseHit only after sidecar, interpreter, ownership, and compatibility checks complete successfully.
  • Does not report a hit for uncertain, stale, malformed, or partially validated entries.
  • Preserves the existing cache reuse and lastUsedAt behavior.

Classifies lifecycle failures without leaking details

envError uses stable low-cardinality categories for outcomes such as:

  • environment discovery failure;
  • no compatible Python;
  • declined compatible-Python installation;
  • uv/Python installation failure;
  • general setup, cache-validation, or metadata-persistence failure;
  • package-install cancellation;
  • cache-lock timeout; and
  • retained, orphaned, or otherwise unavailable lock state.

Errors remain logged through the existing paths; telemetry sends no exception text.

Preserves uv consent and compatibility behavior

  • Adds detailed internal uv lookup/install result types so the manager can distinguish available, declined, failed, and installed outcomes.
  • Keeps existing compatibility wrappers for callers that only need boolean/path results.
  • Does not change prompt text, consent requirements, install selectors, restart-required behavior, or uv's existing telemetry.

Keeps event emission coalesced and deterministic

  • Same-key concurrent creation/reuse callers receive one lifecycle result for the shared operation.
  • Cache inspection, lock acquisition, fallback selection, cancellation, and cleanup retain their existing control flow.
  • Non-applicable create() calls and pre-validation exits do not emit success-shaped lifecycle events.

Event payloads and privacy

Event Data
inlineScript.envCreated build duration; normalized dependency count
inlineScript.envReuseHit normalized dependency count
inlineScript.envError stable failure category

The events send no:

  • script URI or filesystem path;
  • requirement or dependency value;
  • package name;
  • Python/interpreter version;
  • cache key;
  • prompt text; or
  • exception/error-message content.

Script-controlled metadata therefore cannot create unbounded telemetry dimensions.

Lifecycle examples

validated cache hit
→ emit one inlineScript.envReuseHit
cache miss or stale entry
→ start build timer
→ build + install dependencies
→ validate ownership/version
→ persist sidecar
→ emit one inlineScript.envCreated
no compatible Python
→ user dismisses consent prompt
→ emit one inlineScript.envError with the declined category

Tests

Coverage includes:

  • creation emitted only after verified sidecar persistence;
  • validated reuse and rebuild behavior;
  • one event for concurrent/coalesced callers;
  • duration boundaries excluding lock wait/cache inspection;
  • normalized/deduplicated dependency counts;
  • discovery, compatibility, consent, installation, cancellation, setup, and lock categories;
  • no lifecycle telemetry for non-applicable calls;
  • direct detailed uv results for available, declined, failed, and installed outcomes; and
  • preservation of existing uv wrapper behavior.

Validation on the rebased branch:

  • npm run compile-tests
  • npm run compile
  • npm run lint
  • focused lifecycle/detailed-uv suites: 39 passing

The full Windows unit run reaches 1612 passing and 5 pending; the existing concurrent writeMetaJson rename test can still intermittently fail with EPERM on Windows. That writer is unchanged by this PR and the same failure is reproducible on main.

Performance

  • No activation work, scan, timer, watcher, or new filesystem operation is introduced.
  • Instrumentation performs constant-size event construction around operations that already occur.
  • Coalesced setup emits once rather than once per waiter.
  • No user-controlled strings are normalized or transmitted beyond the dependency count already needed for the cache operation.

User impact

No default-path user impact. The inline manager remains behind the undeclared, default-off python-envs.inlineScripts.enabled flag.

When the internal flag is manually enabled, prompts, environment creation/reuse, cancellation, error propagation, and cache behavior remain unchanged. This PR only records privacy-safe lifecycle outcomes.

Scope and follow-up

This PR intentionally does not implement:

  • script detection or automatic routing;
  • activation-time cache discovery;
  • project registration or user-facing setup UX;
  • cache clearing or TTL eviction; or
  • status-bar behavior.

The telemetry is ready for those later entry points to consume once the feature is intentionally exposed.

@rchiodo

Rich Chiodo (rchiodo) commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

🔒 Automated review in progress — Rich Chiodo (@rchiodo) is auto-reviewing this PR.

Comment thread src/common/telemetry/constants.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
Comment thread src/managers/builtin/inlineScript/envManager.ts
@rchiodo Rich Chiodo (rchiodo) added the review-auto:changes-requested Automated review: posted blocking findings to address. label Aug 18, 2026

@rchiodo Rich Chiodo (rchiodo) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Review Center.

@rchiodo Rich Chiodo (rchiodo) added review-auto:approved Automated review: no blocking findings (approval posted). and removed review-auto:changes-requested Automated review: posted blocking findings to address. labels Aug 18, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3cb82ae9-7424-40a4-9156-8c54ac6e0895
Cover detailed uv outcomes and normalized dependency counts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Keep telemetry-only helpers scoped to their consuming test suite.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
Report reuse dependency counts and preserve accurate final failure outcomes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6b12d843-8011-4bfc-9ba9-f75761eadee2
@StellaHuang95
Stella Huang (StellaHuang95) merged commit 655b075 into microsoft:main Aug 19, 2026
43 of 44 checks passed
Stella Huang (StellaHuang95) added a commit that referenced this pull request Sep 1, 2026
> Part of #1602 (PEP 723 inline script env support). Design doc: #1601.

### Roadmap context

This is **PR 14 of 16** in the PEP 723 inline-script roadmap. PR 13
added explicit, user-confirmed cache cleanup; this PR adds the separate
best-effort TTL path.

| Phase 5: Lifecycle and polish | PR | Status |
|---|---|---|
| | PR 7: persistence (`get` / `set` + Memento) | merged (#1697) |
| | PR 13: Clear Script Environment Cache | merged (#1724) |
| | **PR 14: opportunistic 14-day TTL eviction** | **this PR** |
| | PR 15: lifecycle telemetry | merged (#1723) |
| | PR 16: status-bar decision | resolved; no code PR |

### Why this PR

Inline-script environments are dependency/interpreter-keyed and
intentionally rebuilt instead of synchronized in place. Without
lifecycle cleanup, old cache keys accumulate whenever dependencies or
the selected Python change.

The design calls for a pipx-style 14-day TTL in addition to the explicit
clear command. Because TTL cleanup is silent, it must be more
conservative than user-confirmed cleanup and must never prevent the
requested environment from being created.

### What this PR does

- Attempts one TTL sweep per `InlineScriptEnvManager` session.
- Runs the sweep before the first environment creation/reuse reaches the
cache.
- Reads only valid `.meta.json` sidecars and selects entries whose
`lastUsedAt` is strictly older than 14 days.
- Reuses PR 13's physical-root, normal-directory, direct-child, and
entry-lock safety checks.
- Re-reads `lastUsedAt` under the entry lock before deletion.
- Preserves held, retained, malformed, redirected, unavailable, or
failed-to-delete entries.
- Treats all TTL failures as best-effort warnings and continues the
triggering creation.
- Invalidates persisted and warm script associations only for entries
confirmed deleted or definitively removed by another host.
- Removes evicted environments from the discovered collection and emits
the existing environment-change events.
- Prevents a concurrent discovery refresh from publishing a stale
snapshot after eviction.

### Eviction semantics

| State | Behavior |
|---|---|
| `lastUsedAt` older than 14 days | Lock, revalidate, and delete |
| Age exactly 14 days | Keep |
| Recent or future timestamp | Keep |
| Missing, invalid, unsupported, or unreadable sidecar | Keep |
| Timestamp becomes fresh before lock acquisition | Keep |
| Entry is actively locked | Keep |
| Entry has a retained cancellation lock | Keep; only explicit cleanup
may reclaim it |
| Entry deletion fails | Keep and continue creation |
| Another host already deleted the entry | Confirm with `lstat`, then
invalidate local associations |
| Association persistence fails after deletion | Keep in-memory state
consistent, log, and continue creation |
| Refresh started before deletion | Reject its stale collection snapshot
|
| Refresh starts during maintenance | Wait, then scan the post-eviction
cache |

### Concurrency and safety

- The once-per-session latch is set synchronously, so concurrent creates
share one sweep.
- The sweep uses the existing cache-maintenance then selection-queue
ordering.
- The active-create counter is incremented before the sweep, preserving
PR 13's clear-vs-create behavior.
- Each deletion is protected by the existing cross-host cache-entry
lock.
- Physical root and entry containment are revalidated after lock
acquisition.
- A cache mutation revision prevents stale discovery publication without
waiting on, or deadlocking with, a refresh already blocked by
maintenance.
- Only `ENOENT` counts as confirmed cross-host deletion; permission and
transient access errors preserve associations.

### Performance

- The cache is scanned once per extension-host session, not on every
lookup.
- Entries are inspected and deleted sequentially to avoid I/O spikes.
- Activation remains unchanged and is not blocked by TTL work.
- Environments with missing or uncertain metadata are not repeatedly
modified.

### User impact

The feature remains behind `python-envs.inlineScripts.enabled`. Users
without inline-script environments see no behavior change. Existing
environments are retained unless a valid sidecar proves they have not
been successfully reused for more than 14 days.

### Tests

- `npm run compile-tests`
- `npm run compile`
- `npm run lint`
- Complete unit suite: **1,928 passing**, 6 pending

Focused coverage includes:

- strict 14-day cutoff;
- once-per-session and failed-sweep behavior;
- under-lock freshness revalidation;
- held and retained lock preservation;
- deletion and persistence failure isolation;
- cross-host deletion confirmation;
- association and discovered-collection invalidation;
- refresh-versus-eviction publication races;
- unchanged explicit clear-cache behavior.

### Scope and follow-up

This PR does not add telemetry, activation-time cleanup, a cache-root
lock, project-setting cleanup, UX, or retry loops. Explicit clear-cache
behavior remains unchanged.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature-request Request for new features or functionality review-auto:approved Automated review: no blocking findings (approval posted).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants