feat: add logging conformance scenarios (capability, invalid level, threshold filtering) - #449
Conversation
…, and threshold filtering Addresses modelcontextprotocol#435 by adding three new conformance scenarios that test logging/setLevel behavior beyond the existing basic happy-path check: - logging-capability-advertisement: verifies server advertises logging in capabilities before emitting notifications - logging-invalid-level: verifies -32602 rejection for unrecognized severity levels - logging-threshold-filtering: verifies notifications respect the configured severity threshold (suppresses below, allows above) All scenarios target 2025-06-18 through 2025-11-25 (removedIn draft).
|
@joy7758 would appreciate your review on this one when you get a chance. |
|
Thanks for the follow-up. I reviewed current head I found four scope/evidence points worth resolving before these checks are treated as conformance results:
The repository contribution guide also asks for a passing real-SDK run and a specific failing case. The PR currently documents the in-repo everything-server path, so adding the pinned real-SDK output plus a negative fixture/result would complete the evidence loop. The |
|
Thanks @joy7758, all four points are valid. I'll push a revision that:
|
- Downgrade threshold filtering from conformance assertions to interoperability observations (spec has no MUST/SHOULD for receiver-side filtering) - Make capability check conditional: SKIPPED when server does not evidence logging support (spec requires capability only for servers that emit notifications/message) - Use untestableCheck() for zero-notification case per repo convention (issue modelcontextprotocol#248) instead of SKIPPED - Change below-threshold violation from FAILURE to WARNING since there is no normative backing
When capabilities.logging is absent, open a probe connection and call test_tool_with_logging to check whether the server actually emits log notifications. If notifications are observed without capability advertised, report WARNING (MUST violation). If none observed, SKIPPED is correct.
|
All four points from @joy7758 addressed and pushed. Threshold filtering is non-scoring now, capability check probes for logging support first, zero-notification uses untestableCheck(), and wire evidence added for invalid level. Maintainer review welcome when someone has time. |
Summary
Adds three new conformance scenarios for the
logging/setLevelendpoint (issue #435):logging-capability-advertisement— verifies the server advertisescapabilities.loggingin its initialize responselogging-invalid-level— verifies the server rejects unrecognized severity levels (e.g. "banana") with-32602(Invalid Params)logging-threshold-filtering— verifies thatnotifications/messagerespects the configured severity threshold (suppresses below, allows above)Scenarios 2 and 3 are registered in
pendingClientScenariosListbecause the everything-server does not implement invalid-level rejection or threshold-based filtering. They can be run targeted against a compliant fixture server vianpm start -- server --scenario logging-invalid-level --url <fixture>.Scenario 1 is in the active list and passes against the everything-server.
Spec references
Test plan
npm run buildpasseslogging-capability-advertisementpasses against everything-serverlogging-invalid-levelcorrectly skipped (pending) in default suitelogging-threshold-filteringcorrectly skipped (pending) in default suiteCloses #435