Repository navigation
test: fuzz api responses against openapi contracts - #228
Open
aditeyabaral wants to merge 3 commits into
Open
aditeyabaral wants to merge 3 commits into
aditeyabaral wants to merge 3 commits into
Conversation
Generate deterministic offline requests through the real ASGI app and a mocked loopback server, validating status codes, content types, response schemas and server errors. Run the URL and CLI paths through a shared pre-commit and CI runner with owned server cleanup and JUnit, JSON and TraceCov reports. Validate existing credential-driven integration responses against the same contract without adding PESU logins. Normalize unreadable request bodies to the standard 400 error shape and correct metrics JSON and bodyless redirect documentation. Validation: pre-commit passed; 330 tests and 4 fuzz subtests passed with real credentials and 100% application coverage. URL fuzz tests passed and CLI exercised all four operations. Docker build and Swagger/health smoke checks passed.
Raise the minor version once for the OpenAPI fuzz suite and its contract fixes, keeping the application metadata and lockfile aligned.
Use TEST_EMAIL and TEST_PASSWORD loaded from .env for a bounded Schemathesis authentication test against the real PESU Academy backend. Generate six profile and field combinations serially, without mocking upstream calls, and discover the test through the canonical pre-commit runner. Keep actual credentials outside generated cases and validate responses using a separate schema with sanitized, context-free failure messages. Add offline regressions for safe pytest reporting and document the real-account command alongside the arbitrary-input offline suite. Verification: pre-commit passed with 335 tests, 5 fuzz subtests and 100% application coverage. All 12 credential-required tests ran locally, including the generated live test. URL pytest and CLI checks also passed; independent correctness and privacy reviews approved.
Comment on lines
+240
to
241
| live_contract(response) | ||
| assert response.status_code in (200, 401, 500) |
4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 Description
Add deterministic OpenAPI contract fuzzing to the offline suite, plus URL-based pytest and CLI runs against an isolated local server. All four documented operations are checked for server errors, undocumented statuses, content-type mismatches, and response schema violations.
An explicit pre-commit hook and CI job run the URL/CLI checks, own server startup and cleanup, and publish JUnit, JSON, and informational TraceCov HTML reports. A credential-dependent Schemathesis test generates six positive profile/field combinations and calls the real PESU Academy backend using TEST_EMAIL and TEST_PASSWORD from
.env. Existing authentication integration responses are also checked against the contract. The live generator uses no PESU mocks and keeps credentials out of generated cases and failure reproductions.Fuzzing found malformed non-UTF-8 JSON returning FastAPI's
detailbody. It now returns the documented 400 error envelope and records aRequestBodyParseError. Metrics JSON now referencesMetricsModel;/readmecorrectly documents a bodyless 308 and its Location header.Closes #218. Credential length limits remain in #192.
🧱 Type of Change
🧪 How Has This Been Tested?
.envcredentialsLinux, Python 3.14 via uv:
/healthreturned 200.✅ Checklist
🛠️ Affected API Behaviour
app/app.py: unreadable bodies use the standard 400 responseapp/docs/: metrics schema, redirect headers, and parse-error documentationpyproject.toml/uv.lock: dev tools and minor version📸 API Demo
A JSON request body containing an invalid UTF-8 byte now receives status 400 with:
{"status": false, "message": "Could not parse request body.", "timestamp": "<ISO timestamp>"}🧠 Additional Notes
The arbitrary-input gate stays offline as required by the issue. Real-account Schemathesis and ordinary contract checks run serially in the integration suite and use sanitized, title-only failure output with exception context suppressed. Live generation disables schema examples, boundary probes, shrinking, and persistent replay; its budget is six generated examples, while arbitrary offline fuzzing retains the larger 200/500-example budgets. Fork CI skips credential-required tests because it has no secrets; they were verified locally.
Unsanitized reproductions are confined to the synthetic loopback harness, which rejects other target URLs/tokens and blocks PESU HTTP requests. The CLI may emit authentication/validation warnings for generated inputs rejected by the mocked backend; these are not conformance failures.
TraceCov is development-only and uses the TraceCov Community License. Its schema-coverage report is informational and does not replace the existing application-code coverage gate.