Skip to content

[contributor setup 1] Read paid-API test credentials from test-only variables - #1869

Open
edwinyyyu wants to merge 1 commit into
MemMachine:mainfrom
edwinyyyu:test/explicit-paid-api-keys
Open

edwinyyyu wants to merge 1 commit into
MemMachine:mainfrom
edwinyyyu:test/explicit-paid-api-keys

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

The integration tests that call OpenAI, Amazon Bedrock, or Cohere read the SDKs' standard variables. So a developer whose shell already holds one of those keys for other work pays for every pytest -m integration run. These tests now read test-only variables, and spend only when a developer sets keys for the tests.

Stack

PR Change
#1869 (this PR) Paid-API integration tests read test-only variables
#1867 Fold the contributing guides into CONTRIBUTING.md; remove AGENTS.md and STYLE_GUIDE.md

This PR is its one commit, 862a083, on main. #1867 is stacked on it.

Description

Service Before Now
OpenAI OPENAI_API_KEY MEMMACHINE_TEST_OPENAI_API_KEY
Amazon Bedrock AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION or AWS_DEFAULT_REGION MEMMACHINE_TEST_AWS_ACCESS_KEY_ID, MEMMACHINE_TEST_AWS_SECRET_ACCESS_KEY, MEMMACHINE_TEST_AWS_SESSION_TOKEN, MEMMACHINE_TEST_AWS_REGION
Cohere COHERE_API_KEY MEMMACHINE_TEST_COHERE_API_KEY
  • Fixtures: openai_integration_config, bedrock_integration_config, and cohere_integration_config in the server tests' conftest.py read the new names and skip without them. The fixtures already pass the credentials to openai.AsyncOpenAI, boto3.client, and Cohere's AsyncClientV2 explicitly, so the SDKs never fall back to their standard variables. No other test reads a real key: the unit tests construct their clients with dummy keys. Ollama (OLLAMA_HOST) and Nebula (NEBULA_HOST) are local services and keep their variables.
  • CI: pytest-integration.yml sets the new names from the same secrets and variable, so CI runs these tests exactly when it did before. The repository has no OpenAI or AWS secrets, so they skip in CI both before and after this PR.
  • Docs: the Testing section of CONTRIBUTING-CORE.md lists the variables and warns that the tests call the real APIs, billed to those keys. [contributor setup 2] Fold the contributing guides into CONTRIBUTING.md and remove AGENTS.md and STYLE_GUIDE.md #1867 carries the same text into CONTRIBUTING.md.

Type of change

  • Project Maintenance (updates to build scripts, CI, etc., that do not affect the main project)

How Has This Been Tested?

  • Manual verification

  • A scratch test (not committed) requested the three fixtures:

    • With only the standard variables set, to dummy values, all three tests skipped, and each skip message names the new variables.
    • With the test-only variables set, all three fixtures returned those values.
    • Bedrock without MEMMACHINE_TEST_AWS_REGION skipped, even with AWS_REGION set.
  • ruff format --check, ruff check, and ty on packages/server as CI runs it all pass. The server and client unit tests collect (2389), and so do the server integration tests (1725).

  • No test was run against the real APIs.

Test Results: as listed above.

Checklist

  • I have signed the commit(s) within this pull request
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation

Further comments

Anyone who runs these tests locally with the standard variables needs to set the MEMMACHINE_TEST_* names instead.


🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.

The OpenAI, Amazon Bedrock, and Cohere integration fixtures read the
SDKs' standard variables (OPENAI_API_KEY, the AWS_* credentials and
region, and COHERE_API_KEY). A developer whose shell already holds one
of those keys for other work paid for every integration run.

The fixtures now read MEMMACHINE_TEST_OPENAI_API_KEY,
MEMMACHINE_TEST_AWS_ACCESS_KEY_ID, MEMMACHINE_TEST_AWS_SECRET_ACCESS_KEY,
MEMMACHINE_TEST_AWS_SESSION_TOKEN, MEMMACHINE_TEST_AWS_REGION, and
MEMMACHINE_TEST_COHERE_API_KEY, and skip without them. They already
pass the credentials to the clients explicitly, so the SDKs never fall
back to the standard variables. Bedrock's region no longer falls back to
AWS_DEFAULT_REGION.

The integration workflow sets the new names from the same secrets and
variable, so CI runs these tests exactly when it did before.
CONTRIBUTING-CORE.md lists the variables and warns that the tests are
billed to those keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Edwin Yu <edwinyyyu@gmail.com>

@marvinyu-memverge marvinyu-memverge left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 862a083.

Verified: the three integration-config fixtures are the only places the test tree reads a real key, and every client built from them gets its credentials passed explicitly (AsyncOpenAI(api_key=...), AsyncClientV2(api_key=...), both boto3 clients with key, secret, session token and region), so none of the SDKs can fall back to the standard variables; main/conftest.py passes the key into the server config explicitly too. The other test clients use dummy keys under mocks, and the AWS_*/OPENAI_API_KEY reads in the configuration tests are monkeypatched. The workflow maps the same secrets and variable onto the new names, so CI's coverage doesn't change. The red test-install jobs fail the same way on unrelated PRs today (Neo4j download 404 in memmachine-configure), so they aren't from this change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants