Skip to content

adds a vcsim based test suite and runs it in CI - #541

Merged
semx merged 2 commits into
developmentfrom
test/vcsim-harness
Sep 7, 2026
Merged

semx merged 2 commits into
developmentfrom
test/vcsim-harness

Conversation

@semx

@semx semx commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Following up on #474: the test setup, before the remaining PRs.

The VMware source could only be exercised against a live vCenter, which is why a bug like #476 reached users instead of a test run. vcsim, the simulator from the govmomi project, replays an inventory captured with govc object.save, so the real pyVmomi code path runs in CI with no vCenter and no NetBox involved.

What is in here

  • tests/fixtures/vcsim/ holds the two captures @ionoci contributed in Maintainer wanted - sunsetting this repository by 30.11.2026 #474 (a 6.7.0 and an 8.0.3 vCenter). The vcsim backed tests are parametrized over every archive in that directory, so dropping in a new capture covers it without touching test code. The README there documents how to take one.
  • tests/conftest.py starts one vcsim per capture on a free port, gives each test a clean in-memory NetBoxInventory, and writes the source a generated settings.ini. It also opens a second, independent pyVmomi connection to the same simulator, so the tests compare the sync against the SDK rather than against themselves.
  • tests/test_vmware_source.py asserts that every VM and host is synced; host serial, model and manufacturer match the hardware (allowing for manufacturer normalisation, Dell Inc. to Dell); VM cpu and memory match the hypervisor; interface MAC addresses match; guest IPs of VMs with running tools are synced and bound to an interface; two runs produce the same inventory; and a run against objects that already exist in NetBox creates nothing new.
  • .github/workflows/test.yml installs vcsim and runs pytest on pushes and pull requests to main and development.

Without vcsim installed the vcsim backed tests skip rather than fail, so pytest still works on a machine that only has the Python dependencies.

Verification

16 tests pass on Python 3.13 (the version the workflow uses) and on 3.14.

To show the suite actually catches regressions rather than just passing, I injected four faults into the source, one at a time, and ran it against the 6.7.0 capture:

injected fault result
memory reported in GB instead of MB 1 failed, 7 passed
guest IPs never collected (the shape of #476) 1 failed, 7 passed
a VM silently dropped during parsing 4 failed, 4 passed
MAC address dropped from interface data 3 failed, 5 passed

After reverting each one the suite was green again.

Notes

  • The captures are about 1.2 MB each and contain host names, VM names, MAC addresses and guest IPs of the source environments. They were posted publicly in Maintainer wanted - sunsetting this repository by 30.11.2026 #474, so this only moves them into the repository; say the word if you would rather keep them out of git and fetch them in CI instead.
  • While writing the MAC assertions I noticed that one VM in the 6.7.0 capture is parsed three times in a single run, which leaves duplicate unsaved NBMACAddress objects for its interfaces (82 objects for 64 distinct interface and address pairs). It does not survive a rerun against saved objects, so it looks like extra POSTs rather than lasting damage. The test compares address sets so it does not fail on it. I will open a separate issue with the details.

The VMware source could only be exercised against a live vCenter, which
is why bugs like #476 were found by users rather than by a test. vcsim,
the simulator from the govmomi project, replays an inventory captured
with `govc object.save`, so the real pyVmomi code path can run in CI.

Adds:

- tests/fixtures/vcsim: the two captures contributed in #474, a 6.7.0
  and an 8.0.3 vCenter. The tests are parametrized over every archive in
  that directory, so adding a capture needs no test changes.
- tests/conftest.py: starts one vcsim per capture on a free port, gives
  each test a clean in-memory NetBoxInventory, and feeds the source a
  generated settings.ini. A second connection through pyVmomi provides
  an independent view of the same simulator to compare against.
- tests/test_vmware_source.py: every VM and host is synced, host serial,
  model and manufacturer match the hardware, VM cpu and memory match the
  hypervisor, interface MAC addresses match, guest IPs of VMs with
  running tools are synced and bound to an interface, two runs produce
  the same inventory, and a run against objects that already exist in
  NetBox creates nothing new.
- .github/workflows/test.yml: installs vcsim and runs pytest on pushes
  and pull requests.

The suite needs no vCenter and no NetBox. Without vcsim installed the
vcsim backed tests skip rather than fail, so `pytest` still works on a
machine that only has the Python dependencies.

Verified on Python 3.13 (the CI version) and 3.14: 16 passed. To check
the tests actually catch regressions, four faults were injected into the
source one at a time: memory reported in GB, guest IPs never collected
(the shape of #476), a VM silently dropped, and the MAC address dropped
from interface data. Each one turned the suite red, and it went green
again after reverting.
@semx
semx requested a review from bb-Ricardo as a code owner September 2, 2026 21:29
The CI job runs `pytest`, which does not put the working directory on
sys.path the way `python -m pytest` does, so conftest failed to import
the module package. Set pythonpath in the pytest config instead, which
fixes it for a plain `pytest` invocation from a checkout as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant