fix: paginated list requests silently skipping rows - #543
Conversation
NetBox paginates list endpoints with limit/offset, and several models order by a key that is not unique. Rows that tie on the sort key have no defined position between two queries, so walking the pages can return one tied row twice and never return another. A row that is never returned looks absent to the sync, which then creates a second copy of an object that already exists. Each new duplicate adds another tie, so the problem feeds itself. ipam.ipaddress is one example: it orders by address, and the same address can legitimately be present more than once. Measured against a list endpoint holding 1222 rows tied on their sort key, read with a 200 row page size: 5 to 7 rows were missed on every full walk, reproducibly. List requests now ask for ordering=id. The primary key is unique, so the sort is total and every row is walked exactly once. A caller that needs a specific order still gets it, and single object requests are untouched. Verified against the NetBox REST API: 5 rows missed without the parameter, none with it. The tests drive the real request() pagination loop against an HTTP server that reproduces the tie instability. The ordering assertions cover every page, not only the first request, so an ordering dropped from the pagination link is caught.
|
We independently hit the same issue while validating netbox-sync v1.8.1 against an existing VMware / NetBox environment (~579 VMs, NetBox 4.2.x). We saw pagination behaving inconsistently during repeated runs and arrived at the same mitigation: adding explicit I included this as item 5 in #537 before seeing this PR, so it’s reassuring to see the same behaviour reproduced independently. Happy to provide sanitized logs or details from our testing if useful. |
|
Hi @jamespage001 |
|
Merged, thanks. Verified against the vcsim suite: the added |
3 participants
NetBox paginates list endpoints with limit/offset, and several models order by a key that is not unique. Rows that tie on the sort key have no defined position between two queries, so walking the pages can return one tied row twice and never return another.
verified against Netbox, as I was having missing/duplicated rows without it - my sample was around 1200 rows, on average 5-7 were having issue every time.
List requests now ask for ordering=id. The primary key is unique, so the sort is total and every row is walked exactly once.
Added same [tool.pytest.ini_options] to pyproject.toml as in #541 - so it can drop out.