Repository navigation
build!: publish the distribution as usersim - #37
Merged
Merged
Conversation
…call The bundled configs set no timeout on either alias, so their calls used the 60-second Data Designer default, which slow endpoints exceed. Both now get 300 seconds, like the model under test, in all three bundled configs. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…again A row whose judge calls all failed, or returned output that did not parse, had no verdict, and `usersim eval` still exited 0. It now exits 1 when no row got a verdict and otherwise reports how many rows lack one. A re-run judges such rows again instead of skipping them as done. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…errors The preview caveat went into `error`, which consumers read as a failed scorer, so a scored row lost its scores; the caveat already lives in `preview_only`, `preview_topics` and `ground_truth_source`. A row without move/Guard ground truth now uses the `no ... scorer skipped` form, so it reads as not applicable. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
git ran in the working directory, so running inside another repository recorded that commit. The sha now comes from the checkout the package was loaded from, when git tracks the package there, or from the commit a git install recorded (PEP 610 direct_url.json). An install from a package index records none; USERSIM_CODE_SHA still overrides. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The pre-flight required a bank directory named after each run locale, so simulate refused every India variant for the locale-scoped probes, which run on the en_IN bank. It now resolves each locale as the probes and scorers do. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The distribution now has the name of the import package and the command. Install specs that name nemo-usersim resolve only against earlier commits. After pulling, run `make clean && make install-dev`: `make clean` also removes egg-info metadata, which can still carry the old name, and `make install-dev` reinstalls the project, restoring the `usersim` command that uninstalling nemo-usersim removes. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
license is a PEP 639 expression, which setuptools reads from 77 on; older releases fail to build the sdist. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
`usersim.__version__`, `usersim --version` and run manifests read the version of the installed usersim distribution, so pyproject.toml is the only place it is written. The distribution named usersim wins over stale metadata from another name. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
setuptools shipped part of the test suite without its conftest.py files; MANIFEST.in prunes tests/. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
`make dist` builds the sdist, then the wheel from it, into a fresh dist/. `make check-dist` replaces `make check-wheel`: it runs twine check --strict, fails if the wheel holds a file the sdist does not, keeps the asset and embeddings.parquet checks, and runs the installed wheel from an empty directory. `make clean` also removes build/ and dist/. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Relative links break in the PyPI project description. The link check resolves links into the main branch against the checkout, so they stay checked. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Setup installs from PyPI into a virtual environment with a version floor and verifies the install offline with smoke and a dry run. Working from a clone points to CONTRIBUTING.md. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The population is sampled from persona datasets, of which census-grounded ones are an example; the README, docs/methodology.md and the CLI help now say so. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
Publishes the distribution as
usersimand readies the package for PyPI, after five fixes.Fixes
usersim evalexits 1 when no row got a judge verdict and otherwise reports how many rows lack one; a re-run judges such rows again.code_shacomes from User Sim's own checkout or git install, never from the working directory's repository.en_INbank.Packaging and docs
usersim. Breaking: install specs namingnemo-usersimresolve only against earlier commits.pyproject.toml(now0.2.0.dev0); tests stay out of the sdist.make distbuilds the sdist and the wheel from it.make check-distreplacesmake check-wheel, addingtwine check --strictand a check that every wheel file came from the sdist.docs/methodology.mdand the CLI help describe the population as sampled from persona datasets, with census-grounded ones as an example.After pulling:
make clean && make install-dev. Removing the old distribution also removes theusersimcommand it shared with the new one;install-devnow reinstalls it.How it was verified
make check-all,make testandmake smokepass after every commit, and the pre-commit hooks pass on each commit's files. Each fix has a test that fails without it.make check-dist(twine--strictPASSED for both archives),make check-extensions,make check-dependency-licenses,make check-license-headers,make check-doc-links,make check-notebooks-clean.make health-checkpassed (simulate, eval and report, with the judge returning a verdict), and ahealth_disclosurerun with--scorers health_disclosure_concealmentscored its row with no scorer error.Results impact
health_disclosureprobes): a row whose audit fell back to committed intent for some topics was recorded as a scorer error, so its scores dropped out; it now counts, still marked preview-only. A row without move/Guard ground truth reads as not applicable instead of as an error.code_sha; it used to record the commit of whatever git repository it was run from.