scratch: verify the pull request report comment - #1
Closed
adrpo wants to merge 459 commits into
Closed
Conversation
* Improve test.py doc in README * Fixing HTML issues - `<html>` missing `DOCTYPE`. - Replace `<hr />` with `<hr>`. - Replace `<br />` with `<br>` - Element p not allowed as child of element span - Fix missing </tr>
- Error was introduced by 74f1379
- GitHub workflow to run scripts with sanityCheck.json - Test on Ubuntu with Python 3.10 and omc nightly/stable - Resulting HTML and sqlite are archived
* [FMI] Let --fmisimulator be given more than once First step of OpenModelica#78: the FMI jobs come in pairs - v1.26-fmi and v1.26-fmi-fmpy, v1.27-fmi and v1.27-fmi-fmpy, master-fmi and master-fmi-fmpy - and both jobs of a pair build the same Model Exchange FMUs, some 19500 of them, only to simulate them with a different tool. Measured on ExternData here: 164 seconds building the FMUs and 0.6 simulating them with OMSimulator, then another 167 seconds building the very same FMUs to simulate them for 3 seconds with FMPy. --fmisimulator can now be repeated, as "name=command" or as the bare command it has always been, so that one job can simulate every FMU with several tools after building it once. The FMU can only be shared when everything that goes into it is the same - the same compiler, the same library and the same build flags - so what a merged job varies is the simulator and nothing else. That is why the three cs-fmu-cvode jobs stay as they are: their FMUs are Co-Simulation ones built with --fmiFlags=s:cvode, not the same artifact at all. --branch now names the job rather than the table, and every simulator derives its own from it: OMSimulator keeps the plain v1.27-fmi it has always had, FMPy fills v1.27-fmi-fmpy, PyFMI would fill v1.27-fmi-pyfmi. The mapping is keyed by the simulator and not by the order it was given in, so that a job asked for FMPy alone still writes to v1.27-fmi-fmpy instead of taking over v1.27-fmi. No report and no history has to move. A job that passes a single simulator behaves as it always did, and the version of every tool is now read rather than only the first one's. This only adds the option and the naming rules; testmodel.py still simulates with one tool, which the next commit changes. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> * [FMI] Describe the FMI simulators in a configuration file Adding a simulator to the testing was a case in testmodel.py, one in the version check and one in the choice of result file. It is now an entry in configs/fmi-simulators.json and nothing else: "fmusim": { "untested": true, "resultExtension": "csv", "versionArgument": "--version", "stepSizeArgument": " --output-interval {stepSize:g}", "arguments": "--interface-type ModelExchange --output-file {result} ..." } An entry says how the tool is invoked, what it writes, how it prints its version, how it spells a step size and which table it fills. The arguments are a template formatted with the model and the experiment, so a tool with different flags needs no code, and a Python package without a command line needs a driver script that the entry then points "command" at. stepSizeArgument is a key of its own because the tools disagree about an experiment that has no step size: OMSimulator leaves the flag out, while FMPy passes --output-interval 0 all the same. A tool of the second kind writes {stepSize:g} into its arguments and needs no stepSizeArgument at all. The entry for fmusim is there because it is the next one we expect to want. It is marked untested, and asking for a simulator nobody has run yet says so rather than failing every model with a puzzling error. Checked against the format strings testmodel.py uses today: both tools are given exactly the command line they were given before, with a step size and without one. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> * [FMI] Build each FMU once and simulate it with every tool asked for The heart of OpenModelica#78. testmodel.py builds the FMU as before and then runs every simulator the job was given on it, and test.py writes one row per simulator into the branch of that simulator. A job given OMSimulator and FMPy on --branch=v1.27-fmi therefore fills v1.27-fmi and v1.27-fmi-fmpy from a single build, which is what the two jobs of a pair used to do twice. Measured on ExternData, 12 models of which 9 build an FMU: separate 164s building + 0.60s simulating OMSimulator 167s building + 3.09s simulating FMPy merged 178s building + 0.74s + 2.58s both so the FMUs are built once instead of twice and the run costs about half. The results are the same either way: every model's phase, verification time, number of variables compared and number that differ is identical to what the two separate runs produced, for both tables. The verification of a result against its reference file became a function that returns rather than one that ends the run, so that the next simulator of the same FMU can be verified too, and omc is now quit once all of them are done rather than after the first. A simulator that fails or times out records its own failure and leaves the others alone; the first one keeps ending the model as it always has, so put the tool you trust most first. A model that fails before its FMU exists reports the phase the build stopped at for every simulator, rather than pretending each of them failed. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> * [FMI] Publish the results of every simulator to its own branch directory The .err and .sim files of a run are collected per library and rsynced to a directory named after the branch under libraries.openmodelica.org/branches. A job that runs several simulators has one set of results per simulator, so it now does that once per branch: v1.27-fmi and v1.27-fmi-fmpy each get their own directory, their own library page and their own files, as they did when two jobs produced them. A branch directory looks exactly as it did before, down to the file names. A simulator writes <model>_<simulator>.sim in the workspace so that the tools of one job do not overwrite each other, but each branch is published from a directory of its own where those appear under the plain <model>.sim that has always been there. The links are hard, not symbolic, so rsync sees ordinary files and nothing is copied twice on disk; where that is not possible, across file systems, the file is copied. The .err is written by the build and is therefore the same for every simulator of a model, and each branch gets a copy rather than a link into another branch: a job asked for FMPy alone publishes no v1.27-fmi to link into, and a branch that is re-run must not break the pages of another. It costs about 66 MB of 3.4 kB files against the 58 GB such a directory already holds. The report of a library is generated from the results of the simulator whose branch it belongs to, so its phases, times and links describe that simulator and not the first one. The two ways of publishing disagreed about --output and still do: the rsync path is given the directory of the branch, the --noSync path the directory the branches live in. Both now derive the directory of each simulator the way they already derived the one of the job. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> * [FMI] Run each pair of FMI jobs as one job The last part of OpenModelica#78. The three pairs of FMI jobs - v1.26-fmi and v1.26-fmi-fmpy, v1.27-fmi and v1.27-fmi-fmpy, master-fmi and master-fmi-fmpy - are one job each now. Six stages become three, and the FMUs of a release are built once instead of twice. The parameters keep their meaning: fmi_v1_27 asks for OMSimulator, fmpy_fmi_v1_27 for FMPy, and ticking both runs one job that builds every FMU once and simulates it with both, filling v1.27-fmi and v1.27-fmi-fmpy as the two jobs did. Ticking one runs that tool alone and fills only its table, and its results and files still go where they always went: a job given FMPy alone stores them in v1.27-fmi-fmpy and publishes them to branches/v1.27-fmi-fmpy, not to the branch it was started with. Which simulators a job runs is now something it is told rather than something guessed from its name, but a job that says nothing still gets the tool its name implies, so the cs-fmu-cvode jobs and every job that does not test FMI are untouched. OMSimulator is only cloned and built when a job actually asks for it. Checked here on ExternData, for all three ways of asking: OMSimulator alone fills v1.27-fmi, FMPy alone fills v1.27-fmi-fmpy, both fill the two of them from a single build, and the results of a merged run are the same as those of the two separate runs. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> * [FMI] Document testing one FMU with several simulators Two sections in the README: how to run a job with more than one FMI simulator, what each of them fills and where it is published, and what may differ between results that share an FMU; and how to add a simulator, which is an entry in configs/fmi-simulators.json and no change to any script. The description of every key of an entry is there, including why a step size flag may have to disappear rather than be passed empty, and what a Python package without a command line needs. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> --------- Signed-off-by: Adrian Pop <adrian.pop@liu.se>
* Add a sqlite3 to PostgreSQL migration and document the database Towards a network based SQL database for the library testing, issue OpenModelica#295. sqlite2postgres.py copies the per-machine sqlite3 databases into PostgreSQL. The PostgreSQL layout is a 1:1 mirror of the sqlite3 one - one table per branch with the same column names, only the types adapted - so that the test scripts can push new results to the network database with the statements they already use. The two machines are merged into the same tables, ripper1 first and ripper2 with --skip-existing, so that the row already in the database wins whenever a key collides. The keys the migration adds - (date, libname, model) for a branch table - are what makes a shared database possible at all: they hold across all 160 million migrated rows, and they stop a run that is pushed twice from storing its results twice. The script needs nothing but the standard library and the psql client, streams the rows through COPY in batches, and commits each batch together with its progress row in [migration_progress], so a migration that is interrupted continues where it stopped instead of duplicating rows. --index creates the keys and the index the reports need, --verify compares the row counts of both sides. --catch-up copies the runs a machine wrote after the migration read its database, which a test still using its sqlite3 file keeps doing until the jobs are switched over: a wasm-jit run of 19527 models finished right after the migration and was missing from the shared database until it was caught up. It picks the runs by date rather than by continuing from the rowid it stopped at, because VACUUM renumbers the rowids of these tables and clean-empty-omcversion-dates.py runs one after every test, and because two machines write into the same table for the branches they both test. Reading a whole database costs two to three minutes, and the keys make repeating it harmless. COPY is used in its text format rather than CSV because an empty CSV field reads back as NULL, which would silently turn the empty libversion strings of the old data into NULLs. For the same reason only the key columns are NOT NULL: the sqlite3 tables declare every column NOT NULL, but tables created by an older test.py kept their laxer declaration and the historical rows do not hold up. doc/README.md documents what is actually in the database: the per-branch tables, omcversion and libversion, what every column means, how finalphase maps to the phase names, how a test run writes its results, and how to run the migration. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> * Let the testing use a shared PostgreSQL database instead of sqlite3 files Second part of issue OpenModelica#295: the test machines can now write their results to one network database and coordinate through it, instead of each copying a sqlite3 file in, testing, and copying it back - the step that made two machines overwrite each other depending on which one finished last. resultsdb.py holds both backends behind one interface. The scripts write the same statements for either, with "?" as the placeholder, and ask the connection where the dialects genuinely differ: quoting a branch name, testing whether a table exists, concatenating a group, counting a condition, matching a branch without case, skipping a row that is already there. Every script takes --db, which defaults to the LIBTEST_DB environment variable, so Jenkins sets the database once for the whole pipeline rather than on forty invocations. A machine claims a job in [job_claim] before testing a library, keyed by exactly the question the run already asks: which library, in which version, against which compiler and configuration. Only one machine can win the claim, and the others skip that library and move on. The winner refreshes a heartbeat every minute and marks the claim done when the results are written, so a machine that dies parks its jobs for STALE_CLAIM_MINUTES rather than forever. A local sqlite3 file has a single writer, so there claim() always says yes. Two PostgreSQL specifics were needed for the reports: GROUP_CONCAT relies on sqlite keeping the order of the subquery that feeds it, so the ported query orders inside the aggregate, and COUNT(x or null) becomes COUNT(*) FILTER. Running the report queries against ripper1's sqlite file and against the migrated database gives the same rows, down to the phase counts, the per-phase sums and the regression rows: checked on master, whose table holds 43 million of them, and on heavy_tests, conversion and basemodelica_jl_master. The two only render the doubles with a different number of digits, and the report parses them back with float(). The Jenkinsfile gets a "postgres" parameter, on by default, which selects the database and drops the download and the publishing of sqlite3.db. Unticking it restores the old behaviour unchanged. The password comes from a Jenkins secret file credential, omdb-pgpass, bound to PGPASSFILE, so it never reaches a command line or the build log. --- Generated by Claude Code. Signed-off-by: Adrian Pop <adrian.pop@liu.se> --------- Signed-off-by: Adrian Pop <adrian.pop@liu.se>
…penModelica#299) Every build of the library testing has failed since OpenModelica#296 with org.jenkinsci.plugins.workflow.steps.MissingContextVariableException: Required context class hudson.FilePath is missing before a single stage ran. The password of the results database is a secret file, and Jenkins writes such a file into the workspace of a node; OpenModelica#296 bound it in the environment of the pipeline, which has agent none and therefore no workspace at all. It is now bound where a node exists: in the environment of the two report stages, beside the IDA_EMAIL credential they already bind that way, and around the test run and the cleanup in runRegressiontest, the way withSccache already binds its key. LIBTEST_DB stays in the environment of the pipeline; it is a string and needs nothing. Nothing about how the scripts read the password changes: libpq still finds it through PGPASSFILE. --- Generated by Claude Code.
Python 3.13 deprecates giving re.sub its count positionally, so every library
test run printed
test.py:1046: DeprecationWarning: 'count' is passed as positional argument
return (re.sub( ".*model name.*:", "", line,1)).strip()
once per job while reading the processor name out of /proc/cpuinfo. The
substitution is unchanged; only how the argument is spelled.
A run of configs/sanityCheck.json now goes through without a single
DeprecationWarning.
---
Generated by Claude Code.
Two machines testing the same library, with the same compiler and the same library version, claimed it both and tested it both. The claim in OpenModelica#295 is keyed by (branch, libname, libversion, omcversion, confighash), and their confighashes differed, so as far as the database was concerned they were testing different things. The hash is taken over the configuration of the library, and the configuration by then holds the reference files as an absolute path: destinationReal = os.path.realpath(destination) c["referenceFiles"] = destinationReal so two machines whose workspaces sit in different places, or whose paths pass through different symbolic links, hash the same configuration differently. That never mattered while each machine had its own sqlite3 file and the hash only had to be stable on the machine that made it. Sharing one database made it matter, and it defeats both the claim and the check for results we already have. The path says nothing that the hash needs: the contents of the reference files are hashed right after it, and the same files in another directory are the same test. It is replaced by a constant. Checked with the same configuration under two different reference directories: the hashes were 3181697168 and 2732211752 before, are equal after, and still differ when a reference file itself changes. Every confighash changes value once, so the first run after this treats every library as untested and tests it. From then on the machines agree. --- Generated by Claude Code.
A run opens its connection, claims its libraries, and then tests for hours before writing anything: the results of every model go into one transaction at the very end, after the reports have been generated. The connection spends that whole time idle, which is exactly the connection a firewall, a NAT or a restarted server drops. When that happened the first INSERT of the run threw OperationalError, test.py died, and a night of testing was lost - and nothing looked wrong until then, because the heartbeat that holds the claims opens a connection of its own every minute. The connection now asks for TCP keepalives, so an idle one is kept alive and a dead one is noticed. If it is lost anyway, the statement that discovers it reconnects, replays the statements the transaction had so far and carries on. Replaying is safe because everything a run writes is an insert guarded by a unique key: it can only produce the rows that were lost, never a duplicate. Queries are not remembered, so the tens of thousands a run makes do not fill the buffer. Checked against the real thing: with a run's transaction half written, killing its backend with pg_terminate_backend loses nothing - the run reconnects, replays its 250 statements, and all 500 rows are in the database, with the connection usable afterwards. This makes losing the connection survivable; it does not make a run's results arrive any earlier. Writing them per library, as each finishes, would also stop a crashed run from losing the libraries it had already tested, and is the better fix - it needs the results to be gathered per library rather than at the end, so it is worth doing on its own. --- Generated by Claude Code.
…#304) The image the report stages run in was still built on build-deps:v1.16.3, which is the dependency image of OpenModelica 1.16. It now uses ubuntu-22.04, the one the OpenModelica repository builds against in .CI/common.groovy and every cmake Jenkinsfile, so the reports run on the same Ubuntu and the same Python as everything else. That also gets psycopg2-binary a wheel to install rather than a source build, which needs pg_config from libpq-dev; the results database needs psycopg2 since OpenModelica#296. Two things the change makes explicit rather than assume of the base image: apt is told -y, because an image that does not assume it turns the build into a prompt that never gets answered, and python3-pip is installed, because the line after it needs pip and the base may not carry it. --- Generated by Claude Code.
Two machines testing master both claimed five of the same libraries and tested them twice. OpenModelica#303 fixed the confighash being different for no reason; these five differ for a real one. They are the libraries whose reference files come from MAP-LIB_ReferenceResults/v4.1.0, which a maintenance job updates from time to time, and the two machines had fetched them either side of such an update. The libraries pinned to v4.0.0 and v3.2.3, which do not move, agreed. So the configuration really did differ, and the claim was keyed by it: PRIMARY KEY (branch, libname, libversion, omcversion, confighash) which is finer than the question a claim answers. Whether this machine may test this library of this branch has nothing to do with which reference files it has; that belongs to the two questions that are keyed by the configuration and stay as they are - whether results for an exact configuration already exist, and whether two runs' rows can collide. The claim is now one row per (branch, libname). The versions are kept as columns, so a run still says who is testing what and since when. An existing job_claim is narrowed on connect, keeping the freshest claim per library so that a run in progress does not lose its claims and let a second machine in. Checked against a database seeded the way production looks: the live claim survives the migration, the key becomes (branch, libname), a second machine with a different confighash is refused and told who holds it, a free library is granted and then refused to the first machine, and after release the other machine may take it. Plus a sqlite3 run through the CI configuration and a PostgreSQL run with two FMI simulators. --- Generated by Claude Code.
The results of a job went into a per-machine sqlite3 file that had to be downloaded before the run and published back afterwards, and since OpenModelica#296 they go into the shared database instead. Which of the two was a tick box on the job, defaulting to the shared one. Nobody should be able to send a day of testing to a file nobody reads by unticking a box, and keeping the old path around only leaves two ways of doing the same thing for someone to wonder about later. Both are gone: - the "postgres" parameter, and the branches it guarded; - copying ~/TEST_LIBS_BACKUP/<machine>-sqlite3.db into the workspace before a run, copying it back after, and the sshPublisher that uploaded it; - fetching a machine's sqlite3.db in the two report stages, which now read the shared database directly; - keeping a week of dated backups of those files on every node; - dbPrefix and sshConfig, which existed only to name and publish that file, from runRegressiontest and its twenty-three callers. The scripts themselves are untouched and still write a sqlite3 file by default: that is what a developer running test.py gets, and what the GitHub checks use. It is the pipeline that no longer has a second way of doing this. --- Generated by Claude Code.
A library whose reference files are already prepared by another library took a
`continue` that skipped the rest of the configuration loop - and the only thing
after it is the line that turns FMI on:
if allTestsFmi:
c["fmi"] = "2.0"
So four libraries of configs/conf.json have been running as ordinary native
models inside the FMI jobs: PowerGrids_symb_jac, PowerGrids_dev, ClaRa_dev and
ScalableTestSuite_noopt, each of them the second library to want a reference
directory that PowerGrids, ClaRa and ScalableTestSuite had already cloned. They
verify identically in master and in master-fmi - 66, 19, 72 and 244 models -
while every library that prepares its own directory drops as an FMI job should
(242 to 176 for ScalableTestSuite, 65 to 31 for PowerGrids).
The reuse also looked the directory up under a key it had not computed yet, so
it took whatever `destination` the previous library left behind. Both are fixed
by computing the normalised destination first and putting the preparation in an
else branch, which leaves nothing after the loop's body unreachable.
Since OpenModelica#297 this crashed the run rather than quietly mistesting it. Such a model
has no per-simulator results, so the second simulator's branch reported the
phase the *first* simulator reached - 7, verified - with no comparison to go
with it, and building the report died on it after seven hours:
("%s (%d verified)" % (timeSeconds(diff.get("time")), ...)) if s[3]["phase"]>=7
AttributeError: 'NoneType' object has no attribute 'get'
A simulator that never ran a model now reports the phase the shared build
reached and never a phase another simulator went on to reach with the same FMU,
in the report and in the database row alike; and a model with nothing to compare
renders as an empty cell whatever phase it claims, so a mismatch can never again
throw away a finished run.
Fixes the master-fmi failure of build 11369.
A model whose command runs out of time reports nothing for the phase it was in,
so the time disappears from every phase column and only `exectime` still counts
it. In the master run of 2026-08-12 that is 15 models and 1.34 hours, including
three BuildingSystems models that spend the full 660 second budget in
translateModel and are recorded as having spent three seconds parsing and
nothing else.
The caller does try to account for it:
except TimeoutError as e:
execstat["frontend"]=monotonic()-start
but that handler never runs. `sendExpressionTimeout` kills the command and calls
`writeResultAndExit` itself, from inside, so the process is gone before the
exception is raised - the results written are whatever `execstat` held when the
command started.
Recording it where the results are written covers every way out instead of the
one path that remembered to. A phase now says when it started, `writeResult`
charges it if it never reported a time of its own, and a phase that ends
normally still reports exactly what it did before.
Which phase a killed command belongs to cannot be known - the timers of omc are
gone with it - so it is charged to what the command itself is: building an FMU
to the build, simulating to the simulation, translating to the front end, which
is also the phase such a model is reported as having failed in.
Verified on Modelica.Fluid.Examples.HeatExchanger.HeatExchangerSimulation with
ulimitOmc=1, which the watchdog kills:
before {'parsing': 1.73}
after {'parsing': 1.65, 'frontend': 3.01}
and on Modelica.Blocks.Examples.PID_Controller, which succeeds, where every
phase is unchanged.
Part of OpenModelica#308.
An FMI job that runs several simulators over one FMU (OpenModelica#297, OpenModelica#309) gives the first of them a veto over the rest. When it fails, the TimeoutError handler calls writeResultAndExit, the process is gone, and the loop that runs fmisimulators[1:] never starts -- but a row is still written for every simulator, recording the phase the first one stopped at. The other tools are marked as having failed at a model they were never given. checkOutputTimeout raises TimeoutError for a command that exits non-zero as well as for one that runs out of time, so this covers an ordinary failure and not just a slow one. In the master-fmi run of 2026-08-12 this cost 433 models. They had never simulated under OMSimulator -- phase 5 in every master-fmi run back to 2026-08-08 -- but FMPy ran them, and the previous master-fmi-fmpy run has them at phase 6 or 7. In the first combined run they are phase 5 in both tables, with FMPy's simulation time recorded as exactly 0: OMSimulator failed in under a second for 344 of them, and FMPy never started. They show up as 433 of the 653 regressions on the comparison page, next to the 213 that PR OpenModelica#309 really did find. It is also why that page reports OMSimulator's version, and its JSON parse warning, in place of "FMPy version 0.3.29": FMPy never ran to say what it was. The first simulator now records its failure and falls through to the others, which is what every simulator after it already did. Nothing changes when only one simulator is configured: that case still ends the model where it always has, so the non-FMI paths and the single-simulator FMI jobs are untouched. Verified on MyLibrary.Blocks.Examples.PID_Controller with the first simulator replaced by /bin/false: before phase 5, "simulators": {} after phase 5, "simulators": {"fmpy": {"sim": 0.64, "phase": 7}} and, unchanged in both: one failing simulator phase 5, os._exit as before both simulators working OMSimulator phase 7, fmpy phase 7 Claude-Session: https://claude.ai/code/session_01MYvMaAotMy425H7nvKzWX7 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…a#312) - Merge ryzen-5950x-1 and ryzen-5950x-2 report stages into a single stage (Postgres makes this fast) - Add wasm-jit to all-reports.py/all-plots.py so history reports are generated for it - Include wasm-jit in overview-combined.html and overview-special-jobs.html alongside master and newInst-newBackend Co-authored-by: Qwen 27B <qwen@alibaba.com>
The all-reports script uses omcversion from the database to build git range specs like 'v1..v2' for commit log lookups. Rust builds append '-rust' to the version (e.g. 'v1.28.0-dev-308-g1ac9d5ad20-rust') which git cannot resolve as a tag or ref. Strip the trailing '-rust' before the version is used in git commands so the underlying tag (e.g. 'v1.28.0-dev-308-g1ac9d5ad20') is found. Co-authored-by: Qwen3.6-27B-TQ <qwen@alibaba.com>
test.py caps the address space of one test at a time, so nothing bounds the sum of the tests it runs in parallel. That is worst for wasm-jit, whose tests are allowed 16 GB of address space each because the JIT reserves ~4 GB of it per wasm memory. Cap the container the job runs in with `docker run --memory=N --memory-swap=N`, N = 85% of the node's RAM, since these jobs are pinned to the high-memory nodes. A cgroup limit charges memory in use rather than address space, covers every process test.py spawns, and on a breach the kernel kills the greediest omc - so a model can now fail because of a model tested in parallel with it, which the log makes visible: the limit is printed before the run and the peak plus the OOM kill count after it. Only the targets that build their own image (wasm-jit today) get this; the jobs running directly on a node are unchanged. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Its IncludeDirectory annotations name modelica://BuildSysPro/Resources/C-Sources, which the library does not have — the IBPSA C sources are in IBPSA/Resources/C-Sources, and in a stale copy under Resources/IBPSA/C-Sources in 3.5.0 and older. omc therefore emitted no -I for them and every model reaching one failed to compile. Add both directories with setCFlags, plus the -include flags a few of those sources need: they call str*/malloc/ModelicaError without including the header, and only compile for the C target because the surrounding translation unit happens to have it. With --nobuildmodel, translating and building share one simulate() call and only the record's messages says either failed. Tell them apart by the translation clocks, so a wasm-jit build failure — the JIT compile, or building the model's Include C sources — is reported as Compile rather than as SimCode or Simulate. Assisted-by: Claude Opus 5
…#320) The report's "System info" line was CPU model, RAM and distro, which is identical on ryzen-5950x-1 and ryzen-5950x-2, and nothing anywhere said which node a run came from. job_claim.host is the only place the hostname was written, and that is a live claim table - the next run overwrites it, so the question "which machine produced this result" was unanswerable a day later. Put the hostname in front of the report's system info, and store the hostname and the whole system info string in [libversion], which is already keyed (date, branch, libname) - one row per library per run, which is the right granularity now that libraries of the same branch can be claimed by different machines. Both schema changes are ADD COLUMN only: the rows already in the databases keep their results and read back NULL for a run whose machine was never recorded. sqlite migrates on user_version 3 -> 4 (and from 1 and 2, which also had to gain the columns); PostgreSQL uses ADD COLUMN IF NOT EXISTS, since the shared database is created once and never migrated. Verified on a copy of both: the existing rows survive, the new insert shape works, re-running createTables is a no-op, and the NATURAL JOIN in test.py is unaffected - the added names collide with nothing in omcversion or in a branch table. sqlite2postgres.py learns the two columns as well, and now fills a text column the source lacks with NULL instead of the string "0". What made this worth doing: ExternalMedia and Buildings' Utilities.IO.Python_3_8 models have been flipping in exact anti-phase on master since 2026-08-11, ten run pairs out of ten. It turned out that ExternalMedia does not load on the Ubuntu 22.04 node and the Buildings Python library does not load on the Ubuntu 24.04 nodes, so master alternating between two machines is the whole of it - but establishing that took cross-referencing a transient claim table against report timestamps, which is exactly the work this commit makes unnecessary. See OpenModelica/OpenModelica#16376.
Newer clang complains about implicit function declarations
The library testing runs against a branch and says what broke after a change was merged. This tests a pull request before that, and compares it against master. The compiler is built from refs/pull/<N>/merge - the pull request as it would land, not the branch on its own - fetched from GitHub itself, since refs/pull/* is not on the read-only mirror the job clones. It is checked out detached, so nothing is left in the shared workspace for the next run to trip over. The run fills a pr-<N> table like any other branch, which the shared database of OpenModelica#295 makes cheap: the claim is (branch, libname), so it cannot collide with a master run, and the results carry their own omcversion, so nothing else can reuse them by mistake. What did not exist is the comparison. all-reports.py reports a branch against its own previous run, which is exactly what a pull request must not do, and its query reads one table. pr-report.py takes the newest run of pr-<N> and the newest run of the baseline branch and compares them with the same rule and the same thresholds: the phase each model reached, and what each phase cost. Per library it compares the newest run each side has of it, because a run does not necessarily hold every library - one whose version, compiler and configuration were tested before keeps the results of the run that produced them. It writes the report next to the nightly ones, in history/pr-<N>/<baseline run>..<pull request run>.html, and a summary to comment on the pull request with beside it. Two things would make a difference mean something other than "the pull request did this", and the report says so when they apply: - the machine, since two runs produced on different hardware compare the hardware as much as the change. The parameter defaults to the node that produces the master runs, and the report names both machines, which OpenModelica#320 records per library; - the libraries, since two runs that tested different library versions, or verified against different reference files, differ for reasons of their own. The models one run has and the other does not are counted and listed rather than quietly left out of the comparison, since a library that failed to load looks like nothing at all otherwise. In Jenkins it is the pull_request parameter, with pull_request_baseline, pull_request_config and pull_request_node beside it. A full run takes days, so this is on demand and takes a configuration file: testing every pull request is not the idea. The tables accumulate, roughly 19500 rows each, and unlike a branch a pull request is tested once and never again. drop-pr-tables.py drops the tables of pull requests that have been merged or closed, and of those tested more than --older-than days ago, together with the rows their runs left in the other tables; it lists them and does nothing unless it is given --yes. The reports published for them are not touched. It is the drop_stale_pull_request_tables parameter in Jenkins, and the same stage keeps the per-pull-request omc builds on the test node from piling up. --- Generated by Claude Code.
Owner
Author
Library testing for #1 against
|
| Branch | Run | Compiler | Machine | |
|---|---|---|---|---|
| Baseline | master |
2026-08-14 18:01:05 | OpenModelica 1.28.0~dev-284-g5fb7305 | ryzen-5950x-1 |
| Pull request | pr-1 |
2026-08-14 19:01:05 | OpenModelica 1.28.0~dev-284-g5fb7305 | ryzen-9950x |
3 models compared, 0 improved, 1 regressions, performance 0 improved, 1 regressions.
2 models affected
| Library | Model | Change |
|---|---|---|
| MyLibrary_1.0.0 | MyLibrary.Blocks.Examples.PID_Controller | Simulate performance 20.00 -> 0:03:20 |
| MyLibrary_1.0.0 | MyLibrary.Mechanics.MultiBody.Examples.Pendulum | Verify -> Failed |
Caveats
- The two runs were produced on different machines (ryzen-9950x against ryzen-5950x-1), so the timings compare the hardware as much as the pull request. The phases a model reaches are still comparable.
- 1 of the 2 libraries were not tested in the same version, or not with the same configuration and reference files, in the two runs; see Library Changes below.
- The baseline is the newest run of master, not the commit the pull request is based on, so a difference can also come from something merged into master since the pull request was branched.
Generated by the OpenModelica library testing
This was referenced Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway pull request in my own fork, used to check that pr-report.py --comment posts and then updates its own comment. Will be closed.