diff --git a/README.md b/README.md index 4302671..d338066 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,8 @@ The test suite runner can be used in one of two ways: 1. With GitHub Actions, (or Travis, Circle, or another CI service) as the controller that connects to the remote test environment. 2. With the runner cloned to and run directly within the test environment. +For the second way, the [host tools](host-tools/README.md) do most of the work: `host-tools/setup.sh` creates `.env` with prompts, and `host-tools/testrunner.sh` runs the full test cycle in one step for cron. + The test runner is configured through environment variables, documented in [`.env.default`](.env.default). It shouldn't need any code modifications; in fact, please refrain from editing the scripts entirely, as it will make it easier to stay up to date. With a direct Git clone, you can: diff --git a/host-tools/README.md b/host-tools/README.md new file mode 100644 index 0000000..6511e34 --- /dev/null +++ b/host-tools/README.md @@ -0,0 +1,109 @@ +# Host tools + +Ready-made scripts for hosts that run the test runner on their own servers. Use them in place of writing your own setup and cron scripts. + +| Script | What it does | +|---|---| +| [`setup.sh`](setup.sh) | Checks the required software, creates `.env` with prompts, and tests the database connection. | +| [`testrunner.sh`](testrunner.sh) | Runs the full test cycle in one step: update, prepare, test, report and clean up. | + +The scripts do not change the runner itself. They use the same `.env` file and the same PHP files (`prepare.php`, `test.php`, `report.php`, `cleanup.php`) as a manual setup. + +## Before you start + +You need: + +- A Linux server with the usual configuration you give to your customers. +- A non-root user to run the tests. +- A database and a database user that are only for the tests. The test suite deletes and creates tables in this database. +- The software in the [Requirements](../README.md#requirements) section of the main README: PHP, Git, rsync, wget, unzip, Node.js and npm. Composer is optional. + +The scripts do not install software, create the database, or set up cron. + +## Set up and run the tests + +1. Clone the runner as the test user: + + ```bash + git clone https://github.com/WordPress/phpunit-test-runner.git + cd phpunit-test-runner + ``` + +2. Create `.env`: + + ```bash + ./host-tools/setup.sh + ``` + + The script asks for the database credentials, the PHP executable, the checkout directory and the report API key. Press Enter to keep the value in brackets. If it cannot connect to the database, it stops and does not change `.env`. + +3. Run the tests once and read the output: + + ```bash + ./host-tools/testrunner.sh + ``` + + A run can take a long time. Each run clones WordPress and installs the npm and Composer dependencies again. + +4. Run the tests on a schedule. For example, a cron entry for every 4 hours. Replace `/path/to/phpunit-test-runner` with the directory of your clone: + + ``` + 0 */4 * * * /path/to/phpunit-test-runner/host-tools/testrunner.sh >> /path/to/testrunner.log 2>&1 + ``` + + The script finds the runner from its own location, so cron does not need to change to that directory first. + +To report results to the [Host Test Results](https://make.wordpress.org/hosting/test-results/) page, you need a bot user with the "Test Reporter" role. See [How to report](https://make.wordpress.org/hosting/handbook/tests/). After you get the application password, run `./host-tools/setup.sh` again and enter it as `botuser:application password`. + +## setup.sh + +```bash +./host-tools/setup.sh [--non-interactive] [--skip-checks] [--skip-db-check] +``` + +| Option | Effect | +|---|---| +| `--non-interactive` | Do not ask. Use values from the environment and the current `.env`. For provisioning tools. | +| `--skip-checks` | Do not check the required software. | +| `--skip-db-check` | Do not test the database connection. | + +- Values come from the environment first, then from the current `.env`, then from the defaults. +- The script sets `WPT_TEST_DIR` to the same directory as `WPT_PREPARE_DIR`, which a run on the same server requires. +- It saves the old `.env` as `.env.bak-` and makes the new `.env` readable only by its owner. + +Example for a provisioning tool: + +```bash +WPT_DB_NAME=wptests WPT_DB_USER=wptests WPT_DB_PASSWORD=secret \ +WPT_REPORT_API_KEY='botuser:application password' \ +./host-tools/setup.sh --non-interactive +``` + +## testrunner.sh + +```bash +./host-tools/testrunner.sh +``` + +1. Stops if `.env` does not exist. +2. Stops if another run is still active. This uses `flock`, which most Linux systems have. Without `flock`, the script does not check for other runs. +3. Updates the runner with `git pull --ff-only origin master`. If the update fails, the script shows a warning and continues. +4. Runs `prepare.php`, `test.php`, `report.php` and `cleanup.php` with the PHP executable from `WPT_PHP_EXECUTABLE`. + +Behavior when a step fails: + +- When a test fails, the script still runs the report, so the failure shows on the results page. +- When `prepare.php` fails, the script skips the tests and the report. +- The script always runs `cleanup.php`, also when a step fails or the run is stopped. +- The exit status is `0` when every step succeeds and `1` when any step fails. + +| Environment variable | Effect | +|---|---| +| `WPT_RUNNER_DIR` | The runner directory. Default: the directory above `host-tools/`. | +| `WPT_SKIP_UPDATE=1` | Skip `git pull`, for example when you test local changes. | + +## Contributing + +- Keep the scripts in Bash, and do not change the runner's PHP files from them. +- Test changes on a Linux server, with a successful run and with a failed step. +- Update this README when you add a script or an option.