CI: publish a package index of the release wheels on GitHub Pages

.github/workflows/index.yml

@@ -0,0 +1,44 @@
+name: Package index
+
+# Publishes on GitHub Pages a package index (PEP 503) of the wheels attached to
+# the GitHub Releases, so that pip and uv can install them by version.
+on:
+  # Started by the Wheels workflow after a release (a release created with the
+  # GITHUB_TOKEN does not trigger the "release" event), or manually from the
+  # Actions tab, e.g. after editing or deleting a release.
+  workflow_dispatch:
+
+# A deployment in progress is not cancelled, the next one waits for it.
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+permissions:
+  contents: read
+
+jobs:
+  deploy:
+    name: Deploy package index
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}simple/qi/
+
+    steps:
+      - uses: actions/checkout@v7
+
+      - name: Generate index
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        run: python3 ci/generate_index.py site
+
+      - uses: actions/upload-pages-artifact@v5
+        with:
+          path: site
+
+      - id: deployment
+        uses: actions/deploy-pages@v5

.github/workflows/wheels.yml

@@ -93,6 +93,7 @@ jobs:
     runs-on: ubuntu-24.04
     permissions:
       contents: write
+      actions: write
 
     steps:
       - uses: actions/checkout@v7
@@ -122,3 +123,12 @@ jobs:
           prerelease=""
           if [[ "$GITHUB_REF_NAME" == *-* ]]; then prerelease="--prerelease"; fi
           gh release create "$GITHUB_REF_NAME" dist/*.whl --generate-notes $prerelease
+
+      # Adds the new wheels to the package index on GitHub Pages (pre-releases
+      # are not in it). Run on master: the "github-pages" environment only
+      # accepts deployments from the default branch, not from tags.
+      - name: Update package index
+        if: ${{ !contains(github.ref_name, '-') }}
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: gh workflow run index.yml --ref master

README.md

@@ -24,15 +24,44 @@ These LibQi python bindings correspond to the  `qi` Python module.
     - Linux x86_64, with glibc 2.28+ (Ubuntu 20.04+, Debian 10+, RHEL 8+, ...): `manylinux_2_28_x86_64`
     - MacOS 11+ on Apple Silicon: `macosx_11_0_arm64`
     - Windows x64: `win_amd64` (the MSVC C++ runtime is included in the wheel)
-- To install the packages
-    - download the wheel file (.whl) corresponding to your python version and OS
-    - install the package using `pip install path/to/wheel.whl`
+
+### Installing the wheels
+
+The wheels are listed in a package index hosted on GitHub Pages: https://funwithagents.github.io/libqi-python/simple/ (the wheels themselves are downloaded from the GitHub Releases). pip and uv pick the wheel matching your Python version and OS.
+
+With **uv**, in the `pyproject.toml` of your project:
+
+```toml
+[project]
+dependencies = ["qi==3.1.7"]
+
+[[tool.uv.index]]
+name = "funwithagents"
+url = "https://funwithagents.github.io/libqi-python/simple/"
+explicit = true  # only used for the packages below
+
+[tool.uv.sources]
+qi = { index = "funwithagents" }
+```
+
+`explicit = true` makes uv take `qi` only from this index, never from PyPI (where the official `qi` package of Aldebaran is published).
+
+With **pip**:
+
+```bash
+pip install "qi==3.1.7" --extra-index-url https://funwithagents.github.io/libqi-python/simple/
+```
+
+⚠️ Always pin the version with pip: it looks for `qi` in both PyPI and this index, and installs the highest version found.
+
+You can also download a wheel from the [Releases](https://github.com/funwithagents/libqi-python/releases) and install it with `pip install path/to/wheel.whl`.
 
 ## Continuous integration and releases
 
 The wheels are built by the [Wheels](.github/workflows/wheels.yml) GitHub Actions workflow, with [cibuildwheel](https://cibuildwheel.pypa.io) (configured in `pyproject.toml`):
 - on pull requests to `master` changing build-related files, and on manual runs ("Run workflow" in the Actions tab): the wheels are available as artifacts of the run
 - on tags `v*`: the wheels are also attached to a GitHub Release (a tag with a suffix, like `v3.1.7-rc1`, creates a pre-release)
+- after a release (except pre-releases), the [Package index](.github/workflows/index.yml) workflow regenerates the package index from all the releases (`ci/generate_index.py`) and deploys it to GitHub Pages. Run it manually after editing or deleting a release.
 
 The C++ dependencies (libqi, boost, openssl, ...) are built once per platform with Conan by `ci/conan_deps.sh`, then reused for all Python versions. Each wheel is tested by `ci/smoke_test.py`.
 

ci/generate_index.py

@@ -0,0 +1,107 @@
+"""Generate a PEP 503 package index of the wheels attached to the GitHub Releases.
+
+Usage: python ci/generate_index.py OUTPUT_DIR
+
+Writes OUTPUT_DIR/simple/index.html and OUTPUT_DIR/simple/qi/index.html, to be
+deployed to GitHub Pages. The index only contains links: the wheels stay in the
+GitHub Releases. Drafts and pre-releases are left out (a pre-release "v3.1.7-rc1"
+contains wheels with the same file names as the release "v3.1.7").
+
+The repository is read from GITHUB_REPOSITORY, and GITHUB_TOKEN (optional) is
+used to call the GitHub API.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import html
+import json
+import os
+import re
+import sys
+import urllib.request
+from pathlib import Path
+
+PROJECT = "qi"
+API_URL = "https://api.github.com"
+
+
+def api_get(path: str) -> list:
+    request = urllib.request.Request(f"{API_URL}{path}", headers={"Accept": "application/vnd.github+json"})
+    if token := os.environ.get("GITHUB_TOKEN"):
+        request.add_header("Authorization", f"Bearer {token}")
+    with urllib.request.urlopen(request) as response:
+        return json.load(response)
+
+
+def releases(repository: str) -> list:
+    """Published releases, oldest first."""
+    result = []
+    page = 1
+    while batch := api_get(f"/repos/{repository}/releases?per_page=100&page={page}"):
+        result += batch
+        page += 1
+    published = [r for r in result if not r["draft"] and not r["prerelease"]]
+    return sorted(published, key=lambda r: r["published_at"])
+
+
+def sha256(asset: dict) -> str:
+    """SHA-256 of an asset: given by GitHub, or computed for assets uploaded before mid-2025."""
+    if digest := asset.get("digest"):
+        algorithm, _, value = digest.partition(":")
+        if algorithm == "sha256":
+            return value
+    print(f"Computing the SHA-256 of {asset['name']}", flush=True)
+    hasher = hashlib.sha256()
+    with urllib.request.urlopen(asset["browser_download_url"]) as response:
+        while chunk := response.read(1 << 20):
+            hasher.update(chunk)
+    return hasher.hexdigest()
+
+
+def requires_python(wheel_name: str) -> str | None:
+    """Python version of a CPython wheel, from its tag (e.g. "cp313" -> "==3.13.*")."""
+    python_tag = wheel_name.removesuffix(".whl").split("-")[-3]
+    if match := re.fullmatch(r"cp(\d)(\d+)", python_tag):
+        return f"=={match[1]}.{match[2]}.*"
+    return None
+
+
+def page(title: str, links: list[str]) -> str:
+    body = "\n".join(f"    {link}<br>" for link in links)
+    return (f"<!DOCTYPE html>\n<html>\n  <head>\n    <meta name=\"pypi:repository-version\" content=\"1.0\">\n"
+            f"    <title>{html.escape(title)}</title>\n  </head>\n  <body>\n"
+            f"    <h1>{html.escape(title)}</h1>\n{body}\n  </body>\n</html>\n")
+
+
+def main() -> None:
+    output_dir = Path(sys.argv[1])
+    repository = os.environ["GITHUB_REPOSITORY"]
+
+    links = []
+    seen = set()
+    for release in releases(repository):
+        for asset in sorted(release["assets"], key=lambda a: a["name"]):
+            name = asset["name"]
+            if not name.endswith(".whl"):
+                continue
+            if name in seen:
+                sys.exit(f"{name} is attached to several releases")
+            seen.add(name)
+            url = f"{asset['browser_download_url']}#sha256={sha256(asset)}"
+            attributes = f'href="{html.escape(url)}"'
+            if python := requires_python(name):
+                attributes += f' data-requires-python="{html.escape(python)}"'
+            links.append(f"<a {attributes}>{html.escape(name)}</a>")
+        print(f"{release['tag_name']}: {len(release['assets'])} assets", flush=True)
+
+    project_dir = output_dir / "simple" / PROJECT
+    project_dir.mkdir(parents=True, exist_ok=True)
+    (output_dir / "simple" / "index.html").write_text(
+        page("Simple index", [f'<a href="{PROJECT}/">{PROJECT}</a>']))
+    (project_dir / "index.html").write_text(page(f"Links for {PROJECT}", links))
+    print(f"{len(links)} wheels in {project_dir / 'index.html'}")
+
+
+if __name__ == "__main__":
+    main()