Skip to content

Upgrade HAProxy to latest LTS version #15

Closed
Feature
#29
Closed
Upgrade HAProxy to latest LTS version#15
Feature
#29
Assignees
richturner
Labels
EnhancementImprovement of an existing feature

Description

@MartinaeyNL
MartinaeyNL
opened on Feb 4, 2026

Description

We currently run HAProxy on version 2.9, while version 3.2 LTS has came out since.
As new features have been introduced, and bugs have been resolved, it makes sense to upgrade.
For example, the configuration item minimum file size for applying compression above a certain threshold.

The goal is not to take advantage of the new possibilities offered by this version, this will be tackled in separate issues.
If possible, list those new possibilities as suggestions for further investigations as part of this issue.
See Announcing HAProxy 3.2

Acceptance criteria

  • HAProxy has been updated to the latest LTS version
  • The new image has been extensively tested on several platforms, for example:
    • Using a local Docker environment
    • Using a local Kubernetes environment
    • On a remote AWS EC2 instance
    • On a remote Kubernetes cluster
    • Using a custom project with a custom HAProxy config
    • A manual double check whether functionality like Support path based proxying #7 still works after upgrading.

Activity

  1. MartinaeyNL
    added
    EnhancementImprovement of an existing feature
    on Feb 4, 2026
  2. MartinaeyNL
    added the
    Feature
    issue type on Feb 4, 2026

  3. wborn commented on Feb 6, 2026

    @wborn
    wborn
    on Feb 6, 2026
    Member

    I asked Gemini what the impact is based on the CHANGELOG and our Dockerfile:

    Based on the provided Dockerfile which uses haproxy:2.9-alpine , and the CHANGELOG file covering development up to version 3.4-dev4, here are the major changes that concern your deployment.

    1. Current Version Status

    • Current: You are using HAProxy 2.9, which is a stable release.

    • Upgrade Path:

      • HAProxy 3.0 is the current LTS (Long Term Support) version, released in May 2024. This is the recommended target for production stability.
      • HAProxy 3.2 is a newer LTS version released in May 2025.
      • HAProxy 3.4 is currently in development.

    2. Critical Feature: Native ACME (Let's Encrypt) Support

    Your Dockerfile manually installs certbot, python, and an entrypoint.sh script to handle certificate renewals. A major change in the 3.x series is the introduction of a native ACME client, which could allow you to remove Certbot entirely from your image.

    • Native ACME Client: Starting around version 3.2 and 3.3, HAProxy includes a native ACME client to handle certificate generation and renewal without external tools.
    • Configuration: New keywords like crt-store and acme sections allow you to configure certificate authorities and renewal logic directly in haproxy.cfg.
    • Impact: You can simplify your Dockerfile significantly by removing the RUN apk add ... certbot lines and the custom renewal scripts once you upgrade to a version with stable ACME support (likely 3.2+).

    3. HTTP/3 and QUIC Support

    If you plan to support modern protocols, significant work has landed in recent versions:

    • Production Ready: QUIC and HTTP/3 support is officially marked as stable/non-experimental in version 3.0.
    • Ease of Use: Version 3.3 automatically enables SSL on QUIC servers and improves configuration for listeners.
    • Performance: Numerous optimizations for packet handling, congestion control (BBR support), and zero-copy forwarding have been added between 2.9 and 3.4.

    4. Breaking Changes & Deprecations

    Before upgrading, check your haproxy.cfg for these removed or deprecated features:

    • nbproc Removed: The nbproc keyword (multi-process mode) is fully removed. You must use nbthread (multi-threading) instead.

    • Deprecated Keywords:

      • req* and rsp* rules (e.g., reqadd, rspirep) are removed. You must use http-request and http-response actions.
      • option http-tunnel is deprecated.
      • listen sections with mode health are removed.
      • monitor-net is removed in 3.4-dev.
      • transparent and option transparent are deprecated in 3.3.

    5. Security & SSL Updates

    • OpenSSL 3.0+: HAProxy has full support for OpenSSL 3.x. Ensure your Alpine image uses a compatible OpenSSL version.
    • OCSP & Certificate Management: Improved CLI commands for managing certificates (add ssl crt-list, set ssl cert) allow for dynamic updates without reloads.
    • WolfSSL & AWS-LC: Support for these alternative SSL libraries has been added if you need FIPS compliance or specific features.

    6. Lua Scripting

    Your Dockerfile adds a Lua ACME plugin.

    • Lua 5.4: Support for Lua 5.4 was added in 2.4/2.5 eras.

    • Threaded Lua: Support for lua-load-per-thread allows better performance in multi-threaded setups.

    Summary of Action Items

    1. Upgrade Target: Plan an upgrade to HAProxy 3.0 LTS for stability and QUIC support.
    2. Simplify Image: Investigate HAProxy 3.2+ features to replace your certbot setup with the native ACME client.
    3. Config Audit: Verify your haproxy.cfg does not use nbproc or old req*/rsp* keywords.
    4. SSL Config: Ensure your SSL configuration aligns with the new crt-store and ssl-min-ver syntaxes if you plan to use modern TLS features.
  4. denniskuijs
    self-assigned this
    on May 1, 2026
  5. denniskuijs
    removed their assignment
    on May 15, 2026
  6. richturner
    changed the title [-]Upgrade HAProxy to latest version (v3.3)[/-] [+]Upgrade HAProxy to latest LTS version[/+] on May 19, 2026

  7. richturner commented on May 19, 2026

    @richturner
    richturner
    on May 19, 2026
    Member

    I created a 3.2 image and deployed on a test server, the image starts and seems to work without change although I now see this error:

    [WARNING]  (100) : config : parsing [/etc/haproxy/haproxy.cfg:131]: argument number 4 at position 37 is empty and marks the end of the argument list; all subsequent arguments will be ignored:
  acl too_fast fe_sess_rate ge "${MQTT_RATE_LIMIT}"

    The conditional logic for MQTT_RATE_LIMIT now throws a warning but doesn't seem to affect startup.

    Some info from Gemini:

    You are absolutely right. If it is genuinely undefined, then I misdiagnosed why the warning was firing. Thank you for correcting me.

    The issue actually stems from a "chicken-and-egg" quirk in how HAProxy 3.x parses configuration files.

    The Real Root Cause: Parsing Order

    In HAProxy 3.2, the configuration parser tokenizes the entire file and expands all environment variables before it fully evaluates conditional blocks like .if defined(...).

    Because of this order of operations:

    1. The tokenizer reads line 131.
    2. It sees "${MQTT_RATE_LIMIT}", notices the variable is undefined, and expands it to an empty string "".
    3. It validates the syntax ge "", realizes it is missing a required argument, and throws the [WARNING] you are seeing.
    4. Afterward, the configuration logic evaluates .if defined(MQTT_RATE_LIMIT) as false, and correctly skips loading the block into your active routing configuration.

    So the warning is purely a noisy side-effect of the early syntax-checking phase.

    The Solution: Inline Defaults

    HAProxy provides a way to suppress this specifically for environment variable expansions (detailed in section 2.3 of their configuration manual). You can provide a fallback value directly inside the variable expansion using a minus sign (-).

    Update line 131 to look like this:

    acl too_fast fe_sess_rate ge "${MQTT_RATE_LIMIT-0}"

    Why this completely fixes it:

    • When the variable is undefined, the early tokenizer expands the string to "0". It sees ge "0", which is perfectly valid syntax, so it throws no warnings.
    • Because the variable itself is still technically undefined in the environment, your .if defined(MQTT_RATE_LIMIT) check still accurately evaluates to false, and HAProxy will safely ignore the entire block just as you originally intended.

  8. richturner commented on May 19, 2026

    @richturner
    richturner
    on May 19, 2026
    Member

    Tested the above suggestion and it removes the parsing warning, will commit change to default haproxy.cfg

  9. richturner
    linked a pull request that will close this issueMove to HAProxy 3.2 LTS #29on May 19, 2026
  10. MartinaeyNL
    assigned
    richturner
    on May 20, 2026
  11. richturner
    assigned
    ebariaux
    on May 20, 2026

  12. richturner commented on May 20, 2026

    @richturner
    richturner
    on May 20, 2026
    Member

    Should probably update lua plugins

  13. richturner
    marked this as blocking on May 20, 2026

  14. ebariaux commented on May 20, 2026

    @ebariaux
    ebariaux
    on May 20, 2026
    Contributor

    @richturner I did run load tests using that proxy version.
    No problems encountered and performances in the same ballpark as with 2.9 version.
    It's however only publishing events over MQTTS, no HTTPS/WSS calls performed by the load test.

  15. ebariaux
    removed their assignment
    on Jun 1, 2026

  16. richturner commented on Jun 19, 2026

    @richturner
    richturner
    on Jun 19, 2026
    Member

    Now verified gateway tunnels function ok so I think we can get this merged and released and I'll create a separate issue for migrating LUA plugins

  17. wborn
    closed this as completedin #29on Jun 20, 2026
  18. wborn
    added a commit that references this issue on Jun 20, 2026

    Move to HAProxy 3.2 LTS (#29)

    83076ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

EnhancementImprovement of an existing feature

Type

Feature

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions