Support Azure Container Linux hosts with a configurable install prefix - #323
Draft
Philip Lombardi (plombardi89) wants to merge 6 commits into
Draft
Philip Lombardi (plombardi89) wants to merge 6 commits into
Philip Lombardi (plombardi89) wants to merge 6 commits into
Conversation
The agent library places its own binaries and helper scripts under a fixed /usr/local prefix. Azure Container Linux mounts /usr read-only, so nothing can be installed there and a Flex Node cannot run on it at all. The version that makes the prefix configurable is not released yet, so this pins the commit under review upstream rather than a tag. That is temporary and has to become a release version before this merges. The same build also reports host package capability truthfully instead of assuming a package manager exists, which matters here because the image has none.
Hosts that mount /usr read-only cannot take the agent's own binaries and helper scripts in the default location. A config field lets them name a writable one. The prefix is declared, never inferred. Keying off distribution identity would misclassify a hardened host whose /usr happens to be read-only, and would silently relocate files on any host whose os-release changed. A wrong guess is expensive to recover from, because generated systemd units reference these paths absolutely. Validation is delegated to the agent library rather than duplicated. The prefix is interpolated into generated units and into a shell script, neither of which quotes it, so the accepted syntax is deliberately narrow; keeping that rule in one place stops the two projects from disagreeing about what is accepted.
defaultAgentUpgradePaths hardcoded /usr/local/bin and
/usr/local/lib/aks-flex-node, both under a read-only /usr on Azure Container
Linux, and it is called with no override, so the location could not be
changed.
The prefix is read from this project's own config rather than the agent
library's applied config. That distinction was found the hard way: the
library exposes a lookup for exactly this, but bootstrap here never writes
that file, so on a real host it is absent, the lookup silently returns the
default, and the failure surfaces later as
initialize agent binary layout: no executable agent binary found
which names neither the prefix nor the path it looked in.
An unreadable or absent config yields the empty prefix, which selects the
default. That is what a host installed before this existed actually has on
disk, so nothing changes for it.
Callers that already hold a config pass it directly; the lookup exists for
entry points started by systemd, which cannot inherit the prefix from the
environment that ran bootstrap.
recoveryScriptPath was a second hardcoded location under /usr/local/lib. On
Azure Container Linux this fails bootstrap after the node has already joined:
install-service: write /usr/local/lib/aks-flex-node/aks-flex-node-recovery.sh:
mkdir /usr/local/lib/aks-flex-node: read-only file system
which is a worse failure than refusing up front, because the cluster has a
node in it by then.
The constant stays, because it is also the substitution placeholder inside
the embedded recovery unit. Only the install location becomes prefix
relative; the literal in the unit template has to keep matching the asset.
systemdSystemDir is deliberately not prefix relative. Units have to live where
systemd looks for them, and /etc is writable even when /usr is not.
Philip Lombardi (plombardi89)
had a problem deploying
to
e2e-testing
September 21, 2026 18:09 — with 
GitHub Actions
Failure
GitHub Actions
Failure
The prefix work it depends on was consolidated upstream and rebased onto main after the bootstrap ownership change merged, so the previously pinned commit no longer exists on any branch. Still an unreleased commit rather than a tag, and still has to become a release version before this merges.
The e2e offline artifacts scenario builds its bundle with
agent-artifacts-builder, installed from the pinned agent library version, so
raising that pin changed the tool.
Its --legal-files-dir used to default to empty, which skipped materializing
LICENSE and NOTICE. The default is now the working directory, and the
materialization is unconditional whenever the flag is set, so the build fails
on this repository, which has no NOTICE:
materialize "NOTICE": link failed: link NOTICE
.../offline-bootstrap-artifacts/v1.35.0/NOTICE: no such file or directory
The bundle never reaches a user. It is published to a loopback registry on a
throwaway VM for the duration of one test, so there is nothing for it to
carry, and passing the flag empty restores the previous behaviour.
Worth flagging upstream separately: a flag default that changed from skip to
require breaks any consumer that never passed it.
Philip Lombardi (plombardi89)
deployed
to
e2e-testing
September 21, 2026 23:43 — with
GitHub Actions
Active
GitHub Actions
Active
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: WIP draft. Not ready to merge.
Long-lived branch collecting the changes AKS Flex Node needs to run on Azure
Container Linux (ACL). Further commits will land here as the work proceeds
rather than as a series of separate PRs.
The dependency, first
go.modpins an unreleased commit ofgithub.com/Azure/unbounded, not atag. The change that makes the agent's host paths configurable is still under
review upstream, and nothing here compiles without it.
That pin is temporary and must become a release version before this merges.
It is the one thing blocking everything else in this PR.
Why ACL does not work today
/usris mounted read-only and there is no package manager. Two separateplaces install into it unconditionally:
defaultAgentUpgradePathshardcodes/usr/local/binand/usr/local/lib/aks-flex-node, and is called with no override, so thelocation cannot be changed.
recoveryScriptPathhardcodes a second location under/usr/local/lib.This one fails after the node has already joined the cluster:
The host itself is in better shape than expected. Every other prerequisite is
present in stock ACL, and
/opt,/var/liband/etcare writable, so awritable prefix exists.
What is here
agent.hostPrefix, validated by delegating to the agent libraryHosts that set no prefix resolve exactly the paths they do today.
Two decisions worth reviewing
The prefix is declared, never inferred. Keying off distribution identity
would misclassify a hardened host whose
/usrhappens to be read-only, andwould silently relocate files on any host whose
os-releasechanged. A wrongguess is expensive because generated systemd units reference these paths
absolutely.
The prefix is read from this project's own config, not the agent library's
applied config. The library exposes a lookup for exactly this purpose and
using it here is wrong: bootstrap in this project never writes that file, so on
a real host it is absent, the lookup silently returns the default, and the
failure surfaces later as
initialize agent binary layout: no executable agent binary found, naming neither the prefix nor the path it looked in.Still to come on this branch
scripts/install.shreports ACL as supported Azure Linux 3 and then failscopying into a read-only
/usr/local/bin. ACL is distinguishable:VARIANT_ID=azurecontainerlinux, andID_LIKE="flatcar"as a second signalthat no Azure Linux 3 image carries. Neither field is parsed today.
install.shanduninstall.shhardcodeINSTALL_DIR, whilebootstrap.shalready supports
AKS_FLEX_NODE_INSTALL_DIRand--install-dir.payload passed as
customDatais never acted on and nothing reports anerror.
What has and has not been verified
An ACL host provisioned entirely by Ignition joined an AKS cluster and ran
workloads with these changes applied: no SSH, no reboot, binaries under
/opt/aks-flex-node/{bin,lib}with/usr/local/binunused, pod-to-pod trafficworking in both directions.
That run predates this branch being rebased onto current
mainand repointedat the extracted upstream API, so treat it as establishing that the design
works rather than that this exact tree does.
Not covered: ARM Machine registration was deliberately deferred, so it fails
with a 403 and is skipped; and a blue/green agent upgrade and rollback has not
been exercised on ACL.