Skip to content

Harden against remote test VM creation failure #300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dboreham merged 1 commit into from
Sep 1, 2026
Merged

Harden against remote test VM creation failure #300

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions tests/k3s-deploy/cluster.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,22 @@ machines:
EOF

echo "Creating machine $machine_fqdn (mode: $STACK_K3S_MODE, kata: $STACK_K3S_KATA, size: $MACHINE_SIZE)"
$MACHINE_CMD --config-file "$machine_config" create --name "$machine_name" --type k8s-stack-host --wait-for-ip
# A create that fails leaves no VM and no log to read, so the provider's own
# one-line message is the whole of the evidence -- worth restating alongside
# the request that provoked it. The commonest cause is the provider being
# out of capacity for this size in this region just now, and DigitalOcean
# reports that in the same words as a size that is genuinely not offered
# there ("Size is not available in this region"), which reads as a permanent
# misconfiguration and usually is not one. Note also that the kata legs ask
# for a larger size than the others, so a shortfall takes those out alone.
if ! $MACHINE_CMD --config-file "$machine_config" create --name "$machine_name" --type k8s-stack-host --wait-for-ip; then
fail "Error: the provider refused to create the VM: size $MACHINE_SIZE, region $MACHINE_REGION, image $MACHINE_IMAGE.
No VM exists, so no test ran and there is nothing to collect diagnostics from.
If the message above is about the size being unavailable, it may mean the region
is temporarily out of that size rather than never offering it; check the
provider's catalogue before changing anything. MACHINE_SIZE and MACHINE_REGION
override the defaults chosen above."
fi

local machine_id
machine_id=$($MACHINE_CMD --config-file "$machine_config" list --name "$machine_name" --output json | jq -r '.[0].id')
Expand Down Expand Up @@ -323,8 +338,14 @@ EOF
# The machine is destroyed when the run ends, so cluster state has to be captured
# while the failure is fresh -- afterwards there is nothing left to look at.
diagnostics () {
if [ ! -f "$fqdn_file" ]; then
echo "No cluster to collect diagnostics from"
# The id file is what records that a VM was actually created; the FQDN is
# chosen and written before the create call, so a run whose create failed
# still has one, and guarding on that instead sent this ssh to a name that
# was never registered -- reported as "Could not resolve hostname", which
# reads like a DNS fault in the middle of a failing run rather than the
# absence of a machine it actually is.
if [ ! -f "$machine_id_file" ]; then
echo "No cluster to collect diagnostics from: no VM was created"
return
fi
echo "----- cluster diagnostics -----"
Expand Down