monolith-os/monolith
1
1
Fork 0
Code Issues Pull requests Projects Releases 40 Packages 20 Wiki Activity Actions 1

Retry the image build step to ride out the setup-qemu flake #77

Merged
TheDarkWizard merged 1 commit from build-step-retries into main 2026-07-24 19:13:15 +00:00
Conversation 1 Commits 1 Files changed 1 +81 -1
TheDarkWizard commented 2026-07-24 19:11:50 +00:00
Owner
Copy link

Roughly every night one of the matrix editions fails with exit code 1 before the build even starts, in the Set up QEMU step inside blue-build/github-action. That step JSON-parses the stdout of a docker run tonistiigi/binfmt, and when several matrix jobs hit the host's shared binfmt_misc at the same moment, that run occasionally returns empty output and the step dies with Unexpected end of JSON input. Re-running the job by hand always passes.

The action gives us no way to skip or retry its QEMU step (checked v1.12.0 and current main), so this retries the whole build step: up to three attempts, the first two with continue-on-error, only the last one deciding the job. Between attempts the leftover buildah workers get swept the same way the job-start sweep does, so a failed attempt can't wedge the retry when the CLI removes the base image after building.

Tradeoff: a genuine build failure now replays up to two more times, but the podman layer cache makes the replays mostly cache hits on the way back to the failing layer, and fail-fast is already off. The three attempts must keep identical inputs; Renovate bumps the action ref on all three at once, cli_version stays a manual bump in three places.

Roughly every night one of the matrix editions fails with exit code 1 before the build even starts, in the Set up QEMU step inside blue-build/github-action. That step JSON-parses the stdout of a `docker run tonistiigi/binfmt`, and when several matrix jobs hit the host's shared binfmt_misc at the same moment, that run occasionally returns empty output and the step dies with `Unexpected end of JSON input`. Re-running the job by hand always passes. The action gives us no way to skip or retry its QEMU step (checked v1.12.0 and current main), so this retries the whole build step: up to three attempts, the first two with continue-on-error, only the last one deciding the job. Between attempts the leftover buildah workers get swept the same way the job-start sweep does, so a failed attempt can't wedge the retry when the CLI removes the base image after building. Tradeoff: a genuine build failure now replays up to two more times, but the podman layer cache makes the replays mostly cache hits on the way back to the failing layer, and fail-fast is already off. The three attempts must keep identical inputs; Renovate bumps the action ref on all three at once, cli_version stays a manual bump in three places.
Retry the image build step to ride out the setup-qemu flake
Some checks failed
bluebuild / Check if a rebuild is needed (pull_request) Successful in 3s
Details
PR Rebase Hint / Post rebase command (pull_request) Successful in 4s
Details
bluebuild / Release (pull_request) Has been cancelled
Details
bluebuild / Generate Release (pull_request) Has been cancelled
Details
bluebuild / Build COSMIC NVIDIA (COSMIC NVIDIA, recipe-cosmic-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build COSMIC (COSMIC, recipe-cosmic.yml) (pull_request) Has been cancelled
Details
bluebuild / Build GNOME NVIDIA (GNOME NVIDIA, recipe-gnome-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build GNOME (GNOME, recipe-gnome.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Hyprland DMS NVIDIA (Hyprland DMS NVIDIA, recipe-hyprland-dms-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Hyprland DMS (Hyprland DMS, recipe-hyprland-dms.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Hyprland Noctalia NVIDIA (Hyprland Noctalia NVIDIA, recipe-hyprland-noctalia-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Hyprland Noctalia (Hyprland Noctalia, recipe-hyprland-noctalia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build KDE NVIDIA (KDE NVIDIA, recipe-kde-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build KDE (KDE, recipe-kde.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Niri DMS NVIDIA (Niri DMS NVIDIA, recipe-niri-dms-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Niri DMS (Niri DMS, recipe-niri-dms.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Niri Noctalia NVIDIA (Niri Noctalia NVIDIA, recipe-niri-noctalia-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Niri Noctalia (Niri Noctalia, recipe-niri-noctalia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Nourish NVIDIA (Nourish NVIDIA, recipe-nourish-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Nourish (Nourish, recipe-nourish.yml) (pull_request) Has been cancelled
Details
bluebuild / Build XFCE NVIDIA (XFCE NVIDIA, recipe-xfce-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build XFCE (XFCE, recipe-xfce.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Budgie NVIDIA (Budgie NVIDIA, recipe-budgie-nvidia.yml) (pull_request) Has been cancelled
Details
bluebuild / Build Budgie (Budgie, recipe-budgie.yml) (pull_request) Has been cancelled
Details
7c68a64a47
 
Most nights one of the matrix builds dies before the build even starts:
the setup-qemu step embedded in blue-build/github-action JSON-parses the
stdout of a docker run of tonistiigi/binfmt, and with several jobs
poking the host's shared binfmt_misc at the same moment that run
occasionally comes back empty, so the step crashes with "Unexpected end
of JSON input". Manually re-running the job always passes.

The action has no input to skip or retry its QEMU step (checked v1.12.0
and current main), so retry the whole build step instead: up to three
attempts, the first two with continue-on-error, the last one deciding
the job. Between attempts, sweep leftover buildah workers the same way
the job-start sweep does, so a failed attempt can't wedge the retry on
the post-build base image removal.
forgejo-actions commented 2026-07-24 19:11:54 +00:00
Copy link

🧪 Test this PR on a real install

Once the build checks on this PR pass, a signed test image is published for each edition this PR builds. Pick the one matching your hardware and, from an existing Monolith install (which already has the signing policy), rebase onto it:

gnome

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/gnome:pr-77-44
systemctl reboot

kde

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/kde:pr-77-44
systemctl reboot

xfce

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/xfce:pr-77-44
systemctl reboot

cosmic

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/cosmic:pr-77-44
systemctl reboot

budgie

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/budgie:pr-77-44
systemctl reboot

hyprland-noctalia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-noctalia:pr-77-44
systemctl reboot

hyprland-dms

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-dms:pr-77-44
systemctl reboot

niri-noctalia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-noctalia:pr-77-44
systemctl reboot

niri-dms

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-dms:pr-77-44
systemctl reboot

nourish

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/nourish:pr-77-44
systemctl reboot

gnome-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/gnome-nvidia:pr-77-44
systemctl reboot

kde-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/kde-nvidia:pr-77-44
systemctl reboot

xfce-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/xfce-nvidia:pr-77-44
systemctl reboot

cosmic-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/cosmic-nvidia:pr-77-44
systemctl reboot

budgie-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/budgie-nvidia:pr-77-44
systemctl reboot

hyprland-noctalia-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-noctalia-nvidia:pr-77-44
systemctl reboot

hyprland-dms-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-dms-nvidia:pr-77-44
systemctl reboot

niri-noctalia-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-noctalia-nvidia:pr-77-44
systemctl reboot

niri-dms-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-dms-nvidia:pr-77-44
systemctl reboot

nourish-nvidia

rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/nourish-nvidia:pr-77-44
systemctl reboot

The tags are rebuilt on every new commit here, so rpm-ostree upgrade pulls the latest build. When you're done testing, return to your edition's released image (:latest).

The commands cover the editions this PR affected when it was opened; commits that touch more editions publish their tags too, just without a refreshed comment. The test tags stop updating once this PR is merged or closed.

### 🧪 Test this PR on a real install **Once the build checks on this PR pass**, a signed test image is published for each edition this PR builds. Pick the one matching your hardware and, from an existing Monolith install (which already has the signing policy), rebase onto it: **`gnome`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/gnome:pr-77-44 systemctl reboot ``` **`kde`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/kde:pr-77-44 systemctl reboot ``` **`xfce`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/xfce:pr-77-44 systemctl reboot ``` **`cosmic`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/cosmic:pr-77-44 systemctl reboot ``` **`budgie`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/budgie:pr-77-44 systemctl reboot ``` **`hyprland-noctalia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-noctalia:pr-77-44 systemctl reboot ``` **`hyprland-dms`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-dms:pr-77-44 systemctl reboot ``` **`niri-noctalia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-noctalia:pr-77-44 systemctl reboot ``` **`niri-dms`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-dms:pr-77-44 systemctl reboot ``` **`nourish`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/nourish:pr-77-44 systemctl reboot ``` **`gnome-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/gnome-nvidia:pr-77-44 systemctl reboot ``` **`kde-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/kde-nvidia:pr-77-44 systemctl reboot ``` **`xfce-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/xfce-nvidia:pr-77-44 systemctl reboot ``` **`cosmic-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/cosmic-nvidia:pr-77-44 systemctl reboot ``` **`budgie-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/budgie-nvidia:pr-77-44 systemctl reboot ``` **`hyprland-noctalia-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-noctalia-nvidia:pr-77-44 systemctl reboot ``` **`hyprland-dms-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/hyprland-dms-nvidia:pr-77-44 systemctl reboot ``` **`niri-noctalia-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-noctalia-nvidia:pr-77-44 systemctl reboot ``` **`niri-dms-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/niri-dms-nvidia:pr-77-44 systemctl reboot ``` **`nourish-nvidia`** ```bash rpm-ostree rebase ostree-image-signed:docker://forge.waywardinn.com/monolith-os/nourish-nvidia:pr-77-44 systemctl reboot ``` The tags are rebuilt on every new commit here, so `rpm-ostree upgrade` pulls the latest build. When you're done testing, return to your edition's released image (`:latest`). _The commands cover the editions this PR affected when it was opened; commits that touch more editions publish their tags too, just without a refreshed comment. The test tags stop updating once this PR is merged or closed._
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something isn't working

  
dependencies
Pull requests that update a dependency file

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
github_actions
Pull requests that update GitHub Actions code

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
bug
dependencies
documentation
duplicate
enhancement
github_actions
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
monolith-os/monolith!77
No description provided.