Executor with progress updates (HMS-9341) #4903
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
croissanne
requested review from
bcl and
supakeen
and removed request for
a team
croissanne
force-pushed
the
executor-with-logging
branch
2 times, most recently
from
0bc9425 to
854dd7b
Compare
lzap
reviewed
Oct 23, 2025
lzap
reviewed
Oct 23, 2025
croissanne
force-pushed
the
executor-with-logging
branch
from
854dd7b to
66c856d
Compare
Contributor
|
The PR is ready for review but I see no changes, including that |
Member
Author
Composer CI has this thing where it blocks execution if the PR is in a draft state (unless you manually click). Maybe we should change that. |
Member
You should use |
Member
Author
ah thanks, had no idea! |
croissanne
force-pushed
the
executor-with-logging
branch
from
66c856d to
db61572
Compare
croissanne
changed the title
croissanne
force-pushed
the
executor-with-logging
branch
3 times, most recently
from
67363ed to
97efd0f
Compare
Changes with 0.210.0 ---------------- - Make bootupd metadata generation configurable per distro (osbuild/images#1958) - Author: Paul Whalen, Reviewers: Lukáš Zapletal, Simon de Vlieger - many: add support for bootc container based anaconda ISOs (HMS-9495) (osbuild/images#1906) - Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger, Tomáš Hozza - osbuild/osbuild-exec: make osbuild command a variable (osbuild/images#1965) - Author: Sanne Raymaekers, Reviewers: Lukáš Zapletal, Simon de Vlieger Changes with 0.211.0 ---------------- - Update osbuild dependency commit ID to latest (osbuild/images#1968) - Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger - Update snapshots to 20251023 (osbuild/images#1967) - Author: SchutzBot, Reviewers: Achilleas Koutsou, Simon de Vlieger - deps: switch yaml libraries (osbuild/images#1970) - Author: Simon de Vlieger, Reviewers: Achilleas Koutsou, Michael Vogt - osbuild/monitor: increase status scanner buffer size to 16MiB (osbuild/images#1977) - Author: Sanne Raymaekers, Reviewers: Michael Vogt, Simon de Vlieger - osbuild: tweak how we can mock osbuild slightly (osbuild/images#1969) - Author: Michael Vogt, Reviewers: Achilleas Koutsou, Sanne Raymaekers - schutzbot/terraform: bump terraform sha (osbuild/images#1976) - Author: Florian Schüller, Reviewers: Achilleas Koutsou, Sanne Raymaekers, Simon de Vlieger Changes with 0.212.0 ---------------- - distro,manifest: have a single `LoraxTemplates` type (osbuild/images#1975) - Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger - distro: cleanup/reuse YAML for bootc ISO (osbuild/images#1971) - Author: Michael Vogt, Reviewers: Lukáš Zapletal, Simon de Vlieger - distro: cleanup/reuse YAML for bootc distro (osbuild/images#1944) - Author: Michael Vogt, Reviewers: Achilleas Koutsou, Simon de Vlieger - distro: load distros from multiple files (osbuild/images#1974) - Author: Simon de Vlieger, Reviewers: Michael Vogt, Tomáš Hozza - osbuild: switch from a bufio.Scanner to a bufio.Reader (osbuild/images#1978) - Author: Michael Vogt, Reviewers: Lukáš Zapletal, Simon de Vlieger - reporegistry: various cleanups (osbuild/images#1980) - Author: Michael Vogt, Reviewers: Simon de Vlieger, Tomáš Hozza - various: fix `make gh-action-test` and update README (osbuild/images#1981) - Author: Michael Vogt, Reviewers: Lukáš Zapletal, Simon de Vlieger Changes with 0.213.0 ---------------- - data: update fedora 43 repositories (osbuild/images#1983) - Author: Simon de Vlieger, Reviewers: Michael Vogt, Sanne Raymaekers - osbuild-exec: add buildlog and monitor support (osbuild/images#1966) - Author: Sanne Raymaekers, Reviewers: Michael Vogt, Simon de Vlieger
Workers can forward progress information of jobs using a partial result in a job update.
croissanne
force-pushed
the
executor-with-logging
branch
2 times, most recently
from
52fa952 to
a0696d5
Compare
croissanne
force-pushed
the
executor-with-logging
branch
from
5f3f328 to
742b6d0
Compare
Member
Author
croissanne
force-pushed
the
executor-with-logging
branch
3 times, most recently
from
aa3adcc to
0007fbe
Compare
croissanne
force-pushed
the
executor-with-logging
branch
8 times, most recently
from
520b648 to
e1ccac6
Compare
Uses status scanner to monitor osbuild to generate progress updates. Progress updates are sent every 30 seconds. Only one level of subprogress needs to be supported. Because osbuild manifests consist of pipelines and stages, more than 2 levels of progress are not expected. The result is still captured from stdout as before. The worker-executor will stream the progress information over http, and save the result in a separate file in the output archive.
croissanne
force-pushed
the
executor-with-logging
branch
2 times, most recently
from
8dece43 to
aa9f0c5
Compare
The progress output is streamed over HTTP. Stdout, which contains the osbuild result, is captured separately and put in the output archive. A separate log handler was added, so osbuild's stdout is always easily retrievable, even in case of build failures.
If osbuild crashes on the worker-executor side its build endpoint will return an error because it is unable to move the osbuild json result to the output directory and create a tarball. This causes the status scanner to error because it will receive invalid JSON. Because the full stack trace coming from osbuild should be present in the error that is handed back to the worker (so it can construct the job result with all the error details), the osbuild output needs to be fetched in case of build failures. The status scanner will put the line it cannot scan into the error, but it only reads up until the first newline. Thus osbuild's output cannot be forwarded via the status scanner.
croissanne
force-pushed
the
executor-with-logging
branch
9 times, most recently
from
b9f7af9 to
1026a45
Compare
If osbuild fails its stdout should be captured and added to the error details, also when using the ec2 executor. This test case makes sure that the error details contain the stacktrace and the error message from osbuild.
croissanne
force-pushed
the
executor-with-logging
branch
from
1026a45 to
1290bda
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As a followup this will need something in the cloudapi to actually expose the progress, but this is big enough of a change already, and will need some careful monitoring of the stage workers once deployed.