store/cockpit: use ibcli for building image (HMS-9389) #3636

kingsleyzissou · 2025-09-11T15:18:27Z

This PR switches the image building from osbuild-composer to ibcli. It does this by making use of cockpits long-running-process. Under the hood, cockpit creates a one-shot systemd service that executes the image build. We then monitor the systemd service and save the result to a file which we then use for reporting a compose status.

JIRA: HMS-9389

codecov · 2025-09-11T15:36:01Z

Codecov Report

❌ Patch coverage is 42.64706% with 78 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.67%. Comparing base (c0bc808) to head (97ef36d).
⚠️ Report is 42 commits behind head on main.

Files with missing lines	Patch %	Lines
src/test/mocks/cockpit/index.ts	0.00%	21 Missing ⚠️
src/AppCockpit.tsx	0.00%	12 Missing ⚠️
src/test/mocks/cockpit/cockpitFile.ts	8.33%	11 Missing ⚠️
...eImageWizard/steps/TargetEnvironment/Aws/index.tsx	23.07%	10 Missing ⚠️
...teImageWizard/steps/Review/ReviewStepTextLists.tsx	68.00%	8 Missing ⚠️
src/test/mocks/long-running-process/index.ts	63.15%	7 Missing ⚠️
src/Components/CreateImageWizard/validators.ts	14.28%	6 Missing ⚠️
fec.config.js	0.00%	1 Missing ⚠️
...Components/CreateImageWizard/CreateImageWizard.tsx	66.66%	1 Missing ⚠️
...nents/CreateImageWizard/utilities/requestMapper.ts	66.66%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #3636      +/-   ##
==========================================
+ Coverage   82.01%   84.67%   +2.66%     
==========================================
  Files         215      212       -3     
  Lines       25139    24719     -420     
  Branches     2587     2629      +42     
==========================================
+ Hits        20617    20931     +314     
+ Misses       4494     3762     -732     
+ Partials       28       26       -2

Files with missing lines	Coverage Δ
...Components/sharedComponents/ImageBuilderHeader.tsx	`96.87% <ø> (+7.35%)`	⬆️
src/Router.tsx	`0.00% <ø> (ø)`
src/constants.ts	`100.00% <100.00%> (ø)`
fec.config.js	`0.99% <0.00%> (-0.01%)`	⬇️
...Components/CreateImageWizard/CreateImageWizard.tsx	`96.17% <66.66%> (+<0.01%)`	⬆️
...nents/CreateImageWizard/utilities/requestMapper.ts	`91.09% <66.66%> (+0.01%)`	⬆️
src/Components/CreateImageWizard/validators.ts	`64.67% <14.28%> (-2.00%)`	⬇️
src/test/mocks/long-running-process/index.ts	`63.15% <63.15%> (ø)`
...teImageWizard/steps/Review/ReviewStepTextLists.tsx	`89.84% <68.00%> (-0.79%)`	⬇️
...eImageWizard/steps/TargetEnvironment/Aws/index.tsx	`72.02% <23.07%> (-4.26%)`	⬇️
... and 3 more

... and 8 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c0bc808...97ef36d. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kingsleyzissou · 2025-09-18T10:03:27Z

I found a bug in ibcli:
osbuild/image-builder-cli#306

And pushed a fix here:
osbuild/image-builder-cli#307

src/store/cockpit/helpers/imageType.ts

src/store/cockpit/helpers/paginate.ts

This change refactors some of the cockpit api file end extracts all the helper functions to their own file. This just makes the file more manageable to work with.

This is not strictly needed, it just makes life a little easier.

src/AppCockpit.tsx

kingsleyzissou · 2025-09-18T10:38:47Z

/jira-epic HMS-8666

kingsleyzissou · 2025-09-18T13:02:48Z

Lol

Last login: Thu Sep 18 09:43:10 2025 from 10.0.2.2
[systemd]
Failed Units: 4
  cockpit-image-builder-052e427f-4f0a-427c-b4b5-e835a78ef30c.service
  cockpit-image-builder-a978c59a-1a1d-4f34-8085-28387a12981b.service
  cockpit-image-builder-e9f6191c-f69a-4532-b524-adedc4338f0d.service
  cockpit-image-builder-fd9ba52b-b9c2-4065-a922-47b0c903d1e4.service

Use cockpit's `long-running-process` to launch a `one-shot` systemd service that runs the image-build with the image-builder cli. This is a step towards removing our dependency on `osbuild-composer`. Results are saved to a file and are updated with the lifecycle events from the systemd service.

The image-builder cli handles the the available target clouds based on environment variables set on the host. For example, if configs and credentials exist in `~/.aws`, then aws image types will be surfaced as available image types. We will need a way to provide the image-build with the name of the aws bucket, so we need to add this as an option in the AWS step of the wizard.

mvo5

Thank you! One quick inline command about the file spawning.

I also wonder what we can/could do to make testing easier for you? It seems for the most basic testing just having a way to override the location of image-builder and providing a mock script that e.g. fails with an error or that produceses some fake json would be the easiest. I almost wonder if ibcli provide this in its git repo. Like image-builder build failing-image and the mock would behave depedning on file type (i.e. for failing-image fails, for slow-image take a long time etc). But maybe I'm overthinking this :)

mvo5 · 2025-09-19T07:08:34Z

src/store/cockpit/cockpitApi.ts

+              // can't be started when in `init` state
+              process.state = 'stopped';
+
+              process.run(['bash', '-ec', cmd.join(' ')]).catch(async () => {


it seems cockpit works with arrays of strings, so would process.run(cmd) work? The only other change required would be HOME= needs to move into the "options" of process.run(), something like progress.run(cmd, environ={"HOME": ${user.home}}) (assuming python syntax here, I have no clue about typescript). As long as we control all inputs its problably fine as is but once there is user controlled input this turns into a command injection quickly and it seems we can just start with using the array of strings which makes command injections (or files with spaces) a non-issue.

It seems this could then also use the same trick that https://github.com/cockpit-project/cockpit/blob/main/examples/long-running-process/index.js#L20 is using, i.e. just monitor the jounal to get the output of ibcli, with that we could add image-builder build --progress=json (we currently only have verbose,term,debug there) and then you could render your own progress this way. When we design the json we need to make sure that its self-contained, i.e. that we must assume that at any point a client could connect or reconect and that the client will not have state. This is not a property of the underlying osbuild json stream which will often use "context" which needs to be captured and remembered by the consumer of the json (but no worries, just spelling it out here, doing this will not be hard especially if we start simple).

Ya I didn't like that either and adding HOME= is not ideal. But I ran into two issues here:

the list of strings didn't work as expected and it was really hard to debug why

adding the HOME as an env variable in the options also never worked and got ignored.

But those are both valid points and I will dig into it deeper and try figure out why that wasn't working.

With the json progress that seems nice and we could probably implement that in the getComposeStatus endpoint

@kingsleyzissou before you spend/waste too much time in debugging this please leave and maybe we can have a joint session where you can help me to reproduce so that I can dig into it a bit (if you want).

Oh sorry I actually missed this comment. I came back to it today in preparation for Friday's call.

I filed a feature request in cockpit:
cockpit-project/cockpit#22454

avitova

I was trying to test, and it did not work locally on Fedora 42. On main it is okay.

avitova · 2025-09-19T09:35:13Z

src/test/mocks/cockpit/index.ts

      if (args.length && args[0] === 'uname') {
        resolve('x86_64');
      }
+      if (args.length && args[0] === 'image-builder' && args[1] === 'list') {


I was about to give you a comment that this might fail. But wow. TIL that you can access [1] even if the array is shorter.

I think it's because args.length is non-zero :)

avitova · 2025-09-19T09:37:24Z

src/store/cockpit/helpers/imageType.ts

+    qcow2: 'guest-image',
+    ami: 'aws',
+    gce: 'gcp',
+    vhd: 'azure',
+    vmdk: 'vsphere',
+    ova: 'vsphere-ova',


No oci for cockpit?

Not yet. No gcp and azure yet either. Getting there slowly

kingsleyzissou · 2025-09-19T09:51:10Z

I was trying to test, and it did not work locally on Fedora 42. On main it is okay.

Thanks! Ya this happened to me in playwright, it was really hard to debug.

What version of image-builder cli do you have installed?

avitova · 2025-09-19T09:57:31Z

@kingsleyzissou Good question! I did not notice that I did not have image-builder cli installed at all.
Makes me think of 1. we should add some check or installation script into our makefile for setting up cockpit, 2. maybe have a way to detect that? 🤔

kingsleyzissou · 2025-09-19T10:05:37Z

@kingsleyzissou Good question! I did not notice that I did not have image-builder cli installed at all. Makes me think of 1. we should add some check or installation script into our makefile for setting up cockpit, 2. maybe have a way to detect that? 🤔

Ya that's a good point, I added it to the specfile but not to the makefile. I'll have a look at that

croissanne

Thank you for exploring this! Personally I really like this approach. But curious to hear what ppl think.

croissanne · 2025-09-25T08:22:40Z

src/store/cockpit/cockpitApi.ts

+              const cmd = [
+                // the image build fails if we don't set
+                // this for some reason
+                `HOME=${user.home}`,


Might be because the systemd process is running as root? Not sure how we could drop it though.

I put a note there and created a feature request for cockpit, basically the env variables aren't being set for the systemd service.

The solution is to either import the long-running-process script and modify it or add the feature upstream.

croissanne · 2025-09-25T09:04:10Z

src/store/cockpit/cockpitApi.ts

+              const ibBpPath = path.join(composeDir, 'bp.json');
+              await cockpit
+                .file(ibBpPath)
+                .replace(


Oh why rewrite the blueprint here?

We need it so we can pass it to ibcli we need to save the on-prem version of the blueprint not the hosted version.
So we need to keep track of both.

croissanne · 2025-09-25T09:10:50Z

src/store/cockpit/cockpitApi.ts

+              process.state = 'stopped';
+
+              process.run(['bash', '-ec', cmd.join(' ')]).catch(async () => {
+                await cockpit


Could we get image-builder-cli to write some sort of results file? Like I wonder if we can infer the results from the build log, saves us from having to write the result file + in that case I think we can safely close the tab and still get a proper compose status.

Or could we attach updateComposeStatus against all running systemd services on load?

Could we get image-builder-cli to write some sort of results file

Yeah that's what we're doing, there is a callback that is handling this too and should be attached to the systemd service afaict. I'll have to double check this.

kingsleyzissou · 2025-09-25T11:25:41Z

Thank you for exploring this! Personally I really like this approach. But curious to hear what ppl think.

Thanks! Yeah I've scheduled a meeting for tomorrow (sorry I know it's friday) for us to discuss as a team

kingsleyzissou requested review from mvo5, achilleas-k and croissanne September 11, 2025 15:18

kingsleyzissou force-pushed the ibcli branch 26 times, most recently from 9aad14c to b8668ec Compare September 12, 2025 14:25

kingsleyzissou force-pushed the ibcli branch 6 times, most recently from e5c10e6 to cece867 Compare September 17, 2025 17:06

kingsleyzissou changed the title ~~[wip] store/cockpit: use ibcli for building image~~ store/cockpit: use ibcli for building image Sep 18, 2025

kingsleyzissou marked this pull request as ready for review September 18, 2025 10:13

kingsleyzissou requested a review from a team as a code owner September 18, 2025 10:13

kingsleyzissou requested review from regexowl and avitova September 18, 2025 10:13

graphite-app bot reviewed Sep 18, 2025

View reviewed changes

src/store/cockpit/helpers/imageType.ts Outdated Show resolved Hide resolved

src/store/cockpit/helpers/paginate.ts Outdated Show resolved Hide resolved

kingsleyzissou added 2 commits September 18, 2025 11:22

store/cockpitApi: extract helpers into their own file

7df1956

This change refactors some of the cockpit api file end extracts all the helper functions to their own file. This just makes the file more manageable to work with.

store/cockpit: pretty print saved json files

03e901c

This is not strictly needed, it just makes life a little easier.

kingsleyzissou force-pushed the ibcli branch from cece867 to 0d8d38b Compare September 18, 2025 10:22

graphite-app bot reviewed Sep 18, 2025

View reviewed changes

src/AppCockpit.tsx Show resolved Hide resolved

schutzbot changed the title ~~store/cockpit: use ibcli for building image~~ store/cockpit: use ibcli for building image (HMS-9389) Sep 18, 2025

kingsleyzissou added 2 commits September 18, 2025 14:05

kingsleyzissou force-pushed the ibcli branch from 0d8d38b to 97ef36d Compare September 18, 2025 13:05

mvo5 reviewed Sep 19, 2025

View reviewed changes

avitova reviewed Sep 19, 2025

View reviewed changes

croissanne reviewed Sep 25, 2025

View reviewed changes

store/cockpit: use ibcli for building image (HMS-9389) #3636

Are you sure you want to change the base?

store/cockpit: use ibcli for building image (HMS-9389) #3636

Uh oh!

Conversation

kingsleyzissou commented Sep 11, 2025 • edited by schutzbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kingsleyzissou commented Sep 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kingsleyzissou commented Sep 18, 2025

Uh oh!

kingsleyzissou commented Sep 18, 2025

Uh oh!

mvo5 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

avitova left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kingsleyzissou commented Sep 19, 2025

Uh oh!

avitova commented Sep 19, 2025

Uh oh!

kingsleyzissou commented Sep 19, 2025

Uh oh!

croissanne left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kingsleyzissou commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kingsleyzissou commented Sep 11, 2025 •

edited by schutzbot

Loading

codecov bot commented Sep 11, 2025 •

edited

Loading

croissanne left a comment •

edited

Loading

kingsleyzissou commented Sep 25, 2025 •

edited

Loading