reusable Actions workflows?

[deitch]

Would you be interested in a PR to restructure the two separate workflows - build.yml and nightly-releases.yml - into reusable components? Functionality unchanged, but easier to manage.

You would end up with the following under .github/workflows/:

• build-reusable.yml - most of what is in build.yml
• build.yml - just calls the reusable flow
• nightly-releases.yml - mostly the same, but where it duplicates the build logic and matrix entirely from build.yml, it would just call the reusable flow

I have a working copy now, would be happy to open a PR if it would be accepted in principle (subject to review, of course).

[cmuellner]

Sounds interesting! Does this also result in less duplication?
I'd be happy to review the PR.

[deitch]

It does, indeed. That is why it was interesting to me.

For context, I was helping someone with a local clone (experimentation), and helped him adapt it, saw an opportunity to be helpful upstream. Good OSS citizenship and all that.

And by the way, super quick response. Thanks!

[cmuellner]

Thanks for working on this and for the offer to share it!

[deitch]

Hi @cmuellner .

Follow-up question. I am finding that reusable-build quite useful. Would you entertain additional options? Of course, I would leave the defaults as current, but I was thinking of adding some more parameters to control the build:

• report - whether or not to run make report. It already filters only for certain cases, but sometimes you want to suppress it explicitly
• strip - the README says: The toolchain has an option --enable-strip to control strip of host binaries, strip is disabled by default. It would be nice to have an option to enable it. We might even want to think about releasing stripped and non-stripped.

My concern with the second one is that, if we run it as part of the matrix, we rebuild everything, which is a huge waste. Would you consider creating both stripped and non-stripped versions as part of the CI process? I am not quite sure how to do that. Normally ./configure builds everything. We could run ./configure again as part of the same job, which should not quite rebuild the whole thing, but will avoid some of it. It would be even better if it were possible to do ./configure which would configure and generate Makefile etc. but not build, then do make install PREFIX=/mnt/riscv (or whatever the right incantation) followed by make install-strip PREFIX=/mnt/riscv-strip.

Thoughts?

[cmuellner]

Hi @cmuellner .

• report - whether or not to run make report. It already filters only for certain cases, but sometimes you want to suppress it explicitly

Sounds good.

• strip - the README says: The toolchain has an option --enable-strip to control strip of host binaries, strip is disabled by default. It would be nice to have an option to enable it. We might even want to think about releasing stripped and non-stripped.

Sounds also good.
However, I don't see why we would want to release unstripped and stripped binaries.
Still, I don't mind if the CI workflow supports stripped releases, but I don't want that enabled.

In general, I don't mind small features that are not used by the CI builds.
E.g. we also have the --with-$foo-sources= flags to override the submodule sources.
And since we also have --enable-stip, it is fine for me to have it supported in the workflow, even if it is not used when instanciating the workflow for the CI builds.
Happy to include that if it helps active contributors in their downstream integration.

[deitch]

report
sounds good

OK, great, I will open a PR just for that, to keep it simple.

Sounds also good.
Still, I don't mind if the CI workflow supports stripped releases, but I don't want that enabled

For sure. The default, IMHO, always should remain unchanged. If you want to make the default something different in the future, that should be a PR that focused just on changing that flag. I worked hard on that for the initial PR that closed this issue. "Principle of least surprise".

However, I don't see why we would want to release unstripped and stripped binaries.

If this were just one binary, I would agree, mostly. If someone downstream wants it stripped, just strip it. A quick look at the most recent release .tar.xz shows 32 binaries in riscv/bin/, another ~10 in riscv/riscv32-unknown-elf/bin/, and I probably missed some. The "normal" way to do this is just configure --enable-strip, but that would be at least a partial rebuild, might be a lot more. What do you think is the sane way to do this? I don't want to create the flag option, just so that someone using it immediately complains about why it is rebuilding everything.

[deitch]

we also have the --with-$foo-sources= flags to override the submodule sources

Oh, really? I missed that. How do you use that?

[cmuellner]

we also have the --with-$foo-sources= flags to override the submodule sources

Oh, really? I missed that. How do you use that?

                        --with-binutils-src=/home/cm/src/binutils-gdb/riscv-mainline \
                        --with-gcc-src=/home/cm/src/gcc/riscv-mainline \
                        --with-glibc-src=/home/cm/src/glibc/riscv-mainline \
                        --with-qemu-src=/home/cm/src/qemu/riscv-mainline \
                        --with-gdb-src=/home/cm/src/binutils-gdb/riscv-mainline \
                        --with-dejagnu-src=/home/cm/src/dejagnu/riscv-mainline \
                        --with-newlib-src=/home/cm/src/newlib-cygwin/riscv-mainline \
                        --with-llvm-src=/home/cm/src/llvm/riscv-mainline \

[cmuellner]

If this were just one binary, I would agree, mostly. If someone downstream wants it stripped, just strip it. A quick look at the most recent release .tar.xz shows 32 binaries in riscv/bin/, another ~10 in riscv/riscv32-unknown-elf/bin/, and I probably missed some. The "normal" way to do this is just configure --enable-strip, but that would be at least a partial rebuild, might be a lot more. What do you think is the sane way to do this? I don't want to create the flag option, just so that someone using it immediately complains about why it is rebuilding everything.

For me, enabling stripping is similar to having another default march, it is simply a different toolchain built.
We may support that build configuration, but we don't need to have it part of our releases.
Most users will be happy with unstripped binaries.
Users with special needs can be supported by configuring flags or config settings in the workflow, but we don't need to release what they need.

So if you need stripping in your downstream pipeline, everything except the final build process instantiation (similar to build.yaml), which enables stripping, can be upstream.

[deitch]

So if you need stripping in your downstream pipeline, everything except the final build process instantiation (similar to build.yaml), which enables stripping, can be upstream

I think that works.

I wouldn't mind finding a more efficient way, though. Something that says, "if you enable the strip option in the build pipeline, it will just take the existing results and strip them into a new .tar.xz," or "rerun make install as make install-strip", but in a way that doesn't rebuild everything.

Right now configure does configuration and building. How do I split that into 2 steps? Is there a "configure but do not build" flag to ./configure? And if I do that, how do I then invoke the build?

[cmuellner]

So if you need stripping in your downstream pipeline, everything except the final build process instantiation (similar to build.yaml), which enables stripping, can be upstream

I think that works.

I wouldn't mind finding a more efficient way, though. Something that says, "if you enable the strip option in the build pipeline, it will just take the existing results and strip them into a new .tar.xz," or "rerun make install as make install-strip", but in a way that doesn't rebuild everything.

Right now configure does configuration and building. How do I split that into 2 steps? Is there a "configure but do not build" flag to ./configure? And if I do that, how do I then invoke the build?

The configure flags are passed down to the individual projects (Binutils, GCC, etc), where they are affecting build steps of the individual modules and binaries. Therefore, stripping needs to be declared at configure time. Later stripping works as well, but then an enumeration of all binaries that should be stripped is necessary. For that, I would clone the target installation directory and strip all binaries in there, before creating an archive with the stripped binaries.

[deitch]

The configure flags are passed down to the individual projects (Binutils, GCC, etc), where they are affecting build steps of the individual modules and binaries. Therefore, stripping needs to be declared at configure time

Yup, that would work. But it would be almost double the time.

Besides, as far as I could tell, ./configure --enable-strip looked like it just changed the install_target and nothing else, so only the final make install step, which runs after build happens? Or did I misread the configure and Makefile.in?

Later stripping works as well, but then an enumeration of all binaries that should be stripped is necessary. For that, I would clone the target installation directory and strip all binaries in there, before creating an archive with the stripped binaries.

I was hoping that would be the efficient path. How would I do that?