containers fail to start in rawhide

[matthiasclasen]

Issue Description

My toolboxes fail to start in rawhide.

podman-5.7.0~rc1-1.fc44.x86_64
toolbox-0.3-2.fc44.x86_64

Error: unable to start container "e924fa34ad6e561edda26ef2c537ea2cfc751e4dc7d30eb57e4ecc8964ea31fb": mounting storage for container e924fa34ad6e561edda26ef2c537ea2cfc751e4dc7d30eb57e4ecc8964ea31fb: creating overlay mount to /home/mclasen/.local/share/containers/storage/overlay/5197d8fe4aa13451fa6abfbdecbe9f6107f54d58bec187a5706771f16f12a3d1/merged, mount_data="lowerdir=/home/mclasen/.local/share/containers/storage/overlay/l/S2KNMSPQFIGNFT7EW46ZDTHRGK:/home/mclasen/.local/share/containers/storage/overlay/l/S2KNMSPQFIGNFT7EW46ZDTHRGK/../diff1:/home/mclasen/.local/share/containers/storage/overlay/l/ZGL47N2OPFGLUX33ILI2ROAS23:/home/mclasen/.local/share/containers/storage/overlay/l/GLTPEUXR5WXDUQLE6VJC64MHDL:/home/mclasen/.local/share/containers/storage/overlay/l/4CII473YAJHTQRF66LAQH3DFPM:/home/mclasen/.local/share/containers/storage/overlay/l/ZX3UINK3AESXULCLLW655JUQHU:/home/mclasen/.local/share/containers/storage/overlay/l/SOJHIOU42K7C5PTLYS4LGWZVXH:/home/mclasen/.local/share/containers/storage/overlay/l/5247CCTVQXEJPRSCOAVJKST2NU:/home/mclasen/.local/share/containers/storage/overlay/l/CWAPTJUVOK34UG5UQLPJL4CRI5:/home/mclasen/.local/share/containers/storage/overlay/l/2HWVXHY5SPA2HNFGXDTCJUTJG3:/home/mclasen/.local/share/containers/storage/overlay/l/5XLTKVQQRS2K3SQWOYH6ZIKPPM:/home/mclasen/.local/share/containers/storage/overlay/l/THF7W74PD427AAS4SKC3Z2SHA6:/home/mclasen/.local/share/containers/storage/overlay/l/ZIVNMQFV3G3DG2ZXLZC6RKXRFP:/home/mclasen/.local/share/containers/storage/overlay/l/HQKN2QEBZL3KMFRMLBYHEARVBT:/home/mclasen/.local/share/containers/storage/overlay/l/JLIR5FE6FYMK2WK4AUL2MPQHNK:/home/mclasen/.local/share/containers/storage/overlay/l/OL2IDQ5CIMXDFQQKGKRREUXZ2E:/home/mclasen/.local/share/containers/storage/overlay/l/NJKUKCKL3GFXCEQRUTROEDC6FR:/home/mclasen/.local/share/containers/storage/overlay/l/2GO334FZOUNA3CHCZSOPWLRO2P:/home/mclasen/.local/share/containers/storage/overlay/l/RWEBXKPPPAGZNTUWI4OVH26Y5K:/home/mclasen/.local/share/containers/storage/overlay/l/6JBF5UOUIEM7AMCP45G4VIG53T:/home/mclasen/.local/share/containers/storage/overlay/l/5OQAJCATI4PDYC76H4K5TLFT2P:/home/mclasen/.local/share/containers/storage/overlay/l/I3DDWX2VH3JRR53NPEZHLQHVPA:/home/mclasen/.local/share/containers/storage/overlay/l/4HW4LU5TA6A3GQQY6RZVTMFUXU,upperdir=/home/mclasen/.local/share/containers/storage/overlay/5197d8fe4aa13451fa6abfbdecbe9f6107f54d58bec187a5706771f16f12a3d1/diff,workdir=/home/mclasen/.local/share/containers/storage/overlay/5197d8fe4aa13451fa6abfbdecbe9f6107f54d58bec187a5706771f16f12a3d1/work,userxattr,context="system_u:object_r:container_file_t:s0:c1022,c1023"": invalid argument

Steps to reproduce the issue

Steps to reproduce the issue
1.
2.
3.

Describe the results you received

Error: unable to start container "e924fa34ad6e561edda26ef2c537ea2cfc751e4dc7d30eb57e4ecc8964ea31fb": mounting storage for container e924fa34ad6e561edda26ef2c537ea2cfc751e4dc7d30eb57e4ecc8964ea31fb: creating overlay mount to /home/mclasen/.local/share/containers/storage/overlay/5197d8fe4aa13451fa6abfbdecbe9f6107f54d58bec187a5706771f16f12a3d1/merged, mount_data="lowerdir=/home/mclasen/.local/share/containers/storage/overlay/l/S2KNMSPQFIGNFT7EW46ZDTHRGK:/home/mclasen/.local/share/containers/storage/overlay/l/S2KNMSPQFIGNFT7EW46ZDTHRGK/../diff1:/home/mclasen/.local/share/containers/storage/overlay/l/ZGL47N2OPFGLUX33ILI2ROAS23:/home/mclasen/.local/share/containers/storage/overlay/l/GLTPEUXR5WXDUQLE6VJC64MHDL:/home/mclasen/.local/share/containers/storage/overlay/l/4CII473YAJHTQRF66LAQH3DFPM:/home/mclasen/.local/share/containers/storage/overlay/l/ZX3UINK3AESXULCLLW655JUQHU:/home/mclasen/.local/share/containers/storage/overlay/l/SOJHIOU42K7C5PTLYS4LGWZVXH:/home/mclasen/.local/share/containers/storage/overlay/l/5247CCTVQXEJPRSCOAVJKST2NU:/home/mclasen/.local/share/containers/storage/overlay/l/CWAPTJUVOK34UG5UQLPJL4CRI5:/home/mclasen/.local/share/containers/storage/overlay/l/2HWVXHY5SPA2HNFGXDTCJUTJG3:/home/mclasen/.local/share/containers/storage/overlay/l/5XLTKVQQRS2K3SQWOYH6ZIKPPM:/home/mclasen/.local/share/containers/storage/overlay/l/THF7W74PD427AAS4SKC3Z2SHA6:/home/mclasen/.local/share/containers/storage/overlay/l/ZIVNMQFV3G3DG2ZXLZC6RKXRFP:/home/mclasen/.local/share/containers/storage/overlay/l/HQKN2QEBZL3KMFRMLBYHEARVBT:/home/mclasen/.local/share/containers/storage/overlay/l/JLIR5FE6FYMK2WK4AUL2MPQHNK:/home/mclasen/.local/share/containers/storage/overlay/l/OL2IDQ5CIMXDFQQKGKRREUXZ2E:/home/mclasen/.local/share/containers/storage/overlay/l/NJKUKCKL3GFXCEQRUTROEDC6FR:/home/mclasen/.local/share/containers/storage/overlay/l/2GO334FZOUNA3CHCZSOPWLRO2P:/home/mclasen/.local/share/containers/storage/overlay/l/RWEBXKPPPAGZNTUWI4OVH26Y5K:/home/mclasen/.local/share/containers/storage/overlay/l/6JBF5UOUIEM7AMCP45G4VIG53T:/home/mclasen/.local/share/containers/storage/overlay/l/5OQAJCATI4PDYC76H4K5TLFT2P:/home/mclasen/.local/share/containers/storage/overlay/l/I3DDWX2VH3JRR53NPEZHLQHVPA:/home/mclasen/.local/share/containers/storage/overlay/l/4HW4LU5TA6A3GQQY6RZVTMFUXU,upperdir=/home/mclasen/.local/share/containers/storage/overlay/5197d8fe4aa13451fa6abfbdecbe9f6107f54d58bec187a5706771f16f12a3d1/diff,workdir=/home/mclasen/.local/share/containers/storage/overlay/5197d8fe4aa13451fa6abfbdecbe9f6107f54d58bec187a5706771f16f12a3d1/work,userxattr,context="system_u:object_r:container_file_t:s0:c1022,c1023"": invalid argument

Describe the results you expected

Describe the results you expected

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Honny1]

PTAL @mheon @Luap99

[Luap99]

@matthiasclasen Please provide the full info as requested, we always need podman info.

Did it used to work for you or did you just used it for the first time? I assume it is a regression. If so did it start with the podman 5.7-rc1 update? Can you downgrade podman to confirm? Did you reboot to see if it makes a difference?
Also can please provide a full reproducer? Does it happen if you create a new toolbox (maybe try one based on another image)

Also did you try to run a normal non toolbox container?

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[baude]

podman info results might also be helpful.

[debarshiray]

My understanding based on trying to debug it with him is:

Does it happen if you create a new toolbox (maybe try one based on another image)

Yes, a newly created Toolbx container did work.

Also did you try to run a normal non toolbox container?

This had worked:

$ podman run --rm registry.fedoraproject.org/fedora:44 echo "hello world"
hello world