Builds randomly failing without error or log in Teamcity CI

[junekpavel]

Hey, we're experiencing quite specific errors in the Teamcity CI builds. I'm not exactly sure when this started, but it’s definitely happening for more than half of a year now.

---

We’re running build via affected:build command:

nx affected:build --configuration=staging-dev --base=develop --head=HEAD

Sometimes, a random application or library build fails with no reason. There's no output of build command in build log or anywhere else.

The final result of command is something like this:

    >  NX   Running target build for project XXX and 343 tasks it depends on failed
   
      Tasks not run because their dependencies failed or --nx-bail=true:
   
      - XXX:build:staging-dev
      - YYY:build:staging-dev
   
      Failed tasks:
   
      - YYY:build-package:staging-dev
   
      View structured, searchable error logs at [NX cloud link]
   
   info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

But when I search in log for the result of failed task (YYY:build-package:staging-dev), there is no specific reason. It's more like the command start and gets terminated in process, the log looks like this:

   > nx run YYY:build-package:staging-dev
   
       Components styles sourcemaps are not generated when styles optimization is enabled.
   - Generating browser application bundles (phase: setup)...
   
 

And that's it. Nothing extra is available. There is no more additional information, even not in the nx cloud:

    Components styles sourcemaps are not generated when styles optimization is enabled.
- Generating browser application bundles (phase: setup)...

---

Important to say, that it's not just about builds, but any tasks triggered by nx. And it's not any specific library or application, but it seems like it's totally random. When the same command is triggered again, it always fixes the problem and the build succeeds, or it fails because some other project fails with the same problem.

To add one more example, this is partial log of nx affected:test --ci --base=develop --head=HEAD:

11:01:22
11:01:22   > nx run AAA:test --ci
11:01:22
11:01:22   No tests found, exiting with code 0
11:01:24
11:01:24   > nx run XXX:test --ci
11:01:24
11:01:25
11:01:25   > nx run BBB:test --ci
11:01:25
11:01:25   No tests found, exiting with code 0
11:01:25
11:01:25   > nx CCC:test --ci
11:01:25
11:01:25   No tests found, exiting with code 0
...
11:06:02    >  NX   Running target test for 275 projects failed
11:06:02
11:06:02      Failed tasks:
11:06:02
11:06:02      - XXX:test
11:06:02
11:06:03      View structured, searchable error logs at [nx cloud]

---

I'm curious if anyone has experienced similar issues.

Is there someone who is willing to help us figure out the problem, or at least point us in the right direction to find the cause of this issue?

We're using latest version of both Teamcity and nx, but it seems like it's irrelevant to the issue.

If any other relevant information is necessary, I'd be happy to provide it.

[EoinGriffin-AI]

I'm getting this a lot since upgrading from nx 16 to 22.
@junekpavel Did you ever figure out what's going on here or how to work around it?