Add transaction tracing support to block test command #9310

bshastry · 2025-10-14T19:31:08Z

PR description

This PR adds transaction tracing support to the block-test subcommand in evmtool, enabling detailed debugging of block test execution with opcode-level traces.

New Feature: `-t/--trace` flag

The block-test command now supports a -t or --trace flag that outputs detailed transaction execution traces in JSON format compatible with go-ethereum. This enables developers to:

Debug complex block test failures at the opcode level
Analyze gas consumption patterns
Inspect stack and memory state during execution
Compare execution behavior across different Ethereum clients

Usage

# Basic tracing
evmtool block-test -t test.json

# With stack and memory traces
evmtool block-test -t --trace-stack --trace-memory test.json

Example Trace Output

Traces are output to stderr in JSON format:

{"pc":7,"op":62,"gas":"0xf3c12b","gasCost":"0x0","memSize":0,"depth":1,"refund":0,"opName":"RETURNDATACOPY","error":"Out of bounds"}

Implementation Details

Integrates with existing TracerManager and StandardJsonTracer infrastructure
Correctly handles all transaction statuses per Ethereum protocol:
- INVALID transactions (wrong nonce, insufficient balance) → reject block
- FAILED transactions (REVERT, out of gas, stack underflow) → include in block with receipt
- SUCCESSFUL transactions → include in block with receipt
Trace format matches go-ethereum for compatibility with existing analysis tools
Behavior aligns with Besu's standard block import path and reference implementations

Testing

Manually tested with fuzzer derived blocktests (about 800 of them)

Both tests include transactions with runtime errors (stack underflow, out of gas, invalid opcodes) that are correctly traced and handled.

Fixed Issue(s)

N/A - This is a new feature enhancement, not fixing a reported issue.

Thanks for sending a pull request! Have you done the following?

Checked out our contribution guidelines?
Considered documentation and added the doc-change-required label to this PR if updates are required.
Considered the changelog and included an update if required.
For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests (N/A - no database changes)

Locally, you can run these tests to catch failures early:

spotless: ./gradlew spotlessApply
unit tests: ./gradlew build (evmtool has no specific unit tests for this command)
acceptance tests: ./gradlew acceptanceTest (not required for evmtool changes)
integration tests: ./gradlew integrationTest (not required for evmtool changes)
reference tests: ./gradlew ethereum:referenceTests:referenceTests (not applicable)
hive tests: Engine or other RPCs modified? (N/A - evmtool only)

Notes

This is a developer tooling enhancement for evmtool, not a change to core Besu functionality
Manually tested with complex Ethereum reference test cases
No changes to APIs, consensus, networking, or database
Documentation may be beneficial to show developers how to use the new tracing capability

Implements tracing functionality for the block-test subcommand in evmtool, allowing developers to debug block test execution with detailed transaction traces including opcodes, gas usage, stack, and memory state. New features: - Added -t/--trace-transactions flag to enable tracing during block test execution - Added --trace-memory, --trace-stack, --trace-returndata, --trace-storage options - Added --trace-output option to specify output file (default: stderr) - Integrated with StandardJsonTracer infrastructure - Traces are output in JSON format compatible with go-ethereum Implementation: The processBlockWithTracing() method processes transactions with full tracing while maintaining correct transaction status handling. Transactions are categorized as INVALID, FAILED, or SUCCESSFUL per Ethereum protocol: - INVALID transactions (wrong nonce, insufficient balance) reject the block - FAILED transactions (reverted execution) are included with receipts - SUCCESSFUL transactions are included with receipts This distinction ensures blocks with reverted transactions are correctly accepted, matching behavior of the standard block import path and reference implementations like go-ethereum. Conflict resolution: This commit was rebased onto main which added test summary reporting (hyperledger#9246). Both features are now merged: - Test summary reporting tracks pass/fail counts across all tests - Transaction tracing provides detailed execution traces when enabled - Both features work independently and complement each other Usage: evmtool block-test -t --trace-stack --trace-memory <test-file.json> evmtool block-test -t --trace-output=trace.jsonl <test-file.json> Tested with Ethereum reference test suite including complex scenarios with failed transactions. Trace output format matches go-ethereum for compatibility with existing tooling. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Bhargava Shastry <[email protected]>

Signed-off-by: Bhargava Shastry <[email protected]>

lu-pinto · 2025-11-03T15:28:29Z

Hi @bshastry first of all, thank you for your PR. I'm wondering what's the rationale for adding tracing support to block-test? Doesn't the state-test suffice for what you need? Take a look at an example here: https://github.com/hyperledger/besu/blob/main/ethereum/evmtool/src/test/resources/org/hyperledger/besu/evmtool/state-test/clz-opcode.json This already supports tracing out-of-the-box.
What's your use case?

bshastry · 2025-11-04T09:24:51Z

Hi @bshastry first of all, thank you for your PR. I'm wondering what's the rationale for adding tracing support to block-test? Doesn't the state-test suffice for what you need? Take a look at an example here: https://github.com/hyperledger/besu/blob/main/ethereum/evmtool/src/test/resources/org/hyperledger/besu/evmtool/state-test/clz-opcode.json This already supports tracing out-of-the-box. What's your use case?

Hi @lu-pinto Thank you for your feedback. The rationale for this PR is to be able to compare clients' execution outcome across transactions, right now we only do this at the statetest level i.e., a pre state, a single tx, and a post state. The idea is to generalize this to a pre state, a sequence of blocks, and a post state and compare the resulting trace between clients. This PR permits such a trace to be obtained. We use this infrastructure to then obtain and compare traces for multi block processing (still at the tx level, but it can shed light on state effects that are not captured by statetests) between besu and say geth and nethermind. This is usually orchestrated by a fuzzer such as goevmlab that is already capable of comparing clients' processing at the statetest level, but if this PR is merged then even at the blocktest level.

lu-pinto · 2025-11-04T10:33:34Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+  @Option(
+      names = {"--trace-storage"},
+      description = "Include storage changes in traces")
+  private boolean traceStorage = false;


These tracing configs are available from parentCommand see https://github.com/hyperledger/besu/pull/9310/files#diff-487cbe55e187e3d2cbc8008f3d9775534d9e70579345ff154dbebb057e7973aeR140

Thank you, I inherited from the parentCommand instead of creating new options.

lu-pinto · 2025-11-04T10:35:07Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+
+  @Option(
+      names = {"--trace-output"},
+      description = "Output file for traces (default: stderr)")


why did you choose stderr? I think this is a great option to add to all subcommands actually, I would prefer if it were moved to EvmToolCommand instead.

Fuzzers read from process pipes to save disk I/O for performance reasons (maximizes test throughput). stdout may contain diagnostics, hence stderr. Could we may be address the addition of trace-output to evmtoolcommand in a separate PR?

Fuzzers read from process pipes to save disk I/O for performance reasons (maximizes test throughput). stdout may contain diagnostics, hence stderr.

But if you have the option to output to a file you can just pipe the file to the fuzzer no? My main problem is inconsistency with other commands as they all go to System.out by default.

re: moving option: sure, let's treat the move to the parent command separately.

Also, I see you are only sending traces to this output but all the other prints are going to parentCommand.out. Why not replacing parentCommand.out with the PrintStream based on this option and send everything there?

The fuzzer consumes the trace by using a strict jsonl parser. However, if the trace were to be sent to parentCommand.out, the fuzzer's jsonl parser errors out because non jsonl lines (info/error logging that is sent to parentCommand.out is interspersed with actual traces). Hence the separation (and use of stderr).

Considering test_name Block 1 (0x1234...) Imported in 1.23 ms (456.78 MGas/s) {"pc":0,"op":96,"gas":"0x5f5e100","gasCost":"0x1",...} {"pc":2,"op":96,"gas":"0x5f5e0ff","gasCost":"0x2",...} Block 2 (0x5678...) Imported in 0.98 ms (789.01 MGas/s) {"pc":4,"op":96,"gas":"0x5f5e0fd","gasCost":"0x1",...} ... Chain import successful - test_name {"test":"test_name","pass":true,"fork":"mainnet","duration":1234,...}

Signed-off-by: Bhargava Shastry <[email protected]>

lu-pinto · 2025-11-04T11:49:56Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+  @Option(
+      names = {"-t", "--trace-transactions"},
+      description = "Enable transaction tracing to stderr")
+  private boolean enableTracing = false;


Seems to be doing the same as final Boolean showJsonResults = false; from EvmToolCommand

Thank you, fixed it.

lu-pinto · 2025-11-04T11:57:57Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+        output.close();
+      }
+    }
+  }


I think you created this class mostly because of the close? I don't think you need it, since you are already using an autoFlush PrintStream?

If it doesn't work consider using a PrintStream instead of a PrintWriter with a FileOutputStream

lu-pinto · 2025-11-04T12:18:32Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+      } else {
+        parentCommand.out.println("Chain import successful - " + test);
+      }
+    } finally {


finally block adds unnecessary complexity IMO. In Java most exceptions are checked so you have to handle them, for a case like this where you are running tests if there's an unchecked one it's best to let them propagate and crash the app. What would be the benefit of getting a summary of results if there's a developer error?

Removed the finally block. Clean up happens at the end of normal execution flow.

lu-pinto · 2025-11-04T12:22:48Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

-            parentCommand.out.printf(
-                "Block %d (%s) Rejected (correctly)%n",
-                block.getHeader().getNumber(), block.getHash());
+            // Original block import logic (from main branch)


What are these comments about? (from main branch) / (from feature branch) ?

Sorry, fixed

lu-pinto · 2025-11-04T12:40:42Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+          if (enableTracing && tracerManager != null) {
+            // Process block with tracing
+            importResult =
+                processBlockWithTracing(


This feels like it adds duplicate code which is hard to sanity check and maintain in the future. Would you be happy to call blockImporter.importBlock(context, block, validationMode, validationMode); if you could add in the tracer before calling into it?
I think it might be doable and if you are happy with that I can give it a shot.

I have made this change, please let me know if there are any issues.

This commit addresses code review feedback by refactoring the block-test tracing implementation to use Besu's standard BlockImportTracerProvider plugin infrastructure instead of custom transaction processing logic. Changes: - Remove duplicate tracing option definitions (now uses parent command's inherited options via @ParentCommand) - Create BlockTestTracerProvider implementing BlockImportTracerProvider - Create BlockAwareOperationTracerAdapter to wrap StreamingOperationTracer - Remove processBlockWithTracing method (~138 lines of duplicate code) - Remove finally block complexity - cleanup happens at end of normal flow - Remove all merge conflict comments (// from feature branch, etc.) - Use standard blockImporter.importBlock() for both tracing and non-tracing - Register tracer provider with ServiceManager when tracing is enabled Benefits: - Eliminates code duplication between tracing and non-tracing paths - Consistent with Besu's plugin architecture - Transaction boundaries handled automatically in AbstractBlockProcessor - Better maintainability - single source of truth for block processing - Cleaner separation of concerns Tested with: - Block tests with tracing enabled (memory, storage, stack) - Block tests without tracing - Verified test summary metrics (gasUsed, txCount, blockCount) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Bhargava Shastry <[email protected]>

Signed-off-by: Bhargava Shastry <[email protected]>

Address PR review feedback about resource management in BlockTestTracerManager. The close() method was unconditionally closing all PrintWriters, which is problematic when wrapping System.err (a shared system resource). Changes: - Add shouldCloseOutput parameter to BlockTestTracerManager constructor - Only close file-based PrintWriters, not System.err - Document stdout/stderr separation rationale (JSONL purity for fuzzers) - Simplify tracing enablement using parentCommand.showJsonResults Technical context: - Trace output (pure JSONL) must be separated from test status messages - Test status goes to stdout (parentCommand.out), traces to stderr or file - Mixing them would break JSONL parsers used by fuzzing tools - File writers need explicit close() for proper resource cleanup - System.err should never be closed as it's shared across the JVM Tested with: - File output: 53,932 trace lines written and file properly closed - Stderr output: Traces correctly written without closing System.err Signed-off-by: $(git config user.name) <$(git config user.email)> Signed-off-by: Bhargava Shastry <[email protected]>

…ature/blocktest-tracing

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto · 2025-11-11T10:32:48Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

            .getByName(spec.getNetwork());

    final MutableBlockchain blockchain = spec.getBlockchain();
+    ProtocolContext context = spec.getProtocolContext();


there was a compilation issue here. Fixed in my latest commit

lu-pinto · 2025-11-11T10:33:37Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+    @Override
+    public void traceStartBlock(
+        final WorldView worldView,
+        final org.hyperledger.besu.plugin.data.BlockHeader blockHeader,
+        final BlockBody blockBody,
+        final Address miningBeneficiary) {
+      // No-op: StreamingOperationTracer doesn't need block-level events
+    }
+
+    @Override
+    public void traceEndBlock(
+        final org.hyperledger.besu.plugin.data.BlockHeader blockHeader, final BlockBody blockBody) {
+      // No-op: StreamingOperationTracer doesn't need block-level events
+    }
+
+    @Override
+    public void traceStartBlock(
+        final WorldView worldView,
+        final org.hyperledger.besu.plugin.data.ProcessableBlockHeader processableBlockHeader,
+        final Address miningBeneficiary) {
+      // No-op: StreamingOperationTracer doesn't need block-level events
+    }


No need to define this, the implemented interface already has empty methods. Removed in my latest commit

lu-pinto · 2025-11-11T10:35:39Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+      currentTracer =
+          new StreamingOperationTracer(
+              output,
+              OpCodeTracerConfigBuilder.create()


You are creating an OpcodeTracerConfig from scratch so you need to specify every single field. Tracing was failing because traceOpcodes option was not defined. Fixed now in my latest commit.

lu-pinto · 2025-11-11T10:37:12Z

ethereum/evmtool/src/main/java/org/hyperledger/besu/evmtool/BlockchainTestSubCommand.java

+     */
+    public StreamingOperationTracer createTracer() {
+      currentTracer =
+          new StreamingOperationTracer(


nit: doesn't make much sense to me - why do you use a StreamingOperationTracer if you only flush at the end? Is only flushing at the end a requirement?

lu-pinto

This looks pretty good now @bshastry! I also made some simplifications in the code in my last commit.

I will add another commit to print the trace as the code is being traced - that way we can avoid flushing at the end. If this does not work for you feel free to remove the commit.

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto

@bshastry feel free to keep working on it or say the word and I'll merge it

macfarla · 2025-11-16T23:34:58Z

@bshastry are you ready for this to be merged?

bshastry · 2025-11-17T23:46:36Z

@bshastry are you ready for this to be merged?

I'll do one final pass. Hopefully will be done by the end of the week. Sorry to keep you waiting.

macfarla · 2025-11-18T00:29:50Z

@bshastry are you ready for this to be merged?

I'll do one final pass. Hopefully will be done by the end of the week. Sorry to keep you waiting.

no worries - just checking you weren't waiting on us! I'll change it to draft while you're still working on it, you can switch it back to ready for review when you're done

bshastry · 2025-11-18T09:04:49Z

@lu-pinto @macfarla This PR is ready to be merged. I have run a moderate fuzzer test suite on the PR with about 1000 different block tests to ensure the tracing works as expected (and can be compared against other EL clients). Thank you for helping me with this PR 🙏

bshastry force-pushed the feature/blocktest-tracing branch from 6224044 to d35158c Compare October 15, 2025 09:38

bshastry force-pushed the feature/blocktest-tracing branch from d35158c to bd27f72 Compare October 17, 2025 10:27

bshastry added 2 commits October 17, 2025 14:31

fixUp

b7ade94

Signed-off-by: Bhargava Shastry <[email protected]>

fixUp: Mark tx boundary

5c8b5a7

Signed-off-by: Bhargava Shastry <[email protected]>

macfarla assigned lu-pinto Nov 2, 2025

lu-pinto reviewed Nov 4, 2025

View reviewed changes

Inherit parent command subcommands for showmemory/stack etc

1acbdd4

Signed-off-by: Bhargava Shastry <[email protected]>

bshastry force-pushed the feature/blocktest-tracing branch from 2241260 to 1acbdd4 Compare November 4, 2025 11:48

lu-pinto reviewed Nov 4, 2025

View reviewed changes

bshastry and others added 5 commits November 4, 2025 14:16

Merge branch 'main' into feature/blocktest-tracing

aa13c7b

Signed-off-by: Bhargava Shastry <[email protected]>

Merge remote-tracking branch 'fork/feature/blocktest-tracing' into fe…

c5ce885

…ature/blocktest-tracing

Fix issues with OpcodeConfigTracer, compilation and refactorings

66056cd

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto reviewed Nov 11, 2025

View reviewed changes

lu-pinto previously approved these changes Nov 11, 2025

View reviewed changes

use PrintStream instead of PrintWriter

5e1c639

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto dismissed their stale review via 5e1c639 November 11, 2025 11:47

Merge branch 'main' into feature/blocktest-tracing

ce1461a

lu-pinto approved these changes Nov 11, 2025

View reviewed changes

lu-pinto reviewed Nov 11, 2025

View reviewed changes

macfarla assigned bshastry and unassigned lu-pinto Nov 11, 2025

macfarla marked this pull request as draft November 18, 2025 00:30

Merge branch 'main' into feature/blocktest-tracing

5f6b2e4

bshastry marked this pull request as ready for review November 18, 2025 09:03

macfarla assigned lu-pinto and unassigned bshastry Nov 18, 2025

lu-pinto enabled auto-merge (squash) November 18, 2025 10:56

lu-pinto merged commit 5f8ceec into hyperledger:main Nov 18, 2025
46 checks passed

Add transaction tracing support to block test command #9310

Add transaction tracing support to block test command #9310

Uh oh!

Conversation

bshastry commented Oct 14, 2025

PR description

New Feature: -t/--trace flag

Usage

Example Trace Output

Implementation Details

Testing

Fixed Issue(s)

Thanks for sending a pull request! Have you done the following?

Locally, you can run these tests to catch failures early:

Notes

Uh oh!

lu-pinto commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bshastry commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lu-pinto Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lu-pinto Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lu-pinto Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lu-pinto Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lu-pinto Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lu-pinto left a comment

Choose a reason for hiding this comment

Uh oh!

lu-pinto left a comment

Choose a reason for hiding this comment

Uh oh!

New Feature: `-t/--trace` flag

lu-pinto commented Nov 3, 2025 •

edited

Loading

bshastry commented Nov 4, 2025 •

edited

Loading

lu-pinto Nov 4, 2025 •

edited

Loading

lu-pinto Nov 4, 2025 •

edited

Loading

lu-pinto Nov 4, 2025 •

edited

Loading

lu-pinto Nov 11, 2025 •

edited

Loading

lu-pinto Nov 11, 2025 •

edited

Loading

macfarla commented Nov 18, 2025 •

edited

Loading

bshastry commented Nov 18, 2025 •

edited

Loading