REPL & Sandbox: structured “display” channel for rich outputs

[rgbkrk]

Motivation

Right now the REPL and Sandbox only return strings. Two things are missing for notebook use cases:

1. execute_result: the rich form of the final eval result (currently coerced to string).
2. display: a way for sandboxed code to proactively send structured outputs back to the host, not just one final JS value.

Borrowing directly from the Deno Jupyter implementation, IPython, and the Jupyter protocol, I'd love a display() function injected into the sandbox. When user code calls display(obj), the host receives a message with { data: { "text/plain": "...", "image/png": "...", ... }, metadata }. The host decides how to render the text, image, plot, html, etc.

In practice, this is so that we can show:

• Rich tables (text/html, application/vnd.dataresource+json)
• Plots (image/png, application/vnd.vegalite.v5+json)
• Interactive outputs

This lets notebooks and dashboards work with the same primitives, instead of bolting on custom stdout hacks.

The Deno Jupyter kernel has many of the building blocks with display(), format() / media bundle, and the$display symbol for rich outputs and library opt-in formatting. The difference here is transport method.

Proposal

JSON REPL

For the JSON REPL:

• Update RunSuccess to optionally carry a MIME bundle (an ExecuteResult).
• Add a new side-channel for Display and DisplayUpdate messages, emitted when user code calls display()

// Host <- REPL
type ReplMessage =
  | { type: "Run" | "RunSuccess" | "RunFailure" | "Error"; ...existing }
  | {
      type: "Display";
      id?: string;          // display_id for updates
      data: Record<string, unknown>;  // MIME bundle / Multimedia bundle
      metadata?: Record<string, unknown>;
      transient?: Record<string, unknown>; // lightweight hints
      buffers?: Uint8Array[]; // optional binary blobs (avoid base64 when possible)
    }
  | {
      type: "DisplayUpdate";
      id: string;
      data: Record<string, unknown>;
      metadata?: Record<string, unknown>;
      buffers?: Uint8Array[];
    };

Execution semantics

• User code in the REPL can call display(value, opts?). This is separate from the normal return value.
• The normal return still yields an ExecuteResult (rich MIME bundle form of the last expression).
• display() posts a Display message immediately. If opts.display_id is provided, later calls with the same id emit DisplayUpdate.

Ideally, we'd prefer side-band binary for images (buffers) with media types like image/png, arrow, etc. Textual media stays JSON.

Host API

TBD for Sandbox, maybe something like this:

repl.on('display', callback)

Library ecosystem hook

To enable rich rendering without bespoke adapters, standardize on:

$display symbol (already used by Deno Jupyter) for objects to return their own media bundle.

A canonical format(obj, raw?) util that:

• Checks $display
• Handles common cases (HTML elements, SVG, canvas, images, Vega/Vega-Lite, DataFrames → application/vnd.dataresource+json + text/html)
• Falls back to "text/plain" (stringifed inspect) or "application/json" if serializable

This mirrors Python’s _repr_html_ success (pandas popularized it) and lets libs opt-in to richer views without coupling to any front-end. Deno already has much of this code in cli/js/40_jupyter.js.

[rgbkrk]

A little more media type stuff and the "why" jupyter uses it this way. We've got the code here in Deno for Jupyter, but I figured I'd outline this again just to make things more clear.

Generally a single output has multiple representations because different frontends can only handle certain output types. Take for example the common dataframe.

Underneath the hood objects like this will have text/plain and text/html most commonly. For the sake of this example I'm going to add in text/markdown as well.

{
  'text/plain': '    a  b\n0   1  2\n1  10  7',
  'text/html':  '<table border="1" class="dataframe">...',
  'text/markdown': '|    |   a |   b |\n|---:|----:|----:|\n|  0 |   1 |   2 |\n|  1 |  10 |   7 |'
}

Each of those can be used by different "UIs". Terminal can consume plain:

  'text/plain': '    a  b\n0   1  2\n1  10  7',

Web UI can consume HTML

  'text/html':  '<table border="1" class="dataframe">...'

and AI is particularly adept at reading text/markdown:

  'text/markdown': '|    |   a |   b |\n|---:|----:|----:|\n|  0 |   1 |   2 |\n|  1 |  10 |   7 |'

Putting all these together means that each consumer of a library can participate in the REPL whether they're an LLM, a web user, or a terminal fiend.