[SDP] PipelineEventSender

Author: jaceklaskowski

docs/declarative-pipelines/.pages

@@ -1,4 +1,5 @@
 title: Declarative Pipelines
 nav:
     - index.md
+    - configuration-properties.md
     - ...

docs/declarative-pipelines/PipelineEventSender.md

@@ -0,0 +1,48 @@
+# PipelineEventSender
+
+`PipelineEventSender` is used by [PipelinesHandler](PipelinesHandler.md) to [send pipelines execution progress events back to a Spark Connect client asynchronously](#sendEvent).
+
+`PipelineEventSender` uses [spark.sql.pipelines.event.queue.capacity](./configuration-properties.md#PIPELINES_EVENT_QUEUE_CAPACITY) configuration property to control the depth of the event queue.
+
+## Creating Instance
+
+`PipelineEventSender` takes the following to be created:
+
+* <span id="responseObserver"> `StreamObserver[ExecutePlanResponse]`
+* <span id="sessionHolder"> `SessionHolder`
+
+`PipelineEventSender` is created when:
+
+* `PipelinesHandler` is requested to [start a pipeline run](PipelinesHandler.md#startRun)
+
+## queueCapacity { #queueCapacity }
+
+`queueCapacity` is the value of [spark.sql.pipelines.event.queue.capacity](./configuration-properties.md#PIPELINES_EVENT_QUEUE_CAPACITY) configuration property.
+
+Used when:
+
+* `PipelineEventSender` is requested to [shouldEnqueueEvent](#shouldEnqueueEvent)
+
+## Send Pipeline Execution Progress Event { #sendEvent }
+
+```scala
+sendEvent(
+  event: PipelineEvent): Unit
+```
+
+`sendEvent`...FIXME
+
+---
+
+`sendEvent` is used when:
+
+* `PipelinesHandler` is requested to [start a pipeline run](PipelinesHandler.md#startRun)
+
+### shouldEnqueueEvent { #shouldEnqueueEvent }
+
+```scala
+shouldEnqueueEvent(
+  event: PipelineEvent): Boolean
+```
+
+`shouldEnqueueEvent`...FIXME

docs/declarative-pipelines/PipelinesHandler.md

@@ -1,3 +1,7 @@
+---
+subtitle: Spark Connect Endpoint
+---
+
 # PipelinesHandler
 
 `PipelinesHandler` is used to [handle pipeline commands](#handlePipelinesCommand) in [Spark Connect]({{ book.spark_connect }}) ([SparkConnectPlanner]({{ book.spark_connect }}/server/SparkConnectPlanner), precisely).
@@ -87,7 +91,7 @@ startRun(
 
 `startRun` finds the [GraphRegistrationContext](GraphRegistrationContext.md) by `dataflowGraphId` in the [DataflowGraphRegistry](DataflowGraphRegistry.md) (in the given `SessionHolder`).
 
-`startRun` creates a `PipelineEventSender` to send pipeline events back to the Spark Connect client (_Python pipeline runtime_).
+`startRun` creates a [PipelineEventSender](PipelineEventSender.md) to [send pipeline execution progress events back to the Spark Connect client](PipelineEventSender.md#sendEvent) (_Python pipeline runtime_).
 
 `startRun` creates a [PipelineUpdateContextImpl](PipelineUpdateContextImpl.md) (with the `PipelineEventSender`).
 

docs/declarative-pipelines/configuration-properties.md

@@ -0,0 +1,17 @@
+# Configuration Properties
+
+**Configuration properties** (aka **settings**) for [Spark Declarative Pipelines](index.md).
+
+## <span id="PIPELINES_EVENT_QUEUE_CAPACITY"> event.queue.capacity { #spark.sql.pipelines.event.queue.capacity }
+
+**spark.sql.pipelines.event.queue.capacity**
+
+Capacity of the event queue of a pipelined execution. When the queue is full, non-terminal `PipelineEvent`s will be dropped.
+
+Default: `1000`
+
+Use [SQLConf.PIPELINES_EVENT_QUEUE_CAPACITY](../SQLConf.md#PIPELINES_EVENT_QUEUE_CAPACITY) to reference it
+
+Used when:
+
+* `PipelineEventSender` is [created](PipelineEventSender.md#queueCapacity)