[SDP] SinkWrite Flow Execution

Author: jaceklaskowski

docs/declarative-pipelines/DataflowGraph.md

@@ -1,6 +1,6 @@
 # DataflowGraph
 
-`DataflowGraph` is a [GraphRegistrationContext](GraphRegistrationContext.md) with [tables](#tables), [views](#views) and [flows](#flows) fully-qualified, resolved and de-duplicated.
+`DataflowGraph` is a [GraphRegistrationContext](GraphRegistrationContext.md) with [tables](#tables), [sinks](#sinks), [views](#views) and [flows](#flows) fully-qualified, resolved and de-duplicated.
 
 ## Creating Instance
 

docs/declarative-pipelines/FlowPlanner.md

@@ -31,12 +31,15 @@ plan(
  flow: ResolvedFlow): FlowExecution
 ```
 
-`plan` creates a [FlowExecution](FlowExecution.md) (for the given [ResolvedFlow](ResolvedFlow.md)) as follows:
+`plan` [looks up the output](DataflowGraph.md#output) for the [destination identifier](ResolutionCompletedFlow.md#destinationIdentifier) of the given [ResolvedFlow](ResolvedFlow.md) in this [DataflowGraph](#graph).
 
- FlowExecution | ResolvedFlow
--|-
- [BatchTableWrite](BatchTableWrite.md) | [CompleteFlow](CompleteFlow.md)
- [StreamingTableWrite](StreamingTableWrite.md) | [StreamingFlow](StreamingFlow.md)
+`plan` creates a [FlowExecution](FlowExecution.md) (for the given [ResolvedFlow](ResolvedFlow.md) and the [Output](Output.md)) as follows:
+
+ FlowExecution | ResolvedFlow | Output |
+-|-|-
+ [BatchTableWrite](BatchTableWrite.md) | [CompleteFlow](CompleteFlow.md) | [Table](Table.md)
+ [SinkWrite](SinkWrite.md) | [StreamingFlow](StreamingFlow.md) | [Sink](Sink.md)
+ [StreamingTableWrite](StreamingTableWrite.md) | [StreamingFlow](StreamingFlow.md) | [Table](Table.md)
 
 ---
 

docs/declarative-pipelines/SinkWrite.md

@@ -1,3 +1,53 @@
-# SinkWrite
+---
+title: SinkWrite
+---
 
-`SinkWrite` is...FIXME
+# SinkWrite Flow Execution
+
+`SinkWrite` is a [StreamingFlowExecution](StreamingFlowExecution.md) that writes a streaming `DataFrame` to a [Sink](#destination).
+
+`SinkWrite` represents a [StreamingFlow](StreamingFlow.md) with a [Sink](Sink.md) as the [output destination](ResolutionCompletedFlow.md#destinationIdentifier) at execution.
+
+When [executed](#startStream), `SinkWrite` starts a streaming query to append new rows to an [output table](#destination).
+
+## Creating Instance
+
+`SinkWrite` takes the following to be created:
+
+* <span id="identifier"> [TableIdentifier](FlowExecution.md#identifier)
+* <span id="flow"> [ResolvedFlow](StreamingFlowExecution.md#flow)
+* <span id="graph"> [DataflowGraph](DataflowGraph.md)
+* <span id="updateContext"> [PipelineUpdateContext](FlowExecution.md#updateContext)
+* <span id="checkpointPath"> [Checkpoint Location](StreamingFlowExecution.md#checkpointPath)
+* <span id="trigger"> [Streaming Trigger](StreamingFlowExecution.md#trigger)
+* <span id="destination"> [Destination](FlowExecution.md#destination) ([Sink](Sink.md))
+* <span id="sqlConf"> [SQL Configuration](StreamingFlowExecution.md#sqlConf)
+
+`SinkWrite` is created when:
+
+* `FlowPlanner` is requested to [plan a ResolvedFlow](FlowPlanner.md#plan)
+
+## Start Streaming Query { #startStream }
+
+??? note "StreamingFlowExecution"
+
+ ```scala
+ startStream(): StreamingQuery
+ ```
+
+ `startStream` is part of the [StreamingFlowExecution](StreamingFlowExecution.md#startStream) abstraction.
+
+`startStream` builds the logical query plan of this [flow](#flow)'s structured query (requesting the [DataflowGraph](#graph) to [reanalyze](DataflowGraph.md#reanalyzeFlow) this [flow](#flow)).
+
+`startStream` creates a `DataStreamWriter` ([Spark Structured Streaming]({{ book.structured_streaming }}/DataStreamWriter/)) with the following:
+
+`DataStreamWriter`'s Property | Value
+-|-
+ `queryName` | This [displayName](FlowExecution.md#displayName)
+ `checkpointLocation` option | This [checkpoint path](#checkpointPath)
+ `trigger` | This [streaming trigger](#trigger)
+ `outputMode` | [Append]({{ book.structured_streaming }}/OutputMode/#append) (always)
+ `format` | The [format](Sink.md#format) of this [output sink](#destination)
+ `options` | The [options](Sink.md#options) of this [output sink](#destination)
+
+In the end, `startStream` starts the streaming write query to this [output table](#destination).

docs/declarative-pipelines/StreamingFlowExecution.md

@@ -14,14 +14,15 @@ Used when:
 
 * `StreamingTableWrite` is requested to [start a streaming query](StreamingTableWrite.md#startStream)
 
-### Execute Streaming Query { #startStream }
+### Start Streaming Query { #startStream }
 
 ```scala
 startStream(): StreamingQuery
 ```
 
 See:
 
+* [SinkWrite](SinkWrite.md#startStream)
 * [StreamingTableWrite](StreamingTableWrite.md#startStream)
 
 Used when:
@@ -34,8 +35,11 @@ Used when:
 trigger: Trigger
 ```
 
+`Trigger` ([Structured Streaming]({{ book.structured_streaming }}/Trigger))
+
 See:
 
+* [SinkWrite](SinkWrite.md#trigger)
 * [StreamingTableWrite](StreamingTableWrite.md#trigger)
 
 Used when:
@@ -45,6 +49,7 @@ Used when:
 
 ## Implementations
 
+* [SinkWrite](SinkWrite.md)
 * [StreamingTableWrite](StreamingTableWrite.md)
 
 ## executeInternal { #executeInternal }
@@ -60,7 +65,7 @@ Used when:
 `executeInternal` prints out the following INFO message to the logs:
 
 ```text
-Starting [identifier] with checkpoint location [checkpointPath]"
+Starting [identifier] with checkpoint location [checkpointPath]
 ```
 
 `executeInternal` [starts the stream](#startStream) (with this [SparkSession](FlowExecution.md#spark) and [sqlConf](#sqlConf)).

docs/declarative-pipelines/StreamingTableWrite.md

@@ -1,6 +1,10 @@
-# StreamingTableWrite
+---
+title: StreamingTableWrite
+---
 
-`StreamingTableWrite` is a [StreamingFlowExecution](StreamingFlowExecution.md).
+# StreamingTableWrite Flow Execution
+
+`StreamingTableWrite` is a [StreamingFlowExecution](StreamingFlowExecution.md) that writes a streaming `DataFrame` to a [Table](#destination)..
 
 When [executed](#startStream), `StreamingTableWrite` starts a streaming query to append new rows to an [output table](#destination).
 
@@ -14,7 +18,7 @@ When [executed](#startStream), `StreamingTableWrite` starts a streaming query to
 * <span id="updateContext"> [PipelineUpdateContext](FlowExecution.md#updateContext)
 * <span id="checkpointPath"> [Checkpoint Location](StreamingFlowExecution.md#checkpointPath)
 * <span id="trigger"> [Streaming Trigger](StreamingFlowExecution.md#trigger)
-* <span id="destination"> [Output table](FlowExecution.md#destination)
+* <span id="destination"> [Destination](FlowExecution.md#destination) ([Table](Table.md))
 * <span id="sqlConf"> [SQL Configuration](StreamingFlowExecution.md#sqlConf)
 
 `StreamingTableWrite` is created when: