Microsoftstreaming

heartbeat

Stateful Stream Processing demonstration with Fabric Spark.

Uses Fabric RTI EventStreams, Spark Structured Streaming with RocksDB
A guided tutorial for real-time, stateful health monitoring using Microsoft Fabric Spark Structured Streaming.

Architecture

Heartbeat architecture diagram

Demo

Install the Fabric Jumpstart to deploy the EventStreams, Lakehouse and Notebook into your workspace. Run the notebook to get the 4 connection strings.

PowerShell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install git+https://github.com/microsoft/fabric-jumpstart.git@main#subdirectory=src/fabric_jumpstart
$env:FABRIC_JUMPSTART_TOKEN_CREDENTIAL = "AzureCliCredential"
$wsId = Read-Host "Enter your Fabric Workspace ID"
python -c "import fabric_jumpstart as js; js.install('stateful-streaming-rocksdb', workspace_id='$wsId', unattended=True)"

Once the jumpstart deploys, browse to the Fabric UI.

1

Start Spark Stream in Fabric Notebook

Run heartbeat_notebook from the Fabric UI, and copy the 4 connection strings it prints into the fields below. Spark will read the Event Stream and mark the device as healthy. If the device disconnects for 5 seconds, Spark immediately marks it as unhealthy.

↓

Paste the 4 connection strings from the notebook output below. This is a static website with no backend server, all connections will be established without leaving your browser.

1
BROWSER

Producer Write Connection

Your browser's heartbeat producers being added below will send events here.

Not Connected
2
FABRIC SPARK

Producer Read Connection

Spark will read the heartbeats from the browser producers from here.

Not Connected
3
FABRIC SPARK

Consumer Write Connection

Spark writes the computed health state here.

Not Connected
4
BROWSER

Consumer Read Connection

Your browser reads the processed health state from here.

Not Connected
↓
2

Add a Heartbeat Producer

This named heartbeat producer will send Events to EventStream.

↓
3

Health Report

Real-time health monitoring powered by Spark. Events only appear when state changes.

State Change Events

0 events
↓
4

Play around!

Scroll up to pause producers to watch them go unhealthy when heartbeat times out in RocksDB after 5 seconds. Restart to watch them initialize and go to healthy.

Note that Spark Streaming must always have one event arriving to progress the microbatch, so don't pause all producers!

Explore more scenarios at Fabric Jumpstart