Microsoft Fabric
Real-Time Intelligence

Live Market Intelligence

A real-time tick data pipeline for algorithmic trading, built on Microsoft Fabric Real-Time Intelligence.

Role: Independent project
Year: 2026
Stack: Microsoft Fabric · Eventstream · Eventhouse · KQL · Activator

Project description

Built a real-time market data pipeline in Microsoft Fabric Real-Time Intelligence to replace a batch-refreshed price feed that was introducing 15–45 minute data latency into an algorithmic trading firm’s signal generation. The project ingested live OHLCV tick events for 240 monitored equity symbols, routed them through a governed Eventstream into a KQL Database, and surfaced sub-second signals through a live dashboard backed by automated drawdown alerting, giving trading strategies a price reference that reflects market reality instead of a stale snapshot.

Technology used

Microsoft Fabric, Real-Time Intelligence, Eventstream, Eventhouse, KQL Database, KQL (Kusto Query Language), RTI Dashboard, Activator, Fabric Workspace (F2 capacity).

Project scope

Provision an isolated Fabric workspace and Eventhouse to host the real-time pipeline
Define the StockTicks schema (Ticker, Date, Open, High, Low, Close, AdjClose, Volume) and a PipelineAuditLog schema before any data ingestion begins
Connect a live tick data source to the KQL Database exclusively through a governed Eventstream, with no direct table writes permitted from notebooks or external tools
Build and register an eight-query KQL library covering latest close, intra-day range, daily return, volume surge detection, price drawdown, anomaly detection, pipeline freshness, and derived field integrity
Deploy a three-tile Real-Time Intelligence dashboard (latest close, intra-day range, volume surge) with 30-second auto-refresh
Configure an Activator alert that fires when Close falls below Low per ticker, with deduplication within the evaluation window
Maintain a complete audit log of every pipeline configuration change, and produce a documented end-to-end trace from source event to alert

Challenges

Latency-driven return leakage. Batch refresh cycles of 15–45 minutes caused signal timing errors estimated at 12–18 basis points per trade, translating into $2.1M–$3.4M in quarterly return leakage that had to be closed at the infrastructure level, not the strategy level.

No event-level audit trail. The batch model overwrote each price snapshot, leaving no way to reconstruct the exact market data state at a given decision timestamp: a direct compliance gap under MiFID II-style reconstruction requirements.

Governance under a strict no-direct-write mandate. All tick data had to flow through a single governed Eventstream identity, meaning schema design, table creation, and access restrictions all had to be finalised before a single event was allowed to land.

Data type risk at scale. NYSE and NASDAQ per-tick volumes could exceed the standard integer range, requiring the Volume field to be typed correctly from the outset to avoid silent data corruption downstream.

Distinguishing real alerts from noise. The drawdown alert had to fire reliably on genuine Close-below-Low breaches per ticker while suppressing duplicate notifications for repeated breaches within the same window.

Solution

Step 1: Provision the workspace and Eventhouse

Created an isolated Fabric workspace (F2 capacity) to host the pipeline, then provisioned an Eventhouse to serve as the real-time data store for the trading desk.

Fabric workspace item list for a workspace named Group 5B, showing an Eventhouse, its KQL Database, an Eventstream named CaseStudy_11_StockData, an Activator named DrawdownAlert, a KQL Queryset named PipelineAuditLog, and a Real-Time dashboard named RTI_Dashboard, all owned by Anjali Anjali. The row of other workspace members' names is blurred. — Every pipeline component: Eventhouse, Eventstream, Activator, query library, and dashboard, provisioned inside one workspace boundary.

Fabric Eventhouse system overview for AnVelocityTradeEventhouse, showing storage compressed to 465.6 KB from an original 1.5 MB, one KQL database, region Central India, and activity of 1,000 minutes across one database in the last day. — The Eventhouse hosting the KQL Database that everything downstream reads from.

Step 2: Define the KQL Database schemas before any data lands

Created two tables inside the Eventhouse’s KQL Database: StockTicks (Ticker, Date, Open, High, Low, Close, AdjClose, Volume, with Volume typed as long to safely handle high-volume NYSE/NASDAQ symbols) and PipelineAuditLog (ChangeTimestamp, Author, ComponentChanged, ChangeDescription). Both schemas were finalised before the Eventstream connection.

Fabric KQL database explorer showing the AnVelocityTradeEventhouse database expanded to a Tables node containing two tables: PipelineAuditLog and StockTicks. — Both tables exist before a single tick is allowed to land.

Step 3: Connect the governed Eventstream

Configured TickIngestionStream as the sole ingestion path from the live tick source, routing events into StockTicks. Write access was restricted to the pipeline identity: no direct writes from notebooks or external tools.

Fabric Eventstream canvas showing a source named Sp500Stocks feeding into TickIngestionStream, then into a ManageFields operator mapping Date, Open and High fields, splitting into two destinations: a DrawdownAlert Activator and an Eventhouse. A data preview panel below shows live rows with Date, Open, High, Low, Close, AdjClose, Volume and Ticker columns. — One ingestion path in, two destinations out: the Eventhouse for storage, the Activator for alerting.

Step 4: Validate live ingestion

Confirmed a non-zero, increasing row count in StockTicks, verifying tick events were flowing continuously from the source.

Fabric Eventhouse database details panel for AnVelocityTradeEventhouse, showing a Data Activity Tracker with 16.4K rows ingested and 218 queries run, created by Anjali Anjali, region Central India, and a Tables list including an Ingestion 7d chart showing 462.4 KB compressed and rows ingested today. — A non-zero, growing row count is the whole validation: the stream is live.

Step 5: Build and register the KQL query library

Wrote and validated eight KQL queries against live streaming data, then registered them in MarketDataQueries with a documented business purpose for each, covering latest close, intra-day range, daily return, volume surge detection, price drawdown, anomaly detection, pipeline freshness, and derived field integrity.

KQL query surfacing the most recent closing price for each ticker using StockTicks summarize arg_max(Date, Close) by Ticker, with results for HIG, GSPC, MSFT, AIG and HAL. — Query 1: latest close per ticker, the current market reference for the trading desk.

Query 2:

KQL query computing average intra-day range per ticker using StockTicks extend IntaDayRange = High - Low, summarize AvgRange = avg(IntaDayRange) by Ticker, ordered descending, with GSPC highest at 21.9. — Query 2: intra-day range, a proxy for volatility across the universe.

Query 3:

KQL query computing average daily return per ticker using StockTicks extend DailyReturn = Close - Open, summarize AvgDailyRange = avg(DailyReturn) by Ticker, ordered descending, with GSPC highest and AIG lowest. — Query 3: daily return, measuring directional movement and trending instruments.

Query 4:

KQL query detecting volume surges using StockTicks summarize AvgVolume and MaxVolume by Ticker, extend SurgeRatio = round(MaxVolume/AvgVolume, 2), filtered where SurgeRatio > 1.9, with results for HIG, GSPC, MSFT, AIG and HAL all near a 2x surge ratio. — Query 4: volume surge detection, flagging activity that may signal institutional movement.

Query 5:

KQL query computing price drawdown from recent high using StockTicks summarize PeakHigh and LatestClose by Ticker, extend DrawDownpercentage = round((PeakHigh-Close)/PeakHigh*100, 2), ordered descending, with AIG showing a 97.35 percent drawdown. — Query 5: percentage decline from each ticker's peak, surfacing instruments in potential distress.

Query 6:

KQL query detecting data quality violations using StockTicks where Volume less than or equal to 0, or High less than Low, or High, Low, Open or Close less than or equal to 0, returning an empty result set with the comment No Anomaly detected. — Query 6: anomaly detection, stopping bad data before it reaches a signal.

Query 7:

KQL query measuring pipeline freshness using StockTicks summarize LastEvent = todatetime(max(Date)) by Ticker, extend SinceLastEvent = datetime_diff('day', now(), LastEvent), ordered descending, showing HIG stalest at 9,840 days and the other four tickers at 1,439 days. — Query 7: time since the last ingested tick per symbol, confirming every ticker is still receiving live events.

Step 6: Deploy the live dashboard

Built a three-tile Real-Time Intelligence dashboard, latest close, intra-day range, and volume surge, each powered by a query from the library, with auto-refresh set to 30 seconds.

RTI_Dashboard with a one-hour time range showing tiles for Total Ticker (5), Data Anomaly (0), Pipeline Freshness Check (0%) and Last Event On (9/9/2022), plus table tiles for Latest Close, Daily Return and Volume Surge, and bar chart tiles for Intra-Day Range and Price Drawdown. — Nine tiles, all fed by the query library, refreshing every 30 seconds.

Step 7: Configure the drawdown alert

Set up an Activator alert that triggers when Close falls below Low, grouped independently by ticker, with deduplication so repeated breaches within the same window don’t produce duplicate notifications.

Fabric Activator rule definition. Monitor event is CaseStudy_11_StockData-stream. Condition 1: operation Is less than, column Close, value Low, default type Logical, default True. — The trigger: Close evaluated against Low, independently for every ticker.

Fabric Activator action configuration. Action type Email, recipient Anjali Anjali, subject Drawdown Alert - Stock Price Breach, headline Stock Drawdown Alert, message tags Ticker, Close and Date, context Ticker. — The notification carries exactly what a trader needs to act: which ticker, what price, and when.

Step 8: Confirm audit completeness and trace end-to-end

Verified PipelineAuditLog captured every configuration change across all components with no gaps, then documented a full trace of a single tick event from source through stream, table, query, dashboard tile, and alert evaluation.

Ingestion query:

A series of .ingest inline into table PipelineAuditLog statements, each recording a timestamp, the author Anjali, a component name and a description: StockTicks schema, StockTicks table, PipelineAuditLog, TickIngestionStream, ManageFields, MarketDataQueries, VelocityTrade Live Market Intelligence, and DrawdownAlert. — Every component that was built gets its own audit row, backfilled here to seed the log.

Final output of the audit log:

KQL query results for PipelineAuditLog showing eight rows with columns ChangeTimestamp, Author, ComponentChanged and ChangeDescription, covering TickIngestionStream, PipelineAuditLog, StockTicks, ManageFields, MarketDataQueries, VelocityTrade Live Market Intelligence, and StockTicks Schema, all authored by Anjali. — Eight components, eight audit rows, each traceable back to the change that created it.