Cyber Threat Intelligence Lakehouse Pipeline
A cloud-based data engineering project that transforms CISA Known Exploited Vulnerabilities into structured, validated analytical datasets using Databricks and a medallion architecture.
Project overview
This project forms part of the Cyber Threat Intelligence Observatory (CTIO). It demonstrates cloud data ingestion, transformation, analytical modelling, orchestration, data quality controls and independent operational monitoring.
The Databricks pipeline complements a separate PostgreSQL-based CTIO data platform.
Pipeline architecture
Bronze preserves ingested source records. Silver prepares and validates the data. Gold produces curated analytical summaries. A fourth notebook monitors execution outcomes and data quality.
Verified processing results
These results are from a successful manual execution on 9 November 2026, not live counters. The ransomware summary also produced two categories.
Orchestration
- Bronze ingestion: Retrieve and ingest CISA KEV data.
- Silver transformation: Standardise and validate vulnerability records.
- Gold aggregations: Produce vendor, monthly trend and ransomware summaries.
- Pipeline monitor: Record execution results and perform quality checks.
The job is configured to run every six hours at 00:15, 06:15, 12:15 and 18:15 UTC. The monitoring notebook is configured to run even when an upstream task fails.
Data quality and reliability
- Bronze-to-Silver record-count consistency checks.
- Duplicate vulnerability identifier checks.
- Freshness checks and non-empty analytical outputs.
- Pipeline execution history and task-status monitoring.
- Independent Ubuntu home-server watchdog using the Databricks Jobs API.
- Healthchecks.io monitoring for watchdog availability and incident signalling.
The independent watchdog checks execution history every 15 minutes and allows a 30-minute grace period for scheduled executions. Failure and recovery alert behaviour is undergoing final verification.
Technology stack
Python · PySpark · Delta Lake · Databricks SQL · Unity Catalog · Databricks Jobs · Linux · Cron · REST APIs · Healthchecks.io
Project status and next steps
The four-task pipeline has completed a successful manual execution. Automated scheduling and independent monitoring are configured. The first scheduled execution and end-to-end failure/recovery notifications remain to be verified.
Future development may extend the cloud analytical layer to additional threat intelligence sources and cross-feed comparisons.