FROM 14 HOURS TO 100 MINUTES: Telemetry at Databricks Speed

Every day, roughly 132,000 usage log files, 531 million product events, 132 GB of raw XML, arrive from Archicad. The pipeline that turned them into product analytics tables had grown organically into 55 mostly serial tasks across 173 notebooks (48,000 lines of code), with twelve near identical copies of the core parser, more than 4,000 lines of commented out code, and no automated tests. A fixed 10:00 UTC cron fired whether the data had arrived or not, and a single failure meant re running a 14 hour job. A median run took 12.5 hours, so analysts saw yesterday's data only by evening, and every change carried untestable risk.

The new pipeline is a 24 task medallion architecture (raw, cleaned, business ready layers) on Azure Databricks, governed by Unity Catalog. Delta Lake gives safe, repeatable re runs through partition scoped writes (MERGE and replaceWhere); the Photon engine on Databricks Runtime 17.3 (Spark 4) accelerates ten of eleven job clusters; a serverless arrival gate job launches the pipeline only once the day's data has arrived for every required product version; and Databricks system tables provide the DBU level cost evidence behind every number.

United Consult's team designed and delivered the transformation end to end, consolidating twelve duplicated parser notebooks into one parametrized notebook, replacing serial chains with a parallel task graph (the three dominant parser runs now execute concurrently in ~27 minutes instead of one 264 minute serial task), introducing 170 automated tests across 13 suites, and validating the whole system in a 2.5 month side by side production run against the legacy pipeline on identical data.

‍