Knowledge & Data

Knowledge & Data covers the foundation everything else is built on: how organisations store, process, model, and ultimately understand their data. It spans two decades of this blog — from distributed processing on Hadoop and Spark, through real-time and streaming architectures, to decentralised ownership with Data Mesh, and on to the semantic layer that agentic systems now depend on.

The through-line is that the technology keeps changing and the problem doesn’t. Getting the right data to the right place has always been the hard part. Agents made it urgent rather than new: an AI OS hits the context wall for the same reason a query hits a full table scan — the working set was wrong.

Related topics on this blog:

• Big Data — Processing and storing data at scale
• Data Mesh — Decentralised, domain-oriented data ownership
• Apache Spark — Fast, in-memory data processing
• Lambda Architecture — Combining batch and real-time processing
• Kappa Architecture — Stream-first data architectures
• AI OS — Ontology and context as the memory of an organisation

For a sovereign platform where analytics and AI work on one current copy of the truth — open Apache Iceberg tables, no proprietary lock-in — see Scrydon’s Cognitive Enterprise and Analytics.

• The Five Elements of an AI OS

• Data Mesh using Microsoft Fabric at Cloud Scale Analytics '23

• Data Mesh in Azure at Microsoft Customer Success Day '22

• Dataminds Connect: The Synapse Keynote

• Configuring Apache Spark to oversubscribe the # of cpu cores

• ML in Production @ AZUG

• ML in Production @ dataminds.be

• Azure Databricks and Spark @ Techorama '18

• Announcing EventHubs ReIngest

• Cloudera, Azure and Big Data at Cloudera Meetup '17

• Predicting Occupancy on the Belgian railroads, based on spitsgids/iRail data, using Azure ML.

• Microsoft Advanced Analytics @ Data Science Ghent '16 meetup

• Spark on Azure: a Gentle Introduction at Cloudbrew '15

• Virdata: lessons learned from the Internet of Things and M2M Cloud Services @ IBM Big Data Developers Meetup, Silicon Valley

• A Lambda Architecture using Hadoop and Storm @ NoSQL Matters Cologne '14

• A Big Data Overview, from DataCrunchers @ Devoxx 2013

• A real time architecture using Hadoop and Storm @ Devoxx 2013

• A real time architecture using Hadoop and Storm @ JAX London 2013

• Microsoft Big Data @ SQLUG 2013

• A real time architecture using Hadoop and Storm @ FOSDEM 2013

• Microsoft Inspirience '12: Getting more out of your Big Data

• Hadoop Pig: Map Reduce the Easy Way @ FOSDEM 2011

Discover more posts and tags on the overview page.