One pattern, many sources
Most organisations do not have a data problem. They have many separate data problems: a database nobody may query during the day, machines that report in their own formats, a warehouse fed by exports nobody owns, a cloud platform the group has chosen. Each gets its own interface, its own schedule and its own way of failing. The reference architecture replaces that with a single path. Data is captured where it originates, moved once through a governed backbone, and delivered to every consumer from there.
The diagram at the top of this page shows the layers. Each is described below with the terms the industry uses for it, and with what Cloudtech does in it.
How data moves
Capture without touching the source. For databases, change data capture reads the transaction log the database already writes. The application is not modified, no polling queries load the system, and every insert, update and delete arrives in order, including deletes, which nightly extracts miss. Initial loads and backfills are taken as snapshots, with the live stream started first so that nothing falls into the gap.
One backbone, with contracts. Every message on the backbone has a registered schema. Producers and consumers agree on that schema, and the registry checks every change for compatibility. This is how data contracts are enforced technically: a change that would break a consumer is caught before it is deployed.
Processing in event time. Stream-processing jobs compute what the business works with: periodic KPIs, daily reports, master-data projections, enriched event streams. They use the time an event happened, not the time it arrived, so late data lands in the right period. Jobs keep their state in checkpoints and stop with savepoints, so an upgrade or a restart resumes where it left off.
Layers that can be rebuilt. Raw change streams, validated and joined streams, and business aggregates are kept as separate layers. Lakehouses call this the medallion architecture: bronze, silver and gold. Any consumer can read the layer it needs, and any layer can be recomputed from the one beneath it.
Serving from stores, never from production. Reports, dashboards and applications read from serving stores. Live figures are pushed to the screen as they change, and every figure links to the records behind it.
Where AI fits
The AI layer sits on top of the serving layer, not beside the sources. It reads what the pipeline has already made trustworthy: aggregates with known definitions, documents with known permissions. That placement is deliberate. A model that reads raw production tables inherits every ambiguity in them. A model that reads curated outputs inherits their definitions, their access rules and their audit trail.
The IoT and sensor-data variant
Device and sensor data follows the same pattern with a different front end:
- Devices, sensors, PLCs and machines publish over MQTT, OPC UA, AMQP or vendor protocols, or deliver files.
- Edge gateways collect, translate and buffer. Store-and-forward keeps readings while the link to the site is down, within the gateway’s buffer limits.
- A message broker, such as RabbitMQ or an MQTT broker, takes device traffic in. A time-series database such as QuestDB stores readings, while a relational database holds devices, configuration and master data.
- A bridge to the streaming backbone carries the device data that has to meet business data: machine states joined to production orders, sensor readings joined to quality results. From there it uses the same schemas, jobs and stores as everything else.
The second diagram shows the variant. In manufacturing, the same layout can carry a Unified Namespace, the broker-based, ISA-95-structured hub for current state, with the streaming backbone holding history and joins.
Inside the boundary, with doors to the cloud
Everything above runs inside the on-prem private cloud: capacity the customer owns, run with cloud-style automation and service levels. Cloud platforms are targets, not the centre. Selected streams can be delivered to:
- Microsoft Fabric and Azure Event Hubs;
- AWS, including Amazon S3;
- Google Cloud;
- Snowflake.
That covers group reporting, data science workspaces and partner exchange. The primary copy and its history stay on premises.
Under the EU Data Act, providers of data processing services must make it possible for customers to switch to another provider or back to on-premises infrastructure. An architecture with open interfaces and on-premises primaries keeps that option open by design.
What runs it
The architecture is only as good as its operation. Every component, from connectors and brokers to jobs, stores and models, is listed in a declared inventory and monitored against it. Incidents are measured from start to end. The whole stack falls under the same 24/7 service level as the databases it reads from, with a P1 reaction of one hour or less.