Any source, any target, one governed pipeline.
What it is
A data platform is the layer between the systems that produce data and the people and systems that use it. It takes records from wherever they originate, makes them consistent and delivers them where they are needed. It does that continuously, with a known versioned schema and a known owner for every stream.
The starting point is rarely a blank sheet. A typical estate holds transactional databases that nobody may slow down. Machines and sensors report in their own formats. Applications write logs. Exports have run unquestioned for years. The task is to build one common pipeline that connects every source the business needs, and where needed the target system as well: a data warehouse, a reporting system, a KPI dashboard.
Cloudtech designs and builds that layer and, where needed, operates it on the same terms as any other critical system: 24/7, under change management, with a P1 reaction within the hour. It is also where the other Cloudtech service lines meet. Analytics and decision support read its streams, AI integration reads its aggregates, Software development writes its connectors and jobs, DevOps, automation and observability versions and deploys them, and Managed platform and database services runs the sources it reads from.
What we do
- Ingestion from any source. Relational and transactional databases: Oracle, PostgreSQL, MySQL and MariaDB, Microsoft SQL Server. Time-series stores such as QuestDB. Message brokers and queues: Kafka, RabbitMQ, MQTT. Also application and system logs, IoT devices, sensors and plant systems, files and exports, and APIs and SaaS applications.
- Change data capture. Debezium reads changes from the database’s own transaction log. The source application is not modified, production is not queried for reporting, and every insert, update and delete arrives in commit order, seconds after it happened.
- Offloading production databases. Every report, export, search index and integration that reads a production database today moves to the stream and to serving stores fed from it. The transaction system keeps its capacity for transactions, the batch window disappears, and a heavy report can no longer stall the tills or the line. Where an Oracle system carries reporting load, that load moves to PostgreSQL or a warehouse instead of more licensed cores. The load taken off production is measured before and after.
- Streaming and batch pipelines on Apache Kafka and Apache Flink: validation, normalisation, enrichment, joins across sources and aggregation over time windows, computed in event time so that late data lands in the right period. Aggregates can be recomputed without double-counting.
- Data contracts and schema management. Every stream has a contract: schema, keys, meaning, owner, freshness and the rules for changing it. The schema is registered in Apicurio Registry and every change is checked for compatibility before a consumer sees it. A column added in the source reaches consumers as a versioned schema, not as a surprise.
- Masking and pseudonymisation in the pipeline. Log-based capture sees the real values, so a database’s query-time masking does not protect them. Personal and confidential fields are handled in the pipeline instead: dropped, truncated or hashed in the connector before they reach Kafka, tokenised in the stream so records can still be joined, or reduced to aggregates. Keys stay inside your boundary and cloud targets receive only masked streams. Pseudonymised data is still personal data under the GDPR; the pipeline makes data minimisation enforceable, it does not remove the obligations.
- Delivery to any target. Operational databases, data warehouses and lakehouse storage, time-series and search stores, and BI tools. Also cloud analytics platforms: Microsoft Fabric and Azure Event Hubs, AWS including Amazon S3, Google Cloud, and Snowflake. Where no existing system fits, Cloudtech builds one.
- Device and plant data. Sensors, PLCs and machines publish over MQTT, OPC UA or vendor protocols. Edge gateways buffer readings while a link is down, a time-series store holds them, and a bridge carries the readings that have to meet business data onto the same backbone. Where a plant runs a Unified Namespace structured by ISA-95, the backbone holds its history and its joins.
What you get
The scope is set per engagement, from one change stream feeding one report to a backbone that carries every source in the company. What you receive is the same at any scale, and it follows the practices the industry has settled on for data platforms: data contracts, data products, pipelines as code and measured quality.
- A measured map of sources and targets. Every system in scope inventoried and measured: change rate, volume, the load it can take and who owns it. Architecture and sizing follow from those numbers, and the map is the first document you receive.
- A data contract for every stream. Schema, keys, meaning, owner, freshness, retention and the rules for change, registered in the schema registry and enforced before a change reaches a consumer. Readable by the business and by the pipeline.
- Data products instead of extracts. Each derived stream is computed once, documented field by field and read by every consumer from the same place, operational and analytical alike. The screen on the floor and the company’s cloud warehouse show the same figure.
- Pipelines as code. Connectors, jobs, schemas and configuration in version control, deployed through CI/CD, secrets kept out of it, every change reversible. An environment can be rebuilt rather than remembered.
- Quality and freshness you can check. Freshness, volume and schema monitored against what each contract promises, lineage from source to figure documented, incidents recorded with their duration. A breach shows at the pipeline before it shows in a report.
- Reconciled results. Every figure the platform produces runs beside the system it replaces until both agree, and the reconciliation is kept as evidence before anything is switched over.
- Security and data protection by design. Capture reads transaction logs and never queries production. Access control is enforced by the server, administrative actions are logged, personal and confidential fields are masked or pseudonymised before they leave the source’s boundary, and the primary copy stays inside your boundary. Only the streams you choose leave it.
- Operations from day one. Every component in a declared inventory and monitored against it, runbooks for stop, start, restore and replay, and the option of the 24/7 service with a P1 reaction within the hour. Documentation, contracts and repositories are handed over in full if you run it yourself.
Where it runs
Inside your on-prem private cloud by default: your hardware, your network, your jurisdiction. Where your systems already live in Azure, AWS or Google Cloud, the same pipeline runs there, in your tenant, with the same contracts, monitoring and service level. Cloud analytics platforms receive selected streams either way, for company-wide reporting or a data science workspace, and the primary copy and its history stay where you decide, on premises or in your own cloud account.