The Industrial Internet of Things (IIoT) is undergoing a revolution. Sensors deployed throughout factories, grids, and supply chains are generating vast data, making the ability to transform raw signals into actionable intelligence a key competitive edge.
However, integrating legacy systems and compatibility between protocols like MQTT/Sparkplug protocol with modern data platforms like Apache Kafka remains a key challenge.
In this blog, we explore how event-driven architectures, especially those combining MQTT and Kafka, unlock the power of industrial data.
The Industrial Revolution: Industry 4.0
The fourth industrial revolution marks the combination of information technology (IT) and operational technology (OT) to enable smart, autonomous, and highly connected industrial systems using real-time data.
Key characteristics and advancements of Industry 4.0 include:
Real-Time Analytics
Enables continuous monitoring of industrial processes, allowing organizations to detect anomalies, optimize performance, and make instant decisions.
Edge Computing
Bringing computation closer to the source of data. To adhere to government regulations around data privacy and reducing latency and network dependency.
Integration with AI & ML
Helping models learn from both historical and real-time data, enabling smarter and more adaptive industrial systems.
Vendor-Native Infrastructure
Vendor-native platforms that are built to integrate seamlessly with specific industrial systems and protocols.
Interoperability & Open Standards
Adoption of open standards like MQTT, and Sparkplug ensures seamless communication and integration between heterogeneous systems and vendors.
Challenges in the IIoT Industry
Scaling
Scalability is one of the most critical requirements in IIoT architectures due to the sheer volume, velocity, and variety of data generated across industrial environments.
Modern factories, energy grids, and supply chains may involve thousands to millions of connected sensors, PLCs, and machines, each streaming data in real time.
Failing to scale effectively can stall Industry 4.0 initiatives and limit the value of IIoT deployments.
Message Throughput
High message throughput is a critical challenge in IIoT systems, where devices emit vast volumes of data at high frequencies. Especially in time-sensitive applications like monitoring, control, or predictive maintenance.
Legacy middleware and traditional data pipelines are often not designed to handle these cases. As a result, systems can experience message loss, increased latency, and degraded performance.
Data storage
Devices and sensors continuously generate time-series data, logs, metrics, and events often measured in terabytes per day across large-scale deployments.
In this case, storing & managing data at this scale across multiple platforms/clusters inflates costs and also an operational nightmare.
Data Governance
Questions around ownership, access control, data lineage, retention policies, and compliance (e.g., ISO/IEC 27001, NIST, GDPR) become increasingly complex as data flows from each sensor to any platform either on-prem or cloud and across organizational boundaries.
Delivery guarantees
Different industrial applications demand varying delivery semantics such as at-most-once, at-least-once, or exactly-once.
Protocols like MQTT provide Quality of Service (QoS) levels to define these guarantees:
- QoS 0 (at-most-once): No guarantee of delivery.
- QoS 1 (at-least-once): Guaranteed delivery but possible duplicates.
- QoS 2 (exactly-once): Guarantees one-time delivery but with increased overhead.
Achieving consistent end-to-end guarantees across diverse systems is non-trivial.
Legacy Systems & Third-Party Integrations
Legacy OT systems often rely on proprietary protocols, air-gapped deployments, or tightly coupled architectures, making integration with third-party vendors & partners particularly challenging.
These systems lack native support for modern interoperability standards, forcing organizations to build complex bridges that translate, normalize, and secure the data before it can flow into real-time IIoT pipelines.
The Shift-Left Architecture: Edge-Powered IIoT
The Shift-Left Architecture represents a transformative leap in how industrial data is processed and utilized by handling data quality and governance right where the data is created.
Instead of waiting for batch pipelines to push data from OT to IT systems, a shift-left approach processes streaming data in real time, enabling faster insights and lower latency.
By combining edge computing with event streaming, it ensures both Operational (OT) workloads (like real-time monitoring, control, and automation) and Analytical (IT) workloads (like trend analysis, forecasting, and anomaly detection) can access consistent, fresh, high-quality data without delay.
feat. Zilla: Why Native MQTT-Kafka Proxying Matters
Zilla transforms Kafka into a high-performance MQTT broker, enabling direct streaming of MQTT messages. While also preserving QoS, session state, and message metadata into Kafka topics without intermediary brokers or translation layers.
This removes redundant infrastructure, avoids data duplication, and eliminates additional latency introduced by intermediary systems.
In most industrial IoT architectures, MQTT brokers act as intermediaries that collect device data and forward it to Kafka using bridges or connectors.
This introduces additional challenges:
- Operational overhead in maintaining MQTT Brokers.
- Data Duplication.
- Increased Latency (device → MQTT broker → bridge/connector → Kafka).
- Scaling Bottlenecks.
MQTT Kafka Mapping
Zilla’s mqtt-kafka proxy binding natively maps MQTT protocol directly to Kafka wire protocol, eliminating the need for an MQTT broker or external bridge.
This makes Zilla a drop-in replacement for any existing MQTT broker.
Key Capabilities
- Zilla natively supports both MQTT v5.0 and v3.1.1: ensuring compatibility with a wide range of clients.
- Supports Quality of Service (QoS) levels (0, 1, 2): Offers flexible delivery guarantees from best-effort to exactly-once delivery.
- MQTT over WebSocket: Enables MQTT communication in browser-based environments.
- Last Will and Testament (LWT): Handles abrupt disconnection and reconnection issue.
- Correlated Request-Response: Supports v5 request-response paradigm.
- Reconnect and Session Takeover: Seamless remain subscribed to MQTT topics previously subscribed.
Kafka Powered Capabilities
- Pub/Sub with Kafka: Bridges MQTT publish/subscribe semantics to Kafka topics.
- Messages on Kafka: MQTT messages are written directly to Kafka, eliminating intermediary MQTT brokers.
- Topic Routing: Flexibility to route MQTT messages to specific Kafka topics without affecting session or retained topics.
- Retained Messages: dedicated retained Kafka topic, to replays to new subscribers for a consistent experience.
- Session Management: Tracks Client connection states, including connect, disconnect, and subscription details.
Support for Sparkplug
Zilla can natively interpret and route Sparkplug-compliant MQTT messages, enabling seamless integration of industrial systems and SCADA environments into modern event-driven architectures.
Data Governance
Zilla enables centralized schema management and data validation for MQTT topics, payloads, and headers with Schema Registry support.
You can also enforce validation for user-provided properties.
Security
TLS
Zilla secures MQTT communication using TLS binding, ensuring that all data exchanged between clients and brokers is encrypted.
This prevents unauthorized access, data interception, and man-in-the-middle attacks, making it suitable for security-sensitive IIoT deployments.
JWT
Zilla supports JWT-based access control, allowing authentication and fine-grained authorization.
The JWT access tokens, with fine-grained privileges enforced on publish or subscribe to MQTT topics.
AsyncAPI Support
Zilla supports AsyncAPI, the industry-standard specification for defining, designing, and documenting event-driven APIs.
You can use your existing MQTT AsyncAPI specification to instantly spin up Zilla instances, enabling seamless bridging between design and deployment.
Comparison of MQTT-Kafka Integration
Zilla’s MQTT-Kafka Proxy in action
Demo: Smart Buildings powered by SparkplugB, Zilla, and Kafka
Zilla acts as a drop-in replacement for any existing MQTT broker, making the transition to modern data infrastructure seamless and non-disruptive.
To illustrate how Zilla’s native MQTT to Kafka proxy can power real-time IIoT solutions, we have built a Smart Buildings demo that applies the Shift-Left architecture in action.
Steps to spin up the Smart Buildings Demo is quick and effortless as we have used Docker Compose to initialize all the supporting services required as part of this stack.
Refer to this Github Repository for deployment instruction and source code: : Zilla Demos:Smart Buildings
In this setup, simulated devices across multiple buildings generate telemetry such as temperature, humidity, CO₂ levels, and light intensity. These readings are packaged using the SparkplugB MQTT protocol and sent to Edge of Network (EoN) nodes.
Zilla receives these MQTT streams directly from the EoN nodes and maps them natively into Kafka topics without passing through a traditional MQTT broker or connector. This approach reduces latency and operational overhead while preserving Sparkplug semantics, QoS guarantees, and session state.
We are using Node-RED as a Sparkplug client to decode the SparkplugB payloads and store the data in InfluxDB for further analytics in Grafana.
Using Node-RED, we can also send commands to the EoN node and its devices.
In this demo, we monitor devices and sensors of all the buildings using Prometheus and visualize the data with Grafana.
This demo shows how edge-to-Kafka streaming can simplify integration, eliminate extra infrastructure layers, and speed up time to insight, aligning perfectly with the goals of Industry 4.0.
Conclusion
The Fourth Industrial Revolution is being shaped by real-time data, intelligent automation, and interconnected systems. Yet, bridging the gap between widely-deployed industrial protocols like MQTT/Sparkplug and modern event-driven platforms like Kafka continues to be a major hurdle. Traditional architectures often introduce complexity, latency, and operational overhead through intermediary brokers and connectors.
Zilla’s native MQTT-Kafka proxy changes that dynamic. By eliminating the need for a separate MQTT broker and directly mapping MQTT traffic in and out of Kafka, Zilla simplifies infrastructure, reduces latency, prevents data duplication, and enhances scalability.
Its edge-first, shift-left design empowers organizations to process, govern, and act on data closer to the source delivering faster insights and enabling smarter industrial operations.
With support for critical features like QoS, WebSocket, schema validation, JWT-based access control, and AsyncAPI, Zilla offers a robust, future-ready foundation for industrial IoT architectures.
What’s Next?
Check out our documentation on MQTT-Kafka Proxy & IoT Ingest & Control Solution on AWS.
We have many such demos and real-world examples published in mentioned Git repositories for you to explore.
Zilla Examples: Hands-on guides and configurations for real world use cases.
Zilla Demos: End-to-end demonstrations of Zilla powering real-time systems.
Join the Zilla Community! Share your experiences and questions on Slack.
