# Location data Over MQTT to Kafka
Message Queuing Telemetry Transport (MQTT) is a lightweight communication protocol for the Internet of Things (IoT). Zilla implements the MQTT protocol specification and actively translates it to native Kafka, providing a full MQTT broker at scale. IoT devices can communicate through Zilla to Kafka using their native MQTT libraries for home automation, industrial monitoring, weather stations, vehicle telemetry data, and more. Zilla is vendor-neutral and not a connector or plugin. You can deploy Zilla anywhere to securely interface web apps, IoT clients, and microservices to Kafka. Let's check out how we can use Zilla and Kafka to implement a taxi-hailing and tracking app!
## Taxi tracking
Ride-share apps have brought location tracking into the mainstream. The taxi industry can adapt by offering live tracking location data. Installing devices in vehicles is only the first step. IoT devices sending vehicle telemetry data over MQTT means data transmission is lightweight and reliable. However, a large number of vehicles producing data and a variable number of consumers make the pub/sub structure of MQTT fall short. The complicated interaction of riders, drivers, vehicles, and management creates a spaghetti mess of different event streams. Each event producer has unique technical needs, making a one-size-fits-all API difficult.
At the core of the problem, we need a way to get information from all the different taxis. These aren't Teslas with onboard computers and MQTT baked in. A taxi will need a small device sending data over a cellular signal or piggybacking off the driver's phone is the best we can hope for. Communication will need to be cheap and function on unreliable connections. The rider's phone app and the driver must send and receive notifications to each other. The user needs to see only their taxi en route, and a central taxi hub needs a live view of the fleet. MQTT will solve a lot of these issues, but we will need more than a Node.js backend for this, much more.
## The Zilla MQTT Broker
The [Zilla MQTT Kafka proxy](https://docs.aklivity.io/zilla/latest/concepts/kafka-proxies/mqtt-proxy.html) is a drop-in replacement for any existing MQTT broker. It can manage the MQTT protocol interactions and leverage the power of Kafka. Zilla is a stateless proxy that removes the need for an additional MQTT event broker to route data onto Kafka. This demo has a simple implementation of taxi location data tracking using MQTT and how that can enable other event streams.
> If you want to see a dedicated example, our [MQTT broker guide](https://docs.aklivity.io/zilla/latest/how-tos/mqtt/mqtt.kafka.broker.html) will be the fastest way to see Zilla in action. It will start everything needed, and you can connect with whichever MQTT client you prefer.
### Why Kafka
Kafka has a long list of use cases for event-driven architectures, and with a massive industry around it, there is tooling to solve any downstream problem. Now that we know Zilla can get our MQTT data into Kafka, let's look at why they work well together.
MQTT is a pub/sub protocol with features for filtering messages by topic. However, this still means that the publisher and subscriber must always be connected and active to ensure messages reach their destination. Kafka is the perfect historical event data layer. With messages stored in Kafka, the publisher can produce messages that will be stored for however long is needed. Subscribers can then consume these messages in whatever way makes sense for them.
With Kafka, producers and consumers can scale independently without impacting each other. IoT and IIoT environments, where the number of devices may not be static, can leverage Kafka's scalability. Combine that scale with the vast Kafka ecosystem, and any IoT use case becomes possible.
### I heard you like Topics
An MQTT broker uses a topic to decide which subscriber gets a published message. Publishers describe themselves based on the topic they use when sending their data and don't have to create the topic beforehand. Subscribers then can filter data based on wildcard patterns for these topics.
Example home sensor topics:
```
myhome/groundfloor/kitchen
myhome/groundfloor/bedroom1
myhome/groundfloor/bedroom2
myhome/groundfloor/bathroom
myhome/topfloor/bedroom1
myhome/topfloor/bathroom
```
Dynamic topic creation and subscription work for simple situations. What happens when the data landscape evolves? The above topics are well organized initially, but what happens when we add new IoT devices with different topic naming patterns? Some MQTT brokers have this limitation because different systems may need to be merged and come with pre-existing architectures. If one subscriber wants to see all the bottom floor bedrooms, there isn't a simple filter. Multiple subscribers would need to be manually aggregated.
Example new generation home sensor topics:
```
home/first/bedroom/1
home/third/bathroom/4
```
Kafka has predefined topics, and Zilla does not map topics 1:1. This means all messages can be published and subscribed to as usual and routed through a single `messages` Kafka topic. However, separate topics can be created for other use cases, like all bottom-floor bathrooms, where Zilla can route both MQTT topic patterns to the dedicated Kafka topic.
```
#/groundfloor/bathroom#
#/first/bathroom/#
```
Again, all data is stored in the underlying Kafka broker, meaning you get all of the benefits of MQTT and Kafka without needing to store and process messages on a separate MQTT broker.
## The Zilla Taxi Demo
This demo has three main components. Zilla sits in front of Kafka, ensuring all the messages get where they need to be. The web app UI serves as the user request, driver navigation, and management observation tool. The `taxi-service` is an MQTT-simulating microservice that pretends to be a vehicle driving along the requested route.
### Open street routes
The map UI and POIs use [Open Street Maps](https://www.openstreetmap.org/) to show the vehicle and bar locations. The routes are pulled from the [Open Route Service](https://openrouteservice.org/).
### MQTT Simulator
The [open-source Python MQTT simulator](https://github.com/DamascenoRafael/mqtt-simulator) called the `taxi-service` provides a real-world taxi location tracking service. A gRPC microservice creates new simulated vehicles along a designated route. The simulator uses the paho MQTT library to publish messages to the Zilla MQTT broker.
### Zilla proxies MQTT, REST, and gRPC to Kafka
> The config snippets in this section are pulled from the [demo zilla.yaml](https://github.com/aklivity/zilla-demos/blob/main/taxi/zilla.yaml), use it for full context.
Zilla implements native protocols. You don't need to learn new libraries or SDKs to work with the endpoints defined in Zilla. You don't have to convert your existing infrastructure into a whole new framework. In many situations, Zilla will be a drop-in replacement for other legacy or off-the-shelf components.
The `taxi-service` is a native MQTT producer publishing data through Zilla into Kafka. The downstream Kafka consumers interact with that data. You can see in the Zilla config below that we are routing MQTT topics `bus/+/location` and `taxi/+/location` to dedicated Kafka topics. This lets each vehicle publish its topic based on a unique identifier. The MQTT topic is used as the Kafka message key.
```yaml
# MQTT proxy service to Kafka topics
mqtt_kafka_proxy:
type: mqtt-kafka
kind: proxy
options:
topics:
sessions: mqtt-sessions
retained: mqtt-retained
routes:
- when:
- publish:
- topic: bus/+/location
- subscribe:
- topic: bus/+/location
with:
messages: bus-locations
- when:
- publish:
- topic: taxi/+/location
- subscribe:
- topic: taxi/+/location
with:
messages: taxi-locations
```
The taxi map UI uses the [js fetch](https://www.w3schools.com/jsref/api_fetch.asp) method to GET data from REST endpoints defined in Zilla. The REST endpoints `/bus/locations` and `/taxi/locations` expose the corresponding Kafka topics, allowing a URL param to filter the returned messages by the message key.
```yaml
# HTTP proxy service to Kafka topics
http_kafka_proxy:
type: http-kafka
kind: proxy
routes:
- when:
- method: GET
path: /bus/locations
with:
capability: fetch
topic: bus-locations
- when:
- method: GET
path: /taxi/locations
with:
capability: fetch
topic: taxi-locations
- when:
- method: GET
path: /taxi/locations/{id}
exit: kafka_cache_client
with:
capability: fetch
topic: taxi-locations
filters:
- key: ${params.id}
```
The `taxi-gateway` is a standard grpc-gateway providing a JSON REST API for the `taxiroute.TaxiRoute` gRPC microservice. When the gateway sends data to Zilla, it thinks it is talking to a gRPC microservice. The protobuf messages being sent between them are routed through Kafka.
```yaml
# gRPC proxy service to Kafka topics
grpc_kafka:
type: grpc-kafka
kind: proxy
routes:
- when:
- method: taxiroute.TaxiRoute/*
exit: kafka_cache_client
with:
capability: produce
topic: route-requests
reply-to: route-replies
# gRPC Kafka fanout to a remote server
taxi_route_remote_server:
type: kafka-grpc
kind: remote_server
entry: kafka_cache_client
routes:
- when:
- topic: route-requests
reply-to: route-replies
method: taxiroute.TaxiRoute/*
with:
scheme: http
authority: ${{env.TAXI_ROUTE_SERVER_HOST}}:${{env.TAXI_ROUTE_SERVER_PORT}}
exit: taxi_route_server_grpc_client
```
### Running the demo
> Zilla can work with any Kafka broker, including [Redpanda](https://redpanda.com/). The demo startup script allows you to switch between running with a Kafka or Redpanda backend. Zilla natively implements the Kafka protocol and doesn't force you to choose between vendors.
The taxi demo [README](https://github.com/aklivity/zilla-demos/tree/main/taxi) contains instructions on how to start and use the demo. Let's look at it in more detail.
Starting the `taxi-demo` with `docker-compose`:
```bash
% ./startup.sh
[+] Running 1/1
✔ zilla Pulled
[+] Running 7/7
✔ Container zilla Running
✔ Container kafka Running
✔ Container taxi-map-ui Running
✔ Container taxi-gateway Running
✔ Container zilla-taxi-kafka-init-1 Exited
✔ Container taxi-service Running
✔ Container kafka-ui Running
```
The `taxi-map-ui` loads into a full view of all active taxis and buses. Users can hail new taxis. Busses follow a set route and loop back when they reach their destination. Initially, only buses are driving around since users haven't requested any taxis. The `taxi-service` publishes both the taxi and bus locations.
The user can request a taxi by choosing between the bars in downtown San Jose, CA. Selecting a waypoint and hailing a taxi will send an HTTP request to the `taxi-gateway` that converts it into a protobuf object. Zilla will send this request to the MQTT simulator microservice using a Kafka to gRPC message fanout.
Once the taxi is hailed, the live tracking data fetched from the Zilla HTTP proxy is filtered to show the user's route and vehicle. The MQTT simulator publishes updated location information for the taxi, and the user can see the taxi get closer to the desired destination. Once the taxi arrives, it stops sending location updates, and the UI stops showing the active taxi.
To return to the main view with all the bars and taxis, use the "Clear route" button or click on the Aklivity logo on the top left. You can keep this view open in its own tab and watch as new taxis get created. In a separate tab, keep hailing different taxis to the different bars.
The demo `kafka-ui` allows you to browse the messages on Kafka. Check out the location data on the `taxi-locations` topic and see how the topic's key is used when filtering updates for an individual taxi.
## Conclusion
You have saved the taxi industry! Well, at least a simulated one. You have seen how effective MQTT is at transmitting data from many new devices. You set up a Zilla instance that proxied native MQTT, gRPC, and REST protocols. Now, you can imagine all the new ways your data can interface with Kafka without the headache of managing extra code, microservices, or endpoints. Just let Zilla take care of it!
### What's next
Check out our [quickstart](https://docs.aklivity.io/zilla/latest/tutorials/quickstart/kafka-proxies.html) to see some of the other things Zilla can do. More features are in active development on the [Zilla roadmap](https://github.com/orgs/aklivity/projects/4). Star and watch the [Zilla repo](https://github.com/aklivity/zilla/releases) for new releases. Your input and feedback will be critical as we continue to improve. Try it out for yourself and leave us feedback on GitHub, or join the discussion in the Zilla Community.
