Encryption and Structured Field Encryption: Zilla Benchmark Series (Part 3)

Introduction

Part 1 of this series measured Zilla Plus proxying self-managed Apache Kafka. Part 2 ran the same workload against Confluent Cloud.

In both, Zilla Plus forwarded records without changing them. This post measures what happens when Zilla Plus also encrypts record data on produce and decrypts it on fetch.

We ran four configurations against the same Kafka cluster and the same workload:

• Direct: clients connect to Kafka with no proxy.
• TLS passthrough: clients connect through Zilla Plus, records unchanged.
• Whole-payload encryption: Zilla Plus encrypts the entire record value.
• Structured field encryption: Zilla Plus encrypts only the schema fields tagged as sensitive.

TL;DR

• Throughput held in every configuration. All four sustained their target rate for the full 60-minute measurement window with zero publish errors.
• The proxy hop is small. TLS passthrough added 0.9 ms at p50 and 10.2 ms at p99. 
• Whole-payload encryption adds a few milliseconds at the median. Over passthrough, it added 3.4 ms at p50 and 37.9 ms at p99.
• Structured field encryption adds more per record. Over passthrough, it added 14.8 ms at p50 and 56.1 ms at p99, and leaves non-sensitive fields readable downstream.

Why Encrypt at the Gateway

TLS protects records in transit. Once a record is written to a broker, it is stored in plaintext and readable by anyone with access to:

• the topic
• broker disks and snapshots
• backups, mirrors, and replicated clusters
• tiered storage

Encrypting record data before it reaches the broker means all of those hold ciphertext.

Doing that in the gateway keeps the work out of application code:

• No client changes: producers and consumers use standard Kafka clients and the Kafka wire protocol.
• Central key handling: keys are resolved from a vault configured in Zilla Plus, not distributed to every application.
• Identity-based decryption: a guard decides which authenticated consumers receive plaintext.

Zilla Plus applies encryption inside its kafka-proxy binding, the same binding used in Part 1 and Part 2.

Whole-Payload Encryption

Whole-payload encryption treats the record value as opaque bytes and encrypts all of it. It needs no schema, so it works for any payload format.

kafka_proxy_encrypt:
  type: kafka-proxy
  kind: proxy
  options:
    topics:
      - name: "*"
        value:
          model: bytes
          transforms:
            guarded:
              inline_guard: []
            encryption:
              vault:
                secrets:
                  - payload-key

‍

On produce, Zilla Plus encrypts the value before forwarding it to Kafka. On fetch, it decrypts the value for consumers that satisfy the guarded clause.

Structured Field Encryption

Structured field encryption parses each record against a schema and encrypts only the fields tagged for protection. Every other field stays readable, so downstream tools can still filter, route, and index on non-sensitive data.

The schema used in this benchmark tags a single field, ssn:

{
  "type": "object",
  "properties": {
    "id": { "type": "string" },
    "status": { "type": "string" },
    "ssn": { "type": "string", "x-kind": "pii-encrypt" },
    "notes": { "type": "string" }
  }
}

‍

The Zilla Plus configuration selects fields by that tag:

‍

kafka_proxy_encrypt:
  type: kafka-proxy
  kind: proxy
  options:
    topics:
      - name: "*"
        value:
          model: json
          catalog:
            my_catalog:
              - subject: test-topic-value
                version: latest
          transforms:
            - select:
                - type: structure
                  fields:
                    - tagged:
                        - x-kind: pii-encrypt
              guarded:
                inline_guard: []
              encryption:
                vault:
                  secrets:
                    - payload-key

‍

The flow below shows what Zilla Plus does on each path. Dashed steps run only in structured field encryption; whole-payload encryption skips them and encrypts the full value.

Adding a new sensitive field is a schema change. The Zilla Plus configuration and the client applications stay the same.

Why Structured Field Encryption Matters

Most records that carry sensitive data are mostly non-sensitive. A payment event might hold an ID, an amount, a merchant, a timestamp, and one card number.

With whole-payload encryption, every consumer that needs the amount or the merchant also needs decrypt access. Plaintext access to the card number spreads to every one of those consumers.

Structured field encryption scopes protection to the fields that need it:

• Least-privilege access: consumers read non-sensitive fields without decrypt rights. Only consumers that satisfy the guard receive sensitive values in plaintext.
• Downstream processing keeps working: stream processors, connectors, search indexes, and analytics jobs can still filter, join, and aggregate on unencrypted fields.
• Records stay schema-valid: encrypted fields keep their place in the record, so schema registries and schema-aware tools continue to work.
• Sensitivity is declared in the schema: a tag such as x-kind: pii-encrypt marks a field for protection, so data owners define it where the data is defined.

Whole-payload encryption fits records that are sensitive end to end or have no schema, such as secrets or binary blobs.

Benchmark Setup

We used the Aklivity fork of the Linux Foundation's OpenMessaging Benchmark (OMB), as in the previous parts.

Configurations

Security protocol

Every configuration runs TLS end to end.

In the encryption runs, clients authenticate with SASL/PLAIN so the Zilla Plus guard has an identity to evaluate before decrypting on fetch.

Infrastructure

The infrastructure matches Part 1.

• Apache Kafka 3.6.0 with KRaft
• GP3 EBS: 500 GB, 3,000 IOPS, 125 MB/s per broker

Workload

• Message size: 1 KiB JSON event (id, status, ssn, notes)
• Target throughput: about 24 MB/s
• Topics: 45, with 200 partitions each
• Producers: 1 per topic
• Consumers: 3 consumer groups per topic, 1 consumer per group
• Config: acks=all, linger.ms=10, replication factor 3, min.insync.replicas=2
• Warm-up: 30 minutes; measurement: 60 minutes

The same JSON payload was used in all four configurations, so the whole-payload and structured runs encrypt identical records.

Encryption details

• Cipher: AES-GCM with a fresh 12-byte IV per operation and a 16-byte authentication tag, adding 28 bytes per encrypted value.
• Key source: a filesystem vault on each Zilla Plus instance. The key is resolved once and cached, so no external KMS calls happen on the measured path.
• Decryption: consumers authenticate through the same guard, so every fetch is decrypted. With 3 consumer groups per topic, Zilla Plus performs three decryptions for each encryption.

Publish Latency

Publish latency is the time from the producer sending a record to receiving the broker acknowledgment.

Added publish latency relative to TLS passthrough:

End-to-End Latency

End-to-end latency: time from the producer sending a record to a consumer receiving it. In the encryption runs, this includes the encrypt on produce and the decrypt on fetch.

Added end-to-end latency relative to TLS passthrough:

End-to-end numbers track publish latency closely in every configuration. At p50, the gap between publish and end-to-end latency is under 1 ms for whole-payload encryption and about 2 ms for structured field encryption, which covers the decrypt on fetch.

The Cost of Each Layer

Each configuration adds one layer, so the difference between neighbors isolates that layer's publish latency cost:

Whole-payload encryption runs AES-GCM over the full 1 KiB value. Most of its cost shows up past p95, while the median moves by a few milliseconds.

Structured field encryption encrypts less data, since only the ssn value is ciphertext. Its extra latency comes from the work around the cipher:

• parsing each record as JSON
• validating it against the catalog schema
• locating the tagged fields
• re-serializing the record with the encrypted field in place

The same steps run in reverse on each fetch.

Whole-Payload Encryption at 60,000 Messages/Second

To evaluate Zilla Plus under a higher-throughput encryption workload, we ran an additional benchmark using a single Kafka topic at 60,000 messages per second. The test measured throughput and latency with whole-payload encryption on produce and decryption on fetch.

Workload and setup

Results

Zilla Plus sustained the 60,000 msg/s target with zero publish errors.

How Zilla Achieves This Performance

Zilla Plus keeps encryption overhead low through a streaming-native architecture designed to minimize data movement, memory allocation, and contention on the critical path.

• Zero-copy, in-place processing. Kafka frames are parsed using flyweights over pooled buffers, avoiding unnecessary copies and object allocations. Encrypted records are rewritten in place, with affected length fields and the CRC updated rather than rebuilding the entire Kafka frame.
• Pass-through for unchanged traffic. Topics and frames that require no transformation are forwarded directly, avoiding unnecessary processing.
• Envelope encryption with cached keys. AES-GCM encryption runs locally. With KMS-backed key management, only the small data keys interact with the KMS, not the record payloads. Keys are cached per worker, avoiding per-record KMS calls during steady-state operation. In this benchmark, keys came from a filesystem vault and were cached, with no external KMS calls on the measured path.
• Shared-nothing workers. Each worker owns its buffers, ciphers, and key cache, confined to a single thread. This eliminates locking and synchronization overhead on the hot path.

Together, these optimizations allow Zilla Plus to enforce encryption at the gateway while sustaining high throughput and keeping added latency low.

Choosing an Encryption Mode

• Whole-payload encryption fits values that have no schema or are sensitive throughout, such as secrets and binary blobs. It has the lower added latency of the two modes.
• Structured field encryption fits records that mix sensitive and non-sensitive fields, where downstream consumers need the non-sensitive ones. It also validates every record against its schema, so malformed records are rejected at the gateway.
• Both modes can run side by side on the same Zilla Plus instance, on separate listeners or topics.

These results are for 1 KiB records at about 24 MB/s across three m5.xlarge gateways. Larger records, higher rates, or more encrypted fields per record will change the numbers and are worth measuring for a specific workload.

Reproducing the Results

The benchmark tooling, Terraform and Ansible deployment, and Zilla Plus configurations are in the Aklivity OpenMessaging Benchmark fork. The deployment guide covers each scenario.

The four configurations map to these run-workloads.sh scenarios:

./confluent-bin/run-workloads.sh ak workloads/confluent-blog-workloads-2cku-1p-1c-1Kb-45topic-200part-24MBps.yaml

./confluent-bin/run-workloads.sh zpak workloads/confluent-blog-workloads-2cku-1p-1c-1Kb-45topic-200part-24MBps.yaml

./confluent-bin/run-workloads.sh zpak-encrypt workloads/confluent-blog-workloads-2cku-1p-1c-1Kb-45topic-200part-24MBps.yaml

./confluent-bin/run-workloads.sh zpak-structured-field-encrypt workloads/confluent-blog-workloads-2cku-1p-1c-1Kb-45topic-200part-24MBps.yaml

Resources

• Part 1: Proxy Benefits With Near-Zero Latency Tax
• Part 2: Zilla Plus on Confluent Cloud
• Aklivity OpenMessaging Benchmark fork
• Zilla documentation
• Zilla examples
• Join the Aklivity Slack community