AWS Simplifies MSK Custom Domains. You Still Own the Networking and Security.

AWS recently added something Kafka platform teams have wanted for years: a supported way to configure custom domain names for Amazon MSK.

The new custom.advertised.listeners property lets you define custom broker hostnames through a single MSK configuration. AWS validates the configuration, resolves the broker ID for each broker, and handles rollout as part of the cluster configuration.

It works with both ZooKeeper and KRaft clusters and supports MSK’s client-facing listener types, including TLS, SASL/SCRAM, and IAM.

That’s a useful improvement. But it solves one specific part of the custom-domain problem: what MSK advertises back to Kafka clients.

You still have to build and operate the network path behind those domains.

Why custom domains matter for Kafka

Kafka clients work differently from typical HTTP clients.

A client connects to a bootstrap address (something like b-1.cluster-name.kafka.us-east-1.amazonaws.com), retrieves cluster metadata, and then opens connections directly to the brokers responsible for the partitions it needs. Those broker addresses are returned in Kafka metadata.

That makes broker hostnames part of the client-facing interface.

A stable domain becomes useful when you want to:

• migrate or replace clusters without changing client configuration
• support blue/green deployments
• fail over between primary and standby clusters
• expose Kafka to partners or external applications through domains you control

Until now, configuring custom broker hostnames on MSK meant manually overriding advertised.listeners for individual brokers. That was operationally awkward and didn't work with KRaft-based MSK clusters.

AWS's new configuration removes that problem.

What AWS now handles

A configuration can look like this:

custom.advertised.listeners=CLIENT_IAM://b-{broker_id}.example.com:9000+{broker_id}

MSK resolves {broker_id}, validates the setting, and rolls the change across the cluster.

It also survives broker replacement and scaling from the MSK configuration side.

If you're already using custom domains with an NLB and SASL/SCRAM or IAM, this is worth adopting. It replaces a fragile manual configuration step with something MSK manages directly.

The rest of the architecture, however, remains your responsibility.

You still have to build the connectivity layer

For the custom hostnames to work, you still need infrastructure that routes each advertised broker address to the corresponding MSK broker.

In the AWS reference pattern, that typically means:

• a Network Load Balancer
• a TLS listener for every broker plus bootstrap
• target groups mapped to the brokers
• certificates covering the broker hostnames
• DNS records for each broker
• security-group configuration connecting the NLB to MSK

Adding brokers means updating that infrastructure as well.

The new MSK property automatically updates what the broker advertises. It doesn't automatically create the corresponding listener, target group, certificate SAN, or DNS entry.

For a small cluster, that may be manageable. As the cluster grows, it becomes another piece of infrastructure that has to stay synchronized with MSK.

There is also an important security limitation for teams using certificate-based authentication.

mTLS remains difficult with the NLB pattern

When TLS terminates at the NLB, the Kafka broker doesn't receive the original client certificate.

That matters if your Kafka authorization model depends on certificate identity. Broker-side ACLs can no longer authorize the connection based directly on the client's certificate Common Name.

For organizations using mTLS to authenticate external Kafka clients, custom advertised listeners don't change that limitation.

This is where the architecture used by Zilla Plus differs.

A protocol-aware approach

Zilla Plus for Amazon MSK has supported custom domains for external access to private MSK clusters for quite some time.

Instead of changing MSK's own advertised.listeners, Zilla Plus understands the Kafka protocol and rewrites the broker metadata returned to external clients.

When a client bootstraps through Zilla Plus:

1. Zilla forwards the metadata request to MSK.
2. MSK returns its normal private broker addresses.
3. Zilla rewrites those addresses in the Kafka metadata response.
4. The client connects back through Zilla using the external broker domains.

MSK continues advertising its normal private addresses internally.

That removes the need to make external networking part of the broker configuration.

Internal Kafka traffic stays internal

Because MSK's own advertised listeners don't change, applications inside the VPC can continue connecting directly to MSK.

They don't need to route through the external load-balancing topology simply because external clients need custom domains.

External access becomes an edge concern rather than a cluster-wide configuration change.

mTLS identity can reach the broker

Zilla Plus also supports end-to-end identity propagation for mTLS clients.

The gateway validates the external client certificate and forwards the certificate Common Name to MSK as the principal used on the broker-facing connection.

Kafka ACLs can therefore continue enforcing permissions against the client identity rather than the proxy identity.

For teams using certificate-based authorization, especially in regulated environments, this is often the more important issue than the custom domain itself.

The network topology is simpler

Zilla Plus runs as a stateless, autoscaling gateway fleet behind a Network Load Balancer.

The NLB forwards connections to the gateways. Zilla handles Kafka routing at the protocol layer.

As MSK brokers are added or removed, you don't need to create matching NLB listeners, target groups, DNS entries, or certificate SANs for every broker.

The gateway discovers the Kafka topology through the protocol and handles the mapping dynamically.

Certificates can stay in ACM

Zilla Plus integrates with AWS Nitro Enclaves for TLS private-key operations.

That means certificates can remain managed through AWS Certificate Manager while the TLS handshake's private-key operations happen inside the enclave. The gateway process never receives the private key itself.

Issuance and rotation remain within the AWS certificate-management workflow.

This is already running in production

KONE uses Zilla Plus to connect roughly 20 external applications and teams to MSK clusters running in private subnets, including SAP Integration Suite, mobile services, and monitoring systems.

Those connections use mutual TLS while the brokers remain private.

N Brown has also used Zilla Plus to simplify external Kafka integrations, reducing the amount of custom middleware and network integration work required to connect third-party systems into AWS.

The same architecture is used across environments including financial services, manufacturing, retail, and online gaming, where Kafka needs to remain private while external clients still require secure access.

Where the AWS update fits

AWS's new custom.advertised.listeners property is a meaningful improvement.

If you already use the NLB-based architecture with SASL/SCRAM or IAM, it eliminates one of the more awkward parts of maintaining custom domains on MSK.

But custom broker names are only one layer of the problem.

You still need to operate the network topology behind them, keep that topology synchronized as the cluster changes, and decide how to handle client identity when TLS terminates before reaching Kafka.

A Kafka-aware gateway takes a different approach: leave MSK private and unchanged, and handle external addressing, routing, authentication, and identity at the edge.

That is the model Zilla Plus uses.

Beyond custom domains

Once a Kafka-aware gateway is in the data path, the same architecture can handle more than connectivity.

Virtual clusters: Multiple teams or tenants can share one physical MSK cluster while each gets its own topic namespace, consumer groups, ACLs, and quotas, configured entirely through the proxy, with no changes to the underlying cluster. Adding a new tenant is a configuration change, not a new cluster.

Topic aliasing: External clients can address topics under names your organization controls, decoupled from whatever the topics are actually called inside the cluster. That means internal naming conventions can change, or teams can reorganize internal topics, without breaking every external consumer's contract.

Message validation: JSON, Avro, and Protobuf payloads are validated as they stream through the gateway, which catches malformed or poison-pill messages before they ever land in a topic, even in clusters that don't run a schema registry.

The same proxy also speaks MQTT, gRPC, SSE, and REST for non-native clients, supports OAuth 2.0/JWT auth, and integrates with AWS Shield and WAF for DDoS protection at the public entry point. 

AWS has made custom domains on MSK easier.

For teams that need secure external Kafka access, the larger architectural question remains: who owns everything between that domain and the private cluster?

Find out more

For the full technical breakdown, including how advertised.listeners behaves across the reference architectures AWS has published over the years and where each one's own documentation says it stops, see Custom Domains for Amazon MSK: Keep the Cluster Private, Keep mTLS Working on the Aklivity blog. 

For deployment details, including mTLS identity propagation, Nitro Enclave-backed TLS termination, and CDK, Terraform, and CloudFormation templates, head to docs.aklivity.io.