Operations·December 2024·10 min read

Real-Time Data and IoT: Transforming User Engagement on the Web

Real-time data in a commerce system isn't live sports scores or a fitness tracker; those are consumer UX patterns, not an architecture. The actual pattern doing the work is event streaming: a durable log or lightweight broker moving inventory and order events, CloudEvents standardizing what those events look like, and change data capture turning ordinary database writes into signals other services can react to.

What "real-time" means in an architecture, not a marketing slogan

Apache Kafka describes itself as a distributed event streaming platform, built to publish and subscribe to streams of records, store them durably, and process them as they occur (Apache Kafka — Documentation, Introduction). That's the actual technical definition worth anchoring to: a stream is a durable, ordered log, not a vague promise of instant updates.

The mistake in most explanations of "real-time" is collapsing very different systems, a stock ticker, a GPS app, a smart thermostat, into one concept because they all feel fast to a user. Architecturally they're solving different problems: high-throughput durable logging, lightweight low-power messaging, and simple request-response with polling, respectively. Only the first two are genuinely relevant to how a commerce backend should be built.

MQTT and Kafka solve different problems, not competing ones

MQTT is an OASIS standard, a lightweight publish-subscribe protocol designed specifically for constrained devices and bandwidth-limited networks (OASIS — MQTT Version 5.0). Kafka is built for the opposite end of the spectrum: high-throughput, durable, replayable event logs meant for servers, not battery-powered sensors.

Dimension	MQTT	Kafka
Designed for	Constrained devices, intermittent connectivity	Server-side, high-throughput event pipelines
Message retention	Typically transient, broker-dependent	Durable log, configurable retention, replayable
Delivery model	Publish-subscribe with QoS levels 0, 1, 2	Publish-subscribe with consumer-group offset tracking
Typical commerce use	Warehouse scanners, RFID readers, connected shelf sensors	Order events, inventory changes, pricing updates across services

A warehouse barcode scanner publishing a stock-count update over MQTT to a broker, which then forwards it into a Kafka topic for durable processing, is a common and sensible bridge pattern. Trying to run Kafka directly on a battery-powered device, or trying to build a replayable audit log on top of MQTT alone, both fight the tool's actual design.

MQTT and Kafka aren't competing standards. MQTT gets a signal off a constrained device efficiently; Kafka gives that signal a durable, replayable home once it reaches server-side infrastructure.

Partitions and consumer groups are what make this scale

A Kafka topic is split into partitions, and consumers reading that topic are organized into consumer groups where each partition is assigned to exactly one consumer within a group at a time (Apache Kafka — Documentation, Introduction). That's what lets a pricing engine scale horizontally: adding more consumer instances to the same group spreads partition assignments across them, and throughput grows without any change to how events are produced.

The tradeoff is that ordering is only guaranteed within a single partition, not across the whole topic. A producer that wants every event for a given SKU processed in order needs to key those events by SKU, so they consistently land on the same partition, rather than being spread arbitrarily for maximum parallelism.

CloudEvents: the envelope format nobody should invent twice

CloudEvents is a CNCF-graduated specification that fixes a small set of context attributes, id, source, type, specversion, and defines how they map onto concrete transports like HTTP, Kafka, MQTT, and AMQP (CloudEvents — Specification, CloudEvents). Without a shared envelope, every producer and consumer pair in a system ends up agreeing on its own ad hoc event shape, and that agreement quietly breaks the first time a new consumer needs to read an existing stream.

For a commerce backend, the payoff is direct: an inventory.updated event has the same envelope whether it's delivered internally over Kafka or externally as a webhook to a 3PL partner. The consuming code doesn't need a special case for "internal event" versus "external webhook," because CloudEvents' transport bindings normalize both into the same shape.

Change data capture: the real-time signal most commerce teams already have

Change data capture (CDC) tools like Debezium turn a database's own write-ahead log into a stream of change events, without requiring any change to the application code that writes to the database (Debezium — Documentation). Every insert, update, and delete on an orders or inventory table becomes an event other services can subscribe to, in near real time, without the application ever explicitly publishing anything.

This matters because it removes an entire class of "did we remember to emit this event" bugs. A developer adding a new field to the inventory table doesn't have to remember to also update three different event-publishing call sites scattered through the codebase; the CDC pipeline picks up the change at the database level automatically.

Snapshot, then stream

A CDC pipeline typically starts with an initial snapshot, a full read of the existing table's current state, before switching to streaming ongoing changes from the write-ahead log. That sequencing matters for a system going live: consumers get a complete, correct starting point rather than only seeing changes from the moment the pipeline was turned on and missing everything that existed before it.

The operational risk worth planning for is the snapshot's load on the source database. A large orders table can take real time and I/O to snapshot fully, and running that snapshot during a peak traffic window is a self-inflicted performance incident, not a CDC design flaw.

An event streaming architecture for commerce: producers, point-of-sale terminals, warehouse scanners, the storefront app, and change data capture off the orders and inventory database, all publish CloudEvents-shaped events into a durable log. Consumers, pricing, storefront cache invalidation, and 3PL notifications, read from that same log independently, each at its own pace.

Stream processing: reacting to events, not just relaying them

Moving events durably is only half the architecture. Kafka's own documentation describes stream processing as building applications that continuously transform input streams into output streams, rather than a batch job that periodically reads and rewrites a table (Apache Kafka — Kafka Streams Documentation). A pricing engine consuming the inventory-updated stream and continuously recalculating scarcity-based pricing is a stream processing application in this sense, whether it's built on Kafka Streams, a similar framework, or hand-rolled consumer code.

The distinction that matters operationally: a stream processor holds state, a running count, a windowed aggregate, that persists across events rather than treating each event in isolation. Getting that state management wrong, restarting a processor and losing its accumulated counts, is a common source of subtly incorrect aggregates that don't throw an error, they just quietly produce the wrong number.

Where IoT actually shows up in commerce: inventory signals, not smart mirrors

The commerce use case for IoT that actually has production traction is inventory visibility: barcode and RFID scanners at receiving docks and warehouse shelves publishing stock-level changes as events, rather than waiting for a nightly batch reconciliation job. Cloud IoT ingestion services from major providers exist specifically to bridge device-side protocols like MQTT into a cloud provider's own event infrastructure at scale (AWS — What is AWS IoT, Google Cloud — Pub/Sub Overview).

Speculative examples like smart mirrors or beacon-guided in-store navigation get disproportionate press coverage relative to actual deployment. The unglamorous, high-value pattern is a scanner event reaching the pricing engine and the storefront's "in stock" indicator within seconds instead of overnight, which directly reduces the number of orders placed against stock that's already gone.

The device-to-cloud path in practice

A typical path looks like this: a handheld scanner or fixed RFID reader publishes over MQTT to a local or regional broker, the broker forwards into a cloud IoT ingestion endpoint over TLS, and from there the event is normalized into a CloudEvents envelope and written onto a Kafka topic alongside events from POS terminals and the storefront app. Every hop after the device itself is standard server-side infrastructure; the device-specific part of the problem is narrower than it first appears.

Device identity and authentication deserve the same rigor applied everywhere else in a commerce stack. AWS IoT and Google Cloud's IoT-adjacent ingestion services both require per-device certificates or credentials rather than a single shared broker password, specifically because a shared credential compromised on one device compromises every device using it (AWS — What is AWS IoT). A warehouse scanner fleet is still an attack surface, even though it doesn't feel like one the way a public-facing API does.

A CDC pipeline is only as trustworthy as its starting snapshot. Streaming changes from a database that was never fully and correctly snapshotted first means every downstream consumer inherits a silently incomplete picture.

Failure modes that don't show up until production

Ordering guarantees are per-partition, not global. Two events for the same SKU landing in different Kafka partitions can be processed out of order relative to each other unless the producer keys by SKU.
At-least-once delivery means consumers must be idempotent. A consumer that isn't safe to process the same event twice will eventually double-decrement inventory after a retry.
Schema evolution breaks silently without a registry. A producer adding a required field to an event payload can break every consumer that doesn't expect it, unless schema compatibility is enforced centrally.
Backpressure needs an explicit plan. A slow consumer, a pricing engine doing expensive recalculation, can fall behind the event log's retention window and permanently lose events it never got to.

A pre-launch checklist for a new event stream

Confirm every event is keyed so that events needing relative ordering land in the same partition.
Confirm every consumer is idempotent against redelivery, not just correct on the happy path.
Register the event schema centrally and enforce compatibility checks before a producer can ship a breaking change.
Set retention long enough that a consumer down for maintenance can catch up without losing data.
Standardize on CloudEvents' envelope so a new consumer doesn't need a transport-specific parser.

FAQ

Do we need Kafka if we're already using MQTT for our devices?

Usually yes, as a bridge target rather than a replacement. MQTT gets the signal off the device efficiently; Kafka gives it durability, replay, and the ability to fan out to multiple downstream consumers.

What's the actual benefit of adopting CloudEvents instead of a custom event format?

Interoperability across transports and consumers without a bespoke parser for every producer. An event's shape stays consistent whether it travels over Kafka internally or as a webhook to an external partner.

Is change data capture safe to run against a production database?

CDC tools like Debezium read the database's existing write-ahead log rather than querying live tables, which is designed specifically to avoid adding load to the primary database's normal read and write paths.

What's the single most common production incident with event streaming?

A non-idempotent consumer processing a redelivered event twice. At-least-once delivery is the default assumption in most streaming systems, and consumers have to be built to handle duplicates safely.

Is smart-shelf or beacon-based retail IoT worth investing in?

The evidence base for those specific formats is thin relative to the press coverage they get. Warehouse and receiving-dock inventory scanning has the clearer, better-documented production track record.

How should a fleet of warehouse scanners or sensors authenticate to the cloud?

Per-device certificates or credentials, not a single shared broker password. Cloud IoT ingestion services from AWS and Google Cloud are both built around per-device identity specifically to contain a single compromised device rather than exposing the whole fleet.

References

From the Destm engineering archive. For current work on this topic, start at Solutions or the blog.