Real‑Time Data Streaming Patterns Made Simple

What is Real‑Time Data Streaming?

Real‑time data streaming is a way to move data from one place to another the moment it happens. Think of it like a live video feed, but instead of pictures, you are sending bits of information.

Unlike batch jobs that wait until night to process a file, streaming works continuously. Every click, sensor reading, or transaction can be pushed instantly to the next system.

Pattern 1: Simple Push (Fire‑hose)

This is the most straightforward pattern. A data source (like a sensor) pushes each record directly to a consumer.

It works well when you have only one consumer and the volume is manageable.

// Node.js example – pushing temperature readings to a WebSocket
const WebSocket = require('ws');
const ws = new WebSocket('ws://localhost:8080');

setInterval(() => {
  const temp = (20 + Math.random() * 5).toFixed(2);
  ws.send(JSON.stringify({sensorId: 'temp-01', value: temp, ts: Date.now()}));
}, 1000);

Tips:

• Keep the payload tiny – only send what the consumer needs.
• Use a lightweight protocol (WebSocket, MQTT, or gRPC streaming).

Pattern 2: Pub/Sub (Topic‑Based)

Pub/Sub stands for publish/subscribe. The producer publishes a message to a topic. Any number of subscribers can listen to that topic.

This decouples producers from consumers. You can add new services without changing the original source.

// Kafka producer – sending click events
const {Kafka} = require('kafkajs');
const kafka = new Kafka({clientId: 'web', brokers: ['localhost:9092']});
const producer = kafka.producer();

async function run() {
  await producer.connect();
  setInterval(async () => {
    await producer.send({
      topic: 'page-clicks',
      messages: [{value: JSON.stringify({userId: 123, page: '/home', ts: Date.now()})}]
    });
  }, 500);
}
run();

Real‑world example: An e‑commerce site publishes order‑created events. The inventory service, the email service, and the analytics service all subscribe to that same topic.

Tips:

• Give topics clear, business‑oriented names (e.g., sensor‑temp, order‑events).
• Use partitions to scale horizontally – each partition can be processed by a separate consumer.

Pattern 3: Event Sourcing & CQRS

Event Sourcing stores every state‑changing event instead of just the current state. CQRS (Command Query Responsibility Segregation) separates writes (commands) from reads (queries).

When a user updates a profile, you store an event like ProfileUpdated. The read side builds a current view by replaying those events.

// Simple event store – appending to a file (for demo purposes)
const fs = require('fs');
function appendEvent(event) {
  fs.appendFileSync('events.log', JSON.stringify(event) + '\n');
}

// Example command
appendEvent({type: 'ProfileUpdated', userId: 42, changes: {email: 'new@example.com'}, ts: Date.now()});

This pattern shines when you need an audit trail, rollback capability, or want to rebuild projections for new features.

Tips:

• Never delete events – they are your source of truth.
• Build lightweight read models (tables, caches) that are updated by a stream processor.

Pattern 4: Windowed Aggregations (Stream‑Processing)

Sometimes you need to calculate something over a moving time window – like “average clicks per minute”. Windowed aggregation lets you do that in real time.

Frameworks such as Apache Flink, Spark Structured Streaming, or even Kafka Streams handle this.

// Kafka Streams – calculating clicks per minute
const {KafkaStreams} = require('kafka-streams');
const config = {kafkaHost: 'localhost:9092'};
const ks = new KafkaStreams(config);

const clickStream = ks.getKStream('page-clicks');
clickStream
  .window({type: 'tumbling', duration: 60000}) // 1‑minute windows
  .countByKey('page')
  .to('clicks-per-minute');

ks.start();

Real‑world scenario: A mobile game wants to show a live leaderboard that updates every 30 seconds based on scores streamed from devices.

Tips:

• Choose the right window type: tumbling (fixed), hopping (overlapping), or sliding (continuous).
• Be mindful of late‑arriving data; most frameworks let you set a “grace period”.

Choosing the Right Pattern

Not every use case needs a heavy‑weight solution. Ask yourself these questions:

1. How many consumers? One? Use Simple Push. Many? Go Pub/Sub.
2. Do you need an audit trail? If yes, consider Event Sourcing.
3. Do you need real‑time calculations? Use Windowed Aggregations.
4. What is the data volume? High volume → partitioned topics, scalable stream processors.

Start small. You can always evolve from a fire‑hose to a full Pub/Sub system as your product grows.

Actionable Takeaways

• Begin with the simplest pattern that meets your functional need.
• Give your streams clear, business‑focused names.
• Keep messages small – include only the fields the consumer requires.
• Use partitions or multiple topics to scale horizontally.
• Consider an event store if you need replayability or compliance.
• When you need live metrics, implement windowed aggregations with a stream‑processing framework.
• Monitor latency and back‑pressure; tools like Prometheus and Grafana can alert you early.

Real‑time streaming can feel magical, but it’s just a series of simple patterns put together. Pick the right one, stay observant, and you’ll build systems that react instantly to the world around them.