Messager

Tech Stack: Go, gRPC, Protocol Buffers, TypeScript, Node.js, Express

Overview

Messager is a production-grade, push-based message broker built from the ground up in Go. It delivers at-least-once messaging with durable persistence via an append-only Write-Ahead Log (WAL), intelligent consumer load balancing, automatic retries, and dead-letter queue handling. Unlike pull-based brokers, Messager actively pushes messages to consumers over bidirectional gRPC streams — making it ideal for event-driven architectures, order processing pipelines, microservice communication, and any system that needs reliable, fault-tolerant message delivery without the operational overhead of heavyweight middleware.

Key Features

📨 Push-Based Message Delivery

Messages are actively pushed to consumers via gRPC bidirectional streams- No polling — consumers receive messages as soon as they are available- Configurable prefetch limits per consumer to control in-flight message count- Least-in-flight routing distributes load across competing consumers- Graceful handling when no eligible consumer is available

💾 Durable Write-Ahead Log (WAL)

Every message is persisted to disk before entering the in-memory queue- One .wal file per queue for isolated, independent durability- CRC32 checksums detect torn writes and corruption at replay boundaries- Full crash recovery — unacknowledged messages are restored on broker restart- Retry counts and message metadata survive restarts via WAL records

🔄 At-Least-Once Delivery with Retries

Explicit ACK/NACK protocol for consumer-driven delivery confirmation- Automatic implicit NACK via background retry scanner on dispatch timeout- Configurable MAX_RETRIES before messages are dead-lettered- Retry count increments and persists through WAL on each re-enqueue- In-flight tracking ensures messages are never lost mid-delivery

☠️ Dead-Letter Queue (DLQ)

Failed messages automatically routed to <queue>.dlq after max retries- DLQ is a first-class WAL-backed queue — not an in-memory side channel- DLQ messages survive broker restarts and can be consumed independently- Configurable DLQ TTL (default 30 days) with background expiration scanner- Main queues are never affected by DLQ TTL policies

⚖️ Intelligent Load Balancing

Dispatcher routes messages to the consumer with the fewest in-flight messages- Prefetch limits prevent slow consumers from being overwhelmed- Competing consumers on the same queue share work proportionally to capacity- Backpressure-safe — dispatcher blocks cleanly when all consumers are at capacity- No message drops under load; messages queue in the topic until a slot opens

🗜️ WAL Compaction

Atomic compaction rewrites WAL files, removing tombstoned (ACKed) records- Temp-file + rename pattern ensures zero risk of corruption during compaction- Prevents unbounded WAL growth and keeps restart replay times manageable- Per-queue or bulk compaction via the queue manager API- Survivor records verified before atomic swap

🖥️ Node.js Client & HTTP API

TypeScript gRPC client for publishing and consuming from any Node.js service- Express HTTP API for programmatic publish, consumer management, and health checks- Structured logging with component-level traceability- Order-event factory for realistic e-commerce workload simulation- Planned stress dashboard for browser-based load testing and live monitoring

Technical Implementation

Broker Architecture (Go)

-Go 1.26+ for high-performance, concurrent message processing-gRPC with Protocol Buffers for type-safe, streaming transport- Modular internal packages: wal, queue, connmgr, dispatcher, ackmgr, retry, dlqttl- Environment-variable-driven configuration with sensible defaults- Graceful shutdown with configurable drain timeout for in-flight messages- Race-detector-enabled test suite across all core packages

gRPC API

-Publish (unary RPC) — enqueue a message to a named queue, returns message ID-Subscribe (bidirectional stream) — consumer registers with queue + prefetch, receives pushed messages, sends ACK/NACK- Protobuf-defined message envelope with headers, retry count, and enqueue timestamp- Rejects new publishes once graceful shutdown has begun

Persistence Layer

Append-only WAL with [magic][length][protobuf payload][CRC32] record format- Tombstone records mark ACKed messages without physical deletion- WAL replay on startup rebuilds in-memory FIFO topics from disk state- Lazy queue initialization — WAL files created on first publish to a queue name- Directory scan discovers all *.wal files including DLQ queues on restart

Node.js Client

-TypeScript with @grpc/grpc-js and @grpc/proto-loader-Express 5 HTTP server exposing REST endpoints over the gRPC broker- Endpoints: GET /health, GET /status, POST /publish, POST /consumer/start, POST /consumer/stop- Graceful shutdown on SIGINT/SIGTERM — cancels streams and closes gRPC connection- Input validation on publish count (1–1000) and consumer ID conflict detection

Message Processing Pipeline

1.Publish & Validate: Publisher sends message via gRPC; broker rejects if shutting down2.WAL Append: Message record written to disk with status=pending before any in-memory change3.Enqueue: Message added to in-memory FIFO topic for the target queue4.Dispatch: Dispatcher dequeues and routes to least-loaded eligible consumer5.In-Flight Register: Ack manager tracks message before channel send (ordering invariant)6.Consumer Delivery: Message pushed via consumer's gRPC stream; in-flight count incremented7.ACK/NACK/Timeout: Consumer responds, or retry scanner fires implicit NACK on timeout8.Finalize: ACK writes WAL tombstone; NACK requeues with retry+1 or routes to DLQ

Use Cases

E-Commerce & Order Processing

Decouple order placement from fulfillment, inventory, and notification services- Route high-volume order events to multiple worker pools with load balancing- Dead-letter failed orders for manual review without blocking the main pipeline- Survive broker crashes without losing in-flight orders

Microservices & Event-Driven Architecture

Reliable inter-service communication without tight coupling- Push-based delivery eliminates polling overhead in downstream services- Prefetch control lets each service tune throughput to its processing capacity- WAL durability ensures no events are lost during deployments or restarts

Background Job Processing

Distribute tasks across a pool of workers with automatic load balancing- Retry failed jobs with configurable attempt limits before dead-lettering- Slow workers are naturally throttled via prefetch caps and least-in-flight routing- DLQ provides an audit trail of permanently failed jobs

Developers & Platform Engineers

Learn message broker internals: WAL, dispatch, ACK management, DLQ patterns- Five scripted scenarios demonstrate load balancing, crash recovery, durability, retries, and backpressure- Lightweight alternative to RabbitMQ/Kafka for single-node or learning workloads- Extensible gRPC API for building custom publishers and consumers in any language

Technical Highlights

-At-Least-Once Guarantees: Messages redelivered on NACK, consumer disconnect, or dispatch timeout-Crash-Safe Durability: WAL-always-first invariant; replay restores exact broker state on restart-Zero-Copy Compaction: Atomic temp-file rewrite drops tombstones without data loss risk-Concurrency-Safe: Mutex-protected WAL writes, race-detector-tested concurrent dispatch paths-Backpressure by Design: Dispatcher retry loop blocks cleanly — no unbounded memory growth or message drops-First-Class DLQ: Dead-letter queues are ordinary WAL-backed queues, not bolted-on afterthoughts-Observable: Structured logging via slog (Go) and component-tagged logger (Node.js)-Testable: Comprehensive unit and integration tests with five end-to-end shell scenarios

Performance Metrics

WAL throughput ceiling: ~2,000–5,000 messages/second on SSD (fsync-bound)- Dispatch latency: sub-millisecond for in-memory dequeue + consumer routing- Crash recovery: proportional to live WAL size (compaction keeps replay fast)- Consumer scaling: tested with competing consumers across varying prefetch and delay profiles- DLQ TTL scanner: hourly scan interval with configurable expiration window (default 30 days)- Graceful shutdown: configurable 15-second drain window for in-flight message completion

Future Enhancements

Browser-based stress dashboard with live consumer/publisher stats and load-test controls- Concurrent publisher support in the Node.js client for high-throughput load testing- Topic-based pub/sub routing beyond named queues- Prometheus metrics export for broker observability- TLS/mTLS support for secure gRPC transport- Multi-broker clustering and leader election for horizontal scaling- Message priority queues and delayed delivery scheduling- Web UI for DLQ inspection, replay, and manual message management- Client SDKs for Python, Java, and other languages

Collaboration

Messager was developed by Harsh Pandey — covering broker architecture, WAL design, gRPC API, dispatcher logic, retry/DLQ systems, Node.js client, and end-to-end test scenarios.

Impact

Messager makes message broker fundamentals accessible and production-ready without requiring heavyweight infrastructure. By combining push-based delivery, WAL durability, intelligent load balancing, and dead-letter handling in a single cohesive system, it provides a reliable foundation for event-driven applications — from e-commerce order pipelines to microservice communication — while remaining lightweight enough to run on a single machine for development, learning, and small-scale production workloads.

Getting Started

Clone the repository and start the broker:
bashgitclonehttps://github.com/harsh3dev/messager.gitcdmessagermakebuild-allgorun./cmd/broker
Publish messages:
bashgorun./cmd/publisher-queueorders-count20-interval100ms
Start a consumer:
bashgorun./cmd/consumer-queueorders-prefetch5-idworker-1
Run all test scenarios:
bashmakescenarios
Start the Node.js HTTP client:
bashcdnode-client && npminstall && npmrundev

Start the Go broker on port 500512. Publish order events via CLI or POST /publish3. Spawn consumers via CLI or POST /consumer/start4. Watch load distribution, retries, and DLQ routing in action
Open Source: Messager is open-source and available on GitHub. Contributions are welcome!

Technology Stack