Chanh LeIn previous posts, we’ve explored the fundamentals of Message-Oriented Architecture (MOA) and how Redis is utilized as the backbone for our alarm message queue. Now, let’s take a step back and examine the overall architecture. This post provides an overview of the system, highlighting the different components and the reasoning behind key design choices.
Understanding the Big Picture
Our system is designed to handle the reliable delivery of alarm messages from alarm devices to alarm receiving centers (ARCs). The architecture follows a message-oriented pattern, ensuring the system can scale to handle high volumes of messages efficiently while maintaining reliability and low latency.
Here’s a breakdown of the key components:
1. Alarm Devices
These are physical devices that detect and report alarms such as fires or security breaches. Each alarm device is responsible for sending messages with key information to the system:
- Message ID: A unique identifier for each alarm event.
- Signal Code: A predefined code that specifies the type of alarm being triggered.
- Sender ID: The ID of the device that triggered the alarm.
- Criteria Index: Parameters specific to the source device that triggered the alarm.
These messages are then pushed to the message queue for further processing.
2. Redis Message Queue
At the core of the architecture, Redis is used as a Message Queue, but with a specific approach:
Redis Key: Instead of placing the full message in the queue, we only push the Redis key (which points to the message hash) into the queue. The Redis key is prefixed by
message:followed by the unique message ID.Message Hash in Redis: Full details about each message—such as its message ID, signal code, criteria index, device ID, destination group, current state, phase, destination information, and TTL—are stored in Redis as a hash. The message hash is assigned an expiration time equal to the TTL of the message to prevent indefinite growth.
3. Processing Layer
This is where alarm messages are taken from the queue and processed by various services in a controlled and scalable manner.
Message Consumers: Consumers retrieve Redis keys from the message queue, fetch the corresponding message hash from Redis, and process the alarm accordingly. The system updates the message state (e.g., "received", "processing", "dispatched") during each stage.
State and Queue Management: The system tracks the message’s current queue and state to handle different stages of the processing workflow.
Timeouts and Retries: If a message isn't processed within its TTL, the system either retries or escalates the issue to ensure critical messages are addressed.
4. Alarm Receiving Centers (ARCs)
Once an alarm message has been processed and routed, it is delivered to an ARC, responsible for handling the final response to the alarm (e.g., dispatching emergency services). The ARCs are decoupled from the processing system, allowing it to scale and process alarms even if some ARCs are temporarily unavailable.
5. Monitoring and Logging
Real-time monitoring is vital to this system. The system tracks:
- Queue Length: Ensuring the message queue doesn’t grow uncontrollably.
- Processing Time: Monitoring how long each message takes to be processed.
- Health Checks: Ensuring the health of each component and alerting the system to potential failures.
Key Design Choices
Message-Oriented Architecture
The system’s message-oriented architecture allows for decoupling between the various components, improving scalability and resilience. Each layer operates independently, allowing flexibility in how messages are processed and how the system scales to handle high message throughput.
Redis as Message Queue and State Store
Redis provides the necessary performance and flexibility. By using Redis for both the message queue and as a hash-based state store, the system ensures efficient, real-time processing of alarm messages. Only the Redis key is pushed into the queue, and the full message is stored in Redis as a hash with additional fields like the message ID, signal code, criteria index, and destination group.
Expiration-Based Message Cleanup
To prevent indefinite message growth in Redis, message hashes have an expiration time set equal to the TTL of the alarm message. This keeps the system efficient and lightweight by removing old messages when they are no longer needed.
Future Enhancements
As the system scales, several improvements could be implemented:
Redis Streams: Transitioning from Redis Lists to Redis Streams could provide better support for advanced features such as consumer groups and message acknowledgment.
Event-Driven Architecture: Implementing an event-driven system could further enhance the responsiveness of the system by integrating external services like analytics.
Redis Clustering: As the message volume increases, Redis clustering can be introduced to ensure consistent performance under high load.
Conclusion
This architectural overview highlights how the system efficiently processes alarm messages in real-time, leveraging Redis to maintain a high level of performance and scalability. By pushing only the Redis key into the message queue and storing the full message in a hash with an expiration, we maintain efficiency and avoid unbounded growth.
In the next post, we will explore the lifecycle of an alarm message and how it transitions through the different states in our system.
