Overview
The article discusses SafetyCulture's implementation of Event Sourcing (ES) and Command-Query Responsibility Segregation (CQRS) as part of their cloud-based architecture. It highlights the benefits, challenges, and lessons learned from adopting these patterns in their engineering processes.
What You'll Learn
1
How to implement event sourcing using Apache Kafka
2
Why separating reads and writes enhances system performance
3
When to use protocol buffers for schema validation in event streams
Prerequisites & Requirements
- Understanding of event-driven architecture and microservices
- Familiarity with Apache Kafka and PostgreSQL(optional)
Key Questions Answered
How does SafetyCulture implement event sourcing in their architecture?
SafetyCulture implements event sourcing by using Apache Kafka as the event store and protocol buffers for message encoding. They publish all API incoming requests into an event source stream, which allows for stateless validation at the API layer and stateful validation in the stream consumer.
What challenges did SafetyCulture face with event sourcing?
SafetyCulture faced challenges such as computing the current state of entities for stateful validation and dealing with duplicate events in the event source. They addressed these by ignoring invalid events and applying updates to the read model while logging errors.
Why is eventual consistency important in event sourcing?
Eventual consistency is crucial in event sourcing as it allows for asynchronous updates across distributed systems. SafetyCulture manages this by implementing optimistic updates in their client applications, assuming changes succeed and handling errors as needed.
What are the benefits of using CQRS with event sourcing?
Using CQRS with event sourcing has provided SafetyCulture with better performance, scalability, and ease of reasoning about the system. It allows for the recreation of read models at any time and the addition of new read models to meet evolving business needs.
Technologies & Tools
Some links below are affiliate links. We may earn a commission if you make a purchase.
Backend
Apache Kafka
Used as the event store for event sourcing.
Database
Postgresql
Used for the read model database.
Tools
Protocol Buffers
Used for message encoding and schema validation.
Key Actionable Insights
1Implementing event sourcing can significantly improve system reliability and scalability.By adopting event sourcing, teams can maintain a clear history of changes, making it easier to debug and reason about system states over time.
2Using protocol buffers for message encoding ensures schema validation and compatibility.This approach allows for easier evolution of the system without breaking existing functionality, which is crucial in a fast-paced development environment.
3Separating command and query responsibilities can lead to enhanced performance.By using CQRS, SafetyCulture achieved better data governance and the ability to optimize read models independently from write operations.
Common Pitfalls
1
Relying solely on event logs for state reconstruction can lead to challenges in maintaining data integrity.
Without a command stream, teams may struggle with computing the current state of entities, making it essential to implement robust validation mechanisms.
2
Ignoring duplicate events in the event stream can corrupt data projections.
It's crucial to have mechanisms in place to identify and handle duplicates to maintain the accuracy of the data stored in projections.
Related Concepts
Event-driven Architecture
Microservices
Data Governance
Scalability In Distributed Systems