CodeOpinion: Software Architecture & Design

Sponsor: Do you build complex software systems? See how NServiceBus makes it easier to design, build, and manage software systems that use message queues to achieve loose coupling. Get started for free.

  • Don’t Fail Publishing Events!

    Consistency is critical when working with an event-driven architecture. You must ensure that when you make state changes to your database, the relevant events you want to publish are published. You can’t fail publishing events. Events are first-class citizens, and when events drive workflows and business processes, they rely on this consistency between state changes and published events.

    YouTube

    Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

    https://www.youtube.com/watch?v=tcePbob8rrY

    Atomicity

    Here’s an example of placing an order. At the very bottom of this method, we publish an OrderPlaced event and then save our state changes to the database.

    https://gist.github.com/dcomartin/564694fba608188ebc804f38388c7aa1

    These last two lines are problematic. The first reason this is an issue is that we have a race condition. We could process the event before the stat is saved to the database. This is more true if there was any code between publishing and saving the database changes.

    To illustrate this, on line 24, we publish the event to the message broker.

    Race Condition

    Once the event is published to the broker, we could have a consumer immediately process that message.

    Race Condition

    If this consumer was within the same logical boundary as the publisher, let’s say to send out the confirmation email, it might reach out to the database to get the order details.

    Race Condition

    But since we haven’t yet saved our database changes, the order won’t exist yet in our database.

    Finally, line 25 is when the order is saved to our database.

    Race Condition

    But what happens if saving the order to our database (line 25) fails? Now we’ve published an event (which is a fact) that an order was placed, but really, an order wasn’t placed because we didn’t save it.

    If we have downstream services that are apart of the workflow, this is misleading and could have many different implications and failures in different services.

    We need to flip the two lines around to avoid a race condition and not publish an event without saving the order.

    https://gist.github.com/dcomartin/eb318479e183865d7bc59d580932c7a0

    We still have an issue. Suppose we save the order and fail to publish the OrderPlaced event to our broker. If we have downstream services that are part of a workflow, they’ll never know an order was placed.

    We can’t fail to publish an event if we have a state change.

    Fallbacks

    One solution is to have a fallback. If we can’t publish to the message broker, we have another place to store the event durably.

    In my post on McDonald’s Journey to Event-Driven Architecture, they used a fallback for this.

    Fallback Storage

    In the example with McDonald’s, they used DynamoDB as their fallback storage. So if they could not publish to their message broker, they would save the event in DynamoDB. I also reviewed Wix.com – 5 Event Driven Architecture Pitfalls, where they used AWS S3 to save events in case of this type of failure.

    From there, you’d have some retry mechanism that would pull the data from your durable storage and have it try and publish to your broker.

    Fallback Retry

    As an example, you could use a retry and fallback policy.

    https://gist.github.com/dcomartin/235211075d417848602a434036c24aa0

    The downside with a fallback is you have no guarantee that you’ll even be able to save that event to durable storage if there’s a failure to publish the event to your broker. There’s no guaranteed consistency.

    Outbox Pattern

    Another solution, which I’ve talked about before, is the Outbox Pattern: Reliably Save State & Publish Events.

    This allows you to make state changes to your database and save the event to the same database within the transaction. Your event data would be serialized in an “outbox” table/collection within the same database as your business data.

    Outbox

    Then you have a separate process that reads that “outbox” table/collection and deserializes it into an event.

    Then it can publish that event to the message broker. If there are any failures in publishing the event, the publisher would simply keep retrying.

    Outbox Publisher

    Once the event is successfully published, the publisher must update the database to mark that event in the outbox table as being published.

    Outbox Publisher

    If there is a failure to update the outbox table, this will result in the publisher publishing the same event more than once, which requires consumers to be idempotent.

    The downside to the outbox pattern is your adding more load to your primary database since the publisher.

    Workflows

    There are also workflow engines that provide guarantees of the execution of parts of a workflow. Let’s say workflow with 3 distinct activities: Create Order, Publish OrderPlaced Event, and Send Confirmation Email.

    Each one of these activities is executed independently in isolation and the first to execute would create and save our order to the database.

    Workflow

    After the create Order activity completes, the Publish Event activity will execute to publish the OrderPlaced event to our broker.

    Workflow

    Now, if there’s a failure to publish the event, this activity could retry or have various ways to handle this failure depending on your tooling. Once the activity succeeds, it moves to the next which could send out the confirmation email.

    Workflow

    The key is that each activity is guaranteed to run. If the Create Order activity is completed, our Publish Event will execute. This eliminates the need for a fallback or an outbox.

    Your Mileage May Vary

    Two simple lines of code can have a large impact on the consistency of your system. Don’t fail publishing events! As you can see there are different ways to handle reliably publishing events and saving state changes, and which you choose will depend on your context. Hopefully, you can see the trade-offs for each and which will fit best for you.

    Join!

    Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

    Follow @CodeOpinion on Twitter

    Software Architecture & Design

    Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

  • Testing WITHOUT Mocks or Interfaces!

    A common approach people take with testing is mocking. Specifically defining interfaces for dependencies which are then typically mocked so you can test in isolation. While interfaces can be helpful for mocking as well as fakes and stubs, there can be other approaches taken. Meaning you don’t need to create an interface for everything.

    YouTube

    Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

    https://www.youtube.com/watch?v=2oRofOH2NT4

    Deterministic

    Let’s start with a method in an OrderService for creating an Order.

    https://gist.github.com/dcomartin/a9a4df9e6371dc7d7007f0f75d76e31c

    To test this method, there are a few dependencies involved. The OrderRepository, ItemRepository, URIComposer. Here’s what the test might look like by using a mocking library.

    https://gist.github.com/dcomartin/b0653b94fc6bf58ffb09aeb19794c074

    The problem is, as I’ve described above, is this test is flaky. That’s because the OrderDate is being set by DateTime.UtcNow. That’s non-deterministic.

    Sure, we could be more lenient on our assert by maybe using a small range/window, but ultimately we want the result to be deterministic.

    Interfaces

    You could jump directly to an interface, which I’ve seen quite a bit of for this exact usage case with DateTime.

    https://gist.github.com/dcomartin/0d075a21c2ada9e3536113d61b17e98e

    With the appropriate registration with the ServiceCollection, we can now inject a ISystemDateTime instead of calling DateTime.UtcNow.

    https://gist.github.com/dcomartin/be6653473715d1c1fedaa20c18dd8286

    We can be deterministic within our test by returning a specific DateTime for UtcNow.

    https://gist.github.com/dcomartin/5424e0140040ca05fdf85f5b99e43b82

    Function

    If you have a class/interface with one method, you have a function. The ISystemDateTime is exactly that. We have more options than interfaces when it comes to abstractions. In this case, using a delegate is also an option.

    https://gist.github.com/dcomartin/cfd7119c4807a81a6a3b77c2dca40cc2

    Again, by registering this delegate and the static method implementation, we can inject that delegate rather than an interface.

    https://gist.github.com/dcomartin/7c0ba4470ddeee47b7400ba334d152f7

    Our test becomes less cumbersome as we can easily create a stub for returning a deterministic DateTime without additional libraries or dependencies.

    https://gist.github.com/dcomartin/441b9f97ca28ff5bef268788f95a0db2

    Values

    As we break this down, you might also wonder why we even inject some abstraction when creating the order but instead pass the value of the DateTime to the order creation—basically moving up the call stack.

    https://gist.github.com/dcomartin/7a45b8ca51d9269910218efebfbee11c

    Now the caller is responsible for passing a DateTime to the CreateOrder, simplifying the test even more as we no longer have a dependency to pass to the OrderService

    https://gist.github.com/dcomartin/0659b8ffc8e568c6adb44a39341789ac

    Abstract Classes

    Have you ever needed to test an implementation that needed to use HttpClient? If so, you’re faced with the same issue where you want a deterministic result.

    First, here’s an example of an ExchangeRateClient for getting currency exchange rates.

    https://gist.github.com/dcomartin/b335ecf2843132b04cef02fe8336b2ea

    HttpClient doesn’t have an interface. So how do you test? Well, it does support providing your implementation of an HttpMessageHandler, where you must implement the SendAsync method.

    https://gist.github.com/dcomartin/1ae12069476fb5d39a60661a9e7d8363

    Abstractions

    You don’t always need to default to interfaces. You have other abstractions like delegates and abstract classes, and changing your design to move non-deterministic calls up the call stack so you can pass values instead.

    Join!

    Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

    Follow @CodeOpinion on Twitter

    Software Architecture & Design

    Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

  • Domain Events in Event Sourcing? Not Exactly!

    What are domain events for? Domain Driven Design and Event Driven Architecture are nothing new, but they continue to become more popular. Unfortunately, some concepts and terms get a bit murky as they do. Specifically, the term domain events have caused a bit of confusion. I will clarify a misconception about the role and purpose of domain events and how that relates to Event Sourcing. Or does it?

    YouTube

    Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

    https://www.youtube.com/watch?v=dJBTNksQzys

    Event Sourcing

    As a primer on Event Sourcing, it’s a way of recording state. As a comparison, if you were recording the current state in a relational database or document store, you’d think of rows/tables or objects/collections.

    As an example, let’s say we have products in a warehouse. We’re recording the SKU (productID), the quantity on hand, and when we last received and shipped the product from our warehouse.

    Relational Database

    With event sourcing, however, we’re recording events that represent the state transitions.

    Event Sourcing Domain Events

    Each product, represented as a row in our relational database table, has a stream of events in our event store.

    For more basics on event sourcing, check out my post Event Sourcing Example & Explained in plain English

    State

    Event Sourcing is about using events as state. This doesn’t mean they are used as a form of communication.

    Thankfully over the last several years, for the most part, the industry is accepting of not integrating at the database. Meaning instead of being able to interact with another service database directly, you must use a defined API. Gone are the days of calling another database directly.

    Unfortunately, this idea seems lost when Event Sourcing—often thinking of events as am means of communication with Event-Driven Architecture. However, event sourcing is about state. Just because you have an event store, does not mean another service can reach out to your event store.

    Domain Events

    Domain events are inside events. They are internal to a logical boundary. Their purpose is for notification. To express that some business concept has occurred. There are situations where you may want to expose them to other boundaries. Often this is because they are well-established business concepts that aren’t going to change. In this case, exposing a domain event is reasonable. Treating a domain event also as an integration event. For more on this, check out my post Should you publish Domain Events or Integration Events?

    So then, aren’t the events used in event-sourcing domain events? They are internal events. Yes, they can be, but they don’t have to be. Not every event persisted to your event store has to be a domain event. Likely they will, in most cases, represent business concepts, but as mentioned, they are also about state transitions.

    You’re using events in event sourcing to reconstruct an internal state within an aggregate, so when you then perform an action (do this!), it can validate if it’s in a state to perform the action. And if valid, then appends the subsequent event that represents the state transition.

    Events in event sourcing may be more fine grain than what is needed for communication purposes. Domain events may be less granular and represent the completion of a workflow.

    As a typical example of a shopping cart and checkout process, let’s say we have an event stream that is:

    • ItemAddedToCart
    • ItemAddedToCart
    • ItemRemovedFromCart
    • ShippingInformationDefined
    • BillingInformationDefined
    • CheckOutCompleted

    Are these events used to notify inside (and possibly) outside boundaries? We surely need them as our state transitions (event sourcing), but do we need them for communication (domain events)?

    For communication, we probably want an OrderPlaced event to define the completion of our workflow. It likely will be a domain event (or integration event) to notify other boundaries. Still, all the others are probably too fine grain.

    Domain events are about business concepts that the business cares about and understands. They are used for notifications/integrations within or outside a service boundary.

    Events used for Event Sourcing are about state. They are capturing events to represent the transitions in state.

    Domain events can be used for event sourcing, but not all events in event sourcing are domain events.

    Join!

    Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out my Patreon or YouTube Membership for more info.

    Follow @CodeOpinion on Twitter

    Software Architecture & Design

    Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design