Category: HTTP API

Optimistic Concurrency in an HTTP API with ETags & Hypermedia

Derek Comartin Leave a comment

How do you implement optimistic concurrency in an HTTP API? There are a couple of different ways, regardless of what datastore you’re using in the backend. You can leverage the ETag header in the HTTP Response to return a “version” of the resource that was accessed. When a client then needs to perform some operation on the resource, they send an If-Match header apart of the request with the value being the result of ETag from the initial GET request. Another option is to leverage hypermedia by returning URIs for actions relevant to a resource that include the version apart of the URI. This enables concurrency to be completely transparent and does not require any knowledge from the client.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

https://www.youtube.com/watch?v=2Nk3kCCyu3A&list=PLThyvG1mlMzk9bWLp4UzG4NFTYaWsc105&index=5

Optimistic Concurrency

In a concurrent environment like a web application or HTTP API, you have multiple concurrent requests that could be trying to make state changes to the same resource. The normal flow for optimistic concurrency is that clients will specify the latest version they are aware of when attempting to make a state change. As an example, two clients make a request for an HTTP request of GET /products/abc123

Optimistic Concurrency in an HTTP API with ETags & Hypermedia

The HTTP API returns the data along with a “version” property. The version indicates the current version of that resource. Now when a client makes a subsequent call to perform any type of action that’s going to result in a state change, it also includes the version. As an example, the client performs an inventory adjustment by making an HTTP call to POST /products/abc123/quantity but it also includes the version it received from the prior GET request.

Optimistic Concurrency in an HTTP API with ETags & Hypermedia

Now if the second client, which also did a GET request and also had version 15 of the resource, makes a similar HTTP call to do an inventory adjustment. It also includes the version it has, which is 15. However since the first client has already made a successful state change, the version is now 16.

Optimistic Concurrency in an HTTP API with ETags & Hypermedia

This request by the second client will fail because we’ve implemented optimistic concurrency. There are various ways you can implement ways beyond just using a version number, such as a DateTime or Timestamp that represents the last change or most recent version. Using a relational database, this will be implemented by including the version in the WHERE clause and then getting back the value of affected rows. If no rows were affected then the version isn’t what is the current value.

ETags & If-Match

Another way of passing the version around from server to client is by leveraging the ETag and If-Match headers in the HTTP Response and Request.

A good example of this implementation is with Azure CosmosDB. When you request a document, it will return an _etag property but also include the ETag header in the response. This represents the version of the resource.

Here is what the response body looks like:

Here are the headers from the response.

Optimistic Concurrency in an HTTP API with ETags & Hypermedia

Since we now have the ETag value, we can now use it when making a subsequent request to perform a state change. To do so, the request must pass the If-Match header with the ETag value.

The Cosmos SDK uses this exactly as illustrated within its API.

Hypermedia

Another way to pass around the version is simply by using Hypermedia. Hypermedia is about providing the client with information about what other actions or resources are available based on the resources it’s accessing. This means when a client requests the product resource GET /products/abc123, the server will provide it the URI to where it can do an Inventory Adjustment. Since we’re providing the URI, we can include the current version in the URI.

In the example above, I’m using EventStoreDB, which allows optimistic concurrency by passing the current version when appending an event to the event stream. If the version passed is not the current version of the stream, a WrongExpectedVersionException is thrown.

Here’s an example of what the response body looks like when calling GET /products/abc123

When we want to do an Inventory Adjustment, we aren’t constructing a URI, we simply use the response from the GET and find the URI in the commands array.

Optimistic Concurrency

There are many different ways to handle optimistic concurrency, and hopefully, this illustrated a couple of different options by using the ETags/If-Match headers as well as leveraging hypermedia.

Source Code

Developer-level members of my YouTube channel or Patreon get access to the full source for any working demo application that I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

Related Links

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Handling HTTP API Errors with Problem Details

Derek Comartin Leave a comment

If you’re developing an HTTP API, having a consistent way of returning errors can make handling errors on the client-side simpler. I’ve talked about Problem Details in another video, which is a standard for returning machine-readable details of errors in an HTTP Response. In this video, I’ll cover how to consume those HTTP API Errors from the client-side.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

https://www.youtube.com/watch?v=uvTT_0hqhyY

Problem Details

If you’re unfamiliar, Problem Details (https://tools.ietf.org/html/rfc7807) is a standard for providing errors from your HTTP API to consuming clients.

HTTP [RFC7230] status codes are sometimes not sufficient to convey enough information about an error to be helpful. While humans behind Web browsers can be informed about the nature of the problem with an HTML [W3C.REC-html5-20141028] response body, non-human consumers of so-called “HTTP APIs” are usually not. This specification defines simple JSON [RFC7159] and XML [W3C.REC-xml-20081126] document formats to suit this purpose. They are designed to be reused by HTTP APIs, which can identify distinct “problem types” specific to their needs. Thus, API clients can be informed of both the high-level error class (using the status code) and the finer-grained details of the problem (using one of these formats).

If you’re interested in how to produce Problem Details from your HTTP API, check out my other post: Problem Details for Better REST HTTP API Errors.

For the rest of this post, I’m going to use a simple ASP.NET Core route that is returning a problem details response that we’ll interact with.

Here’s a sample of what a Problem Details response body looks like. It will also have a Content-Type header of application/problem+json;

Handling HTTP API Errors with Problem Details

Consumer

To illustrate handling a Problem Details response in C#, we can provide the HttpClient with a custom HttpMessageHandler. This will allow us to inspect the response and check if the Content-Type response is of application/problem+json, and if it is, throw a ProblemDetailsException with the relevant properties.

Now when using an HttpClient, we can pass it our ProblemDetailsHttpMessageHandler that will throw and we can handle that specific exception.

When I run this sample console application, here’s the output from all the Console.Writeline.

Handling HTTP API Errors with Problem Details

Extensions

The real power for consuming Problem Details as HTTP API Errors is in the Type property and Extensions. The Type property in the Problem Details response is a URI that should provide developers at design time information about the error. This can include additional properties, called extensions, that may also exist in the response body. In my example above, “invalidParams” was an extension. My client code, albeit trivial, was looking for that specific type so that it could deserialize the response to a specific validation problem details type that I created, that was fully aware of that “invalidParams” extension.

If you leverage the Type property you can build very specific error types using extensions that your client can be aware of to provide a better experience.

Javascript & Typescript

Alex Zeitler created a nice package for parsing Problem Details that you can use within a Javascript / Typescript client. Check out his package http-problem-details-parser.

Here’s a sample using a mapper for the invalidParams extension.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Related Links

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Problem Details for Better REST HTTP API Errors

Derek Comartin Leave a comment

How do you tell your API Consumers explicitly if there are errors or problems with their request? Everyone creating HTTP APIs seems to implement error responses differently. Wouldn’t it be great if HTTP API Errors had a standard? Well, there is! It’s called Problem Details (https://tools.ietf.org/html/rfc7807)

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

https://www.youtube.com/watch?v=MfTLob6teJE

If you’re creating a RESTful HTTP API, Status Codes can be a good way to indicate to the client if a request was successful or not. The most common usage is 200 range status codes indicate a successful response and using 400 range indicate a client errors.

However, in most client error situations, how do you tell the client specifically what was wrong? Sure an HTTP 400 status code tells the client that there’s an issue, but what exactly is the issue?

Different Responses

Here are two made-up examples that were inspired by 2 different APIs that I’ve recently consumed. Both indicate there a client error, however, they both do this in very different ways.

This first example is using a Status Code of 400 Bad Request, which is good. They provide a response body that has a Message member which is human readable. There is also a documentation member which is a URI, I assume to give the developer more info on why it occurred.

Here’s another example but very different response.

This response has an HTTP 200 OK. Which is interesting, to say the least. Instead, to indicate success or failure, they include in the response body a success member was a Boolean. There is an error object which is useful because it contains an info member, which is human readable. But what’s really nice is the code and type members which appear to be machine-readable. Meaning we can read their documentation, and write the appropriate code to handle when we receive an error with code=101, then we might want to show our end-user some specific message or possibly perform some other type of action.

Commonality

So what do these 2 different HTTP APIs have in common when it comes to providing the client with error information?

Nothing.

They are both providing widely different response body’s and using HTTP status codes completely differently.

This means that every time you’re consuming an HTTP API, you have to write 100% custom code to handle the various ways that errors are returned.

At the bare minimum, it would be nice if there was a consistent and standard way to define the human-readable error message. In one of the responses, this was called “message”, in the other, it was “error.info“.

Problem Details

Wouldn’t it be great if there was a standard for providing error info to your clients? There is! It’s called Problem Details (RFC7807)

   HTTP [RFC7230] status codes are sometimes not sufficient to convey
   enough information about an error to be helpful.  While humans behind
   Web browsers can be informed about the nature of the problem with an
   HTML [W3C.REC-html5-20141028] response body, non-human consumers of
   so-called "HTTP APIs" are usually not.

   This specification defines simple JSON [RFC7159] and XML
   [W3C.REC-xml-20081126] document formats to suit this purpose.  They
   are designed to be reused by HTTP APIs, which can identify distinct
   "problem types" specific to their needs.

   Thus, API clients can be informed of both the high-level error class
   (using the status code) and the finer-grained details of the problem
   (using one of these formats).

Here’s an example using Problem Details for the first example defined above.

Problem Details

“type“: URI or relative path that defines what the problem is. In a similar way, as the first example had a “documentation” member, this is the intent of this member as well. It’s to allow the developer to understand the exact meaning of this error. However, this is meant to also be machine-readable. Meaning this URI should be stable and always represent the same error. This way we can write our client code to handle this specific type of error how we see fit. It’s acting in a similar way as an error code or error key.

“title“: A short human-readable message of the problem type. It should NOT change from occurrence to occurrence.

“status“: The status member represents the same HTTP status code.

“detail“: Human-readable explanation of the exact issue that occurred. This can differ from occurrence to occurrence.

“instance“: A URI that represents the specific occurrence of the problem.

Here’s another example using Problem Details from the second example above.

Problem Details

Type & Extensions

In the example above, traceId is an extension. You can add any members you want to extend the response object. This allows you to provide more details to your clients when errors occur.

This is important because if you use the type member, which is the primary way you identify what the problem is, then you can provide more extension members based on the type you return.

In other words, in your HTTP API documentation, you can specify a problem type by its URI, and let the developer know there will be certain other extension members available to them for that specific problem.

Multiple Problems

As with everything, nothing is perfect. Problem Details has no explicit way of defining multiple problems in a single response. You can achieve this by defining a specific type, which indicates there will be a problems member which will be an array of the normal problem details members. Just as I described above to leverage bot the type and extensions together.

ASP.NET Core

If you’re using ASP.NET Core, you can use Problem Details today in your Controllers. Simply call the Problem() which returns an IActionResult.

If you don’t have thin controllers and have business logic outside of your controllers, then you can use Hellang.Middleware.ProblemDetails by Kristian Hellang which is a middleware that maps exceptions to problem details.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design