Category: .NET

As you start enqueuing more background jobs with Hangfire, you might need to increase the number of Consumers that can process jobs. Scaling Hangfire can be done in a couple of ways that I’ll explain in this post, along with one tip on what to be aware of when starting to scale out.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Producers & Consumers

First, let’s clear up how Hangfire works with producers and consumers.

A Hangfire producer is creating background jobs but not executing them. This is when you’re using the BackgroundJobClient from the Hangfire.

Once you call Enqueue, the job is stored in Hangfire JobStorage. There are many job storages you can use, such as SQL Server, Redis, and others.

The Hangfire Server, which is a consumer, will get that job from JobStorage and them execute the job.

There are two ways of scaling Hangfire to add more consumers. Worker Threads and Hangfire Servers.

Worker Threads

A Hangfire server can be hosted along with ASP.NET Core, or entirely standalone in a Console app using the Generic Host, which is the example below.

The Hangfire Server uses multiple threads to perform background jobs. Meaning it can process a background job per thread within the Hangfire server. This allows you to execute background jobs concurrently.

By default, the number of threads it uses is 5 per Processor Count. With a maximum of 20.

Math.Min(Environment.ProcessorCount * 5, 20);

However, you can configure this by setting the WorkerCount in the AddHangfireServer()

Constraints

Ultimately your constrained by the host where this Hangfire Server is running. Regardless if you’re running in a container, VM, or Physical Server, you’re going to be constrained by the CPU and Memory of the host. Meaning, you cannot just arbitrarily set the WorkerCount to a very high number as you could max out CPU if you had a high number of concurrent jobs that are CPU intensive.

You’ll have to monitor your application and the background jobs specific to your app to determine what the right number is. The default is a good default.

Hangfire Servers

The second option for scaling Hangfire is to simply run more Hangfire Servers.

Now when running two instances of my hangfire Server, the dashboard shows each server with 30 worker threads. This means our application can process 60 jobs concurrently.

Hangfire fully manages dispatching jobs to the appropriate server. You simply need to add servers to increase consumers, which ultimately increases the number of jobs you can process.

Downstream Services

Once you start scaling out by increasing the overall worker count or adding Hangfire servers, you will want to pay attention to downstream services.

If for example, your background jobs interact with a Database, you’re going to now be adding more load to the database because you’re going to be performing more jobs concurrently.

Just be aware that adding more consumers can move the bottleneck to downstream servers. They also become a constraint when trying to scale your system that’s using Hangfire.

Links

• Message Queue Overload from High Processing Latency
• Using Hangfire and MediatR as a Message Dispatcher

Two popular libraries in Hangfire and MediatR can be used together to create a pretty powerful out-of-process messaging dispatcher. I’ve covered this a bit many years ago but I figured I’d give it a refresh since it’s a bit easier in the world of ASP.NET Core.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Hangfire and MediatR Bridge/Wrapper

The first thing you need to do is create a bridge/wrapper around MediatR. At first this may seem completely pointless, but it has a purpose. Hangfire has a lot extensibility in terms of how jobs are executed that are often times defined by using attributes. Because we don’t own the MediatR library, we need to have our own class that we can defined these attributes on.

Here’s a simple start to our Bridge/Wrapper

In the example above, I have an overload for Send() that accepts the jobName as the first parameter. The DisplayName attribute will be used by Hangfire to show in the UI Dashboard the name of the job. This is a simple example of why we need this wrapper.

MediatR Extensions

The next piece of the puzzle is creating extension methods to be able to use Hangfire to create background jobs. In the example below I’ve created Enqueue() extension methods that use the Hangfire BackgroundJobClient to enqueue work using our Bridge/Wrapper

Hangfire Serialization

In the BackgroundJobClient.Enqueue() above, Hangfire will use Json.NET to serialize the IRequest that we are passing to Send() that ultimately gets put into storage. When Hangfire then pulls that job out of storage to perform the work, it needs to deserialize it. Because it’s just an IRequest it has no way to turning that into a concrete type.

To handle this, we need to configure Hangfire to add type handling when it serializes/deserializes.

ConfigureServices

The last thing we need to do is actually configure Hangfire in the ConfigureServices of either your ASP.NET Core Startup or your HostBuilder.

Here’s an example of my Worker process. This is just a console app that is purely a Hangfire server that just processes jobs from Hangfire.

Enqueuing a Request

In order to enqueue a MediatR request to Hangfire is a matter of calling the Enqueue() extension method we created off of IMediator.

In the example above, the HTTP Request to our Controller action will return a 204 NoContent, even though we are throwing in our PlaceOrderHandler because it’s execution is actually done out of the context of the HTTP Request. This could either be in a different thread, an entire different process, or on a totally on a different server).

More

There’s a lot more you can do with this and take it much farther to handle things like event publishing to have each event handler be it’s own job within Hangfire.

You can get all the source code of this running example on my GitHub.

If you have any thoughts or questions, please leave a comment on my blog, Twitter, or on the YouTube video.

Links

• Why use MediatR? 3 reasons why and 1 reason not
• Moving work Out-of-Process using Brighter and RabbitMQ