Replacing Dead-letters with Hangfire Failed jobs
I am a big fan of Hangfire, and my evangelism of the technology at work has earned a few more users.
The dashboard is a great tool for monitoring the health of a system, and the failed jobs in particular allow you to see what calls has gone wrong.
I am working on a stock system (.Net Framework MVC) that is receiving messages sent from scanners used by the factory forklift drivers when they move stock. There have been a few dead-letters in these early days of the project, but I have failed to quickly notice them. As yet, I don not have any dead-letter alerts set-up.
The stock system however, has email alerts sent to our helpdesk whenever a Hangfire job fails.
The easy win therefore, is to pass the messages to the stock system’s message router class via a Hangfire job.
I needed to tweak the existing message router class to allow it to be queued by Hangfire:
- The parameters to the class constructor needed to be moved to an instance method.
- Any logic in the constructor should also be moved to this instancfe method.
- All of the parameters needed to be serialised - a parameter of type ServiceBusReceivedMessage will not work.
Pre-Hangfire message queueing:
static async Task MessageHandler(ProcessMessageEventArgs args)
{
new MessageRouter(args.Message);
await args.CompleteMessageAsync(args.Message);
}
Post-Hangfire message queueing:
static async Task MessageHandler(ProcessMessageEventArgs args)
{
BackgroundJob.Enqueue(() =>
new MessageRouter().Route( args.Message.Body.ToString(),
args.Message.Subject,
Newtonsoft.Json.JsonConvert.SerializeObject(args.Message.ApplicationProperties))
);
await args.CompleteMessageAsync(args.Message);
}
Now, instead of chasing down dead-letters, I can view them in the Hangfire dashboard:
Restoring the application’s database containing the jobs into a test system, allows you to re-run the failed job and debug through the steps taken.