Retrying queue messages

When you implement a queue processing job you might end up in a situation where you can't process the message right now but you know that you'll soon want to try it again. Such as if you detect that a system that you want to send the message to is down at the moment so you want to try again in X minutes.

Nexus lets you handle that by returning ProcessResults.RetryLater() from a queue job or call batch.RetryLater() in a batch queue job. You specify the amount of time to wait and then Nexus will handle the rest.

Nexus will place the message in a new status called AwaitingRetry and it will sit there until it's time to try agian. At that point the message status is set to Pending again and the processing job will pick it up again.

Automatic retries

You can also instruct Nexus to automatically retry messages in a queue. In the Admin UI you can click on the More button on the queue page and set the field "Automatic error retry" to eg 24h (or 1d or 1h 30m etc) which will automatically retry any message that gets the Error status after that time period. Note that the time period is per message, and does not run periodically to batch update all Error messages to Pending.

You can also control this through code by either setting it in the [Queue] attribute:

[Queue("my-queue", AutomaticRetryAfter = "00:15:00")]
public class MyQueueMessage : IQueueMessage 
{

}

AutomaticRetryAfter represents a string that can be passed to TimeSpan.Parse() so in the example above it'll be retried 15 minutes after it first failed.

Another way of setting it through code is to use IQueueMetaDataRepository.SetRetryErrorsIntervalAsync() which lets you set a runtime value for it. This is the same method that gets called when updating it in the Admin UI/API.

If no value has been set through IQueueMetaDataRepository Nexus falls back to what's set in the attribute, and if no value is set in the attribute no automatic retries are performed.

Automatic retries respects the max retries for a queue which is described below.

Max retries

To avoid having messages that end up in an inifinite loop of retries Nexus has a default limit on how many times a message can be retried. The default limit is 10 and once a message has reached that number of retries it will get the status Error and a comment saying Message was retried too many times (10) and has been placed in the Error status. At that point it will not be automatically retried again but you can always manually set it back to Pending which gives the message 10 more tries.

If you want to change the limit of max number of retries you can either set it on all queues like this in your Program.cs:

builder.Services.AddNexus().AddQueues(options =>
{
    options.DefaultMaxRetries = 5;
});

You can also set this per queue using the [Queue] attribute:

[Queue("example", MaxRetries = 5)]
public class ExampleQueueMessage : IQueueMessage
{

}

Note that you can allow an infinite number of retries by setting -1 as the max.

Status when max retries

By default the status becomes Error when a message has been retried too many times. But you can control which status it gets by passing the status as part of the ProcessResults.RetryLater() call, like this:

return ProcessResults.RetryLater(TimeSpan.FromMinutes(5), statusWhenMaxRetried: "MyErrorStatus");