External pending queue storage

The great thing about Nexus is the control and insight you get into queue messages from using an SQL based queue storage. But for some cases the message throughput is just too high for it to make sense to use SQL based queues for everything.

This is where external pending queue storage in Nexus comes in, specifically the IExternalPendingStorage interface. It lets you use Nexus without forcing you to store pending messages in an SQL backed queue. One of the IExternalPendingStorage implementations in Nexus is Redis and using that means that message producers can enqueue directly to Redis or through the Nexus enqueue API and the message won't be stored in SQL at all. When the Nexus queue job starts it will fetch messages from Redis, process them and then delete them from Redis if nothing failed. If a message fails the message is inserted into the SQL version of the queue which gives you the ability to troubleshoot like you're used to with Nexus and typically don't get with a Redis backed queue. If you decide to retry the message it will get deleted from the SQL based queue and added to Redis again.

This is very useful for cases like event ingestion where you get tons of messages/events but don't really care about individual messages when they're in the happy path of Pending -> getting processed -> Processed/deleted. But you do care about failures and want the insights you get from using Nexus.

Supported packages

At the time of writing the Nexus.Redis, Nexus.RabbitMQ, Nexus.Kafka and Nexus.Azure packages offer external pending storage out-of-the-box.

Use it by calling the extension method from the different packages, eg:

builder
    .AddNexus()
    .AddRedisExternalPendingStorage();

And in your queue message set it as external like this:

[Queue("export_order", StorageMode = QueueStorageMode.ExternalPending)]
public class ExportOrderQueueMessage : IQueueMessageWithId
{ }

Using multiple different external storages

Note that you can use both eg Redis and RabbitMQ at the same time on different queues like this:

builder
    .AddNexus()
    .AddRedisExternalPendingStorage()
    .AddRabbitMQExternalPendingStorage();

[Queue("export_order", StorageMode = QueueStorageMode.ExternalPending, ExternalProvider = "redis")]
public class ExportOrderQueueMessage : IQueueMessageWithId
{ }

[Queue("import_order", StorageMode = QueueStorageMode.ExternalPending, ExternalProvider = "rabbitmq")]
public class ImportOrderQueueMessage : IQueueMessageWithId
{ }