Filterable message properties

In some cases you might want to find messages where one or more property contains certain values. Like if you have a queue of inventory messages and you don't want to process it until the article data has come in. If your inventory message class looks like this:

public class InventoryChangeQueueMessage : IQueueMessage
{
    public required int ChangedQuantity { get; set; }
    public required string Sku { get; set; }
}

When the message is processed we check if an article exists with that SKU and if it doesn't we want to skip processing it, but we don't want to delete it. We'll schedule it to be retried later, but we want to minimize the wait time after the article has been created and when we process its inventory messages.

So when the article has been created we can update the status of messages by passing in a lambda like this:

class ProductController(IQueueItemUpdater<InventoryChangeQueueMessage> updater) 
{
    public async Task OnArticleCreatedAsync(string sku)
    {
        await updater.UpdateStatusWhereAsync(QueueItemStatus.Pending, m => m.Sku == sku);
    }
}

The lambda is translated into SQL and by default it will use the JSON functionality in the database to filter by message property. If you have a lot of messages in the queue that can become a performance issue as it won't be able to use any indexes for the query. If you're concerned about performance you should use the [Filterable] attribute as described below.

Reading queue items by property

If you have more advanced use cases than just updating status by message property you can use the IQueueReader to read queue items that matches a lambda filter. Eg:

class ProductController(IQueueReader<InventoryChangeQueueMessage> reader) 
{
    public async Task OnArticleCreatedAsync(string sku)
    {
        var items = await reader.ReadAsync(m => m.Sku == sku);
        foreach (var item in items)
        {
            // Do something interesting
        }
    }
}

[Filterable] attribute

If you want to filter on message properties and you're concerned about performance you should add the [Filterable] attribute on your property like this:

public class InventoryChangeQueueMessage : IQueueMessage
{
    public required int ChangedQuantity { get; set; }
    [Filterable]
    public required string Sku { get; set; }
}

This tells Nexus that you want enhanced filtering on the Sku property which means that we will extract the contents of that property into a separate database column that gets a database index to improve performance of filtering.

The [Filterable] attribute is allowed on properties of primitive type (including DateTime and Guid) as well as lists of primitive types. Eg:

public class InventoryChangeQueueMessage : IQueueMessage
{
    public required int ChangedQuantity { get; set; }
    [Filterable]
    public required List<string> Skus { get; set; }
}

Nexus will create a separate table per list property in order to efficiently filter on values for it.

Nexus will also automatically populate new filterable columns and tables from the messages in the queue when you've added [Filterable] attributes. This is done when the application starts up.

Nexus automatically creates database indexes for these columns and tables, but won't combine multiple columns in these indexes. Which means that if you want to filter on multiple properties in a single query you might want to create a combined index on those columns.

Lambda limitations

The lambda passed to IQueueItemUpdater and IQueueReader can combine and/or comparisons as well as grouping by parentheses such as m => m.Sku == sku && (m => m.Quantity == 1 || m => m.Quantity == 2).

The lambda can filter on deep properties such as m => m.Some.Deep.Object == value but note that the [Filterable] attribute only work on top-level properties on a message class. So deep properties like this will always use the less performant JSON operators in the database.

The lambda can check if a list of primitive values contains a value or not, such as m => m.ListOfSkus.Contains(sku) or m => !m.ListOfSkus.Contains(sku) but no other comparisons on lists are allowed. Dictionaries are not supported for filtering.

If you have a dictionary on your message and you can't convert that to a list you can create a separate filterable property that's derived from the dictionary instead:

public class ArticleChangeQueueMessage : IQueueMessage
{
    public required Dictionary<string, Category> Categories { get; set; }
    [Filterable]
    public required List<string> CategoryIds => Categories.Keys.ToList();
}

Filtering in the Admin UI

By default all properties with the [Filterable] attribute will be extra visible and filterable in the Admin UI. If you don't want a filterable property to show up in the Admin UI like that you can hide it by doing [Filterable(VisibleInAdminUI = false)].

Filtering guarantees

In order to guarantee consistency Nexus stores primitive filterable properties as columns in the queue table. Which means that you won't risk getting out-of-sync between the message itself and the filterable property column since it's committed to the database in the same query.

In the case of list properties Nexus will populate the separate tables after the messages are inserted/updated in a queue table, but it's done inside a transaction to prevent inconsistency. If you're explicitly using an isolation level in the database that allows dirty reads (such as read uncommitted in SQL Server) you risk reading or updating the wrong messages.