Instance events

Nexus is built to work on multiple instances/servers at the same time in a scaled out environment. You can have as many servers running jobs as you like and the system still guarantees that a single job will only run on a single instance at any given time and that only a single queue message is processed by a single server.

In some cases the system needs to communicate with all running instances which is called instance events. At the heart of it is IInstanceEvent and IInstanceEventBroker.

IInstanceEvent represents an event that should be sent to either all or a specific instance. Specific implementations of the interface exists for different cases such as broadcasting a message to cancel a job or a message to start queue processing for a specific queue.

IInstanceEventBroker is the interface to use when you want to listen to or broadcast these events and it's up to the implementation of it to make sure that the events are sent to the correct instance(s).

Single instance mode

There's two implementations of the broker interface built in. One that you can use if you only run a single instance at any given time. It uses a memory based solution to register listeners and broadcast events. You register it by calling this in your DI setup code:

builder.Services.AddNexus().AddSingleServerInstanceEvents();

Multi instance mode

If you have multiple instances running like in an auto scaled environment where there might be multiple servers jobs running you can't use the single instance mode. Your options are to either use database polling, use Azure Service Bus, or write your own broker.

Both the database polling and Azure Service Bus implementation takes an option func that lets you specify BrokerType which is either InstanceEventBrokerType.ProduceAndConsumeEvents or InstanceEventBrokerType.ProduceEvents. For an application that only adds messages to Nexus queues using the CommerceMind.Nexus.Queueing package you can set the broker type to InstanceEventBrokerType.ProduceEvents. This means that Nexus will skip setting upp polling/subscriptions for that application.

builder.Services.AddNexus().AddDatabasePollingInstanceEvents(options =>
{
    options.BrokerType = InstanceEventBrokerType.ProduceEvents;
});

// Or:
builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options =>
{
    options.BrokerType = InstanceEventBrokerType.ProduceEvents;
});

Database polling

If you want to limit external dependencies you can use the database to store instance events and have your servers poll the database for new events. You configure database polling events like this:

builder.Services.AddNexus().AddDatabasePollingInstanceEvents(options =>
{
    options.PollDelay = TimeSpan.FromSeconds(1); // Default polling is every second
});

This implementation will check if there's multiple running instances and then save a copy of the event for each running instance except for itself where it broadcasts the event directly. It knows which instances are running by using the INexusInstanceHeartbeatService. Read more about heartbeats here.

RabbitMQ

If you're using RabbitMQ you can use that as your instance event broken. Install the package CommerceMind.Nexus.RabbitMQ and configure the instance event broker:

builder.Services
    .AddNexus()
    .AddRabbitMq(options =>
    {
        options.UserName = "...";
        options.Password = "...";
        options.HostNames = ["localhost"];

        // Or:
        options.ConnectionString = new Uri("amqp://...");
    })
    .AddRabbitMqInstanceEvents(options => 
    {
        // This is the default value, you only need to change this if you're using
        // multiple Nexus applications with the same RabbitMQ instance.
        options.ExchangeName = "nexus_instance_events";
    });

Azure Service Bus

If you're using Azure there's a NuGet package called CommerceMind.Nexus.Azure which uses a service bus topic to broadcast instance events.

After you've installed it you configure it like this in your Program.cs:

builder.Services.AddNexus().AddAzureServiceBusInstanceEvents();

The default name of the topic is nexus-instance-events but that can be configured by passing an options func like this:

builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options =>
{
    options.TopicName = "my-topic-name";
});

By default Nexus will look for the connection string using IConfiguration.GetConnectionString("azureServiceBus") but you can add a loader func on the options object if you need to fetch it from somewhere else:

builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options =>
{
    options.ConnectionStringLoader = (serviceProvider) => "Endpoint=sb://xxx.servicebus.windows.net/;...";
});

Nexus will automatically create topic subscriptions for each instance which works best in an auto scaled environment where you don't know how many instances you will have. In order to be able to create subscriptions on-the-fly you also need to configure an administration connection string. You do that either by setting the connection string azureServiceBusAdmin with IConfiguration or set the AdminConnectionStringLoader property on the options object.

If you're not using an auto scaled environment and you instead want to create the subscriptions yourself you can skip configuring the admin connection string and instead add SubscriptionNameLoader:

builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options =>
{
    options.SubscriptionNameLoader = (serviceProvider) => "my-subscription";
});

Batching events

By default instance events are batched so that events that happen within 0.5 seconds are sent as a single service bus message. This is done to limit the number of messages since Azure Service Bus has a cost per message. If a 0.5 second delay doesn't work for you it's possible to change the delay like this:

builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options =>
{
    options.DebounceDelay = TimeSpan.FromSeconds(5);
    options.MaxMessageBatchSize = 10; // The default is 10
});

The MaxMessageBatchSize is to guard against getting a message size that is bigger than what's allowed. So if there's more than 10 events within 0.5 seconds they will get sent as mutiple messages. If you're using the standard tier service bus the limit is 256 kB but if you're using the premium tier the limit is 100 MB.

Writing your own broker

If you don't feel like any of the above options works for you then you can always create your own implementation of IInstanceEventBroker.

An important thing to think about is that IInstanceEvent has an InstanceId property that can be null or have a value. If it's null then the event should be sent to all instances. But if it's not null it should only be published as an event on the instance where INexusInstance.InstanceId is the same as IInstanceEvent.InstanceId.

You can of course decide to send the event to all instances and then do the check in your implementation before calling OnInstanceEventRecieved?.Invoke() if the instance id matches.

When you're done with the implementation all you need to do is register it like this:

builder.Services.AddSingleton<IInstanceEventBroker, MyInstanceEventBroker>();