Commerce Mind Nexus

Commerce Mind Nexus is a .NET (6+) based library and solution from Commerce Mind for running scheduled background functions and jobs, and populating and processing queues of messages. The solution is built primarily with e-commerce data flows in mind but it's a general solution that works outside of e-commerce as well.

Key features

• Distributed as NuGet packages, Apache 2.0 licensed
• Execute a class method with retries in the background using Nexus functions
• Schedule jobs to run every X seconds/minutes/hours/etc
• Out-of-schedule job runs
• Scale out and run jobs and functions on multiple servers
• SQL based queues with repository like access
• Versioned queue messages with optional identity to ensure safety
• Access to previous version of messages to enable sophisticated change tracking
• Virtual queues with one or more backing queues
• Processing queue messages in batch or one by one
• Full HTTP API and Admin UI
• Enqueue over HTTP API to let other services create messages
• Great introspection, debuggability, and visibility into the contents of the queues
• Easily re-enqueue failed messages
• Monitoring of failed jobs, queue messages, and functions
• Custom statuses for queue messages to allow grouping and saving for later
• Enqueueing messages for processing in the future
• Optionally keep processed messages
• Retry processing a queue message in X minutes
• Scheduled health checks
• Can easily be embedded into an existing application
• No external dependencies required, everything can be run locally on a dev machine

See the section about Nexus vs FaaS if you read this and thought "Well I already have Azure Functions for this".

Demo environment

There's a demo environment that's deployed to a free tier Azure Web App here:

Admin UI
https://commerce-mind-nexus.azurewebsites.net/admin/

API
https://commerce-mind-nexus.azurewebsites.net/swagger/

You can click on any buttons you like, start any job or function or do anything with the queues to get a feel for what the system does.

Getting started

The first thing you need to do is to get access to the NuGet packages which are hosted on a Github packages. Just talk to us and we'll add you to the repo and give you access to the NuGet feed!

Once you've gotten a Personal Access Token for the feed you need to configure NuGet to look for packages in this feed:
https://nuget.pkg.github.com/Commerce-Mind/index.json

You use your Github username as username and the Personal Access Token as password. Read more about the Github NuGet feed here:
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-nuget-registry

When you have access you need to install these NuGet packages in a .NET Core (6 or later) project:

CommerceMind.Nexus.Functions
CommerceMind.Nexus.Jobs
CommerceMind.Nexus.Queueing

This package is for the API and admin UI which is only relevant if you're using ASP.NET.

CommerceMind.Nexus.Api

When you've installed the packages you open your Program.cs and add this initialization code:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton<ExampleService>();
builder.Services.AddControllers();

builder.Services.AddNexusSingleServerInstanceEvents();
builder.Services.AddNexusFunctions();
builder.Services.AddNexusScheduledJobs(options =>
{
    // Logs are stored forever by default, make sure to set your own retention if you don't want
    // your logs table to fill up
    options.HistoricalRunsRetention = TimeSpan.FromDays(30);
});
builder.Services.AddNexusQueues();

// Only call this line if you're running a ASP.NET Core application
//
// Also make sure to call this last because it picks up all registrations of INexusHealthCheck
builder.Services.AddNexusApi();

var app = builder.Build();

// If you want the admin UI
app.UseNexusAdminUI();

app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.Run();

Now let's create the most simple functions, jobs and queues possible, make sure to place it in the same .NET project as your Program.cs file.

public class ExampleService
{
    public void ExampleMethod(int arg)
    {
        Console.WriteLine("ExampleMethod was called with arg: " + arg);
    }
}

public class ExampleJob : IScheduledJob
{
    private readonly IEnqueuer<ExampleQueueMessage> _enqueuer;
    private readonly INexusFunction _nexusFunction;

    public ExampleJob(IEnqueuer<ExampleQueueMessage> enqueuer, INexusFunction nexusFunction)
    {
        _enqueuer = enqueuer;
        _nexusFunction = nexusFunction;
    }

    public string DefaultSchedule => CronSchedule.TimesPerMinute(10);

    public async Task<JobResults> ExecuteAsync(CancellationToken cancellationToken)
    {
        await _enqueuer.EnqueueAsync(new ExampleQueueMessage());
        await _nexusFunction.RunInBackgroundAsync<ExampleService>(x => x.ExampleMethod(123));
        return JobResults.Completed("This went great!");
    }
}

public class ExampleQueueMessage : IQueueMessage
{

}

public class ExampleQueueProcessingJob : IScheduledQueueJob<ExampleQueueMessage>
{
    public string DefaultSchedule => CronSchedule.TimesPerMinute(10);

    public async Task<ProcessResults> ProcessMessageAsync(ExampleQueueMessage message, CancellationToken cancellationToken)
    {
        return ProcessResults.Completed("The message was processed!");
    }
}

Place a breakpoint in the ExampleMethod(), ExecuteAsync() and ProcessMessageAsync() methods and start the application. The breakpoints should get hit within a couple of seconds. Now we have a very basic example of one job scheduling a function as well as producing messages and placing them in a queue, and another job that process the messages.

If using a web application you'll also be able to access the API on eg /api/jobs and /api/jobs/ExampleJob/historical-runs. And the admin UI is available on /admin. With the above setup you're using SQLite as the database which is great for quickly getting started, but it's not really fit for a heavy production load. Read more about database setup in the database section.

When you've got the above working you should dig deeper into the docs here to understand everything you can do with the system!