# Middleware for Inngest functions > Observe and wrap function runs, steps and event sends with InngestMiddleware, for logging, metrics, tracing, error mapping, event enrichment, encryption and reporting errors to Sentry. Web page: https://damikun.github.io/inngest-dotnet/guides/middleware/ Middleware runs code around every function call, step and event send of the app, without touching the functions themselves. Derive from `InngestMiddleware`, override the hooks you need and register the class. ```csharp title="TimingMiddleware.cs" using System.Diagnostics; using Inngest; public sealed class TimingMiddleware(ILogger logger) : InngestMiddleware { public override async Task WrapStepHandlerAsync(FunctionContext context, StepInfo step, Func> next) { var started = Stopwatch.GetTimestamp(); try { return await next(); } finally { logger.LogInformation("Step {Step} of {Function} took {Elapsed}", step.Id, context.FunctionId, Stopwatch.GetElapsedTime(started)); } } } ``` ```csharp title="Program.cs" builder.Services .AddInngest(options => options.AppId = "shop") .AddMiddleware(); ``` ## Registration and lifetime `AddMiddleware()` registers the class as a scoped service. One instance serves one call from Inngest, including the events its steps send, so it can take scoped dependencies and keep state for the call in fields. `AddMiddleware(instance)` registers one instance for every call instead; keep such an instance free of per-call state. `context.Services` is the call's scope for every hook. A [scoped step](https://damikun.github.io/inngest-dotnet/guides/scoped-steps.md) runs its handler in a scope of its own that middleware does not see: `WrapStepHandlerAsync` wraps the handler together with that scope, which exists only inside `next()`. Middleware runs in registration order. For wrapping hooks, the first one registered is the outermost. Middleware for one function goes in its `FunctionOptions.Middleware` instead. It runs inside the app's middleware, for the function's failure handler too. `FunctionMiddleware.Of()` resolves the class from the call's scope, and registers it as scoped unless the app registered it; `FunctionMiddleware.Of(factory)` calls a factory with the call's scope on every call. ## Observing hooks These run at points of a call and return nothing. An exception thrown in one is logged and the other middleware still run. | Hook | When it runs | | - | - | | `OnRunStartAsync(context)` | On the first call of a run only: the first attempt, with no memoized steps. | | `OnMemoizationEndAsync(context)` | Once the memoized steps have been replayed and the function reaches new code. | | `OnRunCompleteAsync(context, output)` | When the function returns. | | `OnRunErrorAsync(context, exception, isFinalAttempt)` | Every time the function throws; `isFinalAttempt` is true when no retry follows. | | `OnStepStartAsync(context, step)` | Before a step handler runs. Memoized steps do not run and raise no step hooks. | | `OnStepCompleteAsync(context, step, output)` | After a step handler succeeds. | | `OnStepErrorAsync(context, step, exception, isFinalAttempt)` | After a step handler throws. | Steps cannot be called from these hooks; the SDK throws an `InvalidOperationException` if one tries. ## Wrapping hooks These receive a `next` delegate and decide what happens around it. Exceptions they throw propagate. | Hook | What it wraps | | - | - | | `WrapFunctionAsync(context, next)` | The function body, on every call, replays included. | | `WrapStepAsync(context, step, next)` | How a step's result reaches the function, memoized or freshly run. The result can be replaced. | | `WrapStepHandlerAsync(context, step, next)` | Only the actual execution of a step handler, once per attempt. | | `WrapSendEventAsync(context, events, next, cancellationToken)` | Every event send, from `IInngestClient` or a step. Pass changed events to `next` to change what is sent. `context` is `null` outside a run. | `StepInfo` describes the step: `Id` as the code wrote it, with the `:n` suffix of a repeated id; `HashedId`, which is what Inngest stores; `Type`; `Memoized`; and `Name`, the display name the code gave the step, if any. ### When next never returns A call that reaches a new step ends there: the SDK reports the step to Inngest and abandons the rest of the function. In such a call, the `next()` of `WrapFunctionAsync` never completes, and neither does the `next()` of `WrapStepAsync` for the new step. Code after `await next()`, and `finally` blocks around it, do not run in that call. Put cleanup that must happen in `WrapStepHandlerAsync`, whose `next()` always completes, or in the observing hooks. Middleware can call step tools from `WrapFunctionAsync` and `WrapStepAsync`. A step created inside a middleware's `WrapStepAsync` is not wrapped by that same middleware again. Memoized results go back to the function in the order the steps finished, which is how a race replays the way it first went. A `WrapStepAsync` that awaits real I/O for a memoized step, such as a call to a key vault, lets the next result overtake it, and then a `Task.WhenAny` can pick a different winner than the first time. Keep that work out of the memoized path, or rewrite payloads in a payload transform, which runs synchronously. ## Examples ### Map errors to non-retriable ones A `404` from an API will not go away on retry. Turn it into a failure that stops at once: ```csharp title="NotFoundIsFinalMiddleware.cs" using System.Net; using Inngest; public sealed class NotFoundIsFinalMiddleware : InngestMiddleware { public override async Task WrapStepHandlerAsync(FunctionContext context, StepInfo step, Func> next) { try { return await next(); } catch (HttpRequestException exception) when (exception.StatusCode == HttpStatusCode.NotFound) { throw new NonRetriableException($"Step {step.Id} got a 404.", exception); } } } ``` ### Stamp every event `InngestEvent` is a record, so a middleware can change events with `with` before they are sent: ```csharp title="EventVersionMiddleware.cs" using Inngest; public sealed class EventVersionMiddleware : InngestMiddleware { public override Task WrapSendEventAsync( FunctionContext? context, IReadOnlyList events, Func, Task> next, CancellationToken cancellationToken) => next([.. events.Select(e => e.Version is null ? e with { Version = "2026-10" } : e)]); } ``` Attaching the signed-in user to every event is a change of this kind. [`Inngest.UserContext`](https://damikun.github.io/inngest-dotnet/guides/user-context.md) does it with the claims protected, and restores them in the run. ### Count failures Observing hooks suit metrics. This one counts final failures per function with `System.Diagnostics.Metrics`; the instruments are static because the middleware is created for every call: ```csharp title="FailureMetricsMiddleware.cs" using System.Diagnostics.Metrics; using Inngest; public sealed class FailureMetricsMiddleware : InngestMiddleware { private static readonly Meter Meter = new("Shop.Inngest"); private static readonly Counter Failures = Meter.CreateCounter("inngest.run.failures"); public override ValueTask OnRunErrorAsync(FunctionContext context, Exception exception, bool isFinalAttempt) { if (isFinalAttempt) { Failures.Add(1, new KeyValuePair("function", context.FunctionId)); } return ValueTask.CompletedTask; } } ``` The [tracing example](https://damikun.github.io/inngest-dotnet/examples/tracing-middleware.md) wraps steps in OpenTelemetry activities. ## Encryption The `Inngest.Encryption` package encrypts event data, step results and function output in your app, so that Inngest stores and shows ciphertext for them. It writes the format of [`@inngest/middleware-encryption`](https://www.npmjs.com/package/@inngest/middleware-encryption), so .NET and TypeScript services can read each other's events, step results and output. ```bash dotnet add package Inngest.Encryption ``` ```csharp title="Program.cs" builder.Services .AddInngest(options => options.AppId = "shop") .AddEncryption(options => options.Key = builder.Configuration["Inngest:EncryptionKey"]); ``` The middleware covers every function of the app. Functions keep working with plain values: they receive events and memoized results decrypted, and what they return is encrypted on its way to Inngest. ### What is encrypted | Data | What is encrypted | | - | - | | The result of a step | All of it, unless it is `null`. | | The output of the function | All of it, unless it is `null`. | | The data of an event the app sends, from `IInngestClient` or a step | One top-level field of it: `encrypted`, or the field that `EventEncryptionField` names. | The field is matched against the name the serializer writes, so with the default camelCase options the property `Encrypted` is the field `encrypted`: ```csharp title="CardPayment.cs" using Inngest; public sealed record CardDetails(string Number, string Holder); public sealed record CardPayment(string OrderId, CardDetails Encrypted) : IInngestEventData { public static string EventName => "shop/card.paid"; } ``` ```csharp await inngest.SendAsync(new CardPayment("o-7", new CardDetails("4242 4242 4242 4242", "Ada Lovelace"))); ``` Inngest stores `OrderId` as it is and `Encrypted` as ciphertext. A function triggered by the event gets `context.Event.Data.Encrypted` as a `CardDetails`. This also works across languages: when an event arrives, every top-level field of its data that holds an encrypted value is decrypted, whatever it is called, and the same goes for the event a `WaitForEventAsync` matched. `context.RawEvent` stays as Inngest sent it. Everything else is sent as it is: event names, ids and timestamps, the other fields of the data, `User`, step ids and names, and the data passed to `InvokeAsync`. Inngest also stores the messages of exceptions as they are, so keep secrets out of them. It evaluates expressions, such as trigger conditions, concurrency keys and the `If` of a wait, against event data, so they can only use fields that are not encrypted. ### Keys A key is any text. The middleware hashes it with BLAKE2b to the 32 bytes the cipher needs, so its length does not matter but its randomness does: use a long random value, such as the output of `openssl rand -base64 32`, kept in a secret store, and not a password. White space around the key is ignored. Every service that reads the data needs the same key. ### Rotating keys `Key` always encrypts. `FallbackDecryptionKeys` only decrypt: they are tried when `Key` does not decrypt a value, and blank entries are ignored. To replace a key without failing the runs that hold data written with the old one: 1. Deploy every service with the new key as a fallback: `Key` is the current key and `FallbackDecryptionKeys` the new one. 2. Deploy again with the new key first, and the current key as the fallback. 3. Once no run holds data written with the current key, remove it. ```csharp title="Program.cs, step 2" builder.Services .AddInngest(options => options.AppId = "shop") .AddEncryption(options => { options.Key = newKey; options.FallbackDecryptionKeys = [currentKey]; }); ``` A value that no key decrypts throws a `CryptographicException` where it is read, which fails the call, and Inngest retries it. During a rolling deploy the retry can reach an instance that has the key. `DecryptOnly = true` makes a service read encrypted data and write none. Use it to add the middleware to every service before encryption is switched on in any of them, so that none receives data it cannot read, and to phase encryption out again. ### TypeScript services An encrypted value is the object `{"__ENCRYPTED__":true,"__STRATEGY__":"inngest/libsodium","data":"..."}`. `data` is the standard base64 of a random 24-byte nonce followed by what libsodium's `crypto_secretbox_easy` (XSalsa20-Poly1305) makes of the JSON text of the value, with the BLAKE2b hash of the key as its key. The package computes this with the managed primitives of BouncyCastle, which is its only dependency besides `Inngest`. The two sides need the same key, and the same event field for the events they send each other. In TypeScript the middleware is added to the client: ```ts import { Inngest } from "inngest"; import { encryptionMiddleware } from "@inngest/middleware-encryption"; const inngest = new Inngest({ id: "shop", middleware: [encryptionMiddleware({ key: process.env.ENCRYPTION_KEY! })], }); ``` Where the two differ: - The AES values, those of the first versions of the TypeScript package, which have no strategy, and those of its `AESEncryptionService`, are read and never written. They have no authentication, so a value counts as genuine when it decrypts to JSON. There is no counterpart of `legacyV0Service`. - The data of a step that invokes another function is not encrypted in .NET, because the payload hooks do not cover it. A .NET function invoked from TypeScript still reads the data decrypted. - The TypeScript package returns the event that `waitForEvent` matched with its field still encrypted. .NET decrypts it. ### Another cipher `EncryptionOptions.Service` replaces the default strategy for writing values. A service gets the UTF-8 JSON of a value and returns the text for the `data` field of the envelope, and the other way round. Its `Identifier` is written to `__STRATEGY__`, and picks the service that reads a value, so other services need the same identifier and algorithm. ```csharp title="AesGcmEncryptionService.cs" using System.Security.Cryptography; using Inngest; public sealed class AesGcmEncryptionService(byte[] key) : EncryptionService { private const int NonceSize = 12; private const int TagSize = 16; public override string Identifier => "shop/aes-gcm"; public override string Encrypt(ReadOnlySpan json) { var data = new byte[NonceSize + TagSize + json.Length]; RandomNumberGenerator.Fill(data.AsSpan(0, NonceSize)); using var aes = new AesGcm(key, TagSize); aes.Encrypt(data.AsSpan(0, NonceSize), json, data.AsSpan(NonceSize + TagSize), data.AsSpan(NonceSize, TagSize)); return Convert.ToBase64String(data); } public override byte[] Decrypt(string data) { var bytes = Convert.FromBase64String(data); var json = new byte[bytes.Length - NonceSize - TagSize]; using var aes = new AesGcm(key, TagSize); aes.Decrypt(bytes.AsSpan(0, NonceSize), bytes.AsSpan(NonceSize + TagSize), bytes.AsSpan(NonceSize, TagSize), json); return json; } } ``` ```csharp title="Program.cs" builder.Services .AddInngest(options => options.AppId = "shop") .AddEncryption(options => options.Service = new AesGcmEncryptionService(key)); ``` One instance serves every call at the same time, so it must be thread-safe, and it runs synchronously on the call's path: load keys at startup rather than from a vault on each value. `Key` is not needed with a service. When it is set, values that the default strategies wrote are still decrypted, so data can move from one to the other. ### Notes - Events pass through middleware in registration order, so register encryption after middleware that add or change events, or it does not see them. - A middleware that rewrites payloads with `IPayloadTransform` sees plain data if it is registered after encryption and ciphertext if it is registered before it. - A value grows by a third and by about 120 bytes of envelope, which counts against the size limits Inngest puts on events and step results. - The input of an AI wrap step and the request of an inference step are stored in the clear: the payload hooks cover what a step returns, not what it was made with. That includes the messages and options, and the arguments of tool calls, that the [chat client steps](https://damikun.github.io/inngest-dotnet/guides/ai.md#what-the-input-costs) record, which `InngestStepsOptions.RecordInput` turns off. The results of those steps are encrypted like any other. - It works under Native AOT. Event data types go in the serializer context as for any event. - The test host reports what is stored, so `Output` and `SentEvents` show encrypted values when encryption is added to the services it runs. Tests of functions usually leave it out. ## Sentry The [`Inngest.Sentry`](https://www.nuget.org/packages/Inngest.Sentry/) package reports the errors of functions and steps to Sentry. An exception that a function throws never leaves the request: the SDK turns it into the answer to Inngest, so Sentry's ASP.NET Core middleware never sees it. This middleware reports it, with the app, the function, the run and the event as tags. ```bash dotnet add package Inngest.Sentry ``` Set Sentry up as usual, and add the middleware before the others, so that their work falls inside its scope. It reports through the `IHub` the app registered, which `UseSentry` does, and otherwise through the static hub. While Sentry is not enabled it does nothing. ```csharp title="Program.cs" builder.WebHost.UseSentry(options => options.Dsn = "https://key@o0.ingest.sentry.io/0"); builder.Services .AddInngest(options => options.AppId = "shop") .AddSentryReporting(); ``` ### What it reports The middleware reports the exception of a function when the function throws, and the exception of a step when its handler throws and `CaptureStepErrors` is on. By default it reports only the final attempt, after which Inngest does not retry: the retries are used up, or the error is a `NonRetriableException`. Each step the call executed leaves a breadcrumb for the events that follow, in the category `inngest.step`, with the level `Error` for a step that failed. An event it reports has the transaction name `inngest:` and these tags: | Tag | Value | | - | - | | `inngest.client.id` | The app id. | | `inngest.function.id` | The function id with the app id in front, as Inngest names it: `shop-checkout`. | | `inngest.run.id` | The run id. | | `inngest.attempt` | The attempt, counted from `0`. | | `inngest.event.name`, `inngest.event.id` | The triggering event, or the first event of a batch. | | `inngest.error.source` | `run` for the error of a function, `step` for the error of a step. | | `inngest.step.name` | For the error of a step, its display name, or its id when it has none. | A step that fails for good and reaches a function that does not catch it is reported by the function. With `CaptureStepErrors` on, the step reports it as well, so it appears twice. ### Scope and trace Sentry's scope follows the async flow. The middleware pushes one around the function and around every step handler, with the tags of the call, so that what your code reports there carries them too: a `SentrySdk.CaptureException` in a step, or an error logged through Sentry's logging integration. Sentry.AspNetCore locks the scope of a request, so a push there creates nothing, and the tags go on the request's scope, which only this call uses. In Sentry's global mode, which is meant for desktop and mobile apps, the whole app has one scope and a push creates nothing either, so the tags go on it, and calls that run at the same time mix them. All calls of a run report to one trace. A run id is a ULID, which holds 128 bits, as a trace id does, and the middleware makes it the trace id the way the TypeScript middleware does. The events it reports are in that trace, even when the calls reach different instances of the app, and not in the trace of the request that carries the call, which is new for every call. What your own code reports during the call stays in the request's trace while Sentry has a transaction for the request, as Sentry.AspNetCore starts one. ### Options | Option | Default | Effect | | - | - | - | | `OnlyCaptureFinalAttempt` | `true` | Reports an error only on the final attempt. When `false`, every attempt is reported. | | `CaptureStepErrors` | `false` | Reports the error of a step as an event of its own, besides the error of the function. | | `DisableAutomaticFlush` | `false` | Leaves flushing to Sentry. | | `FlushTimeout` | 2 seconds | How long the flush waits for events to be sent, which is Sentry's own default. | ```csharp title="Program.cs" builder.Services .AddInngest(options => options.AppId = "shop") .AddSentryReporting(options => { options.OnlyCaptureFinalAttempt = false; options.CaptureStepErrors = true; }); ``` Unless `DisableAutomaticFlush` is set, the middleware flushes Sentry when the call's service scope ends, so that events are sent before a runtime that stops once a call has answered, as serverless ones may, goes away. ### Sentry's logging integration `UseSentry` also turns on Sentry's logging integration, which reports log entries of the level `Error` and above as events. When a function or a step fails for good, the SDK writes such an entry with the exception: event id 107 for a function and 109 for a step. It comes before the middleware's report, and Sentry keeps one event per exception, so the event is the log entry's. In an ASP.NET Core app it has the tags of the call, but not the error source or the transaction name, and the options above do not apply to it. To have the middleware alone report these errors, filter the entries out: ```csharp title="Program.cs" using Sentry.Extensions.Logging; builder.WebHost.UseSentry(options => options.AddLogEntryFilter((category, level, eventId, exception) => category.StartsWith("Inngest.", StringComparison.Ordinal) && eventId.Id is 107 or 109)); ``` ### Compared with @inngest/middleware-sentry - The TypeScript middleware also starts a Sentry span for each request and each step. This one reports errors only. For spans, use [`Inngest.OpenTelemetry`](https://damikun.github.io/inngest-dotnet/reference/api.md#opentelemetry) with Sentry's OpenTelemetry integration. - There is no `ctx.sentry`. Use `SentrySdk`, or take an `IHub` from the container. - There is no `inngest.function.name` tag, because a function's context holds its id only. - `FlushTimeout` is new: `FlushAsync` needs a timeout. ## Compared with the TypeScript SDK The hooks follow inngest-js v4's middleware, with .NET names. The TypeScript SDK's transform hooks (`transformFunctionInput`, `transformStepInput`) have a .NET counterpart in `IPayloadTransform`: a middleware that implements it rewrites the raw JSON of events, step results, step output and the function's output, which is what [encryption](https://damikun.github.io/inngest-dotnet/guides/middleware.md#encryption) needs. There is no `wrapRequest`: ASP.NET Core middleware or an endpoint filter on `MapInngest` wraps a request, and constructor injection covers most other uses. Inngest's [middleware guide](https://www.inngest.com/docs/durable-execution/guides-and-advanced/middleware) describes the lifecycle in general.