dotnet core - UseExceptionHandler and a custom ClientProvider

I'm having trouble getting my custom ClientProvider to work in my dot net core (3.1) app in conjunction with a custom exception handler. Here's some simplified code from Startup.cs to illustrate the problem: Configure method:

        app.UseRaygun();
        app.UseExceptionHandler(a => a.Run(async context =>
        {

            var feature = context.Features.Get<IExceptionHandlerPathFeature>();
            var exception = feature.Error;
            var result = env.IsDevelopment() ?
                    JsonConvert.SerializeObject(new { message = exception.Message, detail = exception.ToString() }) :
                    JsonConvert.SerializeObject(new { message = "An error has occurred" });

            context.Response.ContentType = "application/json";
            await context.Response.WriteAsync(result);
        }));

And here is the relevant part of ConfigureServices:

 services.AddRaygun(Configuration, new RaygunMiddlewareSettings()
        {
            ClientProvider = new RaygunAspNetCoreClientProvider(Environment)
        });

My custom client provider just adds a couple of tags to the details of the error logged in raygun including some custom errors my application always has set and the environment name.

Raygun seems able to log errors, even if they have been also handled by the custom exception handler, however the custom ClientProvider does not seem to be invoked in this scenario.

Any tips on the best way to achieve this? should I be explicitly sending exceptions to Raygun via SendInBackground inside my custom exception handler? I'd worry that may cause errors to get logged twice in some scenarios.