stackoverflow in .NET Raygun client

andrew

Posted on
Aug 28 2015

We're seeing occasional stackoverflows from the Raygun client. Attached below is a sample stacktrace from production. I also have some dump files available if needed.

We're using .NET 4.5.1 on 64 bit Win2k8 R2 servers, and we're using the 4.0.1 client.

Is this a known issue that's fixed in 5 and above? If so, we can upgrade.

Thanks!

[a ton more of these until the process exits with exception code 0xc00000fd] Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+a3 Mindscape.Raygun4Net.dll!Unknown+16e Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+a3 Mindscape.Raygun4Net.dll!Unknown+16e Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+a3 Mindscape.Raygun4Net.dll!Unknown+16e Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+a3 Mindscape.Raygun4Net.dll!Unknown+16e Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+a3 Mindscape.Raygun4Net.dll!Unknown+16e Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+a3 Mindscape.Raygun4Net.dll!Unknown+16e Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+99 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+183 Mindscape.Raygun4Net.dll!Unknown+ef Mindscape.Raygun4Net.dll!Unknown+214 Mindscape.Raygun4Net.dll!Unknown+58 Mindscape.Raygun4Net.dll!Unknown+2b9 dotMailer.Core.dll!Unknown+7c dotMailer.Core.dll!Unknown+117 dotMailer.Core.dll!Unknown+386 [[HelperMethodFrame]] dotMailer.Core.dll!Unknown+5b1 [[HelperMethodFrame]] dotMailer.Core.dll!Unknown+35a [[HelperMethodFrame]] SystemDatani!System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, Boolean, Int32, System.Threading.Tasks.Task ByRef, Boolean, System.Data.SqlClient.SqlDataReader)+66ddef [[HelperMethodFrame]] SystemDatani!System.Data.SqlClient.SqlCommand.FinishExecuteReader(System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.RunBehavior, System.String)+66d3b5 [[HelperMethodFrame]] SystemDatani!System.Data.SqlClient.SqlConnection.OnError(System.Data.SqlClient.SqlException, Boolean, System.Action1)+188 System_Data_ni!System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(System.Data.SqlClient.TdsParserStateObject, Boolean, Boolean)+32f System_Data_ni!System.Data.SqlClient.TdsParser.TryRun(System.Data.SqlClient.RunBehavior, System.Data.SqlClient.SqlCommand, System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.BulkCopySimpleResultSet, System.Data.SqlClient.TdsParserStateObject, Boolean ByRef)+11a3 System_Data_ni!System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()+3d System_Data_ni!System.Data.SqlClient.SqlDataReader.get_MetaData()+8a System_Data_ni!System.Data.SqlClient.SqlCommand.FinishExecuteReader(System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.RunBehavior, System.String)+ec System_Data_ni!System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, Boolean, Int32, System.Threading.Tasks.Task ByRef, Boolean, System.Data.SqlClient.SqlDataReader)+b6b System_Data_ni!System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Threading.Tasks.TaskCompletionSource1, Int32, System.Threading.Tasks.Task ByRef, Boolean)+24a SystemDatani!System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String)+6b SystemDatani!System.Data.SqlClient.SqlCommand.ExecuteReader(System.Data.CommandBehavior, System.String)+120 SystemDatani!System.Data.SqlClient.SqlCommand.ExecuteReader()+12e


Jason Fauchelle

Raygun

Posted on
Aug 28 2015

Hi Andrew,

Thanks for contacting us about this. There have been no known issues causing stack overflow exceptions in Raygun4Net. Unfortunately the stack trace is not too much help with tracking down how this has been caused. If you can, please find more information about this issue that may help us reproduce this or track this down. If you have any other logs that give insight to what happened before the stack overflow started, or if you can determine if this was caused by a manual or automatic send, or if you can reproduce this locally, that may help.

Also, let me know what type of application this occurred in and I'll look through the code to try determine a scenario that could cause a stack overflow.

Sorry I couldn't provide more immediate assistance.

-Jason Fauchelle


andrew

Posted on
Sep 04 2015

Hi Jason -

I've got a repro case, attached below. It fails in the latest version (5.1). It's slightly contrived but reliably throws a StackOverflowException on .NET 4.6, compiled as 64 bit. The issue is too much recursion in SimpleJson.cs. It seems on my machine, one DataRow isn't enough to trigger the bug, so I added two - you may need to add three depending on what the CLR is doing that day.

Obviously passing in a DataTable is not great practice, and we're changing our external interface to only allow primitive types to be passed in. However, the impact on us was pretty bad and took a while to hunt down with WinDbg, so it feels like some protective measures from you guys would help (simple stack depth counter perhaps?).

Thanks!

Andrew

internal static class WillCauseStackOverflow
{
    public static void Main(string[] args)
    {
        var client = new RaygunClient("API Key here");
        var message = RaygunMessageBuilder.New.Build();
        message.Details.UserCustomData = GetBadData();
        client.Send(message);

        Console.ReadKey();
    }

    private static IDictionary GetBadData()
    {
        var badData = new Dictionary<string, object>();
        DataTable dt = new DataTable();
        dt.Columns.Add("Name");
        dt.Columns.Add("Sandwich");
        DataRow row = dt.NewRow();
        row["Name"] = "Andrew";
        row["Sandwich"] = "Ham hock";
        dt.Rows.Add(row);
        badData.Add("dt", dt);

        return badData;
    }
}

Jason Fauchelle

Raygun

Posted on
Sep 09 2015

Hi Andrew,

Thanks for the repro snippet. I'll have a look at fixing this now and push a new Raygun4Net version as soon as possible.

I'll get back to you once the next version is available.

-Jason Fauchelle


Jason Fauchelle

Raygun

Posted on
Sep 09 2015

Hi Andrew,

I've just pushed a new release of Raygun4Net (v5.1.1) to NuGet which fixes the stack overflow issue. The problem was a bug in the cyclic detection code that I wrote in the SimpleJson file. Sorry for the trouble!

Note that this just solves the serialization bug, but DataTable will still be serialized as an object, rather than having first-class support to be serialized as a table of its values - so moving away from DataTable as you mentioned is still a recommended action.

Hope that helps. Let us know any time you have further questions about Raygun.

-Jason Fauchelle


OmerRaviv

Posted on
Nov 10 2015

Hi Jason,

I've also had an issue where customers complained about my software causing their application to crash, and it turns out the fault was with RayGun throwing a stackoverflow exception. I will try upgrading to v5.1.1 as you suggested, but I would like to also send you a dump file, if you could please analyze it and let me know that indeed it is the exact same issue that was fixed? Where should I send the dump to?

Also, as a RayGun customer, I really would have liked to know about this 2 months ago when it was first discovered, rather than now. Please do consider making a "please upgrade" notice, while I realize doing that is bad PR, I do feel kind of cheated right now, as I'm paying monthly for a service that's supposed to help me fix errors, only to realize it is causing crashes. Mistakes do happen to everyone, and it's perfectly understandable, but not if you do not let your customers know that a mistake's been made and they need to upgrade...

Thanks, - Omer


Jason Fauchelle

Raygun

Posted on
Nov 10 2015

Hi Omer,

Thanks for your feedback, and apologies for not sending a message about this potentially important update. If you send the dump to jason@raygun.io I can take a look. If you could also send any information about the custom data you may be trying to send, that could help too - this bug only occurs from a reasonably particular set of possible cyclic data structures. Hope to hear from you soon.

-Jason Fauchelle


Reply