Fatal error UnwrappingExceptions during heavy threading
chris.maffin
Posted on
Oct 09 2014
One of our applications just crashed while trying to send an error to Raygun with the following stack trace:
System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
at Mindscape.Raygun4Net.RaygunClientBase.StripWrapperExceptions(Exception exception) in e:\\Users\\Jason\\Documents\\GitHub\\raygun4net\\Mindscape.Raygun4Net45\\RaygunClientBase.cs:line 304
at Mindscape.Raygun4Net.RaygunClientBase.BuildMessage(Exception exception, IList`1 tags, IDictionary userCustomData) in e:\\Users\\Jason\\Documents\\GitHub\\raygun4net\\Mindscape.Raygun4Net45\\RaygunClientBase.cs:line 287
at Mindscape.Raygun4Net.RaygunClient.<>c__DisplayClassf.<SendInBackground>b__c(Object c) in e:\\Users\\Jason\\Documents\\GitHub\\raygun4net\\Mindscape.Raygun4Net45\\RaygunClient.cs:line 40
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
This particular application consumes multiple messages from a message queue and in this case was handling a big batch of hundreds of messages which all generated errors. It seems like b/c the service uses multiple threads to handle multiple messages concurrently that the large number of errors across a large number of threads exposed a threading issue in the SendInBackground functionality of the Raygun Client.
Jason Fauchelle
Raygun
Posted on
Oct 09 2014
Hi Chris,
Thanks for contacting us about this issue. We are looking into this and will update you once we have a solution.
-Jason Fauchelle
chris.maffin
Posted on
Oct 10 2014
Thanks Jason, I did a little bit of digging and it appears that the issue stems from the fact that the list of WrapperExceptions is static, but it is also new-ed up every time the constructor is called, which means that if an instance of the client is iterating through the WrapperExceptions list at the same time another instance of the client is being created in another thread, the first instance will be iterating through the collection when the second replaces it with a new one, leading to the error.
We've been able to get around this issue by creating a single client and using it across threads, but this leads to some confusion about how the client is meant to work in a multi-threaded environment. The fact that the WrapperExceptions collection is static seems to indicate that the client is mean to be used as a singleton, but the fact that the affected UserInfo is set on the client would suggest that one is meant to create a new client for every error. Needless to say, this dichotomy has resulted in some serious issues with using Raygun in a multi-threaded environment, and we would appreciate some guidance on how the Raygun team envisioned it being used.
Jason Fauchelle
Raygun
Posted on
Oct 10 2014
Hi Chris,
Thanks for this feedback. To solve this issue, we are going to change the wrapper exception list to be non-static. Doing this would mean that you can continue creating a new RaygunClient instance for each error or thread - which is the best usage for your scenario.
Once we've tested this and pushed this change into the next version, I'll let you know when it's available.
-Jason Fauchelle
Jason Fauchelle
Raygun
Posted on
Oct 24 2014
Hi Chris,
You may have noticed that we've recently released part of Raygun4Net version 4: https://raygun.io/blog/2014/10/heads-raygun4net-4-0-coming-soon/
The Mvc and WebApi support has been moved into their own NuGet packages. So if you are using Raygun in an MVC or/and WebApi project, you'll want to switch to using the appropriate NuGet packages now.
The reason I mention this is because these NuGet packages have the change mentioned in this forum thread - the wrapper exception type list is no longer static. If you are not using Raygun in an MVC or WebApi project, you'll want to continue using the standard Raygun4Net package which will have the non-static wrapper list in V4 releasing next Monday.
-Jason Fauchelle
Jason Fauchelle
Raygun
Posted on
Oct 28 2014
Hi Chris,
As a follow up to my previous message, version 4 of the standard Raygun4Net package has now been released which includes the change to make the wrapper exception list non-static. This means all Raygun4Net NuGet packages are now up to date and have non-static wrapper exception lists.
Please let me know if you have further questions about this.
-Jason Fauchelle