Merging and Resolving Super Slow

Josh

Posted on
May 09 2020

Merging and resolving have been super slow all day today for me. I tweeted about this earlier to @raygunio. It's still happening now. Please advise.


John-Daniel Trask

Raygun

Posted on
May 09 2020

Hi Josh,

Sorry to hear this - nothing is registering as being overly delayed right now so we'll have a team member investigate.

As a note, merge jobs can be slow if the groups are huge, but generally is managed as an offline task. They do queue, so if you have one really massive merge job, and a bunch of small ones, it may look slower than it is.

Changing the status to resolved though shouldn't be slow since that's just updating a status field on a database row, so we'll look into that too.

Usually in these situations it can just be a queue of merge jobs, and there's no action to be taken (other than continuing to think of ways to make it faster for everyone), but I'll update here on if something significant is found.

Kind regards,

John-Daniel Trask


John-Daniel Trask

Raygun

Posted on
May 09 2020

Hi Josh,

I've just tested resolving issues and it's instant on our account. You're just doing this from the main Crash Reporting view?

Kind regards,

John-Daniel Trask


John-Daniel Trask

Raygun

Posted on
May 09 2020

Hi Josh,

FInal update - the merge jobs all just completed as the team were investigating. There were more than 300 queued up, accounting for ~1.9m errors being re-assigned into new groups.

I suspect his one will fall into the 'how can we make this faster', or at least look completed while the heavy lifting grinds through our data stores.

Apologies for the inconvienance. Do let me know about the 'resolve' being slow and how you were doing that as it should be nearly instant.

Kind regards,

John-Daniel Trask


John-Daniel Trask

Raygun

Posted on
May 09 2020

Hi Josh,

Sorry - slip of the tongue, the 1.9m figure was your number over the lifetime of your account. The actual number was tiny - I'll have the team do a deeper dive during the work week to find out why only a few thousand errors being merged took so long and hasn't caused any monitor trips or notice to us on it being this noticably slow for you.

Sorry about that,

John-Daniel Trask


Josh

Posted on
May 09 2020

Thanks for looking in to this! You had me going with the 1.9m for a split second. Yeah, this was mostly just onesy and twosy merges where the id was different with a few slightly bigger ones that may be a few hundred to a few thousand items. It's never taken this long to register in the UI for me before. However, I have noticed that the merged item often times does NOT immediately update with all of the merged info. I immediately suspected that was because it was handled in a background task as you said. However, this delay has been weird. On one of the merges earlier, it updated the UI display and looked like it completed but when checking back later for another reason it looked like the merge had never really happened.


Josh

Posted on
May 09 2020

It does appear to be working normally now.


Josh

Posted on
Jun 02 2020

FYI... Things are merging very slowly again today.


Mitchell

Raygun

Posted on
Jun 02 2020

Hi Josh,

Thank you for getting in touch with us again. We have not detected anything that may be directly impacting the merging of error groups. We will continue to monitor the situation and let you know if we encounter a specific issue within our system that is causing the slow performance.

While performance improvements can be a large undertaking, we may be able to make some improvements to our grouping logic in the short term. Is there particular error groups you are commonly seeing that should be grouped together automatically? If you share with us some examples, we can see what can be done to release to a new grouping version to reduce the need for merging.

Thank you, Mitchell


Aun Ali

Posted on
Aug 28 2020

.


Reply