Disappearing errors
Allen Petersen
Posted on
Jun 06 2013
It seems that most of our errors that were logged yesterday (June 4th) are missing. Yesterday they were in our dashboard and they were searchable but today they are not. There are plenty of old errors and some new ones from today but there is a gab from "10 hours ago" to "3 days ago" in our dashboard.
I had one URL from an email https://app.raygun.io/dashboard/342hb/errors/21044920
A few hours ago I would get the following message from that URL:
"This has most likely occurred because the instances associated with this error have now fallen outside of the data retention scope of your plan"
That message has gone away and I'm able to see the full error report but it is still not searchable or in the app dashboard. It is hard to have confidence in error reporting when things are disappearing.
jeremy
Raygun
Posted on
Jun 06 2013
Hi Allen,
Thanks for reporting this. As you will have noticed the data for that error should now be appearing. We had an issue on our search cluster which is used as the primary source of error data. We also have the data stored in S3 and fall back to fetching from there if the search cluster is offline however due to an error (which we have pushed a fix up for today) this was not working correctly for everyone - sorry you were impacted by this.
Ive looked at the dashboard ordering and we have picked up another issue which again we have pushed up a fix for today where the lists used on the dashboard (which are cached for performance benefits) were not being correctly expired so we have believe this should now be sorted.
One impact from the search cluster being offline for a short period is any errors which came in to the system during that time are stored in S3 and registered with in the system, so no data has been lost as such, but they have not been indexed yet for search. We will be looking in to a refresh of the search cluster to pick up the missing and this will take some time so once this is completed all your error reports should be picked up by search again.
Thanks again for letting us know about these issues :)
Jeremy