Questions on the new deployment tracking feature

I would love to be able to use deployment tracking to track the scope of our crashes. However, this feature seems mainly geared to web deployments, which may happen frequently and where users are almost always guaranteed to be running the latest code. I'm trying to determine (before adding to our shipping tools) how tracking will work with our desktop product, which we ship every 6 weeks and where users don't always upgrade immediately.

Scenario: Users are currently running v1. We ship v2 and publish a deployment to Raygun.

1. If a user running v1 crashes with a new issue after the deployment is published, will that issue be marked as newly occurring in the deployment tracking for v2?
2. If a user running v1 crashes with a resolved issue after the deployment is published, will that be marked as a regression?
3. If a user running v1 crashes with an existing issue after the deployment is published, will that be marked as "still occurring" in v2?

Related, how do the existing version tags (used in filter-by-version) incorporate with the versions that are published to deployment tracking? It's hard to imagine how the feature could work for our needs if there is no relation, since the versions that get logged in crash reports are the key to determining the scope of a crash.

As a side note, I'm curious why the extra publish step is necessary to expose this data. We currently utilize the version tag and existing filter-by-version to answer some of these new error/regression questions though it is a bit cumbersome. Is it not possible to build some of this functionality off of existing versions that get logged automatically?

Thanks!

---

Hey Matt,

You're right - the first release of Deployment Tracking is geared towards "single version in production" type scenarios. We like to get features out so that people can take them for a spin so we don't spend too long building something no-one wants.

There's been a bit of discussion internally about how deployments should interact with the version information we collect from crashes. Our current plan is to merge the two, so that you can filter by versions that only exist as deployments, or see versions that you never pushed as a deployment and fill in the deployment information to make use of deployment tracking for that version.

There are a few drawbacks to just relying on the version in the crash reports - for one, we never find out about versions of your code that never crash! Also if you aren't using Semantic Versioning (some people use git hashes for instance) then we can't order your versions without extra information, which makes it very hard to determine if a new error should be considered a regression or not.

All of that is a bit trickier than the easy case (web deployment scenarios), which is why we put that out first to get feedback. I imagine you'd like the answers to your questions to be No, No, and No, and that's what we will be working towards in the next release of deployment tracking. Unfortunately for now, the answer is Yes, Yes, and Yes. However, I'd suggest getting it set up anyway as once we've got your deployment information in the system it will mean less backfilling of version information once we do get it working for your use case. Your deployments should use the same version number as would be reported in your crashes.

The extra step is there primarily because we only get one piece of information from crashes - a version string. What we really need for this feature to be useful is version, release date, the person who owned the deployment, notes, and a source control system identifier. With this information we can show more information about what changed in the version which the crash occurred - what were the release notes, who made changes, what files were changed, and in the future correlate changes to a file with stack trace information.

---

Jamie,

Thank you for the prompt and detailed response! I'm glad to hear that you're working towards that behavior for v2.

Re: "versions of your code that never crash" - If only! ;)

Re: "correlate changes to a file with stack trace information" - This would be a huge productivity enhancement, very exciting.

---