10 Common Reasons Production Breaks During a Release
Why does it work in staging but not production?
As an online application, Testomato users count on us to consistently release new features to improve our service. But with every release, or new deploy, there’s always a risk that something will break in production.
It happens to every developer and every team at some point. In fact, as we’ve mentioned before, Testomato was built because there are times production breaks despite your best efforts.
We wanted to share some of the more common reasons a build works in staging, but fails in production to help you understand what often causes problems in the first place.
Here’s 10 common reasons production breaks during a release:
1) Your version in staging is different from the version in production.
This is a typical scenario, and it’s a supposed to work this way. Sometimes, you forget to update changes you made in staging. You do want to test out new versions in staging first, but don’t forget to then update the version in production.
2) Your production environment is missing a piece of code or template.
Ever had something work perfectly in staging, but then a template or piece of code disappears in production? This could happen if you deploy a script in your public folders, but have forgotten to copy a non-public folder.
3) You forgot you made changes to the structure and data in your database.
Typically, this occurs when someone forgets to update SQL after they’ve added new data or a column in staging.
4) A feature wasn’t approved and turned on in production.
This situation can occur when a feature is turned on in staging but not in production, which can throw your whole deployment workflow out of sync. Of course, it’s normal to have features turned off in production, but don’t forget to turn them on!
5) A deployment was released too soon.
If you deploy too early, a feature can end up causing a performance problem.
6) You forgot to upload a new software version to your server.
To avoid this, we recommend using an automatic tool to help you, such as envtesting. You can also check out Nagios, but you’ll need to write your own tests.
7) Your app can’t access storage.
If your app can’t access storage in production (e.g. memcache or another service), this could break production. Normally, if an app can’t access storage it’s because your settings are not set to allow your app to connect to anything or you don’t have the correct login information.
Always make sure to double check this before a release.
8) Browser is using cached images, Javascript, and CSS.
If doesn’t work in production, make sure your browser isn’t using cached images, Javascript, and CSS. You can force your browser to reload with fresh data using F5 in most browsers and operating systems. If you’re using a Mac/Apple pressed Command + R.
9) Domain names or DNS records haven’t been updated.
Changes to these settings are usually not as common, which means they’re easier to forget about.
10) The release didn’t deploy fully and breaks the source code.
We’ll let you in on a secret – this is the most common cause for suspicious behavior on Testomato because it’s located on three different servers. Recently, we had a new update deploy on two of the servers, but didn’t fully deploy to the third.
Tips from Team Testomato for Better Deployments
Get everyone on the same page.
Deploy major releases to production at a scheduled time that’s shared with your entire team. Try to do it at a low activity time and when someone is there to verify the changes work in your live version.
Avoid automatic deployments.
There are a few shops like Etsy that use continuous deployment to ship code straight to production, but as much as we love automation, we recommend all deployments be performed by a real person to cut down on mistakes.
Let everyone know about releases.
An email is sent to the whole Testomato team (not just the developers) after every deployment with an update about all the changes that were made. This keeps everyone informed about what’s gone live and also makes it easier for us to explain updates to our users.
Create a deployment workflow.
Similar to your development workflow, deployments require some thought. For example, each of our developers have their own local setup. Features are merged into the staging branch and automatically deployed to our Staging environment. Once changes are implemented and tested thoroughly, we deploy manually to production.
Why else do features break between Staging and Production? Have other tips for better deployments?
Share with us in the comments below or on Facebook. You can always tweet us directly @testomatocom.
