I Don’t Need to Test My Configurations. My Deployments are Automated

This is a pretty common response we get from people we’re explaining our product to. There is logic to it but we don’t believe it’s necessarily reasonable. To illustrate our viewpoint on this we thought we’d paraphrase a conversation we had with a prospective client recently.

“Configuration testing. I get it and I see the value but I simply don’t need it. Our deployments are fully automated with Chef/Puppet.”

“That’s great. You’ve got consistency. You’ve got repeatability. That’s a huge step. At the end of the day though your deployments are still code, right?”

“Yes. Code that we use all the time. We see the output. We know if stuff is working or not.”

“OK. But wouldn’t you agree that’s exactly the same argument that software developers had 10 years ago when arguing against the need for full test suites of their products? It can be unit tested. It can appear to be working but without an automated test suite how can you guarantee it will remain that way.”

“I suppose”.

“With configurations the risk of drift is also higher. It’s generally hard for a dev to get a change into Production. Sysadmins have full access though. How sure are you that every single change made to your Production configurations is reflected in your recipes or manifests?”

“I’m fairly confident. I know if my systems are working.”

“Working, maybe. What about the stuff you can’t easily determine though? Are they secure? Are they compliant? Has someone opened a port they shouldn’t have to fix a critical incident? Have they applied an unapproved patch to save themselves time? Is it possible that some of this stuff is falling between the cracks.”

“It’s possible, sure.”

“We believe very strongly in testing¬†independent¬†of deployment automations. By separating them out you can also remove the responsibility of things like security and compliance from your ops guys. Let them focus on the build. Let your security team provide security tests. Let your compliance team provide compliance tests.”

“Makes sense. We have monitoring as well don’t forget.”

“Yes, but your monitoring system is set up to detect symptoms. Not diagnose causes. A monitoring system won’t troubleshoot a configuration error. A configuration test script will.”

“OK, you’re winning me over. I have to ask though. Is this conversation getting recorded? You sound kinda douchey.”

“I don’t know what you’re talking about.”