Will Maier
CISO
Even Responsible Finance, Inc
You've spent lots of time building separate environments to make testing safe and easy. Maybe you've even Done The Full Kubernetes and have dedicated clusters spin up to validate changes in CI. But your CI bill is too big, bugs keep showing up in prod, and now you have devops for your devops. What do?
Well, I'm a security person and I want you to focus more on testing in prod. Here's why:
1. The building blocks you need for (safe) testing in prod also make your system more resilient to disruptions (caused by AWS or by your friendly neighborhood APT).
2. The time you don't spend building devops for your devops can go into security engineering: anomaly detection, better code primitives, vulnerability scanning and remediation, ...
3. Your tests-in-prod are much more likely to actually validate the changes you make, reducing the likelihood that a change breaks prod (and compromises confidentiality, integrity, or availability).
4. Your tests-in-prod will (should?) flow through existing SDLC controls, giving me confidence that tests are documented, auditable, and reviewed by multiple engineers.
If you already have a spiffy super CI, great! But if you're guiltily building a staging for your staging because a security person gave you grief about testing, I hope to give you some hope and confidence (and some ideas for doing all this safely).