In a recent article on the GovDevSecOpsHub, we shared three key benefits that shifting testing to the left delivers to the application and software development process. Those three benefits came courtesy of a recent Cloudbees blog post from Dave Farinelli, a senior software engineer with almost a decade of experience providing enterprise-level software solutions to the healthcare and insurance marketplace.
Unfortunately, while testing during production can expedite the development process, allow more frequent deployments, improve testing accuracy and allow for more seamless transitions between testing phases, embracing this shift in application development can create some challenges for application development teams. It also comes with some prerequisites.
Here are two of those challenges, courtesy of Dave:
Security Risks
When dealing with testing in production, security can be one of the most challenging things to address. We’re no longer only working with dummy data but real live data. This reality increases the severity of handling data appropriately, meaning we must be careful when testing using existing customer data.
Depending on your application, the severity of data protection changes. Some software requires HIPAA compliance and comes with extreme costs for violations. Others, especially in the financial industry, contain plenty of personally identifiable information (PII). Data leakage can result in major lawsuits and much worse.
A consideration here is to limit the amount of tools that have access to this data. The more places this PII is stored, the worse. A tool such as CloudBees Feature Flags provides this separation out of the box by not requiring this data to be stored for use.
Required Maturity of Deployment Capabilities
Testing in production requires a fairly mature deployment process to already be in place.
First, you’ll need to have the means to make deployments quickly—meaning moving away from heavily manual deployments, as these are riddled with risk due to inconsistency. Next, you’ll need to make more frequent deployments to allow for safely introducing feature flags into your application.
You need to have the means of using feature flags dynamically—so not only turning features on and off, but also allowing for changing functionality based on the user. Finally, these same flags need to be in place in case a feature flag needs to be turned off gracefully.
Achieving granularity with the feature flags being used across the board makes having a positive user experience in managing feature flags even more critical. A few points of importance in this area include having a good user interface for toggling specific feature flags and having the means for control across an enterprise. The latter becomes especially important in a large company as feature flag deployment becomes more widespread.
There are a few things that can occur if you are attempting to test in production without the appropriate deployment capabilities:
– Bad data can populate your production databases and, without a rollback plan, can require manual intervention to fix data. This issue poses risks from a security and functionality perspective.
– There might be unintended consequences of malfunctions or even unintended downtime for external applications. Fixing these side effects requires either using feature flags to change functionality based on request or load balancing multiple systems with different functionality, akin to canary releasing.
This might seem intimidating at first glance, and it might be even harder to get buy-in for a group behind the curve on modern deployment strategies. The difficulty comes in not only implementing these features, but also the buy-in of the culture of continuous delivery.
