Software defects are not uncommon, but what sets great companies apart is how quickly they respond and put the customer first. While that is a challenge in any domain, security software defects are a unique beast. Our products can be the first line of defense or the last, depending on the solution. The cost of failure in security is catastrophic - and I say that without exaggeration.
"For a security product, quality is more than a technical measure; it's a form of security in itself."
This is an important lesson I've learned over the past 25 years in the cybersecurity industry. For example, when McAfee.com was acquired by Network Associates, I received a welcome gift from Chris Bolin - Test-Driven Development by Kent Beck. Chris, then head of engineering, emphasized a core principle that has stayed with me:Developers must own the quality of the products they build. From him, I learned that quality isn't just a responsibility; it's an integral part of the development process.
Another key leader I worked with, Bryan Barney, was instrumental in establishing PSIRT and CSIRT processes. He often said, "No product defect will ever have the same large-scale impact as a bad update from a security vendor." He was, of course, referring to us. Yet, despite that warning, we did cause a large-scale disruption to critical infrastructure around the world when a flawed content update was released. At the time, we were deploying content updates every single day - fully automated across all versions, operating systems, and products.
Recently, when a similar incident occurred with a major vendor's security product, several colleagues - current and former security leaders - shared their own war stories of having to report major incidents to their C-suite and board. These weren't CISOs, but leaders in engineering, responsible for the very products designed to protect businesses from security threats. One thing we all could agree on - any issue a vendor encounters is a reminder to not get complacent, another wake-up call to step up our processes, methodologies of designing, building, testing and releasing code/software. This is why quality is priority zero, and we know the stakes are high if we don't get it right.
This hyper-focus on quality is partly due to the fact that security products operate with elevated privileges, granting them significant access to systems and environments. A failure in quality can introduce vulnerabilities, turning the product from a defense mechanism into an attack surface. Poorly executed security updates can cause the very breaches they are designed to prevent.
Quality impacts the customer experience. What we ought to strive for is the quality of customer experience. Usability issues, stemming from poor quality, can lead to misconfigurations or overlooked critical alerts, reducing the overall effectiveness of a security solution. (Around 80% to 85% of quality issues are due to misconfigurations, policy inconsistencies and poor implementation of software rather than flaws in the security products themselves.) This is especially true for products designed to detect and respond to incidents. If compromised by poor quality, their ability to protect customers is weakened, with potentially disastrous consequences.
Quality also goes hand-in-hand with operational resilience, which is a primary goal for many customers investing in security solutions. But when a security product fails, it does the opposite - disrupting the very operations it was meant to safeguard. In this way, a widespread failure in a security product can, in some cases, cause even more damage than a targeted ransomware attack, which usually affects specific targets.
The consequences of a security failure are not only about service disruptions but also about real-world harm, particularly in industries where downtime can put lives at risk. Think about hospitals, immigration services, utilities like electricity and water - any failure in these sectors can have immediate and severe repercussions. For example, a hospital unable to access patient records due to a security product malfunction could delay critical treatments.
Similarly, in banking, government services and large corporations, the financial and reputational damage of a security product failure can be profound. These sectors rely on security solutions to maintain operational integrity, and a single defect can lead to financial losses, reputational damage and long-term erosion of customer trust. In many industries, compliance with strict regulatory standards is also at stake. A failure in quality can result in non-compliance, leading to penalties, scrutiny, or even exclusion from certain markets.
"The potential for service disruptions, economic impacts, real-world harm and regulatory risks makes it essential to prioritize quality. Quality, then, is not just a technical attribute; it is a fundamental component of security."
The consequences of a quality failure can be more far-reaching than even a malicious attack, highlighting the need for stringent standards and security practices when developing critical security products.
This became even more clear to me during the COVID-19 pandemic when I received a panicked call from an account executive. A drug manufacturing facility's production line had come to a halt due to a defect in my product. The stakes couldn't have been higher. A swift response, however, gets your customers to become loyal customers.
When we can build solutions to defend critical infrastructure against nation-state attacks, we must be equally committed to ensuring the quality and security of our own products and processes. I believe AI will be a key player in helping us meet this challenge.
With the advancements in AI, the five key steps outlined above can now be implemented more efficiently and effectively than ever before.
AI can help automate and streamline these processes, allowing teams to quickly identify and address issues, improve product quality and maintain customer trust. Our teams have rolled up their sleeves and jumped in, leveraging AI to accelerate unit testing, automate compliance steps, review logs to check for anomalies proactively, improve the risk assessment framework to consistently assess risk of builds and automate detection of vulnerabilities.
"By leveraging AI, we can enhance the overall security and resilience of our products, making these critical steps easier to achieve in today's fast-moving cybersecurity environment."
Looking ahead, I cannot envision any scenario where the pursuit of unwavering quality is detached from building great security products. Effective and reliable security solutions are the foundation of digital trust, especially in a world where threats evolve and morph at the speed of AI. This means every security vendor and the industry as a whole must commit to rigorous testing, CI/CD principles and transparent communications with our customers, even - or maybe especially - when the news is really dire. For Cisco Security Engineering, these commitments are not aspirational; they are priority zero.
We'd love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!