Your Secure DevOps Questions Answered

As SANS prepares for the 2nd Annual Secure DevOps Summit, Co-Chairs Frank Kim and Eric Johnson are tackling some of the common questions they get from security professionals who want to understand how to inject security into the DevOps pipeline, leverage leading DevOps practices, and secure DevOps technologies and cloud services.

If you are considering attending the Secure DevOps Summit or have already registered, this post is a good warmup, as these questions will be covered in-depth at the Summit on October 22nd-23rd in Denver, CO.

1. What do security professionals need to do to implement Secure DevOps processes?

SecDevOps requires security and compliance teams to integrate their processes, policies, and requirements into the same workflows being used by Development & Operations. This does not happen without the Security team understanding the current development and operations engineering process, technologies, and tools.

The Continuous Integration (CI) tool, is one of the most important pieces in the DevOps process, manages the workflow, executes steps, and integrates the entire toolchain together. Common examples include Jenkins, Visual Studio Team Services (VSTS), Travis, Bamboo, TeamCity.

Leveraging the CI tool, SecDevOps teams will start to integrate important security touch points such as:

• Pre-Commit Security Hooks - Code checkers to scan for secrets (private keys, system passwords, cloud access keys, etc.) before committing code to version control.
• Static Source Code Scanning - Scanning source code files (infrastructure templates, application source code) for security vulnerabilities in the build pipeline.
• Security Unit Tests - Custom security unit tests written by the security and engineering teams to provide coverage for abuser stories and negative test cases.
• Dynamic Application Security Testing - Scanning a running application for common security vulnerabilities (OWASP Top 10) and misconfigurations and enforcing acceptance criterial using tools such as Gauntlt / BDD Security.
• Secrets Management - Automating the provisioning and storage of secrets using tools such as Vault, Conjure, AWS KMS, Azure Key Vault.
• Continuous Monitoring - Monitoring the production environment for vulnerabilities, anomalies, and in progress attacks using tools such as statsd, graphite, graphana, etc.

2. What Secure DevOps tools should I use?

The authors of the SANS Institute's SEC540 Cloud Security and DevOps Automation course put together a Cloud Security and DevSecOps Best Practices poster. Our SecDevOps Toolchain breaks the SecDevOps process down into 5 key phases with a detailed list of associated tools.

• Pre-Commit: Security controls that can take place before code is checked into version control.
• Commit: Fast, automated security checks that are invoked by continuous integration tools during the build.
• Acceptance: Automated security acceptance tests, functional tests, and out of band security scans that are invoked by continuous integration tools as artifacts are delivered to testing / staging environments.
• Production: Security smoke tests and configuration checks that are invoked by continuous integration tools as artifacts are delivered to the production environment.
• Operations: Continuous security monitoring, testing, auditing, and compliance checks running in production and providing feedback to SecDevOps teams.

3. How important is culture in an organization moving towards SecDevOps?

John Wills, Damon Edwards, and Jez Humble came up with the CALMS to understand and describe DevOps:

• Culture — People come first
• Automation — Rely on tools for efficiency + repeatability
• Lean — Apply Lean engineering practices to continuously improve
• Measurement — Use data to drive decisions and improvements
• Sharing — Share ideas, information and goals across organizational silos

There is a reason that Culture is first on the list. DevOps creates an open, diverse working environment that enables working together to solve problems, empower teams, fail fast, and continuously improve.

Here is the biggest challenge integrating the "Sec" into "DevOps": traditional security cultures are always ready to say "no", fail to share information across the organization, and do not tolerate failure.

This is also evidenced in how CISOs lead their organizations and interact with development teams and the rest of the business. There are three different types of CISOs: the dictator, the collaborator, and the partner.

By saying "no" the dictator doesn't fully consider how security introduces unnecessary friction into existing development and business processes. For SecDevOps to be successful we need to move to be more of a collaborator by understanding engineering practices, process, and tools. Even better, security leaders can become partners by understanding business goals to earn a seat at the executive table.

4. How do you measure the success of SecDevOps?

SecDevOps teams often measure success by monitoring change frequency, deployment success / failure rates, and the mean time to recover (MTTR) from a failure.

For the "Sec" component, assume that a vulnerability assessment or penetration test uncovers a critical patch missing from a system. How long does it take the feedback from the security team to enter the operations workflow, build a new gold image, run automated acceptance tests to validate the changes, and roll the gold image into production? Mean time to recover (MTTR) is a critical metric for measuring success.

Measuring the # of vulnerabilities discovered via automated testing in the pipeline versus vulnerabilities escaping to production. This shows the effectiveness of the pipeline and improvements over time.

Tracking the window of exposure, or how long vulnerabilities remain open in production, provides important metrics for management to see progress.

These metrics also help handle managerial issues as you ask above because can show the process working and helping make the organization more secure.

5. What does "Security as code" mean?

A prerequisite for automation requires all steps to be written into code and checked into version control. Development engineering teams have been writing code since the beginning. Modern operations teams are now writing "infrastructure as code" using tools like Chef, Puppet, and Ansible to create and configure cloud infrastructure, on-premise infrastructure, gold images, and network devices.

Security as code takes this approach a step further by converting manual security and compliance steps into automated, repeatable scripts that can be executed inside a CI pipeline. Security tools are quickly evolving to have APIs and command line interfaces to support "security as code" instead of manually configuring a scanner and pressing a button.

In a SecDevOps world, security tools that cannot be automated through an API or from the command line are not worth purchasing.

Summary

Many security teams jump in too quickly and disrupt the engineering workflows. It is important to tackle one step at a time and start slowly. Security teams must ensure new security steps are working properly, evaluate results, fine tune scanners, and minimize false positives before even considering disrupting the workflow.

To learn more about DevOps and Cloud Security, check out SEC540: Cloud Security and DevOps Automation! Eric Johnson

Eric is a Co-founder and Principal Security Engineer at Puma Security  and a Senior Instructor with the SANS Institute. His experience includes cloud security assessments, cloud infrastructure automation, static source code analysis, web and mobile application penetration testing, secure development lifecycle consulting, and secure code review assessments. Eric is the lead author and an instructor for SEC540: Cloud Security and DevOps Automation, a co-author and instructor for both the brand new SEC510: Multicloud Security Assessment and Defense, and the upcoming SEC584: Defending Cloud Native Infrastructure.  To learn more about Eric, read his full bio here. Frank Kim