Microsoft is embracing the cloud and we’re adopting agile methodology—DevOps—for cloud app development. This transition has challenged traditional security methods. To build security into our agile development process and provide a baseline for security in cloud apps, we created the Secure DevOps Kit for Azure. It offers tools and best practices for building security into every stage of cloud app development.

Core Services Engineering (CSE, formerly Microsoft IT) created the Secure DevOps Kit for Azure to help build security best practices into enterprise cloud application development and operations. The kit contains automation, extensions, plugins, templates, modules, and other tools that seamlessly add security to cloud applications during development process. Additionally, the kit helps our engineering teams save time and money, increase security awareness in Azure, and create a simpler, more structured, and consistent security environment in the CSE Azure app infrastructure.

Adopting modern engineering with DevOps

CSE has been on a steady journey to the cloud over the last few years. In fact, we plan to have 90 percent of our IT resources hosted in the cloud as of July 2017. Continual progress in cloud technology and cloud security readiness allows us to migrate to Microsoft Azure and come closer to our cloud-first, mobile-first transformation strategy.

During this period, our engineering teams have adopted modern practices and a DevOps-centric culture, using the cloud as the default platform for IT solutions. DevOps has brought our development and operations teams together and started a grassroots movement that has led to this new, agile culture. Together, we create innovative solutions using cloud technologies with a goal to deliver continuous, rapid, and incremental value to business.

Understanding the security challenges of DevOps

The digital transformation to DevOps in unison with our move to the cloud hasn’t been without challenges for enterprise security. DevOps in the cloud changes the IT ecosystem in ways that significantly affect security. We questioned the future relevance of how we had traditionally developed and managed IT security and risk management, and it became clear to us early on that enterprise security also needed to transform for a smooth and complete transition. There were several challenges to consider, including:

• Engineering teams have increased autonomy. In the past, engineering teams waited weeks or months for development resources. Now that IT no longer provisions development environments, we don’t have a significant impact on scheduling or capital expense. With DevOps in the cloud, autonomy and decentralization allows engineering teams to work end to end with almost complete independence from IT. Engineering teams can instantly provision test environments, and solutions can be deployed and published with an Azure subscription at whatever pace suits the team and business stakeholders. Traditional security methods hinder this agility.
• More development technologies are available. Developing in the cloud opens up a huge opportunity for connecting different platforms and frameworks, but as flexibility has increased, so has the number of APIs and services used to make those connections. The cloud app development environment is more complex, and maintaining security in that environment using traditional methods is also more complex—and sometimes isn’t possible.
• Constant change is the norm. With the shift to agile sprints and DevOps, constant change is the norm. The platform components on which applications run keeps changing, improving, and growing—often at a cadence dictated by individual Azure service teams. On top of that, dedicated business unit application teams regularly add new functionality and improve existing functionality following the agile philosophy of incremental but continuous improvement. Traditional security and the associated tollgate procedures aren’t designed for such continuous change.
• DevOps has wide-ranging operational responsibilities. In the DevOps era, there isn’t a hard boundary between development and operations. The engineer who developed a feature is also responsible for the operational aspects of the feature. Operational considerations, including security, are a high priority for the development team in a DevOps culture.

Addressing DevOps security challenges

Faced with these DevOps security challenges, we set out to determine how security could be managed in a DevOps ecosystem. We wanted to change our thinking, methods, and tools to adapt to a development environment and culture that was in harmony with the nuances inherent in cloud DevOps. To do this, we adopted a number of imperatives.

Automate security

Automation gives us a chance to keep pace with the constantly changing cloud environment. DevOps is heavily centered on end-to-end automation, and we need to complement it with automated security. Automated security saves significant time and cost for apps that update much more often than their traditional counterparts, and it allows us to ensure that security configuration and deployment in DevOps can be achieved quickly and consistently.

Empower engineering teams

In an environment where change is constant, we want to empower our engineering teams to make meaningful, consistent changes without a tedious approval process. Our engineers need to be able to build security into their applications from the start. We need security integrated into the DevOps workflow. Developers don’t have to take extra measures to be secure, nor do they need to wait for a central security team to approve an app.

Maintain continuous assurance

When development and deployment are continuous, everything that goes with them needs to follow suit, including security assurance. The age-old requirements for sign-offs or compliance checks create tension in the modern engineering environment. We want to define a security state and track drift from that state to maintain a consistent level of security assurance across the entire environment. This helps ensure that builds and deployments that are secure at the time they are delivered, stay secure from one release iteration to the next and beyond.

Set up operational hygiene

We need to have a clear view of our DevOps environment to ensure that operational hygiene is in place. In addition to understanding operational risks in the cloud, DevOps operational hygiene in the cloud requires a different perspective than the traditional development environment. We need to create the ability to see the security state across DevOps stages and establish capabilities to receive security alerts and reminders for important periodic activities.

Understanding the Secure DevOps Kit for Azure

The Secure DevOps Kit for Azure is a set of automation, extensions, plugins, templates, modules, and other tools that combine to offer a security-focused development workflow for our DevOps engineering teams working in the cloud. The goal of the kit is to empower our teams to build and use Azure-based solutions in a consistent, repeatable, and efficient manner with security integrated at every stage.

Figure 1 shows how the six main tools in the DevOps toolkit work together to support secure development in the cloud.

 

The Secure DevOps Kit for Azure is designed to approach cloud development security in the following areas:

• Subscription security. This ensures that a subscription is configured and provisioned with necessary security controls.
• Secure development. This provides the ability to write secure code and spot check secure configuration of cloud resources.
• Continuous integration and continuous deployment (CI/CD) extensions. These integrate security testing into CI/CD workflows.
• Continuous assurance. This ensures that the security state stays compliant and doesn’t drift over time.
• Alerting and monitoring capabilities. These check for security events and provide an effective remediation path for subscription and application security issues.
• Telemetry dashboards. These get aggregate views of security patterns and trends to make concerted improvements.

Breaking down the Secure DevOps Kit for Azure

To help you understand the nature of the DevOps Kit for Azure, we’ve broken the toolset into six main categories. These components will help facilitate secure development in your Azure environment.

Subscription security

The subscription security component is a package of scripts and programs that help ensure secure provisioning, configuration, and administration of an Azure subscription. Using these capabilities, you can set up and configure a compliant, secure subscription from the very start and have a solid foundation upon which to develop, deploy, and run secure solutions. You can also check the subscription configuration to see if various settings are compliant to an expected level. The primary tools in subscription security include:

• Health check script. The subscription health check script runs automated steps to examine a subscription and flag conditions that indicate your subscription may be at risk due to security issues, misconfigurations, or obsolete settings.
• Provisioning script. The provisioning script is a master script, which coordinates several smaller components that work together to provision a DevOps Kit environment. These components include:
  • Mandatory role-based access control accounts for important functions.
  • High-level alerts for critical or severe security events.
  • Azure Resource Manager policies that help secure otherwise insecure actions.
  • Default enterprise policy settings for Azure Security Center.
  • Security contact information.

Secure development

The secure development components help ensure that security is integrated into the day-to-day development process. The primary components include:

• Security Verification Tests. These tests automatically verify most built-in security controls for common Azure services such as App Services, Azure Storage, Azure SQL Database, Azure Key Vault, or Azure Virtual Machines.
• Security IntelliSense. This feature augments traditional IntelliSense with secure coding best practices and offers corrections, tips, and guidelines while a developer writes code. The secure coding rules covered vary from Azure platform as a service (PaaS) APIs to traditional web application security and cryptography best practices.

Security in CI/CD

Build/Release Tasks for CI/CD workflows allow us to check subscription and resource security during automated build/deployment flows. These workflows integrate security coverage within the Visual Studio Team Services (VSTS) CI/CD pipeline via VSTS build/release extensions for security verification tests and other security tools.

Continuous assurance

Continuous assurance prevents security state drift, helps to stay current with Azure security feature improvements. It also encourages adherence to security best practices such as key rotation and separation of duties. The tools in this section include:

• Azure Automation runbooks that identify and correct security configuration drift.
• Azure Resource Manager templates used to securely deploy pre-configured Azure resources.
• A set of PowerShell scripts to create the Automation account, apply the templates, and install and configure the Runbooks.

Alerting and monitoring

The alerting and monitoring solution for the DevOps Kit uses Operations Management Suite (OMS) to offer a central dashboard where teams can view the security state and trends for their Azure subscriptions and applications, as reported by the different components of the kit. The OMS solution is created from an Azure Resource Manager template that builds all the necessary components needed for security state monitoring. The OMS views include:

• Summary views of critical tasks that need immediate attention.
• Outcomes of the most recent continuous assurance scans.
• Summary of recent role-based access control activity (important role assignments, access revocation, and others).
• Trends of various security metrics and activity over time.
• Common useful queries for alerting, and other activities.
• Pre-configured alerts in OMS.
• Runbooks for auto-healing when certain alerts are triggered.

Cloud risk governance

The Secure DevOps Kit generates telemetry events from all stages that use automation, scripts, or extensions. The telemetry is routed to an Application Insights account where it’s processed through web jobs that integrate organization mapping information and then viewed on a Power BI dashboard. The telemetry supports a data-driven approach to agile development and DevOps by allowing us to make measured and accurate security improvement decisions in a continuous fashion. Cloud risk governance focuses on three primary views:

• We can see adoption and usage of the DevOps Kit across the enterprise. These views give us a picture of the company’s secure DevOps maturity in the cloud.
• We can view aggregate cloud-related risks across service lines. Aggregation of control failures for different cloud resource types helps us understand which areas of cloud use are leading to higher risk exposure for the company due to vulnerable configuration. This information can be used to target risk reduction.
• We get visibility into common errors and challenges that developers face while using the kit. Information about errors and exceptions helps the Secure DevOps Kit team improve features and the user experience.

Using the Secure DevOps Kit in CSE

We’ve encouraged adoption of the Secure DevOps Kit within CSE for any of our business groups working in Azure. One such team is Field Mobility and Cloud Services (FMCS). The FMCS team, consisting of 200 developers, supports approximately 120 apps hosted in Azure. As part of the transition to DevOps, FMCS uses the Secure DevOps Kit to incorporate secure cloud development practices in the application life cycle. They have realized several benefits:

• Reduced development time and money. The Secure DevOps Kit puts security best practices and tools at our fingertips. It saves our developers the time and effort of researching, cataloging, and implementing Azure security practices manually, and it provides a set of consistent security practices for them to follow.
• Higher awareness of security. Because the Secure DevOps Kit builds security automation and best practices into the development process, our engineers are aware of security requirements and capabilities from the beginning of a project. Security has become an integral piece of the development process, rather than something that’s scrutinized near the end of the development cycle and might require significant re-work of solution components.
• Easier transition to DevOps. FMCS is in the midst of transitioning to DevOps, and the Secure DevOps Kit has simplified that transition. By incorporating security automation in our toolset, we know that security is built in to the entire life cycle.
• Simple processes for checking existing solutions. We’ve used the manual Service Validation and Testing (SVT) processes several times with existing projects to confirm that Azure security configuration is correct.
• Convenient assurance checks and problem resolution. The OMS dashboards in the Secure DevOps Kit enable us to view security assurance across our app portfolio and see where attention is needed. The alert package helps us ensure that Azure resources security configuration drift is kept in check.

Planning for the future

• Adoption schedule for CSE. At present, the Secure Dev Ops Kit has been used in over 25 percent of Microsoft IT Azure subscriptions. Our goal is to push adoption to over 75 percent by September 2017 and close to 100 percent by December 2017. So far, much of the adoption has been through our security governance program for applications that process critical company data. However, we’re increasingly encouraging teams to do due diligence on security regardless of data classification. We also want to drive teams to onboard continuous assurance features more rapidly to ensure sustained security in production environments.
• External requests. The Secure DevOps Kit for Azure is available for external customers who may be interested in using our approach and tools to accelerate Azure adoption. Download the Secure DevOps Toolkit for Azure from Github:
  http://aka.ms/azsdkossdocs
• Feature expansion. We’re working on a few features to help accelerate adoption. Future enhancements include the ability to remember user attestation for failed controls and the ability to allow access for other downstream systems through event hubs or webhooks.