SOFTWARE SUPPLY CHAIN SECURITY: CI/CD/CT PIPELINES AND SECURITY TOOLS: PART 1

Uncategorized

Introduction

The DevOps movement of the last decade more-or-less led to the DevSecOps movement of this decade. This focus on automation to create efficient, end-to-end software publishing pipelines combined with incidents like the SolarWinds Hack in 2020 and the recent ‘S1ngularity’ attack on the NPM ecosystem (along with litany of similar attacks) led to a new focus on security tooling within CI/CD/CT pipelines, software supply chain security, and overall security of the build / test environments for software products within organizations. Even with this recent emphasis software supply chain security, there are still many developers and other technical resources within organizations who blindly let package management tools (npm, maven, etc) import thousands of packages with millions of lines of unreviewed / unsupported code that is deployed to production in short order. This sets the stage for unfortunate things to happen.

This is blog post is primarily considering software supply chain concerns in the context of web applications and APIs, but generally can be extended to native desktop apps, native mobile apps, etc.

The examples given in this blog post, either as specifications or specific-products, are primarily based on what I have first-hand experience with. If a specific product or standard isn’t mentioned, it means that it isn’t something I’ve worked with. When I’ve written similar posts in the past, I was inundated with comments of “What about my favorite tool XYZ?” Feel free to share in the comments, but I cannot add every tool in every capability category to the list / diagram. So, I draw the line at tooling / specifications that I’ve personally worked with.

This post is not meant to be an exhaustive exploration of each of the topics mentioned — opportunities for future blog posts. Furthermore, just because you setup all of the security capabilities mentioned here in your software supply chain doesn’t automatically mean you are secure. Most of the security capabilities described here are process-oriented that need to be repeatable and built into the way your development teams do their work — it’s not enough to have a static code scanning tool, you have to actually address the issues described in the generated reports in a timely manner.

What is a Software Supply Chain? 

From here we have, “Software Supply Chain is composed of the components, libraries, tools, and processes used to develop, build, and publish a software artifact.” That definition works for our purposes. It isn’t too difficult to imagine what that means. If it is involved in building, testing, or distributing your software, it’s part of your Software Supply Chain.

What is a Software Supply Chain Attack?

From the United States CyberSecurity & Infrastructure Security Agency’s (CISA’s) “Defending Against Software Supply Chain Attacks”, we have “A software supply chain attack occurs when a cyber threat actor infiltrates a software vendor’s network and employs malicious code to compromise the software before the vendor sends it to their customers. The compromised software then compromises the customer’s data or system. Newly acquired software may be compromised from the outset, or a compromise may occur through other means like a patch or hotfix. In these cases, the compromise still occurs prior to the patch or hotfix entering the customer’s network. These types of attacks affect all users of the compromised software and can have widespread consequences for government, critical infrastructure, and private sector software customers.” This mostly works as a definition for this discussion, but it is critically important to understand the the organization building the software could be your own. This can include:

  • Compromising Development Tools or Infrastructure
  • Infiltrating Source Code Repositories
  • Compromising Software Updates
  • Exploiting Third-Party Dependencies
  • Manipulating Distribution Channels

How Does One Mitigate This Type of Attack?

Anyone who has been reading my blog posts for a while has probably heard me use the phrase “exercise in the basics”. I know, it’s boring, but it goes a long way to addressing the issues.

  • Governance (IT, Data, Corporate, SDLC, etc)
  • Risk Management
  • Source Code Management
  • Integrated Development Environments (IDEs)
  • Package Tools
  • Artifact Repository
  • Container Image Repository
  • Container image scanning service
  • Build Automation/Management Technologies
  • (CI/CD/CT) Pipelines
  • Static Application Security Testing (SAST)
  • Dynamic Application Security Testing (DAST)
  • Software Composition Analysis (SCA)
  • Software Bill of Materials (SBOM)
  • Patch Management
  • Vulnerability Management
  • Peer Code Reviews
  • Software scanning tools
  • Release Management
  • Software Distribution
  • Secure Software Architecture
  • Automated Testing
  • Identity (User SSO for each tool)
  • Identity (system-to-system authentication)
  • Secure Credential Storage
  • Audit capabilities of build process.

Most of this is stuff you should be doing already. If you are like most organizations, some of these are strengths, some are weaknesses. It’s a journey.

The Modern Software Supply Chain

The Modern Software Supply Chain Security Architecture

Governance

Any mature, professionally-managed (by adults) software development organization has an effective governance paradigm that encompasses:

  • Corporate
  • IT
  • Data
  • Software Development Life Cycle
  • Regulatory Compliance

SOC / SOC2

SOX (Sarbanes-Oxley)

HIPAA/HITRUST

PCI-DSS

Numerous banking regulations (USA)

Numberous banking regulations (EU)

Common Criteria

ISO-27001

ISO-27002

NIST CSF

FISMA

GDPR

NIS2

FIPS

CIS

There is much more to governance than these, but this is relevant to the current topic — governance and lack thereof is a future blog post.

Software Development Life Cycle (SDLC)

The Software Development Life Cycle (SDLC) is a structured, repeatable process that describes all phases of the software creation process (designing, building, testing, and delivering, etc). It outlines each stage of development step by step, with defined checkpoints to ensure compliance and quality. The ultimate goal is to produce reliable, maintainable software that meets the specified requirements.

The exact steps and checkpoints for your organization’s SDLC will differ to accommodate that organizations needs. It will most likely be similar. The control gates will likely correspond to prescribed environments the organization uses (dev, qa, ppe, prod, etc, etc, etc). It’s also common to require a sign off from a QA test manager, load testing, change management, etc. This sign off process should be as simple as possible. Click of a button. Maybe email is used with links to approvals; however, this can create phishing opportunities.

Risk Management

There are a number of Risk Management frameworks and standards in industry and government. Here are some of the common ones.

There is some overlap with the list in the earlier governance section.

I will mention one pet peeve of mine in this section. Risk management cannot be pushed down to individual teams / groups (or management thereof) who are financially and reputationally-incentivized to accept any and all risk no matter the potential consequence. I’ve watched that happen too many times. That is the “bad joke” version of risk management. In the whole history of humanity, that has never ended well. Stop doing it!

Asset Management

An organization’s IT department is responsible for maintaining a database of all technical assets owned by that organization. This includes hardware (servers, switches, routers, UPSes, network appliances, etc), virtual components (Virtual Machines, virtual network components, etc), services (Microservices, containers, APIs, etc), Software as a Service (SaaS) services in use, etc. The Asset Management database and Configuration Database are often combined in the same product. The Asset Management database / system is a building block of several other capabilities mentioned in this post: Configuration Management, Vulnerability Management. Having up to date and accurate information is essential to the proper functioning of these capabilities that your software supply chain depends upon.

The Asset Management database will contain meta data such as:

  • device name
  • IP address
  • hardware type
  • hardware specs (memory, CPUs, storage, etc)
  • product name(s)
  • vendor
  • support contract information
  • asset owner
  • asset technical contact

The asset meta data that is tracked differs from one organization to the next. Keeping this information up to date requires constant vigilance. Updating the asset management database should be a defined step in the creation of any additional asset that is tracked.

This is a foundational component of the IT Governance that was mentioned earlier.

Configuration Management

Generally, I view Configuration Management as an extension of Asset Management. I’ve seen the terms used interchangeably across different organizations to describe more-or-less the same thing. It tends to be very fluid what is kept in an Asset Management database versus a Configuration Management database. Again, in many organization, especially smaller ones, it’s the same database / product.

A sub-discipline of Configuration Management is Security Configuration Management (SCM). This may be managed entirely separately from the IT Governance concept of Configuration Management or it could all flow up into the same tools and processes. SCM ensures devices on the network are configured in a secure manner. It defines the baseline security configuration (gold standard) for each device type. It has the ability to to scan devices, look for deviations from policy, and automatically restore the default configuration. Likewise, it may have the ability to scan devices on the network for vulnerabilities. Then, generate reports about compliance with security policy. 

Some of this overlaps with other security capabilities mentioned in this post. It depends on your tool suite, IT Governance strategy, vendors in use, and overall security architecture. Avoid duplication of capabilities and be consistent.

Configuration Management is a foundational component of the IT Governance that was mentioned earlier.

It’s also critical to the Vulnerability Management capability that’s described further down.

Source Code Management

Let’s focus on git as the source code management system. There is no single implementation. There is no single formal spec for git, but the original implementation’s source code serves as a de facto standard. There are some pieces of git that have formal specs:

Integrated Development Environments (IDEs)

There are no specs in this space. There are some specs that IDEs implement for certain capabilities (such as a debugger attaching to a running process).

There are many products / solutions in this space. Here are some I’ve used.

Language / Platform

There are many languages that can be chosen. This is a matter of the right-tool-for-the-right-job, available skill sets, and potentially domain-specific needs. Some of the common languages (and specs) are:

Package Tools

Unique to each programming language, but there is some overlap:

Artifact Repository

An artifact repository is a centralized storage system used to manage and store all related artifacts (outputs from stages of the build process), such as compiled code, libraries, dependencies, configuration files, documentation and build outputs. What these artifacts are vary by language, platform, run times, etc. The artifact repository will ensure these artifact are easily accessible, versioned, and tracked through all stages of the Software Development Life Cycle (SDLC).

Here are a few examples:

The same technology / solution that provides an artifact repository will likely also serve as the Container Image Repository.

Container Image Repository

A Container Image Repository is a specific type of Artifact Repository (see above).

Azure Container Registry (ACS)

AWS Elastic Container Registry (ECR)

Google Artifact Registry

Container Image Scanning Service

Most Container Image Registries provide some type of malware / vulnerability scanning service and reporting mechanism. These services generally operate asynchronously of the pipelines that build / push the container images. For example, see:

Build / Deployment Automation/Management Technologies

There are many options in this space. Depending on your preferences, one tool may be used for the build step and another tool used for the deployment step. I’ve seen many differing opinions on this. When I’m working at a client site, I tend to just go with whatever the prevailing preference is. For my personal projects, I’ll use the pipeline technology available in the Source Code Management platform (ie, GitHub Actions if your using GitHub, GitLab Pipelines with GitLab) to limit the number of different tools being used. For some examples, see:

If there is one recommendation I could leave with you dear reader, pick one and stick with it. I’ve been in organizations that had five or ten different tools in this category. Trying to standardize or change anything in the software supply chain generally ended in nothing getting done and big drama.

(CI/CD/CT) Pipelines

The pipeline technology(ies) in use could have three different types of pipelines defines: Continuous Integration (CI), Continuous Delivery / Deployment (CD), and Continuous Testing (CT).

Continuous Integration: The automated process of taking all changes developers have applied / created across feature branches and combining them into a single release or environment branch. The single combined branch is used to scan, build, test, generate reports for, etc the code base. This may involve spinning up a temporary environment to perform end-to-end testing. If successful, the resulting artifacts are placed into the appropriate artifact repository.

Continuous Delivery: The next step in the automated pipelines can deploy the build artifacts into the target environment. This will involve whatever tooling is needed to deploy / push the artifact to the target environment. However, an approval process is required for production. There may also be an approval required for deployment to any environment; this approval may be baked into the Pull / Merge Request process.

Continuous Deployment: Same basic idea as Continuous Delivery except that there is no manual approval process. True continuous deployment requires a sophisticated and reliable testing automation phase. Continuous Delivery / Deployment tends to look slightly different in every organization. 

Continuous Testing: In some organizations and products, automated testing is setup in a distinct pipeline. Whether this is baked into the CI pipeline or a CT pipeline, what’s important is having a comprehensive automated testing suite.

I tend to generically refer to all of this as CI/CD/CT pipelines. The exact structure of these pipelines has been different in every organization I’ve been in; so, the best advice I can give is to follow best practices and setup a system that satisfies your organization’s needs and maps to its SDLC.

Summary

In the interest of keeping this blog post to a manageable size, it has been split into two parts. Please continue to Part 2.

Feel free to post any comments or suggestions below.

AI / GenAI / ChatGPT / etc were not used to generate this article.

Diagrams created with Lucid.App.

Leave a Reply