In a previous blog post, we discussed what qualities are required in an enterprise code signing system. Most of the questions we received on that post were about hash validation of deterministic builds and device authentication.
Given the news surrounding the SolarWinds attack, we decided to provide more details on hash validation of deterministic builds. For a deeper dive on securing the software supply chain, check out this post on preventing malware injections.
An Overview Of The Problem
Code signing needs to be directly tied to software development. However, for many enterprises, the two systems are disjoint or only very loosely coupled. This opens up the possibility of signing potentially malicious code that doesn’t originate from the source code repository (or other form of artifact repository).
To eliminate this potential threat, a secure code signing system must verify that the code being signed precisely matches the code in the repository and that the code in the repository is worthy of signing.
Prerequisites For An Enterprise Solution
In order for a code signing system to be both secure and scalable, it’s necessary to implement a technique known as client-side hashing or simply as hash signing. In short, this method enables clients to generate a cryptographic hash of the code they are requesting to sign before sending it over the network to complete the digital signature, improving performance by reducing bandwidth consumption.
With a client-side hashing architecture in place, the problem of verifying that the code to sign matches what is in the repository translates to verifying that the hashes match. Since the hashes must match exactly, it is necessary to use reproducible builds. This means that whenever a given set of software is built (i.e., the source code is converted to its final binary format) it must build to the exact same value each time. Many modern compilers support deterministic building, as it is considered more secure, so this usually isn’t a road block.
Learn how a client-side hash signing architecture can accelerate digital signatures for a variety of use cases throughout your enterprise’s environment.
Solution 1: Manual Approvals
The obvious way to verify that the requested hash from the signing client matches the hash from the source code repository is to manually ensure that the hashes are the same. This is most easily accomplished by putting the signing request into a pending state and requiring a quorum of approvers to approve the request before the digital signature is generated. The approvers see the entirety of the request data, including the hash to sign, the signing client who is requesting the signature, which key is to be used, etc.
Using this information, the approvers can manually validate the request. The signing client that continuously polls the server until it receives a successful signature or an indication of failure (typically because one of the approvers rejected the request).
This approach can be expanded on with:
- tiers of approvers, each having their own quorum (e.g., the software development team first approves the request, then the change control board, etc.).
- a requirement that approvers manually submit the hash to sign to avoid “blind approvals.”
Solution 2: Automated Hash Validation
Manual code signing approvals are great for signatures needed infrequently (e.g., quarterly production releases) but they don’t scale well when the number of required code signatures increases significantly (e.g., in the CI/CD pipeline). A high volume of code signatures demands a scalable solution that can still ensure top-notch security. As with many solutions in a DevOps environment, automation is the name of the game.
An automated code signing solution must be able to verify that the hash of the code a client is requesting to sign matches the hash of the code in the repository, without manual intervention. To do this, the signing client includes the project identifier and source control revision number, along with the signing request. A dedicated portion of the code signing system uses this information to retrieve the relevant source code from the source control repository, perform a (deterministic) build of the software, and then compute the hash to sign to see if it matches the hash submitted by the signing client. The code signing system can then determine whether to grant or deny access, depending on whether or not the hashes match.
GaraSign, Garantir’s flagship product, was designed with a client-side hashing architecture for superior performance as well as automated hash validation for increased security. As the following sections will make clear, automated hash validation is an effective technique for improving both the speed and security of code signing.
Code Signing Analysis
By the end of the automated build process, the code signing system has access to both the source code and the built binaries for that particular build. It can then run third-party scanning tools on those artifacts to assess the software for vulnerabilities, code quality, etc. The results of these scans can also be incorporated into the decision on whether or not to sign the code.
Performance
A natural concern with an automated approach is the performance impact it has, especially on CI/CD environments where every second counts. Indeed, the obvious approach to automated hash validation is to validate the hash before the signature is generated, known as Pre-Sign Validation. This preventative approach has the greatest security benefits but comes with the highest performance cost.
Another approach is to validate the hash after the signature is generated, known as Post-Sign Hash Validation. This detective approach slightly lessens the security guarantees (something invalid could get signed but it would eventually be detected) but allows the CI/CD pipeline to continue operating at the same high speed as it does without any hash validation technique in place.
Post-Sign Hash Validation can be further improved to accelerate the CI/CD pipeline beyond the speed possible without any code signing solution in place.
At first, this may sound like snake oil, but the solution is actually quite simple, albeit clever. First, let’s look at a CI/CD pipeline without code signing at all:
Now, here’s a CI/CD pipeline with code signing.
The general goal is to make code signing as secure as possible while minimizing Y, the additional time taken for code signing, to the greatest extent possible.
However, with Automated Post-Sign Hash Validation, the secure code signing system gets its own copy of the source and binaries, so it can offload some of the processing that the CI/CD pipeline would otherwise take on, and complete this process in parallel. Using this approach the CI/CD pipeline now looks like:
Now, the overall time to complete a build is less than the original time to build. This is, of course, measured in wall-clock time (aka, elapsed real time). The overall CPU time (aka, processing time) will increase.
SolarWinds Attack
The current question du jour is whether automated hash validation would have thwarted the SolarWinds attack. The answer is, it likely would have.
According to SolarWinds, the malware “was not present in the source code repository of the Orion Platform products”. This means that the hash validation would have detected a mismatch and either not have signed the build (if validating in pre-sign mode) or alerted about the issue (if validating in post-sign mode).
If, instead, the attackers had injected their malware into the source control repository then the answer would be different. Under this scenario, the requested hash to sign would have matched what is in the repository so that would not have resulted in any validation issues. However, if hash validation were combined with static or dynamic code analysis, then this approach’s ability to have detected and/or prevented the attack would depend on the capabilities of the scanners.
Regardless of whether this approach would have stopped the SolarWinds attack, automated hash validation is a highly secure technique that can additionally be configured to improve the performance of your CI/CD pipeline.
For a deeper dive on secure software development, check out this post on preventing malware injections.
GaraSign, Garantir’s flagship product, uses client-side hashing and automated hash validation to make enterprise code signing processes both faster and more secure. To learn more about deploying GaraSign in your environment, get in touch with the Garantir team.
