At Spearbit, we have established a reputation for conducting very thorough smart contract and protocol security reviews. Our focus often centers around the reviews themselves, the dedicated researchers conducting them, and the significant findings they reveal. However, one aspect that merits additional emphasis is Security Review Readiness for protocols where there is an intentional effort to prepare the codebase, technical documentation, resources, and communication structure to streamline the security review and maximize potential results. This guide will answer the following question in detail:
Protocol’s Lifecycle and Security Research Needs
Stage-Dependent Security Needs:
-
Early Stages of System Design: It's too soon for a code review. However, this is an ideal phase for an architecture review.
-
During the MVP: A full team review may be premature. Instead, one or two researchers could conduct a preliminary review. Often termed “solo reviews”, they typically involve more than one reviewer in practice.
-
Frozen Feature Set w/o Solid Test Coverage: Opting for a full review might be tempting. Yet, without comprehensive test coverage, security researchers might waste time identifying issues that could be revealed through tests like unit, e2e, or property/fuzz tests. In such scenarios, engaging a security researcher with expertise in fuzzing can enhance automated code coverage.
-
Preparation for Mainnet Launch: A third-party security review is essential before going live on the mainnet. The key consideration here is whether the protocol is sufficiently prepared for this step.
-
Post-refactor or Incremental Feature Addition: Options such as solo/pair reviews, public competitions, or team security reviews are suitable. The crucial aspect to evaluate is the protocol's readiness for such reviews.
General Consideration: The cliché answer to when to involve security researchers is that it depends. The development stage of a protocol dictates distinct security needs. Engaging with security researchers can be beneficial even before reaching full review readiness. Protocol developers should aim for a bug-free protocol before going live and ensure that their protocol is in a condition ready for review.
Defining 'Review Ready' in Protocol Development
Understanding the Diversity in Development Teams: Protocol development teams come in various sizes and possess a wide range of skill sets, from hardware engineering and front-end development to crypto-native expertise. Their resources are equally diverse. Given this variety, there is no universal checklist that can define when a protocol is 'Review Ready'.
Core Criteria for Review Readiness: The primary indicator of review readiness is when a protocol team has exhausted all within their capacity to identify and eliminate bugs. This milestone signals that the project is 'Review Ready'. At Spearbit, we offer the following general guidance.
A Realistic Perspective on Bugs: Achieving review readiness does not imply the codebase is entirely free of bugs. It's a common experience in development that some bugs elude detection by their creators. Importantly, reaching the 'Review Ready' stage does not signify the end of the development process. Rather, it marks a critical transition point where external review becomes highly beneficial.
Guidelines to Achieve Review Readiness
Understanding QA and QC in Project Management: In project management, a key distinction to consider is between Quality Assurance (QA) and Quality Control (QC). QA is about implementing processes to prevent issues from occurring, while QC focuses on detecting issues in existing products or services. This understanding is crucial for framing our steps toward review readiness.
Categorizing Steps into QA and QC: The steps we advocate for achieving review readiness align with these two critical aspects of quality management:
-
Quality Assurance Steps: These are proactive steps that aim to prevent issues in the protocol development process. They involve setting up the right procedures, standards, and methodologies from the start of the project. These include:
-
System Specification
-
Removing Attack Vectors
-
Identifying Risks
-
Refactoring
-
Code Reuse
-
-
Quality Control Steps: These are reactive measures, implemented to detect and address issues in the already developed aspects of the protocol. This includes various forms of testing and reviews to identify and rectify any problems.
-
Code Freeze
-
Test Coverage
-
Peer Code Review
-
Checklists
-
Automated Code Analysis
-
Quality Assurance (Prevention)
A. System Specification
Why is System Specification Important?
-
Role in Bug Prevention: A comprehensive protocol specification is pivotal in identifying potential vulnerabilities early in the development process. It serves as a roadmap that delineates the system's intended behavior, crucial for effective testing and bug prevention.
-
Facilitating Testing and Review: Detailed specifications provide a clear reference point for testers and reviewers. Without such guidance, they may have to rely on guesswork to infer intended behaviors, which can lead to oversight of critical issues.
Advantages of Specification Over Implementation
-
Ease of Revision: Modifying a specification is typically less complex and resource-intensive than revising an implemented system. This accessibility allows for more frequent and thorough revisions, enhancing the overall robustness of the protocol.
-
Encouragement for Developers: Protocol developers are advised to invest time in meticulously detailing their system specifications. This practice not only aids in the early detection of potential issues but also opens opportunities for refining and simplifying the protocol.
Specification Tools and Techniques
-
Plain English: Utilize simple, clear language to describe the protocol's functionality. This approach ensures that the intent and operations of the protocol are easily understandable.
-
State Machines: Employ state machine models to visually represent the protocol's states and transitions, aiding in the identification of potential issues and logical inconsistencies.
-
Diagramming: Create diagrams to map out the architecture and workflow of the protocol, providing a visual representation that can be invaluable for understanding complex interactions.
-
Interfaces with NatSpec: Use NatSpec (Natural Language Specification) to document code and interface details, ensuring that they are well-defined and easily interpretable.
-
Placeholder Tests: Develop preliminary tests to simulate protocol operations even before full implementation, helping to identify potential failures and design flaws early on.
-
Formal Definitions: Where applicable, include formal definitions to rigorously specify the protocol's behavior, ensuring precision and reducing ambiguity.
Please explore the following thread for further guidance.
Additional Resources
- Discussing Preconditions, Postconditions, and Invariants: For more insights into ensuring protocol integrity and reliability, explore this linked thread. It delves into the significance of preconditions, postconditions, and invariants in protocol development.
B. Removing Attack Vectors
Understanding your Attack Surface
-
Principle of Elimination: The most effective way to mitigate an attack vector is to remove it entirely. Eliminating a potential threat is more efficient than defending against it.
-
Example - Admin Key Management: Consider the management of admin keys. The simplest method to secure these keys isn't through hardware wallets or secret stores but by eliminating admin-level privileges. Without protocol admin roles, there's no need to secure an account that doesn't exist.
-
Application to Protocol Design: Assess your protocol for existing attack vectors. Ask if they can be completely removed instead of just fortified against potential breaches.
C. Identifying Risks
Risk and Threat Assessment
-
Threat Modeling: This is an ongoing process crucial throughout the protocol's lifecycle. It involves a systematic approach to identify potential threats.
-
OWASP’s Four Question Framework: These questions guide the process of identifying and addressing potential risks. For further details please see the official OWASP website.
Practical Activities for Risk Identification:
-
Review incidents, bug disclosures, and security reviews from similar protocols.
-
Conduct internal team discussions and brainstorming sessions.
-
Consult with security researchers to pinpoint known risks.
-
Explore other relevant activities that contribute to a comprehensive understanding of potential vulnerabilities.
D. Refactoring
Refactoring Advantages
-
Emulating Pascal: The famous quote from Pascal, "I only made this one longer because I didn’t have the time to make it shorter," resonates deeply with software development. Much like writing, developing software requires a focus on brevity and clarity.
-
The Need for Refactoring: During the development cycle, especially when adding features and testing protocols, allocating time for refactoring is critical. This process should aim to simplify the system, streamline its implementation, and minimize potential attack surfaces.
-
Advantages of Simplicity: Simple protocols are often safer, or at least, their safety (or lack thereof) is easier to ascertain. Refactoring with the goal of simplification is a proactive step towards bug prevention.
E. Code Reuse
Don’t Reinvent the Wheel
-
Leveraging Existing Resources: Utilizing well-tested and peer-reviewed codebases, rather than building from scratch, is a strategic approach to bug prevention.
-
Practical Application: When features are already available in established libraries like OpenZeppelin or Solady, prefer using these resources over creating new implementations. This not only saves development time but also reduces the likelihood of introducing new bugs.
Quality Control (Detection)
A. Code Freeze
Advantages of a Code Freeze
-
Balancing Flexibility and Discipline: While a code freeze is more a guideline than a strict rule, its importance in the development process cannot be overstated. It serves as a checkpoint to evaluate the maturity and stability of the codebase.
-
Indicators of Insufficient Preparation: Frequent commits just days before a security review are often telltale signs that other crucial prevention and detection steps might have been overlooked or rushed.
-
Halting Feature Addition: The first and foremost action in moving towards review readiness is to stop introducing new features. This allows the focus to shift from expansion to examination.
-
Stabilizing the Codebase: Freezing the codebase provides a stable platform for a thorough and effective bug detection phase. It ensures that the code being reviewed is the final version, barring critical fixes, and not a moving target.
B. Test Coverage
The Importance of Testing in Uncovering Bugs
-
Testing as a Primary Tool: A crucial strategy in identifying bugs is the examination of untested code segments. We often share this insight with bug bounty hunters.
-
Complementing Other Security Measures: While testing alone might not catch all bug types (hence the need for security reviews, fuzzing campaigns, etc.), it plays a vital role in detecting a wide array of errors.
The Significance of Comprehensive Test Coverage
-
Targeting High Coverage Standards: In the realm of protocol development, aiming for 100% statement and branch coverage is not just advisable but necessary. Given the immutable nature of contracts and the tangible value of data in our systems, even higher coverage is often pursued, with certain code areas requiring multiple tests.
-
System Specification as a Testing Guide: The previously discussed System Specification section serves as an excellent foundation for test coverage. Protocols should strive to develop test suites that verify the implementation's adherence to the specified standards.
-
Incorporating Threat Modeling: Review your threat model to write tests that ensure specific issues are either non-existent or effectively mitigated.
Additional Resources for Enhancing Test Coverage
- Expert Insights: For further guidance, refer to the works of Paul R. Berg's BTT and Alexander González's Bulloak, which provide valuable perspectives on test coverage strategies.
C. Peer Code Review
Understanding the Value of Peer Code Review
-
Similarity to Security Reviews: A security review is, in essence, an advanced form of peer code review. Therefore, it should not be the first instance where peer code reviews are conducted in the development process.
-
Integrating Regular Review Processes: Establish a routine of conducting feature-by-feature pull request (PR) code reviews. This practice not only enhances the immediate quality of each feature but also prepares the team for more comprehensive reviews.
Comprehensive Code Base Review
- Emulating Third-Party Security Reviews: Besides feature-specific reviews, allocate time for thorough, top-to-bottom reviews of the entire codebase. This approach should mirror the process undertaken in third-party security reviews, providing an internal check before external evaluation.
Further Reading and Resources
- Expanding Expertise: For additional insights into effective peer code reviewing and its role in smart contract auditing, refer to cmichel's post on "How to Become a Smart Contract Auditor."
D. Checklists
Advantages of Checklists
-
Cross-Referencing for Comprehensive Coverage: Security checklists serve as an effective tool to compare your existing coverage with key areas of concern identified by experts. These lists encapsulate critical points deemed essential for thorough security analysis.
-
Timing in the Review Process: Conducting a self-assessment using one or more checklists is a highly recommended step before proceeding to the internal Peer Code Review. This ensures that major areas have been addressed and sets a solid foundation for more detailed reviews.
Here is some guidance on potential checklists for a protocol team to explore including but not limited to:
-
General Checklists:
-
Subject Matter-Specific List:
E. Automated Code Analysis
Effective Use of Tooling
-
Overview of Tooling: Before undergoing a security review, it is prudent to identify and rectify obvious vulnerabilities. This can be efficiently done using static and automated analysis tools. Be aware that these tools are not infallible and might generate false positives. However, the presence of multiple warnings should not be a deterrent.
-
Enhancing the Review Process: Running these analyses proactively allows for the early addressing of potential issues. This preparation not only streamlines the subsequent manual review process but also enables security researchers to dedicate more time to uncovering complex, less obvious issues.
-
Recommended Tools:
Conclusion
By following the steps and guidelines above, your protocol will be prepared for a security review and able to streamline the relevant processes within that review for an effective and efficient engagement.
Looking for a Security Review?
Please contact us below and we will turn around to you with a quote within 24 hours:
You may also reach out to us via our Twitter: