Funded by a DARPA-sponsored SBIR opportunity through Phases I and II+Option (2017–2021), Stealth’s DEBLOOM solution was designed to satisfy the objective of the Secure Messaging Platform (SMP) program:
“Create a secure messaging and record platform that separates the message creation, from the transfer (transport) and reception of the message using a decentralized messaging backbone to allow anyone anywhere the ability to send a secure message or conduct other records across multiple channels traceable in a decentralized ledger.”
The DEBLOOM platform that we developed across the four years of the SMP program was built on a blockchain framework, which we chose as our backbone based on its ability to meet the program goals of decentralization and immutability of records once entered into the system. However, although we employed the blockchain paradigm as our starting point, DEBLOOM features a number of enhancements over traditional blockchain systems:
- Scalable Consensus. A key challenge for any distributed system, especially one in which individual users do not know or trust one another, is reaching agreement on the current system state and ensuring consistency between each of the decentralized servers that maintain the system. Traditional blockchain solutions rely on proof-of-work protocols (and related variants), which are slow and resource intensive. By placing reasonable assumptions on the underlying system (namely, the existence of a subset of users, some of whom will behave “honestly”), DEBLOOM is able to draw from techniques in cryptography – specifically the well-studied subfield of Byzantine agreement – to provide a far more efficient solution to the consensus problem.
- Fast Search. Unlike many traditional blockchain storage systems, DEBLOOM supports fast (sub-linear) search capabilities, whereby users can quickly find the records they are searching for without performing a linear scan of the entire blockchain.
- Separate, Linked Private Storage. In addition to the storage of public information that traditional blockchain systems provide, DEBLOOM offers separate storage for private information. This private information is encrypted at all times – only the users who generated the data have the decryption key – and is linked with “on-chain” public versions, ensuring consistency between the two.
- Zero-Knowledge Policy Enforcement and Auditing. Because DEBLOOM provides storage for private information, the tasks of ensuring that this information is accurate (consistent with the public/on-chain version), legitimate (well-formed and/or conforming to policy rules), and auditable (accessible for inspection by authorized users to ensure compliance) require special tools to perform these tasks over encrypted records.
- Secure Computation Backend. Because we use Stealth’s special encryption mechanism for private storage, we can link the database(s) storing this private information to Stealth’s secure computation engine, which allows for (authorized) analyses of users’ aggregate private records.
DEBLOOM is a general-purpose platform with a robust design that makes it appropriate for a number of target applications. Namely, DEBLOOM supports a range of privacy and security requirements, as well as network and usage parameters, and provides functionality for a spectrum of users with different goals, responsibilities, and privileges. The DEBLOOM platform comprises three core components, and there is flexibility in the design both within each component as well as in how the components interoperate with each other:
Component 1: Authentication and Consensus
Component 2: Secure Storage
Component 3: Accessibility Toolkit
Authentication and Consensus
Our network design allows a dynamic network with mutually untrusting participants. As such, authenticating the validity of new messages and records and maintaining the integrity of the blockchain ledger is a core feature of DEBLOOM. Additionally, as with any distributed ledger technology, ensuring a consistent view across all participants is crucial. DEBLOOM provides strong assurances of these properties, even in the presence of some users who may deliberately try to disrupt or threaten them, through the use of rigorous techniques from cryptography. Specifically, to support authentication of sensitive data, we use techniques from the subfield of cryptography known as “zero-knowledge proofs,” through which the satisfaction of policy rules or data properties can be verified without requiring raw (decrypted) access to the data. Meanwhile, consensus is supported by incorporating techniques from the subfield of cryptography known as “Byzantine agreement,” which enables DEBLOOM to guarantee that the data stored and accessed is up-to-date and mutually consistent across all servers – even if there is limited trust in the authorities in charge of monitoring and approving data updates.
Secure Storage
The DEBLOOM platform is designed to accommodate a variety of usage scenarios, scaling from a small handful of users to networks of millions of users and petabytes of data. Additionally, there are tunable parameters that allow customers to select a range of privacy settings regarding access to the message and record contents. In order to support these various use cases, we support two flavors of data storage: “on-chain” (for public, succinct information) and “off-chain” (for complete, private information). Both the on-chain and off-chain storage structures are designed to facilitate efficient search, auditing, and analysis, and are distributed across multiple databases to ensure redundancy and decentralized availability to authorized users.
This stores “public” information, for example message and record metadata (which may consist of participants, timestamp, succinct summary, and similar data). While DEBLOOM offers a wide range of capabilities and user roles for interacting with the system, the most common use will likely be by ordinary users (as opposed to maintainers with special roles such as auditors or analysts), and these users will interact with the DEBLOOM platform in three main ways: (1) making new data entries, (2) viewing and searching on-chain (public) data, and (3) maintaining their off-chain (private) data. Therefore, from an ordinary user’s perspective, having an intuitive and easy-to-access interface for entering, viewing, and searching the on-chain data is crucial. As part of development in the SMP program, we developed multiple user interfaces and software plug-ins to support user front-ends, including an Excel add-on and a web-based plug-in. Ultimately, the specific front-end of a deployment will depend on a potential customer’s existing workflow and requirements, and the DEBLOOM system was designed to be readily incorporated into users’ existing systems, with a lightweight API and minimal resource requirements. Additionally, in contrast to most traditional blockchain storage systems, DEBLOOM supports fast (sub-linear) search capabilities for on-chain data, whereby users can quickly find the records they are searching for without doing a linear scan of the entire blockchain.
This stores “private” information, for example full message and record details, including payloads and possibly unstructured data. Off-chain data can serve a variety of purposes depending on the application, including providing a more thorough auditing mechanism, enabling research and analytics, and more generally storing important information that is too large (in terms of memory footprint) or too private for storage on-chain.
At the core of DEBLOOM’s off-chain storage design is a strong guarantee of privacy and security. For privacy, all data is encrypted, and restrictions on access to the raw data are strictly enforced based on application-specific rules. For security, DEBLOOM includes a mechanism for ensuring that the data stored off-chain is authenticated and immutable, and also that it is consistent with its on-chain counterpart (see On-Chain/Off-Chain Consistency below). Additionally, in support of both security and privacy, off-chain data is spread across multiple servers (which may be co-located and/or geographically isolated for faster access as well as for an additional layer of security), with no single server able to decrypt the data they store. This means that an attacker attempting to gain unauthorized access, to disrupt access, or to destroy or alter off-chain information must simultaneously attack servers in multiple locations. As such, DEBLOOM’s multi-server distribution of off-chain data provides both redundancy and resilience.
Finally, DEBLOOM offers several authorized tools to interact with off-chain data; see the Integrated Toolsuite section below.
Due to the sensitive nature of off-chain information, only the users who are involved in the generation of data changes (insertions, deletions, and updates) are able to make the appropriate changes to the encrypted off-chain database. Therefore, DEBLOOM must protect against negligent or deliberately malicious individuals who do not faithfully maintain their off-chain records in accordance with the information that was actually registered on-chain. Furthermore, access to the off-chain database is restricted even for “super-users” such as system administrators or auditors, who have a limited ability to view off-chain records – in particular, in general no super-user has raw access to off-chain data or the ability to decrypt it. Due to this limited access, dictating that such super-users will shoulder the burden of maintaining on-chain/off-chain compatibility still requires design work to specify how these users will perform this task.
DEBLOOM is able to guarantee on-chain/off-chain consistency by introducing a special user role of “off-chain auditor”. In terms of resource overhead, this new role likely has a small impact, as auditors need only periodically perform audits (for example only at the end of each day, and only on users who were involved in records that day). In terms of privacy and information-leakage considerations, auditors do not learn anything about any user’s private off-chain data, except for a yes-or-no indication of whether that user has faithfully maintained their records in the off-chain database (in accordance with records that appear in the on-chain ledger). DEBLOOM enables such off-chain auditors to perform their role without leaking private information, by leveraging a tailor-made zero-knowledge (ZK) protocol with minimal communication and computation overhead.
Since the on-chain information is different from – often a strict subset of – off-chain information, the “consistency” between the on-chain and off-chain information that DEBLOOM provides includes:
- Any record that has been entered on-chain also appears in off-chain storage. Implicit in this statement is that record details (including for example the set of participants, the time/date of the record, the record amount, goods exchanged in the recorded transaction, and the like) are the same. Note that the on-chain ledger might only contain metadata and/or a commitment/hash of the full record, while the off-chain SSD stores all record details (in encrypted form). Thus, “matching record details” means that the metadata and/or commitment/hash that are stored on-chain are consistent with the (encrypted) record details that are stored off-chain.
- Every modification made to an off-chain record has a matching entry on-chain. In particular, changes made to the off-chain database are in one-to-one correspondence with corresponding changes to the on-chain ledger.
Accessibility Toolkit
The DEBLOOM platform offers a suite of search, auditing, and analysis tools, which can be used to verify properties of the on-chain ledger and to provide controlled access to private off-chain data. These tools include zero-knowledge verification of adherence to rules and policies, accumulators with witness verification, secure search and private information retrieval, real-time aggregation/count statistics via function secret sharing (FSS), zero-knowledge auditing, and secure computation of logical or arithmetic functions via secure multi-party computation (MPC).
For applications that desire it, DEBLOOM offers a search mechanism for the off-chain database (“Stealth Secure Database (SSD)”). This search mechanism has strong privacy guarantees, namely, it is:
- Privacy-preserving. No records are decrypted in performing the search; decryption occurs only at the end of a search, whereby the searcher is able to decrypt the results returned from their search query.
- Oblivious. Database servers learn nothing about which – or even how many – records matched the query.
In addition to these privacy guarantees, DEBLOOM also supports additional security features, including privacy for the searcher (the query is hidden from the SSD server); policy enforcement (for, e.g., query rules, searcher quotas, and data-dependent policies); and private policies (policy enforcement is performed even while hiding policy rules and even policy decisions for a given query).
While certain auditing tasks may be possible just by inspection of the public on-chain information, other auditing tasks may require accessing private off-chain information. DEBLOOM supports several such applications:
- Ensuring validity of new private records as they are entered into the system (discussed above)
- Ensuring consistency of on/off-chain data (discussed above)
- Direct access via Searchable Encryption (discussed above)
- Auditing and analysis via generic secure computation (discussed below)
- “Break-the-glass” emergency raw access
Each of the above capabilities can be turned on and off, and they can be customized to meet the specific requirements of the given DEBLOOM deployment. For example, for the “break-the-glass” scenario, DEBLOOM can allow a configuration in which a group of super-users/system maintainers can come together to reconstruct the decryption key for a user who refuses to cooperate with a valid audit request, and thus obtain raw access to that user’s private data.
DEBLOOM connects with Stealth’s secure computation engine to provide support for running arbitrary computations (that are representable by circuits) on top of the (aggregated) private off-chain user data. Furthermore, DEBLOOM allows for the results of a database query to the Searchable Encryption tool (see above) to be piped in as the inputs to the secure computation engine, which greatly enhances the power of the aggregate analysis tool, by selecting on the relevant records for analysis (as well as keeping the actual search results private/not directly accessible).
This work was supported by DARPA under contract numbers W911NF-16-P-0032 and W911NF-18-C-0036, Distribution Statement A: “Approved for Public Release, Distribution Unlimited. If you have any questions, please contact the Public Release Center”. The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.