Hyperledger Global Forum is the most important annual event for enterprises that adopt consortium blockchain technologies. Hundreds of blockchain enthusiasts come together at the annual Hyperledger Forum to share their user cases and the latest progress on enterprise blockchain technologies. During the conference, I presented on the 10 critical problems and requirements to consider based on numerous enterprise blockchain implementation projects using Hyperledger Fabric-based Oracle Blockchain Platform (OBP). These projects cover the gamut of industries, including financial services, supply chain, healthcare, and government, and range from custom developments supported by the Oracle solutions team to SI-led projects and ISV solutions.
These critical issues include:
- Using SQL for rich queries in Smart Contracts
- Data Backup/Recovery
- Ledger checkpoint and pruning/archiving
- Byzantine Fault Tolerant consensus
- Governance
- Performance
- Privacy & Confidentiality Protection
- Inter-network Support
- Pluggable Crypto Implementations
- Auditing Capability
Although the original public blockchains rely on a self-sovereign management style with complete decentralization and rules governed by consensus algorithms, permissioned blockchains have a different structure. In the enterprise-permissioned blockchains used in private or consortium deployments, the participating enterprises are often concerned about maintaining their own nodes with efficiency and resiliency while, at the same time, working as part of a cross-company blockchain network. This requires a secure and flexible governance model and on-chain collaboration mechanisms to address the many operational issues at different layers of the blockchain network – from interoperable connections, to storage management, membership governance, chaincode distribution, etc. As organizations set up their blockchain networks, there are several things they need to pay particular attention to and design their networks with in mind.
SQL and Smart Contracts
The first issue is about supporting SQL language for rich queries in smart contracts – queries of Key/Value data that apply conditions to the values. While some are neutral on this issue, my team views this as an important option for enterprise users, especially those who want to migrate their existing SQL-based business logic into blockchain smart contracts quickly. Using extensive SQL SELECT capabilities minimizes the code complexity and enables a single query to aggregate results that would otherwise involve multiple queries (necessitating multiple network hops between the chaincode container and peer container where the world state database resides). Based on the open-sourced BerkeleyDB K/V store with SQL-Lite, Oracle Blockchain Platform allows customers to enjoy the power of SQL, while also achieving more than 2000 TPS. For interoperability purposes, CouchDB Query Language is also supported in the chaincodes on top of Berkeley DB world state database.
Blockchain Data Backup & Recovery
For production customers, questions about how to backup and recover the data for blockchain networks are very critical. Although it’s easy to create a new node in theory, it is not practical to wait hours for a new node to sync up the ledger data from existing blockchain nodes and transfer configuration metadata. And when there’s the requirement to migrate a node across datacenters, this recovery operation becomes critical. Oracle has already resolved this problem by designing specific tools for OBP customers, which avoid service disruptions when migrating or upgrading the blockchain nodes.
Ledger Pruning and Archiving
Some of our customer projects expect to handle very large transaction volumes (e.g., a maritime shipping network tracking 10s of millions of shipments/year, each recording on-chain a large number of documentation and logistics events, could exceed 3B transactions/year). Depending on the payload size and number of digital signatures (endorsers) attached to the transaction, the total storage requirements could grow to the levels that are going to be difficult to manage. There’s a strong interest by some of the leading customers in a solution that can prune the ledger, archive old content, and keep the overall storage requirements manageable. Some work is underway in Fabric (e.g., FAB-106 Jira). We believe a solution to the ledger pruning requirements is mandatory and are exploring different approaches consistent with the overall architecture of Fabric in order to be prepared when customer blockchains grow big enough to need a solution.
Byzantine Fault Tolerant Consensus
Byzantine Fault Tolerance (BFT) is defined as the feature of a distributed network to reach consensus (agreement on the same value) even when some of the nodes in the network fail to respond or respond with incorrect information.
An important consideration to be aware of while setting up a blockchain network is the requirements of Byzantine Fault Tolerant (BFT) consensus, compared with the Crash Fault Tolerant (CFT) one. Due to the underlying complexity of BFT consensus algorithms, a best practice is for the community to leverage the latest academically-proven consensus algorithms based on rigorous and peer-reviewed demonstrations of the safety and liveness properties. Such algorithms include the Tendermint, Algorand, Mir-BFT and HotStuff. There is also some on-going work on Golang-based implementation of the BFT-SMART algorithm for Hyperledger Fabric. These are important reference points for blockchain architects and developers interested in adopting BFT consensus in the future. At Oracle ,we are actively exploring the available options to ensure they meet the rigorous proof requirements as well as deliver operational characteristics, including performance and resilience required in enterprise applications.
Governance
Governance is another strong requirement, particularly in consortium blockchains, and all audiences believe this is a feature they seriously need. The lack of governance creates challenges in real applications (e.g., how to agree on adding new members, creating new channels and setting their policies, deploying or upgrading smart contracts, etc.). Although users can negotiate these out-of-band using email, conference calls, etc., it is untrustworthy in theory and very inconvenient in practice, especially to achieve agreements across different organizations. And it certainly doesn’t scale for consortiums with more than a dozen members.
We recommend several practical solutions to address the issue using on-chain mechanisms to provide an audit trail for governance-related agreements and chaincodes to automate the processes for reaching these agreements through voting and other means. This could be supported via a special governance channel for persisting transactions, votes, and policy evaluation results, similar to the system channel that is used to help govern the ordering services; the other part would be a Governance System chaincode (GSCC); and the last piece is to use External distributed governing service provided by vendors.
The GSCC option is very promising because in the Hyperledger Fabric 2.0 release organizations can vote for chaincode lifecycle with the help of the implicit collection. Thus, it is feasible to extend the functionality to allow users to vote for other operations, such as adding new members or creating channels. It could also track proposals, manage vote tabulations, and evaluate them against policy requirements. Part of the GSCC implementation has been verified by the Fabric interop team.
Performance of a Blockchain Network
Although customers often ask for performance numbers, sometimes there is a misunderstanding of the performance metrics. For example, in blockchain, the performance metrics include throughput and latency, and results will vary under different network size and hardware configurations and the tradeoffs made between higher throughput vs. lower latency. In analyzing typical blockchain scenarios from finance to supply chain and healthcare to IoT, we can see that the performance requirement is not the same for each scenario. A suggested best practice is to optimize the performance from a systematic view. For example, the chaincode often does lots of calculation, while the ledger will require storage, and the consensus is sensitive to network latency. Hence, a reasonable performance optimization solution will be based on the thoughtful understanding of the business demands and the platform architecture. In customer benchmarks we have observed significant impact of the payload size as transactions move between clients, peers, and orderers; number of peer nodes and their CPU capacity in an instance; ordering block size (more transactions/block can significantly improve throughput but at some cost to latency); network capacity between ordering service and the peers, and other factors. Properly tracking all of these factors and optimizing for those that are most important in the context of a specific application enabled us to meet the highest levels of customer requirements, including exceeding 2000 TPS in a maritime logistics blockchain network.
Data Privacy and Business Confidentiality
Data privacy, which is becoming an important bottleneck in blockchain, has been discussed for quite a while. Enterprise blockchains that plan to share confidential business data are particularly concerned about confidentiality between certain participants and shielding this data from access by other participants. There are a number of approaches, but no perfect solution. Even with the private data collections (PDCs) in Fabric, there’s complexity of configuring PDCs when transactions span different members and when new members join the network. Implicit PDCs in Fabric 2.0 help to some extent. But even with PDCs, the hashed results are still recorded in the ledger, enabling those with access to detect the relationships and frequency of transactions, if not their content.
When blockchain nodes run as a managed cloud service with users only able to access them through the APIs and event subscriptions (and not by accessing the underlying files systems where the ledgers are stored), data privacy can be managed easier. The channel access policies can be set to prevent unauthorized members from seeing detailed block content through event subscriptions and to enable only authorized members to deploy or upgrade chaincode. When coupled with a unique feature of Oracle Blockchain Platform that enables the use of on-chain fine-grained Access Control Lists (ACLs) in smart contracts, the chaincode can control the access based on the user’s identity. And we suggest customers encrypt important data before putting them on the private database (shared by PDC members.) Ultimately, if you care about data privacy, a good rule to follow is to encrypt all your data before putting it on the blockchain.
Inter-Network Operations
This topic spurs a number of active conversations, but the answers aren’t clear today. Let’s break this down into a few more specific questions about what Inter-network operations mean. First, is it about multiple nodes and networks built on the same technology stack (e.g., Hyperledger Fabric)? The Hyperledger Interop Working Group kicked off at the past Hyperledger forum in Basel has been working on testing and documenting how multiple members’ implementations can interoperate. The testing done as part of this working group by Oracle, SAP, and IBM has shown that their nodes can join each other’s network and interop with a common ordering service and channels. The next question is about interop between separate Fabric networks with their own ordering services and channels. This is possible within Fabric architecture, whereby a peer node can be configured to connect with multiple ordering services and see channels supported by each of them. When coupled with inter-chaincode query capability, this enables a chaincode on one network to query a chaincode from another network for data from its ledger.
Moving to heterogeneous technology stacks, what about interop between Fabric and other blockchain technologies, such as Ethereum, Corda, Quorum, etc.? Again, we need to break this down to more specific questions. For example, do we want to:
- Read/query data from multiple ledgers?
- Write data to multiple ledgers with consistent results (i.e., both are committed or none)?
- Involve nodes and members from multiple networks in consensus mechanisms?
The answers vary. Querying the data is possible using “oracles,” gateways that can retrieve data from one trusted source and make it available to another network, while mediating trust relationships (e.g., using Chainlink). Writing data consistently is possible using centralized intermediary gateways, which is not wholly satisfying, but even then the diverse models of finality across these stacks make this challenging as deterministic finality in Fabric is not always possible in other networks. Cross-network consensus remains at the level of academic research and will not be achieved in the near term.
Pluggable Crypto Implementations
The regulatory compliance requirements in some regions (e.g., Asia, Russia and, potentially, the EU) require that blockchains align with regional crypto standards. Today this is handled by scanning Fabric implementation for all uses of specific crypto libraries and the painstaking effort to replace these with regional/country crypto standards. Making the crypto service pluggable will save a huge programming effort, and it can promote the open source technologies into a broader market. Fabric Jira FAB-5496 has been created to track this.
Auditing Capability
Finally, the issues of supporting auditing must be considered. Compared with logging, metrics and tracing, auditing is specific for business and legal regulation and compliance (used by auditors and accountants). In some projects involving Intercompany accounting and billing reconciliation, the information on the ledger related to how invoices are settled and paid feeds into financial systems that are subject to SOX-404 compliance requirements. This means that internal or external audit teams need to be able to verify the source data and the integrity of the systems that maintain it. While we provide tamper-evident block chaining capability in Fabric, we do not currently provide tools to allow a user to re-verify on-demand the cryptographic hashes linking the blocks or the validity of the digital signatures (and the current status of the certificates) attached to past transactions. In discussions with the SIs that are implementing blockchain projects for some of our customers and are also involved in audits, it became clear that having such APIs for auditing purposes would be very useful and could promote the use of blockchain in the financial departments of many enterprises.
Summary
Enterprise blockchain is rapidly gaining adoption in many industries and for many use cases. In our experience, practical applications of blockchain in production deployments by early adopters raise a number of key considerations and challenges that have to be addressed for greater acceptance of enterprise blockchain in the mainstream. Our blockchain team and partner ecosystem is exploring a number of these topics with customers. As an active Hyperledger member, Oracle is providing recommendations and suggesting approaches to these “bleeding-edge” issues through the various working groups and the TSC. Jointly with the rest of the Hyperledger community, we continue to evolve Fabric and other Hyperledger projects to meet these requirements for the broader set of companies and other organizations who will be adopting these technologies in the coming years.
In case you missed my session at the Hyperledger Forum, or were unable to attend, you can watch the session replay of “Practical Issues in Blockchain Implementation.”
Developers can learn more helpful tips by downloading the complimentary Oracle Blockchain Developer eBook.
About the Author
Baohua Yang, Oracle, Principal Architect, Oracle Blockchain Platform
Baohua has served the Hyperledger community as a member of the Technical Steering Committee (TSC) and co-chair of the Technical Working Group in China (TWGC) from 2017-2019. As an open-source developer, he contributes to numbers of projects (e.g., Hyperledger, OpenStack and OpenDaylight), and is leading several (e.g., Hyperledger Cello, fabric-sdk-py). More about him can be found at yeasy.github.com.