We’re excited to announce that Caliper has been accepted by the Technical Steering Committee as the latest Hyperledger project! Hyperledger Caliper is a blockchain benchmark tool, which allows users to measure the performance of a specific blockchain implementation with a set of predefined use cases.
Why Caliper?
The performance of a blockchain solution is one of the most concerned features from blockchain users. However, currently there is not a general tool that serves to provide performance evaluations for different blockchain solutions based on a set of neutral and commonly accepted rules.
Some reports exist on the performance of different blockchain implementations in various scenarios, however there is not a commonly accepted blockchain benchmarking tool. Hyperledger Caliper is a benchmark tool for blockchain frameworks and relies on a functioning blockchain implementation as the benchmarking target. However, tools that can quickly bring up a blockchain network are ideal to cooperate with Caliper.
Hyperledger Caliper will produce reports containing a number of performance indicators, such as TPS (Transactions Per Second), transaction latency, resource utilisation etc. The intent is for Caliper results to be used by other Hyperledger projects as they build out their frameworks, and as a reference in supporting the choice of a blockchain implementation suitable for a user’s specific needs.
Clearly, every blockchain framework or platform proposed nowadays has its own unique strengths. In the light of such variety, we consider the absence of a common benchmarking tool as harmful. There are three main reasons why we think it’s necessary to have a general purpose benchmark tool:
- Lack of source code. There are some performance reports for different projects but since they do not provide the source code, it is hard to validate the evaluation and impossible to perform the same evaluation on different projects.
- There is no common definition of performance indicators (e.g. TPS, latency, resource utilization, etc.) The Hyperledger Performance & Scalability Working Group (PWSG) is the ideal place to define these. Hyperledger Caliper will be a good starting point and place to hold the implementation.
-
-
- A good example of having a well-defined performance indicator is TPS (Transaction per Second), as in any blockchain solution that supports smart contracts, it is a common agreement that the complexity of a smart contract is a key factor of TPS. Benchmarks that use different trial smart contracts, no matter how simple these smart contracts could be, are inadvertently affecting the results.
-
- There is no common accepted benchmark use cases for benchmarking. Users are always curious about use cases, with proper use cases, they help the users to understand the blockchain itself as well as the performance indicators. The use cases for benchmarking are also open to discussion and welcome for contribution.
Hyperledger Caliper is not intended to make judgments and will not publish benchmark results, but provide benchmark tools for users. Users should not claim the result is tested by Caliper unless the test environment is revealed. The Hyperledger Caliper project references the definitions, metrics and terminology as defined by the Hyperledger PWSG.
Technical details of Caliper
The key component in Hyperledger Caliper is the adaptation layer, which is introduced to integrate multiple blockchain solutions into the Caliper framework. An adaptor is implemented for each blockchain system under test (SUT), the adaptor is responsible for translation of Caliper NBIs into corresponding blockchain protocol. Caliper NBI is a set of common blockchain interfaces, which contains operations to interact with backend blockchain system, for example, to install smart contract, invoke contract, query state from the ledger, etc. The NBIs can be used for upstream applications to write tests for multiple blockchain systems. For more information, please see the documentation of Hyperledger Caliper.
For now, Hyperledger Fabric, Hyperledger Sawtooth and Hyperledger Iroha are in scope and we sincerely welcome contributions for integrations to other blockchain solutions.
Who will work on Caliper?
Collaboration with PSWG in the Hyperledger community is extremely important to the success of Hyperledger Caliper and we propose to have regular discussions about Caliper during PSWG meetings. Caliper is not going to limit the benchmark environment, but will leave it to user’s choice depending on their own use cases. Developers from Huawei, Hyperchain, Oracle, Bitwise, Soramitsu, IBM and the Budapest University of Technology and Economics are all contributing to the project. More are always welcome.
Getting started with Caliper
The hope is that Hyperledger Caliper provides the community with a functioning benchmark tool capable of running against many Hyperledger frameworks. With the help of the community, we will be putting continuous effort into the definition of performance indicators and benchmark use cases. The success of the project will depend on many users within the community using it as the benchmark tool.
You can get started contributing to Hyperledger Caliper on github (coming soon): https://github.com/hyperledger/caliper You can also join the Hyperledger Caliper community on the #caliper channel in Rocket.Chat and on the hyperledger-caliper mailing list.
As always, we encourage developers to join our efforts on other projects, via github, Rocket.Chat, the wiki or the mailing lists. You can also follow Hyperledger on Twitter or email us with any questions: info@hyperledger.org.