Background
In the evolving landscape of blockchain technology and smart contracts development, one of the key components that plays a pivotal role is the "Keeper." A Keeper is a programmable bot or script that automatically performs tasks on a Blockchain Network, such as executing smart contract functions when certain conditions are met, with the result of maintaining efficiency and stability in decentralized systems. At TechOps Services, crafting custom keepers is part of our regular day-to-day activity, providing essential bridges between on-chain and off-chain worlds. So far we have built many keepers with different uses. Here are some examples:
- Governance Keeper - executing the latest governance passed proposal at the proper time when it has to be executed
- Poker Keeper - posting the latest Oracle prices
- CDP Keeper - actively manage open Collateral Debt Positions
- Starknet Teleport Keeper - the bridge between Ethereum and Starknet networks
In this post, we describe one instance of this process when developing these specialized components for a customer. Here’s our first-hand experience resolving the challenges we had to tackle with this project. Let’s dive in!
Initiating the Keeper Development
The development process started with ensuring all necessary environment variables were set. Since we planned on Running this keeper in Docker, it meant this would be a trivial process. One of the first critical steps was establishing a connection to Etherscan to fetch ABIs (Application Binary Interfaces) for the smart contract in question. Choosing a dynamic retrieval of ABI, rather than static storage, ensured that the keeper would always interact with the latest version of the smart contract, accommodating updates and changes without manual intervention, thereby enhancing flexibility and reducing maintenance effort, which streamlined our design and ensured efficiency right from the get-go.
Designing the Keeper Workflow
To begin we worked with smart contracts developers to clearly understand the desired flow keeper should follow. We usually end up with a very clear and easy to read flow diagram, we confirm it with all stakeholders and always keep it up to date. It would look a bit like this:
An example of what it ended up looking like
We need the keeper to operate in a continuous loop, constantly checking for new Ethereum block confirmations. After some monitoring and testing, the choice of a five-second sleep interval struck the right balance between effective monitoring and resource optimization, especially in the context of our Infura usage limitations. (Although, you will see in a moment that we opted for a more cost effective solution.)
Carefully managing our Infura request quota was essential for long-term operation considering resources and costs. We fine-tuned our polling intervals to maintain an effective balance between active monitoring and quota conservation, a critical aspect of a keeper's sustainable operation.
The Use of WebSockets with Infura
To maintain real-time updates, we considered using Websocket with Infura. This would have established a persistent connection for event consumption, reducing latency and network load, and enabling a more responsive and efficient operational model. However, it necessitated a shift to asynchronous programming, which we decided to reserve for a future iteration, considering our existing progress and time constraints.
Self-Hosted Ethereum Nodes as an alternative
Utilizing our own Ethereum nodes offered significant cost savings compared to relying solely on Infura, so we decided to go with our own nodes. This strategy enhanced our operational flexibility and scalability, allowing us to manage a higher volume of requests without the constraints of third-party quotas. The move towards self-hosting was a strategic decision to optimize for our use case with the added benefit of supporting a more decentralized blockchain infrastructure.
Embracing Python’s Multithreading Capabilities
A highlight of this project for us was leveraging the enhanced multi-threading capabilities in Python 3.11. This advancement brought us closer to the multi-threading efficiency of languages like Java, allowing each keeper to run in separate threads with improved performance. This makes sense because it can significantly improve the efficiency and responsiveness of the keeper. Multi-threading allows the keeper to perform multiple operations concurrently, such as monitoring different contracts or executing various tasks simultaneously. Enhanced multi-threading can lead to better resource utilization, faster execution, and overall improved performance of the keeper. As an example, this level of efficiency can be a huge win for DeFi when performing multiple market order liquidations.
Keeper Flow, Event Management and Exceptions in Logical Flow
Our keepers start by aggregating all relevant past events from the blockchain, processing each sequentially. This approach required an in-depth understanding of blockchain's unique operational environment, different from the more traditional architectures.
In traditional programming, exceptions are errors that occur during the execution of a program. Software Developers try to minimize exceptions because they often indicate something has gone wrong. However, in the context of keepers interacting with smart contracts, exceptions can be reinterpreted in a more nuanced way.
Intriguingly, smart contract exceptions in our keeper's flow weren't indicators of errors but signals of timing for function execution. For us, this novel use of exceptions exemplified our adaptive approach to utilizing blockchain's functionalities effectively.
Understanding the Nuances of Solidity Functions
Differentiating between static and dynamic functions in Solidity was vital. While static functions can't initiate transactions (and don’t cost any gas), dynamic functions can. This understanding was crucial for our interactions with the blockchain, ensuring cost-effectiveness and functional accuracy.
Rigorous Testing for Optimal Performance
Testing was a cornerstone of our development process. Tools like CodeCov helped us identify untested code segments, guiding our testing strategy. Mock calls were used to simulate blockchain interactions, ensuring the reliability and accuracy of our keepers.
Deployment and Monitoring Strategies
Leveraging Kubernetes, we were able to deploy our keeper service efficiently using GitHub Actions and managing them with Terraform for any future continuous deployments. However, the real challenge lies in monitoring our wallet activities on-chain. Having developed a monitoring Lambda function to track success (or failure) of all the transactions via Etherscan API, we ensured the operational integrity of this service by closely monitoring its behavior over time.
This venture into blockchain keeper development wasn't just about understanding blockchain intricacies; it was about integrating modern software engineering practices into the decentralized world. Through this post, we aim to provide some valuable insights for those looking to navigate similar paths in the blockchain and smart contract arena.
If you're considering developing your own keepers and need expert guidance or assistance, TechOps Services is here to help. Our experienced team is well-equipped, up to date and ready to support projects in this innovative and rapidly evolving field. Feel free to reach out to us for collaboration, advice, or services tailored to your blockchain development needs. info@techops.services
