Background

As blockchain technology and DeFi ecosystems grew, there was a significant demand for tools and platforms that could provide deep insights into blockchain data. Token Flow was developed to meet this demand by offering comprehensive tools, data, and insights to help users navigate the complex on-chain landscape of blockchains through their platform TokenFlow.live.

“Our mission is to give blockchain ecosystem participants the analytical tools, contextualized data and insights that they need to deeply understand and navigate the maze of on-chain activities.”

Starting out as a young startup, Token Flow was seeking to benefit from having an advanced team, for a fraction of the cost of a full-time one, that was ready to scale when they were. As the project matured, Token Flow has remained stable in its scope of work and the only change was to add a DevOps engineer internally to handle some of the more immediate tasks and to plan and delegate work to TechOps Services. This Hybrid option has proven to be an excellent choice for this relationship and remains this way to this day.

The Scope of Service

Although not limited to this list, the scope of work agreed upon included:

• Keeping Blockchain nodes operational and in sync with strict uptime requirements. TechOps does this by employing advanced 24/7 monitoring and providing immediate support whenever needed.
• Creating infrastructure quickly and effortlessly and providing fast response support. An example of how this has been done is by the Node deployment templates that TechOps has created in Terraform which allows any engineer to spin up new nodes in a matter of minutes, deploy them and have monitoring for these nodes in place right from the start.
• Cloud infrastructure required for optimal operation of all the Products. You can think of this as housekeeping for all the services used to run TokenFlow.live, like health monitoring, updates, cost and efficiency optimization, etc.
• Production, staging, testing, and development environments required for operating and developing websites, applications, GUIs, APIs, databases, etc.
• Managing and maintaining Domains and DNS
• Administrating Productivity and collaboration tools and software, including but not limited to Atlassian (JIRA, Confluence), Discord, Google Workspace.
• Creating and managing accounts and permissions for the above-mentioned Infrastructure, Environments, Domains and Tools;
• 24/7 monitoring for the above-mentioned Infrastructure, Environments, Domains and Tools;
• Continuous Integration and Continuous Deployment pipelines;
• Functional, performance and load testing of Token Flow products.
• Enforcing Security policies, assessments and training.
• Live support to all TokenFlow employees.

In further details; Token Flow's top priority is the availability and correct functioning of the services they offer to their customers, including their homepage. These services rely heavily on blockchain nodes that collect data and run scans. An example of a task we often see related to this is ensuring sufficient disk space, CPU and memory for blockchain data processing without overpaying for unused resources.

Deploying new nodes and especially nodes of newer blockchains has also taken up much of the work time completed for Token Flow. During the deployment of Optimism nodes, a lot had to be learnt on how to work with them in the early stages which ended up getting easier as updates were released that improved their stability.

When it comes to more traditional infrastructure work, TechOps Services has helped build K8s clusters, security measures and control plans for Token Flow. Initially, our engagement focused on leveraging Amazon Web Services (AWS) Elastic Container Service (ECS) for hosting Token Flow's applications. This task, while foundational, set the stage for more intricate and advanced projects. The turning point came with the introduction of an internal DevOps engineer at Token Flow, which transitioned our role towards more specialized Kubernetes cluster management.

Our work encompassed early-stage tooling preparation, aiding in the creation of a robust Kubernetes infrastructure. This not only involved basic setup, but extended to intricate aspects like logging, security, and aiding developers in deploying their applications efficiently.

Another nice solution we created was the ability to use templates for cloning some of Token Flow’s Ethereum nodes. For running one time processes for a section of the blockchain. Token Flow sometimes needs to create short-lived nodes, and this approach using templates saves them money since they avoid having idle resources, they can quickly spin up a new node, use it and delete it when they are done. This also improves the whole process for them as they don’t need to ask us for new nodes to be deployed.

One of the things we’re really good at at TechOps is creating advanced alerting systems with integrations into our clients messaging tools like Discord and Slack. On top of that we created the ability for anyone on the channels to be able to create support tickets or request status updates on tickets by using keywords like “all my tickets” or “TECH-999” (ticket key).

The correct functioning of these communication services also falls under the responsibilities of TechOps Services. We handle the access, verification process and the use of Spam filters.

Insights Gained

Since stability and heavy server usage is at the core of Token Flow’s engineering, the efficiency and cost of these resources is always a topic of conversation.

Over time we have researched many alternatives and competing products to use that could help cut costs or make operations easier to run. For instance, switching to serverless wherever possible or considering using a 3rd party node service. In this later case, it was concluded that the reliability of the nodes needed would be better if we ran our own nodes and with so much usage, the costs would also be in our favour.

The proprietary services that Token Flow creates push us to help create cutting edge tooling to accomplish their requirements. A lot of R&D has gone into this and has been fun to work on.

Even the Kubernetes design from Token Flow’s internal team has been next level, for which we processed deploying the designs and pushed to meet their requirements.

Conclusion

Our collaboration with Token Flow exemplifies how we support not just the operation of services, but also the underlying technologies that power them. From managing Ethereum nodes on Kubernetes to offering scalable solutions, TechOps work has significant implications. It highlights our commitment to providing services that cater to large-scale, high-impact applications, distinguishing us from competitors.

Being quick to reply and to get to work, we've managed to handle all requests, even if it required doing R&D at our own cost.

Customers like Token Flow are the reason we focus on Web3. Building next generation architecture and finding the best ways to support it gives us an immense amount of satisfaction that we hope to share with more projects in Web3.