Bluzelle Bi-Weekly Sprint Update: Ensuring Network Stability Amid Upgrades- December 20, 2024
As we close out the two-week sprint ending December 20, 2024, we’ve encountered unexpected challenges but also made significant progress in maintaining and improving the Bluzelle mainnet. This sprint underscored the importance of balancing innovative development with robust maintenance, a core philosophy that ensures our network remains reliable and efficient.
Addressing Mainnet Disruptions
In the early days of this sprint, our mainnet faced a sudden disruption when several nodes were automatically stopped by AWS. This unforeseen incident became our team’s top priority, as it not only impacted ongoing operations but also had potential implications for the quality of our planned Cosmos upgrades.
Despite this disruption, the network’s redundant architecture ensured seamless performance. Our design’s inherent resilience allowed the mainnet to remain operational without compromising service quality, even as we addressed the underlying issues.
Our team focused not only on immediate resolutions but also on preventive measures to safeguard against similar occurrences in the future. This work highlights the dual nature of our responsibilities: maintaining a stable, high-performing mainnet while driving forward the technological advancements outlined in our roadmap.
Cosmos SDK Upgrades: Progress and Next Steps
The upgrade to the latest Cosmos SDK is critical for bringing the bridge to Osmosis online and ensuring our network remains at the cutting edge of blockchain technology.
Completed Work:
- New Auth Module: Successfully added to the Bluzelle SDK, providing enhanced functionality.
- Governance Module Overhaul: The SDK governance module was completed, requiring substantial rework to address new dependency requirements in the upgraded Cosmos SDK.
- Infrastructure Stability: Fixed an internal Jenkins issue causing SDK and Curium-CLI build failures in test suites.
Next Steps:
- Continue the upgrade process to Cosmos SDK 0.47 after resolving current mainnet issues.
Mainnet Issues: Progress and Priorities
Maintaining a robust mainnet is as crucial as developing new features. Here’s a summary of our work on resolving mainnet issues and ensuring long-term stability:
Completed Work:
- Node Analysis: Conducted a detailed review of which nodes were suspended and which remained operational.
- Gateway Sentry Node Fix: Resolved storage issues caused by an NVMe volume misconfiguration, allowing the node to sync and operate as expected.
- Infrastructure Automation: Refactored infrastructure-control code to automate sentry node setups on existing networks.
- Node Recovery:
- Restored three Big Dipper nodes that were previously suspended.
- Refactored Docker start scripts for increased reliability.
- Performed a health check of all active nodes, ensuring block creation and adequate storage capacity.
Upcoming Tasks:
- Resolve Connection Issues: Address IP-throttling preventing development machines from connecting to nodes.
- Node Resyncing and Redeployment: Reactivate suspended nodes, many of which require resyncing or full redeployment.
- Explorer Fixes: Repair Big Dipper explorers to properly communicate with client sentries.
- Validator Recovery: Bring back and unjail a suspended internal validator.
- Additional Resources: Set up an extra machine for developers to expedite issue resolution.
- Version Mismatch Investigation: Determine why one node is operating on an outdated version of Curium.
- AWS Elastic IP: Apply Elastic IPs to AWS nodes for increased stability.
- PingPub Node Restoration: Reactivate and stabilize suspended PingPub nodes.
Looking Ahead
This sprint highlighted the resilience of our network and the dedication of our team. While we continue advancing our Cosmos SDK upgrades, we remain deeply committed to maintaining and improving the Bluzelle mainnet. Our proactive approach ensures that unforeseen challenges strengthen our network and pave the way for long-term stability and success.
And keep on top of the latest with our public roadmap: https://trello.com/b/vqwJLgvz/bluzelle-public-roadmap
