While Kubernetes is still the de-facto standard in container orchestration, a lot of people who work on it would agree that it is in fact quite hard and in need of a lot of polishing.
Kubernetes is the best way to orchestrate containers, period. For a quick recap on containers and microservices, you might want to read our previous post on how most enterprise organizations are switching from monoliths to microservices, or this post on why containers are the best way to deploy software.
According to the CNCF, Kubernetes has a massive 92% market share of container orchestration tools for businesses. While a lot of people still use a Docker engine to create and share containers, most organizations prefer Kubernetes (though it’s hard) over Docker Swarm for orchestration requirements.
The part where it gets hard, however, is when you need to run multiple clusters across hybrid environments that often include multiple public and private clouds, as well as on-premise facilities and edge locations.
TikTok to the rescue
Reminiscent of how Google used its experience running large-scale distribution systems for over a decade to build Kubernetes, ByteDance, the organization behind TikTok has used its experience working with KubeFed for multicluster management to bring us KubeAdmiral. Now while Kubernetes was originally designed as an internal large-scale cluster management system for Google before it was released to the world, it wasn’t intended for multi-cluster management, which is where KubeFed comes into the picture.
KubeFed is an Open Source project that allows for multi-cluster management in Kubernetes and isn’t without its limitations. one of which is the fact that KubeFed’s RSP (ReplicaSchedulingPreference) only supports static cluster weights and stateless workloads. This makes multi-cluster scheduling of diverse workloads quite problematic.
Getting back to our main story, Like most organizations that switch to Kubernetes to orchestrate containers, ByteDance witnessed an unprecedented expansion of its work environment which had diversified from stateless microservices to include both stateful and stateless services, as well as offline, storage, and machine learning workloads.
With frequent service disruptions that followed reschedulings, ByteDance decided it was time to build its own cluster federation system based on KubeFed v2. Born out of the need for a more robust and scalable cluster federation system to deal with these diverse workloads, KubeAdmiral is named after its efficiency in handling multiple clusters across diverse environments (like an admiral commanding a fleet of ships).
KubeAdmiral
Compared to the community version of Kubernetes which is limited to 5,000 nodes per cluster, Dancebyte currently uses KubeAdmiral to manage over 100,000 microservices with more than 10 million pods running on multiple federated Kubernetes clusters. That is in addition to about 30,000 upgrades per day with a 95-98% stable deployment rate. Many would agree that orchestrating 10 million pods on Kubernetes is in itself a feat of software engineering.
This is undoubtedly accredited to the enhanced multi-cluster orchestration and scheduling capabilities of KubeAdmiral which is powered by the scheduler which is a core component of KubeAdmiral and also referred to as its “brain.” The scheduler is responsible for ensuring the accurate and timely placement of workloads in member clusters.
Another area in which KubeAdmiral outshines its predecessor KubeFed is in the inclusion of native Kubernetes APIs. While KubeFed is an open-source project that was built specifically to enable multi-cluster orchestration on Kubernetes, it is not compatible with native Kubernetes APIs. That’s right, while KubeFed requires users to use federated APIs that increase the costs associated with onboarding, Kubeadmiral’s federate-controller is compatible with Kubernetes APIs. This allows users to seamlessly transition to multi-cluster environments with lower costs and without the hassle associated with using incompatible “federated” APIs.
Other features of KubeAdmiral include automatic propagation of dependencies, override policies, scheduling plugins, status aggregation of member cluster resources, scalability, stability, and most importantly, an enhanced user experience.
Gleaming the Kube
In conclusion, while Kubernetes is still the de-facto standard in container orchestration, a lot of people who work on it would agree that it is in fact quite hard and in need of a lot of polishing. That being said, however, it’s nice to see organizations like ByteDance that have acquired a treasure trove of valuable practical experiences using Kubernetes at scale, giving back to the open-source community. While books and videos can teach you a lot, there’s nothing that can substitute for true battle-hardened experience using Kubernetes in production and orchestrating millions of pods across dozens of clusters in a live environment.
The fact that they would package that experience and share it with the world says a lot about ByteDance, as well as the Kubernetes and open-source communities around the world. Other open-source projects from ByteDance include Byconity, KubeWharf, CloudWeGo, ByteIR, and RsPack.
In case you missed:
- Introducing Nvidia’s L4 GPU profiles in the cloud
- In the realm of Shadow, Zombie, and Rogue APIs
- Bacteria That Can Generate Electricity from Wastewater!
- This AI Camera Takes Pictures Without a Lens!
- Meet Einstein Copilot, Salesforce’s new conversational AI
- AI can already read your mind
- Could Quantum Gaming Become a Thing?
- NavIC, India’s Aatmanirbhar answer to GPS
- This Device Literally Makes You a Telepath to Computers
- Are Drones the New Fireworks?
