Building a Kubernetes Edge (Ingress) Control Plane for Envoy v2

Building a Kubernetes Edge (Ingress) Control Plane for Envoy v2

Kubernetes has become the de facto runtime for container-based microservice applications, but this orchestration framework alone does not provide all of the infrastructure necessary for running a distributed system. Microservices typically communicate through Layer 7 protocols such as HTTP, gRPC, or WebSockets, and therefore having the ability to make routing decisions, manipulate protocol metadata, and observe at this layer is vital. However, traditional load balancers and edge proxies have predominantly focused on L3/4 traffic. This is where the Envoy Proxycomes into play.

Envoy proxy was designed as a universal data plane from the ground-up by the Lyft Engineering team for today’s distributed, L7-centric world, with broad support for L7 protocols, a real-time API for managing its configuration, first-class observability, and high performance within a small memory footprint. However, Envoy’s vast feature set and flexibility of operation also makes its configuration highly complicated – this is evident from looking at its rich but verbose control plane syntax.

With the open source Ambassador API Gateway, we wanted to tackle the challenge of creating a new control plane that focuses on the use case of deploying Envoy as an forward-facing edge proxy within a Kubernetes cluster, in a way that is idiomatic to Kubernetes operators. In this article, we’ll walk through two major iterations of the Ambassador design, and how we integrated Ambassador with Kubernetes.