Today I will demonstrate how to use OpenResty Edge as a powerful ingress controller for Kubernetes Clusters. That is, managing traffic going to the backend applications running inside the Kubernetes containers.

Embeded image

In this tutorial, we will create a Kubernetes upstream in an Edge application. The Edge gateway servers may run inside or outside the Kubernetes Clusters. The Edge Admin servers continuously monitor the Kubernetes Clusters via its API servers. And it updates the upstream server list automatically according to the online and offline events of the Kubernetes Nodes (or containers).

Embeded image

How to create and use Kubernetes upstream

Now let’s go to a web console of OpenResty Edge. This is our sample deployment of the console. Every user has their own local deployment.

Screenshot

First, we need a gateway server in a dedicated gateway partition that can connect to the Kubernetes services.

Screenshot

Let’s go to the Gateway Partitions page.

Screenshot

We have already created a gateway partition, kubernetes-test-partition,

Screenshot

which contains a gateway cluster, kubernetes-test-cluster.

Screenshot

We need a separate gateway partition here because not all our gateway servers can connect to our Kubernetes services.

Go to the kubernetes-test-cluster cluster.

Screenshot

There is one gateway server already defined in this gateway cluster. We need to make sure that this gateway server can reach our Kubernetes Cluster.

Screenshot

Create Kubernetes Cluster

Before we can create a new Kubernetes upstream in an Edge application, we first need to register our Kubernetes Cluster globally.

Goto the Kubernetes page.

Screenshot

Click this button to add a new Kubernetes Cluster.

Screenshot

The OpenResty Edge Admin needs to establish connections to Kubernetes' HTTPS API servers for any new notifications.

Enter the host name or IP address of our Kubernetes API server.

Screenshot

Then enter its port number.

Screenshot

Next we enter the name of the kubernetes Cluster

Screenshot

We disable the SSL certificate verification. This is because our Kubernetes API server’s certificate is self-signed.

Screenshot

We also need a token with enough privileges to access the Kubernetes API server.

Screenshot

Click on the link “How to generate this token”.

Screenshot

This window explains how to generate one from your own Kubernetes deployment. Now let me demonstrate the process.

Close this window.

Screenshot

We need to prepare a token.yml specification file. One sample of this file is as follows. It is for creating a service account with reading access to the

Screenshot

namespaces,

Screenshot

services,

Screenshot

endpoints, and

Screenshot

Pods objects.

Screenshot

Then we create the service account using the following command.

kubectl apply -f /root/token.yml

Screenshot

Finally, we use the following command to get the token of the service account.

Screenshot

The text after “token:” is the token we need.

Screenshot

Now we have a token to access the Kubernetes API server.

Paste the API token we just created.

Screenshot

Click the Create button.

Screenshot

Create Kubernetes upstream

Now that we have the Kubernetes Cluster registered globally. It’s time to create an upstream in an Edge application to make use of this cluster.

Screenshot

Go to the application list page.

Screenshot

We already prepared an Edge application named www.kubernetes-edge-test.com.

Screenshot

Our application is mapped to the gateway partition we showed earlier.

Screenshot

Enter the application.

Screenshot

Go to the Upstreams page.

Screenshot

Click New Kubernetes Upstream.

Screenshot

Enter the name of the Kubernetes upstream, “my kubernetes backend”.

Screenshot

Select a Kubernetes Cluster as the target.

Screenshot

We select the one we just created at the beginning of this tutorial.

Screenshot

Now select the target Kubernetes namespace.

Screenshot

Choose the namespace named “default”. You may have a different namespace though.

Screenshot

Select a Kubernetes service from a list.

Screenshot

We select the “test-hello” service.

Screenshot

And also the service port.

Screenshot

It’s port 80 in our case.

Screenshot

Save the new upstream.

Screenshot

Now we have successfully created a new Kubernetes upstream.

Screenshot

The upstream server addresses have already appeared.

Screenshot

They are Kubernetes Nodes or containers automatically synchronized from the Kubernetes Cluster.

Screenshot

Create a page rule that uses Kubernetes upstream

Just as with ordinary upstreams, we still need to create a new Page Rule to make use of this new Kubernetes upstream.

Go to the Page Rules page.

Screenshot

Click the “New Rule” button.

Screenshot

Enable the Proxy action.

Screenshot

Click on the “Proxy to upstream” drop-down list.

Screenshot

Select the Kubernetes upstream, “my kubernetes backend”, we just created.

Screenshot

Save this page rule.

Screenshot

We need to make a new configuration release for this Edge application, as always.

Screenshot

Click this button.

Screenshot

Ship it!

Screenshot

It is fully synchronized.

Screenshot

Now the new page rule has been pushed to all the gateway clusters and servers.

Embeded video

Embeded video

Embeded video

Our configuration changes do NOT require server reload, restart, or binary upgrade. So it’s very efficient and scalable.

Embeded image

Test

Next, let’s find the Edge gateway server in the right partition to actually test it.

Go to the Gateway Clusters page.

Screenshot

Find the right gateway server here.

Screenshot

Its public IP address ends with .196.

Screenshot

Copy this IP address so that we can use it on the command line.

On the terminal, let’s try to access the Kubernetes service through this gateway server.

Screenshot

Note that we use the gateway server IP address we just copied.

Screenshot

Run this command.

Screenshot

Now we see that the access to the service was successful. In the rest of the tutorial, we’d modify the configuration of the Kubernetes service.

Screenshot

We will increase the number of the Kubernetes Nodes to 3 on the fly.

kubectl scale --replicas=3 deployment test-hello

Screenshot

It was scaled successfully.

Screenshot

Let’s check the new Nodes.

kubectl get pods -o wide

Screenshot

There are indeed three Kubernetes Nodes now.

Screenshot

The Kubernetes upstream should be automatically updated by Edge Admin to reflect this change.

Let’s go back to our Edge application page to check it out.

Screenshot

Screenshot

Return to the Upstreams page.

Screenshot

Refresh the Upstreams page to avoid any stale data.

Screenshot

Yay! The Kubernetes upstream indeed has 3 servers now!

Screenshot

Similarly, if there are fewer Kubernetes Nodes, Edge Admin will also automatically update the upstream. This is what I’d like to cover today.

Screenshot

What is OpenResty Edge

OpenResty Edge is our all-in-one gateway software for microservices and distributed traffic architectures. It combines traffic management, private CDN construction, API gateway, security, and more to help you easily build, manage, and protect modern applications. OpenResty Edge delivers industry-leading performance and scalability to meet the demanding needs of high concurrency, high load scenarios. It supports scheduling containerized application traffic such as K8s and manages massive domains, making it easy to meet the needs of large websites and complex applications.

If you like this tutorial, please subscribe to this blog site and/or our YouTube channel. Thank you!

About The Author

Yichun Zhang (Github handle: agentzh), is the original creator of the OpenResty® open-source project and the CEO of OpenResty Inc..

Yichun is one of the earliest advocates and leaders of “open-source technology”. He worked at many internationally renowned tech companies, such as Cloudflare, Yahoo!. He is a pioneer of “edge computing”, “dynamic tracing” and “machine coding”, with over 22 years of programming and 16 years of open source experience. Yichun is well-known in the open-source space as the project leader of OpenResty®, adopted by more than 40 million global website domains.

OpenResty Inc., the enterprise software start-up founded by Yichun in 2017, has customers from some of the biggest companies in the world. Its flagship product, OpenResty XRay, is a non-invasive profiling and troubleshooting tool that significantly enhances and utilizes dynamic tracing technology. And its OpenResty Edge product is a powerful distributed traffic management and private CDN software product.

As an avid open-source contributor, Yichun has contributed more than a million lines of code to numerous open-source projects, including Linux kernel, Nginx, LuaJIT, GDB, SystemTap, LLVM, Perl, etc. He has also authored more than 60 open-source software libraries.