Virtual Kubernetes API Compatibility

Objective

This documentation provides you with details about the F5® Distributed Cloud Services Virtual Kubernetes (vK8s) APIs supported in this release. For more information, see Virtual Kubernetes (vk8s).

Distributed Cloud Services vK8s APIs

Distributed Cloud Services supports a Kubernetes compatible API for centralized orchestration of applications across a fleet of sites (Customer Edge sites or Regional Edge sites). This API is “Kubernetes compatible” because not all Kubernetes APIs or resources are supported. However, for the APIs that are supported, they are fully compatible. Distributed Cloud Services has implemented a distributed control plane within its global infrastructure to manage scheduling and scaling of applications across multiple (tens to hundreds of thousands of) sites, where each site is also a system-managed physical K8s cluster.

Note: The supported Kubernetes version is v1.23.

Workload APIs

Deployment v1 APPs

A Deployment provides declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets or to remove existing Deployments and adopt all their resources with new Deployments.

Write Operations

Create POST /apis/apps/v1/namespaces/{namespace}/deployments Patch PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name} Replace PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name} Delete DELETE /apis/apps/v1/namespaces/{namespace}/deployments/{name} Delete Collection DELETE /apis/apps/v1/namespaces/{namespace}/deployments

Read Operations

Read GET /apis/apps/v1/namespaces/{namespace}/deployments/{name} List GET /apis/apps/v1/namespaces/{namespace}/deployments Watch GET /apis/apps/v1/watch/namespaces/{namespace}/deployments/{name} Watch List GET /apis/apps/v1/watch/namespaces/{namespace}/deployments

Status Operations

Read Status GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status

Misc Operations

Read Scale GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale Replace Scale PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale Patch Scale PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale

Job v1 Batch

A Job creates one or more Pods and ensures that a specified number of them successfully terminate. As pods successfully complete, the Job tracks the successful completions. When a specified number of successful completions is reached, the task (in other words, Job) is complete. Deleting a Job will clean up the Pods it created. A simple case is to create one Job object in order to reliably run one Pod to completion. The Job object will start a new Pod if the first Pod fails or is deleted (for example, due to a node hardware failure or a node reboot).

Write Operations

Create POST /apis/batch/v1/namespaces/{namespace}/jobs Patch PATCH /apis/batch/v1/namespaces/{namespace}/jobs/{name} Replace PUT /apis/batch/v1/namespaces/{namespace}/jobs/{name} Delete DELETE /apis/batch/v1/namespaces/{namespace}/jobs/{name} Delete Collection DELETE /apis/batch/v1/namespaces/{namespace}/jobs

Read Operations

Read GET /apis/batch/v1/namespaces/{namespace}/jobs/{name} List GET /apis/batch/v1/namespaces/{namespace}/jobs Watch GET /apis/batch/v1/watch/namespaces/{namespace}/jobs/{name} Watch List GET /apis/batch/v1/watch/namespaces/{namespace}/jobs

Status Operations

Read Status GET /apis/batch/v1/namespaces/{namespace}/jobs/{name}/status

CronJob v1beta1 Batch

A cronjob is a scheduled job and written in cron format and is useful for running periodic tasks.

Write Operations

Create POST /apis/batch/v1beta1/namespaces/{namespace}/cronjobs Patch PATCH /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name} Replace PUT /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name} Delete DELETE /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name} Delete Collection DELETE /apis/batch/v1beta1/namespaces/{namespace}/cronjobs

Read Operations

Read GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name} List GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs List All Namespaces GET /apis/batch/v1beta1/cronjobs Watch GET /apis/batch/v1beta1/watch/namespaces/{namespace}/cronjobs/{name} Watch List GET /apis/batch/v1beta1/watch/namespaces/{namespace}/cronjobs Watch List All Namespaces GET /apis/batch/v1beta1/watch/cronjobs

Status Operations

Patch Status PATCH /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status Read Status GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status Replace Status PUT /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status

Pod v1 Core

A Pod (as in a pod of whales or pea pod) is a group of one or more containers (such as Docker containers), with shared storage/network, and a specification for how to run the containers. A Pod’s contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific “logical host.” It contains one or more application containers which are relatively tightly coupled. In a pre-container environment, applications executed on the same physical or virtual machine are executed on the same logical host.

Read Operations

Read GET /api/v1/namespaces/{namespace}/pods/{name} List GET /api/v1/namespaces/{namespace}/pods Watch GET /api/v1/watch/namespaces/{namespace}/pods/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/pods

Status Operations

Read Status GET /api/v1/namespaces/{namespace}/pods/{name}/status

Misc Operations

Read Log GET /api/v1/namespaces/{namespace}/pods/{name}/log

Proxy Operations

Create Connect Portforward POST /api/v1/namespaces/{namespace}/pods/{name}/portforward Get Connect Portforward GET /api/v1/namespaces/{namespace}/pods/{name}/portforward Create Connect Exec POST /api/v1/namespaces/{namespace}/pods/{name}/exec Get Connect Exec GET /api/v1/namespaces/{namespace}/pods/{name}/exec

ReplicaSet v1 Apps

A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

Read Operations

Read GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name} List GET /apis/apps/v1/namespaces/{namespace}/replicasets Watch GET /apis/apps/v1/watch/namespaces/{namespace}/replicasets/{name} Watch List GET /apis/apps/v1/watch/namespaces/{namespace}/replicasets

Status Operations

Read Status GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/status

Misc Operations

Read Scale GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/scale

StatefulSet v1 Apps

StatefulSet is the workload API object used to manage stateful applications. This workload API object also manages the deployment and scaling of a set of Pods, and provides guarantees about the ordering and uniqueness of these Pods. Like a Deployment, a StatefulSet manages Pods that are based on an identical container specification. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same specification but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

Write Operations

Create POST /apis/apps/v1/namespaces/{namespace}/statefulsets Patch PATCH /apis/apps/v1/namespaces/{namespace}/statefulsets/{name} Replace PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name} Delete DELETE /apis/apps/v1/namespaces/{namespace}/statefulsets/{name} Delete Collection DELETE /apis/apps/v1/namespaces/{namespace}/statefulsets

Read Operations

Read GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name} List GET /apis/apps/v1/namespaces/{namespace}/statefulsets Watch GET /apis/apps/v1/watch/namespaces/{namespace}/statefulsets/{name} Watch List GET /apis/apps/v1/watch/namespaces/{namespace}/statefulsets

Status Operations

Read Status GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/status

Misc Operations

Read Scale GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale Replace Scale PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale Patch Scale PATCH /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale

Service APIs

Service v1 Core

Service is an abstract way to expose an application running on a set of Pods as a network service. With Kubernetes, you do not need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.

Write Operations

Create POST /api/v1/namespaces/{namespace}/services Patch PATCH /api/v1/namespaces/{namespace}/services/{name} Replace PUT /api/v1/namespaces/{namespace}/services/{name} Delete DELETE /api/v1/namespaces/{namespace}/services/{name}

Read Operations

Read GET /api/v1/namespaces/{namespace}/services/{name} List GET /api/v1/namespaces/{namespace}/services Watch GET /api/v1/watch/namespaces/{namespace}/services/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/services

Status Operations

Read Status GET /api/v1/namespaces/{namespace}/services/{name}/status

Endpoints v1 Core

Endpoints track the IP addresses of the objects the service sends traffic to. When a service selector matches a Pod label, that IP address is added to your endpoints.

Read Operations

Read GET /api/v1/namespaces/{namespace}/endpoints/{name} List GET /api/v1/namespaces/{namespace}/endpoints Watch GET /api/v1/watch/namespaces/{namespace}/endpoints/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/endpoints

Config and Storage APIs

Secret v1 Core

Kubernetes secret objects let you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image. See Secrets design document for more information.

Write Operations

Create POST /api/v1/namespaces/{namespace}/secrets Patch PATCH /api/v1/namespaces/{namespace}/secrets/{name} Replace PUT /api/v1/namespaces/{namespace}/secrets/{name} Delete DELETE /api/v1/namespaces/{namespace}/secrets/{name} Delete Collection DELETE /api/v1/namespaces/{namespace}/secrets

Read Operations

Read GET /api/v1/namespaces/{namespace}/secrets/{name} List GET /api/v1/namespaces/{namespace}/secrets Watch GET /api/v1/watch/namespaces/{namespace}/secrets/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/secrets

ConfigMap v1 Core

ConfigMap enables you to decouple configuration artifacts from image content to keep containerized applications portable. The information below provides a series of usage examples demonstrating how to create a ConfigMap and configure Pods using data stored in that ConfigMap.

Write Operations

Create POST /api/v1/namespaces/{namespace}/configmaps Patch PATCH /api/v1/namespaces/{namespace}/configmaps/{name} Replace PUT /api/v1/namespaces/{namespace}/configmaps/{name} Delete DELETE /api/v1/namespaces/{namespace}/configmaps/{name} Delete Collection DELETE /api/v1/namespaces/{namespace}/configmaps

Read Operations

Read GET /api/v1/namespaces/{namespace}/configmaps/{name} List GET /api/v1/namespaces/{namespace}/configmaps Watch GET /api/v1/watch/namespaces/{namespace}/configmaps/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/configmaps

PersistentVolumeClaim v1 Core

Managing storage is a distinct problem from managing compute. The PersistentVolume subsystem provides an API for users and administrators that abstracts the details of how storage is provided from how it is consumed. To do this, K8s introduces two new API resources: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster in the same manner as a node is a cluster resource. PVs are volume plugins, like Volumes, but have a lifecycle independent of any individual Pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (like CPU and memory). Claims can request specific size and access modes (for example, they can be mounted once read/write or many times read-only).

Write Operations

Create POST /api/v1/namespaces/{namespace}/persistentvolumeclaims Patch PATCH /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name} Replace PUT /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name} Delete DELETE /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name} Delete Collection DELETE /api/v1/namespaces/{namespace}/persistentvolumeclaims

Read Operations

Read GET /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name} List GET /api/v1/namespaces/{namespace}/persistentvolumeclaims Watch GET /api/v1/watch/namespaces/{namespace}/persistentvolumeclaims/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/persistentvolumeclaims

Cluster APIs

Node v1 Core

A node is a worker machine in Kubernetes, previously known as a minion. A node may be a virtual machine (VM) or a physical machine, depending on the cluster. Each node contains the services necessary to run pods and is managed by the master components. The services on a node include the container runtime, kubelet, and kube-proxy.

Read Operations

Read GET /api/v1/nodes/{name} List GET /api/v1/nodes Watch GET /api/v1/watch/nodes/{name} Watch List GET /api/v1/watch/nodes

Status Operations

Read Status GET /api/v1/nodes/{name}/status

Namespace v1 Core

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.

Read Operations

Read GET /api/v1/namespaces/{name} List GET /api/v1/namespaces Watch GET /api/v1/watch/namespaces/{name} Watch List GET /api/v1/watch/namespaces

PersistentVolume v1 Core

Managing storage is a distinct problem from managing compute. The PersistentVolume subsystem provides an API for users and administrators that abstracts the details of how storage is provided from how it is consumed. To do this, K8s introduce two new API resources: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster in the same manner as a node is a cluster resource. PVs are volume plugins, like Volumes, but have a lifecycle independent of any individual Pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (like CPU and memory). Claims can request specific size and access modes (for example, they can be mounted once read/write or many times read-only).

Read Operations

Read GET /api/v1/persistentvolumes/{name} List GET /api/v1/persistentvolumes Watch GET /api/v1/watch/persistentvolumes/{name} Watch List GET /api/v1/watch/persistentvolumes

Status Operations Read Status GET /api/v1/persistentvolumes/{name}/status

Metadata APIs

Event v1 Core

Read GET /api/v1/namespaces/{namespace}/events/{name} List GET /api/v1/namespaces/{namespace}/events List All Namespaces GET /api/v1/events Watch GET /api/v1/watch/namespaces/{namespace}/events/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/events Watch List All Namespaces GET /api/v1/watch/events

Access Control APIs

ServiceAccount v1 Core

Write Operations

Create POST /api/v1/namespaces/{namespace}/serviceaccounts Patch PATCH /api/v1/namespaces/{namespace}/serviceaccounts/{name} Replace PUT /api/v1/namespaces/{namespace}/serviceaccounts/{name} Delete DELETE /api/v1/namespaces/{namespace}/serviceaccounts/{name} Delete Collection DELETE /api/v1/namespaces/{namespace}/serviceaccounts

Read Operations

Read GET /api/v1/namespaces/{namespace}/serviceaccounts/{name} List GET /api/v1/namespaces/{namespace}/serviceaccounts List All Namespaces GET /api/v1/serviceaccounts Watch GET /api/v1/watch/namespaces/{namespace}/serviceaccounts/{name} Watch List GET /api/v1/watch/namespaces/{namespace}/serviceaccounts Watch List All Namespaces GET /api/v1/watch/serviceaccounts

Resource Management for Distributed Cloud Services vK8s

The Distributed Cloud Services vK8s resources can be managed using the ves.io/ annotations in the deployments. These annotations are supported for the deployments on both network cloud and edge cloud.

The information below describes the supported annotations to manage Distributed Cloud Services vK8s resources.

• ves.io/sites: namespace/name: The namespace is the site's namespace and name is the site's name. In the case of multiple sites, the ves.io/sites: namespace/name,namespace/name format can be used. In the case of Customer Edge (CE) sites, the system string should be specified as a namespace along with the CE name. In the case of Regional Edge (RE) sites, the ves-io-system string should be specified as the namespace.

The following is an example annotation for launching vK8s resources to CE sites:

    "annotations": {
      "ves.io/sites": "system/ce01, system/ce02",
    }

Note: When both ves.io/sites and ves.io/virtual-sites annotations are configured, the ves.io/sites annotation gets priority. This can be used to manage all vK8s resources.

• ves.io/virtual-sites: namespace/name: The namespace is the virtual site's namespace and name is the virtual site's name. In the case of multiple virtual sites, the ves.io/virtual-sites: namespace/name,namespace/name format can be used.

• The following can be managed using the annotation:

  • Deployments
  • Jobs
  • StatefulSets
  • Services
  • PVCs
  • ConfigMaps
  • Secrets

This annotation selects the sites where the K8s object is instantiated. In case you do not specify this annotation, then the sites are selected based on the virtual sites reference in the vK8s object.

Note: In case of using the virtual sites in the shared namespace of the ves.io tenant, the ves-io-shared string should be specified as the namespace.

• ves.io/workload-flavor : workload-flavor-name: The workload-flavor-name can be ves-io-tiny, ves-io-medium, or ves-io-large. The workload flavor maps to CPU and memory requests and limits for a pod. Each container in a Pod consumes compute resources - the CPU and memory (RAM). All containers launched using the vK8s APIs have limits on the compute resources. Memory is measured in units of MiB (mebibyte), power of 2. One MiB is 1,048,576 bytes. CPU is measured in units of vCPU. Each vCPU is a thread on a CPU core.

The following Workload Flavor types are supported:

The following can be managed using the annotation:

• Deployments
• Jobs
• CronJobs
• DaemonSets
• StatefulSets

Custom Workload Flavors are also supported. Workload Flavors can be chosen by adding the annotation ves.io/workload-flavor with the value set to the flavor name.

This example provides a workload-flavor with ves-io-medium:

{ves.io/workload-flavor : "ves-io-medium"}

By default, if no ves.io/workload-flavor annotation is present, then vK8s uses the compute resources from the ves-io-tiny flavor.

This annotation determines the CPU and memory requests and limits for pods created by the above controllers and applies to all containers within the Pod. Individual containers within the Pod can be overridden using the ves.io/workload-flavor-container-name: workload-flavor-name annotation, where container-name is the name of the container in the Pod specification. If no annotation is specified, the default flavor used for all containers in the Pod is ves-io-tiny.

• ves.io/proxy-type:proxy-type-name: The proxy-type-name can be HTTP_PROXY, TCP_PROXY, or TCP_PROXY_WITH_SNI. This applies to Services and determines the proxy type used for the internal virtual host associated with the K8s service. If no annotation is specified, the TCP_PROXY value is used as the default proxy.

• ves.io/http2-enable: true|false: This applies to services and determines whether HTTP2 upstream connection proxying is enabled for this service. If no annotation is specified, the default value is false, indicating that the HTTP2 connection is not enabled for the service.

In case the pods are exceeding the set resource limits, and you specified container-level resource limits in the Pod specification, the container-level resource limits are overwritten by that of the workload-flavor annotation.

Note: The annotation ves.io/workload-flavor only takes effect on the Distributed Cloud Services global network. The container-level resource limits in the Pod specification are honored in deployments on your cloud (public or private) or your edge F5® Distributed Cloud App Stack node.

In case of network cloud Regional Edge (RE) sites, a vK8s CPU and memory resource limit is supported per RE site. If that is exceeded, no new Pods can be instantiated on the RE site.

Concepts

• Virtual Kubernetes Concepts
• vK8s APIs
• Kubernetes Concepts

Kubernetes References

• Cluster Architecture
• Namespaces
• Deployments
• Jobs
• Pods
• ReplicaSet
• Persistent Volumes
• Service
• Secrets
• Nodes