What is Kubectl
If you’re reading this article, you’re probably familiar with Kubernetes and want to interact with a Kubernetes cluster. Regardless of how you provisioned the cluster, kubectl is the standard command line used to communicate with it. This article assumes that you have a basic understanding of Kubernetes and the kubectl command.
Kubectl offers three techniques.
The “
- Imperative commands” technique directly tells Kubernetes which operation to perform on which
- Imperative Object Configuration” technique is identical to the first, except that it works on manifest files instead of objects directly, for example kubectl create -f manifest.yaml. The
- Declarative Object Configuration” technique again takes manifest files as input, but uses “upsert” logic and creates objects if they do not exist, or updates existing objects that are different from the specifications in the input manifest files. The command for this is kubectl apply -f manifest.yaml.
objects, for example: kubectl create pod or kubectl delete service. The “
“
Note that there is no true declarative approach yet because kubectl cannot delete objects automatically. The -prune option for the kubectl apply command allows you to achieve a fully declarative approach, but this option is currently in alpha at the time of writing and is therefore not considered suitable for general use. Before diving into the code, it’s important to keep a few things in mind.
Object notation is usually in the form of an object type, followed by a forward slash, followed by the object name. For example, to go to the “mypod” pod, the notation will be “pods/mypod”. Some commands accept different notations (for example, “kubectl get pod mypod”), so you might see different notations used throughout this article.
Most important Kubectl commands
This is the TL;DR section of this article, a quick access section to remind you of the most important commands. No explanations are given, but the commands are explained later in the article.
To configure the current contexts in the
kubeconfig file
: $ kubectl config get-contexts To change the context: $ kubectl config
use-context
minikube To get the name of
the containers in a running pod:
$ kubectl get pod MYPOD -o ‘jsonpath={.spec.containers[*].name}’
To get the value of a secret (if you have the base64 command available
): $ kubectl -n mynamespace get secret MYSECRET -o ‘jsonpath={.data. DB_PASSWORD}’ | base64 -d SuperSecretPassword
If you don’t have the base64 command available, you can use the go:
$ kubectl -n mynamespace get secret MYSECRET -o ‘go-template={{.data. DB_PASSWORD | base64decode}}’
An example of a filter in jsonpath
: $ kubectl get pod nginx -o ‘jsonpath={.spec.containers[?( @.name==”nginx”)].image}’ nginx:1.9.1 To create a secret from
your current Docker credentials to extract images from a private registry
: $ kubectl create secret generic SECRETNAME -from-file=.dockerconfigjson=$HOME/.docker/config.json -type=kubernetes.io/dockerconfigjson To forward pod port 8080
to the local computer on the port
8888: $ kubectl port-forward MYPOD 8888:8080
To test RBAC rules
: $ kubectl -as=system: serviceaccount:MYNS:MYSA auth can-i get configmap/MYCM yes Kubectl
Contexts Kubectl
uses contexts to know how to communicate with the cluster. Contexts are stored in a kubeconfig file, which can store multiple contexts. Contexts will usually be provided by some other commands related to the control plane or some other management commands.
Such commands will usually add context to your kubeconfig file. The default kubectl configuration file is located in $HOME/.kube/config. You can use a different kubectl configuration file by specifying the -kubeconfig=PATH arguments on the kubectl command line.
To list the contexts available in your kubeconfig file, use
“get-contexts”: $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE * arn:aws:eks:… arn:aws:eks:… arn:aws:eks:… minikube minikube default minikube
To switch from one context to another,
use “use-context”: $ kubectl config use-context minikube Changed to context “minikube”. $ kubectl config get-contexts CURRENT NAME CLUSTER AUTHINFO NAMESPACE arn:aws:eks:… arn:aws:eks:… arn:aws:eks:… * minikube minikube minikube default If you
only have one context (i.e. you are managing a single cluster), then you are not concerned about contexts. If you are working with two or more clusters, you may want to consider using the -context command-line option.
The reason is that if you switch between clusters often, you are virtually guaranteed that one day you will run a kubectl command that you intended for another cluster, and the consequences could be dramatic.
With this in mind, you might want to set your default context to something innocuous, like minikube, and force yourself to explicitly provide the -context option in every kubectl command.
Kubectl Get
Kubectl get is probably the command you’ll use the most, so it deserves its own section. Kubectl get can retrieve information about all Kubernetes objects as well as nodes in the Kubernetes data plane. The most common Kubernetes objects you’re likely to query are pods, services, deployments, stateful sets, and secrets.
Kubectl get offers a range of output formats:
-o wide just adds more information (which
- depends on the type of objects being queried
- generate the full current state of the object (and therefore usually includes more information than the original manifest files).
- allows you to select the information you want from the full JSON from the -o json option using jsonpath notation.
- -o go-template allows you to apply Go templates for more advanced functions.
). -o yaml and -o json
-o jsonpath
Here are some examples of commonly used commands:
List all pods in the default namespace
(in this case, there are no pods in the
default namespace): $ kubectl get pod No resources found in default namespace.
Learn more about a
given pod: $ kubectl -n mynamespace get po mypod-0 -o wide NAME READY STATE RESTARTS AGE IP NODE NOMINATED AVAILABILITY DOORS mypod-0 2/2 Running 0 4d3h 192.168.181.98 node1.lan [none] [none]
The READY column shows how many containers are in the “ready” state in the given pod. The IP column shows the assigned IP address of this pod within the Kubernetes cluster. The NODE column shows on which node the pod is running (or scheduled).
Get the full YAML status of the same pod above
: $ kubectl -n mynamespace get pods/mypod -o yaml apiVersion: v1 Type: Pod metadata: [ – crop – ]
To get the services in the default namespace:
$ kubectl get svc NAME TYPE CLUSTER-EXTERNAL IP-IP PORT(S) AGE kubernetes ClusterIP 10.100.0.1 [none] 443/TCP 24d mysql ClusterIP 10.100.78.115 [none] 3306/TCP 22d mysql-headless ClusterIP None [none] 3306/TCP 22d
The kubernetes service is used to access the Kubernetes API from the cluster and is typically in the default namespace.
To get the value of a secret:
$ kubectl -n mynamespace get secrets MYSECRET -o ‘jsonpath={.data. DB_PASSWORD}’ | base64 -d SuperSecretPassword
Kubernetes stores secrets as base64-encoded values, hence the | base64 -d at the end to display a human-readable value. The base64 command is available on many operating systems, but you may need to change it for your operating system.
Other commands to
get information
Imperative
commands
These commands directly instruct Kubernetes to perform a specific operation on a given object. This section will display the most common commands in no particular order. These are useful during the development stages, but you should definitely avoid them in a production environment. Use kubectl create to create a Kubernetes object, except for pods that are created with the kubectl run command. So, to create a pod directly:
$ kubectl run debug -image=busybox – sleep infinity pod/debug created $ kubectl get pod NAME READY STATUS RESTARTS AGE debug 1/1 Running 0 6s
Using the run command is good enough to run simple pods. For more complex pods with multiple containers, persistent volumes, mounted ConfigMaps or Secrets, and many environment variables, this method will become intractable very quickly and it is much easier to use a manifest file. To
delete the pod: $ kubectl delete pod/debug pod “debug” deleted
You can directly create all other Kubernetes objects using the kubectl create command. Here’s a (silly) example for creating a
deployment: $ kubectl create deployment nginx -image=nginx -replicas=2 deployment.apps/nginx created $ kubectl get pod NAME READY STATE RESTART AGE nginx-6799fc88D8-6CLHP 1/1 Running 0 9s nginx-6799fc88d8-cjz56 1/1 Running 0 9s
Again, you can delete the deployment by using the delete command:
$ kubectl delete deployment nginx deployment.apps “nginx” deleted
You can modify the implementation as follows (this will start your
favorite text editor): $ kubectl edit deployment nginx If you
don’t want kubectl to start a text editor, you can use the replace command. You will first need to get the status of the object in YAML format:
$ kubectl get deploy/nginx -o yaml ] tmp.yaml $ # edit tmp.yaml $ kubectl replace -f tmp.yaml
Note that not all elements of a Kubernetes object can be modified after it is created. Another method to modify a Kubernetes object is to
use the patch command. With the above deployment object, such a command might look like this
: $ kubectl patch deployment/nginx -p ‘{“spec”: {“replicas”: 3}}’ deployment.apps/nginx patched $ kubectl get pod NAME READY STATE RESTART AGE nginx-6799fc88d8-8jtw9 1/1 Running 0 11m nginx-6799fc88d8-dgkqr 1/1 running 0 24s nginx-6799fc88d8-vqnkj 1/1 running 0 24s
The -p argument accepts JSON or YAML, although JSON tends to be a bit easier to write. In any case, you’re probably thinking this isn’t a user-friendly way of doing things, and you’d be right. Here’s an example where working with manifest files and using declarative style makes things much easier.
You can also use the kubectl set command to easily modify certain objects, but it has a limited scope and is not generic. It is mainly used to modify the image of pods or deployments, for example
: $ kubectl set image deploy/nginx nginx=nginx:1.9.1 deployment.apps/nginx image updated $ kubectl get pod nginx-684c8b4f65-c6ws6 -o ‘jsonpath={.spec.containers[?( @.name==”nginx”)].image}’ nginx:1.9.1
By the way, the jsonpath used above contains a filter in the array “containers” to extract only the information we want. Another way to modify objects is to add or change the attached labels and annotations, for example
: $ kubectl label deploy/nginx -overwrite app=frontend deployment.apps/nginx labeled $ kubectl get deploy nginx -o jsonpath='{.metadata.labels}’ {“app”:”frontend”} $ kubectl annotate deploy/nginx test=yes deployment.apps/nginx annotated $ kubectl get deploy/nginx -o ‘jsonpath={.metadata.annotations}’ {“deployment.kubernetes.io/revision”:”2″, “test”:”yes”}
Declarative commands
When you want to work in declarative mode, you will have a manifest file (or a set of them) and essentially use a single command:
$ kubectl apply -f X
Here, X is a file or a directory (you can add several -f arguments if necessary). If you want to make some changes to your existing Kubernetes objects, simply modify the manifest files and run kubectl apply again. This will calculate the differences between your desired state and the existing state and make the necessary changes to reconcile them. The only caveat is that it will not delete objects that you have deleted from the manifest files.
If you use
kustomization files, you must use the -k option instead, and the argument must be a directory: $ kubectl apply -k
DIR
There is a special case where reconciliation is problematic, which has to do with the Kubernetes authorization system. The reasons for this are quite obscure and are touched on here if you are interested. If you make changes to RBAC resources or roles, you must use the kubectl auth reconcile command. More information can be found in the official Kubernetes documentation.
You can delete objects declared in manifest files by using the following command:
$ kubectl delete -f X
Deployment Management
Here are some useful commands for managing deployments (in a general sense, so it also includes stateful sets and
daemon sets). When you upgrade
a deployment (or stateful set or daemon set), you can view the status of the upgrade by using kubectl rollout status deploy/myapp. You can cancel a deployment using this command: kubectl rollout undo statefulset/myapp and get a history of changes using kubectl rollout history deployment/myapp. In practice, however, these commands are rarely used because you would manage deployments using Helm or some other similar software.
You can modify the number of
pods running for a given deployment by using kubectl scale -replicas=N deploy/myapp, where N is the desired new number of replicas. The end result is the same as using kubectl edit deploy/myapp and modifying the number of replicas there.
Again, in practice, you are likely to use Helm to perform static manual changes or the pod autoscaler and not perform any manual operations. By the way, you can set up basic autoscaling using kubectl autoscaling, although that only works with one metric: CPU utilization.
If your Docker registry is private, you’ll need to pass some credentials to Kubernetes. The easiest way to do this is to sign in first with the Docker command line, for example, on AWS:
$ aws ecr get-login-password | docker login -u AWS -password-stdin ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com
Once logged in, you can manually
create the secret like this: $ kubectl create secret generic SECRETNAME -from-file=.dockerconfigjson=$HOME/.docker/config.json -type=kubernetes.io/dockerconfigjson
You can then use the SECRETNAME secret in the pod specification within your manifest file as follows
: imagePullSecrets: [name: SECRETNAME]
Please note that such secrets are usually valid only for a limited period of time.
It is very common to forget to set imagePullSecrets, in which case your pods will show an “image extraction error”. Therefore, if you see that error and are using a private registry, you may want to verify that you set imagePullSecrets and that the secret has not expired.
Interact
with pods
This section will list a bunch of commands that are very useful for interacting with your pods.
There is nothing better than taking a projectile to a working capsule! Here’s how to do this (you can omit the -c argument if you only have one container running inside the pod):
$ kubectl -n NS exec -it POD -c CONTAINER – sh
Note that the shell will run as the default user specified by the container image. Kubectl does not currently allow you to run the shell as another user (e.g. root), although there are some plugins that can. Also note that you need an executable shell available inside the container image (so you can’t use it on undistributed images, for example).
Consequently, you may need to think about how such containers will be fixed. You may want to have two versions of such containers: one for development (with the shell and maybe other programs) and one for production (which would be without distribution).
You can quickly create a link between a given port number in a given pod running within the Kubernetes cluster and your computer (this is called port forwarding). Here’s how to do it, assuming
your pod exposes port 8080: $kubectl port-forward MYPOD 8888:8080
You can then open port 8888 on your local computer and that will forward traffic to the MYPOD pod on your port 8080. This is very useful for doing a quick check of your pod to verify that it looks good.
You can copy files and directories using the kubectl cp command. We won’t go into detail here, but sometimes it’s better to inspect large and/or binary files on your computer rather than struggling with the shell and a couple of utilities in an executive session.
Finally, you can use kubectl attach to connect to the
container terminal, although this is usually of limited use because it gets the output from kubectl logs, and applications generally do not read stdin.
Miscellaneous commands
It’s useful to be able to complete your shell commands, and kubectl has a solution ready for that: $ source kubectl
completion bash
Kubectl supports multiple shells, and you’ll need to refer to the Kubernetes documentation for more details. Note that you will also need to install the bash-completion package for your operating system (if it is not already installed).
In addition, the above command will enable autocomplete for the current session only; to make it permanent, add this command to the shell initialization file.
Pro tip: If, like me, you use bash and like to have an alias to abbreviate “kubectl” in “k”, add the following line to your ~/.bashrc file to make autocompletion work when you only use
“k”: complete -F __start_kubectl k
You can display the APIs and resource types available in your cluster as follows:
$ kubectl api-versions admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1 [ – snip – ] $ kubectl api-resources ABBREVIATED NAME APIVERSION NAMESPACED KIND bindings v1 true Binding componentstatuses cs v1 false ComponentStatus configmaps cm v1 true ConfigMap endpoints ep v1 true Endpoints events ev v1 true Event [ – snip – ]
Finally, a very useful command to test your RBAC rules is kubectl auth can-i. Here’s an example to test if a given service account can read a certain configuration map
: $ kubectl -as=system:serviceaccount:MYNS:MYSA auth can-i get configmap/MYCM yes
Obviously, you’ll need to replace the uppercase placeholders with names that make sense for your configuration.
Troubleshooting Kubernetes
with Komodor Kubernetes
It’s a complex system, and often, something will go wrong, simply because it can. In situations like this, you’ll likely start the troubleshooting process by going back to some of the kubectl commands above to try to determine the root cause. This process, however, can often spiral out of control and become a stressful, ineffective and time-consuming task.
This is why we created Komodor, a tool that helps development and operations teams stop wasting their precious time looking for needles in hay piles whenever things go wrong.
Acting as a single source of truth (SSOT) for all your k8s troubleshooting needs, Komodor offers
:
- Change intelligence: Every problem is the result of change. In a matter of seconds we can help you understand exactly who did what and when.
- In-depth visibility: A complete upstream timeline, showing all code and configuration changes, deployments, alerts, code differences, pod logs, and more. All within a glass panel with easy to break down options.
- Information about service dependencies: An easy way to understand changes between services and visualize their ripple effects throughout the system.
- Seamless notifications: Direct integration with your existing communication channels (e.g. Slack) so you have all the information you need, when you need it.
If you’re interested in watching Komodor, use this link to sign up for a free trial.
