This page shows how to delete Pods that are part of a stateful set and explains considerations when doing so.
Before you
begin
- This is a fairly advanced task and has the potential to violate some of the properties inherent in StatefulSet
- Before proceeding, familiarize yourself with the considerations listed below.
.
StatefulSet considerations
In normal StatefulSet operation, never it is necessary to force the deletion of a StatefulSet Pod. The StatefulSet controller is responsible for creating, scaling, and deleting StatefulSet members. Try to make sure that the specified number of Pods from ordinal 0 to N-1 is alive and ready. StatefulSet ensures that, at any given time, there is at most one Pod with a given identity running in a cluster. This is known as at most one semantics provided by StatefulSet.
Forced manual deletion should be done with caution, as it has the potential to violate the semantics of at most one inherent in StatefulSet. StatefulSets can be used to run distributed and clustered applications that need a stable network identity and stable storage. These applications often have a configuration that is based on a set of a fixed number of members with fixed identities. Having multiple members with the same identity can be disastrous and can lead to data loss (e.g., split-brain scenario in quorum-based systems).
Remove pods
You can perform a successful pod removal with the following command:
For the above to lead to a correct termination, the Pod should not specify a pod. Spec.TerminationGracePeriodSeconds of 0. The practice of establishing a pod. 0-second Spec.TerminationGracePeriodSeconds is not safe and strongly discouraged for StatefulSet pods. Proper removal is safe and will ensure that the Pod shuts down properly before the kubelet removes the apiserver name.
A Pod is not automatically deleted when a node cannot be accessed. Pods running on an inaccessible node enter the ‘Terminating’ or ‘Unknown’ state after a timeout. Pods can also enter these states when the user successfully attempts to delete a pod on an unreachable node. The only ways a Pod in such a state can be removed from the apiserver are as follows:
The Node
- object is deleted (either by you or by the Node controller). The
- kubelet on the unresponsive node starts responding, kills the pod, and deletes the apiserver entry
- Force the removal of the Pod by the user.
.
The recommended best practice is to use the first or second approach. If a node is confirmed to be dead (for example, permanently disconnected from the network, shutdown, etc.), delete the Node object. If the node suffers from a network partition, try to resolve this or wait for it to be resolved. When the partition is healed, the kubelet will complete the removal of the Pod and release its name on the apiserver.
Typically, the system completes the removal once the Pod is no longer running on a Node, or the Node is deleted by an administrator. You can override this by forcing the Pod to be removed.
Force delete
Forced deletions do not wait for confirmation from the kubelet that the Pod has been finished. Regardless of whether a forced delete succeeds in killing a Pod, it will immediately release the apiserver name. This would allow the StatefulSet controller to create a replacement Pod with the same identity; this can lead to duplication of a Pod still running, and if such a Pod can still communicate with the other members of the StatefulSet, it will violate the semantics at most one that StatefulSet is designed to guarantee.
When you force the removal of a StatefulSet pod, you are claiming that the Pod in question will never again contact other Pods in the StatefulSet and its name can be safely released for a replacement to be created.
If you
want to remove a Pod forcibly
using kubectl version >= 1.5, do the following:If you
are using any version of kubectl <= 1.4, you should skip the -force option
and use:
If even after these commands the pod is stuck in Unknown state, use the following command to remove the pod from the cluster:
Always perform forced removal of StatefulSet Pods carefully and with full knowledge of the risks involved.
What’s next
Learn more about debugging a StatefulSet.
