Understanding concurrency control in Kubernetes
June 1, 2025
Kubernetes is the de-facto standard for orchestrating thousands of containers in a cluster. It was originally designed for distributed environments, so it uses etcd as the database to store the state of Kubernetes resources such as Pods, ConfigMaps, and Services.
Because of its distributed nature, Kubernetes always struggles with concurrency. For example, when you try to update a resource without considering concurrency, you'll get an error like:
The object has been modified; please apply your changes to the latest version and try again
Why does this happen? How does the Kubernetes API server notice changes? How does it determine that an updated resource is outdated? In this article, I'll walk you through the details of concurrency control in Kubernetes and how it works under the hood.
etcd
etcd is a key-value store designed for distributed systems. It was created by the CoreOS team and is now a graduated project in the CNCF. It is well-known as the database for Kubernetes.
etcd is a replicated state machine (RSM) that copies the state of the database to multiple nodes and synchronizes them to keep data resilient even when several nodes fail. When a client inserts new data, the RSM sends the data to all nodes, so the data remains consistent across all nodes.
The critical part of implementing RSM is a consensus algorithm. In distributed systems, it's not guaranteed that all inputs are received in the same order, and messages may be lost during transmission. This is where Consensus comes into play.
etcd uses Raft as its consensus algorithm. In Raft, one server acts as the leader while others are followers. The leader handles all client requests and ensures data stays consistent across all servers.
When a client wants to add new data, the leader first saves it locally with a sequence number. Then it sends this data to all followers. Each follower saves the data and confirms back to the leader.
Once more than half of the servers have confirmed they saved the data, the leader considers it officially committed. This majority rule ensures the data is safe even if some servers fail.
resourceVersion
Kubernetes employs optimistic concurrency control, meaning it does not lock resources when they are updated. To implement this, it leverages resourceVersion. All Kubernetes resources have a resourceVersion field as part of their metadata. It is a string that serves as a monotonically increasing identifier used for optimistic concurrency control. If the versions don't match when a client tries to update a resource, the API server returns an error with 409 StatusConflict.
Kubernetes doesn't actually maintain resourceVersion itself. Instead, it is maintained by etcd. It comes from etcd's mod_revision, which is one of the revision metadata fields managed by etcd for distributed synchronization concurrency control. A revision is a 64-bit cluster-wide counter that increments each time the key space is modified. It serves as a global logical clock, sequentially ordering all updates to the store.
The resourceVersion is currently backed by etcd's mod_revision. However, it's important to note that applications should not rely on the implementation details of the versioning system maintained by Kubernetes. We may change the implementation of resourceVersion in the future, such as changing it to a timestamp or per-object counter.
The server changes the resourceVersion whenever a resource is updated. If resourceVersion is provided in a PUT request, the server verifies that the mutation is being applied to the current version of the resource by comparing whether the current value of resourceVersion matches the one in the request.
The only way for clients to know the value of resourceVersion is through GET requests to the Kubernetes API server. It is always obtained from the API server's response and provided again in further requests.
GET /api/v1/namespaces/default/pods/nginx-deployment
---
200 OK
Content-Type: application/json
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {"resourceVersion": "738", ...},
...
}Now we can answer the above question. The "Object Has Been Modified" error is occurred because the resource version in the request differs from the one on the server.
resourceVersion in Watch API
The Kubernetes API allows clients to make watch requests to detect changes on resources. Once a client starts a watch request, the server and client establish a long-lived connection, and the server streams events to the client.
The resourceVersion also plays a crucial role in the Watch API. Since the order of events may not match the order in which requests were made, Kubernetes uses resourceVersion to determine that events occurred after the client made the watch request. In other words, only events with a larger resourceVersion than the one in the watch request will be streamed to the client.
GET /api/v1/namespaces/test/pods?watch=1&resourceVersion=10245
---
200 OK
Transfer-Encoding: chunked
Content-Type: application/json
{
"type": "ADDED",
"object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
}
{
"type": "MODIFIED",
"object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...}
}
...Conclusion
resourceVersionplays a crucial role in Kubernetes for managing concurrency and detecting changes.- Currently,
resourceVersionis backed by etcd'smod_revision.