Environment
Deployment: Kubernetes (RKE2), Bitnami MongoDB Sharded Helm chart MongoDB: 8.0.8 mongosh: 2.5.0 Topology: Config servers: 3 (CSRS) Mongos: 3 Shards: 3 shards total; each shard is a replica set with 2 data-bearing members + 1 arbiter Constraint: Topology is fixed (2 data + 1 arbiter per shard). We need resilience to a single data pod (node) outage. When a shard loses one data-bearing member and a step-up occurs (new PRIMARY), the first access to some databases via mongos fails with:
MongoServerError[WriteConcernFailed]: waiting for replication timed out Shard logs show that mongos (or the shard) triggers a databaseVersion metadata refresh on the shard (config.cache.databases) to handle StaleDbVersion. That refresh writes with writeConcern: "majority" on the shard. In a 2 data + 1 arbiter RS with one data node down, “majority” is 2 but only one data-bearing member can acknowledge, so the refresh times out.
Oddly, some databases “work” (no refresh needed at that moment), while others consistently fail until the shard regains data-bearing majority.
On the affected shard (source DB “credential” below), listCollections via mongos triggers a refresh that fails:
{"t":{"$date":"2025-11-13T12:45:24.455+00:00"},"s":"D1","c":"SH_REFR","id":24100,"svc":"S","ctx":"CatalogCache-23","msg":"Error refreshing cached database entry","attr":{"db":"credential","durationMillis":60010,"error":"WriteConcernFailed: waiting for replication timed out"}} {"t":{"$date":"2025-11-13T12:45:24.455+00:00"},"s":"E","c":"SHARDING","id":6697205,"svc":"-","ctx":"Sharding-Fixed-14","msg":"Failed database metadata refresh","attr":{"db":"credential","error":"WriteConcernFailed: waiting for replication timed out"}} {"t":{"$date":"2025-11-13T12:45:24.455+00:00"},"s":"I","c":"SHARDING","id":22065,"svc":"S","ctx":"conn725","msg":"Failed to refresh databaseVersion","attr":{"db":"credential","error":"WriteConcernFailed: waiting for replication timed out"}} {"t":{"$date":"2025-11-13T12:45:24.455+00:00"},"s":"I","c":"COMMAND","id":51803,"svc":"S","ctx":"conn725","msg":"Slow query","attr":{"type":"command","ns":"credential.$cmd","command":{"listCollections":1,"nameOnly":true,"databaseVersion":{"uuid":{"$uuid":"c511729b-0757-4e92-a94a-6052482226dc"},"timestamp":{"$timestamp":{"t":1763022815,"i":2}},"lastMod":1}},"databaseVersionRefreshDurationMillis":60011,"ok":0,"errName":"WriteConcernFailed","errCode":64,"errMsg":"waiting for replication timed out"}} On the same shard, hello shows majority cannot advance while single data-bearing member is up:
{ opTime: { ts: Timestamp({ t: 1763039352, i: 1 }), t: Long('12') }, lastWriteDate: ISODate('2025-11-13T13:09:12.000Z'), majorityOpTime: { ts: Timestamp({ t: 1763034662, i: 1 }), t: Long('12') }, majorityWriteDate: ISODate('2025-11-13T11:51:02.000Z') } Replica set status reflects the topology:
writeMajorityCount: 2 votingMembersCount: 3 // 2 data + 1 arbiter Arbiters vote but do not acknowledge writes; with one data node down, shard-majority writes cannot succeed. Is there a configuration option to set the writeMajorityCount in a replica set?
