Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
MongoDB production checklist: replica sets, indexes and backups What we check before a MongoDB deployment carries real traffic: replica set layout, security settings, indexes that match the queries, the oplog window and…
A MongoDB deployment that worked well in development often goes live as a single mongod on the application server, with authentication off and port 27017 reachable from more places than anyone intended. It runs fine until the disk fills, a slow query pins the CPU, or someone asks for yesterday's data. This is our MongoDB production checklist, the one we work through before a database carries real traffic.
The short answer: run a three-member replica set on separate machines, enable authentication, TLS and a narrow bind address, size the WiredTiger cache deliberately, build indexes from real query shapes using the equality, sort, range rule, keep an oplog window of at least a day, and back up off the server with scheduled restore tests.
Replica set layout A standalone mongod is fine for development and nothing else. Even a small production system should be a replica set: it gives you automatic failover, lets you take one member down for maintenance, and is required for multi-document transactions and change streams.
Use three data-bearing members (primary, secondary, secondary). Avoid the primary-secondary-arbiter layout: when one data-bearing member is down, majority writes cannot be acknowledged and cache pressure builds on the primary. Put each member on a different machine, and ideally in a different zone or data centre. Three members on one host is a single point of failure with extra steps. Address members by DNS names, not IP addresses, so you can replace a machine without reconfiguring the set. Use internal authentication between members, either a keyfile or x.509 certificates. Give applications a connection string that lists the members and the replica set name, so drivers discover the topology and follow failover. # Application connection string (one line in practice)
mongodb://app_user:****@db1.internal:27017,db2.internal:27017,db3.internal:27017/appdb
?replicaSet=rs0&tls=true&authSource=admin&w=majority
# Health checks from mongosh
rs.status().members.map(m => [m.name, m.stateStr, m.health])
rs.printSecondaryReplicationInfo() // replication lag per secondarySince MongoDB 5.0 the default write concern is majority for most topologies, so an acknowledged write has reached most members and survives a failover. Keep it that way for anything you care about. Retryable writes are on by default in current drivers, which smooths over the few seconds of an election.
Server and security settings Most MongoDB incidents we are asked to look at are not clever. They are an open port, a cache that fights the application for memory, or a disk that filled up. A short mongod.conf covers the essentials:
storage:
dbPath: /var/lib/mongodb
wiredTiger:
engineConfig:
cacheSizeGB: 8 # set explicitly in containers or shared hosts
net:
port: 27017
bindIp: 127.0.0.1,db1.internal
tls:
mode: requireTLS
certificateKeyFile: /etc/mongodb/tls/db1.pem
CAFile: /etc/mongodb/tls/ca.pem
security:
authorization: enabled
keyFile: /etc/mongodb/keyfile # or clusterAuthMode: x509
replication:
replSetName: rs0
oplogSizeMB: 20480
operationProfiling:
mode: slowOp
slowOpThresholdMs: 100Authentication on, with separate users per application and the built-in roles they need: readWrite on their own database, backup for the backup job, clusterMonitor for the metrics exporter. No application connects as root. bindIp limited to localhost and the private interface. The firewall allows 27017 only from application servers and the other members, never from the internet. WiredTiger cache: by default it takes the larger of 50% of (RAM minus 1 GB) or 256 MB. On a dedicated server that is sensible. Current versions read a container's memory limit, but on a shared host or a container that also runs other processes, set cacheSizeGB yourself so the cache leaves room for everything else. Data on XFS, which MongoDB recommends for WiredTiger, on its own volume so a full log partition cannot stop the database. Raise the open files limit (MongoDB suggests 64000) and keep swappiness low. The official packages' systemd unit sets the limits for you. Transparent huge pages: MongoDB 8.0 and later recommend enabling them, while 7.0 and earlier recommend disabling. Check the production notes for your exact version rather than copying an old tuning script. Indexes that match your queries Without the right index, MongoDB scans the whole collection. That is invisible at ten thousand documents and painful at ten million. Build compound indexes from the query shapes your application actually runs, following the ESR rule: fields matched by equality first, then the sort fields, then fields filtered by range.
// Query: open orders for one customer, newest first, from the last 90 days
db.orders.find({ customerId: cid, status: "open", createdAt: { $gte: since } })
.sort({ createdAt: -1 })
// Equality (customerId, status), then sort/range (createdAt)
db.orders.createIndex({ customerId: 1, status: 1, createdAt: -1 })
// Confirm the plan: look for IXSCAN, not COLLSCAN,
// and totalKeysExamined close to nReturned
db.orders.find({ customerId: cid, status: "open", createdAt: { $gte: since } })
.sort({ createdAt: -1 })
.explain("executionStats")With the profiler in slowOp mode, operations slower than 100 ms are written to the log and to system.profile, which is where you find the queries that need an index. Run it the other way too. Every index costs memory and slows writes, so check usage with $indexStats and remove what nothing uses. The counters are per member and reset on restart, so check every member that serves reads. Hide an index first with hideIndex, watch for a week, then drop it; unhiding is instant if something slows down.
// Index usage since the last restart
db.orders.aggregate([{ $indexStats: {} }])
.forEach(i => print(i.name, i.accesses.ops))
// Safe removal: hide, observe, then drop
db.orders.hideIndex("status_1")
db.orders.dropIndex("status_1")Two more habits help. Use TTL indexes for data that should expire, such as sessions or temporary tokens, instead of a cleanup cron. And build large indexes during quieter hours: since 4.2 builds no longer block the collection for their whole duration, but they still compete for disk and cache with live traffic.
Oplog window and backups The oplog is a capped collection of recent writes that secondaries replay. Its time window decides how long a secondary can be offline and still catch up without a full resync, and how far back oplog-based point-in-time recovery can reach. A heavy batch job can shrink a comfortable window to an hour.
// Current oplog size and the time range it covers
rs.printReplicationInfo()
// Resize online (in MB), run on each member
db.adminCommand({ replSetResizeOplog: 1, size: 20480 })We aim for at least 24 hours of window at peak write load, and alert when it drops below that. Backups then depend on size and how much data you can afford to lose:
Small databases (a few GB): nightly mongodump to a compressed archive, run against a secondary, with --oplog for a consistent point in time. Our post on backups that actually restore shows the script and the restore test. Larger self-hosted replica sets: Percona Backup for MongoDB, which is open source, takes consistent logical backups and adds point-in-time recovery from the oplog. Its faster physical backups require Percona Server for MongoDB rather than the Community server. mongodump gets slow and heavy on cache well before the terabyte range. Filesystem snapshots (LVM or cloud disk snapshots) of a secondary are consistent as long as the journal is on the same volume as the data. MongoDB Atlas: turn on cloud backup with continuous backup if you need point-in-time restores, and check the retention policy matches what you agreed with the business. Whatever the method, copy backups off the server to storage the database host cannot delete from, and restore one into a clean instance on a schedule. A backup you have never restored is only a hopeful file.
Monitoring and upgrades For self-hosted deployments we run Percona's mongodb_exporter into Prometheus and Grafana; Atlas has equivalent built-in charts and alerts. The handful of signals worth alerting on:
A member not in PRIMARY or SECONDARY state, or no primary at all. Replication lag on any secondary above a minute or two. Oplog window below your target. Disk usage above 80% on the data volume. Connections climbing towards the limit, which usually means an application creating a client per request instead of reusing a pool. WiredTiger dirty cache staying high, a sign that disk cannot keep up with writes. Upgrades follow a fixed order. Go one major release at a time (6.0 to 7.0 to 8.0, never skipping). Upgrade secondaries first, then step down the primary and upgrade it. Only once the application has run cleanly on the new binaries for a while, raise the feature compatibility version, which from 7.0 onwards requires confirm: true. Until then, rolling back is a binary swap; after it, downgrading is far harder.
// After all members run the new binaries
db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })
db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true })The checklist Three data-bearing replica set members on separate machines, addressed by DNS name, no arbiter. Write concern majority and retryable writes left at their defaults. Authentication, TLS and internal member authentication enabled; one least-privilege user per application. Port 27017 reachable only from application servers and other members. WiredTiger cache sized explicitly in containers; data on its own XFS volume. Slow operation logging on; every frequent query covered by an ESR-ordered index; unused indexes hidden, then dropped. Oplog window of 24 hours or more at peak, with an alert below that. Off-server backups and a scheduled restore test with timing recorded. Alerts for member state, lag, oplog window, disk, connections and cache. A written upgrade path: one major version at a time, FCV raised last. If several items on this list are open on a database that already holds production data, fix them in this order: network exposure and authentication, then backups and a tested restore, then the replica set, then indexes. The first two protect the data you have. The rest make it faster and easier to keep running.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789