If you’ve worked with MongoDB for some time, you’ve probably had this thought at least once:
“We already created a unique index… so duplicates shouldn’t happen.”
It sounds logical. It feels safe.
And honestly, most of us trust that — until we see duplicate data sitting in production where everyone is confident uniqueness is already enforced.
At that moment, the first reaction is:
“Something is wrong with MongoDB”
But if you pause and actually look at what’s happening, you realize something important:
MongoDB is not failing.
It’s doing exactly what we asked it to do — just not what we thought we asked.
I’ve seen this in real production systems, and I also tested these scenarios locally to understand the gap clearly.
Let’s walk through it — step by step, the way it actually unfolds in real environments.
What a unique index really does
When you create:
db.users.createIndex({ email: 1 }, { unique: true })MongoDB builds an index (B-tree via WiredTiger).
From that moment:
- Every insert/update checks the index
- If the value already exists → operation fails (E11000)
So the guarantee is simple:
No two indexed entries can have the same value
But there are three important realities:
- MongoDB does not clean existing data
- It only enforces rules on indexed entries
- And it enforces exactly what you define — nothing more
That last line is where most confusion begins.
The first mistake — assuming the index exists
Let’s start with a very common situation.
db.users.insertMany([{ email: "alice@example.com" },{ email: "alice@example.com" }])Now someone tries to fix it:
db.users.createIndex({ email: 1 }, { unique: true })MongoDB responds with a duplicate key error.
At this point, something subtle but dangerous usually happens. The command fails, but that failure is often not verified properly. From the application side, everything continues as if the index was successfully created. In reality, no such protection exists, and duplicate data continues to get inserted silently over time.
If you explicitly check using db.users.getIndexes(), you’ll see that the unique index was never created at all.
This is one of those issues that doesn’t show up immediately but slowly corrupts data integrity if unnoticed.
A step that should never be skipped
Before creating a unique index, the safest thing you can do is ask MongoDB a simple question:
“Do duplicates already exist?”
db.users.aggregate([ { $group: { _id: "$email", count: { $sum: 1 } } }, { $match: { count: { $gt: 1 } } }])This shows you exactly what the database considers as duplicates. MongoDB is not going to fix this for you — it will only enforce rules on clean data. If duplicates exist, they need to be resolved first before any uniqueness constraint can be applied.
Where things start getting misunderstood
This is where most real-world confusion begins.
Because at this stage, teams usually say:
“We have a unique index… but still seeing duplicates”
In almost every case, the index is not wrong. The misunderstanding lies in how MongoDB interprets data internally.
Missing fields quietly become null
Take a simple case — a field like sku that is not always present.
You insert a document like this:
db.products.createIndex({ sku: 1 }, { unique: true })db.products.insertOne({ name: "Item1" })db.products.insertOne({ name: "Item2" })The second insert fails with a duplicate key error referencing sku: null.
Even though the sku field is not present in either document, MongoDB internally treats missing fields as null for indexing purposes. From the index perspective, both documents effectively look identical.
This is expected behavior, but it often feels counterintuitive. From an application point of view, the field does not exist, so nothing appears duplicated. But from MongoDB’s perspective, both documents share the same indexed value.
This usually becomes visible when a field gradually becomes optional. Older records may not contain the field at all, while newer logic assumes it is always present. That mismatch leads to situations where inserts start failing in places where nothing looks duplicated at first glance.
Sparse index — same data, different rule
Then someone enables sparse — and behavior changes completely
To “fix” the null issue, someone enables sparse:
db.products.createIndex( { sku: 1 }, { unique: true, sparse: true })Now inserting multiple documents without sku works without any issue.
The reason is that MongoDB completely skips indexing documents where the field is missing. Since no index entries are created for those documents, no uniqueness check is performed.
However, this changes the rule in a subtle but important way. What was previously a strict constraint now becomes conditional. Uniqueness is enforced only for documents where the field exists.
If this change is not intentional, it can lead to silent data inconsistencies. Over time, multiple documents without the field can accumulate, and enforcing strict uniqueness later becomes much harder.
Partial index — uniqueness with conditions
To enforce uniqueness only for a subset of documents, we create a partial index.
db.orders.createIndex( { email: 1 }, { unique: true, partialFilterExpression: { status: "active" } })If you insert multiple documents with the same email but with status: inactive, MongoDB allows them without any issue.
This is because only documents matching the filter condition are indexed. Everything else is ignored from a uniqueness perspective.
At the application level, it may still feel like “email should be unique,” but the database is enforcing a more specific rule. This difference becomes visible later when the status changes to active, and conflicts suddenly start appearing.
What initially looks like a safe design can turn into a data consistency problem if this behavior is not clearly understood.
Same value… different type
To see how MongoDB treats different data types, we insert values that look similar but are stored differently.
db.users.insertOne({ email: "123" })db.users.insertOne({ email: 123 })Both documents are inserted successfully.
MongoDB treats values with different BSON types as completely distinct. Even though they look similar, a string and a number are not equal from the database’s point of view.
This often happens in distributed systems where different services or APIs send data in slightly different formats. Over time, this inconsistency bypasses uniqueness constraints and creates unexpected duplicates.
Case sensitivity — looks same, but not same
To observe case sensitivity, we insert values that differ only in letter case.
db.users.insertOne({ email: "ABC@test.com" })db.users.insertOne({ email: "abc@test.com" })Both values are treated as different.
MongoDB performs case-sensitive comparisons by default. If your application logic treats emails or usernames as case-insensitive, this difference can lead to duplicate logical entries.
To enforce case-insensitive uniqueness, a collation must be explicitly defined during index creation.
Compound index — uniqueness is not per field
To enforce uniqueness across multiple fields together, we create a compound index.
db.orders.createIndex( { email: 1, region: 1 }, { unique: true })This does not enforce uniqueness on email alone. Instead, it enforces uniqueness on the combination of email and region.
So the same email can exist in different regions without any issue. This is correct behavior, but it often leads to incorrect assumptions when the index is designed without fully considering the requirement.
Compound index + missing field
To see how missing fields behave in a compound index, we insert documents without one of the indexed fields.
db.compound_test.createIndex({ tenantId: 1, userId: 1 },{ unique: true })db.compound_test.insertOne({ tenantId: 1 })db.compound_test.insertOne({ tenantId: 1 })The second insert fails, even though userId was never provided.
Internally, MongoDB treats the missing field as null, making both documents identical from the index perspective. This is the same behavior seen earlier, but now it applies within a compound index.
This becomes confusing because the data appears incomplete rather than duplicated, yet the uniqueness constraint still applies.
Arrays — uniqueness does NOT go inside a document
Let’s take a simple scenario where a single document contains multiple account entries.
Here, the expectation is that even inside the same document, duplicate values should be prevented.
To test that, we first create a unique index on the account fields:
db.array_test.createIndex( { "accounts.bank": 1, "accounts.number": 1 }, { unique: true })# Now insert a document with one account:db.array_test.insertOne({ name: "Naga", accounts: [{ bank: "HDFC", number: "123" }]})#Next, we try to add the exact same account again into the same document:db.array_test.updateOne( { name: "Naga" }, { $push: { accounts: { bank: "HDFC", number: "123" } } })If you check the document, you’ll see duplicate entries inside the same array.
MongoDB allows this because uniqueness is enforced across documents, not within a single document. From the index perspective, there is still only one document containing that value.
This is important when working with arrays like phone numbers or accounts. Even with a unique index, duplicates can still exist within the same document if not handled explicitly.
Sharded clusters
In sharded environments, uniqueness behaves slightly differently.
unique index behavior depends on how the shard key is defined.
If a unique index does not include the shard key as a prefix, MongoDB cannot enforce global uniqueness across shards. In such cases, either the index creation fails or uniqueness is enforced only within individual shards.
This is a common source of confusion in distributed setups and should always be considered during design.
A safer write pattern — Upsert
Instead of relying on inserts that may fail under concurrency, we can use upsert.
db.users.updateOne( { email: "user@test.com" }, { $set: { name: "User" }, $setOnInsert: { createdAt: new Date() } }, { upsert: true })Instead of blindly inserting and handling duplicate errors, this pattern ensures that existing documents are updated while new ones are inserted safely.
In high-concurrency systems, this reduces race conditions and makes write operations more predictable.
Final takeaway
After going through all these scenarios, one thing becomes very clear.
MongoDB is not randomly allowing duplicates. It is consistently enforcing the exact rules defined in the index.
The confusion comes from the difference between how we think the data behaves and how MongoDB actually interprets it internally.
Once that gap is understood, the behavior becomes predictable. And once it’s predictable, it becomes reliable.
If there’s one thing to take away:
Don’t assume data integrity.
Define it clearly, verify it, and understand how MongoDB interprets your data.
Because once you do that, issues like this stop being surprises — and start becoming design decisions.