Being a data company is never easy, especially when you reach larger scale. An appropriate data store is one of the most important engineering and financial decision you’ll have to take. We decided to go with MongoDB to store parts of our aggregated data because we love MongoDB’s aggregation framework and the flexibility in it’s data modeling. We find it to be a good solution for dynamic and fast growing companies like us. When our Mongo deployment started to get “fat,” so did our problems with it. This is exactly where we decided we need a change, or did we? MongoDB 3.0 sets us straight.
MongoDB 3.0 has been available for download since March 2015 and since it’s release it had seen several updates with performance improvements and bug fixes. Here is a selection of topics from what’s new in MongoDB 3.0 and why you should consider upgrading.
What’s new in MongoDB 3.0?
Pluggable Storage Engine
MongoDB 3.0 introduces a pluggable storage engine API that allows third parties to develop storage engines for MongoDB. The vision: “One data model, one API, one set of operations concerns. Under the hood – many options for every use case under the sun.” It is even possible to use different storage engines for different members of a single deployment’s Replica Set, making it easy to evaluate the migration of a new engine. The possibility to mix different storage engines on the same replica set allows, for example, to direct operational data (requiring low latency and high throughput performance) to replica set members using in-memory storage engine, while exposing the data to analytical processes running in a Hadoop cluster through a member configured with an HDFS storage engine, which is executing interactive or batch operations rather than real time queries.
So the platform is now open for new players to come in and integrate / implement new specialized storage engines to address a variety of requirements from different architectures and/or applications.
Facebook’s Parse, for example, reported on April this year that they are now running a RocksDB storage engine on MongoDB 3.0 in production. Another already supported 3rd party engine is Percona’s TokuMX. Other storage engines in development are an HDFS storage engine and a FusionIO engine that bypasses the filesystem.
We are very curious to see what kinds of new engines will rise in the future.
MongoDB 3.0 introduces support for the new WiredTiger storage engine out of the box, joining the classic MMAPv1 storage engine that was available in previous versions.
The new WiredTiger engine supports all previously available MongoDB features, introducing new features like document-level locking and data and index compression. WiredTiger performs more work per CPU core than alternative engines. To minimize on-disk overhead and I/O, WiredTiger uses compact file formats, additionally to optional compression.
According to MongoDB, for write-intensive applications, the new engine gives users an up to 10x increase in write performance, with 80 percent reduction in storage utilization, helping to lower costs of storage, achieve greater utilization of hardware, improve predictability of performance and minimize query latency. Migrating to the WiredTiger storage engine, MongoDB says, will deliver the most noticeable performance gains on highly write-intensive applications, such as an analytics platform like we operate at SOOMLA.
With document-level locking, a lock is being put on a single document (instead of a collection/database lock) while a write is being made to that document, in which case the operation is queued up and waits until the previous operation is completed. This brings far better utilization of the CPU and scales vertically much better with threading. WiredTiger uses various algorithms to minimize contention between threads.
Upgrades to the WiredTiger storage engine will be non-disruptive for existing deployments and can be performed with zero downtime.
The WiredTiger product and team was acquired by MongoDB December last year and continues development on the next version of the WiredTiger engine.
In 3.0, MMAPv1 adds support for collection-level locking. Previously, the storage engine had a write lock at the database-level, so each database only allowed one writer at a time. The new version of the engine improves concurrency, giving almost linear performance scaling with growing concurrent inserts.
Record allocation behavior has been improved to better handle large document sizes. The new allocation strategy called ‘power of 2 sized allocation’ can efficiently reuse freed records to reduce fragmentation, increasing the probability that an insert will fit into the free space created by an earlier deletion or document relocation.
Introduction of Ops Manager
Ops Manager is a new management and monitoring tool introduced with MongoDB 3.0 that incorporated best practices to help keep managed databases healthy and optimized. It converts complex manual tasks into automated procedures which are easy to execute by API calls or through dashboard.
Ops Manager helps operations teams with: deploying a new cluster or upgrading one, backing it up and recovering from backup, dynamically resizing capacity by adding shards and replica set members and it does all that with zero downtime. Some of these tasks would require hundreds of manual steps in the past, and now can be done in a single step.
Ops Manager allows to monitor more than a hundred key database and system health metrics including operations, counters, memory and CPU utilization, replication status, open connections, queues and any node status.
Increased Number of Replica Set Members
In MongoDB 3.0 replica sets can have up to 50 members instead of 12.
Replica Set Step Down Behavior Changes
Before stepping down the step-down procedure will attempt to terminate long running operations that might block the primary from stepping down, such as index build, or a map-reduce job.
To help prevent rollbacks the procedure will wait for an electable secondary to catch up to the state of the primary before stepping down. Previously, a primary would wait for a secondary to catch up to within 10 seconds of the primary.
Other Replica Set Operational Changes
Initial replica sync build indexes more efficiently applying epilog entries in batches using threads.
Provides a more predictable read preference behavior. Instances no longer pin connections to members of replica sets when performing read operation. Instead, mongos reevaluates read preferences for every operation.
Improved visibility of balancer operations. sh.status() includes information about the state of the balancer.
MongoDB adds a new SCRAM-SHA-1 challenge-response user authentication mechanism. This mechanism represents an improvement over the previously used MONGODB-CR providing a tunable work factor, per-user random salts rather than server-wide salts, stronger hashing (SHA-1 rather than MD5) and authentication of the server to the client as well as the client to the server.
New Query Introspection System
An improved explain introspection system that provides better output and a finer-grained introspection. The query plan can now be calculated and returned without actually running the query like before.
MongoDB now categorizes some log messages under specific components or operations and provides the ability to set the verbosity level for these components.
MongoDB Tools Enhancements
Key MongoDB tools mongoimport, mongoexport, mongodump, mongorestore, mongostat, mongotop and mongooplog have been rewritten as multi-threaded processes, allowing faster operation and smaller binaries.
The mongorestore tool can now accept BSON data input from standard input in addition to reading BSON data from file.
How we feel about it?
The new features introduced in MongoDB 3.0 look promising and should make it a better performing solution with substantially lower operation costs for our deployment. We chose to go ahead and upgrade. What about you?