Author

A server core senior, enthusiastic about quality, creativity and growth. Java-grown, web-enhanced and bash-infused. Onn holds a B.Sc. in Computer Engineering and an MBA from The Hebrew University of Jerusalem. He used to work for various startup companies in the field of mobile platforms and lately for IBM MobileFirst.

Tech Resources

Being a data company is never easy, especially when you reach larger scale. An appropriate data store is one of the most important engineering and financial decision you’ll have to take. (300x300) MongoDB-UpgradeWe decided to go with MongoDB to store parts of our aggregated data because we love MongoDB’s aggregation framework and the flexibility in it’s data modeling. We find it to be a good solution for dynamic and fast growing companies like us. When our Mongo deployment started to get “fat,” so did our problems with it. This is exactly where we decided we need a change, or did we? MongoDB 3.0 sets us straight.

MongoDB 3.0 has been available for download since March 2015 and since it’s release it had seen several updates with performance improvements and bug fixes. Here is a selection of topics from what’s new in MongoDB 3.0 and why you should consider upgrading.

What’s new in MongoDB 3.0?

Major Changes

Pluggable Storage Engine

MongoDB 3.0 introduces a pluggable storage engine API that allows third parties to develop storage engines for MongoDB. The vision: “One data model, one API, one set of operations concerns. Under the hood – many options for every use case under the sun.” It is even possible to use different storage engines for different members of a single deployment’s Replica Set, making it easy to evaluate the migration of a new engine. The possibility to mix different storage engines on the same replica set allows, for example, to direct operational data (requiring low latency and high throughput performance) to replica set members using in-memory storage engine, while exposing the data to analytical processes running in a Hadoop cluster through a member configured with an HDFS storage engine, which is executing interactive or batch operations rather than real time queries.

So the platform is now open for new players to come in and integrate / implement new specialized storage engines to address a variety of requirements from different architectures and/or applications.

Facebook’s Parse, for example, reported on April this year that they are now running a RocksDB storage engine on MongoDB 3.0 in production. Another already supported 3rd party engine is Percona’s TokuMX. Other storage engines in development are an HDFS storage engine and a FusionIO engine that bypasses the filesystem.

We are very curious to see what kinds of new engines will rise in the future.

@MongoDB 3.0 introduces a pluggable storage engine API that allows 3rd parties to develop storage… Click To Tweet

WiredTiger

MongoDB 3.0 introduces support for the new WiredTiger storage engine out of the box, joining the classic MMAPv1 storage engine that was available in previous versions.

The new WiredTiger engine supports all previously available MongoDB features, introducing new features like document-level locking and data and index compression. WiredTiger performs more work per CPU core than alternative engines. To minimize on-disk overhead and I/O, WiredTiger uses compact file formats, additionally to optional compression.

According to MongoDB, for write-intensive applications, the new engine gives users an up to 10x increase in write performance, with 80 percent reduction in storage utilization, helping to lower costs of storage, achieve greater utilization of hardware, improve predictability of performance and minimize query latency. Migrating to the WiredTiger storage engine, MongoDB says, will deliver the most noticeable performance gains on highly write-intensive applications, such as an analytics platform like we operate at SOOMLA.

With document-level locking, a lock is being put on a single document (instead of a collection/database lock) while a write is being made to that document, in which case the operation is queued up and waits until the previous operation is completed. This brings far better utilization of the CPU and scales vertically much better with threading. WiredTiger uses various algorithms to minimize contention between threads.

Upgrades to the WiredTiger storage engine will be non-disruptive for existing deployments and can be performed with zero downtime.

The WiredTiger product and team was acquired by MongoDB December last year and continues development on the next version of the WiredTiger engine.

MMAPv1 Improvements

In 3.0, MMAPv1 adds support for collection-level locking. Previously, the storage engine had a write lock at the database-level, so each database only allowed one writer at a time. The new version of the engine improves concurrency, giving almost linear performance scaling with growing concurrent inserts.

Record allocation behavior has been improved to better handle large document sizes. The new allocation strategy called ‘power of 2 sized allocation’ can efficiently reuse freed records to reduce fragmentation, increasing the probability that an insert will fit into the free space created by an earlier deletion or document relocation.

Introduction of Ops Manager

Ops Manager is a new management and monitoring tool introduced with MongoDB 3.0 that incorporated best practices to help keep managed databases healthy and optimized. It converts complex manual tasks into automated procedures which are easy to execute by API calls or through dashboard.

Ops Manager helps operations teams with: deploying a new cluster or upgrading one, backing it up and recovering from backup, dynamically resizing capacity by adding shards and replica set members and it does all that with zero downtime. Some of these tasks would require hundreds of manual steps in the past, and now can be done in a single step.

Ops Manager allows to monitor more than a hundred key database and system health metrics including operations, counters, memory and CPU utilization, replication status, open connections, queues and any node status.

Replica Sets

Increased Number of Replica Set Members

In MongoDB 3.0 replica sets can have up to 50 members instead of 12.

Replica Set Step Down Behavior Changes

Before stepping down the step-down procedure will attempt to terminate long running operations that might block the primary from stepping down, such as index build, or a map-reduce job.

To help prevent rollbacks the procedure will wait for an electable secondary to catch up to the state of the primary before stepping down. Previously, a primary would wait for a secondary to catch up to within 10 seconds of the primary.

Other Replica Set Operational Changes

Initial replica sync build indexes more efficiently applying epilog entries in batches using threads.

Sharded Clusters

Provides a more predictable read preference behavior. Instances no longer pin connections to members of replica sets when performing read operation. Instead, mongos reevaluates read preferences for every operation.

Improved visibility of balancer operations. sh.status() includes information about the state of the balancer.

Security Improvements

MongoDB adds a new SCRAM-SHA-1 challenge-response user authentication mechanism. This mechanism represents an improvement over the previously used MONGODB-CR providing a tunable work factor, per-user random salts rather than server-wide salts, stronger hashing (SHA-1 rather than MD5) and authentication of the server to the client as well as the client to the server.

Improvements

New Query Introspection System

An improved explain introspection system that provides better output and a finer-grained introspection. The query plan can now be calculated and returned without actually running the query like before.

Enhanced Logging

MongoDB now categorizes some log messages under specific components or operations and provides the ability to set the verbosity level for these components.

MongoDB Tools Enhancements

Key MongoDB tools mongoimport, mongoexport, mongodump, mongorestore, mongostat, mongotop and mongooplog have been rewritten as multi-threaded processes, allowing faster operation and smaller binaries.

The mongorestore tool can now accept BSON data input from standard input in addition to reading BSON data from file.

How we feel about it?

The new features introduced in MongoDB 3.0 look promising and should make it a better performing solution with substantially lower operation costs for our deployment. We chose to go ahead and upgrade.   What about you?

Sources:

https://docs.mongodb.org/manual/release-notes/3.0/

https://www.mongodb.com/press/wired-tiger

http://www.mongodb.com/presentations/webinar-an-introduction-to-mongodb-3-0

https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-1-95-reduction-operational-overhead-and-security-enhancements

https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-2-richer-query-language-enhanced-tools-and-global-multi-data

https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-3-performance-efficiency-gains-new-storage-architecture

https://www.mongodb.com/presentations/webinar-an-introduction-to-mongodb-3-0

http://s3.amazonaws.com/info-mongodb-com/MongoDB-Whats-New-3.0.pdf

http://www.wiredtiger.com/

http://blog.parse.com/announcements/mongodb-rocksdb-parse/

http://www.zdnet.com/article/mongodb-3-0-gets-ready-to-roll-with-wiredtiger-engine-onboard/

http://www.tomsitpro.com/articles/mongodb-big-data-document-store,1-2485.html

http://www.acmebenchmarking.com/2015/01/diving-deeper-into-mongodb-28.html

Feel free to share:
Announcement

SOOMLA’s Whales Report introduces a unique way to understand how your game is performing when looking specifically at your paying users. It is a known fact that in the mobile gaming industry the majority of revenue comes from whales, which constitutes a tiny margin of games players. These whales will easily be dissolved in the big numbers of your game’s analytics. We, at SOOMLA, think you should concentrate your efforts on whales when targeting your users. Studios with published games that are part of the Grow network are sent weekly reports.  The Whales Report helps identify inefficiencies in your game’s player-to-payer conversion funnel, improve your user retention performance and ultimately increase revenue.

Why is the Whales Report Unique?

The Whales Report harnesses knowledge about user behavior from other games and provides developers with crowdsourced intelligence about paying users that would otherwise be hard to come across.  A self-published studio might be able to see 100,000 users across two to three titles, but when that data is combined and analyzed from thousands of games, you suddenly have a much more accurate understanding of your users.  Traditional analytics and mobile marketing platforms can only show you data that you the developer measured in your game.  The Whales Report, as part of the Grow network, redefines this limitation by collecting and analyzing payer insights across many games from different vendors.  With user level insights about who will pay and who won’t, your retention and monetization practices can become truly data driven instead of guesswork driven.

Whales Report – Explained

Let’s go over the different parts of the Whales Report with an example:

The Whales Report showing new paying users and a growing graph.

  1. New paying users: players who are new to your game this week and have paid in the game. You can see how well you are doing at getting more players to pay your game week-over-week.
  2. New paying users graph: this graph shows your historical count of new paying users going 12 weeks back. You can see how well you are doing acquiring payers over time.More of the Whales Report showing total new users and looking at the various growth methods.
  3. New users: players that played your game for the first time during this week. Out of these new users:
  4. Seen in GROW: players that were seen (have played) in the the Grow network before they played your game. Use SOOMLA Insights to know these players better as soon as they enter your game.
  5. Paid in GROW: players that paid before in other games in the Grow network. These are the whales that have entered your game. You should focus your efforts on converting these players to paying users in your game.
  6. Paid in your game: players that have paid exclusively in your game and didn’t pay in other games in the Grow network. If you can raise this metric, then it means you’re doing something right compared to the rest of Grow.
  7. Missed potential sales: this section shows you how well you are doing at converting known whales (those that paid before on Grow) to pay in your game. It shows how many of the whales from Grow that played in your game didn’t pay this week in your game. If a large number of whales came into your game and did not pay, it suggests that you might consider doing something differently to get them to pay in your game as well. Every whale that pays elsewhere and not in your game is practically money that you’ve left on the table. The missed potential sales figure comes from multiplying number of whales not paying in your game with the Weekly ARPPU (= Weekly Average Revenue Per Paying User = Weekly Revenue divided by Weekly Paying Users Count).The Whales Report showing weekly payers retention and the weekly break down.
  8. Retention: this section shows your payer retention performance over the report’s week days using a cohort analysis chart. The left axis shows the date in which each cohort entered your game and the top shows how many days passed since that entry day. The matching cell for these axis values shows how many payers stayed in your game after this many days since their entry.
  9. Weekly users played: weekly count of users who played your game.
  10. Weekly users paid: weekly count of users who paid in your game.
  11. Weekly conversion: conversion rate for your game this week. Shows what percentage of new users you’ve been able to convert to paying users this week.
  12. All-time users played: total count of users who played your game since you joined the Grow network.
  13. All-time users paid: total count of users who paid in your game since you joined the Grow network.
  14. Average days to payment: how many days on average (on all days since you joined Grow) does it take a paying user to start paying in your game since he started playing it.The Whales Report showing the total revenue for last week and this week.
  15. Revenue: shows how much revenue you made this week and last week side by side.
  16. Revenue graph: shows your historical revenue over the last 12 weeks.

Tell us what you think

The SOOMLA Whales Report is ever evolving and will continue to grow and offer more insights in the future. We are also diligently working on cleaning up the various figures from fraudulent contribution, so with time you should expect these reports to be even more accurate.

We always love to hear what you think about our products and we are inviting you to suggest new features that can be valuable for you to have in the weekly reports. Feel free to contact me at: onn@soom.la

Feel free to share:

DOWNLOAD OUR RECENT REPORT

Join 8060 other smart people who get email updates for free!

We don't spam!

Unsubscribe any time

Categories

Archives