Marketing, Resource, Tech Resources, Tips and Advice

Complete Mobile Advertising Glossary – Over 70 Terms Defined

SOOMLA's complete mobile advertising glossary - we've broken down the top 70 terms in ad tech to be easily understood.

From newcomers to mobile industry veterans, the amount of acronyms and terms that exist can be daunting, and frankly, often times confusing. The industry is constantly evolving, and with that, comes a steady flow of new terminology. We’ve compiled a list of 70 terms that are most prevalent in adtech and broken them down into easy to understand terms.

Here is the complete glossary to mobile advertising at your fingertips. Read, take notes and prosper.

1st Impression

Refers to the first ad displayed to a user within a session. Considered to be the most valuable in terms of eCPM.

1st Look

Ad-networks are prioritized in a a waterfall where the 1st ad-network gets to “look” at the ad request and decide if they want to provide an ad or not. If the ad-network passes the turn goes down the waterfall to the 2nd ad-network. When an ad-network asks for “1st look” it means they want to sit on the top of the waterfall and have the opportunity to see all the ad requests.

1st Price Auction

Where a bidder pays exactly what they bid. Often times leading to inflated prices and a lower demand for that publisher’s inventory.

2nd Price Auction

Where the top bidder pays the what the second highest bid was + $0.01. Allows for the saving of money due to overestimation of the value of a publisher’s inventory.

A/B Testing

A test in which all variables are identical aside from one. The purpose is to compare the two versions and see which performs better according to the established KPI.

Ad Exchange

An ad exchange refers to a platform that helps facilitate the buying and selling of advertiser inventory from multiple ad networks.

Ad Inventory

The amount of (virtual) space a publisher has available to be sold to advertisers. Also known as “ad space”.

Ad Mediation

A platform that sends requests for advertisements to multiple ad networks for publishers, ensuring the ad space is filled with the best possible deal.

Ad Network

An ad network connects advertisers and publishers looking to generate revenue by serving ads in a mobile app or website.

Ad Revenue

The amount of revenue generated from placing advertisements in an application / website.

Ad Whales

Refers to a group of users who make the most amount of ad revenue. Typically this is defined as the top 10% or 20% of the users who make 70% or 80% of the ad revenue. In other cases, the ad-whales are defined as a threshold of $0.7 (the equivalent of a $1 IAP).

Advertiser Blacklisting

The process of publishers blacklisting specific advertisers from appearing in their app. Some ad networks provide this as an option to publishers. The blacklisting is often based on competition, poor eCPMs or inappropriate content.

CASE STUDY ON OPT-IN RATES & SOOMLA INSIGHTS

Advertiser Identity

Identifying which advertisers are advertising in one’s app and their respective performance.

API

Application Program Interface. An API opens access to a limited part of a piece of software, allowing 3rd party developers to access previously unaccessible information.

ARPU

(Average Revenue Per User). Calculated by dividing the total revenue by the number of users across a time perdiod (generally monthly or annually).

ASO

(App Store Optimization). The process of improving the visibility of an app in the respective app store (Google Play, iTunes, etc.)

Attribution

The process of accrediting a traffic source with the conversion. Each platform has it’s own methodologies for doing so.

Audience

A group of consumers / users within a specific target market which are targetted for a specific ad campaign.

Blacklist

A list of advertisers that a publisher has decided to not allow to appear within their app.

Churn Rate

The rate at which users have stopped using an app during a specific time period

CLV

(Customer Lifetime Value). Has the same meaning to LTV – the total value of a user if given enough time to fully exhaust all opportunities to pay or watch ads in the game. This value is often averaged across a group of users (cohort) and most publishers try to create models for future prediction of CLV/LTV based on the activity of the first few days so they can make quick decisions on the marketing side.

Cohort

A group of users that share one or more similar characteristics. Used for grouping in data analysis.

CPA

(Cost Per Acquisition). The cost a publisher incurs to bring a new user through various paid channels.

CPC

(Cost Per Click). A campaign where the advertiser is charged everytime a user clicks on the ad they are shown.

CPI

(Cost Per Install). A campaign where a price is paid when a user views an ad, goes to the app store and installs an app.

CPM

(Cost per Mile). A campaign were advertisers pay for every thousand times the advertisement is shown.

CTR

(Click-Through-Rate). The ratio of users who click on an ad / opt-in to the total users who were displayed an ad.

DAU

(Daily Active Users). How many users open your app on a daily basis. This is one of the key metrics used to measure ad revenue.

Direct Deal

Rather than going through an ad network or mediation platform, publishers can make direct deals with advertisers, cutting out the middle men.

DSP

(Demand Side Platform). A platform in which marketers can buy ad inventory from multiple ad exchanges in one place.

eCPM

(Effective Cost Per Thousand). A metric used to measure ad revenue generated. Generally calculated by dividing total revenue earned by total number of impressions in the thousands.

eCPM Decay

The inevitable decrease in eCPM as ads begin to be displayed to your users. There are however methods proven to help slow it down.

Fill Rate

The number of ads requested that are successfully filled in relation to the total number of ads requested. Tends to be displayed in a percentage format.

Frequency / Frequency Capping

The rate at which users are shown ads. For example, some apps place a frequency cap of 4 ads per day for users.

GDPR

(General Data Protection Regulation). A comprehensive new set of regulations designed to give EU citizens more control / oversight on their personal data and the entities that collect it. Under GDPR, companies will need to notify their customers when collecting personal data. Consent can be given or denied based on the purpose of the data usage.

Header Bidding

The process by which publishers open ad space to be bid upon by multiple ad exchanges, resulting in an increased yield and revenue.

IAP Cannibalization

The notion that the integration of incentivized rewarded videos are damaging to the revenue gained from in-app purchases.

IDFA

The “Identifier for Advertisers” is a random identifier number given by Apple to a user’s device. It is used for advertising targeting.

Impression

Refers to the moment when an ad is fetched from its source and dispalyed to the user. Each ad type has a varying distinction of when the impression occurs. For banner ads, once an ad is fetched from its source and the user sees the ad. For video ads, the impressions is logged once the first frame of the video is displayed.

Interstitials Ads

Ads that are full-screen which are typically displayed at natural transition points in the flow of an app. For example, between levels or when the game is paused.

Q1 2018 MONETIZATION BENCHMARKS

Lookalikes

A method of targeting additional users based on similarities to existing users. Generally based on geo, interests, gender, age etc.

LTV

(Lifetime Value). Has the same meaning to CLV – the total value of a user if given enough time to fully exhaust all opportunities to pay or watch ads in the game. This value is often averaged across a group of users (cohort) and most publishers try to create models for future prediction of CLV/LTV based on the activity of the first few days so they can make quick decisions on the marketing side.

MAU

(Monthly Active Users) How many users open your app on a monthly basis. This is another of the key metrics used to measure ad revenue.

Mobile Ad Fraud

The process which fraudsters cheat advertisers into displaying ads to fake users / bot traffic.

Multi-Touch Attribution

Allows for the attiributing of all channels in the conversion process. Different from the existing last touch approach.

Offer Wall

A list of incentives which a user can receive by performing a specific action, typically downloading another app, watching a video ad, or sharing content on social media. The rewards for doing so tend to be some form of digital currency or other in-app incentives.

Opt In

When a user actively agrees to participate in viewing an ad.

Pipelining

The process of moving data from one point to another.

Playable Ads

Ads that request the user to interact, often showing off the basic gameplay of the game being advertised.

Postbacks

The ability to notify a third party of a specific event that occurs within an app (install, in-app event, etc.)

Programmatic

An automated process for purchasing digitial advertising, eliminating the need for a human touch.

Reach

The number of unique users who can potentially be targeted by advertising.

Real-Time Bidding

Bidding on inventory in real-time. The bid is often dynamically generated based on past performance of creatives, inventory, user groups, and other parameters.

Retargeting

A marketing effort in which targeted online advertisements are shown to users based on their previous behaviour.

Rewarded Video Ads

Video ads that are offer the user the option to opt-in to view them in order to receive some incentive. Can range from free coins, extra lives to free uprades.

ROI

(Return on investment). A performance measure used to determine the efficiency of an investment / effort.

SDK

(Software Development Kit). A set of software development tools that can be added to existing apps to support new capabilities.

SDK Mediation

Mediation of ad-networks allows publishers to use a few ad-networks in parallel and maximize fill rates and eCPM. SDK mediation is a form of mediation where each ad-networks installs an SDK on the publisher’s app and the mediation is done on the client side.

Server Side Mediation

Server side mediaiton is a form of mediation. Unlike SDK mediation, the goal of maximizing fill rates and eCPMs for the publisher is acheived with a single SDK provided by the mediaiton company and the ad-networks have to serve their ads through that SDK. This means that the mediaion is done on the server side

Share of Voice

Refers to the portion that one company controls out of the total. In terms of ads, can refer to an ad network who maintains a larger share of voice in rewarded video displays in comparison to other ad networks.

SSP

(Supply Side Platform) A platform that allows publishers to sell their advertising inventory in an automated way.

Target Audience

A group of consumers / users within a specific target market which are targetted for a specific ad campaign.

Traceback

SOOMLA’s unique ability to traceback individual user’s ad behavior.

Tracker

A tracker is a link structure generated by the attribution platform on behalf of the advertiser. The tracker is than passed along to the ad-network so clicks on the ads of the advertiser which are associated with the ad-networks can be identified through the unique link structure of the tracker.

Unified auction

On open auction where all ad networks are given an equal opportunity to bit on an app’s inventory, without any preferential treatment, in which the highest bid tends to win.

UDID

(Unique User ID). A unique identifier assigned to each device. Apple and Google have their own methods for doing so.

User Acquisition

The marketing efforts to acquire new users through advertising campaigns.

VAST Tag

(Video Ad Serving Template). A universal XML-based protocol specification created by the IAB for serving video ads.

Whitelisting

The opposite of blacklisting in which certain entities are explicitly approved.

Yield Management

The use of tools and business practices to maximize revenue.


Find a term that we didn’t define here? Let us know in the comments below and we’ll get them added ASAP.

Feel free to share:
Analytics, Announcement, App Monetization, Resource, Tech Resources

Industry’s First Monetization Benchmark

Header image - the SOOMLA ads and churn case study is out for Q4 2017, full of insights

We are excited to announce our industry first “Q1 2018: Monetization Benchmarks” report today. This is one of the many industry data reports that we will continue to publish providing important insights related to monetization through ad revenue. This report gives an in-depth comparison of eCPMs for 1st impressions and overall and providing a ranking of monetization providers in the mobile industry.

You are welcome to download the report through this link.

Here are the quick take-aways from the report:

  • Advertisers and monetization providers are clearly paying a premium for first impressions. The premium can be as high as 100% of the average eCPM, sometimes higher
  • Monetization providers and advertisers have different bidding strategies when it comes to first impressions. Some are more aggressive while others seem indifferent to the impression sequence
  • Games tend to have a bigger focus on getting the 1st impression in comparison to non-gaming advertisers who appear to be indifferent to whether or not they are shown 1st.
  • There are a few advertisers who repeatedly show up in the top 10 across different ad formats and platforms. They are able to do that by having a clear data advantage. When negotiating prices for 1st impression – make sure you have enough data.

 

Feel free to share:
Analytics, App Monetization, Tech Resources, Tips and Advice

Rewarded Video Ads Strategy for F2P Games

A Look Into Social Point's F2P Rewarded Video Ads Strategy With Sharon Biggar

CLARIFICATION: Social Point are not a customer of SOOMLA at the moment

Video ads have been taking the mobile industry by storm as the new business model and with that, comes an abundance of questions as to how to optimize and what are their effects on users. Depending if you are a monetization manager or a product manager, your KPIs may vary, however the ultimate goal is the same: A successful app that maximizes it’s potential.

In a recent panel at GIAF (Games Industry Analytics Forum), Sharon Biggar, Chief Analytics Officer at Social Point led a captivating talk on their personal journey through a series of tests with their users on the effects of video ads and their various formats. The official title was “Video Advertising Strategy in F2P Mobile Gaming”. The video runs for about 25 mins, so if you have the time, give it a watch. Here are the topics covered:

Social Point sought to answer 3 main questions internally before coming to any conclusions regarding their F2P video advertising strategy and I’ll give a break down for each one.

What design options exist for video ads in F2P games?

Generally speaking, there are 4 main ways to integrate video ads into F2P games. Each one has it’s pluses and minuses and are used in different ways. Sharon broke them down into two main categories:

  • Pull Advertising – Pull referring to giving the user the choice to “pull” the video towards them, thus initiating the video ad.
  • Push Advertising – Push referring to forcing the user to watch the entirety of the video in order to continue.

Method 1 – Pull – Gain Currency

Rewarded Video Ads Pull 1 - Gain Currency

I’m sure we are all familiar with this method as its possibly the most prevalent in mobile games. While there are usually daily limits to prevent abuse, the general idea is a user can request to watch a video in return for a reward in forms of gold coins, gems or some other form of in game currency.

Method 2 – Pull – Double Rewards

Rewarded Video Ads Pull 2 - Double Rewards

These rewarded videos tend to appear in two forms as well. One being before initiating a level, the player can watch a video to double the effect bonuses received (gold, gems, experience) upon completing a level. Alternatively, the player is given the option once they’ve completed the level to double their rewards. Generally speaking there are no daily limits on these as they are based upon a user’s direct engagement with the app. Why limit your user’s engagement right?

Method 3 – Pull – Speed Up

Rewarded Video Ads Pull 3 - Speed Up

I’ve encountered (and used) these countless times. Sometimes they come in forms of using in game currency, but specifically here we are talking about watching a video ad in order to speed up an upgrade or improvement. Why wait 5 hours for an upgrade when you can watch a 30 second video to complete the upgrade immediately. Overall it’s a win – win and keeps user’s active.

Method 4 – Push – Forced Video

Rewarded Video Ads Push 4 - Forced Video

An intrusive and often causes some heat from users, due to it’s method, these video interstitials take over the full screen, and force the user to watch / interact with the ad. Some show a countdown before the X appears, others will appear after a few seconds. No reward / incentive is given to the user aside from waiting it out to allow them to continue to play.

Most Requested Video Ad Type

Looking specifically at their game “Dragon City”, in October, they took note of “pull” types of video ads and found some interesting things. *Note: “Dragon Cinema” refers to the gain currency type of rewarded video.

Video Ad Types and Engagement

  • Double Rewards was the most requested type of video ad
  • Interestingly enough that the “Speed Up” type of video ad was less used considering the time spent waiting vs. time spent watching the ad

Do Video Ads Increase Churn?

I think this is one of the most discussed topics right now in the industry. Before I dive into Social Point’s specific study, make sure you check out SOOMLA’s recent report on Lotum and their study on the effect on advertising direct competitors as well. It goes a bit more in depth into the effects on eCPM, churn and has surprising results. We’ve also published a series of blog posts on rewarded video ads which you can find here.

Touching upon Social Point’s study, they found that pulled video ads on similar cohorts (users that played 7 days and that had 14 sessions) had some interesting results. The users that did in fact engage with video ads within the first 7 days, churned significantly less than those who did not watch any video ads. Clearly indicating that for Social Point, video ads were NOT causing an increase in user churn.

Social Point's Results on Do Video Ads Increase Churn

FREE REPORT – VIDEO ADS RETENTION IMPACT

What About Push Video Ads?

Social Point ran an A/B test where they allowed a control group to see 0 video ads and the second group that saw skippable video ads. There was a clear decrease in retention once forced (push) video ads were displayed. More interestingly, as pointed out by Sharon, is the minor difference between 1 and 3 skippable video ads displayed effect on retention.

Do Push Video Ads Increase Churn?


Do Video Ads Cannibalise IAP Revenue?

Social Point ran an A/B test intending to test the theory that video ads cannibalize the in-app purchase revenue. The results were rather surprising as they did in fact find that the video ads resulted in an 8% decrease in IAP revenue in the control group, however when looking at the overall revenue, the ad revenue resulted in an overall 5% increase in total revenue despite the drop in in-app purchases.

Do Video Ads Canniblize In-App Revenue?

CASE STUDY ON ADVERTISERS CHURN & eCPM

Oooops!

The tests ran were under the condition that users could watch up to 4 video ads a day, as was the limit set at Social Point. However due to an intentional development change, users were able to watch up to 8 per day. Therefore the results changed a bit.

They quickly found that 39% of their payers (users who were doing IAPs), were watching up to 8 videos resulting in a significant drop in IAP revenue, and even further, the ad revenue was not off setting the drop in IAP revenue and cannibalising the revenue.

Too many rewarded videos results in drop in revenue


Conclusion

  • ”PULL” video ads have no impact on churn
  • ”PUSH” video ads have a small negative impact on churn
  • Video ads DO canniablise IAP revenue
  • IAP + AD revenue can be greater than IAP alone
  • Be careful – don’t allow Payers to watch too many ads

There were two great questions raised by the audience members which really touch upon the importance of analyzing advertisers, churn, eCPM as key drivers.

  • Q1: In the place where you raised the cap on video ads, did you see a drop in the eCPM and was that what impacted the video games short fall in the IAP?
  • A: Yes exactly. As soon as you go to a higher number of video ads, the players are less likely to convert on those video ads.
  • Q2: It is possible to optimize your ad stack to make up for the drop in eCPM?
  • A: Possibly, one of the challenges we’ve had is quite often the mediation platforms don’t have enough inventory.

To close, SOOMLA provides a series of tools helping monetizers, marketers and even product managers analyze critical KPIs on their app’s performance.

 

Feel free to share:
Research, Tech Resources

Top 10 Parse Alternatives For Your Game Backend

(1200x600) Top 10 Parse Alternatives

Contrary to popular belief among mobile game developers, the recent Parse announcement that it’s shutting its services down is not the end of the world.

Wait, what?

Yes, you heard me. A few days ago, Parse announced that it is retiring, sending out ripples of disbelief and discontent across the development world. No need to panic, though, keep reading.

First of all, Parse will have a year-long cooldown period – the final shutdown is scheduled for January 28, 2017, so you have plenty of time. Second of all, the company released a database migration tool (you can find it here), as well as an open-source Parse Server, which lets you run most of the Parse API from your own Node.js server.

Third of all, we’ve created a list of the best Parse alternatives for your mobile game you can find.

Why should you care?

Parse is a mobile backend as a service (MBaaS). That is a model that has grown to become an essential part of (almost) any game, even though it is a fairly new product category, one that’s been around for roughly five years. Its services might vary to some degree, from company to company, but the basics are the same – every MBaaS will offer a cloud storage solution, push notifications, file sharing and social integrations (Facebook, Twitter), as well as messaging and communications options. In today’s world of mobile and (quite often) social gaming, you can see why these features are essential to a mobile game’s success. It removes the burden of building in-app purchase item ownership data, building player progression storage or in-game communications, to name a few, and allows the developer to focus on more pressing matters like art, game design, innovation and monetization.

This is why we can’t have nice things

Parse was an important figure in the chain – it was loved by developers for having tons of features, good documentations and quality customer support. And after it got acquired by Facebook back in 2013 for $85 million, game developers were certain the company would have a bright future ahead – flocking to use its service.

Now, panic and fear has crept up on hearts and souls of mobile developers everywhere, as they raise their hands in despair and wonder why bad things always happen to good people </drama>.

But seriously, don’t worry. While the Parse announcement spawned a lot of lists with alternatives, those mostly revolve around general apps, with little to no focus on gaming. And with gaming being a specific industry in its own right, we feel a specific list is needed. We’ve got you covered. Below you will find the top 10 Parse alternatives for your game backend (listed in no particular order).

FREE REPORT – VIDEO ADS RETENTION IMPACT

gamesparksGameSparks

GameSparks, which launched in 2013 and now has over 72 million players using their platform, is a good mobile backend as a service option, and one of the more popular ones. It is flexible and has a good set of features such as analytics, a management dashboard, leaderboards, and real-time and turn-based muliplayer. It runs a MAU (Monthly Active Users) cost which can be confusing, leading people to think it’s too expensive when, in fact, it offers quite a competitive price. GameSparks isn’t a prescriptive service. They provide a highly flexible, configurable and extensible platform that allows developers to build and manage their own projects.

playfabPlayFab

PlayFab launched in September 2014, though behind the veil the’ve been in business for 3 years as the in-house backend for Uber Entertainment. Some will say it is the most complete backend platform, especially after it partnered with Photon, the multiplayer cloud service. With 20M players on their system and a top game holding 1 million DAU (confirmed with their team), PlayFab is no stranger to scale. Features include player accounts, virtual goods management, in-game messaging, and game data storage. Another unique PlayFab aspect is their recently launched marketplace, which makes it easy to integrate with key 3rd party services beyond Photon, such as attribution-tracking, advanced analytics, community tools, and more.

heroiclabHeroic Labs

The key selling point of HeroicLabs is the API which allows game developers to easily integrate multiplayer and social elements without needing a server backend. It focuses and optimizes mostly for massive games, games of high volume. HeroicLabs also has a code sample with SOOMLA in our knowledge base.

gamedoniaGamedonia

Gamedonia is another complete backend solution for mobile games. The cloud platform for game developers does not require a server and offers many social games and real-time elements such as PvP (player versus player) modules, in-game chat or social sharing. Gamedonia was founded in 2012 and besides offering mobile support, also works in the browser.

kiiKii

Kii is another developer sweetheart and a Unity partner, making its community support quite strong. Its key selling point is a burst limit of 150 API calls per second, which is quite important. On the other hand, it does not allow anonymous users. Other features include server extensions, push notifications, leaderboards and achievements. It supports iOS, Android and Windows 8.

kinveyKinvey

Kinvey is one of the pioneers in the MBaaS game, which by default makes it a strong contestant for the best service out there. Compared to Parse, I’d say the two are quite similar in features: it offers cloud storage and push notifications. There’s also an easy way to integrate Facebook Open Graph for all those apps without websites. However, like Parse, it’s a general purpose MBaaS for all mobile apps, not just games.

braincloudLogo_80brainCloud

brainCloud might make your brain hurt of all the features it offers. It calls itself “backend in a box.” It is a ready-made, cloud-based backend designed for game developers, allowing them to jumpstart their game creation with various pre-built features. Its features include Cloud Data, including user and global statistics, shared data and custom files, Multiplayer, with support for turn-by-turn and one-way offline (clash-style) multiplayer. Other features include Achievements, Leaderboards and Monetization features.

floxGamua Flox

Flox is a scalable and lightweight cloud backend for mobile games built by Gamua. It runs on all mobile devices supported by Adobe AIR, and also allows offline play. Players can be authenticated through Google+, Facebook, email or the iOS GameCenter API. It comes with rich documentation and a powerful customer support.  If you’re developing with AIR, or specifically the Starling framework, this is the backend for you.

app42App42

App42 is another popular solution. It has many features, including all the usual ones like leaderboards, cloud storage or social sharing. It used to be cheaper than Parse (now it definitely is), while offering the same burst limit. A great solution for any mobile game developer.

pun_logo_bigPhoton

Photon is a cross-platform multiplayer game backend – a service tailored especially for game developers. It allows you to easily add multiplayer to your games and run them in the Global Photon Cloud. You can also host your own Photon servers, if that kind of hybrid is your thing. It is a good choice for game developers of all sizes, from indies to AAA studios.

Basic pricing plans

Company Free Tier? Minimum Price
GameSparks Yes $0.02/player (MAU – applies when a game has reached 10,000 users)
PlayFab Yes Free. (Support + Enterprise are paid)
HeroicLabs Yes $69 / Month / $0.02 (MAU)
Gamedonia Yes 89€ / Month
Kii Yes $1,200 / Month
Kinvey No $2,000 / Month
brainCloud Yes $30 / Month
Gamua Flox No $29 / Month
App42 Yes $99 / Month
Photon Yes $95 / Month
Feel free to share:
Tech Resources

Why We Decided to Upgrade to MongoDB 3.0

Being a data company is never easy, especially when you reach larger scale. An appropriate data store is one of the most important engineering and financial decision you’ll have to take. (300x300) MongoDB-UpgradeWe decided to go with MongoDB to store parts of our aggregated data because we love MongoDB’s aggregation framework and the flexibility in it’s data modeling. We find it to be a good solution for dynamic and fast growing companies like us. When our Mongo deployment started to get “fat,” so did our problems with it. This is exactly where we decided we need a change, or did we? MongoDB 3.0 sets us straight.

MongoDB 3.0 has been available for download since March 2015 and since it’s release it had seen several updates with performance improvements and bug fixes. Here is a selection of topics from what’s new in MongoDB 3.0 and why you should consider upgrading.

What’s new in MongoDB 3.0?

Major Changes

Pluggable Storage Engine

MongoDB 3.0 introduces a pluggable storage engine API that allows third parties to develop storage engines for MongoDB. The vision: “One data model, one API, one set of operations concerns. Under the hood – many options for every use case under the sun.” It is even possible to use different storage engines for different members of a single deployment’s Replica Set, making it easy to evaluate the migration of a new engine. The possibility to mix different storage engines on the same replica set allows, for example, to direct operational data (requiring low latency and high throughput performance) to replica set members using in-memory storage engine, while exposing the data to analytical processes running in a Hadoop cluster through a member configured with an HDFS storage engine, which is executing interactive or batch operations rather than real time queries.

So the platform is now open for new players to come in and integrate / implement new specialized storage engines to address a variety of requirements from different architectures and/or applications.

Facebook’s Parse, for example, reported on April this year that they are now running a RocksDB storage engine on MongoDB 3.0 in production. Another already supported 3rd party engine is Percona’s TokuMX. Other storage engines in development are an HDFS storage engine and a FusionIO engine that bypasses the filesystem.

We are very curious to see what kinds of new engines will rise in the future.

@MongoDB 3.0 introduces a pluggable storage engine API that allows 3rd parties to develop storage… Click To Tweet

WiredTiger

MongoDB 3.0 introduces support for the new WiredTiger storage engine out of the box, joining the classic MMAPv1 storage engine that was available in previous versions.

The new WiredTiger engine supports all previously available MongoDB features, introducing new features like document-level locking and data and index compression. WiredTiger performs more work per CPU core than alternative engines. To minimize on-disk overhead and I/O, WiredTiger uses compact file formats, additionally to optional compression.

According to MongoDB, for write-intensive applications, the new engine gives users an up to 10x increase in write performance, with 80 percent reduction in storage utilization, helping to lower costs of storage, achieve greater utilization of hardware, improve predictability of performance and minimize query latency. Migrating to the WiredTiger storage engine, MongoDB says, will deliver the most noticeable performance gains on highly write-intensive applications, such as an analytics platform like we operate at SOOMLA.

With document-level locking, a lock is being put on a single document (instead of a collection/database lock) while a write is being made to that document, in which case the operation is queued up and waits until the previous operation is completed. This brings far better utilization of the CPU and scales vertically much better with threading. WiredTiger uses various algorithms to minimize contention between threads.

Upgrades to the WiredTiger storage engine will be non-disruptive for existing deployments and can be performed with zero downtime.

The WiredTiger product and team was acquired by MongoDB December last year and continues development on the next version of the WiredTiger engine.

MMAPv1 Improvements

In 3.0, MMAPv1 adds support for collection-level locking. Previously, the storage engine had a write lock at the database-level, so each database only allowed one writer at a time. The new version of the engine improves concurrency, giving almost linear performance scaling with growing concurrent inserts.

Record allocation behavior has been improved to better handle large document sizes. The new allocation strategy called ‘power of 2 sized allocation’ can efficiently reuse freed records to reduce fragmentation, increasing the probability that an insert will fit into the free space created by an earlier deletion or document relocation.

Introduction of Ops Manager

Ops Manager is a new management and monitoring tool introduced with MongoDB 3.0 that incorporated best practices to help keep managed databases healthy and optimized. It converts complex manual tasks into automated procedures which are easy to execute by API calls or through dashboard.

Ops Manager helps operations teams with: deploying a new cluster or upgrading one, backing it up and recovering from backup, dynamically resizing capacity by adding shards and replica set members and it does all that with zero downtime. Some of these tasks would require hundreds of manual steps in the past, and now can be done in a single step.

Ops Manager allows to monitor more than a hundred key database and system health metrics including operations, counters, memory and CPU utilization, replication status, open connections, queues and any node status.

Replica Sets

Increased Number of Replica Set Members

In MongoDB 3.0 replica sets can have up to 50 members instead of 12.

Replica Set Step Down Behavior Changes

Before stepping down the step-down procedure will attempt to terminate long running operations that might block the primary from stepping down, such as index build, or a map-reduce job.

To help prevent rollbacks the procedure will wait for an electable secondary to catch up to the state of the primary before stepping down. Previously, a primary would wait for a secondary to catch up to within 10 seconds of the primary.

Other Replica Set Operational Changes

Initial replica sync build indexes more efficiently applying epilog entries in batches using threads.

Sharded Clusters

Provides a more predictable read preference behavior. Instances no longer pin connections to members of replica sets when performing read operation. Instead, mongos reevaluates read preferences for every operation.

Improved visibility of balancer operations. sh.status() includes information about the state of the balancer.

Security Improvements

MongoDB adds a new SCRAM-SHA-1 challenge-response user authentication mechanism. This mechanism represents an improvement over the previously used MONGODB-CR providing a tunable work factor, per-user random salts rather than server-wide salts, stronger hashing (SHA-1 rather than MD5) and authentication of the server to the client as well as the client to the server.

Improvements

New Query Introspection System

An improved explain introspection system that provides better output and a finer-grained introspection. The query plan can now be calculated and returned without actually running the query like before.

Enhanced Logging

MongoDB now categorizes some log messages under specific components or operations and provides the ability to set the verbosity level for these components.

MongoDB Tools Enhancements

Key MongoDB tools mongoimport, mongoexport, mongodump, mongorestore, mongostat, mongotop and mongooplog have been rewritten as multi-threaded processes, allowing faster operation and smaller binaries.

The mongorestore tool can now accept BSON data input from standard input in addition to reading BSON data from file.

How we feel about it?

The new features introduced in MongoDB 3.0 look promising and should make it a better performing solution with substantially lower operation costs for our deployment. We chose to go ahead and upgrade.   What about you?

Sources:

https://docs.mongodb.org/manual/release-notes/3.0/

https://www.mongodb.com/press/wired-tiger

http://www.mongodb.com/presentations/webinar-an-introduction-to-mongodb-3-0

https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-1-95-reduction-operational-overhead-and-security-enhancements

https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-2-richer-query-language-enhanced-tools-and-global-multi-data

https://www.mongodb.com/blog/post/whats-new-mongodb-30-part-3-performance-efficiency-gains-new-storage-architecture

https://www.mongodb.com/presentations/webinar-an-introduction-to-mongodb-3-0

http://s3.amazonaws.com/info-mongodb-com/MongoDB-Whats-New-3.0.pdf

http://www.wiredtiger.com/

http://blog.parse.com/announcements/mongodb-rocksdb-parse/

http://www.zdnet.com/article/mongodb-3-0-gets-ready-to-roll-with-wiredtiger-engine-onboard/

http://www.tomsitpro.com/articles/mongodb-big-data-document-store,1-2485.html

http://www.acmebenchmarking.com/2015/01/diving-deeper-into-mongodb-28.html

Feel free to share:
Announcement, Tech Resources

Introducing Simplified Retention for Game Developers and Marketers

SOOMLA Retention Reports

SOOMLA introduces retention reports – the best way to explain the behavior of your users in terms of how many stayed in your game and how many left it. There are many services out there that present the ability to investigate user retention, SOOMLA’s retention reports come with 3 kinds of retention types, each of them shows you your users behavior from a different angle and gives you information so you can react accordingly to prevent users’ churn.

Investigating your new users

The first type of report is the Regular Retention, here you can see how many first time (new) users visited your game.

Regular_Retention

The two leftmost columns represent how many users started playing your game and the specific day they came in. Any other column “i” represents how many users visited your game in the i-th day after the starting day.

Rolling retention of first time users

The next retention report is Rolling Retention which was first introduced by Flurry. Rolling Retention shows you how many users are still “in your game.” For example, a user started playing on day 0, and came back only on day 3. Rolling Retention will count this user also on day 1 and 2 as if the user played throughout all these days.  While this is a more “optimistic” analysis of user behavior, it treats users as equals even if they didn’t come back each and every day.  Rolling Retention is particularly interesting to look at when coupled with other marketing activities used by your studio.  For example, a push notification campaign is likely to get users back and to boost this metric.  This metric has been much debated, which is why we offer multiple retention types for developers looking to optimize their retention.

Rolling Retention

Like in the regular retention, in Rolling the two leftmost columns show the number of users who start to play on those days. Every other column “i,” represents the number of users who visited your game after “i” days or more according to the date range you’re looking at.

Take a deeper look into returning users

The last type of retention you can find on the Grow dashboard, is the Return Retention. This kind of retention is different from the others because it doesn’t consider each date as a new “day-0” cohort.  Instead, it accounts for all active users on that certain date and shows you their return rate from that day on.  In that sense, Return Retention captures a snapshot of users from multiple cohorts and observes their future retention.  It also lets you identify weak and strong days of gameplay over time.

Return_Retention

The leftmost column represents the total number of users which visited your game on the exact day, while the other columns tell you how many of them came after the i-th day.

Retention is the cornerstone metric every studio should be tracking.  The importance of this metric can be emphasized by two observations:

  1. Retention is the foundation for calculating user lifetime value, which lends its hand to revenue projection and ROI calculations of a game.  It’s also necessary to know LTV in order to conduct ROI-positive user acquisition
  2. Retention expresses your users’ delight from your product.  It’s the ultimate metric for understanding if your game is truly entertaining to the extent of keeping users coming back for more.

We encourage studios to explore retention metrics in the Grow dashboard and to understand their users’ behavior.

Feel free to share:
Tech Resources

Setting up a new Mac for full stack development – UPDATED

This post is an update of an older post we had about how to set up shop on your mac: http://blog.soom.la/2014/04/setting-up-shop.html

Here we’ll describe the basic setup of every one of the engineers here in SOOMLA. This basic installation includes the basic essentials every full stack engineer needs.

Go through these stages to install the full stack

XCode installation

Using The OSX App Store, Install The Latest Version Of XCode, And Run It For The First Time

Command Line Tools

Next, Paste The Following Code In A New Terminal Window

xcode-select --install

Make your terminal work for you (dotfiles)

Everyone in SOOMLA works with the terminal. It’s the fastest and most efficient way to work as an engineer that needs to install things and run a lot of commands all the time.

dotfiles is a great project that incorporates many amazing people’s terminal configurations. We especially like mathiasbynens‘s stuff. It’s a great starting point to the perfect personal terminal configuration.

 Homebrew setup

Paste The Following Code In a Terminal Window

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  • Update via the brew update Command
  • Verify Successful Installation via The brew doctor Command

Install different packages

Now we’ll be installing various programs using brew, for each program enter the following command brew install xxx, replacing each time.
You can easily verify installation by running xxx --version after, where xxx is the name of the installed package.

The programs we’ll install using brew (click for further info):

$ git version
git version 2.3.0
$ openssl version
OpenSSL 0.9.8zd 8 Jan 2015
$ redis-cli --version
redis-cli 2.8.19
$ mongod --version
db version v2.6.7

Downloading JDK 1.6

Download And Install JDK 1.6 For OSX here

Installing the Android SDK

SDK
  • Enter brew install android-sdk into Terminal
  • Run The android Command From Terminal And Follow Instructions To Continue Installing
NDK

Enter brew install android-ndk into Terminal to install.

Setting up your IDEs

These are the IDEs we work with at SOOMLA

Rubymine

Download and install here

Android Studio

Download and install here

Text Editor

There’s a great debate which text editor is good for you. I guess it’s everyone’s choice. This is how we rank the test editors (from best to pretty good):

  1. Atom
  2. Sublime
  3. Textmate 2
  4. Brackets – especially for web development.

Additional utilities you will need

Robomongo – MongoDB Management

Download and install here

Skitch – Image Annotation

Download and install here

StackEdit – Markdown Editor

Get it here

Meld – Use it for git diffs

Follow instructions here

Gitx – Use it to see your git history

Download and install here

Sequel Pro – MySql client

Download and install here

Keep – For notes

You should really try it here

Git-ify your command line

Go over this great post by @gurdotan: http://rubyglazed.tumblr.com/post/15772234418/git-ify-your-command-line
Your command line should be your main tool when using git. Get ready to git in the speed of light.

Online Services Sign-Up

Every team member should be signed up to these services

  • Github
  • Trello

Done

Now you should be up and running, the SOOMLA way!

Feel free to share:
Resource, Tech Resources, Tips and Advice

Engineer – Don’t Over Sell Yourself in Interviews

Over the last few months, we’ve been conducting interviews for various positions and it’s been difficult. Exceptional engineers are hard to come binterview handshake_shadey. So between mobile game reviews and other technical posts, I wanted to take a moment and point out some do’s and don’ts for all you smart candidates out there:

Don’t Over Sell Yourself!

Sell yourself. Of course. But be careful, over selling yourself will lead to people not assessing your character and expertise correctly, which isn’t good.  This is bad not only for the hiring company, but also for you, as the candidate. A wrong perception of who you are may cause you to:

  • … be assigned to a position that doesn’t fit your knowledge.
  • … work in a company that doesn’t fit your character.
  • … be assigned a position that you don’t want.
  • … not get the job by seeming arrogant or sometimes over qualified.

What does over selling yourself look like?

I’m not saying that candidates are lying. If I had a feeling a candidate was lying to me then, of course, I wouldn’t want to work with him. I’m talking about candidates that create an image of someone they aren’t just to get the position. When you create a wrong image, you:

  • … say you know how to do something that you actually don’t.
  • … say you know how to do something that you’re actually just 20% experienced in.
  • … will try to put on a show and act like somebody you’re not. (If you’re not a loving & caring person, don’t try to be)
  • … will try to answer questions you weren’t asked, thinking it will impress your interviewer.
  • … will try to correct your interviewer and be wrong about it. (Oh god that’s a turn off 🙂 )

How To Not Over Sell Yourself

The evident result of over selling yourself is that you will very quickly be back on the market. And that’s if people in the company you joined really care about you. Smart managers know when a new candidate isn’t right and can recognize when they’ve hired the wrong person. Therefore, the best thing is for both sides to say goodbye nicely. It becomes even harder for managers to do this when they know they took the candidate from a previous workplace, but still the smart thing to do is part ways.

In order not to over sell yourself in an interview, think how you can help your interviewer. The guy sitting across from you asking questions and assessing your abilities is not there to fail you. He’s there to actually hire you. Give him the right reasons to hire you and not just reasons to hire you. Ask your interviewer: “What do you need?” and see if you fit that description. If the description is not clear enough, ask him to rephrase and make sure you have a clear understanding of the requirements and that they apply to you.

Another smart thing to do in an interview, is stay humble. Be patient, calm and answer what you’re asked. Don’t try to show how smart you are, if you’re smart it’ll be shown.

So stay focused, answer every question you’re asked and make sure you’re the right man for the job. The goal is not to win the interview, but to win the position that fits you most.

Good Luck!!!
🙂

Feel free to share:
Tech Resources

How We Boosted Counting Performance by 7410x with Redis


This is the second of a 2 part post about how we improved query performance on our analytics dashboard by over 7000x. All just by moving some of our data from MySQL to Redis. Part 1 was a technical explanation of the setup, while part 2 shows the benchmarks we saw when comparing fetching data from both systems.


We use Redis a lot. It is fast, stable, effective and awesome! This time, we found Redis useful in solving a painful problem: counting unique users for multiple different filters.

We recently found a new feature in Redis (new for us at least): HyperLogLog. HyperLogLog is a growth arrowprobabilistic data structure which makes estimating the number of distinct objects in a set very fast (Actually, more like blazing fast), but with a minor standard error (You can read more about it here). The moment we read about HyperLogLog we knew there’s something in it. And now that Redis has made it so easy to use, our testing started almost immediately.

We Want Real-Time Data

Until now, we used to keep all data about unique users in MySQL. The data was saved in different variations and ready for filtering (country, day …). As time went by, our queries became slower and slower. It was a pretty grim situation when all our different optimizations on MySQL showed us there’s no real solution here. We were offered to take many different approaches using Redshift, Hadoop or ElasticSearch but we didn’t want to have our data presented in any delay to our users. We wanted a complete, real-time data presentation in our dashboard that is being instantly updated using our background workers.

Redis to The Rescue

Once we had Redis running and migrated the MySQL data in, the results were astonishing. We’ve been tweaking MySQL to try to make distinct counting faster for a couple of months now, and results were mediocre at best (not to MySQLs fault, we were counting cardinality in 10 million+ row tables), but Redis was FAST. Although speed wasn’t the only thing we had to benchmark, we weren’t sure how well the 0.8% error deviation Redis promises for HyperLogLog stood up when we ran queries on our data.

MySql is Under Performing

To get us started, here is a benchmark of part of the many many different ways we tried tweaking MySQL specifically for COUNT DISTINCT

mysql-performance

We tried different query and index structures, the conclusions we drew from the process:

  • SELECT COUNT(*) FROM (SELECT * GROUP BY id) seemed to constantly work better than SELECT COUNT(DISTINCT id).
  • MySQLWorkbench is awesome.
  • With 10M rows and getting larger every day, MySQL just wasn’t the tool for counting the cardinality of our user-data.

Revelation of Goodness

Once we migrated all of our MySQL data into Redis Keys, we saw Redis zip by MySQL in a blink of the eye.

mysql-redis

There’s no mistake in that graph. We tried to chart both performance times of MySql and Redis on the same graph, but you probably can’t see redis’s values there. Here’s a close up of Redis performance times.

redis-performance

Amazing!

The Fly in The Ointment

This can’t be all so good. HyperLogLog only gives an estimate, so then it was time to compare the estimates to the actual MySQL counts. For most queries, the difference was much smaller than the 0.8% error deviation (the smallest was 0.03%), but after benchmarking many different queries, we also had 2 that reached an error of 1.1% and 1.7%.

redis-difference

In the end, these error deviations were acceptable for some of our use cases. We’re still saving exact counts outside of Redis … Just in case.

HyperLogLog is a very powerful tool for counting unique entities. You should definitely use it if you’re willing to accept its minor standard error.

Feel free to share:
Tech Resources

Installing Redis Master-Slave on Amazon EC2


This is the first out of a 2 part post about how we improved query performance on our analytics dashboard by over 7000x. All just by moving some of our data from MySQL to Redis. Part 1 will be a technical explanation of the setup, while part 2 will show the benchmarks we saw when comparing fetching data from both systems.


 

GROW is SOOMLA’s new user intelligence services presented by a brand new Analytics Dashboard. We wanted to provide mobile gaming studios with various ways to investigatRedis Featured Imagee their games using informative and important data metrics. The problem was, queries were slow and the user experience was bad. Most of the slowness stemmed from the fact we used MySql to calculate unique users in multiple different filters which was a bad choice for real-time uniqueness calculations. We tried to figure out ways to improve that until we stumbled upon a new method for calculating unique users with Redis.

We were already using Redis at this point, but only for internal purposes, and not for serving data to our web app, so we decided on setting up different servers that would only serve data for the dashboard. We looked at different options (clusters coming to Redis 3, which has officially been released since this blog post was written, Redis Sentinel using at least 3 different servers) but decided that for our usage, setting up a simple master-slave duo would be enough. Failover will be taken care of semi-manually instead of the overhead of sentinels, once a crash is identified we run a script that notifies the slave to be master, and switches the IP on all servers that connect to Redis (we based some of our approach on some great advice by the awesome @jondot).

When looking to set up a few Redis servers, we saw 2 major options:

  • Setting up our own machines and running Redis off them
  • Using a cloud Redis service such as Amazon Elasticache/Azure Cache/redislabs

After considering pricing and our specific needs we decided to manage our own machine, which came to a third of the price of other self managed cloud services.

Here is our process of setting up 2 Redis machines as Master/Slave

We start with a fresh instance on Amazon EC2, using Ubuntu 14.04


EC2 was just our choice for testing purposes… you can absolutely select your preferred cloud service provider.


Setup

SSH in and add all necessary keys to ~/.ssh/authorized_keys

Install the latest version of Redis via Chris Lea’s PPA:

sudo add-apt-repository ppa:chris-lea/redis-server
sudo apt-get update
sudo apt-get install redis-server

redis-server should be running now

Run redis-benchmark -q -n 100000 -c 50 -P 12 to make sure everything is running ok

Config

Open /etc/redis/redis.conf and change the following settings:

  • tcp-keepalive 60

  • comment bind
    This makes the machine accessible to anyone on the web

  • requirepass choose extremely secure password
    The extreme speediness of Redis is a double edged sword when it comes to password protection.
    An attacker can be able to try as much as 150,000 passwords/second so make that password secure.

  • maxmemory-policy noeviction
    For our needs, no key can ever be deleted

  • appendonly yes
    We will be using both AOF and RDB backups

  • appendfilename redis-[prod/stg]-s[1/2]-ao.aof

If configuring slave

This part is just for our slave

  • slaveof ip port
  • masterauth master password

Save and exit.

Restart redis with sudo service redis-server restart.

You should now be able to connect to redis via

redis-cli -h 127.0.0.1 -p [your port]
AUTH ֿ

Machine Config

Install Git:

sudo apt-get update
sudo apt-get install git

Install Node.JS + NPM:
We will migrate our data to Redis with some node scripts

sudo apt-get install nodejs
sudo apt-get install npm
sudo ln -s /usr/bin/nodejs /usr/sbin/node

Setting up semi-auto failover

We will set up the failover in a way where the redis server IP is an environment variable. On failure we will receive notification and set up a script that will:

  • switch the environment variable to be the IP of the slave machine
  • send the slave a SLAVEOF NO ONE command
  • update the master’s configuration to be slave of the original slave machine

After that we can take our time and figure out why the master server crashed.

To setup the local variable

create a new file in /etc/profile.d with a .sh extension, the file content should be

export REDIS_SERVER=

To make changes have effect immediately run

source .sh

And make sure everything is set by running

printenv REDIS_SERVER

Now, in your server environment configuration, set the Redis server url to be

server: process.env.REDIS_SERVER

Now your server should successfully connect to Redis via the environment variable.

If you’re running your server as a service

In this case the daemon service will not recognise your environment variables, therefor you should inject the .sh in your daemon script in /etc/init.d/yourservice

the injection should look like this

source /etc/profile.d/.sh

inserted before the start/stop/restart functions.

That’s it for setting up our server, stay tuned for the 2nd part, how using Redis HyperLogLog made our queries 7000x faster.

Feel free to share: