BillRun is open-source billing solution for enterprise, which built on top of MongoDB. This presentation took place on Israeli MongoDB User Group meeting, February 20, 2014, at John Bryce college, Tel Aviv. Find out more: http://billrun.net

1. Billing on top of MongoDB
Ofer Cohen, S.D.O.C. Ltd.
MUG IL, February 2014

2. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintainence
Q&A

3. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintainence
Q&A

4. Who am I
●
●
●
●
●
Open Source Evangelist
Board member
Started from DevOp
Shifted to Biz
Co-Founder at S.D.O.C. Ltd.
○ open source solutions for enterprises
● Mongo user since 2012

5. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintainence
Q&A

6. How it all began
● Partnering a startup Golan Telecom (GT)
● Telecom CDR first use - Anti-Fraud
○ CDR = Call/Charge detail record
● CDR Types: MOC, MTC & Data
○ 3 Different data structure
○ MOC/MTC duration=0 means SMS

7. SQL Implementation
$base_query = "SELECT imsi FROM moc WHERE callEventStartTimeStamp >=" . $start
. " UNION SELECT imsi FROM mtc WHERE callEventStartTimeStamp >=" . $start
$base_query = "SELECT imsi FROM (" . $base_query . ") AS qry ";
if (isset($args['imsi']))
$base_query .= "WHERE imsi = '" . $this->_connection->real_escape_string($args['imsi']) . "'";
$base_query .= "GROUP BY imsi ";
$mtc_join_query = "SELECT 'mtc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(callingNumber)<=10, callEventDuration, 0))) AS israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(callingNumber)<=10, CEILING
(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(callingNumber, 1, 3)!='972', IF(CHAR_LENGTH(callingNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS
non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM mtc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";

8. Implementation cont.
$moc_join_query = "SELECT 'moc' AS type, imsi, SUM(callEventDuration) AS duration, SUM(CEILING(callEventDuration/60)*60) AS duration_round "
. ", SUM(chargeAmount) charge "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', callEventDuration, IF(CHAR_LENGTH(connectedNumber)<=10, callEventDuration, 0))) AS
israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)='972', CEILING(callEventDuration/60)*60, IF(CHAR_LENGTH(connectedNumber)<=10, CEILING
(callEventDuration/60)*60, 0))) AS israel_duration_round "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, callEventDuration, 0), 0)) AS non_israel_duration "
. ", SUM(IF(SUBSTRING(connectedNumber, 1, 3)!='972', IF(CHAR_LENGTH(connectedNumber)>10, CEILING(callEventDuration/60)*60, 0), 0)) AS
non_israel_duration_round "
. ", SUM(IF(callEventDuration = 0, 1, 0)) AS sms_count "
. "FROM moc "
. "WHERE callEventStartTimeStamp >=" . $start . " "
. "GROUP BY type, imsi";
//
$gprs_join_query = $this->build_query($args, "gprs");

9. It still continues
$group_query = "SELECT base.imsi, moc.duration AS moc_duration, moc.charge AS moc_charge, "
. "mtc.duration AS mtc_duration, mtc.charge AS mtc_charge, "
. "mtc.duration_round AS mtc_duration_round, moc.duration_round AS moc_duration_round, "
. "moc.israel_duration AS moc_israel_duration, moc.non_israel_duration AS moc_non_israel_duration, "
. "moc.israel_duration_round AS moc_israel_duration_round, moc.non_israel_duration_round AS moc_non_israel_duration_round, "
. "mtc.israel_duration AS mtc_israel_duration, mtc.non_israel_duration AS mtc_non_israel_duration, "
. "mtc.israel_duration_round AS mtc_israel_duration_round, mtc.non_israel_duration_round AS mtc_non_israel_duration_round, "
. "mtc.sms_count AS mtc_sms_count, moc.sms_count AS moc_sms_count "
. "FROM "
. "( " . $base_query . " ) AS base "
. " LEFT JOIN (" . $mtc_join_query . " ) AS mtc ON base.imsi = mtc.imsi "
. " LEFT JOIN (" . $moc_join_query . " ) AS moc ON base.imsi = moc.imsi " ;
if (isset($args['limit'])) {
$limit = (int) $args['limit'];
} else {
$limit = 100000;
}

10. How it works with Mongo?
$base_match = array(
'$match' => array(
'source' => 'nrtrde',
'unified_record_time' => array('$gte' => new MongoDate($charge_time)),
)
);
$where = array(
'$match' => array(
'record_type' => 'MOC',
'connectedNumber' => array('$regex' => '^972'),
'event_stamp' => array('$exists' => false),
'deposit_stamp' => array('$exists' => false),
'callEventDurationRound' => array('$gt' => 0), // not sms
),
);
$group = array(
'$group' => array(
"_id" => '$imsi',
"moc_israel" => array('$sum' => '$callEventDurationRound'),
'lines_stamps' => array('$addToSet' => '$stamp'),
),
);
$project = array(
'$project' => array(
'imsi' => '$_id',
'_id' => 0,
'moc_israel' => 1,
'lines_stamps' => 1,
),
);

11. How it works with Mongo?
$having = array(
'$match' => array(
'moc_israel' => array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.moc.israel'))
),
);
$moc_israel = $lines->aggregate($base_match, $where, $group, $project, $having);
//sms out to all numbers
$where['$match']['record_type'] = 'MOC';
$where['$match']['callEventDurationRound'] = 0;
$group['$group']['sms_out'] = $group['$group']['mtc_all'];
unset($group['$group']['mtc_all']);
unset($having['$match']['mtc_all']);
$group['$group']['sms_out'] = array('$sum' => 1);
$having['$match']['sms_out'] = array('$gte' => Billrun_Factory::config()->getConfigValue('nrtrde.thresholds.smsout'));
$project['$project']['sms_out'] = 1;
unset($project['$project']['mtc_all']);
$sms_out = $lines->aggregate($base_match, $where, $group, $project, $having);

12. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintainence
Q&A

13. Next - Small (ILDs) Billing
● Billing only for ILDs (A.K.A. MABAL)
○ Received 6 types of CDRs *almost* the same
○ Each MABAL do what is under his 7th letter
● We decide from day 1 to go with Mongo

14. Small Billing - TCO
● 2 weeks of dev
● 2 days of deployment and first run
● 10,000 invoices
● Low maintenance=>Avg of 2 hours per month
● Base architecture for the main billing

15. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintainence
Q&A

16. Main Billing storage requirements
● 10 types of CDRs
○ Some binary hierarchical - Nokia Siemens
specification
● 500,000,000 CDRs per month
● Requirement to save each CDR in DB
storage.
● Fields can be added on the fly

17. Main Billing storage size
● 500,000,000/28 days/24 hr/60 min/60 sec
○ 232 CDRs per second

18. Main Billing storage size
● 500,000,000/5 days/24 hr/60 min/60 sec
○ 1152 CDRs per second

19. Main Billing storage size
● Today GT have 400,000 subscribers
● Might be 800,000 in the near future
● Data usage (CDRs) increasing (40% of the storage)
● Zero downtime
● 2 data centers redundancy

20. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintainence
Q&A

21. Infrastructure
Thanks to Eran Uzan

22. Sharding
● Shard key
○ Requirement for good balancing
● Updates are really fast with shard key
○ No collection lock yet
○ But lock only part of the DB (1 shard)

23. The size [GB]
> db.lines.stats(1024*1024*1024)
{
"sharded" : true,
"ns" : "billing.lines",
"count" : 1436122538,
"numExtents" : 771,
"size" : 1275,
"storageSize" : 1302,
"totalIndexSize" : 355,
"indexSizes" : {
"_id_" : 36,
"aid_1" : 36,
"aid_1_urt_1" : 48,
"sid_1" : 36,
"stamp_1" : 126,
"type_1" : 30,
"urt_1" : 30
},
"avgObjSize" : 8.878072492167796e-7, //954.7864099064851 Bytes
"nindexes" : 7,
"nchunks" : 21516,

24. The size [GB] - each shard
"rs0" : {
"rs1" : {
"ns" : "billing.lines",
"ns" : "billing.lines",
"count" : 238827456,
"count" : 239694511,
"size" : 212,
"size" : 213,
"avgObjSize" : 8.876701345426549e-7,
"avgObjSize" : 8.886311126248528e-7,
"storageSize" : 217,
"storageSize" : 217,
"numExtents" : 130,
"numExtents" : 130,
"nindexes" : 7,
"nindexes" : 7,
"lastExtentSize" : 1,
"lastExtentSize" : 1,
"paddingFactor" : 1.0000000000335132,
"paddingFactor" : 1.0000000000337341,
"systemFlags" : 1,
"systemFlags" : 1,
"userFlags" : 0,
"userFlags" : 0,
"totalIndexSize" : 59,
"totalIndexSize" : 59,
"indexSizes" : {
"indexSizes" : {
"_id_" : 6,
"_id_" : 6,
"stamp_1" : 21,
"stamp_1" : 21,
"urt_1" : 5,
"urt_1" : 5,
"aid_1" : 6,
"aid_1" : 6,
"sid_1" : 6,
"sid_1" : 6,
"type_1" : 5,
"type_1" : 5,
"aid_1_urt_1" : 8
"aid_1_urt_1" : 8
},
},
"ok" : 1
},
"ok" : 1
},

25. Pay attention
● With SSD you get x20 more performance
○ 5000 lines/second inserted during peak
● Mongo loves RAM
○ All used indexes must to be in RAM
○ Means, each shard have 64 GB in our env

26. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintainence
Q&A

27. Billing - Transactions
●
●
●
●
●
●
●
write concern = 1 (by default)
Unique key
findAndModify (A.K.A FAM)
value=oldValue
2 phase commit on app side
Auto increment (invoice id)
Transaction lock by design

28. Billing - Transactions
● More in our blog post, written by Shani Dalal
http://www.billrun.net/mongo-acid

29. Agenda
1.
2.
3.
4.
5.
6.
7.
8.
Who am I
How it all began
Small Billing
Main Billing
Infrastructure
Transactions
Maintenance, BI & TCO
Q&A

30. Billing - Maintenance
● MongoDB subscription
○ Leverage your professionalism
○ Special for enterprise or start-up
○ Low TCO
● Monitoring
○ Built in commands for kick-off
○ MMS the best tool for this purpose

31. Billing - Maintenance
● Backup - hot or cold
○ With replica it’s pretty easy (without downtime)
○ Can be done with MMS or simple script

32. Billing - BI
● MongoDB to SQL
○ Script convert hierarchy to one dimensional object
○ Script to CSV to MySQL
● Pentaho BI tool

33. Billing - TCO
● 3 months dev
● 1 month without SSD
● 2 months QA
● Solution can be extends easily
○ Without app change

34. Q&A
MUG IL, February 2014

35. Thank you
Ofer Cohen, S.D.O.C. Ltd.
ofer@billrun.net
@oc666
MUG IL, February 2014