2. WHAT’S IN STORE
•
•
•
•
•
•
•
•
•
•

3. A DISCLAIMER
•
•
•
•
•
•

4. MICHAEL PEACOCK

5. SMITH ELECTRIC VEHICLES

6. NO. NOT MILK FLOATS
ALL ELECTRIC, COMMERCIAL VEHICLES. (ANYMORE)
Photo courtesy of kenjonbro: http://www.flickr.com/photos/kenjonbro/4037649210/in/set-72157623026469013

7. ALL-ELECTRIC COMMERCIAL VEHICLES

8. ELECTRIC VEHICLES
•
•
•
•
•
•
•
•
•
•

9. DATA CHALLENGES FROM ELECTRIC VEHICLES
•
•
•
•
•
•
•
•
•

10. ENTER TELEMETRY
•
•
•
•
•
•
•
•
•
•

11. HUGE DATA VOLUMES

12. CURRENT STATS
•
•
•
•
•
•

13. INITIAL MANDATE

14. INITIAL SOLUTION
•
•
•

15. NEW MANDATE
•
•
•
•
•
•
•
•
•
•
•

16. DATA STARTS TO INCREASE
•
•
http://www.flickr.com/photos/robotapocalypse/245508884/

17. http://www.flickr.com/photos/holyoutlaw/5920882576

18. SCALING

19. PROBLEM #0: INSERTS
•
•
•
•
•

20. INITIAL ARCHITECTURE

21. PROBLEM #1: AVAILABILITY

22. PROBLEM #2: CAPACITY
www.flickr.com/photos/eveofdiscovery/3149008295

23. CAPACITY

24. OPTION 1: CLOUD INFRASTRUCTURE
•
•
•
•

25. PROBLEM WITH CLOUD INFRASTRUCTURE
•
•
•

26. SOLUTION #1: MQ
www.flickr.com/photos/gadl/89650415/inphotostream

27. CLOUD BASED MESSAGE QUEUE

28. “THE CLOUD”
ISN’T PERFECT
www.flickr.com/photos/brapps/403257780

29. PROBLEM #3: STORAGE SYSTEM STARTS TO CRAC
http://www.flickr.com/photos/mknott/2855987266

30. SOLUTION: GENIUS DBA & VENDOR
CONSULTANCY
Sam Lambert – DBA Extraordinaire
•
•
•
•
•
•

31. LIVE, REAL-TIME INFORMATION

32. LIVE DATA: PROBLEMS
•
•
•
•
•
•

33. LIVE, REAL TIME INFORMATION: PROBLEM
•
•
•
•
•

34. LIVE DATA: GLOBAL MAP

35. REAL TIME INFORMATION: CONCURRENT
•
•
•
•
•
•

36. RACE CONDITIONS
•
•
•
•
•
•
•

37. LOTS OF DATA: RACE CONDITIONS
•
•
•
•

38. RACE CONDITIONS: PHP & SESSIONS

39. LIVE: STABLE BUT SLOW

40. CACHE THE DATA
•
•
•

41. MEMCACHE
// instantiate
$mc = new Memcache();
// set the memcache server and port
$mc->connect( MEMCACHE_SERVER, MEMCACHE_PORT );
// get data based on a key
$value = $mc->get(‘key’);

42. MEMCACHE FAILOVER

43. CACHING WITHIN LEGACY CODE

44. LAZY LOADING REGISTRY
public function getObject( $key )
{
if( in_array( $key, array_keys( $this->objects ) ) )
{
return $this->objects[$key];
}
elseif( in_array( $key, array_keys( $this->objectSetup ) ) )
{
if( ! is_null( $this->objectSetup[ $key ]['abstract'] ) )
{
require_once( FRAMEWORK_PATH . 'registry/aspects/' . $this-
>objectSetup[ $key ]['folder'] . '/' . $this->objectSetup[ $key]['abstract'] .'.abstract.php' );
}
require_once( FRAMEWORK_PATH . 'registry/aspects/' . $this->objectSetup[ $key ]['folder'] . '/' . $this-
>objectSetup[ $key]['file'] . '.class.php' );
$o = new $this->objectSetup[ $key ]['class']( $this );
$this->storeObject( $o, $key );
return $o;
}
elseif( $key == 'memcache' )
{
// requesting memcache for the first time, instantiate, connect, store and return
$mc = new Memcache();
$mc->connect( MEMCACHE_SERVER, MEMCACHE_PORT );
$this->storeObject( $mc, 'memcache' );
return $mc;
}
}

45. LLR: NOT FIT FOR PURPOSE

46. REAL TIME INFORMATION: # OF REQUESTS
•
•
•

47. RACE CONDITIONS: USE A TEMPLATE ENGINE
•
•
•

48. RACE CONDITIONS: USE A SINGLE ENTRY
POINT
•
•
•
•

49. A LITTLE BREATHING SPACE
•
•
•
•
•
•
•

50. THINGS WERE STILL SLOW
•
•
•
•
•
•

52. GENERATING PERFORMANCE REPORTS
•
•
•
•
•
•
•
•

53. NOSQL
•
•
•

54. REGULAR PROCESSING
•
•
•
•

56. QUERY SPEED BECOMES MORE IMPORTANT
•
•

57. SHARDING
•
•
•

58. WHICH BUCKET?
public function getTableNameFromDate( $date ) {
// we can’t query for future data, so if the date is in the future
// reset it to today
$date = ( $date > date( 'Y-m-d') ) ? date('Y-m-d') : $date;
// get the time in seconds since epoc
$stt = strtotime( $date );
// is the query date since we implemented sharding?
if( $date >= $this->switchOver ) {
// calculate the year this week is from
$year = ( date( 'm', $stt ) == 01 && date( 'W', $stt ) == 52 ) ? date('Y',
$stt ) - 1 : date('Y', $stt );
// add the year and the week number to the table, and return
return ‘data_' . $year . '_' . date('W', $stt );
} else {
// return the legacy table
return 'data';
}
}

59. QUERY MAGIC
private function parseQuery( $sql, $date=null )
{
$date = ( is_null( $date ) ) ? date('Y-m-d') : $date;
return sprintf( $sql, $this->shardHelper->getTableNameFromDate( $date ) );
}

60. BACKUPS & ARCHIVES
•
•
•
•
•
•

61. SHARDING: AN EXCUSE
•
•
•

62. DATA TYPES
•
•
•
•
•

63. INDEX OPTIMISATION
•
•
•

64. QUERIES: OPTIMISATION TIPS
•
•
•

65. CONCURRENT DATABASE CONNECTIONS
•
•

66. REPORTS: DB CONNECTION MANAGEMENT
•
•
•
•
•

67. CONCURRENT DATA PROCESSING
$counter = 1;
foreach( $vehicles as $vehicle )
{
if( ( $counter % $concurrent ) == $instance )
{
$vehiclesToProcess[] = $vehicle;
}
$counter++;
}

68. SYSTEM ARCHITECTURE

69. SYSTEM ARCHITECTURE

70. APPLICATION QUALITY
•
•
•
•

71. DEPLOYMENT
•
•

72. ADDITIONAL POINTS

73. DON’T LOG “EVERYTHING”: EXTRAPOLATE AND
ASSUME
•
•
•
•
•
•
•

74. EXTRAPOLATE AND ASSUME: “INTERLATION”
•
•
•
•

75. INTERLACE
* Add an array to the interlation
public function addArray( $name, $array )
* Get the time that we first receive data in one of our arrays
public function getFirst( $field )
* Get the time that we last received data in any of our arrays
public function getLast( $field )
* Generate the interlaced array
public function generate( $keyField, $valueField )
* Beak the interlaced array down into seperate days
public function dayBreak( $interlationArray )
* Generate an interlaced array and fill for all timestamps within the range of
_first_ to _last_
public function generateAndFill( $keyField, $valueField )
* Populate the new combined array with key fields using the common field
public function populateKeysFromField( $field, $valueField=null )
http://www.michaelpeacock.co.uk/interlation-library

76. ABSTRACTION

77. TECHNICAL TIPS
•
•
•
•

78. WHAT DO WE HAVE NOW?

79. CONCLUSIONS
•
•
•
•
•
•
•
•
•
•

80. Q&A