Hadoop Conference Taiwan 2014 Presentation.

1. Hadoop
BigQuery Connector
Simon Su & Sunny Hu @ MiCloud

2. I am Simon Su
var simon = {};
simon.aboutme = 'http://about.me/peihsinsu';
simon.nodejs = ‘http://opennodes.arecord.us';
simon.googleshare = 'http://gappsnews.blogspot.tw'
simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw';
simon.blog = ‘http://peihsinsu.blogspot.com';
simon.slideshare = ‘http://slideshare.net/peihsinsu/';
simon.email = ‘simonsu.mail@gmail.com’;
simon.say(‘Good luck to everybody!');

3. I am Sunny Hu
var sunny = {};
sunny.aboutme = 'https://plus.google.com/u/0/+sunnyHU/posts';
sunny.email = sunnyhu@mitac.com.tw’;
sunny.language =[‘Java’,’.NET’,’NodeJS’,’SQL’ ]
sunny.skill = [ ‘Project management’,’System Analysis’,
’System design’,’Car ho lan’]
sunny.say(‘寫code太苦悶，心情要sunny');

4. ● We are 蘇 胡 二人組 ...

5. We are MiCloud
● 2011/11 MiCloud Launch
● 2013/2 Google Apps Partner
● 2013/9 Google Cloud Partner
● 2014/4 Google Cloud Launch

9. 緣起
● Dremel (BigQuery) 能提供大量及穩定服務
● 2013, 平均每日服務量: 5,922,000,000 人次
● 2012, 平均每日服務量: 5,134,000,000 人次
● 2011, 平均每日服務量: 4,717,000,000 人次
● 2010, 平均每日服務量: 3,627,000,000 人次
● 2009, 平均每日服務量: 2,610,000,000 人次
● 2008, 平均每日服務量: 1,745,000,000 人次

10. What is the components of Hadoop...
Strategy
MapReduce
HDFS
Your idea for filtering information from the
given datasets
Mass computing power to parallel load and
process the requirements
Persistence storage for parallel access, better
with good performance...

11. You have better choice in Cloud...
Strategy
MapReduce
HDFS
Nothing can replace a good idea…, but fast...
Cloud machines with unlimited resources,
better with lower and scalable pricing...
Object storage services, like: Google Cloud
Storage, AWS S3...

12. ● The fast way run hadoop - docker

13. Google Provide
Resources

14. ● GCE Hadoop Utility

15. ● GCE Cluster Tool - bdutil

17. Before Demo… Prepare
1. Install google_cloud_sdk
2. Install bdutil

18. google cloud sdk
curl https://sdk.cloud.google.com | bash

19. ● Auth the gcloud utility

20. ● Setup default project
● Test configuration….

21. Using bdutil...
https://developers.google.com/hadoop/setting-up-a-hadoop-cluster

22. bdutil scopes
● Design for fast create hadoop cluster
● Quick run a hadoop task
● Quick integrate google’s resources
● Quick clear finished resources

23. Demo start first….

24. ● Config your bdutil env.

25. ● bdutil deploy -e bigquery_env.sh

29. ● Checking the result...

30. ● The Administration console

31. TeraSort
https://www.mapr.com/fr/company/press/mapr-and-google-compute-engine-set-new-world-record-hadoop-terasort

33. You can win the game, too...
…. (skip)

34. BigQuery Connector
https://developers.google.com/hadoop/running-with-bigquery-connector

35. hadoop-w-0 hadoop-m hadoop-w-1

36. Demo start first….

37. Run a BigQuery Connector job...

38. Workflow...
1. Dump sample data from [publicdata:samples.shakespeare]
2. MapReduce to count the word display
3. Update result to BigQuery specific table

39. Look into source code...
● BigQueryInputFormat class
● Input parameters
● Mapper
● BigQueryOutputFormat class
● Output parameters
● Reducer

40. BigQueryInputFormat
● Using a user-specified query to select the appropriate
BigQuery objects.
● Splitting the results of the query evenly among the Hadoop
nodes.
● Parsing the splits into java objects to pass to the mapper

41. Input parameters
● Project Id : GCP project id , eg. hadoop-conf-2014
● Input Table Id :[optional projectId]:[datasetId].[table id]

43. BigqueryOutputFormat Class
● Provides Hadoop with the ability to write JsonObject
values directly into a BigQuery table
● An extension of the Hadoop OutputFormat class

44. Output parameters
● Project Id : GCP project id ,eg. hadoop-conf-2014
● Output Table Id :[optional projectId]:[datasetId].[table id]
● Output Table Schema :[{'name': 'Name','type': 'STRING'},
{'name': 'Number','type': 'INTEGER'}]

46. bdutil house keeping...
https://developers.google.com/hadoop/setting-up-a-hadoop-cluster

47. ● GDamee ovleer - Dteelet e tthhe haedo ohp claustedr oop cluster

48. ● Check project….

49. You cost in this lab...
VM (n1-standard-1) * machines *
hours
$0.070 USD/Hour 24 1
* *

50. Today’s Demo
Using Docker...

51. ● Using google optimized docker container
localhost:~$ gcloud compute instances create simon-docker
> --image https://www.googleapis.com/compute/v1/projects/google-containers/global/images/container-vm-v20140522
> --zone asia-east1-a
> --machine-type f1-micro
localhost:~$ gcloud compute ssh simon-docker
simonsu@simon-docker:~$ sudo docker search bdutil
simonsu@simon-docker:~$ docker run -it peihsinsu/bdutil bash

52. Other connectors
BigQuery connector for Hadoop
$ ./bdutil deploy -e bigquery_env.sh
Datastore connector for Hadoop
$ ./bdutil deploy -e datastore_env.sh
To use both BQ & Datastore
$ ./bdutil deploy -e datastore_env.sh,bigquery_env.sh

53. http://goo.gl/PbHdDc

54. http://micloud.tw

55. http://jsdc-tw.kktix.cc/events/jsdc2014