O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Google Cloud: Data Analysis and Machine Learningn Technologies

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Carregando em…3
×

Confira estes a seguir

1 de 94 Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Google Cloud: Data Analysis and Machine Learningn Technologies (20)

Anúncio

Mais de Andrés Leonardo Martinez Ortiz (20)

Mais recentes (20)

Anúncio

Google Cloud: Data Analysis and Machine Learningn Technologies

  1. 1. Google Data Analysis Technologies Andres L. Martinez a.k.a almo Google Developer Relations Manager @davilagrau
  2. 2. Contents ● BigQuery ● Data Analysis ● Machine Learning ● Cloud ML ● Google Assistant ● Cloud ML APIs
  3. 3. BigQuery
  4. 4. BigQuery: 100% serverless data warehouse Google BigQuery Fully Managed and Serverless Google Cloud’s Enterprise Data Warehouse for Analytics Petabyte-Scale and Fast Convenience of SQL Encrypted, Durable and Highly Available
  5. 5. BigQuery is a great choice because: Near-real time analysis of massive datasets No-ops; Pay for use Durable (replicated), inexpensive storage Immutable audit logs Mashing up different datasets to derive insights
  6. 6. 10 B rows Sample query - Processes over 10 billion rows in less than 10 seconds SELECT language, SUM(views) as views FROM wikipedia_benchmark.Wiki10B WHERE regexp_match(title,"G.*o.*o.*g") GROUP by language ORDER by views DESC
  7. 7. BigQuery = Massively Parallel Processing query with the petabit network and thousands of servers SQL QueryPetabit Network BigQuery Storage Compute Streaming Ingest Fast Batch Load DataFlow DataProc
  8. 8. Load data using bq tool, web UI, or API Create, append or overwrite table CSV, JSON or AVRO format
  9. 9. Data Studio
  10. 10. For business analysts Beautiful reports Drive-based collaboration experience No technical expertise required Connects to many sources: BigQuery, Adwords, Google Analytics, Google Sheets, YouTube Analytics, etc.
  11. 11. Integrating with Google Data Studio 1 Navigate to DataStudio to create a new dashboard 2 Create a new Data Source 3 Select the type of Data Source to use 4 Authorize
  12. 12. Person A Person B Collaboration
  13. 13. Sample Dashboard
  14. 14. Machine Learning
  15. 15. Why Machine Learning? ★ Allows to solve problems we don’t have exact solution for. ○ E.g. recommendations, predictions, clustering. ★ Given y = F(X), where we observe y, we can estimate F. ★ Becomes better with more data ○ when hard coded solution usually becomes worse with more code :)
  16. 16. Google Products Using Machine Learning
  17. 17. Big Data Challenges ★ Variety of data ★ Learning many things at once ★ Small data where matters
  18. 18. Why is this time different?Photo by Emily Morter on Unsplash
  19. 19. Photo by Sharon Pittaway on Unsplash Innovation
  20. 20. Photo by Alex Holyoake on Unsplash
  21. 21. Fast{er} Photo by Josh Calabrese on Unsplash
  22. 22. Focus on the user
  23. 23. Deep Learning Input Out
  24. 24. Input Out Deep Learning for Perception Tasks
  25. 25. Input Out Deep Learning for Perception Tasks
  26. 26. Input Out Deep Learning for Perception Tasks
  27. 27. Input Out Deep Learning for Perception Tasks
  28. 28. Input Out Deep Learning for Perception Tasks
  29. 29. Deep Learning combines many components Predictions GoogLeNet Recurrent Neural Network
  30. 30. The Machine Learning Spectrum TensorFlow Cloud Machine Learning Machine Learning APIs BYOML skills (Friendly Machine Learning) Pre-packaged ML
  31. 31. TensorFlow Ecosystem Researchers Developers Data Scientists
  32. 32. TensorFlow Core Execution Engine CPU GPU Android iOS ... C++ FrontendPython Frontend ...
  33. 33. TensorFlow: Google backed ★ Google supported (growing army of engineers are working on improving it). ★ Used in 100s of products across Google
  34. 34. Simple example: How can I use this?
  35. 35. Predictive Example Going to deep neural network is easy:
  36. 36. Understanding Images
  37. 37. Image Classification
  38. 38. Scaling Out TensorFlow scales with number of Machines. You can use Google Cloud ML or Docker containers in VMs. https://arxiv.org/abs/1604.00981
  39. 39. TensorFlow Serving: Serving models in production Open Source project. Check it out: http://github.com/tenso rflow/serving
  40. 40. Google Assistant
  41. 41. Voice Kit
  42. 42. Google Home — voice-activated speaker powered The Google Assistant — A conversation between you and Google that helps you get more done in your world. Actions on Google — How developers can extend the assistant (via Conversation Actions)
  43. 43. Direct Actions
  44. 44. Conversation Actions
  45. 45. Use machine learning to understand what users are saying
  46. 46. “Ok Google, talk to personal chef” Conversation API, Actions SDK Invoke “personal chef” action “Sure, here’s personal chef. Hi, I’m your personal chef, what are you in the mood for?” Speech to Text “What protein would you like to use?” “Well, it’s kind of cold outside, so I’d like something to …” Text to Speech “Sure, here’s your personal chef” Speech to Text, NLP, Knowledge Graph, ML Ranking, User Profile, Text to Speech
  47. 47. APIs
  48. 48. Confidential & ProprietaryGoogle Cloud Platform 53 So…. Why APIs? { Google Cloud Platform } 1. We want to offer businesses the tools to differentiate by offering a powerful set of APIs that enable apps to see, hear and understand the world 2. Reduce your Time to Market (TMM) when launching your next-generation app 3. Provide you easy access to machine learning technology to give any developer the freedom to work in the language and tools they want 4. Provide virtually limitless scalability to your application without needing to manage back-end servers running deep learning
  49. 49. Pre-Trained Machine Learning Models Fully trained ML models from Google Cloud that allow a general developer to take advantage of rich machine learning capabilities with simple REST based services.
  50. 50. Introducing Cloud Natural Language API Sentiment analysis and entity recognition for text
  51. 51. Confidential & ProprietaryGoogle Cloud Platform 56 Features Extract sentence, identify parts of speech and create dependency parse trees for each sentence Identify entities and label by types such as person, organization, location, events, products and media Understand the overall sentiment of a block of text Access via REST API. Text can be uploaded in the request or integrated with Google Cloud Storage Syntax Analysis Entity Recognition Sentiment Analysis Integrated REST API
  52. 52. Cloud Vision API Insight from images with our powerful Cloud Vision API
  53. 53. Confidential & ProprietaryGoogle Cloud Platform 58 Faces: Faces, facial landmarks, emotions OCR: Read and extract text, with support for > 10 languages Photo credit Getty Images Label: Detect entities from furniture to transportation Logos: Identify product logos Landmarks & Image Properties Detect landmarks & dominant color of image Safe Search: Detect explicit content - adult, violent, medical and spoof Cloud Vision API Call API from anywhere, with support for embeddable images, and Google Cloud Storage
  54. 54. Use case
  55. 55. Let’s Party
  56. 56. Party planning ● Finding people @Twitter ● Cloud Vision API ● Custom classifier (k-means)
  57. 57. Google Cloud Console We need to have access so we can add hash tag to intro slide
  58. 58. Google Cloud Console
  59. 59. Show Me the code!
  60. 60. Main shellplus_contacts = get_plus_contacts() print "Processing %d contacts" % len(plus_contacts) for plus_id in plus_contacts: plus_profile = get_plus_profile(plus_id) image_uri = plus_profile['image']['url'].replace("?sz=50","?sz=250") image_data = analyze_img(image_uri) if image_data is not None: print(image_uri) if 'labelAnnotations' in image_data['responses'][0]: for label in image_data['responses'][0]['labelAnnotations']: print label['description']; label['score']; image_uri
  61. 61. get_plus_contacts: oAuth storage = Storage('/home/almo/dev/keys/ex1/oAuth_credentials.dat') credentials = storage.get() if credentials is None or credentials.invalid: PEOPLE_API='https://www.googleapis.com/auth/contacts.readonly' flow = flow_from_clientsecrets('/home/almo/dev/keys/ex1/oAuth_key.json', scope=[PEOPLE_API]) credentials = run_flow(flow, storage) http = credentials.authorize(httplib2.Http()) service = build('people','v1',http=http) request = service.people().connections().list(resourceName='people/me', pageSize=500)
  62. 62. analyze_image api_key = json.load(open('/home/almo/dev/keys/ex1/api_key.json'))['api_key'] service = discovery.build('vision','v1',developerKey=api_key) service_request = service.images().annotate(body={ 'requests': [{ 'image': { 'content': image_content.decode('UTF-8') }, 'features': [{ 'type': 'LABEL_DETECTION', 'maxResults': 3 }] }] }
  63. 63. Data
  64. 64. face; 0,92830354; https://lh3.googleusercontent.com/-c3M1gn6ougg/AAAAAAAAAAI/AAAAAAAAAds/cTIrpGhktfw/photo.jpg?sz=250 text; 0,93046468; https://lh4.googleusercontent.com/-GFVyrVlgMy4/AAAAAAAAAAI/AAAAAAAAABE/u3xVd9eJgf8/photo.jpg?sz=250 font; 0,85384184; https://lh4.googleusercontent.com/-GFVyrVlgMy4/AAAAAAAAAAI/AAAAAAAAABE/u3xVd9eJgf8/photo.jpg?sz=250 line; 0,70535356; https://lh4.googleusercontent.com/-GFVyrVlgMy4/AAAAAAAAAAI/AAAAAAAAABE/u3xVd9eJgf8/photo.jpg?sz=250 eyebrow; 0,98022038; https://lh5.googleusercontent.com/-5c9gdP9nX9M/AAAAAAAAAAI/AAAAAAAAGt4/FoZEEVA8F68/photo.jpg?sz=250 hair; 0,96653992; https://lh5.googleusercontent.com/-5c9gdP9nX9M/AAAAAAAAAAI/AAAAAAAAGt4/FoZEEVA8F68/photo.jpg?sz=250 face; 0,95101357; https://lh5.googleusercontent.com/-5c9gdP9nX9M/AAAAAAAAAAI/AAAAAAAAGt4/FoZEEVA8F68/photo.jpg?sz=250 person; 0,92170084; https://lh4.googleusercontent.com/-yVWpXcqQfXU/AAAAAAAAAAI/AAAAAAAAB5w/rqxRrJHgk_0/photo.jpg?sz=250 news; 0,63342041; https://lh4.googleusercontent.com/-yVWpXcqQfXU/AAAAAAAAAAI/AAAAAAAAB5w/rqxRrJHgk_0/photo.jpg?sz=250 professional; 0,61274487; https://lh4.googleusercontent.com/-yVWpXcqQfXU/AAAAAAAAAAI/AAAAAAAAB5w/rqxRrJHgk_0/photo.jpg?sz=250 drawer; 0,80023241; https://lh6.googleusercontent.com/-Qf9SSsIUktA/AAAAAAAAAAI/AAAAAAAAABg/u6zPUNXCYFs/photo.jpg?sz=250 furniture; 0,79278195; https://lh6.googleusercontent.com/-Qf9SSsIUktA/AAAAAAAAAAI/AAAAAAAAABg/u6zPUNXCYFs/photo.jpg?sz=250 product; 0,76023591; https://lh6.googleusercontent.com/-Qf9SSsIUktA/AAAAAAAAAAI/AAAAAAAAABg/u6zPUNXCYFs/photo.jpg?sz=250 eyewear; 0,97702742; https://lh3.googleusercontent.com/-ihQNk3ewmzQ/AAAAAAAAAAI/AAAAAAAAAMk/EEEylEyriNE/photo.jpg?sz=250 hair; 0,96766639; https://lh3.googleusercontent.com/-ihQNk3ewmzQ/AAAAAAAAAAI/AAAAAAAAAMk/EEEylEyriNE/photo.jpg?sz=250 sunglasses; 0,96445274; https://lh3.googleusercontent.com/-ihQNk3ewmzQ/AAAAAAAAAAI/AAAAAAAAAMk/EEEylEyriNE/photo.jpg?sz=250 person; 0,92747426; https://lh4.googleusercontent.com/--_BxhkQPYfA/AAAAAAAAAAI/AAAAAAAAACA/1pN6-Chy8EI/photo.jpg?sz=250 person; 0,96007371; https://lh3.googleusercontent.com/-sX8l_lv_-7w/AAAAAAAAAAI/AAAAAAAAAPU/ApQpBMPbcdc/photo.jpg?sz=250 face; 0,95332307; https://lh3.googleusercontent.com/-sX8l_lv_-7w/AAAAAAAAAAI/AAAAAAAAAPU/ApQpBMPbcdc/photo.jpg?sz=250 Raw Data
  65. 65. 160 different labels Max Freq.: 200 Min Freq. : 1
  66. 66. person 200 0,9320951099 hair 140 0,9609928544 face 139 0,9489352931 font 136 0,7606724908 text 130 0,925080287 blue 114 0,9112923658 facial hair 36 0,8802876539 nose 34 0,8859786603 profession 30 0,569073382 hairstyle 25 0,7532089968
  67. 67. cartoon 23 0,8588066957 professional 23 0,6149233535 glasses 20 0,8234816515 facial expression 14 0,9502550086 eyebrow 12 0,9559630675 black and white 11 0,9199305709 eyewear 11 0,9767648145 logo 11 0,7749610755 head 9 0,7432496333 clothing 7 0,9151270214
  68. 68. [{}]
  69. 69. Planning our next Party
  70. 70. "hair", 0.9559916, "person", 0.94347906, "face", 0.92830354 "text", 0.9304647, "font", 0.85384184, "line", 0.70535356 "eyebrow", 0.9802204, "hair", 0.9665399, "face", 0.9510135 "person", 0.92170084, "news", 0.63342035, "professional", 0.61274487 "drawer", 0.8002325, "furniture", 0.792782, "product", 0.760235 "eyewear", 0.9770274, "hair", 0.9676664, "sunglasses", 0.96445274 "person", 0.92747426, "https://lh4.googleusercontent.com/--_BxhkQPYfA/AAAAAAAAAAI/AAAAAAAAACA/1pN6-Chy8EI/photo.jpg ?sz=250" "green", 0.9307698, "text", 0.92834556, "font", 0.8631033 "hair", 0.98155975, "face", 0.95545304, "eyebrow", 0.93590355 "face", 0.9523797, "person", 0.94760686, "hair", 0.94507515 "hair", 0.9731342, "face", 0.94925183, "person", 0.9371813 "hair", 0.94741917, "person", 0.9436425, "hairstyle", 0.7414854 "person", 0.925232, "people", 0.9086431, "male", 0.83032143 "person", 0.95530343, "face", 0.94757956, "nose", 0.86752254 "face", 0.96074444, "hair", 0.9606222, "eyebrow", 0.9451414 "face", 0.9664352, "hair", 0.9561741, "nose", 0.9222636 "phenomenon", 0.94444287, "celestial event", 0.53744316, "aurora", 0.52995497 "face", 0.9625666, "hair", 0.9514838, "facial expression", 0.94977105 "product", 0.80306137, "font", 0.77923214, "logo", 0.69078964 "black and white", 0.9267871, "person", 0.8998944, "photography", 0.8296365
  71. 71. Training mode!
  72. 72. “invited”, "hair", 0.9559916, "person", 0.94347906, "face", 0.92830354 “excluded”, "text", 0.9304647, "font", 0.85384184, "line", 0.70535356 “excluded”, "eyebrow", 0.9802204, "hair", 0.9665399, "face", 0.9510135 “invited”, "person", 0.92170084, "news", 0.63342035, "professional", 0.61274487 “excluded”, "drawer", 0.8002325, "furniture", 0.792782, "product", 0.760235 “excluded”, "eyewear", 0.9770274, "hair", 0.9676664, "sunglasses", 0.96445274 “excluded”, "green", 0.9307698, "text", 0.92834556, "font", 0.8631033 “excluded”, "hair", 0.98155975, "face", 0.95545304, "eyebrow", 0.93590355 “invited”, "face", 0.9523797, "person", 0.94760686, "hair", 0.94507515 “invited”, "hair", 0.9731342, "face", 0.94925183, "person", 0.9371813 “invited”, "hair", 0.94741917, "person", 0.9436425, "hairstyle", 0.7414854 “invited”, "person", 0.925232, "people", 0.9086431, "male", 0.83032143 “invited”, "person", 0.95530343, "face", 0.94757956, "nose", 0.86752254 “excluded”, "face", 0.96074444, "hair", 0.9606222, "eyebrow", 0.9451414 “excluded”, "face", 0.9664352, "hair", 0.9561741, "nose", 0.9222636
  73. 73. Prediction Mode!
  74. 74. "hair", 0.9559916, "person", 0.94347906, "face", 0.92830354 “invited” "text", 0.9304647, "font", 0.85384184, "line", 0.70535356 “excluded” "eyebrow", 0.9802204, "hair", 0.9665399, "face", 0.9510135 “excluded” "person", 0.92170084, "news", 0.63342035, "professional", 0.61274487 “invited” "drawer", 0.8002325, "furniture", 0.792782, "product", 0.760235 “excluded” "eyewear", 0.9770274, "hair", 0.9676664, "sunglasses", 0.96445274 “excluded” "green", 0.9307698, "text", 0.92834556, "font", 0.8631033 “excluded” "hair", 0.98155975, "face", 0.95545304, "eyebrow", 0.93590355 “excluded” "face", 0.9523797, "person", 0.94760686, "hair", 0.94507515 “invited” "hair", 0.9731342, "face", 0.94925183, "person", 0.9371813 “invited” "hair", 0.94741917, "person", 0.9436425, "hairstyle", 0.741485 4 “invited” "person", 0.925232, "people", 0.9086431, "male", 0.83032143 “invited” "person", 0.95530343, "face", 0.94757956, "nose", 0.86752254 “invited” "face", 0.96074444, "hair", 0.9606222, "eyebrow", 0.9451414 “excluded” "face", 0.9664352, "hair", 0.9561741, "nose", 0.9222636 “excluded”
  75. 75. Beyond AI
  76. 76. Machine Learning find new ways for our data center to save energy
  77. 77. Global Fishing Watch (GFW)
  78. 78. A.I. Experiments
  79. 79. Progress of AI and Machine Learning https://goo.gl/FWWBv2
  80. 80. Photo by Gerome Viavant on Unsplash … And the future?
  81. 81. Thank you! Andres L. Martinez a.k.a almo Google Developer Relations Manager @davilagrau

×