O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
INTERACTIVE
SPARK IN YOUR
BROWSER
Spark	
  Lightning	
  Night	
  -­‐	
  Hadoop	
  World
September	
  2015
GOAL

OF HUE
WEB INTERFACE FOR ANALYZING DATA
WITH APACHE HADOOP	
  
SIMPLIFY AND INTEGRATE



FREE AND OPEN SOURCE
—> “BI...
SPARK

ARCHITECTURE
Livy
Spark
Spark
Spark
YARN
Backend	
  partWeb	
  part
SPARK

ARCHITECTURE
Livy
Spark
Spark
Spark
YARN
Spark	
  ServerUI
HADOOP NOTEBOOK
UI
• Married	
  with	
  full	
  ecosystem	
  
• File,	
  Job	
  browsers	
  
• Create	
  table	
  wizards	
  
• Any	
  langua...
LIVY SPARK SERVER
Backend
• REST	
  Web	
  server	
  in	
  Scala	
  
• InteracTve	
  Spark	
  Sessions	
  and	
  Batch	
  Jobs	
  
• Type	
  Introsp...
LIVY WEB SERVER

ARCHITECTURE YARN	
  
Master
Spark	
  
Client
YARN

Node
Spark

Interpreter
Spark

Context
YARN

Node
Spa...
LIVY WEB SERVER

ARCHITECTURE
Livy	
  Server
YARN	
  
Master
Scalatra
Spark	
  
Client
Session	
  Manager
Session
YARN

No...
LIVY WEB SERVER

ARCHITECTURE YARN	
  
Master
Spark	
  
Client
YARN

Node
Spark

Interpreter
Spark

Context
YARN

Node
Spa...
LIVY WEB SERVER

ARCHITECTURE YARN	
  
Master
Spark	
  
Client
YARN

Node
Spark

Interpreter
Spark

Context
YARN

Node
Spa...
LIVY WEB SERVER

ARCHITECTURE YARN	
  
Master
Spark	
  
Client
YARN

Node
Spark

Interpreter
Spark

Context
YARN

Node
Spa...
LIVY WEB SERVER

ARCHITECTURE YARN	
  
Master
Spark	
  
Client
YARN

Node
Spark

Interpreter
Spark

Context
YARN

Node
Spa...
LIVY WEB SERVER

ARCHITECTURE YARN	
  
Master
Spark	
  
Client
YARN

Node
Spark

Interpreter
Spark

Context
YARN

Node
Spa...
LIVY WEB SERVER

ARCHITECTURE YARN	
  
Master
Spark	
  
Client
YARN

Node
Spark

Interpreter
Spark

Context
YARN

Node
Spa...
SESSION CREATION

AND EXECUTION
%	
  curl	
  -­‐XPOST	
  localhost:8998/sessions	
  	
  
	
  	
  -­‐d	
  '{"kind":	
  "spa...
DEMO
TIME

• Stability	
  and	
  Scaling	
  
• Security	
  
• iPython/Jupyter	
  interpreter	
  
and	
  file	
  format	
  
COMING SOON
TWITTER
@gethue
USER GROUP
hue-­‐user@
WEBSITE
h[p://gethue.com
LEARN
h[p://learn.gethue.com
THANKS!

Interactive Apache Spark in Your Browser
Interactive Apache Spark in Your Browser
Próximos SlideShares
Carregando em…5
×

Interactive Apache Spark in Your Browser

2.879 visualizações

Publicada em

Presented by Romain Rigaux at Spark Lightning Night 2015

Publicada em: Software

Interactive Apache Spark in Your Browser

  1. 1. INTERACTIVE SPARK IN YOUR BROWSER Spark  Lightning  Night  -­‐  Hadoop  World September  2015
  2. 2. GOAL
 OF HUE WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP   SIMPLIFY AND INTEGRATE
 
 FREE AND OPEN SOURCE —> “BIG NOTEBOOK” FOR HADOOP
  3. 3. SPARK
 ARCHITECTURE Livy Spark Spark Spark YARN Backend  partWeb  part
  4. 4. SPARK
 ARCHITECTURE Livy Spark Spark Spark YARN Spark  ServerUI
  5. 5. HADOOP NOTEBOOK UI
  6. 6. • Married  with  full  ecosystem   • File,  Job  browsers   • Create  table  wizards   • Any  language  (Hive,  Spark...)   • Graphing   • Export/Import/Sharing   • MulT  users   • ImpersonaTon   HADOOP WITH SPARK
 NOTEBOOK
  7. 7. LIVY SPARK SERVER Backend
  8. 8. • REST  Web  server  in  Scala   • InteracTve  Spark  Sessions  and  Batch  Jobs   • Type  IntrospecTon  for  VisualizaTon   • Running  sessions  in  YARN    local   • Backends:  Scala,  Python,  R   • Open  Source:  h[ps://github.com/cloudera/hue/ tree/master/apps/spark/java   • Play  with  Curl  h[p://gethue.com/how-­‐to-­‐use-­‐the-­‐ LIVY
 SPARK SERVER
  9. 9. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker Livy  Server Scalatra Session  Manager Session
  10. 10. LIVY WEB SERVER
 ARCHITECTURE Livy  Server YARN   Master Scalatra Spark   Client Session  Manager Session YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1
  11. 11. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 Livy  Server Scalatra Session  Manager Session
  12. 12. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 Livy  Server Scalatra Session  Manager Session
  13. 13. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 4 Livy  Server Scalatra Session  Manager Session
  14. 14. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 4 5 Livy  Server Scalatra Session  Manager Session
  15. 15. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 2 3 4 5 6 Livy  Server Scalatra Session  Manager Session
  16. 16. LIVY WEB SERVER
 ARCHITECTURE YARN   Master Spark   Client YARN
 Node Spark
 Interpreter Spark
 Context YARN
 Node Spark
 Worker YARN
 Node Spark
 Worker 1 7 2 3 4 5 6 Livy  Server Scalatra Session  Manager Session
  17. 17. SESSION CREATION
 AND EXECUTION %  curl  -­‐XPOST  localhost:8998/sessions        -­‐d  '{"kind":  "spark"}'   {      "id":  0,      "kind":  "spark",      "log":  [...],      "state":  "idle"   }   %  curl  -­‐XPOST  localhost:8998/sessions/0/statements  -­‐d  '{"code":  " {      "id":  0,      "output":  {          "data":  {  "text/plain":  "res0:  Int  =  2"  },          "execution_count":  0,          "status":  "ok"      },      "state":  "available"   }  
  18. 18. DEMO TIME

  19. 19. • Stability  and  Scaling   • Security   • iPython/Jupyter  interpreter   and  file  format   COMING SOON
  20. 20. TWITTER @gethue USER GROUP hue-­‐user@ WEBSITE h[p://gethue.com LEARN h[p://learn.gethue.com THANKS!


×