O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Spark Day 2017- Spark 의 과거, 현재, 미래

874 visualizações

Publicada em

한국 스파크 사용자 모임 Spark Day 2017 의 발표자료입니다.

Publicada em: Tecnologia
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Spark Day 2017- Spark 의 과거, 현재, 미래

  1. 1. https://github.com/apache/spark/commits/master?after=9e50a1d37a4cf0c34e20a7c1a910ceaff41535a2+1574&author=mateiz http://blog.madhukaraphatak.com/history-of-spark/
  2. 2. package org.myorg; import java.io.IOException; import java.util.*; import org.apache.hadoop.fs.Path; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; import org.apache.hadoop.util.*; public class WordCount { public static class Map extends MapReduceBase implements Mapper<Lon private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, OutputCollector<Tex String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); output.collect(word, one); } } } public static class Reduce extends MapReduceBase implements Reducer public void reduce(Text key, Iterator values, OutputCollector<Tex int sum = 0; while (values.hasNext()) { sum += values.next().get(); } output.collect(key, new IntWritable(sum)); } } public static void main(String[] args) throws Exception { JobConf conf = new JobConf(WordCount.class); conf.setJobName("wordcount"); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(IntWritable.class); conf.setMapperClass(Map.class); conf.setCombinerClass(Reduce.class); conf.setReducerClass(Reduce.class); conf.setInputFormat(TextInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); JobClient.runJob(conf); } } wordcount
  3. 3. file.flatMap(line => line.split(" “)) .map(word => (word, 1)) .reduceByKey(_ + _) wordcount Logistic regression in Hadoop and Spark
  4. 4. 2016 Spark survey
  5. 5. 2016 Spark survey
  6. 6. 2016 Spark survey
  7. 7. 2016 Spark survey https://www.slideshare.net/SparkSummit/trends-for-big-data-and-apache-spark-in-2017-by-matei-zaharia
  8. 8. 2016 Spark survey
  9. 9. 2016 Spark survey
  10. 10. 2016 Spark survey
  11. 11. 2016 Spark survey
  12. 12. Learning 52 Streaming 39 Machine 38 Deep 33 Science 30 Analyics 27 Scale 21 Developer 20 Enterprise 20 Ecosystem 20 Research 13 Applications 9 Processing 8 Pipeline 8 Platform 7
  13. 13. 2017 Spark summithttps://www.slideshare.net/SparkSummit/trends-for-big-data-and-apache-spark-in-2017-by-matei-zaharia
  14. 14. 2017 Spark summithttps://www.slideshare.net/SparkSummit/trends-for-big-data-and-apache-spark-in-2017-by-matei-zaharia
  15. 15. • http://blog.madhukaraphatak.com/history-of-spark/ • http://cdn2.hubspot.net/hubfs/438089/DataBricks_Surveys_-_Content/2016_Spark_Survey/ 2016_Spark_Infographic.pdf • http://cdn2.hubspot.net/hubfs/438089/DataBricks_Surveys_-_Content/Spark-Survey-2015- Infographic.pdf

×