961万人の食卓を支えるデータ解析

•Transferir como KEY, PDF•

36 gostaram•5,248 visualizações

Tatsuya Sasaki

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

Tecnologia Espiritual

•
•
• Hadoop (Cloudera)

• Elastic MapReduce
•
•

• (@sasata299)

• 2009 8 JOIN

•
• Hadoop

• Hadoop MySQL

•
- GROUP BY

• 7000 …

• (´Д` )

Hadoop
• Google MapReduce OSS

•
-
-
-
-

Hadoop
master ( )

slave ( )

<key,value>
Map
Shufﬂe & Sort

Hadoop
master ( )

slave ( )

<key,value>
Map Reduce
Shufﬂe & Sort

• Hadoop Streaming (Ruby )

• EC2 Cloudera Hadoop
- Cloudera CDH1
- Hadoop 0.18.3

• S3

MySQL → Hadoop
•
• GROUP BY MapReduce
- ( )
- key

• JOIN MapReduce
•

(1) master
master slave scp

(2) S3

S3 slave scp

MySQL vs Hadoop

7000

MySQL Hadoop
MySQL Hadoop

MySQL vs Hadoop

( Д )
7000
30

MySQL Hadoop
MySQL Hadoop

• Hadoop
-
• Hadoop (HADOOP-6254)
- S3
- SocketTimeoutException

• EMR (Elastic MapReduce)
- Amazon Hadoop

• Cloudera CDH2
-

AMI
(Amazon Machine
UP Image)

EMR

CDH2

EMR
BootStrap Action

Step (Hadoop Job)

Job Flow ( )

•
-
- --alive
• AMI
- AMI

- BootStrap Action

•
- mapred.child.java.opts
- streaming

•
-
- ElasticMapReduce-master 5100

Mais conteúdo relacionado

Semelhante a 961万人の食卓を支えるデータ解析

COOKPADでのHadoop利用

Hadoop導入事例 in クックパッド

Amebaサービスのログ解析基盤

Hadoop入門とクラウド利用

Hadoop and mysql by Chris Schneider

Dmitry Makarchuk

Hadoop london

Yahoo Developer Network

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

npinto

Storage and computation is getting cheaper AND easily accessible on demand in the cloud. We now collect and store some really large data sets Eg: user activity logs, genome sequencing, sensory data etc. Hadoop and the ecosystem of projects built around it present simple and easy to use tools for storing and analyzing such large data collections on commodity hardware. Topics Covered * The Hadoop architecture. * Thinking in MapReduce. * Run some sample MapReduce Jobs (using Hadoop Streaming). * Introduce PigLatin, a easy to use data processing language. Speaker Profile: Mahesh Reddy is an Entrepreneur, chasing dreams. Works on large scale crawl and extraction of structured data from the web. He is a graduate frm IIT Kanpur(2000-05) and previously worked at Yahoo! Labs as Research Engineer/Tech Lead on Search and Advertising products.

HadoopThe Hadoop Java Software Framework

ThoughtWorks

Qubole Overview at the Fifth Elephant Conference

Joydeep Sen Sarma

Hadoop Overview & Architecture

EMC

Hadoop Conference Japan 2011 Fall

Ryu Kobayashi

Hadoop Overview kdd2011

Milind Bhandarkar

データ解析技術入門(Hadoop編)

Takumi Asai

サンプルから見るMap reduceコード

Shinpei Ohtani

サンプルから見るMapReduceコード

Shinpei Ohtani

Hadoop

Rajesh Piryani

Brust hadoopecosystem

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

Semelhante a 961万人の食卓を支えるデータ解析 (20)

COOKPADでのHadoop利用

Hadoop導入事例 in クックパッド

Amebaサービスのログ解析基盤

Hadoop入門とクラウド利用

Hadoop and mysql by Chris Schneider

Hadoop london

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

HadoopThe Hadoop Java Software Framework

Qubole Overview at the Fifth Elephant Conference

Hadoop Overview & Architecture

Hadoop Conference Japan 2011 Fall

Hadoop Overview kdd2011

データ解析技術入門(Hadoop編)

サンプルから見るMap reduceコード

サンプルから見るMapReduceコード

Hadoop

Brust hadoopecosystem

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

Mais de Tatsuya Sasaki

からあげエンジニアについて

クックパッドでのemr利用事例

からあげとビーチと私

メタプログラミングでDSLを書こう

NoSQLデータベースが登場した背景と特徴

Tatsuya Sasaki

Hadoopをemr経由で利用する方法

Tatsuya Sasaki

YUI

Tatsuya Sasaki

Mais de Tatsuya Sasaki (7)

からあげエンジニアについて

クックパッドでのemr利用事例

からあげとビーチと私

メタプログラミングでDSLを書こう

NoSQLデータベースが登場した背景と特徴

Hadoopをemr経由で利用する方法

YUI

Último

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

The Raspberry Pi 5 was announced on October 2023. This new version of the popular embedded device comes with a new iteration of Broadcom’s VideoCore GPU platform, and was released with a fully open source driver stack, developed by Igalia. The presentation will discuss some of the major changes required to support this new Video Core iteration, the challenges we faced in the process and the solutions we provided in order to deliver conformant OpenGL ES and Vulkan drivers. The talk will also cover the next steps for the open source Raspberry Pi 5 graphics stack. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://eoss24.sched.com/event/1aBEx

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Igalia

Sara Mae O’Brien Scott and Tatiana Baquero Cakici, Senior Consultants at Enterprise Knowledge (EK), presented “AI Fast Track to Search-Focused AI Solutions” at the Information Architecture Conference (IAC24) that took place on April 11, 2024 in Seattle, WA. In their presentation, O’Brien-Scott and Cakici focused on what Enterprise AI is, why it is important, and what it takes to empower organizations to get started on a search-based AI journey and stay on track. The presentation explored the complexities of enterprise search challenges and how IA principles can be leveraged to provide AI solutions through the use of a semantic layer. O’Brien-Scott and Cakici showcased a case study where a taxonomy, an ontology, and a knowledge graph were used to structure content at a healthcare workforce solutions organization, providing personalized content recommendations and increasing content findability. In this session, participants gained insights about the following: Most common types of AI categories and use cases; Recommended steps to design and implement taxonomies and ontologies, ensuring they evolve effectively and support the organization’s search objectives; Taxonomy and ontology design considerations and best practices; Real-world AI applications that illustrated the value of taxonomies, ontologies, and knowledge graphs; and Tools, roles, and skills to design and implement AI-powered search solutions.

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Enterprise Knowledge

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today) Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers. Core Insights Include: Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base. Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes. Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies. Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively. This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Earley Information Science

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

961万人の食卓を支えるデータ解析

1. 961

2. • • • Hadoop (Cloudera) • Elastic MapReduce • •

3. • • • Hadoop (Cloudera) • Elastic MapReduce • •

4. • (@sasata299) • 2009 8 JOIN • • Hadoop

5. • • • Hadoop (Cloudera) • Elastic MapReduce • •

7. • 961 • 30 3 1 • - -

8. • • ( , , ...) - - ( , , , …) -

10.

11.

12.

13.

14. • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )

15. • MySQL - • - -

16.

17. • • • Hadoop (Cloudera) • Elastic MapReduce • •

18. Hadoop • Google MapReduce OSS • - - - -

19. Hadoop master ( ) slave ( )

20. Hadoop master ( ) slave ( ) Map

21. Hadoop master ( ) slave ( ) <key,value> Map Shufﬂe & Sort

22. Hadoop master ( ) slave ( ) <key,value> Map Reduce Shufﬂe & Sort

23. • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3

24. MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •

25. (1) master (2) S3

26. (1) master (2) S3

27. (1) master master slave scp (2) S3

28. (1) master master slave scp (2) S3 S3 slave scp

29. MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop

30. MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop

31. Hadoop++ ←Hadoop ↓MySQL

32. • • • Hadoop (Cloudera) • Elastic MapReduce • •

33. • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException

34. • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -

35. AMI (Amazon Machine UP Image) EMR CDH2

36. AMI (Amazon Machine UP Image) EMR CDH2

37. EMR Job Flow ( )

38. EMR BootStrap Action Job Flow ( )

39. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

40. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

41. • - - --alive • AMI - AMI - BootStrap Action

42.

43. Created job ﬂow j-8IXS98OW1WEE ID

44. Hadoop

45.

46. • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100

47. • • • Hadoop (Cloudera) • Elastic MapReduce • •

48. • Map - • Reduce - key Reduce -

49. UU Map Reduce

50. UU Map ID Reduce

51. UU Map Reduce ID

52. UU Map Reduce ID

53. Map Reduce

54. Map ID key Reduce

55. Map Reduce key Reduce

56. Map 100 100 Reduce key Reduce

57. × Map 100 × 100 Reduce key Reduce

58. × Map 100 ×100 Reduce Reduce key sort

59. × Map 100 ×100 Reduce Reduce key sort

60. Hadoop • - Hadoop

61. Hadoop • - Hadoop

62. Hadoop • - Hadoop

63. Hadoop • - Hadoop

64. Hadoop • - Hadoop

65. • • • Hadoop (Cloudera) • Elastic MapReduce • •

66. • Hadoop - - - Reduce

961万人の食卓を支えるデータ解析

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a 961万人の食卓を支えるデータ解析

Semelhante a 961万人の食卓を支えるデータ解析 (20)

Mais de Tatsuya Sasaki

Mais de Tatsuya Sasaki (7)

Último

Último (20)

961万人の食卓を支えるデータ解析

Notas do Editor