Apache Hadoop On-Demand on Cloud Infrastructure at Java2Days

•Transferir como KEY, PDF•

2 gostaram•1,530 visualizações

Apache Hadoop runs best on bare metal but there are many cases when it makes sense to run clusters on cloud infrastructure. In this talk I discuss advantages, disadvantages and common problems. At Axemblr.com our mission is to make the process of provisioning a fully functional cluster on any cloud provider as simple as possible. We are building on Cloudera Manager and the API we provide is fully compatible with Amazon Elastic Map Reduce. Get in touch with us at hello@axemblr.com

Tecnologia

Apache™ Hadoop® On-Demand
on Cloud Infrastructure

Why? How? Tools?

Andrei Savu @ Cloud2Days 2012
asavu@axemblr.com

Me
• Co-Founder of Axemblr.com
• Co-Organiser of Bucharest JUG (bjug.ro)
• OSS contributor (Apache Whirr, jclouds)
• Worked at Adobe, Facebook

• Connect with me on LinkedIn

Outline
• Quick Introduction
• 7 Reasons Why
• 4 Reasons Why Not
• How (practical lessons)
• OSS Projects & Resources
• Q&A

Apache™ Hadoop®

• Stack of Components (HDFS,YARN, MR,
Hive, Pig, Mahout etc.)
• Distributed Data Storage & Processing
• Designed to Scale to 1000s nodes
• Highly available at application level

Useful for
• Click stream analysis, transactions etc.
• General Business Intelligence
• Advanced Analytics
• Machine Learning (data driven features)
and many more (data crunching)

IaaS
• Infrastructure As A Service
• Programatic access to virtual machines,
servers, storage, load balancers, network
etc.
• PAYG. Good hardware utilisation. Flexible
• Public & Private

IaaS + Hadoop = HaaS
Hadoop-As-A-Service
Parallel Data Processing as a Service

#1 Data Locality
“Work near Data” in the same Cloud

#2 Low Usage
Nightly processing. Short lived clusters. Development

#3 CAPEX
No upfront investment requirements

#4 Multi-Tenancy
Enforced by IaaS mechanisms (hypervisor & network)

#5 Security
VPNs, VLANS, Authentication, ACLs

#6 Easy to Expand
Standard instances types. “Inﬁnite” capacity.
Rapid provisioning.

#7 Self-Service Access
Increase Team Agility. Fast experiments.

#1 Low I/O Performance
Requires more machines for the same workload

#2 Assumes Static Infrastructure
Scheduling & Recovery. Works ﬁne for
short-lived clusters with a ﬁxed size.

#3 Difﬁcult Access to Legacy
Systems
Like databases or log ﬁles

#4 Topology Awareness
Hidden by the IaaS layer

How?
Provisioning. Conﬁguration.
Workﬂow Management. Ongoing Monitoring

Provisioning
Not trivial for 10s or 100s of VMs
Should be: persistent, granular, idempotent
Each provider is like a snowﬂake (normalisation)

Conﬁguration
Usual suspectes: Puppet, Chef, Bash
Or ofﬂoad: Cloudera Manager, Ambari (HPD)

Workﬂow Management
In-House, Oozie, Azkaban
API Integration

Ongoing Monitoring
Automatic repair by reacting to IaaS events
Ganglia, CloudWatch etc.

Tools?
Emerging Market. Growth determined
by rate of cloud adoption.

OSS Projects #1

• Apache Whirr
http://whirr.apache.org/

• Juju (Ubuntu)
https://juju.ubuntu.com/

OSS Projects #2

• Pallet
http://palletops.com/

• Serengeti
http://serengeti.cloudfoundry.com/

Resources
• Hadoop in Cloud Infrastructure (Steve
Loughran)
http://steveloughran.blogspot.ro/2012/03/
hadoop-in-cloud-infrastructures.html
• What factors justify the use of Hadoop
http://redmonk.com/sogrady/2011/01/13/
apache-hadoop/

Thanks! Questions?
Andrei Savu / asavu@axemblr.com
Twitter: @andreisavu

Mais conteúdo relacionado

Mais de Andrei Savu

Recap on AWS Lambda after re:Invent 2015

Andrei Savu

One Hadoop, Multiple Clouds - NYC Big Data Meetup

Andrei Savu

Introducing Cloudera Director at Big Data Bash

Andrei Savu

APIs & Underlying Protocols #APICraftSF

Andrei Savu

Nowadays we've got all the tools we need to spin-up and tear-down clusters with hundreds of nodes in minutes and this puts more pressure on the tools we use to configure and monitor our applications. This challenge is even more interesting when we have to deal with long running distributed data storage and processing systems like Hadoop. In this talk we will look into some of the challenges we need to deal with when creating and managing Hadoop clusters in AWS, we will discuss improvement opportunities in monitoring (e.g. detecting and dealing with instance failure, resource contention & noisy neighbors) and a bit about the future and how we should go about disconnecting workload dispatch from cluster lifecycle.

Challenges for running Hadoop on AWS - AdvancedAWS Meetup

Andrei Savu

Cloud as a Data Platform

Andrei Savu

Apache Provisionr (incubating) - Bucharest JUG 10

Andrei Savu

Creating pools of Virtual Machines - ApacheCon NA 2013

Andrei Savu

Data Scientist Toolbox

Andrei Savu

Simple Service for Managing Pools of 10s or 100s of Virtual Machines With Provisionr we want to solve the problem of cloud portability by hiding completely the API and only focusing on building a cluster that matches the same set of assumptions on all clouds, assumptions like: running a specific operating system (e.g. Ubuntu LTS), having the same set of pre-installed packages and binaries, sane dns settings (forward & reverse ip resolution - as needed for Hadoop), ntp settings, networking settings, ssh admin access, vpn access etc.

Axemblr Provisionr 0.3.x Overview

Andrei Savu

2012 in Review - Bucharest JUG

Andrei Savu

Metrics for Web Applications - Netcamp 2012

Andrei Savu

Counters with Riak on Amazon EC2 at Hackover

Andrei Savu

Simple REST with Dropwizard

Andrei Savu

Guava Overview Part 2 Bucharest JUG #2

Andrei Savu

Guava Overview. Part 1 @ Bucharest JUG #1

Andrei Savu

Polyglot Persistence & Big Data in the Cloud

Andrei Savu

Building a Great Team in Open Source - Open Agile 2011

Andrei Savu

Apache Whirr

Andrei Savu

Automated Testing for Web Applications - Wurbe #36

Andrei Savu

Mais de Andrei Savu (20)

Recap on AWS Lambda after re:Invent 2015

One Hadoop, Multiple Clouds - NYC Big Data Meetup

Introducing Cloudera Director at Big Data Bash

APIs & Underlying Protocols #APICraftSF

Challenges for running Hadoop on AWS - AdvancedAWS Meetup

Cloud as a Data Platform

Apache Provisionr (incubating) - Bucharest JUG 10

Creating pools of Virtual Machines - ApacheCon NA 2013

Data Scientist Toolbox

Axemblr Provisionr 0.3.x Overview

2012 in Review - Bucharest JUG

Metrics for Web Applications - Netcamp 2012

Counters with Riak on Amazon EC2 at Hackover

Simple REST with Dropwizard

Guava Overview Part 2 Bucharest JUG #2

Guava Overview. Part 1 @ Bucharest JUG #1

Polyglot Persistence & Big Data in the Cloud

Building a Great Team in Open Source - Open Agile 2011

Apache Whirr

Automated Testing for Web Applications - Wurbe #36

Último

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

Apache Hadoop On-Demand on Cloud Infrastructure at Java2Days

1. Apache™ Hadoop® On-Demand on Cloud Infrastructure Why? How? Tools? Andrei Savu @ Cloud2Days 2012 asavu@axemblr.com

2. Me • Co-Founder of Axemblr.com • Co-Organiser of Bucharest JUG (bjug.ro) • OSS contributor (Apache Whirr, jclouds) • Worked at Adobe, Facebook • Connect with me on LinkedIn

3. Outline • Quick Introduction • 7 Reasons Why • 4 Reasons Why Not • How (practical lessons) • OSS Projects & Resources • Q&A

4. Apache™ Hadoop® • Stack of Components (HDFS,YARN, MR, Hive, Pig, Mahout etc.) • Distributed Data Storage & Processing • Designed to Scale to 1000s nodes • Highly available at application level

5. Useful for • Click stream analysis, transactions etc. • General Business Intelligence • Advanced Analytics • Machine Learning (data driven features) and many more (data crunching)

6. IaaS • Infrastructure As A Service • Programatic access to virtual machines, servers, storage, load balancers, network etc. • PAYG. Good hardware utilisation. Flexible • Public & Private

7. IaaS + Hadoop = HaaS Hadoop-As-A-Service Parallel Data Processing as a Service

8. 7 Reasons Why

9. #1 Data Locality “Work near Data” in the same Cloud

10. #2 Low Usage Nightly processing. Short lived clusters. Development

11. #3 CAPEX No upfront investment requirements

12. #4 Multi-Tenancy Enforced by IaaS mechanisms (hypervisor & network)

13. #5 Security VPNs, VLANS, Authentication, ACLs

14. #6 Easy to Expand Standard instances types. “Inﬁnite” capacity. Rapid provisioning.

15. #7 Self-Service Access Increase Team Agility. Fast experiments.

16. 4 Reasons Why Not

17. #1 Low I/O Performance Requires more machines for the same workload

18. #2 Assumes Static Infrastructure Scheduling & Recovery. Works ﬁne for short-lived clusters with a ﬁxed size.

19. #3 Difﬁcult Access to Legacy Systems Like databases or log ﬁles

20. #4 Topology Awareness Hidden by the IaaS layer

21. How? Provisioning. Conﬁguration. Workﬂow Management. Ongoing Monitoring

22. Provisioning Not trivial for 10s or 100s of VMs Should be: persistent, granular, idempotent Each provider is like a snowﬂake (normalisation)

23. Conﬁguration Usual suspectes: Puppet, Chef, Bash Or ofﬂoad: Cloudera Manager, Ambari (HPD)

24. Workﬂow Management In-House, Oozie, Azkaban API Integration

25. Ongoing Monitoring Automatic repair by reacting to IaaS events Ganglia, CloudWatch etc.

26. Tools? Emerging Market. Growth determined by rate of cloud adoption.

27. OSS Projects #1 • Apache Whirr http://whirr.apache.org/ • Juju (Ubuntu) https://juju.ubuntu.com/

28. OSS Projects #2 • Pallet http://palletops.com/ • Serengeti http://serengeti.cloudfoundry.com/

29. Resources • Hadoop in Cloud Infrastructure (Steve Loughran) http://steveloughran.blogspot.ro/2012/03/ hadoop-in-cloud-infrastructures.html • What factors justify the use of Hadoop http://redmonk.com/sogrady/2011/01/13/ apache-hadoop/

30. Thanks! Questions? Andrei Savu / asavu@axemblr.com Twitter: @andreisavu

Apache Hadoop On-Demand on Cloud Infrastructure at Java2Days

Recomendados

Recomendados

Mais conteúdo relacionado

Mais de Andrei Savu

Mais de Andrei Savu (20)

Último

Último (20)

Apache Hadoop On-Demand on Cloud Infrastructure at Java2Days

Notas do Editor