Big data architectures

•

5 likes•1,968 views

Daan Gerits

Presentation given at @DamnData discussing different architecture types for BigData environments

Technology

Volume
IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOO
OIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOI
OIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOII
IOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIII
OIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOII
OIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOII

We already have that:
- NAS/SAN
- High Performance Computing

Variety

IOII IOIIIOIIIOII

IOII
IOII
IOII

IOII

IOII

We already have that:
- Meta-modeling
- NAS/SAN

Velocity

OIII
IOII
IOII OO
OIII
We already have that:
- Complex Event Processing

But do you have all of that in 1
platform?

Analytical Big Data

Analysis Oriented
Optimize
Non-intrusive

Delta
Apps
Dashboards
Distributed
Database

Data
Sources
Ingestion
Engine

Enrich

Data
Systems

Delta
Impala,
Hive, ...

Apps
Dashboards

Distributed
Database

Data
Sources
Flume,
Sqoop,
Scribe, ...

MR, Pig,
Crunch,
Mahout, ...

MR, Pig,
Crunch, ...

Data
Systems

Delta

Analytical Big Data
architecture for enriching mostly
structured data with the goal to
optimize business processes.

Delta
Apps
Dashboards
Distributed
Database

Data
Sources
Ingestion
Engine

Enrich

Overload!

Data
Systems

Delta
Be
write-heavy
or
read-heavy
NOT both!

Operational Big Data
Focussed on Day-today business
Innovate
(Non-)intrusive
(Thx Nathan Marz!)

Lambda
Realtime
View A
Realtime
Processing

Apps

Realtime
View B

Dashboard

Realtime
View C

Data
Sources

Batch
View A
Fact
Store

Just In Time
Combiner

Batch
View B
Batch
View C

Reports

Lambda
Cassandra*
Storm

Apps

Cassandra*

Dashboard

Cassandra*
Custom
Code*

Data
Sources
ElephantDB
HDFS

ElephantDB
ElephantDB

Reports

Lambda

Operational Big Data
architecture for storing and processing

multi-structured and
immutable data with the goal to
Innovate business

Advice
Pilots, PoC, PoT, … do them!
Be pragmatic, start skinny
In Belgium: Variety > Volume
Be prepared to pivot on technologies

Questions?
Thoughts?
Ideas?
Disagreements?
...

daan.gerits@dasos.be
www.dasos.be
@daangerits

All images are used merely for illustrational means. In no
way was it my purpose to violate any rights by using

Lambda
Multistructured

Unstructured

Restructured

Viewers also liked

A presentation cum workshop on Real time Analytics with Apache Kafka and Apache Spark. Apache Kafka is a distributed publish-subscribe messaging while other side Spark Streaming brings Spark's language-integrated API to stream processing, allows to write streaming applications very quickly and easily. It supports both Java and Scala. In this workshop we are going to explore Apache Kafka, Zookeeper and Spark with a Web click streaming example using Spark Streaming. A clickstream is the recording of the parts of the screen a computer user clicks on while web browsing.

Real time Analytics with Apache Kafka and Apache Spark

Rahul Jain

Getting more out of your big data

Nathan Bijnens

Apache storm vs. Spark Streaming

P. Taylor Goetz

Big Data

NGDATA

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. It is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Storm is a distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. This session presents the main concepts of Kafka and Storm and then shows how a simple stream processing application is implemented using these two technologies.

Kafka and Storm - event processing in realtime

Guido Schmutz

Big Data & Hadoop Tutorial

Edureka!

Hadoop Summit Europe 2014: Apache Storm Architecture

P. Taylor Goetz

Big data ppt

Nasrin Hussain

Viewers also liked (8)

Real time Analytics with Apache Kafka and Apache Spark

Getting more out of your big data

Apache storm vs. Spark Streaming

Big Data

Kafka and Storm - event processing in realtime

Big Data & Hadoop Tutorial

Hadoop Summit Europe 2014: Apache Storm Architecture

Big data ppt

Similar to Big data architectures

Big data - Apache Hadoop for Beginner's

senthil0809

This Edureka Hadoop Tutorial ( Hadoop Tutorial Blog Series: https://goo.gl/zndT2V ) helps you understand Big Data and Hadoop in detail. This Hadoop Tutorial is ideal for both beginners as well as professionals who want to learn or brush up their Hadoop concepts. This Edureka Hadoop Tutorial provides knowledge on: 1) What are the driving factors of Big Data and what are its challenges? 2) How Hadoop solves Big Data storage and processing challenges with Facebook use-case? 3) The overview of Hadoop YARN Architecture and its Components. 4) A real-life implementation of a complete end to end Hadoop Project on a Reddit use case on a Hadoop Cluster. Check our complete Hadoop playlist here: https://goo.gl/ExJdZs

Hadoop Tutorial | What is Hadoop | Hadoop Project on Reddit | Edureka

Edureka!

The Big data ecosystem is thriving. Driven by the productivity of the open source approach and by the markets, the community is constantly developing new tools, plugins and functionalities to improve and simplify all the aspects of data science. Unfortunately the life of system administrators is increasingly becoming more complicated. From the selection of the right architecture to the installation of the tools, their configuration and testing, a very large amount of time needs to be spent setting up the system. The burden is even more demanding when the infrastructure has high availability and strong security requirements. For the enterprise this means committing a considerable amount of resources to the maintenance of the systems, which require dedicated Unix sysadmin support. At Keedio we like simplicity, and in this workshop we shall demonstrate how you can, within minutes, easily deploy a full big data stack which is highly available and secure. Session presented at Big Data Spain 2015 Conference 16th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/fri/slot-38.html

Keedio Stack: BDaaS deployment for dummies by Alessio Comisso at Big Data Spa...

Big Data Spain

Introduction to Big Data by Manouj Bongirr

Pranav Kulkarni

IRJET- Systematic Review: Progression Study on BIG DATA articles

IRJET Journal

For 20 years, Blancco has offered solutions that support compliance with data protection and privacy regulations. We support the need for heavily-regulated industries to stay compliant with these regulations with data erasure solutions that satisfy (and often exceed) those requirements. Blancco Data Erasure works across the entire enterprise and lifecycle of every IT asset. We support highly-regulated industries, such as finance, healthcare and government, by assisting in the development and enforcement of data removal policies to prove compliance and achieve data sanitization.

Why Blancco ?

Christophe Elut

Big data introduction

Chirag Ahuja

Birst Cloud BI Data Sheet

Barbara Snowball

Seagate Solution Brief Highlights axle ai Integration with Seagate Lyve Cloud

flashnewsrelease

Geode is Not a Cache, it's an Analytics Engine

VMware Tanzu

big data

Jisha Aravind

BigData Analytics

Mayank Kumar Sharma

This paper compares the performance of scalable predictive analysis models using XGBoost in Big Data. The performance measurement is based on the training computing time and accuracy with AUR and Precision of a model. We developed XGBoost classification models with Airbnb listing dataset that predict the recommendation of the listings. The models are built in PySpark Rapids, BigDL, and H2O Sparkling with CPU and GPU on AWS EMR. We observed that BigDL with GPU is 25 – 50% faster training time than other platforms. H2O Sparkling has 5 - 7% better AUC and 0.7% better Precision than others.

Comparing Scalable Predictive Analysis using Spark XGBoost Platforms

Jongwook Woo

IoT and BigData

Daan Gerits

Red Hat Middleware Solutions for SAP Systems

Ted Jones

Big Data: Its Characteristics And Architecture Capabilities

Ashraf Uddin

Copy Data Management & Storage Efficiency - Ravi Namboori

Ravi namboori

NidhiGupta8431

You’re hearing a lot about big data these days. And big data and the technologies that store and process it, like Hadoop, aren’t just new data silos. You might be looking to integrate big data with existing enterprise information systems to gain better understanding of your business. You want to take informed action. During this session, we’ll demonstrate how Red Hat JBoss Data Virtualization can integrate with Hadoop through Hive and provide users easy access to data. You’ll learn how Red Hat JBoss Data Virtualization: Can help you integrate your existing and growing data infrastructure. Integrates big data with your existing enterprise data infrastructure. Lets non-technical users access big data result sets. We’ll also provide typical uses cases and examples and a demonstration of the integration of Hadoop sentiment analysis with sales data.

Big data insights with Red Hat JBoss Data Virtualization

Kenneth Peeples

Building a Big Data Analytics Platform- Impetus White Paper

Impetus Technologies

Similar to Big data architectures (20)

Big data - Apache Hadoop for Beginner's

Hadoop Tutorial | What is Hadoop | Hadoop Project on Reddit | Edureka

Keedio Stack: BDaaS deployment for dummies by Alessio Comisso at Big Data Spa...

Introduction to Big Data by Manouj Bongirr

IRJET- Systematic Review: Progression Study on BIG DATA articles

Why Blancco ?

Big data introduction

Birst Cloud BI Data Sheet

Seagate Solution Brief Highlights axle ai Integration with Seagate Lyve Cloud

Geode is Not a Cache, it's an Analytics Engine

big data

BigData Analytics

Comparing Scalable Predictive Analysis using Spark XGBoost Platforms

IoT and BigData

Red Hat Middleware Solutions for SAP Systems

Big Data: Its Characteristics And Architecture Capabilities

Copy Data Management & Storage Efficiency - Ravi Namboori

Big data insights with Red Hat JBoss Data Virtualization

Building a Big Data Analytics Platform- Impetus White Paper

Recently uploaded

New customer? New industry? New cloud? New team? A lot to handle! How to ensure the success of the project? Start it well! I've created the 3 areas of focus at the beginning of the project that helped me in multiple roles (BA, PO, and Consultant). Learn from real-world experiences and discover how these insights can empower you to deliver unparalleled value to your customers right from the project's start.

Powerful Start- the Key to Project Success, Barbara Laskowska

CzechDreamin

WebRTC and SIP not just audio and video @ OpenSIPS 2024

Lorenzo Miniero

This our Twelfth semiannual report on the global Cryptocurrency mining industry. Bitcoin is the world’s largest special purpose supercomputer. And it is globally decentralized. Millions of nodes all run the same open-source code to secure the Bitcoin network, create value, and put new transactions onto the distributed ledger. The latest Top500 list has just been announced at the ISC 2024 conference in Hamburg, and once again the Frontier supercomputer with 1.2 Exaflops peak performance is number one on the list. If assigned to SHA-256 hashing, Frontier would provide only the equivalent hash rate of about three cabinets of the latest high-end Bitcoin mining systems, costing less than 0.1% of Frontier’s cost. Michael Saylor, Chairman of MicroStrategy, has pointed out that GPUs are two orders of magnitude slower than the 5-nanometer technology of custom ASICs used for Bitcoin mining today. He makes the point that the Bitcoin network is unassailable by all of the hyperscale computing resources combined in AWS, Google, and Microsoft Azure cloud data centers today.

TopCryptoSupers 12thReport OrionX May2024

Stephen Perrenod

IESVE for Early Stage Design and Planning

IES VE

Delivered by Sreenath Gopalakrishna, Director of Software Engineering at BT, and Dr Jim Webber, Chief Scientist at Neo4j, at Gartner Data & Analytics Summit London 2024 this presentation examines how knowledge graphs and GenAI combine in real-world solutions. BT Group has used the Neo4j Graph Database to enable impressive digital transformation programs over the last 6 years. By re-imagining their operational support systems to adopt self-serve and data lead principles they have substantially reduced the number of applications and complexity of their operations. The result has been a substantial reduction in risk and costs while improving time to value, innovation, and process automation. Future innovation plans include the exploration of uses of EKG + Generative AI.

BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx

Neo4j

The presentation underscores the strategic advantage of treating design systems not just as technical assets but as vital business components that require thoughtful management, robust planning, and strategic alignment with organizational goals. Key Points Covered: - Understanding Design Systems as Business Entities: Conceptualizing design systems as internal business entities can streamline their integration and evolution within a company. - Adoption and Expansion: Elaborating on the importance of tactical adoption across organizational structures, enhancing product suites to cater to user needs and broadening scope to mobile and content authoring solutions. - Data-Driven Development: Utilizing data insights for component development ensures that resources are allocated to create valuable, widely used features. - Financial Modeling for Design Systems: Developing sustainable funding models is crucial for long-term support and success of design systems. - Promoting Internal Buy-In: Stressing on strategies for promoting design systems within the organization to increase engagement and investment from internal stakeholders.

A Business-Centric Approach to Design System Strategy

UXDXConf

The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf

FIDO Alliance

Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf

FIDO Alliance

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “Enterprise Knowledge Graphs: The Importance of Semantics” on May 9, 2024, at the annual Data Summit in Boston. In her presentation, Hedden describes the components of an enterprise knowledge graph and provides further insight into the semantic layer – or knowledge model – component, which includes an ontology and controlled vocabularies, such as taxonomies, for controlled metadata. While data experts tend to focus on the graph database components (RDF triple store or a label property graph), Hedden emphasizes they should not overlook the importance of the semantic layer.

Enterprise Knowledge Graphs - Data Summit 2024

Enterprise Knowledge

Google I/O Extended 2024 Warsaw

GDSC PJATK

Webinar Recording: https://www.panagenda.com/webinars/alles-neu-macht-der-mai-wir-durchleuchten-den-verbesserten-notes-eigenschaftendialog/ Haben Sie sich schon einmal über den zu kleinen Eigenschaftendialog in Notes geärgert? Mussten Sie einen Agenten oder eine Aktion erstellen, um schnell mal ein Feld zu ändern? Haben Sie jedes mal endlos nach dem zu vergleichenden Feld gesucht, nachdem Sie ein neues Dokument ausgewählt haben? Wollten Sie das verdammte Ding einfach nur größer machen? Zum Glück gibt es dafür eine Lösung – und sie ist wahrscheinlich bereits installiert! Mit dem kostenlosen panagenda Document Properties (Pro) erhalten Sie den Eigenschaftendialog, den Sie schon immer haben wollten. Größer, anpassbar, und im Volltext durchsuchbar. Sehen Sie mehrere Dokumente gleichzeitig oder vergleichen Sie mit einem Diff-Viewer. Ändern Sie beliebige Felder und haben Sie endlich eine einfache Möglichkeit, Profildokumente für alle Benutzer zu verwalten. Entdecken Sie mit HCL Ambassador Marc Thomas, wie Document Properties Ihre Arbeit vereinfachen und Sie bei der täglichen Verwendung von Domino-Anwendungen unterstützen kann – im Client oder im Designer. Sie werden es nicht bereuen! Für Sie in diesem Webinar - Was Document Properties ist, welche Editionen es gibt und wo es in Notes und Domino Designer zu finden ist - Wie Sie nach einem beliebigen Feld suchen und es bearbeiten, Dokumente vergleichen oder alle Daten per CSV exportieren können - Suchen, Bearbeiten und auch Löschen von Profildokumenten - Welche Konfigurationseinstellungen verfügbar sind, um Funktionen anzupassen - Wie Ihre Endbenutzer davon profitieren - Sehen Sie alles in einer Live-Demo

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...

panagenda

AI presentation and introduction - Retrieval Augmented Generation RAG 101

vincent683379

The Metaverse: Are We There Yet?

Mark Billinghurst

Unlock the mysteries of successful Salesforce interviews in this insightful session hosted by Hugo Rosario (Salesforce Customer), a seasoned hiring manager that leads the Salesforce Department of multinational company with over 100 interviews under their belt. Step into the manager's chair and gain exclusive behind-the-scenes insights into what makes a Salesforce consultant stand out during the interview process. From deciphering the unspoken cues to mastering key strategies, we'll explore the intricacies of the interview process and provide practical tips for consultants looking to not only pass interviews but also thrive in their roles. Whether you're a seasoned professional or just starting your Salesforce journey, this session is your backstage pass to the secrets that hiring managers wish you knew.

Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...

CzechDreamin

Join me in this session where I'll share our journey of building a fully serverless application that flawlessly managed check-ins for an event with a staggering 80 thousand registrations. We'll dive into three key strategies that made this possible. Firstly, by harnessing DynamoDB global tables, we ensured global service availability and data replication across regions, boosting performance and disaster recovery. Next, we'll explore how we seamlessly integrated real-time updates into the app using Appsync subscriptions, making the experience dynamic and engaging for users. Finally, I'll discuss how provisioned concurrency not only improved performance but also kept costs in check, highlighting the cost-effectiveness of serverless architectures. Through these strategies and the inherent scalability of serverless technology, our application effortlessly handled massive user loads without manual intervention. This session is a real world example to the power and efficiency of modern cloud-based solutions in enabling seamless scalability and robust performance with Serverless

How we scaled to 80K users by doing nothing!.pdf

Srushith Repakula

Intro in Product Management - Коротко про професію продакт менеджера

Mark Opanasiuk

Delivered by Michael Down at Gartner Data & Analytics Summit London 2024 - Your enemies use GenAI too: Staying ahead of fraud with Neo4j. Fraudsters exploit the latest technologies like generative AI to stay undetected. Static applications can’t adapt quickly enough. Learn why you should build flexible fraud detection apps on Neo4j’s native graph database combined with advanced data science algorithms. Uncover complex fraud patterns in real-time and shut down schemes before they cause damage.

Your enemies use GenAI too - staying ahead of fraud with Neo4j