Walking Around the Data Lake

•Transferir como PPTX, PDF•

1 gostou•69 visualizações

All Things Open

Presented at: All Things Open 2019 Presented by: Fatima Naqui, Fidelity Investments

Walking around the
Data Lake
Fatima Naqui
Sr Big Data Engineer at
Fidelity Investments
Naqui.Fatima@gmail.com
linkedin.com/in/fatimanaqui

Agenda
• What is a Data Lake?
• Why do I need a Data Lake?
• High Level Architecture
• Principles of a Data Lake
• Data Lake layers and components
• Ingestion
• Metadata management
• Data Lineage and Governance
• Data Curation
• Technical Architectures from Cloud providers
• Q&A

What is a
Data Lake?
Gartner’s definition of a Data Lake
“A data lake is a collection of storage instances of
various data assets. These assets are stored in a
near-exact, or even exact, copy of the source
format and are in addition to the originating data
stores.”

Why do I need
a Data Lake?
• Cost effective way to deal with big data
challenges
• Single source of truth that can be reused
• Cheaper and flexible alternative to batch
workloads
• Increase analytics and operational agility

High level
architecture

Principles of a
Data Lake
• Separation of storage and compute
• Bridging information silos
• Schema on read

Data Lake layers and
components

Ingestion
• Land everything without modification
• Dump information from source
systems
• Structured, unstructured, historical
data

Metadata Management
Foundation for
distributed data
management
Supports operational use
of data
Data semantic variations
over time
Monitoring changes in
data usage

Data Lineage and Governance
Data lineage deals with data's origins
Where data moves over time?
What happens to the data?
How is the data transformed?
Data governance is the process of
managing
Availability
Usability
Security
Integrity

Data Curation
Optimize data from its raw
form to be made consumable
Source of insights Enrich the data with analytic
and algorithmic functions

Technical
Architectures

Cloud
Architectures

Cloud
Architectures

Q & A

Mais conteúdo relacionado

Mais procurados

Struggling to keep up with an ever-increasing demand for data at your organisation? Do you spend hours tinkering with your streaming data pipelines? Does that one data scientist with direct EDW access keep you up at night? Introducing Snowflake, a brand new SQL data warehouse built for the cloud. We’ve designed and implemented a unique cloud-based architecture that addresses the most common shortcomings of existing data solutions. With Snowflake, you can unlock unlimited concurrency, enable instant scalability, and take advantage of built-in tuning and optimisation. Join us and find out what Netflix, Adobe, and Nike all have in common.

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...

Amazon Web Services

Building A Self Service Analytics Platform on Hadoop

Building A Self Service Analytics Platform on Hadoop

Building A Self Service Analytics Platform on Hadoop

Snowflake Company Presentation

Snowflake Company Presentation

Snowflake Company Presentation

Real Use Cases - Pentaho & Big Data Ecosystem

Real Use Cases - Pentaho & Big Data Ecosystem

Real Use Cases - Pentaho & Big Data Ecosystem

Original event: https://www.meetup.com/NYC-Machine-Learning/events/256605862/ -- "Doing large scale ML in production is hard" – Everyone who's tried This talk is focussed on ML Systems. Especially the less obvious pitfalls, which have caused us troubles at Spotify. This talk assumes a certain level of familiarity with ML: You'll get the most out of if you've some experience with applied ML, ideally on production systems. Romain Yon is a Staff ML Engineer at Spotify. Over the years, Romain has worked on many of the core ML systems that power Spotify today (Music Recommendation, Catalog Quality, Search Ranking, Ads, ..). During the past year, Romain has been mostly focusing on designing reusable ML Infrastructure that can be leveraged throughout Spotify. Prior to Spotify, Romain co-founded the startup https://linkurio.us while getting his MSc in ML from Georgia Tech.

ML Infra @ Spotify: Lessons Learned - Romain Yon - NYC ML Meetup

ML Infra @ Spotify: Lessons Learned - Romain Yon - NYC ML Meetup

ML Infra @ Spotify: Lessons Learned - Romain Yon - NYC ML Meetup

EDB Postgres in DBaaS & Container Platforms

EDB Postgres in DBaaS & Container Platforms

EDB Postgres in DBaaS & Container Platforms

Optimize the performance, cost, and value of databases.pptx

Optimize the performance, cost, and value of databases.pptx

Optimize the performance, cost, and value of databases.pptx

Watch the presentation on-demand now: https://goo.gl/kceFTe Today’s digital economy demands a new way of running business. Flexible access to information and responses in real time are essential for outpacing competition. Watch this Denodo DataFest 2017 session to discover: • Data access challenges faced by organizations today. • How data virtualization facilitates real-time analytics. • Key use cases and customer success stories.

Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses

Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses

Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses

Watch Daves' presentation on-demand from Fast Data Strategy Virtual Summit here: https://buff.ly/2Kj7muc Denodo’s new dynamic catalog is the new black. It combines the power of data delivery infrastructure with data catalog for contextual information and collective intelligence. Attend this session to discover: • What is unique about Dynamic Data Catalog? • How it empowers a community of analysts and decisions makers? • How it facilitates data curation and data stewardship in your organization?

A Dynamic Data Catalog for Autonomy and Self-Service

A Dynamic Data Catalog for Autonomy and Self-Service

A Dynamic Data Catalog for Autonomy and Self-Service

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

Patrick Van Renterghem

Ready to take the plunge into a cloud data lake? Don’t dive in just yet. Most of the data filing your cloud data lake is raw, unstructured, and rife with data quality problems. If you’re going to get maximum value from your cloud data lake investment, data preparation is critical. We have created a checklist to guide you through six steps you should take to improve agility, productivity, and consistency when preparing data for a cloud data lake: - Choose a data preparation solution that natively integrates with your cloud platform - Use data preparation to increase efficiency and coordination among stakeholders - Don’t rebuild what works; choose the right use cases - Empower users with self-service while maintaining data governance - Ensure data quality with at-scale and continuous validation - Publish and operationalize preparation for downstream analytics and machine learning Join this tech demo and learn how to use Data Preparation to accelerate Cloud Data Lake adoption.

Six Data Prep steps to Optimize Cloud Data Lakes - Big Data Expo 2019

Six Data Prep steps to Optimize Cloud Data Lakes - Big Data Expo 2019

Six Data Prep steps to Optimize Cloud Data Lakes - Big Data Expo 2019

webwinkelvakdag

How to build a successful Data Lake

How to build a successful Data Lake

How to build a successful Data Lake

DataWorks Summit/Hadoop Summit

Auckland SQL Saturday - Azure Data Lake

Auckland SQL Saturday - Azure Data Lake

Auckland SQL Saturday - Azure Data Lake

Sergio Zenatti Filho

Chug building a data lake in azure with spark and databricks

Chug building a data lake in azure with spark and databricks

Chug building a data lake in azure with spark and databricks

Brandon Berlinrut

Julie Strauss. This session introduces the newest services in the Cortana Analytics family. The Azure Data Catalog is an enterprise-wide metadata catalog that enables self-service data source discovery. Data Catalog is a fully managed service that stores, describes, indexes, and provides information on how to access any registered data source in your organization. This session presents an overview of the Data Catalog and how – by using it to register, enrich, discover, understand and consume data sources – you can close the gap between those seeking information and those creating it.

Cortana Analytics Workshop: Azure Data Catalog

Cortana Analytics Workshop: Azure Data Catalog

Cortana Analytics Workshop: Azure Data Catalog

A world-leading manufacturer was in search of an IoT solution that could ingest, integrate, and manage data being generated from various types of connected machinery located on factory floors around the globe. The company needed to manage the devices generating the data, integrate the flow of data into existing back-end systems, run advanced analytics on that data, and then deliver services to generate real-time decision making at the edge. In this session, learn how Clairvoyant, a leading systems integrator and Red Hat partner, was able to accelerate digital transformation for their customer using Internet of Things (IoT) and machine learning in a hybrid cloud environment. Specifically, Clairvoyant and Eurotech will discuss: • The approach taken to optimize manufacturing processes to cut costs, minimize downtime, and increase efficiency. • How a data processing pipeline for IoT data was built using an open, end-to-end architecture from Cloudera, Eurotech, and Red Hat. • How analytics and machine learning inferencing powered at the IoT edge will allow predictions to be made and decisions to be executed in real time. • The flexible and hybrid cloud environment designed to provide the key foundational elements to quickly and securely roll out IoT use cases.

Delivering digital transformation and business impact with io t, machine lear...

Delivering digital transformation and business impact with io t, machine lear...

Delivering digital transformation and business impact with io t, machine lear...

As companies grow, so does the volume of their data. Without the proper solutions in place to quickly store, measure and analyze that data, its usefulness quickly declines. See our latest webinar to learn about how companies are increasingly turning towards cloud-based data warehousing to derive more value out of their data and apply their findings to make smarter business decisions. The webinar covers core topics including: - The benefits of using Snowflake’s unique architecture for interacting with data. - How Matillion can help you quickly load and transform your data to maximize its value. - Expert advice on how to apply data warehousing and ETL best practices. Watch the full webinar: https://youtu.be/mIOm3j431OQ

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

AzureDay - Introduction Big Data Analytics.

AzureDay - Introduction Big Data Analytics.

AzureDay - Introduction Big Data Analytics.

Clinical genomic analytics pipelines using Databricks and the Delta Lake for the benefit of loading individual reads from raw sequencing or base-call files have significant advantages over more traditional methods. Analysis pipelines that perform genomic mapping to purpose-built reference data artifacts persisted to tables allows for enhanced performance that is magnitudes greater than previous mapping methods. These scalable, reproducible, and potentially open sourced methods have the ability to transform bioinformatics and R&D data management / governance.

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Mais procurados (19)

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...

Snowflake: Your Data. No Limits (Session sponsored by Snowflake) - AWS Summit...

Building A Self Service Analytics Platform on Hadoop

Building A Self Service Analytics Platform on Hadoop

Building A Self Service Analytics Platform on Hadoop

Snowflake Company Presentation

Snowflake Company Presentation

Snowflake Company Presentation

Real Use Cases - Pentaho & Big Data Ecosystem

Real Use Cases - Pentaho & Big Data Ecosystem

Real Use Cases - Pentaho & Big Data Ecosystem

ML Infra @ Spotify: Lessons Learned - Romain Yon - NYC ML Meetup

ML Infra @ Spotify: Lessons Learned - Romain Yon - NYC ML Meetup

ML Infra @ Spotify: Lessons Learned - Romain Yon - NYC ML Meetup

EDB Postgres in DBaaS & Container Platforms

EDB Postgres in DBaaS & Container Platforms

EDB Postgres in DBaaS & Container Platforms

Optimize the performance, cost, and value of databases.pptx

Optimize the performance, cost, and value of databases.pptx

Optimize the performance, cost, and value of databases.pptx

Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses

Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses

Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses

A Dynamic Data Catalog for Autonomy and Self-Service

A Dynamic Data Catalog for Autonomy and Self-Service

A Dynamic Data Catalog for Autonomy and Self-Service

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

Six Data Prep steps to Optimize Cloud Data Lakes - Big Data Expo 2019

Six Data Prep steps to Optimize Cloud Data Lakes - Big Data Expo 2019

Six Data Prep steps to Optimize Cloud Data Lakes - Big Data Expo 2019

How to build a successful Data Lake

How to build a successful Data Lake

How to build a successful Data Lake

Auckland SQL Saturday - Azure Data Lake

Auckland SQL Saturday - Azure Data Lake

Auckland SQL Saturday - Azure Data Lake

Chug building a data lake in azure with spark and databricks

Chug building a data lake in azure with spark and databricks

Chug building a data lake in azure with spark and databricks

Cortana Analytics Workshop: Azure Data Catalog

Cortana Analytics Workshop: Azure Data Catalog

Cortana Analytics Workshop: Azure Data Catalog

Delivering digital transformation and business impact with io t, machine lear...

Delivering digital transformation and business impact with io t, machine lear...

Delivering digital transformation and business impact with io t, machine lear...

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

Simplifying Your Journey to the Cloud: The Benefits of a Cloud-Based Data War...

AzureDay - Introduction Big Data Analytics.

AzureDay - Introduction Big Data Analytics.

AzureDay - Introduction Big Data Analytics.

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Managing R&D Data on Parallel Compute Infrastructure

Semelhante a Walking Around the Data Lake

So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a modern data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. They all may sound great in theory, but I'll dig into the concerns you need to be aware of before taking the plunge. I’ll also include use cases so you can see what approach will work best for your big data needs. And I'll discuss Microsoft version of the data mesh.

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Harness the power of Data in a Big Data Lake

Harness the power of Data in a Big Data Lake

Harness the power of Data in a Big Data Lake

Saurabh K. Gupta

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Enterprises have been rapidly adopting data lakes as a complement or replacement of data warehouses. Many of the Data lake implementations are ignoring the inherent drawbacks and limitations of Data Lakes and ending up as data swamps with little or no benefit to the businesses. In this session we will go through some of challenges and the key aspects that need to be considered for successful Data lake implementations.

Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data...

Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data...

Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data...

Avinash Ramineni

Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020. Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms. Data lakes will be built in cloud object storage. We’ll discuss the options there as well. Get this data point for your data lake journey.

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

Data warehouse

Shwetabh Jaiswal

Data warehouseold

Data warehouseold

Data warehouseold

Shwetabh Jaiswal

Big Data is the reality of modern business: from big companies to small ones, everybody is trying to find their own benefit. Big Data technologies are not meant to replace traditional ones, but to be complementary to them. In this presentation you will hear what is Big Data and Data Lake and what are the most popular technologies used in Big Data world. We will also speak about Hadoop and Spark, and how they integrate with traditional systems and their benefits.

How to use Big Data and Data Lake concept in business using Hadoop and Spark...

How to use Big Data and Data Lake concept in business using Hadoop and Spark...

How to use Big Data and Data Lake concept in business using Hadoop and Spark...

Institute of Contemporary Sciences

Data Warehouse

kalyani.ppt

Data Warehouse

kalyani.ppt

Today’s data-driven companies have a choice to make – where do we store our data? As the move to the cloud continues to be a driving factor, the choice becomes either the data warehouse (Snowflake et al) or the data lake (AWS S3 et al). There are pro’s and con’s for each approach. While the data warehouse will give you strong data management with analytics, they don’t do well with semi-structured and unstructured data with tightly coupled storage and compute, not to mention expensive vendor lock-in. On the other hand, data lakes allow you to store all kinds of data and are extremely affordable, but they’re only meant for storage and by themselves provide no direct value to an organization. Enter the Open Data Lakehouse, the next evolution of the data stack that gives you the openness and flexibility of the data lake with the key aspects of the data warehouse like management and transaction support. In this webinar, you’ll hear from Ali LeClerc who will discuss the data landscape and why many companies are moving to an open data lakehouse. Ali will share more perspective on how you should think about what fits best based on your use case and workloads, and how some real world customers are using Presto, a SQL query engine, to bring analytics to the data lakehouse.

Data Warehouse or Data Lake, Which Do I Choose?

Data Warehouse or Data Lake, Which Do I Choose?

Data Warehouse or Data Lake, Which Do I Choose?

Thirty years is a long time for a technology foundation to be as active as relational databases. Are their replacements here? In this webinar, we say no. Databases have not sat around while Hadoop emerged. The Hadoop era generated a ton of interest and confusion, but is it still relevant as organizations are deploying cloud storage like a kid in a candy store? We’ll discuss what platforms to use for what data. This is a critical decision that can dictate two to five times additional work effort if it’s a bad fit. Drop the herd mentality. In reality, there is no “one size fits all” right now. We need to make our platform decisions amidst this backdrop. This webinar will distinguish these analytic deployment options and help you platform 2020 and beyond for success.

ADV Slides: Platforming Your Data for Success – Databases, Hadoop, Managed Ha...

ADV Slides: Platforming Your Data for Success – Databases, Hadoop, Managed Ha...

ADV Slides: Platforming Your Data for Success – Databases, Hadoop, Managed Ha...

Choosing Right datastorage.pptx

Choosing Right datastorage.pptx

Choosing Right datastorage.pptx

AlekhyaAchanta3

Choosing right data storage.pptx

Choosing right data storage.pptx

Choosing right data storage.pptx

AlekhyaAchanta3

Architecting a datalake

Architecting a datalake

Architecting a datalake

Laurent Leturgez

Building Data Warehouse in SQL Server

Building Data Warehouse in SQL Server

Building Data Warehouse in SQL Server

Antonios Chatzipavlis

Introduction to no sql database

Introduction to no sql database

Introduction to no sql database

Heman Hosainpana

Enterprises and organizations from every industry and scale are working to leverage data to achieve their strategic objectives — whether they are to be more profitable, effective, risk-tolerant, prepared, sustainable, and/or adaptable in an ever-changing world. Data has exploded in volume during the last decade as humans and machines alike produce data at an exponential pace. Also, exciting technologies have emerged around that data to improve our abilities and capabilities around what we can do with data. Behind this data revolution, there are forces at work, causing enterprises to shift the way they leverage data and accelerate the demand for leverageable data. Organizations (and the climates in which they operate) are becoming more and more complex. They are also becoming increasingly digital and, thus, dependent on how data informs, transforms, and automates their operations and decisions. With increased digitization comes an increased need for both scale and agility at scale. In this session, we have undertaken an ambitious goal of evaluating the current vendor landscape and assessing which platforms have made, or are in the process of making, the leap to this new generation of Data Management and integration capabilities.

The Shifting Landscape of Data Integration

The Shifting Landscape of Data Integration

The Shifting Landscape of Data Integration

Semelhante a Walking Around the Data Lake (20)

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Data Lakehouse, Data Mesh, and Data Fabric (r2)

Harness the power of Data in a Big Data Lake

Harness the power of Data in a Big Data Lake

Harness the power of Data in a Big Data Lake

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data...

Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data...

Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data...

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

ADV Slides: When and How Data Lakes Fit into a Modern Data Architecture

Data warehouse

Data warehouseold

Data warehouseold

Data warehouseold

How to use Big Data and Data Lake concept in business using Hadoop and Spark...

How to use Big Data and Data Lake concept in business using Hadoop and Spark...

How to use Big Data and Data Lake concept in business using Hadoop and Spark...

Data Warehouse

kalyani.ppt

Data Warehouse

kalyani.ppt

Data Warehouse or Data Lake, Which Do I Choose?

Data Warehouse or Data Lake, Which Do I Choose?

Data Warehouse or Data Lake, Which Do I Choose?

ADV Slides: Platforming Your Data for Success – Databases, Hadoop, Managed Ha...

ADV Slides: Platforming Your Data for Success – Databases, Hadoop, Managed Ha...

ADV Slides: Platforming Your Data for Success – Databases, Hadoop, Managed Ha...

Choosing Right datastorage.pptx

Choosing Right datastorage.pptx

Choosing Right datastorage.pptx

Choosing right data storage.pptx

Choosing right data storage.pptx

Choosing right data storage.pptx

Architecting a datalake

Architecting a datalake

Architecting a datalake

Building Data Warehouse in SQL Server

Building Data Warehouse in SQL Server

Building Data Warehouse in SQL Server

Introduction to no sql database

Introduction to no sql database

Introduction to no sql database

The Shifting Landscape of Data Integration

The Shifting Landscape of Data Integration

The Shifting Landscape of Data Integration

Mais de All Things Open

Presented at the ATO RTP Meetup Presented by Jeremy Proffit, Director of DevSecOps & SRE for Customer Care and Communications, Ally Title: Building Reliability - The Realities of Observability Abstract: Join me as we discuss true observability, learn what works and what doesn't. We'll not only discuss dashboards, monitoring and alerting, but how these can be built by automation or included in your IAC modules. We'll talk about how to properly alert staff based on priority to keep your staff and yourself sane. And even discuss architecture and how it impacts reliably and why serverless isn't always the best at being reliable.

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

All Things Open

Presented at the ATO RTP Meetup Presented by Peter Zaitsev, Founder of Percona Title: Modern Database Best Practices Abstract: There are now more Database choices available for developers than ever before - there are general purpose databases and specialized databases, single node and distributed databases, Open Source, Proprietary databases and databases available exclusively in the cloud. In this presentation we will cover the best practices of choosing database(s) for your applications, best practices as it comes to application development as well as managing those databases to achieve best possible performance, security, availability at the lowest cost.

Modern Database Best Practices

Modern Database Best Practices

Modern Database Best Practices

All Things Open

All Things Open 2023 Presented at All Things Open 2023 Presented by Deb Bryant - Open Source Initiative, Patrick Masson - Apereo Foundation, Stephen Jacobs - Rochester Institute of Technology, Ruth Suehle - SAS, & Greg Wallace - FreeBSD Foundation Title: Open Source and Public Policy Abstract: New regulations in the software industry and adjacent areas such as AI, open science, open data, and open education are on the rise around the world. Cyber Security, societal impact of AI, data and privacy are paramount issues for legislators globally. At the same time, the COVID-19 pandemic drove collaborative development to unprecedented levels and took Open Source software, open research, open content and data from mainstream to main stage, creating tension between public benefit and citizen safety and security as legislators struggle to find a balance between open collaboration and protecting citizens. Historically, the open source software community and foundations supporting its work have not engaged in policy discussions. Moving forward, thoughtful development of these important public policies whilst not harming our complex ecosystems requires an understanding of how our ecosystem operates. Ensuring stakeholders without historic benefit of representation in those discussions becomes paramount to that end. Please join our open discussion with open policy stakeholders working constructively on current open policy topics. Our panelists will provide a view into how oss foundations and other open domain allies are now rising to this new challenge as well as seizing the opportunity to influence positive changes to the public’s benefit. Topics: Public Policy, Open Science, Open Education, current legislation in the US and EU, US interest in OSS sustainability, intro to the Open Policy Alliance Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Open Source and Public Policy

Open Source and Public Policy

Open Source and Public Policy

All Things Open

Presented at All Things Open 2023 Presented by Ashpak Shaikh & Lucy Shen - Intuit Title: Weaving Microservices into a Unified GraphQL Schema with graph-quilt Abstract: The magic of GraphQL is that it provides data access through a single endpoint—clean and easy. But as the number of GraphQL microservices your tech stack depends on starts to grow, that single-endpoint purpose becomes a new multi-endpoint problem. Ideally, we would have an orchestrator that could aggregate schemas from multiple microservices into a unified GraphQL schema and route the requests to the appropriate microservice. Enter graph-quilt, an open source Java library that provides recursive schema stitching and Apollo Federation style schema composition. In this talk, we’ll walk through our GraphQL journey and show you how to use graph-quilt to simplify your data orchestration needs. We will also share our open sourced reference implementation of a highly performant graph-quilt gateway currently being used in production here at Intuit, where we’ve had incredible success in scaling the gateway with 50+ microservices and 150+ clients. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

All Things Open

Presented at All Things Open 2023 Presented by Phil Nash - Sonar Title: The State of Passwordless Auth on the Web Abstract: Can we get rid of passwords yet? They make for a poor user experience and users are notoriously bad with them. The advent of WebAuthn has brought a passwordless world closer, but where do we really stand? In this talk we'll explore the current user experience of WebAuthn and the requirements a user has to fulfil to authenticate without a password. We'll also explore the fallbacks and safeguards we can use to make the password experience better and more secure. By the end of the session you'll have a vision of how authentication could look in the future and a blueprint for how to build the best auth experience today. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

All Things Open

Presented at All Things Open 2023 Presented by Phil Nash - Sonar Title: Total ReDoS: The dangers of regex in JavaScript Abstract: Regular expressions are complicated and can be hard to learn. On top of that, they can also be a security risk; writing the wrong pattern can open your application up to denial of service attacks. One token out of place and you invite in the dreaded ReDoS. But how can a regular expression cause this? In this talk we’ll track down the patterns that can cause this trouble, explain why they are an issue and propose ways to fix them now and avoid them in the future. Together we’ll demystify these powerful search patterns and keep your application safe from expressions that behave in a way that is anything but regular. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

All Things Open

Presented at All Things Open 2023 Presented by Karl Mozurkewich - Storj Title: What Does Real World Mass Adoption of Decentralized Tech Look Like? Abstract: We delve into the transformative potential of decentralized technology. Beginning with a brief overview of the rise of centralization with the advent of the internet and the counter-shift marked by blockchain we explore the intrinsic characteristics of decentralized and distributed systems, such as trustless operations, peer-to-peer networks, and enterprise application scalability. Various sectors, including finance, supply chains, media and entertainment, data science and cloud infrastructure are on the brink of disruption. The societal implications are vast, with the potential for greater individual empowerment, a greener planet and more viable resource utilization, but concerns about data security persist. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

All Things Open

Presented at All Things Open 2023 Presented by Anastasia Lalamentik - Kaleido Title: How to Write & Deploy a Smart Contract Abstract: In this talk, Anastasia Lalamentik, Full Stack Engineer at Kaleido, will walk through how Ethereum smart contracts work and go over related concepts like gas fees, the Ethereum Virtual Machine (EVM), the block explorer, and the Solidity programming language. This is vital to anyone who wants to build a blockchain app and is a great introduction to blockchain technology for newcomers to the space. By the end of the talk, attendees will better understand how to: - Write a simple smart contract - Deploy their smart contract to an Ethereum test network through the latest tools like Hardhat and the MetaMask wallet - Test interactions with their deployed smart contract and ensure that everything is working properly Additionally, participants will get to interact with Anastasia's deployed smart contract at the end of the talk. Anastasia’s past talks have attracted and have been attended by a diverse group of participants with a range of experience in the space. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

All Things Open

Presented at All Things Open 2023 Presented by Paul Brebner - Instaclustr (by Spot by NetApp) Title: Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Abstract: In this talk we’ll build a Drone delivery application, and then use it to do some Machine Learning “on the fly”. In the 1st part of the talk, we'll build a real-time Drone Delivery demonstration application using a combination of two open-source technologies: Uber’s Cadence (for stateful, scheduled, long-running workflows), and Apache Kafka (for fast streaming data). With up to 2,000 (simulated) drones and deliveries in progress at once this application generates a vast flow of spatio-temporal data. In the 2nd part of the talk, we'll use this platform to explore Machine Learning (ML) over streaming and drifting Kafka data with TensorFlow to try and predict which shops will be busy in advance. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

All Things Open

Presented at the All Things Open 2023 Inclusion and Diversity in Open Source Event Presented by Efraim Marquez-Arreaza - Red Hat Title: DEI Challenges and Success Abstract: In today's world, many companies and organizations have Diversity, Equity and Inclusion (DEI) communities. Red Hat Unidos is a DEI community focused on advocating for the Hispanic/Latine community. In this talk, we would like to share our challenges and success during the past 4-years and plans for the future. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

DEI Challenges and Success

DEI Challenges and Success

DEI Challenges and Success

All Things Open

Presented at All Things Open 2023 Presented by Lydia Cupery - HubSpot Title: Scaling Web Applications with Background Jobs: Takeaways from Generating a Huge PDF Abstract: Do you need to perform time-consuming or CPU-intensive processes in your web application but are concerned about performance? That’s where background jobs come in. By offloading resource-intensive tasks to separate worker processes, you can improve the scalability of your web application. In this talk, I'll share my experience of using background jobs to scale our web application. I'll discuss the challenges my team faced that led us to adopt background jobs. Then, I'll share practical tips on how to design background jobs for CPU-intensive or time-consuming processes, such as generating huge PDFs and batch emailing. I'll wrap up by going over the performance and cost tradeoffs of background jobs. I'll use Typescript, Express, and Heroku as examples in this talk, but the concepts and best practices that I'll share are applicable to other languages and tools. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Scaling Web Applications with Background

Scaling Web Applications with Background

Scaling Web Applications with Background

All Things Open

Presented at All Things Open 2023 Presented by Robert Aboukhalil - CZI Title: Supercharging tutorials with WebAssembly Abstract: sandbox.bio is a free platform that features interactive command-line tutorials for bioinformatics. This talk is a deep-dive into how sandbox.bio was built, with a focus on how WebAssembly enabled bringing command-line tools like awk and grep to the web. Although these tools were originally written in C/C++, they all run directly in the browser, thanks to WebAssembly! And since the computations run on each user's computer, this makes the application highly scalable and cost-effective. Along the way, I'll discuss how WebAssembly works and how to get started using it in your own applications. The talk will also cover more advanced WebAssembly features such as threads and SIMD, and will end with a discussion of WebAssembly's benefits and pitfalls (it's a powerful technology, but it's not always the right tool!). Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

All Things Open

Presented at All Things Open 2023 Presented by K.S. Bhaskar - YottaDB LLC Title: Using SQL to Find Needles in Haystacks Abstract: Database journal files capture every update to a database. A database of a few hundred GB can generate GBs worth of journal files every minute at busy times. Troubleshooting and forensices, especially of rare and intermittent problems, such as which process made what update and when, is an exercise of finding needles in haystacks. A similar problem exists with syslogs. A solution is to load the journal files and syslogs into a database, and use SQL to query the database. Bhaskar will present and demonstrate this with a 100% FOSS stack. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

All Things Open

Presented at All Things Open 2023 Presented by Wes Widner - Automox Title: Configuration Security as a Game of Pursuit Intercept Abstract: In this session we will take a look at the emerging field of cloud security posture management and how we can approach the problem space using a class of board games known as pursuit/intercept. Using the game Scotland Yard as a visual illustration we'll explore the cognitive and technical limitations that all CSPM systems face and what you should look for when evaluating the strengths and weakness of CSPM vendors and approaches. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

All Things Open

Presented at All Things Open 2023 Presented by Carol Huang & Mike Fix - Stripe Title: Scaling an Open Source Sponsorship Program Abstract: We already know this: the open-source ecosystem needs further monetary investment from the companies that benefit most from it. Likewise, companies say they want to participate in these initiatives, but find it hard to dedicate resources to open source funding when there isn’t a clear ROI. This talk discusses how the Open Source Program Office at Stripe built a scalable, sustainable open source sponsorship model that aligns internal company incentives with those of open source maintainers and the community at large. We go over the unique “platformization” of our OSPO that allowed us to create multiple funding models, such as BYOB (Bring Your Own Budget), and share lessons learned from this experience as well as other OSPOs. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

All Things Open

Presented at All Things Open 2023 Presented by Arundeep Nagaraj - Amazon Web Services (AWS) Title: Build Developer Experience Teams for Open Source Abstract: Open Source has become the default strategy for many IT organizations and Enterprises. However, the constant challenge with Open Source leaders of these organizations has been - How is my product's developer experience? Is this the right metric to track? How can I scale my team to support our products better? How can I add automation to scale redundant workflows? If my product involves working with developers, how can I scale to the complexity of the requests and reduce Engineering bandwidth? The challenges within support of open source products continues to magnify depending on the end user persona whether they are consumers or contributors to your product. Consumers utilize your product, SDK's and API's and are blocked with using it or run into issues, whereas contributors are advanced users of your software that understands the codebase to provide a meaningful contribution back to the product. The answer to the above is to look at Open Source support as a first-class citizen of your corporate support strategy. To employ the right level of developer focused support as opposed to traditional infrastructure based support is key to scale to the amount of developers using your product. Supporting customers in the open involves more than pure support - building customer / developer experiences (DX) in the open (across platforms and communities) that pivots over the ability of your product's users or developers to be focused on the end-to-end value add. This helps with your active developer growth and retention of users. Key Takeaways: - IT leaders of Open Source will learn to employ strategies to build a DX team that engages on multiple platforms - Work on identifying accurate metrics for product and organization - Innovate on platforms such as Discord to build a bot and a dashboard - Ability to leverage customer feedback and iterate over the customer success flywheel - Distinguish between DX and Developer Advocacy (DA) Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

All Things Open

Presented at All Things Open 2023 Presented by Danny McCormick - Google Title: Deploying Models at Scale with Apache Beam Abstract: Apache Beam is an open source tool for building distributed scalable data pipelines. This talk will explore how Beam can be used to perform common machine learning tasks, with a heavy focus on running inference at scale. The talk will include a demo component showing how Beam can be used to deploy and update models efficiently on both CPUs and GPUs for inference workloads. An attendee can expect to leave this talk with a high level understanding of Beam, the challenges of deploying models at scale, and the ability to use Beam to easily parallelize their inference workloads. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

All Things Open

Presented at All Things Open 2023 Presented by Peter Czanik - One Identity Title: Sudo – Giving access while staying in control Abstract: Sudo is used by millions to control and log administrator access to systems, but using the default configuration only, there are plenty of blind spots. Using the latest features in sudo let you watch some previously blind spots and control access to them. Here are four major new features, which arrived since the 1.9.0 release, allowing you see your blind spots: - configuring a working directory or chroot within sudo often makes full shell access redundant - JSON-formatted logs give you more details on events and are easier to act on - relays in sudo_logsrvd make session recording collection more secure and reliable - you can log and control sub-commands executed by the command run through sudo Let us take a closer look at each of these. Previously, there were quite a few situations where you had to give users full shell access through sudo. Typical examples include when you need to run a command from a given directory, or running commands in a chroot environment. You can now configure the working directory or the chroot directory and give access only to the command the user really needs. Logging is a central role of sudo, to see who did what on the system. Using JSON-formatted log messages gives you even more information about events. What is even more: structured logs are easier to act on. Setting up alerting for suspicious events is much easier when you have a single parser to configure for any kind of sudo logs. You can collect sudo logs not only by local syslog, but also by using sudo_logsrvd, the same application used to collect session recordings. Speaking of session recordings: instead of using a single central server, you can now have multiple levels of sudo_logsrvd relays between the client and the final destination. This allows session collection even if the central server is unavailable, providing you with additional security. It also makes your network configuration simpler. Finally, you can log sub-commands executed from the command started through sudo. You can see commands started from a shell. No more unnoticed shell access from text editors. Best of all: you can also intercept sub-commands. These are just a few of the most prominent features helping you to watch and control previous blind spots on your systems. See these and other possibilities in action in some live demos during our presentation. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

All Things Open

Presented at All Things Open 2023 Presented by Christine Abernathy - F5, Inc. Title: Fortifying the Future: Tackling Security Challenges in AI/ML Applications Abstract: As Artificial Intelligence (AI) and Machine Learning (ML) applications continue to surge, it is crucial to be aware of and address the security risks associated with these technologies. In this talk, Christine will explore AI/ML failure modes, threats, and mitigation strategies. She will guide you through the fundamentals of ML models then introduce you to key security challenges such as adversarial attacks, data poisoning, model inversion, model stealing, and membership inference attacks, using real-world examples to demonstrate their potential impact. Christine will also discuss privacy and ethical considerations in ML, touching upon techniques like federated learning and shedding light on the current regulatory landscape surrounding security risks. If you are developing AI/ML applications or incorporating AI/ML components into your technology stack, check out this talk. You will walk away with a deeper understanding of the current AI/ML security landscape and a toolkit to help you address these risks, enabling you to build safer, more secure, and privacy-aware applications. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

All Things Open

Presented at All Things Open 2023 Presented by Carlos Santana - AWS Title: Securing Cloud Resources Deployed with Control Planes on Kubernetes using Governance and Policy as Code Abstract: Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you! We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance. By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

All Things Open

Mais de All Things Open (20)

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

Building Reliability - The Realities of Observability

Modern Database Best Practices

Modern Database Best Practices

Modern Database Best Practices

Open Source and Public Policy

Open Source and Public Policy

Open Source and Public Policy

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

The State of Passwordless Auth on the Web - Phil Nash

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

Total ReDoS: The dangers of regex in JavaScript

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

What Does Real World Mass Adoption of Decentralized Tech Look Like?

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

How to Write & Deploy a Smart Contract

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

DEI Challenges and Success

DEI Challenges and Success

DEI Challenges and Success

Scaling Web Applications with Background

Scaling Web Applications with Background

Scaling Web Applications with Background

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

Supercharging tutorials with WebAssembly

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

Using SQL to Find Needles in Haystacks

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

Configuration Security as a Game of Pursuit Intercept

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

Scaling an Open Source Sponsorship Program

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

Build Developer Experience Teams for Open Source

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

Deploying Models at Scale with Apache Beam

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

Sudo – Giving access while staying in control

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Último

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

In an era where artificial intelligence (AI) stands at the forefront of business innovation, Information Architecture (IA) is at the core of functionality. See “There’s No AI Without IA” – (from 2016 but even more relevant today) Understanding and leveraging how Information Architecture (IA) supports AI synergies between knowledge engineering and prompt engineering is critical for senior leaders looking to successfully deploy AI for internal and externally facing knowledge processes. This webinar be a high-level overview of the methodologies that can elevate AI-driven knowledge processes supporting both employees and customers. Core Insights Include: Strategic Knowledge Engineering: Delve into how structuring AI's knowledge base is required to prevent hallucinations, enable contextual retrieval of accurate information. This will include discussion of gold standard libraries of use cases support testing various LLMs and structures and configurations of knowledge base. Precision in Prompt Engineering: Learn the art of crafting prompts that direct AI to deliver targeted, relevant responses, thereby optimizing customer experiences and business outcomes. Unified Approach for Enhanced AI Performance: Explore the intersection of knowledge and prompt engineering to develop AI systems that are not only more responsive but also aligned with overarching business strategies. Guiding Principles for Implementation: Equip yourself with best practices, ethical guidelines, and strategic considerations for embedding these technologies into your business ecosystem effectively. This webinar is designed to empower business and technology leaders with the knowledge to harness the full potential of AI, ensuring their organizations not only keep pace with digital transformation but lead the charge. Join us to map a roadmap to fully leverage Information Architecture (IA) and AI chart a course towards a future where AI is a key pillar of strategic innovation and business success.

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Earley Information Science

Choosing the right accounts payable services provider is a strategic decision that can significantly impact your business's financial performance and operational efficiency. By considering factors such as expertise, range of services, technology infrastructure, scalability, cost, and reputation, businesses can make informed decisions and select a provider that aligns with their unique needs and objectives. Partnering with the right provider can streamline accounts payable processes, drive cost savings, and position your business for long-term success. https://katprotech.com/accounts-payable-and-purchase-order-automation/

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Katpro Technologies

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

A Call to Action for Generative AI in 2024

A Call to Action for Generative AI in 2024

A Call to Action for Generative AI in 2024

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Friends Colony Women Seeking Men

08448380779 Call Girls In Friends Colony Women Seeking Men

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

Advantages of Hiring UIUX Design Service Providers for Your Business

Advantages of Hiring UIUX Design Service Providers for Your Business

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

Sara Mae O’Brien Scott and Tatiana Baquero Cakici, Senior Consultants at Enterprise Knowledge (EK), presented “AI Fast Track to Search-Focused AI Solutions” at the Information Architecture Conference (IAC24) that took place on April 11, 2024 in Seattle, WA. In their presentation, O’Brien-Scott and Cakici focused on what Enterprise AI is, why it is important, and what it takes to empower organizations to get started on a search-based AI journey and stay on track. The presentation explored the complexities of enterprise search challenges and how IA principles can be leveraged to provide AI solutions through the use of a semantic layer. O’Brien-Scott and Cakici showcased a case study where a taxonomy, an ontology, and a knowledge graph were used to structure content at a healthcare workforce solutions organization, providing personalized content recommendations and increasing content findability. In this session, participants gained insights about the following: Most common types of AI categories and use cases; Recommended steps to design and implement taxonomies and ontologies, ensuring they evolve effectively and support the organization’s search objectives; Taxonomy and ontology design considerations and best practices; Real-world AI applications that illustrated the value of taxonomies, ontologies, and knowledge graphs; and Tools, roles, and skills to design and implement AI-powered search solutions.

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Enterprise Knowledge

Slack Application Development 101 Slides

Slack Application Development 101 Slides

Slack Application Development 101 Slides

Último (20)

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

Automating Google Workspace (GWS) & more with Apps Script

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

Histor y of HAM Radio presentation slide

A Call to Action for Generative AI in 2024

A Call to Action for Generative AI in 2024

A Call to Action for Generative AI in 2024

08448380779 Call Girls In Friends Colony Women Seeking Men

08448380779 Call Girls In Friends Colony Women Seeking Men

08448380779 Call Girls In Friends Colony Women Seeking Men

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

Finology Group – Insurtech Innovation Award 2024

Advantages of Hiring UIUX Design Service Providers for Your Business

Advantages of Hiring UIUX Design Service Providers for Your Business

Advantages of Hiring UIUX Design Service Providers for Your Business

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

A Year of the Servo Reboot: Where Are We Now?

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

Scaling API-first – The story of a global engineering organization

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

GenCyber Cyber Security Day Presentation

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Slack Application Development 101 Slides

Slack Application Development 101 Slides

Slack Application Development 101 Slides

Walking Around the Data Lake

1. Walking around the Data Lake Fatima Naqui Sr Big Data Engineer at Fidelity Investments Naqui.Fatima@gmail.com linkedin.com/in/fatimanaqui

2. Agenda • What is a Data Lake? • Why do I need a Data Lake? • High Level Architecture • Principles of a Data Lake • Data Lake layers and components • Ingestion • Metadata management • Data Lineage and Governance • Data Curation • Technical Architectures from Cloud providers • Q&A

3.

4. What is a Data Lake? Gartner’s definition of a Data Lake “A data lake is a collection of storage instances of various data assets. These assets are stored in a near-exact, or even exact, copy of the source format and are in addition to the originating data stores.”

5. Why do I need a Data Lake? • Cost effective way to deal with big data challenges • Single source of truth that can be reused • Cheaper and flexible alternative to batch workloads • Increase analytics and operational agility

6. High level architecture

7. Principles of a Data Lake • Separation of storage and compute • Bridging information silos • Schema on read

8. Data Lake layers and components

9. Ingestion • Land everything without modification • Dump information from source systems • Structured, unstructured, historical data

10. Metadata Management Foundation for distributed data management Supports operational use of data Data semantic variations over time Monitoring changes in data usage

11. Data Lineage and Governance Data lineage deals with data's origins Where data moves over time? What happens to the data? How is the data transformed? Data governance is the process of managing Availability Usability Security Integrity

12. Data Curation Optimize data from its raw form to be made consumable Source of insights Enrich the data with analytic and algorithmic functions

13. Technical Architectures

14. Cloud Architectures

15. Cloud Architectures