Evoluindo a Plataforma de Dados do Nubank TDC SP 2019
•
2 gostaram•1,690 visualizações
O documento descreve a evolução da plataforma de dados do Nubank. Apresenta a infraestrutura de dados da empresa, incluindo o uso do Datomic e Kafka para armazenamento e processamento de dados. Também explica os processos de ingestão, análise e uso dos dados para tomada de decisões e modelos, além das ferramentas utilizadas como Databricks, Looker e Jupyter Notebooks.
Denunciar
Compartilhar
Denunciar
Compartilhar
Baixar para ler offline
Recomendados
[Pcamp19] - Scaling Nubank`s customer service with machine learning - Gustavo...
1) Em qual momento em sua estratégia de produto, você deve considerar apostas como Machine Learning?
2) Como trazer um mindset de AI para o seu time e passar a considerar essas apostas no backlog;
3) Como investir com poucos recursos e testar se AI é uma boa solução para o seu problema;
4) Como conciliar automação e uma excelente experiência para o seu cliente;
5) Os problemas de produto que você só vai descobrir quando decidir usar machine learning como uma solução. Quais são (muitas vezes parecidos em várias empresas) e como solucionamos alguns desses problemas no Nubank, que permitiu que a gente saisse de 3% para mais de 55% das dúvidas dos clientes resolvidas por uma solução com ML;
6) Como vender a importância do investimento de Machine Learning na sua empresa.
Developer Experience no Nubank
Você já ouviu falar sobre Developer Experience? Entenda como provemos uma melhor experiência para as pessoas desenvolvedoras aqui no Nubank e possibilitamos que os times de produtos se concentrem no que é mais importante: liberar novas funcionalidades para nossos clientes, de maneira rápida, consistente e confiável.
Straight Talk to Demystify Data Lineage
Are you sure you trust the data you just used for that $10 million decision? To trust data authenticity we must first understand its lineage. However, the term "Data Lineage" itself is ambiguous since it is used in different contexts. "Business Lineage" links metadata constructs to specific terms in a business glossary. This approach is used by numerous Data Governance solutions. This approach alone comes up short, since it doesn't trace the real flow of information through an organization. "Technical Lineage" traces data's journey through different systems and data stores, providing an audit trail of the changes along the way. True "Data Lineage" combines both aspects, providing context to fully understand the data life cycle. Every step in data's journey is a potential source for introduction of error that could compromise Data Quality, and hence, business decisions. In this session, Ron Huizenga offers a comprehensive discussion of data lineage and associated Data Quality remediation approaches that are essential to build a foundation for Data Governance.
Data Governance & Data Steward Certification
Becoming certified means that you have been provided some form of external review, education, assessment, or audit and that you passed that review. Being certified can make the difference in getting a job or that desirable position. Many people are seeking certification to differentiate themselves from their competition. It makes sense.
Join Bob Seiner for this month’s installment of Real-World Data Governance to explore the depth of necessity of certification in the field of data governance and the responsibility of the data stewards. Bob will talk about the different certifications available and direct you to the one that is appropriate according to your responsibilities. It may not be as easy as you think. Learn why in this webinar.
In this webinar Bob will talk about:
The Value of Being Certified
Categories of Available Certification
What to look for from Certification
Whether Certification is Right for You
Internal Versus External Certification
Image models infrastructure at OLX
OLX is an online classifieds company with over 200 million monthly active users across 30+ countries. It receives around 10 million new image uploads per day across its various marketplaces. The speaker works on Apollo, OLX's machine learning system for analyzing images. Apollo uses computer vision models to extract metadata like categories and image quality from uploaded photos. The models are trained on Amazon SageMaker but served using Kubernetes due to cost and operational advantages over SageMaker hosting. One challenge is detecting duplicate images, as 30% of images and 10% of ads are duplicates. Hashing algorithms are used to identify duplicates while being resilient to minor image changes.
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
Data Product Management at Nubank
O documento discute a gestão de produtos de dados no Nubank. Resume os desafios de implantar uma cultura de dados em uma empresa de rápido crescimento, como medir o sucesso das ferramentas de dados, e lições aprendidas como empoderar times verticais e tratar os próprios dados como produtos.
Data Infra and Data Access in Nubank
This document summarizes Nubank's data infrastructure and access. It notes that Nubank has 55 squads processing 40 TB of data daily across 190 microservices and 29 models. The goals are to make data easy and safe to use for decisions and improve decision making. The infrastructure includes platforms like Datomic, Redshift, Spark and services like Metabase, Looker, Jupyter notebooks. Over 500 people access data tools daily. There are over 300 ETL job contributions monthly to extract, transform and load data from various sources into the data warehouse. Tools like Metabase, Looker and Jupyter notebooks are used for analysis and Databricks for complex tasks. Training, support channels and meetings help users.
Recomendados
[Pcamp19] - Scaling Nubank`s customer service with machine learning - Gustavo...
1) Em qual momento em sua estratégia de produto, você deve considerar apostas como Machine Learning?
2) Como trazer um mindset de AI para o seu time e passar a considerar essas apostas no backlog;
3) Como investir com poucos recursos e testar se AI é uma boa solução para o seu problema;
4) Como conciliar automação e uma excelente experiência para o seu cliente;
5) Os problemas de produto que você só vai descobrir quando decidir usar machine learning como uma solução. Quais são (muitas vezes parecidos em várias empresas) e como solucionamos alguns desses problemas no Nubank, que permitiu que a gente saisse de 3% para mais de 55% das dúvidas dos clientes resolvidas por uma solução com ML;
6) Como vender a importância do investimento de Machine Learning na sua empresa.
Developer Experience no Nubank
Você já ouviu falar sobre Developer Experience? Entenda como provemos uma melhor experiência para as pessoas desenvolvedoras aqui no Nubank e possibilitamos que os times de produtos se concentrem no que é mais importante: liberar novas funcionalidades para nossos clientes, de maneira rápida, consistente e confiável.
Straight Talk to Demystify Data Lineage
Are you sure you trust the data you just used for that $10 million decision? To trust data authenticity we must first understand its lineage. However, the term "Data Lineage" itself is ambiguous since it is used in different contexts. "Business Lineage" links metadata constructs to specific terms in a business glossary. This approach is used by numerous Data Governance solutions. This approach alone comes up short, since it doesn't trace the real flow of information through an organization. "Technical Lineage" traces data's journey through different systems and data stores, providing an audit trail of the changes along the way. True "Data Lineage" combines both aspects, providing context to fully understand the data life cycle. Every step in data's journey is a potential source for introduction of error that could compromise Data Quality, and hence, business decisions. In this session, Ron Huizenga offers a comprehensive discussion of data lineage and associated Data Quality remediation approaches that are essential to build a foundation for Data Governance.
Data Governance & Data Steward Certification
Becoming certified means that you have been provided some form of external review, education, assessment, or audit and that you passed that review. Being certified can make the difference in getting a job or that desirable position. Many people are seeking certification to differentiate themselves from their competition. It makes sense.
Join Bob Seiner for this month’s installment of Real-World Data Governance to explore the depth of necessity of certification in the field of data governance and the responsibility of the data stewards. Bob will talk about the different certifications available and direct you to the one that is appropriate according to your responsibilities. It may not be as easy as you think. Learn why in this webinar.
In this webinar Bob will talk about:
The Value of Being Certified
Categories of Available Certification
What to look for from Certification
Whether Certification is Right for You
Internal Versus External Certification
Image models infrastructure at OLX
OLX is an online classifieds company with over 200 million monthly active users across 30+ countries. It receives around 10 million new image uploads per day across its various marketplaces. The speaker works on Apollo, OLX's machine learning system for analyzing images. Apollo uses computer vision models to extract metadata like categories and image quality from uploaded photos. The models are trained on Amazon SageMaker but served using Kubernetes due to cost and operational advantages over SageMaker hosting. One challenge is detecting duplicate images, as 30% of images and 10% of ads are duplicates. Hashing algorithms are used to identify duplicates while being resilient to minor image changes.
Building a Data Strategy – Practical Steps for Aligning with Business Goals
Developing a Data Strategy for your organization can seem like a daunting task – but it’s worth the effort. Getting your Data Strategy right can provide significant value, as data drives many of the key initiatives in today’s marketplace – from digital transformation, to marketing, to customer centricity, to population health, and more. This webinar will help demystify Data Strategy and its relationship to Data Architecture and will provide concrete, practical ways to get started.
Data Product Management at Nubank
O documento discute a gestão de produtos de dados no Nubank. Resume os desafios de implantar uma cultura de dados em uma empresa de rápido crescimento, como medir o sucesso das ferramentas de dados, e lições aprendidas como empoderar times verticais e tratar os próprios dados como produtos.
Data Infra and Data Access in Nubank
This document summarizes Nubank's data infrastructure and access. It notes that Nubank has 55 squads processing 40 TB of data daily across 190 microservices and 29 models. The goals are to make data easy and safe to use for decisions and improve decision making. The infrastructure includes platforms like Datomic, Redshift, Spark and services like Metabase, Looker, Jupyter notebooks. Over 500 people access data tools daily. There are over 300 ETL job contributions monthly to extract, transform and load data from various sources into the data warehouse. Tools like Metabase, Looker and Jupyter notebooks are used for analysis and Databricks for complex tasks. Training, support channels and meetings help users.
Technical Product Management at Nubank
Venha descobrir como é praticado Technical Product Management no Nubank. Entenda porque criamos equipes que atendem a engenharia, e os desafios de gerenciar produtos de alta complexidade técnica.
Vamos compartilhar exemplos reais dos nossos times, práticas e ferramentas, ilustrando como é possível alinhar as necessidades técnicas e de negócio.
Data Governance and Metadata Management
Metadata is a tool that improves data understanding, builds end-user confidence, and improves the return on investment in every asset associated with becoming a data-centric organization. Metadata’s use has expanded beyond “data about data” to cover every phase of data analytics, protection, and quality improvement. Data Governance and metadata are connected at the hip in every way possible. As the song goes, “You can’t have one without the other.”
In this RWDG webinar, Bob Seiner will provide a way to renew your energy by focusing on the valuable asset that can make or break your Data Governance program’s success. The truth is metadata is already inherent in your data environment, and it can be leveraged by making it available to all levels of the organization. At issue is finding the most appropriate ways to leverage and share metadata to improve data value and protection.
Throughout this webinar, Bob will share information about:
- Delivering an improved definition of metadata
- Communicating the relationship between successful governance and metadata
- Getting your business community to embrace the need for metadata
- Determining the metadata that will provide the most bang for your bucks
- The importance of Metadata Management to becoming data-centric
Lessons Learned Developing and Managing Massive (300TB+) Apache Spark Pipelin...
This document discusses lessons learned from developing and managing massive (300TB+) Apache Spark pipelines in production. It covers using Spark for its performance, testability, and modularity benefits. It also discusses managing large scale through automation, simple interfaces, planning for growth by persisting data to HDFS, using efficient serialization and data structures, and testing on sampled data to address data skew issues.
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Netflix processes trillions of events and petabytes of data a day in the Keystone data pipeline, which is built on top of Apache Flink. As Netflix has scaled up original productions annually enjoyed by more than 150 million global members, data integration across the streaming service and the studio has become a priority. Scalably integrating data across hundreds of different data stores in a way that enables us to holistically optimize cost, performance and operational concerns presented a significant challenge. Learn how we expanded the scope of the Keystone pipeline into the Netflix Data Mesh, our real-time, general-purpose, data transportation platform for moving data between Netflix systems. The Keystone Platform’s unique approach to declarative configuration and schema evolution, as well as our approach to unifying batch and streaming data and processing will be covered in depth.
Building a Data Strategy Your C-Suite Will Support
Being a data leader in any industry is an advantage that creates measurable financial benefits. Many studies have shown this – I’ve seen them from Bain, McKinsey, MIT and more. Since most firms are measured on profit, getting good at making data driven decisions is a key to being competitive. You can't get there without a plan. That is where a data strategy comes in.
In speaking with ~300 firms who indicated that their organizations were effective in using data and analytics, McKinsey found that construction of a data strategy was the number one contributing factor to their success. Being good at using data to drive decisions creates a meaningful profit advantage and those who are leaders indicated that the number one driver of their success was their data strategy.
This presentation will cover what a data strategy is, how to construct one, and how to get buy in from your executive team. The author is a former Fortune 500 Chief Data Officer and has held senior data roles at Capital One and Markel.
Here are a few helpful links for your data journey:
Free Data Investment ROI Template:
https://www.udig.com/digging-in/roi-calculator-for-it-projects/
Real world data use cases:
https://www.udig.com/our-work/?category=data
Contact Me:
https://www.udig.com/contact/
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
The document discusses how data-driven companies are performing better financially and outlines the benefits of big data and analytics. It provides examples of companies using big data and analytics to improve customer experience through personalization, predict maintenance needs, and identify at-risk veterans to prevent suicide. The challenges of big data are also reviewed. Finally, it proposes a seven-step methodology for leveraging big data and analytics to address critical business challenges.
Solution Blueprint - Customer 360
This document provides an overview of the conceptual data flow and architecture for a Customer 360 solution. Key components include extracting data from various admin systems, transforming and loading it into a data quality repository, matching and merging records in MDM, propagating updates to downstream systems like Salesforce, and enabling data steward review of matches and merges. The data flows both systematically and in response to user changes in various applications and portals.
A Carreira de um Scrum Master
O documento fornece informações sobre a carreira de um Scrum Master, incluindo suas certificações e habilidades necessárias. Ele também discute o mercado de trabalho para Scrum Masters e as principais certificações em Scrum.
DMBOK and Data Governance
Peter Vennel presents on the topic of DAMA DMBOK and Data Governance. He discusses his background and certifications. He then covers some key topics in data governance including the challenges of implementing it and defining what it is. He outlines the DAMA DMBOK knowledge areas and introduces the concept of a Data Management Center of Excellence (DMCoE) to establish governance. The DMCoE would include steering committees for each knowledge area and a data governance council and team.
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
It is common for consumer Internet companies to start off with popular third-party tools for analytics needs. Then, when the user base and the company grows, they end up building their own analytics data pipeline and query engine to cope with their data scale, satisfy custom data enrichment and reporting needs and achieve high quality of their data. That’s exactly the path that was taken at Grammarly, the popular online proofreading service.
In this session, Grammarly will share how they improved business and marketing analytics, previously done with Mixpanel, by building their own in-house analytics engine and application on top of Apache Spark. Chernetsov wil touch upon several Spark tweaks and gotchas that they experienced along the way:
– Outputting data to several storages in a single Spark job
– Dealing with Spark memory model, building a custom spillable data-structure for your data traversal
– Implementing a custom query language with parser combinators on top of Spark sql parser
– Custom query optimizer and analyzer when you want not exactly sql
– Flexible-schema storage and query against multi-schema data with schema conflicts
– Custom aggregation functions in Spark SQL
Introducción a Scrum@Scale Scrum Day Peru 2018
Este documento presenta los conceptos fundamentales de Scrum@Scale, un marco para implementar Scrum a gran escala en organizaciones. Explica los componentes clave como los diferentes niveles de Scrum Masters, Product Owners y equipos, así como el rol del Executive Action Team en la coordinación entre equipos y eliminación de impedimentos. También incluye un caso de estudio sobre cómo una compañía de gas y petróleo implementó con éxito Scrum@Scale para mejorar la eficiencia y colaboración.
Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa...
Learning is an analytic process of exploring the past in order to predict the future. Hence, being able to travel back in time to create features is critical for machine learning projects to be successful. To enable this, we built a time machine that computes features for any arbitrary time in the recent past for offline experimentation. We also built a real-time stream processing system to capture the interests of members during different times of the day and to quickly adapt to changes in the collective interests of members as it happens in case of real-world events.
Building the time machine for offline experimentation and the real-time infrastructure for online recommendations with Apache Spark (Streaming) and Apache Cassandra empowered us to both scale up the data size by an order of magnitude and train and validate the models in less time. We will delve into the architecture, use case details, data models used for cassandra and share our learnings.
About the Speakers
Prasanna Padmanabhan Engineering Manager, Netflix
Prasanna leads the Data Systems for Personalization team at Netflix. His primary focus is on building various big data infrastructure components that help their algorithmic engineers to innovate faster and improve personalization for Netflix members. In the past, he has built distributed data systems that leverages both batch and stream processing.
Roopa Tangirala Engineering Manager, Netflix
Roopa Tangirala is an experienced engineering leader with extensive background in databases, be they distributed or relational. She manages the database engineering team at Netflix responsible for operating cloud persistent and semipersistent runtime stores for Netflix, which includes Cassandra, Elasticsearch, Dynomite and MySQL databases, by ensuring data availability, durability, and scalability to meet the growing business needs.
ApacheCon Europe Big Data 2016 – Parquet in practice & detail
Apache Parquet is among the most commonly used column-oriented data formats in the big data processing space. It leverages various techniques to store data in a CPU- and I/O-efficient way. Furthermore, it has the capabilities to push-down analytical queries on the data to the I/O layer to avoid the loading of nonrelevant data chunks. With various Java and a C++ implementation, Parquet is also the perfect choice to exchange data between different technology stacks.
As part of this talk, a general introduction to the format and its techniques will be given. Their benefits and some of the inner workings will be explained to give a better understanding how Parquet achieves its performance. At the end, benchmarks comparing the new C++ & Python implementation with other formats will be shown.
SFA2018 Project to Product - Carmen DeArdo
Carmen DeArdo presentation at 2018 Souther Fried Agile conference on moving from a Project delivery model to a Product Model
Introduction to DCAM, the Data Management Capability Assessment Model - Editi...
DCAM stands for Data management Capability Assessment Model. DCAM is a model to assess data management capabilities within the financial industry. It was created by the EDM Council in collaboration with over 100 financial institutions. This presentation provides an overview of DCAM and how financial institutions leverage DCAM to improve or establish their data management programs and meet regulatory requirements such as BCBS 239. Also the benefits of DCAM are described as part of this presentation.
Data Democratization at Nubank
Nubank is the leading fintech in Latin America. Using bleeding-edge technology, design, and data, the company aims to fight complexity and empower people to take control of their finances. We are disrupting an outdated and bureaucratic system by building a simple, safe and 100% digital environment.
In order to succeed, we need to constantly make better decisions in the speed of insight, and that’s what We aim when building Nubank’s Data Platform. In this talk we want to explore and share the guiding principles and how we created an automated, scalable, declarative and self-service platform that has more than 200 contributors, mostly non-technical, to build 8 thousand distinct datasets, ingesting data from 800 databases, leveraging Apache Spark expressiveness and scalability.
The topics we want to explore are:
– Making data-ingestion a no-brainer when creating new services
– Reducing the cycle time to deploy new Datasets and Machine Learning models to production
– Closing the loop and leverage knowledge processed in the analytical environment to take decisions in production
– Providing the perfect level of abstraction to users
You will get from this talk:
– Our love for ‘The Log’ and how we use it to decouple databases from its schema and distribute the work to keep schemas up to date to the entire team.
– How we made data ingestion so simple using Kafka Streams that teams stopped using databases for analytical data.
– The huge benefits of relying on the DataFrame API to create datasets which made possible having tests end-to-end verifying that the 8000 datasets work without even running a Spark Job and much more.
– The importance of creating the right amount of abstractions and restrictions to have the power to optimize.
Camunda BPM 7.2 - English
This document summarizes a webinar about the Camunda BPM 7.2 release. It discusses key features of the new release like performance improvements, an enhanced tasklist user interface, improved support for connectors, data formats, scripting and templates, and integration with the CMMN standard for case management. Examples and live demos were provided to illustrate how these features can be used. The webinar encouraged attendees to try out the open source version and also mentioned options for the enterprise edition and support services.
Data Observability.pptx
The document discusses data observability at Intuit and how it helps prevent data incidents. It introduces Intuit's data ecosystem and challenges around data quality. It then describes Intuit's data observability model of curing, detecting, preventing and eradicating issues through techniques like data quality checks, anomaly detection, infrastructure monitoring and data triage at multiple layers. These techniques help reduce incorrect reporting, incidents, troubleshooting time and ensure SLAs are met by improving data reliability across Intuit's data platforms and products.
Understanding IT Strategy, Sourcing and Vendor Relationships
This document discusses IT strategy, sourcing, and vendor relationships. It covers key topics such as:
1. Aligning IT strategy with business strategy and exploring different IT governance models.
2. Examining IT operating plans and sourcing strategies, including discussions of outsourcing, offshoring, and the outsourcing lifecycle.
3. Important considerations for commencing relationships with IT vendors, including vendor selection, contracts, and financial stability.
Agile Placemat v9
This document provides an overview of numerous agile concepts, frameworks, and practices. It includes concepts related to lean, scrum, kanban, design thinking, and scaled agile frameworks. The document also discusses challenges organizations may face in implementing agile and how an agile delivery partner can help address common issues through various agile services.
Sql Server
O documento descreve a parceria entre a Microsoft e a Sybase para lançar o SQL Server, desde a primeira versão em 1988 até versões mais recentes. Inicialmente, a Microsoft entrou com credibilidade e a Sybase com a tecnologia do DataServe. Posteriormente, a Microsoft adquiriu os direitos da tecnologia e passou a liderar o desenvolvimento do SQL Server.
Sistemas de Informação - Aula05 - cap5 bd e business intelligence
Sistemas de Informação - Aula05 - cap5 bd e business intelligenceCEULJI/ULBRA Centro Universitário Luterano de Ji-Paraná
O documento discute sistemas de banco de dados e business intelligence. Apresenta os principais tópicos a serem abordados na aula, incluindo gestão de dados, modelagem de dados, sistemas de gerenciamento de banco de dados e aplicações como data warehouse e data mining.Mais conteúdo relacionado
Mais procurados
Technical Product Management at Nubank
Venha descobrir como é praticado Technical Product Management no Nubank. Entenda porque criamos equipes que atendem a engenharia, e os desafios de gerenciar produtos de alta complexidade técnica.
Vamos compartilhar exemplos reais dos nossos times, práticas e ferramentas, ilustrando como é possível alinhar as necessidades técnicas e de negócio.
Data Governance and Metadata Management
Metadata is a tool that improves data understanding, builds end-user confidence, and improves the return on investment in every asset associated with becoming a data-centric organization. Metadata’s use has expanded beyond “data about data” to cover every phase of data analytics, protection, and quality improvement. Data Governance and metadata are connected at the hip in every way possible. As the song goes, “You can’t have one without the other.”
In this RWDG webinar, Bob Seiner will provide a way to renew your energy by focusing on the valuable asset that can make or break your Data Governance program’s success. The truth is metadata is already inherent in your data environment, and it can be leveraged by making it available to all levels of the organization. At issue is finding the most appropriate ways to leverage and share metadata to improve data value and protection.
Throughout this webinar, Bob will share information about:
- Delivering an improved definition of metadata
- Communicating the relationship between successful governance and metadata
- Getting your business community to embrace the need for metadata
- Determining the metadata that will provide the most bang for your bucks
- The importance of Metadata Management to becoming data-centric
Lessons Learned Developing and Managing Massive (300TB+) Apache Spark Pipelin...
This document discusses lessons learned from developing and managing massive (300TB+) Apache Spark pipelines in production. It covers using Spark for its performance, testability, and modularity benefits. It also discusses managing large scale through automation, simple interfaces, planning for growth by persisting data to HDFS, using efficient serialization and data structures, and testing on sampled data to address data skew issues.
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Netflix processes trillions of events and petabytes of data a day in the Keystone data pipeline, which is built on top of Apache Flink. As Netflix has scaled up original productions annually enjoyed by more than 150 million global members, data integration across the streaming service and the studio has become a priority. Scalably integrating data across hundreds of different data stores in a way that enables us to holistically optimize cost, performance and operational concerns presented a significant challenge. Learn how we expanded the scope of the Keystone pipeline into the Netflix Data Mesh, our real-time, general-purpose, data transportation platform for moving data between Netflix systems. The Keystone Platform’s unique approach to declarative configuration and schema evolution, as well as our approach to unifying batch and streaming data and processing will be covered in depth.
Building a Data Strategy Your C-Suite Will Support
Being a data leader in any industry is an advantage that creates measurable financial benefits. Many studies have shown this – I’ve seen them from Bain, McKinsey, MIT and more. Since most firms are measured on profit, getting good at making data driven decisions is a key to being competitive. You can't get there without a plan. That is where a data strategy comes in.
In speaking with ~300 firms who indicated that their organizations were effective in using data and analytics, McKinsey found that construction of a data strategy was the number one contributing factor to their success. Being good at using data to drive decisions creates a meaningful profit advantage and those who are leaders indicated that the number one driver of their success was their data strategy.
This presentation will cover what a data strategy is, how to construct one, and how to get buy in from your executive team. The author is a former Fortune 500 Chief Data Officer and has held senior data roles at Capital One and Markel.
Here are a few helpful links for your data journey:
Free Data Investment ROI Template:
https://www.udig.com/digging-in/roi-calculator-for-it-projects/
Real world data use cases:
https://www.udig.com/our-work/?category=data
Contact Me:
https://www.udig.com/contact/
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
The document discusses how data-driven companies are performing better financially and outlines the benefits of big data and analytics. It provides examples of companies using big data and analytics to improve customer experience through personalization, predict maintenance needs, and identify at-risk veterans to prevent suicide. The challenges of big data are also reviewed. Finally, it proposes a seven-step methodology for leveraging big data and analytics to address critical business challenges.
Solution Blueprint - Customer 360
This document provides an overview of the conceptual data flow and architecture for a Customer 360 solution. Key components include extracting data from various admin systems, transforming and loading it into a data quality repository, matching and merging records in MDM, propagating updates to downstream systems like Salesforce, and enabling data steward review of matches and merges. The data flows both systematically and in response to user changes in various applications and portals.
A Carreira de um Scrum Master
O documento fornece informações sobre a carreira de um Scrum Master, incluindo suas certificações e habilidades necessárias. Ele também discute o mercado de trabalho para Scrum Masters e as principais certificações em Scrum.
DMBOK and Data Governance
Peter Vennel presents on the topic of DAMA DMBOK and Data Governance. He discusses his background and certifications. He then covers some key topics in data governance including the challenges of implementing it and defining what it is. He outlines the DAMA DMBOK knowledge areas and introduces the concept of a Data Management Center of Excellence (DMCoE) to establish governance. The DMCoE would include steering committees for each knowledge area and a data governance council and team.
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
It is common for consumer Internet companies to start off with popular third-party tools for analytics needs. Then, when the user base and the company grows, they end up building their own analytics data pipeline and query engine to cope with their data scale, satisfy custom data enrichment and reporting needs and achieve high quality of their data. That’s exactly the path that was taken at Grammarly, the popular online proofreading service.
In this session, Grammarly will share how they improved business and marketing analytics, previously done with Mixpanel, by building their own in-house analytics engine and application on top of Apache Spark. Chernetsov wil touch upon several Spark tweaks and gotchas that they experienced along the way:
– Outputting data to several storages in a single Spark job
– Dealing with Spark memory model, building a custom spillable data-structure for your data traversal
– Implementing a custom query language with parser combinators on top of Spark sql parser
– Custom query optimizer and analyzer when you want not exactly sql
– Flexible-schema storage and query against multi-schema data with schema conflicts
– Custom aggregation functions in Spark SQL
Introducción a Scrum@Scale Scrum Day Peru 2018
Este documento presenta los conceptos fundamentales de Scrum@Scale, un marco para implementar Scrum a gran escala en organizaciones. Explica los componentes clave como los diferentes niveles de Scrum Masters, Product Owners y equipos, así como el rol del Executive Action Team en la coordinación entre equipos y eliminación de impedimentos. También incluye un caso de estudio sobre cómo una compañía de gas y petróleo implementó con éxito Scrum@Scale para mejorar la eficiencia y colaboración.
Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa...
Learning is an analytic process of exploring the past in order to predict the future. Hence, being able to travel back in time to create features is critical for machine learning projects to be successful. To enable this, we built a time machine that computes features for any arbitrary time in the recent past for offline experimentation. We also built a real-time stream processing system to capture the interests of members during different times of the day and to quickly adapt to changes in the collective interests of members as it happens in case of real-world events.
Building the time machine for offline experimentation and the real-time infrastructure for online recommendations with Apache Spark (Streaming) and Apache Cassandra empowered us to both scale up the data size by an order of magnitude and train and validate the models in less time. We will delve into the architecture, use case details, data models used for cassandra and share our learnings.
About the Speakers
Prasanna Padmanabhan Engineering Manager, Netflix
Prasanna leads the Data Systems for Personalization team at Netflix. His primary focus is on building various big data infrastructure components that help their algorithmic engineers to innovate faster and improve personalization for Netflix members. In the past, he has built distributed data systems that leverages both batch and stream processing.
Roopa Tangirala Engineering Manager, Netflix
Roopa Tangirala is an experienced engineering leader with extensive background in databases, be they distributed or relational. She manages the database engineering team at Netflix responsible for operating cloud persistent and semipersistent runtime stores for Netflix, which includes Cassandra, Elasticsearch, Dynomite and MySQL databases, by ensuring data availability, durability, and scalability to meet the growing business needs.
ApacheCon Europe Big Data 2016 – Parquet in practice & detail
Apache Parquet is among the most commonly used column-oriented data formats in the big data processing space. It leverages various techniques to store data in a CPU- and I/O-efficient way. Furthermore, it has the capabilities to push-down analytical queries on the data to the I/O layer to avoid the loading of nonrelevant data chunks. With various Java and a C++ implementation, Parquet is also the perfect choice to exchange data between different technology stacks.
As part of this talk, a general introduction to the format and its techniques will be given. Their benefits and some of the inner workings will be explained to give a better understanding how Parquet achieves its performance. At the end, benchmarks comparing the new C++ & Python implementation with other formats will be shown.
SFA2018 Project to Product - Carmen DeArdo
Carmen DeArdo presentation at 2018 Souther Fried Agile conference on moving from a Project delivery model to a Product Model
Introduction to DCAM, the Data Management Capability Assessment Model - Editi...
DCAM stands for Data management Capability Assessment Model. DCAM is a model to assess data management capabilities within the financial industry. It was created by the EDM Council in collaboration with over 100 financial institutions. This presentation provides an overview of DCAM and how financial institutions leverage DCAM to improve or establish their data management programs and meet regulatory requirements such as BCBS 239. Also the benefits of DCAM are described as part of this presentation.
Data Democratization at Nubank
Nubank is the leading fintech in Latin America. Using bleeding-edge technology, design, and data, the company aims to fight complexity and empower people to take control of their finances. We are disrupting an outdated and bureaucratic system by building a simple, safe and 100% digital environment.
In order to succeed, we need to constantly make better decisions in the speed of insight, and that’s what We aim when building Nubank’s Data Platform. In this talk we want to explore and share the guiding principles and how we created an automated, scalable, declarative and self-service platform that has more than 200 contributors, mostly non-technical, to build 8 thousand distinct datasets, ingesting data from 800 databases, leveraging Apache Spark expressiveness and scalability.
The topics we want to explore are:
– Making data-ingestion a no-brainer when creating new services
– Reducing the cycle time to deploy new Datasets and Machine Learning models to production
– Closing the loop and leverage knowledge processed in the analytical environment to take decisions in production
– Providing the perfect level of abstraction to users
You will get from this talk:
– Our love for ‘The Log’ and how we use it to decouple databases from its schema and distribute the work to keep schemas up to date to the entire team.
– How we made data ingestion so simple using Kafka Streams that teams stopped using databases for analytical data.
– The huge benefits of relying on the DataFrame API to create datasets which made possible having tests end-to-end verifying that the 8000 datasets work without even running a Spark Job and much more.
– The importance of creating the right amount of abstractions and restrictions to have the power to optimize.
Camunda BPM 7.2 - English
This document summarizes a webinar about the Camunda BPM 7.2 release. It discusses key features of the new release like performance improvements, an enhanced tasklist user interface, improved support for connectors, data formats, scripting and templates, and integration with the CMMN standard for case management. Examples and live demos were provided to illustrate how these features can be used. The webinar encouraged attendees to try out the open source version and also mentioned options for the enterprise edition and support services.
Data Observability.pptx
The document discusses data observability at Intuit and how it helps prevent data incidents. It introduces Intuit's data ecosystem and challenges around data quality. It then describes Intuit's data observability model of curing, detecting, preventing and eradicating issues through techniques like data quality checks, anomaly detection, infrastructure monitoring and data triage at multiple layers. These techniques help reduce incorrect reporting, incidents, troubleshooting time and ensure SLAs are met by improving data reliability across Intuit's data platforms and products.
Understanding IT Strategy, Sourcing and Vendor Relationships
This document discusses IT strategy, sourcing, and vendor relationships. It covers key topics such as:
1. Aligning IT strategy with business strategy and exploring different IT governance models.
2. Examining IT operating plans and sourcing strategies, including discussions of outsourcing, offshoring, and the outsourcing lifecycle.
3. Important considerations for commencing relationships with IT vendors, including vendor selection, contracts, and financial stability.
Agile Placemat v9
This document provides an overview of numerous agile concepts, frameworks, and practices. It includes concepts related to lean, scrum, kanban, design thinking, and scaled agile frameworks. The document also discusses challenges organizations may face in implementing agile and how an agile delivery partner can help address common issues through various agile services.
Mais procurados (20)
Lessons Learned Developing and Managing Massive (300TB+) Apache Spark Pipelin...
Lessons Learned Developing and Managing Massive (300TB+) Apache Spark Pipelin...
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Virtual Flink Forward 2020: Netflix Data Mesh: Composable Data Processing - J...
Building a Data Strategy Your C-Suite Will Support
Building a Data Strategy Your C-Suite Will Support
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
The Data Driven Enterprise - Roadmap to Big Data & Analytics Success
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa...
Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa...
ApacheCon Europe Big Data 2016 – Parquet in practice & detail
ApacheCon Europe Big Data 2016 – Parquet in practice & detail
Introduction to DCAM, the Data Management Capability Assessment Model - Editi...
Introduction to DCAM, the Data Management Capability Assessment Model - Editi...
Understanding IT Strategy, Sourcing and Vendor Relationships
Understanding IT Strategy, Sourcing and Vendor Relationships
Semelhante a Evoluindo a Plataforma de Dados do Nubank TDC SP 2019
Sql Server
O documento descreve a parceria entre a Microsoft e a Sybase para lançar o SQL Server, desde a primeira versão em 1988 até versões mais recentes. Inicialmente, a Microsoft entrou com credibilidade e a Sybase com a tecnologia do DataServe. Posteriormente, a Microsoft adquiriu os direitos da tecnologia e passou a liderar o desenvolvimento do SQL Server.
Sistemas de Informação - Aula05 - cap5 bd e business intelligence
Sistemas de Informação - Aula05 - cap5 bd e business intelligenceCEULJI/ULBRA Centro Universitário Luterano de Ji-Paraná
O documento discute sistemas de banco de dados e business intelligence. Apresenta os principais tópicos a serem abordados na aula, incluindo gestão de dados, modelagem de dados, sistemas de gerenciamento de banco de dados e aplicações como data warehouse e data mining.Power Tuning - Portfolio de BI
Serviços prestados de BI
Criação e manutenção de rotinas de ETL / ELT utilizando T-SQL, Integration Services (SSIS), Azure Data Factory, Azure Synapse Pipelines ou Pentaho (PDI)
Migração de projetos de BI On-premisses (local) para Azure como serviço (SaaS)
Modelagem e arquitetura de projetos de BI e banco de dados
Performance Tuning em processos de carga de ETL demorados ou pesados para o banco
Criação de rotinas de ETL para automatizar processos manuais que extraem dados do sistema e realizam diversos tratamentos/Macro/Fórmulas no Excel
Automatização de processos utilizando serviços do Azure para ler dados de origens diversas (Ex: E-mail) e gravar em tabelas do banco para consumir no Power BI
Criação e manutenção de cubos OLAP do Analysis Services (Multidimensional e Tabular) para carregar grandes volumes de dados para o Power BI de forma instantânea
Criação e manutenção de rotinas Integration Services
Criação/manutenção de relatórios utilizando Transact-SQL e envio por e-mail (Excel ou CSV)
Criação/manutenção de relatórios utilizando Power BI (Pro, Premium, Embedded ou Report Server)
Criação/manutenção de relatórios paginados utilizando Report Builder (Reporting Services ou Power BI)
Apoio na implantação de projetos de BI e Power BI
Análise de erros e problemas em relatórios, fórmulas, gateway, pacotes do SSIS e em cubos OLAP do Analysis Services
Treinamentos internos personalizados de banco de dados, T-SQL, Integration Services, Analysis Services, Reporting Services e Power BI
Venda de Licenças de Power BI (Conta Pro, Premium, Embedded e Report Server)
Mudança de paradigma no monitoramento de banco de dados
O documento descreve uma apresentação sobre a ferramenta Aimbetter, que fornece monitoramento inteligente de bancos de dados em tempo real. A Aimbetter monitora o desempenho do banco de dados, identifica problemas e envia alertas para permitir uma resolução rápida. Ela também fornece análises avançadas de consultas para ajudar desenvolvedores a identificar e corrigir erros no código. A ferramenta oferece maior produtividade e visibilidade para equipes de DBA, desenvolvedores e centros de dados.
Comprei um SQL Server e agora?
Apresentação realizada no 5º do SQL Server Local Group do Espírito Santo, onde abordei sobre as ferramentas que estão inclusas no SQL Server e que ao adquirir as empresas podem utilizar para desenvolver muitos projetos em diversas áreas como cloud, in-memory, analytics, BI, dentre outros.
AWS Innovate 2020 - Entenda como o Data Flywheel pode apoiá-lo em sua estraté...
AWS Innovate 2020 - Entenda como o Data Flywheel pode apoiá-lo em sua estraté...Amazon Web Services LATAM
AWS Innovate 2020 - Entenda como o Data Flywheel pode apoiá-lo em sua estratégia de dados - Hugo RozestratenWebinar: Ask the Experts - Big Data (Português)
Na sessão sobre Big Data, apresentamos as soluções baseadas na nuvem e o portfólio dos produtos/serviços da AWS.
SQL SAT Salvador - Arquitetando Data Lake Multicloud
O documento discute a arquitetura de um Data Lake multicloud. Primeiro, explica os conceitos de Data Lake e as diferenças em relação a data warehouse. Em seguida, aborda os desafios na arquitetura de um Data Lake multicloud, incluindo a captura e processamento de dados de múltiplas fontes, o consumo dos dados e a governança. Por fim, apresenta exemplos de arquiteturas em nuvens diferentes e estratégias como virtualização e gravidade dos dados.
Apresentação Power Embedded - Descubra uma nova forma de compartilhar relatór...
O que é o Power Embedded?
Uma aplicação web que permite reduzir o custo do Power BI em até 90%
➢ Usuários que precisam apenas visualizar relatórios acessam o Power Embedded
ao invés do portal do Power BI, e por isso, não precisam de licença PRO
Usuários que criam relatórios continuam acessando powerbi.com para configurar
atualizações, gateways e tudo mais.
➢ Permissões de acesso aos relatórios e RLS são configurados no Power Embedded
e não mais no portal do Power BI
Investimento necessário
Pagamento único
➢ Setup inicial e instalação: R$ 1.500,00 – Power Tuning
Mensal
➢ Sistema: R$ 5,00 (por usuário) – Power Tuning
➢ Capacidade Embedded ou Fabric: A partir de R$ 500,00* – Microsoft
O que são essas capacidades?
São recursos do Power BI que licenciam a solução do Power Embedded junto à
Microsoft e permitem utilizar as API’s, de forma legal e ilimitada, a inserir relatórios
publicados em aplicações terceiras e não precisar de licença para visualizar os relatórios
Tipos de capacidades (todas são suportadas pelo Power Embedded):
➢ Fabric: Mais nova, flexível, mais recursos e pode começar mais barata.
➢ Power BI Embedded: Mais estável, confiável e pode ser mais barata no pago pelo uso.
➢ Premium por Capacidade: A mais completa, mas o custo inicial é 32 mil reais por mês.
• As capacidades são contradas direto com a Microsoft (ou parceiro) pelo Azure.
• Ter uma capacidade é um requisito para o Power Embedded funcionar.
• Microsoft libera um período de avaliação gratuito de 60 dias para o Fabric.
Por quê preciso do Power Embedded?
Apenas contratando a capacidade, você ainda precisará de licenças PRO para
visualizar os relatórios.
➢ O Power Embedded que cria essa comunicação com as API’s do Power BI e permite
visualizar os relatórios pelo Sistema, sem precisar de licença.
➢ Também é o Power Embedded que implementa o controles de segurança (RLS e
OLS) e garante que apenas os usuários com permissão podem visualizar o relatório.
➢ Se você não quiser gastar centenas de horas de programação para criar o seu portal e
todos os controles de segurança, você precisa do Power Embedded. ☺
Apresentação Power Embedded - Descubra uma nova forma de compartilhar relatór...
O que é o Power Embedded?
Uma aplicação web que permite reduzir o custo do Power BI em até 90%
➢ Usuários que precisam apenas visualizar relatórios acessam o Power Embedded
ao invés do portal do Power BI, e por isso, não precisam de licença PRO
Usuários que criam relatórios continuam acessando powerbi.com para configurar
atualizações, gateways e tudo mais.
➢ Permissões de acesso aos relatórios e RLS são configurados no Power Embedded
e não mais no portal do Power BI
Investimento necessário
Pagamento único
➢ Setup inicial e instalação: R$ 1.500,00 – Power Tuning
Mensal
➢ Sistema: R$ 5,00 (por usuário) – Power Tuning
➢ Capacidade Embedded ou Fabric: A partir de R$ 500,00* – Microsoft
O que são essas capacidades?
São recursos do Power BI que licenciam a solução do Power Embedded junto à
Microsoft e permitem utilizar as API’s, de forma legal e ilimitada, a inserir relatórios
publicados em aplicações terceiras e não precisar de licença para visualizar os relatórios
Tipos de capacidades (todas são suportadas pelo Power Embedded):
➢ Fabric: Mais nova, flexível, mais recursos e pode começar mais barata.
➢ Power BI Embedded: Mais estável, confiável e pode ser mais barata no pago pelo uso.
➢ Premium por Capacidade: A mais completa, mas o custo inicial é 32 mil reais por mês.
• As capacidades são contradas direto com a Microsoft (ou parceiro) pelo Azure.
• Ter uma capacidade é um requisito para o Power Embedded funcionar.
• Microsoft libera um período de avaliação gratuito de 60 dias para o Fabric.
Por quê preciso do Power Embedded?
Apenas contratando a capacidade, você ainda precisará de licenças PRO para
visualizar os relatórios.
➢ O Power Embedded que cria essa comunicação com as API’s do Power BI e permite
visualizar os relatórios pelo Sistema, sem precisar de licença.
➢ Também é o Power Embedded que implementa o controles de segurança (RLS e
OLS) e garante que apenas os usuários com permissão podem visualizar o relatório.
➢ Se você não quiser gastar centenas de horas de programação para criar o seu portal e
todos os controles de segurança, você precisa do Power Embedded. ☺
Construção da Plataforma de dados Autônoma e em Escala
Vamos falar um pouco sobre minha trajetória na construção de uma plataforma de dados autônoma, agnóstica e escalável. Vou contar um pouco sobre o que tínhamos com soluções de Data Warehousing clássicas, os problemas que buscamos resolver e a solução que chegamos.
PowerBI na Pártica com Indicadores Elicitados com MindMap e Canvas consumidos...
O documento apresenta uma palestra sobre Inteligência de Negócios (BI) realizada pelo professor Aislan Honorato. A palestra abordou tópicos como banco de dados relacional e OLAP, framework de BI, levantamento de indicadores, data warehouse, ETL, cubos dimensionais e ferramentas de visualização como Power BI. O professor também apresentou seu currículo e realizou uma demonstração prática no Power BI.
Aula tecnologia da informacao 6 banco de dados
O documento explica os objetivos da aula sobre gerenciamento de bancos de dados, incluindo explicar a importância da implementação de processos e tecnologias de gerenciamento de dados, as vantagens de uma abordagem de gerenciamento de bancos de dados e como o software de gerenciamento de banco de dados auxilia as empresas. Também dará exemplos para ilustrar conceitos-chave sobre bancos de dados.
Apresentacao Vertical De Alocacao Consultor Adriana
Adriana apresentou seu trabalho atual de segmentação de clientes para a CTBC, incluindo a definição de regras de negócio, extração e análise de dados, e construção de modelos estatísticos. O projeto tem como objetivo aumentar a participação de mercado em segmentos importantes e melhorar a lucratividade da empresa.
VTEX @ Splunk Live! São Paulo
O documento discute como a VTEX usa o Splunk para coletar e analisar logs, métricas e dados de máquinas para monitoramento e fornecer insights de negócios aos clientes. Antes do Splunk, a VTEX enfrentava desafios para centralizar e analisar grandes volumes de dados gerados. O Splunk permitiu a criação de um ambiente centralizado para logs e o desenvolvimento de aplicativos para análises específicas.
Vtex - Splunk live! 2014 São Paulo
O documento discute como a VTEX usa o Splunk para coletar e analisar logs, métricas e dados de máquinas para monitoramento e fornecer insights de negócios aos clientes. Antes do Splunk, a VTEX enfrentava desafios para centralizar e analisar grandes volumes de dados gerados. O Splunk permitiu a criação de um ambiente centralizado para logs e o desenvolvimento de aplicativos para análises específicas.
Apresentação Opensys Serviços especializados em Bancos de Dados
A Opensys é uma empresa de tecnologia fundada em 2003 que oferece serviços de administração de banco de dados, soluções de TI e hardware. A empresa emprega especialistas qualificados e utiliza ferramentas avançadas para fornecer soluções inovadoras e personalizadas aos clientes. A Opensys tem como missão ser referência na elaboração de projetos tecnológicos que agreguem valor aos clientes.
AWS Data Immersion Webinar Week - Planeje e entenda como criar um repositório...
AWS Data Immersion Webinar Week - Planeje e entenda como criar um repositório...Amazon Web Services LATAM
O documento discute a construção de data lakes de maneira segura seguindo boas práticas. Ele explica o que é um data lake e como a AWS Lake Formation facilita a criação, ingestão de dados, governança e segurança em data lakes na nuvem da AWS de maneira gerenciada. O documento também aborda desafios comuns na construção de data lakes e como a solução da AWS ajuda a superá-los de forma escalonável.Escalando infra em ops em um ambiente de hiper crescimento
O documento descreve a jornada da Nubank na escalada de sua infraestrutura e operações para acompanhar o hipercrescimento da empresa. Apresenta os desafios enfrentados no início com poucos engenheiros e a evolução para equipes maiores e novas estruturas com o Spotify model e tribos. Também destaca a ênfase contínua em melhorar a confiabilidade, desempenho e escalabilidade dos serviços.
Como Implementar a Análise de Dados em Tempo Real
Watch full webinar here: https://bit.ly/31rnBD0
As técnicas de análise em tempo real prometem enriquecer a análise tradicional de dados. Isto é fundamental para muitos cenários, tais como gerenciamento dos processos de produção ou atendimento ao cliente. A virtualização de dados é bem conhecida por oferecer conectividade em tempo real a diversas fontes e recursos de federação - os dois ingredientes básicos para análises de dadis em tempo real. No entanto, construir uma estratégia em torno destes conceitos pode ser um desafio. O impacto de fontes de dados sensíveis, questões de segurança e desempenho são freqüentemente mencionados.
Participe deste webinar e descubra:
- Quais são os cenários onde o valor da análise de dados em tempo real pode fazer a diferença
- As principais capacidades que as tornam possíveis
- As melhores práticas para torná-las bem sucedidas
Semelhante a Evoluindo a Plataforma de Dados do Nubank TDC SP 2019 (20)
Sistemas de Informação - Aula05 - cap5 bd e business intelligence
Sistemas de Informação - Aula05 - cap5 bd e business intelligence
Mudança de paradigma no monitoramento de banco de dados
Mudança de paradigma no monitoramento de banco de dados
AWS Innovate 2020 - Entenda como o Data Flywheel pode apoiá-lo em sua estraté...
AWS Innovate 2020 - Entenda como o Data Flywheel pode apoiá-lo em sua estraté...
SQL SAT Salvador - Arquitetando Data Lake Multicloud
SQL SAT Salvador - Arquitetando Data Lake Multicloud
Apresentação Power Embedded - Descubra uma nova forma de compartilhar relatór...
Apresentação Power Embedded - Descubra uma nova forma de compartilhar relatór...
Apresentação Power Embedded - Descubra uma nova forma de compartilhar relatór...
Apresentação Power Embedded - Descubra uma nova forma de compartilhar relatór...
Construção da Plataforma de dados Autônoma e em Escala
Construção da Plataforma de dados Autônoma e em Escala
PowerBI na Pártica com Indicadores Elicitados com MindMap e Canvas consumidos...
PowerBI na Pártica com Indicadores Elicitados com MindMap e Canvas consumidos...
Apresentacao Vertical De Alocacao Consultor Adriana
Apresentacao Vertical De Alocacao Consultor Adriana
Apresentação Opensys Serviços especializados em Bancos de Dados
Apresentação Opensys Serviços especializados em Bancos de Dados
AWS Data Immersion Webinar Week - Planeje e entenda como criar um repositório...
AWS Data Immersion Webinar Week - Planeje e entenda como criar um repositório...
Escalando infra em ops em um ambiente de hiper crescimento
Escalando infra em ops em um ambiente de hiper crescimento
Evoluindo a Plataforma de Dados do Nubank TDC SP 2019
- 1. Evoluindo a Plataforma de Dados do Nubank 16/07/2019 Felipe Sabino André Tavares
- 2. Felipe Guilherme Sabino Engenheiro de Dados www.linkedin.com/in/fgsabino/
- 3. André de Lannoy Tavares Gerente de Produto www.linkedin.com/in/andre-lannoy-tavares/
- 4. Nubank Infraestrutura Uso de Dados Plataforma O Futuro Dados
- 5. Nubank
- 6. • Maior fintech fora da Asia • 10 milhões de clientes • 1800 funcionários • 16 Tribes e 70 Squads
- 7. • Cartão de crédito sem anuidade • Conta Digital • Programa de recompensas • Empréstimo pessoal
- 9. Para que o Nubank usa dados? • Para tomar decisões de crédito • Para melhor a experiência dos clientes com produtos mais inteligentes • Para gerar controles e relatórios para a empresa, seus acionistas e reguladores • Para ajudar os nubankers a tomarem decisões melhores
- 10. • 250 microsserviços • 30 modelos em produção • 70Tb de dados processados por dia • 1100 usuários semanais nas ferramentas de dados
- 11. Dados como uma plataforma Aquisição de Clientes Atendimento ao Cliente Cartão de Crédito Plataforma de Dados Empréstimos Pessoais Relatórios Financeiros ....
- 12. Tribo de Dados ! Times de Dados Acesso a Dados "Infra de Dados #
- 15. Datomic e Entity Long a Attribute String v Value Any t Transaction point in time Long tx Transaction entity id Long txInstant Transaction wall-clock time java.util.Date op Operation (assertion / retraction) Boolean • Imutabilidade • Dados não são sobrescritos • ACID
- 16. [ 28 ':name' 'felipe' ] Datomic
- 17. entidade atributo valor Datomic [ 28 ':name' 'felipe' ]
- 18. [ 28 ':name' 'felipe' ] entidade atributo valor 'sabino' Datomic
- 19. [ 28 ':name' 'felipe' Tx₁] transação Datomic entidade atributo valor [ 28 ':name' 'sabino' Tx₂]
- 20. Plataforma
- 21. Por que construir um ambiente analítico?
- 22. Ambos são necessários AMBIENTE TRANSACIONAL AMBIENTE ANALÍTICO Dificuldade na escritaDificuldade na leitura Unificado, GlobalFragmentado, Local Alta acessibilidade SQL somente leitura com ACL Baixa acessibilidade Datalog, bancos de produção, segurança
- 23. Data Processing Data WarehouseData Ingestion Data Analysis Batch Models
- 25. Ingestão de Dados do Datomic 1. Descoberta 2. Alocação dos bancos 3. Extração dos Logs 4. Empacotamento para S3
- 26. Ingestão de Outros Dados via Kafka Serviço que acumula as mensagens recebidas salva no Data Lake
- 27. • Utilizamos Amazon S3 • Ambiente Isolado • Gestão de Metadados Nosso Data Lake
- 28. Serviço de Gestão de Metadados • "Mapa" do ETL: contém os metadados de cada dataset, incluindo onde eles estão armazenados no S3 • Os serviços não tem interferência direta nos dados
- 30. Nossa engine de processamento de dados • O “motor" do ETL; descreve todas as transformações para produzir os datasets • Feito para processar grandes quantidades de dados • Em Scala, fácil de usar porque abstrai as partes complexas do Spark • Declarativo
- 31. Organização dos Datasets DB1 Log S0 DB1 Log S1 DB2 Log S0 Dataset Series contract 1 contract 2 dataset 1 dataset 2 model policy
- 32. Datasets • Qualquer pessoa na empresa pode criar um Dataset e contribuir com um Pull Request em nosso monorepo • As equipes são responsáveis por escrever e manter seus Datasets • Os Datasets são escritos em Scala; DSLs são fornecidas • Usamos Databricks para iterar • Revisão de PRs são feitas por pares para garantir qualidade e consistência • 150 contribuidores fazendo 300+ contribuições por mês
- 33. Gerenciando DAGs
- 34. Uso de Dados
- 35. Modelos
- 36. Modelos em Batch • São executados com o ETL • Python • Ferramentas que facilitam: • Validar configuração • Treinar e testar diferentes parâmetros • Simular comportamento de produção localmente • Enviar para Produção Dados Modelos Score / Prob. Políticas Decisões
- 37. Serving Layer
- 38. Mandando dados de volta para produção • Serviço na stack de dados carrega as políticas em um Dynamo DB • Serviço na stack de produção acessa o mesmo banco e oferece interfaces para o consumo dessas políticas
- 39. Data Warehouse
- 40. BigQuery • Serviço que transfere do S3 para o GCS • Carrega os datasets no BigQuery • Fully managed
- 42. Looker
- 44. Databricks
- 45. Democratização e Cultura de Dados
- 46. Treinamentos
- 47. Suporte
- 49. Recap
- 50. Datomic Extractor Online Models Production Services Jupyter Notebooks Looker Batch Models Spark Engine Production Services Datomic DB Logs Amazon S3 Metadata Manager Non-datomic Datasources Non-datomic Extractor Static Inputs Databricks Loader Databricks Serving Layer Loader Amazon Dynamo DB Serving Layer Plataforma de Dados Nubank BigQuery BigQuery Loader
- 51. O Futuro
- 52. O Futuro • Plataforma Internacional • Governança de Dados • Portal de Metadados • Continuar Escalando