1. IBM Netezza TwinFin ® Líder em Appliances para Data Warehouse Silvio Ferrari IBM Netezza Systems Engineer [email_address]
2. Conteúdo Integrate & Cleanses Dados Estruturados Analisar Integrar Governança Dados Aplicações Transacionais & Colaborativas Gerenciar Informação Streaming Aplicações Analíticas de Negócio Streams Big Data Data Warehouses Fontes de informação Externas www Qualidade Gerenciamento de Lifecycle Segurança & Privacidade Netezza, IM e BAO Data Warehouse Appliances Master Data
7. A arquitetura IBM Netezza AMPP™ ( parte de Hardware ) Analíticos Avançados Loader ETL BI Applicações FPGA Memory CPU FPGA Memory CPU FPGA Memory CPU Discos S-Blades™ Rede Interna Netezza Appliance Hosts Host
10. Nosso segredo: FPGA CPU Descomprime Elimina colunas não usadas Restringe Visibilidade Operações complexas: ∑ Joins, Aggs, etc. select DISTRICT, PRODUCTGRP, sum(NRX) from MTHLY_RX_TERR_DATA where MONTH = '20091201' and MARKET = 509123 and SPECIALTY = 'GASTRO' Parte da tabela MTHLY_RX_TERR_DATA (comprimida) where MONTH = '20091201' and MARKET = 509123 and SPECIALTY = 'GASTRO' sum(NRX) select DISTRICT, PRODUCTGRP, sum(NRX)
12. Arquitetura IBM Netezza TwinFin™ Hardware+Software Otimizados Projetado (e não simplesmente adaptado) para tarefas analíticas de alta performance; Não necessita ajustes; Dados Streaming Aceleradores de query por Hardware, para resultados mais rápidos Verdadeiro MPP Todos os processadores totalmente utilizados para máxima eficiência e velocidade Analíticos avançados Analíticos complexos executados in-database
13.
14. Complexidade versus Simplicidade IBM Netezza Criando um database: 0. CREATE DATABASE TEST LOGFILE 'E:raDataESTOG1TEST.ORA' SIZE 2M, 'E:raDataESTOG2TEST.ORA' SIZE 2M, 'E:raDataESTOG3TEST.ORA' SIZE 2M, 'E:raDataESTOG4TEST.ORA' SIZE 2M, 'E:raDataESTOG5TEST.ORA' SIZE 2M EXTENT MANAGEMENT LOCAL MAXDATAFILES 100 DATAFILE 'E:raDataESTYS1TEST.ORA' SIZE 50 M DEFAULT TEMPORARY TABLESPACE temp TEMPFILE 'E:raDataESTEMP.ORA' SIZE 50 M UNDO TABLESPACE undo DATAFILE 'E:raDataESTNDO.ORA' SIZE 50 M NOARCHIVELOG CHARACTER SET WE8ISO8859P1; 1. Oracle* table and indexes 2. Oracle tablespace 3. Oracle datafile 4. Veritas file 5. Veritas file system 6. Veritas striped logical volume 7. Veritas mirror/plex 8. Veritas sub-disk 9. SunOS raw device 10. Brocade SAN switch 11. EMC Symmetrix volume 12. EMC Symmetrix striped meta-volume 13. EMC Symmetrix hyper-volume 14. EMC Symmetrix remote volume (replication) 15. Days/weeks of planning meetings Mudar pata 6data!!!!!!! IBM Netezza: ZERO parâmetros: CREATE DATABASE my_db;
15.
16. Complexidade Tradicional versus a Simplicidade Netezza (RDBMS 101) CREATE TABLE EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT ( RPT_PERIOD_DIM_ID NUMBER NOT NULL, SRVY_WEEK_DIM_ID NUMBER NOT NULL, DATE_DIM_ID NUMBER NOT NULL, SRVC_MKT_SEG_DIM_ID NUMBER NOT NULL, RESPD_HHLD_DIM_ID NUMBER NOT NULL, MDOTLT_DIM_ID NUMBER NOT NULL, LSTN_LOC_DIM_ID NUMBER NOT NULL, EXPSR_MIN_CNT NUMBER NOT NULL, RESPD_WGHT_NMBR NUMBER, PRELIM_DAILY_WGHT_NMBR NUMBER, FINAL_DAILY_WGHT_NMBR NUMBER, TIMESHIFT_SECOND_CNT NUMBER, BGN_EXPSR_UTC_TS DATE, END_EXPSR_UTC_TS DATE, BGN_EXPSR_LOCAL_TS DATE, END_EXPSR_LOCAL_TS DATE, BGN_BCST_UTC_TS DATE, END_BCST_UTC_TS DATE, BGN_BCST_LOCAL_TS DATE, END_BCST_LOCAL_TS DATE, SOURCE_ID VARCHAR2(50 BYTE), ACTIVE_IND CHAR(1 BYTE) DEFAULT 'Y‘ NOT NULL, INSERT_TS DATE NOT NULL, UPDATE_TS DATE NOT NULL, METADATA_ID NUMBER, MEDIA_CODE VARCHAR2(10 BYTE), MDOTLT_HIER_DIM_ID NUMBER, OUT_OF_MKT_IND CHAR(1 BYTE) ) CREATE TABLE EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT ( RPT_PERIOD_DIM_ID INTEGER NOT NULL, SRVY_WEEK_DIM_ID INTEGER NOT NULL, DATE_DIM_ID INTEGER NOT NULL, SRVC_MKT_SEG_DIM_ID INTEGER NOT NULL, RESPD_HHLD_DIM_ID INTEGER NOT NULL, MDOTLT_DIM_ID INTEGER NOT NULL, LSTN_LOC_DIM_ID INTEGER NOT NULL, EXPSR_MIN_CNT NUMERIC(9,2) NOT NULL, RESPD_WGHT_NMBR NUMERIC(9,2), PRELIM_DAILY_WGHT_NMBR NUMERIC(9,2), FINAL_DAILY_WGHT_NMBR NUMERIC(9,2), TIMESHIFT_SECOND_CNT INTEGER, BGN_EXPSR_UTC_TS TIMESTAMP, END_EXPSR_UTC_TS TIMESTAMP, BGN_EXPSR_LOCAL_TS TIMESTAMP, END_EXPSR_LOCAL_TS TIMESTAMP, BGN_BCST_UTC_TS TIMESTAMP, END_BCST_UTC_TS TIMESTAMP, BGN_BCST_LOCAL_TS TIMESTAMP, END_BCST_LOCAL_TS TIMESTAMP, SOURCE_ID VARCHAR(50), ACTIVE_IND CHAR(1) DEFAULT 'Y‘ NOT NULL, INSERT_TS TIMESTAMP NOT NULL, UPDATE_TS TIMESTAMP NOT NULL, METADATA_ID INTEGER, MEDIA_CODE VARCHAR(10), MDOTLT_HIER_DIM_ID INTEGER, OUT_OF_MKT_IND CHAR(1) ) distribute on random; 516 BASE TABLE PARTITIONS… TABLESPACE AT_EDW_REXMIN PCTUSED 0 PCTFREE 10 INITRANS 1 MAXTRANS 255 LOGGING PARTITION BY RANGE (RPT_PERIOD_DIM_ID) ( PARTITION RP0000 VALUES LESS THAN (0) NOLOGGING NOCOMPRESS TABLESPACE AT_EDW_REXMIN PCTFREE 10 INITRANS 1 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), PARTITION RP0001 VALUES LESS THAN (2) NOLOGGING NOCOMPRESS TABLESPACE AT_EDW_REXMIN PCTFREE 10 INITRANS 1 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), PARTITION RP0002 VALUES LESS THAN (3) NOLOGGING NOCOMPRESS TABLESPACE AT_EDW_REXMIN PCTFREE 10 INITRANS 1 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), … … PLUS DDL FOR 513 MORE PARTITIONS Index REXMIN_SOURCE_ID_I on 515 PARTITIONS… CREATE INDEX EDW_PROD.REXMIN_SOURCE_ID_I ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (SOURCE_ID) TABLESPACE AI_EDW_REXMIN INITRANS 2 MAXTRANS 255 LOGGING LOCAL ( PARTITION RP0000 NOLOGGING NOCOMPRESS TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), PARTITION RP0001 NOLOGGING NOCOMPRESS TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), PARTITION RP0002 NOLOGGING TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), … … PLUS DDL FOR 512 MORE PARTITIONS Index REXMIN_LLOC_FK_BI on 515 PARTITIONS… CREATE BITMAP INDEX EDW_PROD.REXMIN_LLOC_FK_BI ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (LSTN_LOC_DIM_ID) TABLESPACE AI_EDW_REXMIN INITRANS 2 MAXTRANS 255 LOGGING LOCAL ( PARTITION RP0000 NOLOGGING TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), PARTITION RP0001 NOLOGGING TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), … … PLUS DDL FOR 513 MORE PARTITIONS Index REXMIN_REHH_FK_BI on 515 PARTITIONS… CREATE BITMAP INDEX EDW_PROD.REXMIN_REHH_FK_BI ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (RESPD_HHLD_DIM_ID) TABLESPACE AI_EDW_REXMIN INITRANS 2 MAXTRANS 255 LOGGING LOCAL ( PARTITION RP0000 NOLOGGING TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), PARTITION RP0001 NOLOGGING TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), … … PLUS DDL FOR 513 MORE PARTITIONS Index REXMIN_SMS_FK_BI on 515 PARTITIONS… CREATE BITMAP INDEX EDW_PROD.REXMIN_SMS_FK_BI ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (SRVC_MKT_SEG_DIM_ID) TABLESPACE AI_EDW_REXMIN INITRANS 2 MAXTRANS 255 LOGGING LOCAL ( PARTITION RP0000 NOLOGGING TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), … … PLUS DDL FOR 514 MORE PARTITIONS Index REXMIN_SRWK_FK_BI on 515 PARTITIONS… CREATE BITMAP INDEX EDW_PROD.REXMIN_SRWK_FK_BI ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (SRVY_WEEK_DIM_ID) TABLESPACE AI_EDW_REXMIN INITRANS 2 MAXTRANS 255 LOGGING LOCAL ( PARTITION RP0000 NOLOGGING TABLESPACE AI_EDW_REXMIN PCTFREE 10 INITRANS 2 MAXTRANS 255 STORAGE ( INITIAL 96K NEXT 96K MINEXTENTS 1 MAXEXTENTS UNLIMITED PCTINCREASE 0 BUFFER_POOL DEFAULT ), … … PLUS DDL FOR 514 MORE PARTITIONS Index REXMIN_RP_FK_BI on 515 PARTITIONS… CREATE BITMAP INDEX EDW_PROD.REXMIN_SRWK_FK_BI ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (SRVY_WEEK_DIM_ID) TABLESPACE AI_EDW_REXMIN INITRANS 2 MAXTRANS 255 LOGGING LOCAL ( … … PLUS DDL FOR 515 PARTITIONS Index REXMIN_DATE_FK_BI on 515 PARTITIONS… CREATE BITMAP INDEX EDW_PROD.REXMIN_DATE_FK_BI ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (DATE_DIM_ID) TABLESPACE AI_EDW_REXMIN INITRANS 2 MAXTRANS 255 LOGGING LOCAL ( … … PLUS DDL FOR 515 PARTITIONS Index REXMIN_MEDO_FK_BI on 515 PARTITIONS… CREATE BITMAP INDEX EDW_PROD.REXMIN_MEDO_FK_BI ON EDW_PROD.EDW_RESPD_EXPSR_MIN_FACT (MDOTLT_DIM_ID)… … PLUS DDL FOR TABLESPACE + 515 PARTITIONS Oracle: 34,500 KB de DDLs Netezza: 250 KB de DDLs
17. Comparação de requerimentos de redes (internas e externas) Total: 9 endereços IP Total: 90 endereços IP 4 network drops 10 network drops minimum (with 50+ reported as being typical 5 IP addresses 68 IP addresses for Ethernet (for a single cluster) - 22 IP addresses for the InfiniBand network TwinFin12 (full rack) Exadata (full rack)
22. The IBM Netezza TwinFin™ - Expansão Em caso de expansão: - um novo sistema completo é enviado - dados migrados ONLINE - IPs são redirecionados - servidor original é desligado e devolvido
26. Família de Appliances para todo o ciclo de gerenciamento: Skimmer Sistemas de Desenvolvimento e Testes 1 TB to 10 TB TwinFin Data Warehouse Analítico de alta Performance 1 TB to 1.5 PB Cruiser Archiving acessível por SQL, Back-up / DR 100 TB to 10 PB
27. 15,000 users running 800,000+ queries per day 50X faster than before Speed Source: http:// www.youtube.com/watch?v =yOwnX14nLrE&feature= player_embedded “… when something took 24 hours I could only do so much with it, but when something takes 10 seconds, I may be able to completely rethink the business process…” - SVP Application Development, Nielsen
28. Simplicity 200X faster than Oracle system ROI in less than 3 months Up and running 6 months before having any training DAYS WEEKS MONTHS “ Allowing the business users access to the Netezza box was what sold it.” Steve Taff, Executive Dir. of IT Services
29. Scalability Source: http://www.computerweekly.com/Articles/2008/04/14/230265/NYSE-improves-data-management-with-datawarehousing.htm 1 PB on Netezza 7 years of historical data 100-200% annual data growth “ NYSE … has replaced an Oracle IO relational database with a data warehousing appliance from Netezza, allowing it to conduct rapid searches of 650 terabytes of data.” ComputerWeekly.com
30. Smart Coupon redemption rates as high as 25% Predicts what shoppers are likely to buy in future visits “ Because of (Netezza’s) in-database technology, we believe we'll be able to do 600 predictive models per year (10X as many as before) with the same staff." Eric Williams, CIO and executive VP
31.
32.
33. Indice de sucesso nas PoCs: 86% One of “ The five most important M&A Deals of 2010 ” - Wall Street Journal
34. Page Digital Media Financial Services Governo Health & Life Sciences Retail / Consumer Products Telecom Other
Before we get into what Netezza appliances are, let’s agree on what appliances are in general Appliances—ie black-box solutions—are commonplace in the IT industry We are all familiar with how they have simplified operations and revolutionized entire markets With networking appliances, even the most incompetents when it comes to IT can create a multi-user network in their homes in a matter of minutes The iPod is a great example of an appliance that simplified and revolutionized digital entertainment instead of using a PC to do the same function These appliances have some common attributes that make them very attractive compared to the old way of doing things They do only one thing, but do it better than any alternative They are plug-and-play, with a simple interface that anyone can operate And they are generally much cheaper than the alternative
Constant 2TB per hour loads with little adverse impact on queries. Opportunity: move from overnight batch loads to trickle-feeds as events occur. Wide range of complementary vendors For Oracle customers – move from row-based PL/SQL to set-based Extract – Load – Transform in-place of ETL. AOL example replace Sun / Oracle ETL stages Saves more than 7 million dollars a year Data integration partners support this “push-down processing” technique.
Once loaded – data is available There are no indexes and aggregates to update before data can be queried Partnerships with all major BI vendors While SQL-based reports are common analytics using tools such as SAS and SPSS derive greater value from the same data. I’ll now investigate TwinFin’s in-database analytics
A key component of Netezza’s performance is the way in which its streaming architecture processes data. The Netezza architecture uniquely uses the FPGA as a turbocharger … a huge performance accelerator that not only allows the system to keep up with the data stream, but it actually accelerates the data stream through compression before processing it at line rates, ensuring no bottlenecks in the IO path. You can think of the way that data streaming works in the Netezza as similar to an assembly line. The Netezza assembly line has various stages in the FPGA and CPU cores. Each of these stages, along with the disk and network, operate concurrently, processing different chunks of the data stream at any given point in time. The concurrency within each data stream further increases performance relative to other architectures. Compressed data gets streamed from disk onto the assembly line at the fastest rate that the physics of the disk would allow. The data could also be cached, in which case it gets served right from memory instead of disk. The first stage in the assembly line, the Compress Engine within the FPGA core, picks up the data block and uncompresses it at wire speed, instantly transforming each block on disk into 4-8 blocks in memory. The result is a significant speedup of the slowest component in any data warehouse—the disk. The disk block is then passed on to the Project engine or stage, which filters out columns based on parameters specified in the SELECT clause of the SQL query being processed. The assembly line then moves the data block to the Restrict engine, which strips off rows that are not necessary to process the query, based on restrictions specified in the WHERE clause. The Visibility engine also feeds in additional parameters to the Restrict engine, to filter out rows that should not be “seen” by a query e.g. rows belonging to a transaction that is not committed yet. The Visibility engine is critical in maintaining ACID (Atomicity, Consistency, Isolation and Durability) compliance at streaming speeds in the Netezza. The processor core picks up the uncompressed, filtered data block and performs fundamental database operations such as sorts, joins and aggregations on it. It also applies complex algorithms that are embedded in the snippet code for advanced analytics processing. It finally assembles all the intermediate results together from the entire data stream and produces a result for the snippet. The result is then sent over the network fabric to other S-Blades or the host, as directed by the snippet code.
We do not have indexes. They are not an option, they simply do not exist. There is no disk administration or SA administraion. Day 2, the customer has a pool of disk performant ready. Upgrades are performed by Netezza as standard maintenance tech support call. Does Oracle help you go from 9i to 10g? Instead of spending time and effort on tedious DBA tasks, use the time for higher BUSINESS VALUE tasks: Bring on new applications and groups Quickly build out new data marts Provide more functionality to your end users
Traditional architectures are much more compicated then just Stoarge + HW + RDBMS. There are multiple hops for the data. Mutliple areas of tuning. Either the customer does this themselves or pays someone to do it.
As data volumes grow, oracle complexity increases. As new indexes are created in oracle, you break existing reports. All of this (indexes, partitioing) is an attempt to out guess the user’s data access. Netezza is database 101. This is as complicated as it gets.
Updates: 10/29/03: J. Feinsmith – System no longer chooses the first column by default.
Predictability
XO Communications offers a variety of communications services including voice over internet protocol (VoIP), data and internet services, network transport, broadband wireless access, and hosted and managed services. Its high capacity IP network and advanced transport network support more than 50 percent of the Fortune 500 and many of the world’s largest telecommunications companies.
This is a number we like to boast about A number that we hope you’ll come to cherish as well and help us maintain and grow in the future This is our win-rate against Oracle, both historic and current, as of last quarter … with and without Exadata! In fact, even when we lost deals, we lost them on business grounds … against Oracle ELAs and business relationships .. and not on the technical merits of their products Obviously our obsession has paid off very well Click to proceed The acquisition is naturally making Larry nervous He knows that the success of Exadata is key to his ambitions against IBM He also knows that if he couldn’t beat Netezza as a standalone company, he doesn’t stand a chance with the combination of Netezza and IBM When it comes to data warehousing, we have the right technology leadership, experience, proven customer successes and the right formula for winning … every single time!
A Company is judged by the Company they keep. Those were just a few examples from over 500 Netezza customers Our customers span a variety of vertical industries and sizes
![IBM Netezza TwinFin ® Líder em Appliances para Data Warehouse Silvio Ferrari IBM Netezza Systems Engineer [email_address]](https://image.slidesharecdn.com/netezzatechnicaloverviewportugues-111116100541-phpapp02/85/Netezza-technicaloverviewportugues-1-320.jpg)
