2. Novedades en el manejo de
Grandes volúmenes de datos
con SQL Server 2014
Enrique Puig Nouselles
e.puig@outlook.es
@epuignouselles
Blog: www.sqlserverpasion.com
DBA at RipLife Gaming Technologies
3. Objetivos
• ¿Qué desafíos plantea trabajar con grandes volúmenes de datos?
• ¿Qué técnicas existen para afrontar estos desafíos?
• ¿Qué trae SQL Server 2014 para ayudarnos?
4. Agenda
• Desafíos de las VLT
• Mantenimiento
• Rendimiento
• Novedades de SQL Server 2014
• Reconstrucción online de Índices particionados
• Estadísticas incrementales
• Índices columnares / In-Memory DW
5. Very Large Tables (VLT)
• Gran cantidad de datos
• Millones/Billones de filas
• Datos estructurados
• OLTP
• Aplicaciones de misión crítica
• Sistemas con elevado número de TPS
• Sistemas con tracking/Logging
• DataWarehouse
• Repositorio histórico de procesos de negocio
• Procesos de carga perdiódicos
6. Desafíos de las VLT (Very Large
Tables)
Transaccional
Reportes
Índices
Estadísticas
Historificación
RendimientoMantenimiento
7. Desafíos: Mantenimiento de Índices
• Imprescindible para evitar fragmentación
• Sobrecarga E/S
• Full Scan de las tablas
• Escritura intensiva en Log
• CPU
• Bloqueos
• Soluciones actuales:
• Reorganización vs. Rebuild
• Particionado de datos
8. Desafíos: Mantenimiento de Índices (II)
Con SQL Server 2014…
• Rebuild Online a nivel de partición
• Lock Priority
• Prioridad del proceso
• MAX_DURATION
• ABORT_AFTER_WAIT
• Reducimos Bloqueos
• Mayor control de Bloqueos
9. Desafíos: Mantenimiento de
Estadísticas• Ayudan al optimizador -> Planes de ejecución
• Histogramas
• Aconsejable FULLSCAN
• Requieren de E/S
• Impactan en performance
• Soluciones actuales:
• Actualizar estadísticas con muestreos de datos
• Estrategia de actualizaciones
• Varias ventanas de mantenimiento
12. Desafíos: Rendimiento consultas
Soluciones actuales
• Índices de cobertura
• Compresión
• Particionado de datos
• Código T-SQL Eficiente
OLTP
Insert/Delete/Update
Select by Key (1 row)
24 x 7 x 365
DataWarehouse
Consultas de negocio
Group By / MAX(), MIN()…
Cargas periódicas
Híbridos
Cargas de trabajo mixtas
Impacto en rendimiento
13. Desafíos: Rendimiento consultas
Con SQL Server 2014…
• Índices columnares
• Mejoras desde 2012
• Clustered Columnstore Indexes
• Read-Write
• Particionado de datos
• Nuevo tipo de compresión de datos
14.
15. Desafíos: Historificación de datos
• Movimiento de datos “no activos”
• Almacenamiento de alto rendimiento
• limitado y caro
• Posibles soluciones actuales
• Particionado + Compresión
• Particionado + Compresión + Tabla Hist
• Particionado + compresión + BBDD hist
• Particionado + compresión + Servidor hist
• Otras….
16. Particionado + Compresión
<=M-6 M-5 M-2M-3M-4 >= MM-1
Particiones comprimidas (PAGE)
Datos “no activos”
Filegroup Readonly
Almacenamiento bajo rendimiento
Particiones sin compresión
Datos “activos”
Primary Filegroup
Almacenamiento Alto rendimiento
dbo.Ventas
17. Particionado + Compresión + Tabla
Hist
M-2 >= MM-1
dbo.Ventas
<=M-6 M-5 M-3M-4
dbo.Hist_Ventas
Particiones comprimidas (PAGE)
Datos “no activos”
Filegroup Readonly
Almacenamiento bajo rendimiento
Particiones sin compresión
Datos “activos”
Primary Filegroup
Almacenamiento Alto rendimiento
Switch IN/OUT
In this session we will take a closer look at the unique design points of Microsoft SQL Server’s in-memory solution and the significant impact it can have on your business.
Now lets take a closer look at how we can impact your business with our in-memory technology. We are the only provider to date that can speed transactions as well as queries and insights with in-memory technology optimized for each workload: OLTP, data warehousing, and analytics. With our new in-memory OLTP engine in SQL Server 2014, we have customers that have seen up to 30x faster transaction processing. I am not talking about query speed, but actual transaction write speed, up to 30x faster. I know many of you might be thinking, well Oracle and other database vendors are talking 100x. What they are talking about is query speed, not transactional speed. We are the only vendor that delivers an in-memory engine designed for OLTP transaction performance increase. There’s also built-in In-Memory columnstore for data warehousing workloads to speed queries. We were already benchmarking over 100x performance gains with many customers in the SQL Server 2012 release of the in-memory columnstore. With SQL Server 2014, the in-memory columnstore gets even better, we will talk about that in just a few minutes. Again, we can also increase query speed by over 100x.Finally we offer business users the ability to analyze data and data models much faster with built-in in-memory capabilities for Excel through PowerPivot, and Analysis Services. The benefit is that you can analyze billions of rows of data per second in Excel. Meaning a business user can analyze data of nearly any size with the tools they are most familiar with.This is what we mean when we say “driving real-time business with real-time insights.” We can significantly speed your transaction business tied to your revenue stream. We can massively speed the process to analyze both real-time transaction data, along with historical and third party data from IT or business users. This is why we are already seeing in-memory technologies transforming the way businesses run.
Now lets take a closer look at how we can impact your business with our in-memory technology. We are the only provider to date that can speed transactions as well as queries and insights with in-memory technology optimized for each workload: OLTP, data warehousing, and analytics. With our new in-memory OLTP engine in SQL Server 2014, we have customers that have seen up to 30x faster transaction processing. I am not talking about query speed, but actual transaction write speed, up to 30x faster. I know many of you might be thinking, well Oracle and other database vendors are talking 100x. What they are talking about is query speed, not transactional speed. We are the only vendor that delivers an in-memory engine designed for OLTP transaction performance increase. There’s also built-in In-Memory columnstore for data warehousing workloads to speed queries. We were already benchmarking over 100x performance gains with many customers in the SQL Server 2012 release of the in-memory columnstore. With SQL Server 2014, the in-memory columnstore gets even better, we will talk about that in just a few minutes. Again, we can also increase query speed by over 100x.Finally we offer business users the ability to analyze data and data models much faster with built-in in-memory capabilities for Excel through PowerPivot, and Analysis Services. The benefit is that you can analyze billions of rows of data per second in Excel. Meaning a business user can analyze data of nearly any size with the tools they are most familiar with.This is what we mean when we say “driving real-time business with real-time insights.” We can significantly speed your transaction business tied to your revenue stream. We can massively speed the process to analyze both real-time transaction data, along with historical and third party data from IT or business users. This is why we are already seeing in-memory technologies transforming the way businesses run.
Las very Large Tables son, comosunombreindica, tablasmuygrande… tablasquecontienen del orden de Millones o inclusobillones de filas.El origen o la generación d eestastablas en el mundo de los datosestructurados (el mundorelacional), suelevenirpor dos vias:Bien poraplicaciones de negocio con unacargatransaccionalmuyalta, normalmenteaplicaciones d emission críticaque se ejecutan 24x7x365O bienpordiseño, comoes el caso del datawarehousequeestadiseñado para almacenarinformaciónhistórica de uno o variosprocesos de negocio.
Estas Very Large tables suelengenerarproblemas y presentandesafios a los DBAs, quetenemosqueingeniarnoslas y adaptarnos a laslimitacionesquetenemos de Hardware.Los principals desafios se puedenclaisificar en términos de rendimiento y Mantenimiento… en cuanto a rendimientodistinguiriamos entre rendimientoTransaccional o de Reporting, dependiendo de la carga de trabaja del Sistema en el quetrabajeY en cuanto a mantenimientopues un poco lo de siempre… mantenimiento de índices, estadísticas y la historificación de datos.
Vamos a empezar hablando del mantenimiento de índices. El mantenimiento de índices y el de estadísticas es muy importante para los DBAs, creo que esto es algo que ya sabemos
Ahora con SQL Server ya podemos reconstruir índices a nivel de partición con la Opción ONLINE, que como ya sabéis reduce los bloqueos y además al ser en una única partición reduce el tiempo.Adicionalmente también se han añadido opciones para asignar prioridades a las operaciones Online, como reconstrucción de índices o switch de particiones, en caso de que hayan transacciones que bloqueen estas operaciones. Or lo que tenemos mas capacidad para decidir como actuar en cada momento.
Now lets take a closer look at how we can impact your business with our in-memory technology. We are the only provider to date that can speed transactions as well as queries and insights with in-memory technology optimized for each workload: OLTP, data warehousing, and analytics. With our new in-memory OLTP engine in SQL Server 2014, we have customers that have seen up to 30x faster transaction processing. I am not talking about query speed, but actual transaction write speed, up to 30x faster. I know many of you might be thinking, well Oracle and other database vendors are talking 100x. What they are talking about is query speed, not transactional speed. We are the only vendor that delivers an in-memory engine designed for OLTP transaction performance increase. There’s also built-in In-Memory columnstore for data warehousing workloads to speed queries. We were already benchmarking over 100x performance gains with many customers in the SQL Server 2012 release of the in-memory columnstore. With SQL Server 2014, the in-memory columnstore gets even better, we will talk about that in just a few minutes. Again, we can also increase query speed by over 100x.Finally we offer business users the ability to analyze data and data models much faster with built-in in-memory capabilities for Excel through PowerPivot, and Analysis Services. The benefit is that you can analyze billions of rows of data per second in Excel. Meaning a business user can analyze data of nearly any size with the tools they are most familiar with.This is what we mean when we say “driving real-time business with real-time insights.” We can significantly speed your transaction business tied to your revenue stream. We can massively speed the process to analyze both real-time transaction data, along with historical and third party data from IT or business users. This is why we are already seeing in-memory technologies transforming the way businesses run.
Now lets take a closer look at our unique in-memory design points—from our engineers deciding to make in-memory pervasive by building it in to the data platform to how we have made it easy to implement in-memory into your applications.