Variant-Apache Spark for Bioinformatics with Piotr Szul
•
1 gostou•807 visualizações
This talk will showcase work done by the bioinformatics team at CSIRO in Sydney, Australia to make Spark more useful and usable for the bioinformatics community. They have created a custom library, variant-spark, which provides a DSL and also a custom implementation of Spark ML via random forests for genomic pipeline processing. We’ve created a demo, using their ‘Hipster-genome’ and a Databricks notebook to better explain their library to the world-wide bioinformatics community. This notebooks compares results with another popular genomics library (HAIL.io) as well.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Variant-Apache Spark for Bioinformatics with Piotr Szul
Semelhante a Variant-Apache Spark for Bioinformatics with Piotr Szul (20)
Mais de Spark Summit
Mais de Spark Summit (20)
Último
Último (20)
Variant-Apache Spark for Bioinformatics with Piotr Szul
- 6. By 2025 it is estimated that 50% of the world population will have been sequenced. Spark Summit Europe 2017 | Piotr Szul6 | Frost&Sullivan
- 7. Genomics will outpace other BigData disciplines Spark Summit Europe 2017 | Piotr Szul7 | Stephens et al. PLOS Biology 2015 Astronomy Twitter YouTube Genomics
- 8. BMC Genomics 2015, 16:1052 PMID: 26651996 (IF=4) VariantSpark learns from 3000 individuals and 80 million mutations in under 30 minutes Cited 7 Spark Summit Europe 2017 | Piotr Szul8 | In the top 5% of all research outputs scored by Altmetric 31 Spark Core Spark ML MLlib Variant Spark RESEARCH 0 1000 2000 Python R H adoop Adam AD M IXTU R E VariantSpark method timeinseconds task binary−conversion clustering pre−processing
- 12. Machine learning on 1.7 Trillion data points Spark Summit Europe 2017 | Piotr Szul12 | 80 Million features Individuals Genomic profile Disease status22,500 samples Disease association identified by GWAS Spark Summit 2017 by Cotton Seed (MIT)
- 13. Look at the data Typical GWAS: 1M variants x 5K samples Full genome: 80M variants x 2.5K samples 0 1 0 … 1 1 1 1 … 1 0 0 0 … 0 0 0 1 … 1 0 1 1 … 1 0 0 0 … 0 1 2 0 … 0 ......... ......... 0 0 0 … 2 1 2 0 … 0 samples (103) variants (106) 0 1 0 0 0 0 1 ... 0 1 1 1 0 0 1 0 2 ... 0 2 0 1 0 1 1 0 0 ... 0 0 ..................... 1 1 0 1 1 0 0 ... 2 0 variants x samples transpose D N D . N 1 x samples predictors response associate 0 10,000 20,000 30,000 40,000 50,000 100,000 1,000,000 10,000,000 100,000,000 Studies 1000 Genomes samples variants
- 14. Why we needed to re-implement RF Spark Summit Europe 2017 | Piotr Szul14 | • Spark ML’s RF was designed for ‘Big’ low dimensional data. • The full genome-wide profile does not fit into the executors memory rendering the approach infeasible. “Cursed” BigData: e.g. Genomics Moderate number of samples with many features Feature set too large to be handled by single executer
- 15. How do other people try to solve this issue Spark Summit Europe 2017 | Piotr Szul15 | Firas Abuzaid (Spark Summit 2016) YGGDRASIL: Faster Decision Trees Column Partitioning in SPARK Flip the matrix: partition by column
- 16. “Cursed Forest” Spark Summit Europe 2017 | Piotr Szul16 | Flip and chop: partition by variables broadcast aggregate 1 2,1 2,2 Executors v1 v2 v3v3v3 vn … var, pointlocal best split var1, point1 var21, point21 var22, point22 global best split … initial sample split subsets Driver • Columns are “small” – easy partition • An executor can find (an exact) best split for many variables • Finding global best split is efficient
- 19. Improving Research Collaboration • Quickly access a managed Spark cluster - AWS EC2 / spot instances • Link to your data and perform whole genome analysis in real-time Jupyter Notebook Phenotype = ((2 + B6) * (1.5 + R1)) + ((0.5 + C2) * (1 + B2)) Demonstration
- 20. Try it on your data HipsterScore = ((2 + B6) * (1.5 + R1)) + ((0.5 + C2) * (1 + B2)) ; HipsterScore >10 =1 Spark Summit Europe 2017 | Piotr Szul20 | https://aehrc.github.io/VariantSpark/notebook-examples/VariantSpark_HipsterIndex_Spark2.html
- 23. Spark Summit Europe 2017 | Piotr Szul23 | Transformational Bioinformatics Denis Bauer, PhD Oscar Luo, PhD Rob Dunne, PhD Piotr Szul Team Aidan O’BrienLaurence Wilson, PhD Adrian White Andy Hindmarch Collaborators David Levy News Software Dan Andrews Kaitao Lai, PhD Arash Bayat John Hildebrandt Mia Chapman Ian Blair Kelly Williams Jules Damji Gaetan Burgio Lynn Langit Natalie Twine, PhD