At the StampedeCon 2013 Big Data conference in St. Louis, Jeff Melching, Big Data Engineer and Architect at Monsanto, discussed Legacy Analysis: How Hadoop Streaming Enables Software Reuse – A Genomics Case Study. The bioinformatics domain and in particular computational genomics has always had the problem of computing analytics against very large data sets. Traditionally, these analytics have leveraged grid and compute farm technologies. Additionally, the analytics software and algorithms have been built up over the past 30 years by contributions from both the public and private domain and written in a number of programming languages. When these software packages are brought in house and combined with the skills and preferences of internal bioinformatics researchers, what you get is a myriad of different technologies linked together in an analytics pipeline. The rise of technologies like MapReduce in hadoop have made the execution of such pipelines much more efficient, but what about all those analytic pipelines I have built up over the years that aren’t written in MapReduce? Do I have to rewrite them? Do I have to know java? This talk will explain how hadoop streaming can help you reuse instead of rewriting. It will also touch on techniques for packaging and deploying hadoop applications without having to centrally manage software versions on the cluster.

1. Jeff Melching
Monsanto: Lead Big Data Engineer
Twitter: @melchbox
8/7/2013

2. Intro to the genomic space @Monsanto
Strategy for Legacy Analysis
Example Use Cases
htseq-count: Counting reads in features
Genotype Scoring
Crossbow

3. “We succeed when farmers succeed.”
-Hugh Grant, Monsanto CEO
Monsanto Company is a leading global provider of
technology-based tools and agricultural products
that improve farm productivity and food quality.
We work to deliver agricultural products and
solutions to:
• Meet the world’s growing food needs
• Conserve natural resources
• Protect the environment
Monsanto Company Confidential

4. Genomics : a discipline in genetics that applies
recombinant DNA, DNA sequencing
methods, and bioinformatics to
sequence, assemble, and analyze the function
and structure of genomes (the complete set of
DNA within a single cell of an organism)
http://en.wikipedia.org/wiki/Genomics

5. New gene discovery, gene expression
Evolutionary population genetics
Insect Control
Disease resistance targets
Marker discovery and variation analysis
Genotyping and fingerprinting
New vegetable reference genomes
Marker discovery and variation analysis
Disease resistance
Viral and fungal resistance
Targets for topical RNAi
Seed Treatments
Yield & Stress
Agricultural Traits
Molecular Breeding
Vegetable Quality &
Disease
Plant Health
Chemistry

6. 30+ years of increasing computational
power, open source tools and knowledge
Two distinct workloads
Production workflows
Discovery analytics
Computational pipelines

8. Grid Computing
High Performance
Storage
File Processing
Hadoop
HDFS
Block processing
perl
python
C/C++
R
bash
MapReduce
Java
Pig
Hive

9. The work is done, why port it to java?
Can I get it done quickly?
Where’s the value?

10. Math is hard
Genomic algorithms are harder
Coding it is harder still
Coding it correctly…

11. 11
http://gapingvoid.com/2008/06/13/now-what/

12. Minimizing change in order to leverage the
existing pipelines, tools and knowledge in their
natural state, requires a common platform that
is language neutral and easily consumable
stdin & stdout

13. Creates map and reduce tasks
Controls map and reduce defined executables
Feeds data to stdin of the process, collects output
from stdout
Equivalent to using pipes
Input Mapper Reducer
Map Exe
Reduce
Exe
stdoutstdin
Output
stdoutstdin

14. Algorithm of existing executables
parallelizable?

15. Can existing code operate on or be easily
modified to support stdin & stdout?
If not, can you wrap it?

16. Identify decision
points to split
code into
MapReduce style
http://www.recessframework.org/blog/category/PHP

17. Minimize Change

18. Test in local mode first
$ cat inputFile | mapper.sh | reducer.sh > outputFile
http://wiki.apache.org/hadoop/HadoopStreaming

20. “Given a file with aligned sequencing reads and
a list of genomic features, a common task is to
count how many reads map to each feature.”
http://www-huber.embl.de/users/anders/HTSeq/doc/count.html
gene
read
read
read
2

21. hadoop jar $HADOOP_STREAMING
-input my_experiment.sam -output count
--mapper 'mapper -q -s no - features.gtf'
--reducer ‘reducer.py'
-file dist/mapper -file features.gtf -file reducer.py
htseq-count –q –s no my_experiment.sam
features.gtf

22. #… do crazy parsing using python libs and stuff…
try:
read_seq = iter( HTSeq.SAM_Reader( sys.stdin ) )
first_read = read_seq.next()
read_seq = itertools.chain( [ first_read ], read_seq )
pe_mode = first_read.paired_end
for r in read_seq:
# do more algorithm and validation stuff…
fs = set()
for iv in iv_seq:
if iv.chrom not in features.chrom_vectors:
raise UnknownChrom
for iv2, fs2 in features[ iv ].steps():
fs = fs.union( fs2 )
if fs is None or len( fs ) == 0:
empty += 1
elif len( fs ) > 1:
ambiguous += 1
else:
counts[ list(fs)[0] ] += 1
for fn in sorted( counts.keys() ):
print "%st%d" % ( fn, counts[fn] )

23. #!/usr/bin/env python
import sys
current_fn = None
current_count = 0
fn = None
for line in sys.stdin:
fn, count = line.split('t',1)
count = int(count)
if current_fn == fn:
current_count += count
else:
if current_fn:
print '%st%s' % (current_fn, current_count)
current_count = count
current_fn = fn
if current_fn == fn:
print '%st%s' % (current_fn, current_count)

24. Split python script into mapper and reducer
No Change to command line args
Reused all dependent libraries
Run in MR mode or local mode
$ cat my_experiment.sam | mapper -q -s no - features.gtf | sort |
python reducer.py

26. Analysis determines variants and quality scores
Legacy code written in Java and R part of a larger
pipeline
Embedded on app server and responds to JMS

27. Parallelizable
Wrapped input to read from streamed files
Map operates on a plate of data at one time
Minimal change only in how the data passed

28. Reused existing Java and R code by simply writing
a transformation class
2 days of work during “Innovation Days” to modify
for Hadoop by following this strategy
Value: 75k plates in 4 minutes on 14 node cluster

30. Crossbow is a scalable software pipeline for whole
genome resequencing analysis. It combines
Bowtie, an ultrafast and memory efficient short
read aligner, and SoapSNP, and an accurate
genotyper. These tools are combined in an
automatic, parallel pipeline…
http://bowtie-bio.sourceforge.net/crossbow/index.shtml

32. “Hadoop also supports a 'streaming' mode of
operation whereby the map and reduce
functions are delegated to command-line scripts
or compiled programs written in any language.
…
This allows Crossbow to reuse existing software
for aligning reads and calling SNPs while
automatically gaining the scaling benefits of
Hadoop. “
http://genomebiology.com/2009/10/11/R134
http://www.cbcb.umd.edu/~mschatz/Posters/Crossbow_WABI_Sept2009.pdf

33. Crossbow-specific output format was
implemented that encodes an alignment as a
tuple where the tuple's key identifies a
reference partition and the value describes the
alignment.
http://genomebiology.com/2009/10/11/R134
http://bowtie-bio.sourceforge.net/crossbow/

34. A new input format (option --12) was added, allowing
Bowtie to recognize the one-read-per-line format
produced by the Crossbow preprocessor.
http://genomebiology.com/2009/10/11/R134

35. The version of SOAPsnp used in Crossbow was
modified to accept alignment records output by
modified Bowtie ... None of the modifications made to
SOAPsnp fundamentally affect how consensus bases or
SNPs are called
http://genomebiology.com/2009/10/11/R134

36. Adoption of Hadoop for discovery
Rapid development times
Executables as stand-alone packages

37. Assess fit
It’s still MapReduce
Minimize change
Test in local mode
http://i.qkme.me/3pgc1j.jpg