Indexing big data in the cloud
Presented by Scott Stults | OpenSource Connections. See conference video - http://www.lucidimagination.com/devzone/events/conferences/lucene-revolution-2012 Amazon Web Services offers a quick and easy way to build a scalable search platform, a flexibility is especially useful when an initial data load is required but the hardware is no longer needed for day-to-day searching and adding new documents. This presentation will cover one such approach capable of enlisting hundreds of worker nodes to ingest data, track their progress, and relinquish them back to the cloud when the job is done. The data set that will be discussed is the collection of published patent grants available through Google Patents. A single Solr instance can easily handle searching the roughly 1 million patents issued between 2010 and 2005, but up to 50 worker nodes were necessary to load that data in a reasonable amount of time. Also, the same basic approach was used to make three sizes of PNG thumbnails of the patent grant TIFF images. In that case 150 worker nodes were used to generate 1.6 Tb of data over the course of three days. In this session, attendees will learn how to leverage EC2 as a scalable indexer and tricks for using XSLT on very large XML documents.
Recommended
Recommended
More Related Content
Viewers also liked
Similar to Indexing big data in the cloud
Similar to Indexing big data in the cloud (20)
More from lucenerevolution
More from lucenerevolution (20)
Recently uploaded
Recently uploaded (20)
Indexing big data in the cloud
- 1. Indexing Big Data in the Cloud
- 2. Me Scott Stults Co-Founder of OpenSource Connections Solr / Lucene Bash / Python / Java 2 Indexing Big Data in the Cloud
- 3. Eric 3 Indexing Big Data in the Cloud
- 4. Big Data Indexing Big Data in the Cloud 4
- 5. Big Data Wrangler 5 Indexing Big Data in the Cloud
- 6. How? Address a Real Project Be Agile Make Small Mistaeks Fast Succeed BIG 6 Indexing Big Data in the Cloud
- 7. USPTO Goals Prototype Search UX Prove Solr: Scales Integrates Excels 7 Indexing Big Data in the Cloud
- 8. Scale? 8 Indexing Big Data in the Cloud
- 9. Our Approach KISS YAGNI 9 Indexing Big Data in the Cloud (This space intentionally left blank)
- 10. Minimal Flair 10 Indexing Big Data in the Cloud
- 11. Record Everything! 11 Indexing Big Data in the Cloud
- 12. Some Numbers Doc Count 1.1 Million Zip Files 313 Docs per Zip File 4,000 Zip File Size 75M File Size 300M 12 Indexing Big Data in the Cloud
- 13. Testing Start some servers Process a batch Check the clock 13 Indexing Big Data in the Cloud
- 14. start_nodes start_nodes() { ec2-run-instances ami-1b814f72 --block-device-mapping '/dev/sdb=snap-48adde35::true' --block-device-mapping '/dev/sdi1=:10:false' --block-device-mapping '/dev/sdi2=:10:false' --block-device-mapping '/dev/sdi3=:20:false' --instance-type m1.large --key uspto-proto --instance-count $MAX_NODES --group default > ~/run-output } 14 Indexing Big Data in the Cloud
- 15. Gut Check How fast can we do this? What can we do in parallel? 15 Indexing Big Data in the Cloud
- 16. Scaling Raise our instance limit xargs -P GNU parallel 16 Indexing Big Data in the Cloud
- 17. Shortcomings SSH? Error recovery One Solr 17 Indexing Big Data in the Cloud
- 18. Alternatives CloudFormation Puppet / Chef Multiple Cores / Shards Hadoop 18 Indexing Big Data in the Cloud
- 19. Success 19 Indexing Big Data in the Cloud
- 20. Victory Lap 20 Indexing Big Data in the Cloud
- 21. Instances / Time Indexing Big Data in the Cloud 21
- 22. Thank You https://github.com/sstults/patent-indexing @scottstults #o19s 22 Indexing Big Data in the Cloud