1. High Level System Overview
Crawler Preprocessor DBProcessor Store in DB

2. Infrastructure Requirements
• Component Independence
• Messaging
• Scalability
• Minimize code written, use as much open
source code as possible

3. Infrastructure Choices
• Jboss/Java vs .NET
• Spring Framework vs Plain-old Java
• Oracle vs MySql
• Hibernate ORM vs Plain-old SQL

4. Logging Structure
• Entering and exiting methods (of
reasonable importance)
• Catching Java checked exceptions
• Uniform structure
• org.fydproject.component.mainMethod.su
bMethod1.subMethod2…
• Ex. org.projectnlp.preprocessor.stemmer

5. Pseudo-Code for Crawler Manager
• Begin infinite loop
– For each messageBoard in List
• crawlAll
– End For loop
• End infinite loop

6. High-Level Crawler Strategy
• Failed messages are
persisted
• Message markers
(right-hand side
labels) are persisted
• Algorithm prevents
crawling duplicate
messages
Old Message
Threshold
Oldest Message
Crawled
Last Successful
crawl
Last Successful
message extracted
Newest Message
Newly Crawled
Messages
Old successful
Crawled
Messages
Old Messages
Yet to be
Crawled
Messages from
Crash
Highest Message
Id
Lowest Message
Id

7. Crawler Strategy Algorithm
• Crawl all previous failed messages
• Crawl ‘crashed messages’
• Crawl new messages
• Crawl new failed messages
• Crawl old messages

8. Preprocessor Block Diagram
Lowercase
HTML
Parser
Cleanup
HTML
Parser
Cleanup
Contractions
Dictionary
Slang
Dictionary
Punctuation
Cleaner
Stop Words
Dictionary
Negation
Engine
Stemmer
Out to DB Processor
In from
Crawler