8. ETL Project by Maximilian Butterer

1. DMDW - ETL-PROJECT
THE GROOVY-WAY
BY MAXIMILIAN BUTTERER
Samstag, 14. Mai 2011

2. WHAT WAS THE JOB
EXTRACT THE DATA FROM ROOM-PLAN-FILE (EXCEL)
TRANSFORM IT INTO NEW STRUCTURE (DATATYPES)
LOAD IT INTO NEW TARGET (E.G. DATABASE-SYSTEM)
CREATE KIND OF DOCUMENTATION (HOW-TO) OR
PRESENT IT TO YOU
Samstag, 14. Mai 2011

3. #1: EXPORT THE DATA
EXPORTING DATA FROM EXCEL IS PRETTY EASY
SO I EXPORTED HOLE FILE AS CSV-FILE
1ST PROBLEM: COMMA IS SEMICOLON
2ND PROBLEM: ENCODING IS NOT UTF-8
Samstag, 14. Mai 2011

4. #2: A HELPER
BECAUSE CSV IS PLAINTEXT IT‘S EASY TO PARSE
I CREATED A GROOVY-SCRIPT FOR CONVERTING
MY SOLUTION IS NOT THE CLEANEST WAY BUT SOME
KIND THE EASIEST
3RD PROBLEM: DATA IS NOT CONSISTENT / THERE ARE
CORRUPTED DATA-SETS
Samstag, 14. Mai 2011

5. #3: PUT IT INTO
HAVING ALL THE OBJECTS PARSED AND CONVERTED
JUST PUT ALL THE STUFF INTO THE DATABASE
4TH PROBLEM: HOW?
Samstag, 14. Mai 2011

6. HOW I SOLVED THE PROBLEMS
PARSING A FILE WITH GROOVY IS EASY AS 1,2,3
EACH LINE I HAD TO SPLIT UP BECAUSE OF THE
SEMICOLONS
TO CONVERT THE DATE-STRING INTO A REAL DATE WE
HAVE TO TRICK
Samstag, 14. Mai 2011

7. LET‘S HAVE A LOOK BEHIND
LOOK INTO THE CODE
HOW DOES THE CODE WORK
QUESTIONS
Samstag, 14. Mai 2011