OOPSLA 2011 talk about variability-aware parsing (i.e., parsing C code without preprocessing it first) Corresponding paper: http://www.informatik.uni-marburg.de/~kaestner/oopsla11_typechef.pdf

1. Variability-Aware Parsing
in the Presence of
Lexical Macros and Conditional Compilation
Christian Kästner
Paolo G. Giarrusso
Tillmann Rendel
Sebastian Erdweg
Klaus Ostermann
Thorsten Berger

2. Parsing C Code
without Preprocessing
Christian Kästner
Paolo G. Giarrusso
Tillmann Rendel
Sebastian Erdweg
Klaus Ostermann
Thorsten Berger

3. Viewer Discretion Is Advised

5. Linux
Kernel

6. 10,000 features, 6 million lines of C code

7. Parse and Type check
all configurations of the
entire Linux kernel
10,000 features, 6 million lines of C code

8. greet.c
printf … VWORLD VBYE main
AST with msg ε msg ε printf
Variability Information ¬ (WORLD ˄BYE)
msg
true -> (WORLD v BYE)

9. Variability-Aware Analysis
Type System
Static Analysis
Bug Finding
Testing
Model Checking
Theorem Proving
…

10. Parsing C without Preprocessing

11. Macro expansion Undisciplined
needed for parsing annotations Alternative macros
?
? ?
greet.c
+ printf VWORLD VBYE + main
+ msg ε + msg ε printf
msg

14. Previous Solutions
Disciplined Subset
Requires Code Preparation
Heuristics and Partial Analysis
Inaccurate, False Positives
Brute Force
Infeasible Effort

15. https://github.com/ckaestne/TypeChef
TypeChef
Variability-Aware
Parser-Framework
( +
2
Variability-Aware * Variability-Aware
3 * VA
Lexer )
+ Parser
4A
5¬A 2 3 4 5
Variability-Aware
Analysis

16. Variability-Aware Lexer
(
2
*

17. Variability-Aware Lexer

18. 353 included header files
0 per C file
8590 macros
0 per C file
1387 conditional macros
0 per C file
335490 token
0 per C file
72 % conditional
0 2.6.33.3
X86

19. Variability-Aware Parsing
( 2 * 3 ) + 4A 5¬A
+
* VA
2 3 4 5

20. Undisciplined Annotations
2 * (A 3 + 4 )A VA
* +
2 + * 4
3 4 2 3

21. Partial Errors
2 * (A 3 + 4 ) VA
* Parse Error
Expected + or EOF,
but found ) at 1:7
2 +
3 4

22. true
( 3 + 4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )

23. true
( 3 + 4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )

24. 3
true
( 3 + 4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )

25. 3
true
( 3 + 4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )

26. A
3 4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

27. 4 A
3 4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

28. 4 A
3 4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
( 3 +
Next Token:
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
ctx -> tok: Consume
¬A ctx ->¬tok: Skip
else: Split (ctx ˄pc) andctx ˄ ¬pc)
(

29. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

30. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

31. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

32. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

33. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

34. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4A (¬A 4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A

35. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 + ¬A˄B
4¬A˄ +¬A˄ 6¬A
B B )¬A )
4A (¬A
4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A˄¬B

36. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 + 4 ¬A˄B
4¬A˄ +¬A˄ 6¬A
B B )¬A )
4A (¬A
4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A˄¬B

37. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 + 4 ¬A˄B
4¬A˄ +¬A˄ 6¬A
B B )¬A )
4A (¬A
4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A˄¬B

38. 4 A
3 4A (¬A + 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 + 4 6 ¬A˄B
4¬A˄ +¬A˄ 6¬A
B B )¬A )
4A (¬A
4¬A˄ +¬A˄ 6¬A
B B )¬A )
¬A˄¬B

39. 4 A
3 4A (¬A + 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 + 4 6 ¬A˄B
4¬A˄ +¬A˄ 6¬A
B B )¬A )
4A (¬A
4¬A˄B +¬A˄ 6¬A
B )¬A )
¬A˄¬B

40. 4 A
3 4A (¬A + 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 + 4 6 ¬A˄B
4¬A˄ +¬A˄ 6¬A
B B )¬A )
4A (¬A
4¬A˄B +¬A˄ 6¬A
B )¬A )
¬A˄¬B

41. 4 A
3 4A (¬A + 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 + 4 6 ¬A˄B
4¬A˄ +¬A˄ 6¬A
B B )¬A )
4A (¬A
4¬A˄B +¬A˄ 6¬A
B )¬A )
6 ¬A˄¬B

42. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4¬A˄ +¬A˄ 6¬A
B B
4A (¬A )¬A )
4¬A˄B +¬A˄ 6¬A
B
VB ¬A
+ 6
4 6

43. 4 A
3 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A )
( 3 +
4¬A˄ +¬A˄ 6¬A
B B
4A (¬A )¬A )
4¬A˄B +¬A˄ 6¬A
B
¬A
VB
+ 6
4 6

44. 3 4A (¬A 4¬A˄B +¬A˄ 6¬A )¬A
B true
( 3 + )
4¬A˄ +¬A˄ 6¬A
B B
4A (¬A )¬A
VA 4¬A˄B +¬A˄ 6¬A
B
4 VB
+ 6
4 6

45. 4A (¬A 4¬A˄B +¬A˄ 6¬A
B )¬A true
( 3 + )
4¬A˄ +¬A˄ 6¬A
B B
+ 4A (¬A )¬A
3 VA 4¬A˄B +¬A˄ 6¬A
B
4 VB
+ 6
4 6

46. Variability-Aware
Parser Combinator Library
for Scala

47. Variability-Aware Parsers
GNU C + cpp Java + Antenna

48. Parsing Linux
Variability in Build System (kbuild)
and with #ifdef
Variability Model (kconfig) 2.6.33.3
X86

49. 7665 C files (x86)
0
30 seconds per file
0 (median)
85 hours total
0
4.1 % overhead
0 (undiscipl.)
0
0 syntax errors 2.6.33.3
X86

50. Parser
Variability-Aware Type System
Module System
Static Analysis
Bug Finding
Testing
Model Checking
Theorem Proving
Editor Support
Code Transform.
…

51. Variability-Aware Parsing
in the Presence of
Lexical Macros and Conditional Compilation
Variability-Aware
Parser-Framework
( +
2
Variability-Aware * Variability-Aware
3 * VA
Lexer )
+ Parser
4A
(macros, includes)
5¬A (split & join) 2 3 4 5
https://github.com/ckaestne/TypeChef