SlideShare a Scribd company logo
1 of 51
Download to read offline
Pyston tech talk
November 10, 2015
What is Pyston
High-performance Python JIT, written in C++
JIT: produces assembly “just in time” in order to accelerate the program
Targets Python 2.7
Open source project at Dropbox, started in 2013
Two full time members, plus part time and open source members
The team
Marius Wachtler
Kevin Modzelewski
Lots of important contributors:
Boxiang Sun, Rudi Chen, Travis Hance,
Michael Arntzenius, Vinzenz Feenstra, Daniel Agar
Pyston current status
25% better performance than CPython
Compatibility level is roughly the same as between minor versions (2.6 vs 2.7)
- Can run django, much of the Dropbox server, some numpy
Next milestone is Dropbox production!
Talk Outline
Pyston motivation
Compatibility
Python performance
Our techniques
Current roadmap
Pyston motivation
Why Pyston
Python is not just “IO-bound”; at scale, Dropbox (and others) have many cores
running Python
Many existing Python-performance projects, but not suitable for large Python
codebases
Existing Landscape
Baseline: CPython
If you want more performance:
- C extension
- Cython
- Numba
- PyPy
- Rewrite (Go? C++?)
How we fit in
Focus on large web-app case (specifically Dropbox):
- Require very low required edits per kLOC
- Implies good C API support
- Good performance scalability to large codebases
Non-goal: crushing microbenchmarks
Compatibility
Compatibility challenges
Some things expected:
- Language documentation but no formal spec
- C API challenges
- Every feature exists because someone wanted it
Compatibility challenges
Some not expected:
- Lots of program introspection
- Some core libraries (pip) are the most dynamic
- Code will break if you fix even the worst warts
- Community accepts other implementations, but assumes
is_cpython = not is_pypy
Our evolution
Started as a from-scratch implementation, is now CPython-based.
Got to experiment with many things:
- showed us several things we can change
- and several things we cannot :(
Evolution result
We use lots of CPython code to be “correct by default”
We support:
- django, sqlalchemy, lxml, many more
- most of the Dropbox server
- some numpy
Aside: the GIL
I don’t want it either but… it’s not just an implementation challenge.
- Removing it is a much bigger compatibility break than we can accept
We have a GIL. And Dropbox has already solved its Python parallelism issue anyway.
Maybe Python 4?
Python performance
What makes Python hard
Beating an interpreter sounds easy (lots of research papers do it!), but:
CPython is well-optimized, and code is optimized to run on it
Hard to gracefully degrade to CPython’s behavior
What makes Python hard
Python doesn’t have static types
But…
What makes Python hard
Python doesn’t have static types
But…
Statically typed Python is still hard!
What makes Python hard
Statically-typed Python is still hard
var_name = var_parser_regex.match(s)
setting = getattr(settings, var_name, None)
What makes Python hard
Statically-typed Python is still hard
Knowing the types does not make getattr() easy to evaluate
var_name = var_parser_regex.match(s)
setting = getattr(settings, var_name, None)
What makes Python hard
Statically-typed Python is still hard
Knowing the types does not make getattr() easy to evaluate
Many other examples:
- len()
- constructors
- binops
var_name = var_parser_regex.match(s)
setting = getattr(settings, var_name, None)
What makes Python hard
- Types are only the first level of dynamicism
- Functions themselves exhibit dynamic behavior
- Traditional “interpreter overhead” is negligible
So what can we get from a JIT?
What makes Python hard
- Types are only the first level of dynamicism
- Functions themselves exhibit dynamic behavior
- Traditional “interpreter overhead” is negligible
So what can we get from a JIT?
- We need to understand + avoid the dynamicism in the runtime
Our techniques
Pyston architecture
Parser Bytecode Interpreter
Baseline
JIT
LLVM JIT
Runtime
Tracer
Our workhorse: tracing
Very low tech tracing JIT:
- single operation (bytecode) at a time
- no inlining
- manual annotations in the runtime
Our workhorse: tracing
Manual annotations
- are difficult to write
+ require less engineering investment
+ are very flexible
+ have very high performance potential
Tracing example def foo(x):
pass
foo(1)
Tracing example
1.Verify the function is the same
2.Call it
def foo(x):
pass
foo(1)
Tracing example
1.Verify the function is the same
a.Check if “foo” still refers to the same object
b.Check if foo() was mutated
2.Call it
a.Arrange arguments for C-style function call
b.Call the underlying function pointer
def foo(x):
pass
foo(1)
Tracing example
1.Verify the function is the same
a.Check if “foo” still refers to the same object
b.Check if foo() was mutated
2.Call it
a.Arrange arguments for C-style function call
b.Call the underlying function pointer
def foo(x):
pass
foo(1)
Can skip hash table lookup
Rare, use invalidation
Can skip *args allocation
Tracing example #2 o = MyCoolObject()
len(o)
Tracing example #2
1.Verify the function is the same
a.Check if “len” refers to the same object
2.Call it
a.len() supports tracing
o = MyCoolObject()
len(o)
Tracing example #2
1.Verify the function is the same
a.Check if “len” refers to the same object
2.Call it
a.len() supports tracing. Decides to:
i.Call arg.__len__()
o = MyCoolObject()
len(o)
Tracing example #2
1.Verify the function is the same
a.Check if “len” refers to the same object
2.Call it
a.len() supports tracing. Decides to:
i.Call arg.__len__()
1.Verify the function is the same
2.Call it
o = MyCoolObject()
len(o)
Tracing example #2
1.Verify the function is the same
a.Check if “len” refers to the same object
2.Call it
a.len() supports tracing. Decides to:
i.Call arg.__len__()
1.Verify the function is the same
...
2.Call it
o = MyCoolObject()
len(o)
Why use tracing
We started with a traditional method-at-a-time JIT, but quickly ran into issues, and our
tracing system kept being the best way to solve them.
- We need a rich way of representing the expected path through the runtime
- We want to let C functions specify alternate versions of themselves that are either
more specialized or more general
- We want to keep the tracing code close to the runtime code it needs to match
PyPy comparison
PyPy
Missing:
- C extension support (80k LOC used at Dropbox)
- performance scalability and consistency
We’ve been measuring our catch-up in “years per month”
PyPy performance scalability
Their performance degrades quite a lot when run on large “real” (non-numeric)
applications, and often ends up slower than CPython
- Initial testing of PyPy at Dropbox shows no clear improvement
One indicator: average benchmark size.
- PyPy: 36 lines
- Pyston: 671 lines
PyPy performance scalability
Simple attribute-lookup example:
PyPy performance scalability
Simple attribute-lookup example:
PyPy performance scalability
Simple attribute-lookup example:
PyPy performance scalability
Simple attribute-lookup example:
8x faster!
PyPy performance scalability
Simple attribute-lookup example:
38x slower :(
8x faster!
Current roadmap
Current roadmap
Focusing on getting ready for Dropbox’s production use. Last “1%” features
- Inspecting exited frames
- Signals support
- Refcounting?
Current roadmap
Continue performance work
- Integrate tracing and LLVM JITs
- Optimized bytecode interpreter
- Function inlining
How to get involved
Just pick something! We have a good list of starter projects
Or just hop on our gitter channel and say hi
Questions?
kmod@dropbox.com
marius@dropbox.com
https://github.com/dropbox/pyston
https://gitter.im/dropbox/pyston
We’re hiring!

More Related Content

Recently uploaded

Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기Chiwon Song
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionsNirav Modi
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9Jürgen Gutsch
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?AmeliaSmith90
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Neo4j
 

Recently uploaded (20)

Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기20240330_고급진 코드를 위한 exception 다루기
20240330_고급진 코드를 위한 exception 다루기
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
eAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspectionseAuditor Audits & Inspections - conduct field inspections
eAuditor Audits & Inspections - conduct field inspections
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9About .NET 8 and a first glimpse into .NET9
About .NET 8 and a first glimpse into .NET9
 
How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?How Does the Epitome of Spyware Differ from Other Malicious Software?
How Does the Epitome of Spyware Differ from Other Malicious Software?
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Sustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire ThornewillSustainable Web Design - Claire Thornewill
Sustainable Web Design - Claire Thornewill
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!Webinar - IA generativa e grafi Neo4j: RAG time!
Webinar - IA generativa e grafi Neo4j: RAG time!
 

Pyston talk 11-10-15

  • 2. What is Pyston High-performance Python JIT, written in C++ JIT: produces assembly “just in time” in order to accelerate the program Targets Python 2.7 Open source project at Dropbox, started in 2013 Two full time members, plus part time and open source members
  • 3. The team Marius Wachtler Kevin Modzelewski Lots of important contributors: Boxiang Sun, Rudi Chen, Travis Hance, Michael Arntzenius, Vinzenz Feenstra, Daniel Agar
  • 4. Pyston current status 25% better performance than CPython Compatibility level is roughly the same as between minor versions (2.6 vs 2.7) - Can run django, much of the Dropbox server, some numpy Next milestone is Dropbox production!
  • 5. Talk Outline Pyston motivation Compatibility Python performance Our techniques Current roadmap
  • 7. Why Pyston Python is not just “IO-bound”; at scale, Dropbox (and others) have many cores running Python Many existing Python-performance projects, but not suitable for large Python codebases
  • 8. Existing Landscape Baseline: CPython If you want more performance: - C extension - Cython - Numba - PyPy - Rewrite (Go? C++?)
  • 9. How we fit in Focus on large web-app case (specifically Dropbox): - Require very low required edits per kLOC - Implies good C API support - Good performance scalability to large codebases Non-goal: crushing microbenchmarks
  • 11. Compatibility challenges Some things expected: - Language documentation but no formal spec - C API challenges - Every feature exists because someone wanted it
  • 12. Compatibility challenges Some not expected: - Lots of program introspection - Some core libraries (pip) are the most dynamic - Code will break if you fix even the worst warts - Community accepts other implementations, but assumes is_cpython = not is_pypy
  • 13. Our evolution Started as a from-scratch implementation, is now CPython-based. Got to experiment with many things: - showed us several things we can change - and several things we cannot :(
  • 14. Evolution result We use lots of CPython code to be “correct by default” We support: - django, sqlalchemy, lxml, many more - most of the Dropbox server - some numpy
  • 15. Aside: the GIL I don’t want it either but… it’s not just an implementation challenge. - Removing it is a much bigger compatibility break than we can accept We have a GIL. And Dropbox has already solved its Python parallelism issue anyway. Maybe Python 4?
  • 17. What makes Python hard Beating an interpreter sounds easy (lots of research papers do it!), but: CPython is well-optimized, and code is optimized to run on it Hard to gracefully degrade to CPython’s behavior
  • 18. What makes Python hard Python doesn’t have static types But…
  • 19. What makes Python hard Python doesn’t have static types But… Statically typed Python is still hard!
  • 20. What makes Python hard Statically-typed Python is still hard var_name = var_parser_regex.match(s) setting = getattr(settings, var_name, None)
  • 21. What makes Python hard Statically-typed Python is still hard Knowing the types does not make getattr() easy to evaluate var_name = var_parser_regex.match(s) setting = getattr(settings, var_name, None)
  • 22. What makes Python hard Statically-typed Python is still hard Knowing the types does not make getattr() easy to evaluate Many other examples: - len() - constructors - binops var_name = var_parser_regex.match(s) setting = getattr(settings, var_name, None)
  • 23. What makes Python hard - Types are only the first level of dynamicism - Functions themselves exhibit dynamic behavior - Traditional “interpreter overhead” is negligible So what can we get from a JIT?
  • 24. What makes Python hard - Types are only the first level of dynamicism - Functions themselves exhibit dynamic behavior - Traditional “interpreter overhead” is negligible So what can we get from a JIT? - We need to understand + avoid the dynamicism in the runtime
  • 26. Pyston architecture Parser Bytecode Interpreter Baseline JIT LLVM JIT Runtime Tracer
  • 27. Our workhorse: tracing Very low tech tracing JIT: - single operation (bytecode) at a time - no inlining - manual annotations in the runtime
  • 28. Our workhorse: tracing Manual annotations - are difficult to write + require less engineering investment + are very flexible + have very high performance potential
  • 29. Tracing example def foo(x): pass foo(1)
  • 30. Tracing example 1.Verify the function is the same 2.Call it def foo(x): pass foo(1)
  • 31. Tracing example 1.Verify the function is the same a.Check if “foo” still refers to the same object b.Check if foo() was mutated 2.Call it a.Arrange arguments for C-style function call b.Call the underlying function pointer def foo(x): pass foo(1)
  • 32. Tracing example 1.Verify the function is the same a.Check if “foo” still refers to the same object b.Check if foo() was mutated 2.Call it a.Arrange arguments for C-style function call b.Call the underlying function pointer def foo(x): pass foo(1) Can skip hash table lookup Rare, use invalidation Can skip *args allocation
  • 33. Tracing example #2 o = MyCoolObject() len(o)
  • 34. Tracing example #2 1.Verify the function is the same a.Check if “len” refers to the same object 2.Call it a.len() supports tracing o = MyCoolObject() len(o)
  • 35. Tracing example #2 1.Verify the function is the same a.Check if “len” refers to the same object 2.Call it a.len() supports tracing. Decides to: i.Call arg.__len__() o = MyCoolObject() len(o)
  • 36. Tracing example #2 1.Verify the function is the same a.Check if “len” refers to the same object 2.Call it a.len() supports tracing. Decides to: i.Call arg.__len__() 1.Verify the function is the same 2.Call it o = MyCoolObject() len(o)
  • 37. Tracing example #2 1.Verify the function is the same a.Check if “len” refers to the same object 2.Call it a.len() supports tracing. Decides to: i.Call arg.__len__() 1.Verify the function is the same ... 2.Call it o = MyCoolObject() len(o)
  • 38. Why use tracing We started with a traditional method-at-a-time JIT, but quickly ran into issues, and our tracing system kept being the best way to solve them. - We need a rich way of representing the expected path through the runtime - We want to let C functions specify alternate versions of themselves that are either more specialized or more general - We want to keep the tracing code close to the runtime code it needs to match
  • 40. PyPy Missing: - C extension support (80k LOC used at Dropbox) - performance scalability and consistency We’ve been measuring our catch-up in “years per month”
  • 41. PyPy performance scalability Their performance degrades quite a lot when run on large “real” (non-numeric) applications, and often ends up slower than CPython - Initial testing of PyPy at Dropbox shows no clear improvement One indicator: average benchmark size. - PyPy: 36 lines - Pyston: 671 lines
  • 42. PyPy performance scalability Simple attribute-lookup example:
  • 43. PyPy performance scalability Simple attribute-lookup example:
  • 44. PyPy performance scalability Simple attribute-lookup example:
  • 45. PyPy performance scalability Simple attribute-lookup example: 8x faster!
  • 46. PyPy performance scalability Simple attribute-lookup example: 38x slower :( 8x faster!
  • 48. Current roadmap Focusing on getting ready for Dropbox’s production use. Last “1%” features - Inspecting exited frames - Signals support - Refcounting?
  • 49. Current roadmap Continue performance work - Integrate tracing and LLVM JITs - Optimized bytecode interpreter - Function inlining
  • 50. How to get involved Just pick something! We have a good list of starter projects Or just hop on our gitter channel and say hi

Editor's Notes

  1. Other companies that use Python: YouTube, Pinterest, Reddit. (Yelp, Venmo, Digg)
  2. sys._getframe, skip type readying; exceptions, GC
  3. (negligible on web-app workloads) ceval.c is 5k LOC out of 40k in Python/. Cython is about the same speed as CPython when used without annotations.
  4. (negligible on web-app workloads) ceval.c is 5k LOC out of 40k in Python/. Cython is about the same speed as CPython when used without annotations.
  5. 80kloc figure was from March ‘15 what do I mean by consistency? they tend to have worse perf on real large programs.
  6. There are other dimensions as well. another one I tried is number of different control flow paths. 512 seems like a lot, but studies on open source projects found up to 300-some types, and the Dropbox codebase is much larger. (think: ORM code which will process #types=#prod tables)