Feedback about 5 years of Foreman Experience to manage different kinds of infrastructure. A story about Open Source. Given for the 7th Birthday of The Foreman.
1. Managing a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with ForemanManaging a R&D Lab with Foreman
Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto
Foreman 7th Birthday Party
Inuits, Antwerp
July 13th, 2016
2. whoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoamiwhoami
Julien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien PivottoJulien Pivotto
• Sysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.euSysadmin at inuits.eu
• FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004FLOSS user since 2004
• Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011Foreman user since 2011
• DevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believerDevOps believer
• @roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie@roidelapluie on irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/githubon irc/twitter/github
9. Foreman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choiceForeman: A good choice
• OOOOOOOOOOOOOOOOOpen-Source
• LLLLLLLLLLLLLLLLLarge, active community
• RRRRRRRRRRRRRRRRRest API and cli tools
10.
11.
12.
13.
14.
15. Behind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenesBehind the scenes
Licensed under a Creative Commons Attribution 2.0 License
https://www.flickr.com/photos/reuver/10105949326
24. What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?What can be hard?
Licensed under a Creative Commons Attribution 2.0 License
https://www.flickr.com/photos/130811041@N04/19114856463
27. How to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The ForemanHow to install The Foreman
• 11111111111111111. yum install foreman-installer
• 22222222222222222. run foreman-installer
• 33333333333333333. done
31. State is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforcedState is not enforced
• SSSSSSSSSSSSSSSSSystem is not up to date
• NNNNNNNNNNNNNNNNNo confidence that the state is still correct
• SSSSSSSSSSSSSSSSSolution A: integrate within the puppet tree
• SSSSSSSSSSSSSSSSSolution B: Re-run the foreman-installer
32. Importing the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your treeImporting the modules in your tree
• SSSSSSSSSSSSSSSSState is enforced
• OOOOOOOOOOOOOOOOOnly thing to care about: updating the
modules
• TTTTTTTTTTTTTTTTThey are linked to the foreman
33. Building and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuildingBuilding and rebuilding
Licensed under a Creative Commons Attribution 2.0 License
https://www.flickr.com/photos/artbystevejohnson/6405400351
34. Building a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a hostBuilding a host
• CCCCCCCCCCCCCCCCCreate/cycle VM
• CCCCCCCCCCCCCCCCCreate/change DNS Config
• CCCCCCCCCCCCCCCCCreate/change DHCP lease
• CCCCCCCCCCCCCCCCCreate/change TFTP files
35. An expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operationAn expensive operation
• IIIIIIIIIIIIIIIIIf something is wrong, it rollbacks
• SSSSSSSSSSSSSSSSSolution: Foreman 1.10 allows you to force
config rebuild
• HHHHHHHHHHHHHHHHHidden in the UI in the `All hosts' view
38. Foreman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entriesForeman managing DNS entries
• KKKKKKKKKKKKKKKKKeeps the data consistent
• CCCCCCCCCCCCCCCCCreates only hostnames that exist
• AAAAAAAAAAAAAAAAAlso does the reverse entries
39. Adding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom recordsAdding custom records
• AAAAAAAAAAAAAAAAAt some point you want CNAME records
• ooooooooooooooooor change NS records
40. Manipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zonesManipulating zones
• FFFFFFFFFFFFFFFFForeman uses Dynamic Zones
• rrrrrrrrrrrrrrrrrndc freeze
• CCCCCCCCCCCCCCCCChange the zone (incr the serial)
• rrrrrrrrrrrrrrrrrndc thaw
41. Org-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNSOrg-level reverse DNS
• YYYYYYYYYYYYYYYYYou own DNS servers should forward the
unknown to your org
• WWWWWWWWWWWWWWWWWe could not make it work for PTR records
• NNNNNNNNNNNNNNNNNeed to set empty-zones-enable=no
• PPPPPPPPPPPPPPPPPR theforeman/puppet-dns#47
43. Managing hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the labManaging hosts outside the lab
• SSSSSSSSSSSSSSSSSecurity purpose
• RRRRRRRRRRRRRRRRRun Demos
• HHHHHHHHHHHHHHHHHandover to other teams
44. Foreman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutionsForeman-proxies solutions
• DDDDDDDDDDDDDDDDDNS Proxy, DHCP proxy, TFTP…
• KKKKKKKKKKKKKKKKKickstart proxying?
• DDDDDDDDDDDDDDDDDigging into the documentation
• FFFFFFFFFFFFFFFFFeature is there but not really visible
• IIIIIIIIIIIIIIIIImprove docs: PR
theforeman/theforeman.org#547
58. What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?What's wrong then?
• LLLLLLLLLLLLLLLLLibvirt servers are not a group
• TTTTTTTTTTTTTTTTThey are separated Compute Resources
• AAAAAAAAAAAAAAAAA lot of work (UI and API)
59. Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?Is it libvirt specific?
• YYYYYYYYYYYYYYYYYes: other providers are `centralized'
• eeeeeeeeeeeeeeeeec2, gce, openstack…
61. VMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integrationVMWare integration
• MMMMMMMMMMMMMMMMMost of the settings, Foreman will take user
defaults
• FFFFFFFFFFFFFFFFForeman is not for VMWare Power users
• SSSSSSSSSSSSSSSSSounds like a lack of interest from
community
62. Logging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging mattersLogging matters
• FFFFFFFFFFFFFFFFForeman requires on Fog
• FFFFFFFFFFFFFFFFFog is a gem for the `clouds'
• FFFFFFFFFFFFFFFFFog for vmware is not as advances as we'd
like
63. Empowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the DevelopersEmpowering the Developers
Licensed under a Creative Commons Attribution 2.0 License
https://www.flickr.com/photos/julesdphotographie/8421289746
64. Distributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the powerDistributing the power
• RRRRRRRRRRRRRRRRRebuilding a host is simple
• OOOOOOOOOOOOOOOOOne clic operation
• FFFFFFFFFFFFFFFFForeman as a VM shop
65. The DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movementThe DevOps movement
• DDDDDDDDDDDDDDDDDevOps is a movement born in 2009
• CCCCCCCCCCCCCCCCCollaboration between Developers and
Operations
• NNNNNNNNNNNNNNNNNothing new, just common sense
• DDDDDDDDDDDDDDDDDevOpsDays, a serie of conferences all
around the world
67. The C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMSThe C of CAMS
• DDDDDDDDDDDDDDDDDevOps is a Cultural change
• EEEEEEEEEEEEEEEEEveryone is in the team
• OOOOOOOOOOOOOOOOOps and Devs work together
• SSSSSSSSSSSSSSSSShare the responsibilities
68. Foreman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the pictureForeman in the picture
• FFFFFFFFFFFFFFFFForeman empowers the developers
• WWWWWWWWWWWWWWWWWhile still providing enough security
• OOOOOOOOOOOOOOOOOrganisations in Foreman
73. Foreman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updatesForeman updates
• AAAAAAAAAAAAAAAAAs any software:
• TTTTTTTTTTTTTTTTTake a backup first
• TTTTTTTTTTTTTTTTTry on your dev environment
• FFFFFFFFFFFFFFFFFirst upgrade the main UI
• TTTTTTTTTTTTTTTTThen update the proxies (#12506)
74. Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems
• 11111111111111111.5->1.6: DHCP config file not readable by
foreman proxy
WWWWWWWWWWWWWWWWWe did a DHCP upgrade at the same time
FFFFFFFFFFFFFFFFFile ownership was changed
rrrrrrrrrrrrrrrrre-run the foreman installer fixed it
75. Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems
• 11111111111111111.5->1.6: Puppet reports not coming into the
foreman
FFFFFFFFFFFFFFFFForeman report preprocessor has changed
NNNNNNNNNNNNNNNNNeed to update foreman.rb + config
nnnnnnnnnnnnnnnnnode.rb also needed an update
78. Upgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problemsUpgrade problems
• 11111111111111111.11: DHCP bugs
TTTTTTTTTTTTTTTTTwo DHCP bugs
11111111111111111. Do not create DHCP reservation is host is static
22222222222222222. Ruby 1.8 issues (EL6 support to be removed in 1.13)
hhhhhhhhhhhhhhhhhttps://theforeman.org/2016/06/foreman-1.11-dhcp-
bugs.html
79. Community and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-SourceCommunity and Open-Source
Licensed under a Creative Commons Attribution-2.0 License
https://www.flickr.com/photos/maicosnap/4393929855
80. Foreman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-SourceForeman is Open-Source
• GGGGGGGGGGGGGGGGGPLv3+
• WWWWWWWWWWWWWWWWWe've got around 20 patches integrated
• TTTTTTTTTTTTTTTTThey rely mostly on open-source tools
• CCCCCCCCCCCCCCCCCI, testing, bugtracker
81. Story: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global StatusStory: Foreman Global Status
• PPPPPPPPPPPPPPPPPut in build mode, go back home
• RRRRRRRRRRRRRRRRRestart on the next day
• LLLLLLLLLLLLLLLLLooping reboots
82. Foreman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global statusForeman global status
• FFFFFFFFFFFFFFFFForeman keeps several status for hosts
• PPPPPPPPPPPPPPPPPlugins can register a status
• BBBBBBBBBBBBBBBBBut reporting and provisioning also
• LLLLLLLLLLLLLLLLLet's use that for expired tokens (bug#5883)
84. Advantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreamingAdvantages of upstreaming
• FFFFFFFFFFFFFFFFFeature will be available in next releases
• WWWWWWWWWWWWWWWWWe won't patch in the future
• CCCCCCCCCCCCCCCCCalls in the patch will evolve as well
• PPPPPPPPPPPPPPPPPeer review with the Foreman team
• FFFFFFFFFFFFFFFFForced to write tests, doc, etc…
85. 2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after2 weeks after
• PPPPPPPPPPPPPPPPPatch is accepted for 2 weeks
• AAAAAAAAAAAAAAAAA new bug is opened: performance problem
(bug#14050)
• SSSSSSSSSSSSSSSSSomeone else fixed the bug
• WWWWWWWWWWWWWWWWWe applied the second patch
87. Where can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improveWhere can the Foreman improve
• PPPPPPPPPPPPPPPPPerformance
• SSSSSSSSSSSSSSSSSome features are missing from API
• DDDDDDDDDDDDDDDDDecoupling from Puppet
• ……………………………………………
88. Where did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improveWhere did the Foreman improve
• TTTTTTTTTTTTTTTTTests reliability
• TTTTTTTTTTTTTTTTTesting Infra
• CCCCCCCCCCCCCCCCCommunity
• SSSSSSSSSSSSSSSSStability
89. The ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe ForemanThe Foreman
• SSSSSSSSSSSSSSSSStability, Maturity, Flexibility
• MMMMMMMMMMMMMMMMMade out of bricks: complex but not
complicated
• IIIIIIIIIIIIIIIIIt meets our needs
• IIIIIIIIIIIIIIIIIt is very active
• PPPPPPPPPPPPPPPPPeople are welcoming