In this presentation you will learn how Farm Credit Services of America/Frontier Farm Credit transformed their quality practices and tooling to bring visibility and consistency to Enterprise Quality, including: testing as a team approach, creating an automated test architecture, measuring progress with dashboards and standardizing on a set of testing tools.
3. About Farm Credit Services of America
• Serving farmers and
ranchers in rural America
• To be Agriculture’s Most
Valued Financial Partner
• ~$28 billion in loan volume
(2018)
• ~140 Applications
Development teammates
across 17 teams
• ~110 Technology Services
teammates
• Primarily a .NET shop with
JavaScript/Angular + Mobile
tech.
7. It is not the strongest of
species that survives, nor the
most intelligent, but the one
most responsive to change.
- One interpretation of Charles Darwin’s writings – “On the
Origin of Species”
12. Success Takes People!
• Quality Assurance role – not just for testing any more, also…
• System health analysis
• Visibility & understanding
• Peer group
• Collaboration
• Development
• Ability to understand and write code
13. Focus on Enterprise Quality
• Solutions/Enterprise Architect with focus on enterprise quality…
• …and health
• …and visibility
• …and metrics/KPIs
• …and consistency
15. Lean Quality Delivery (LQD)
• Vision
• To Be the Chosen Partner
• Goals
• Functional quality – is it doing what it’s supposed to do?
• Structural quality – is the code testable, secure, maintainable, etc.?
• Process quality – are the software development processes efficient and effective?
• Strategies
• Consistent across teams
• Tactics
• Can vary by team and project
• Teams revisit LQD/tactics at the beginning/end of projects
16. Lean Quality Delivery (LQD) – Cont’d
Lean Quality Delivery (LQD)
Vision Goal Strategy Tactics
Score
(1-5)
ToBetheChosenPartner
FunctionalQuality
1. Delivers the desired value
Story writing and review
UI automation testing
Performance testing (Does it meet SLA needs?)
Service-level testing
Unit testing
Discussion with development teams
2. Deliberate testing plan at the
story and project level
Testing starts with the creation of the story and review
Testing is a first-class citizen
Tests should answer questions
o What makes the story done?
o How is the user going to use it?
3. High reliability, minimal defects
Pairing/Code reviews
Health monitoring (Are we notified when something goes awry?)
4. Delightful user experience
User interviews (in our case, this is typically developers)
Process mapping
Prototyping
Follow-up discussions/surveys
StructuralQuality
1. Design for testability
Incorporate design patterns (as appropriate)
TDD/Code reviews
Continuous Integration
Continuous Delivery
2. Security best practices
Security training for developers
Static and dynamic code scanning (for public-/partner-facing
apps)
3. Follows project guidance
Application reviews
Reuse of enterprise components
Reference Tech Radar (and follow the process for new
technologies)
Create Project Start Architecture document
ProcessQuality
1. Responsive support
Proactive support (e.g. Do we send “ack” within 15 minutes?)
Someone takes ownership of issue and “walks it through”
2. Consistent Process
The team can describe the process and how it impacts quality
The team implements the agreed upon tactics
Reviewing and grading of the team this document
3. Predictable Delivery
Measure Cycle Time
Groom Backlog
Manage Budget
Manage Timeline
Automated deployments
4. Visibility & Transparency of
Count defects and kickbacks
Count production-related issues
System Health (for systems under our team’s control)
5. Routinely follows process for
feedback and learning
Retrospectives
Demos
5 Why's for production defects
22. Version Control
• Use version control as
the single source of
truth for Automation
Scripts.
Dev
• Azure DevOps
• Visual Studio
• VS Code
• GIT
• Common lib
• Project Repo
QA
• Azure DevOps
• Visual Studio
• VS Code
• GIT (SourceTree)
• QA lib
• QA Repo
Software Development/Quality Tools
23. Communication Dev
• Lead Dev Meeting
• AppDev Team Blog
• SharePoint
• Microsoft Teams
• Email Distribution Lists
• DOUG
QA
• QAD Meeting
• AppDev Team Blog
• Microsoft Teams
• QA Website
• QA SharePoint
• Email distribution lists
Communication as a Tool
24. Learning Dev
•On-site Training
•Dev Pairing
•Pluralsight
•Website/Blogs
•Tech Conferences
•DOUG
•.NET/Other User Groups
QA
•On-site Training
•QAD/Dev Pairing
•Pluralsight
•Website/Blogs
•Tech Conferences
•Book Club
•DOUG
Learning Opportunities
26. Jenkins
• Why Jenkins?
• Flexibility!
• Single Platform
• Highly configurable
• Automates the build and test (we use it for scheduling/running tests)
• Get source code from repository (Azure DevOps)
• Dashboard - everyone can see what’s happening
• Generate report and notify stakeholders of build status
• Plug-in extensibility: Unit test coverage, last success and failure, trends
28. Azure DevOps
• Why Azure DevOps?
• Integration with development, build & deployment (CI/CD) processes
• Familiar platform to developers
• Highly configurable
• Automates the build, test and deployments
• Get source code from repository (Azure DevOps)
• Dashboard - everyone can see what’s happening
• Plug-in extensibility: Unit test coverage, last success and failure, trends
31. QA VM Image
Target Install
• Telerik Runtime/ Chrome Plugin
• Postman(Node/Newman)
• jMeter
• NUnit
Common Software
• GIT
• Jenkins
• Beyond Compare
• Notepad++
Group Policies
• Auto-login with test account, Disable screen lock, UAC
• Browser settings, Screen resolution
Base Windows 10 Image
• With standard configuration
32. Summary
• The problem: No unified approach to quality
• Solution:
• Adopted shift-left approach to software development and quality
• Implemented the testing pyramid
• Increased collaboration and communication between QA and Devs
• Testing architecture using Jenkins and Azure DevOps
• In other words… consistency
• Impact:
• Friction leads to lower adoption rates
• We increased test automation from 400 tests suites per day 2,300
• Standardized on a common set of tools
• Increased collaboration across teams
• Increased confidence in delivering software
33. What’s next ?
• Continue to focus on enterprise health & visibility
• Further grow our shift-left practices
• Automate our test data scenarios
• Grow the quality community
• Deliver great, high-quality software and experiences… faster!
We can’t do this without the sponsors! Thanks, sponsors!
Let’s have a dialog!
We can all learn from each other.
While we are not to the same scale as the “Microsofts” or “Googles” of the world, we have been increasing our # of deployments dramatically year over year. As we move into more of a micro-/mini-service architecture mindset, it is expected that our number of deployments will increase more quickly than ever before.
$28.4 billion in loan volume in 2018 up from $22.1 billion in loan volume in 2014
No matter how much value you deliver, the number of requirements you implement, the amount of functionality you build into your products, your business users and customers will always want more. If it’s not new features in your existing products, it’s new products that you’ve yet to create.
Provide leadership and help bring focus and visibility
B2B Strategy: Basement living to Balcony
Quality has been an after thought, treat it as a first-class citizen
Image Source: https://xebia.com/blog/its-2017-test-automation-is-not-optional-when-building-mobile-apps/
Get QA involved up front – during the initial stages
Test earlier in the process
We shifted our Quality Assurance Engineer (QAE) role to a Quality Assurance Developer (QAD) role. This decision was made to help our transition to a shift-left mindset.
Provides a consistent mechanism for all software development teams to apply focus toward quality.
Another view of the same LQD from the previous page.
Are you collaborating with your QA(D)?
Are quality efforts discussed as part of your project planning?
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
Continuous Delivery (CD) is a software development discipline where you build software in such a way that the software can be released to production at any time - http://martinfowler.com/bliki/ContinuousDelivery.html
Continuous Deployment means that you are engaging in actual deployment.
Continuous Delivery is a great milestone! However, there is more to be achieved by taking the next step toward Continuous Deployment.
“Demo-ware”
“Demo-ware”
“Demo-ware”
There was no unified approach in our Quality Practices(tooling and process) across our 10 AppDev teams. We were good at Continuous Integration but inefficient at Continuous Delivery.