DOES16 San Francisco - Susanna Brown & Ben Chan - DevOps in the Midst of an Airline Merger
•Download as PPTX, PDF•
0 likes•709 views
DevOps in the Midst of an Airline Merger Susanna Brown, Managing Director Operations Technology, American Airlines Ben Chan, Director Shared Services, American Airlines Description: DevOps as a cultural change agent to bring enterprise/federated, infrastructure/development, employees/vendors together, while merging two major airlines. DevOps as a cultural change agent to bring enterprise/federated, infrastructure/development, employees/vendors together, while merging two major airlines. DevOps Enterprise Summit San Francisco 2016
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to DOES16 San Francisco - Susanna Brown & Ben Chan - DevOps in the Midst of an Airline Merger
Similar to DOES16 San Francisco - Susanna Brown & Ben Chan - DevOps in the Midst of an Airline Merger (20)
More from Gene Kim
More from Gene Kim (18)
Recently uploaded
Recently uploaded (20)
DOES16 San Francisco - Susanna Brown & Ben Chan - DevOps in the Midst of an Airline Merger
- 1. DevOps in the Midst of Merger Susanna Brown, Managing Director – Operations Technology Ben Chan, Director – Operations Technology
- 2. Integration: no small feat Set Date with Header & FooterFooter Changed Under Insert>Header & Footer 660 712AA Systems US Systems 2
- 3. slow delivery different cultures Everyone doing things differently various apps for same function steep learning curves disconnected teams LOTS of heavy lifting 3
- 4. Integration: no small feat 105,000 40,000 front-end mobile devices computers need to be integrated 2,200 kiosks 4
- 5. 2,007 projects } Integration: no small feat 5 1,224 complete!
- 6. 6 “US Airways Makes Smooth Transition to American Reservation System” “Are you booked on American this weekend? 5 things to watch” “American Airlines passing first big merger test – so far” “US Airways’ last flight takes over social media” “Computer merger of US Airways and American Airlines completed” “Final US Airways Flight makes stop in Charlotte”“No glitches as American Airlines computer switch appears successful” “One last flight under US Airways logo” “American Airlines’ Reservations Test: Smooth Travel This Weekend” “American Airlines Flies Through Cutover as Reservations System Shines” 1Q 2015 Single Loyalty Pgm 2Q 2015 Single Operating Cert 3Q 2015 Co-located IOC’s 4Q 2015 Single PSS
- 7. 7 It’s all about Integration INTEGRATE 7 • Laser-focused on the most critical projects • Risk averse • Resource management This served us very well!
- 8. 8 “We can’t keep throwing bodies at every problem” - Maya Leibman
- 9. 9 Single Flight Operating System
- 11. Connect Collaborate Create Communicate! share knowledge / use all tools Face-to-face sweet spots / visit on-site Do good together volunteer 11 Educate get rid of FUD
- 12. Anyone can add value to any team from anywhere 12
- 13. 11/21/2016 13
- 14. 14 Leveraging tools to communicate!
- 16. Connect Collaborate Create Automate the mundane unleash the team’s potential Create foundations invest in tools all can use Partner with partners hack a user story Enable adoption make it easy 16
- 17. GitHub DevOps Toolchain DevOps Toolchain - Java 17
- 18. Visual Studio TFS GIT TeamBuild MSTest NuGet TFS DevOps Toolchain DevOps Toolchain - .NET 18
- 19. 19 RHEL patching of the Development and Test environments Time for patching 200 RHEL Nodes: Puppet Scripts Manual 5 minutes 3 hours 17 hours Identified systems vulnerable to OpenSSH bug in just under 5 minutes of receiving the notification. Configuration automation
- 20. IT 20
- 23. Partner with partners Hack the user stories 23
- 24. Partner with partners Hack the user stories 24
- 25. Partner with partners Hack the user stories 25As ImplementedAs Designed
- 26. Partner with partners Hack the user stories 26
- 27. Enable adoption: show the opportunity FACET – FA Customer Experience Tool 60% 26% 14% Automation Coverage – Phase 1 Automation Coverage - Phase 1 Not Automated Out of Scope 86% 14% Automation Coverage – Phase 2 Automation Coverage - Phase 2 Out of Scope 27
- 28. Create Lighthouse Examples FACET – FA Customer experience Tool Manual Execution – 162 Cases Automated Execution - Current Automated Execution - Target Execution Time (Calendar Days) 3 Execution Time (Calendar Days) 2 Execution Time (Calendar Days) 1 # of Tablets 3 # of Tablets 1 # of Tablets 2 FTU Consumption 648 FTU Consumption 97 FTU Consumption 97 Execution Cost ($) 7K Execution Cost ($) 1K Execution Cost ($) 1K Execution cost reduced by 85% Execution cost reduced by 85% Execution time reduced by 33% Execution time reduced by 33% Execution time reduced by 33% Execution time reduced by 66% 28
- 29. Make it easier: 29 • Insert experts in with development teams to help get to consistency: • Matrix in architect, tech lead, program office and operations • Sing from the same hymn book! • Use the same test harness • Use the same offshore teams • Use the same tools
- 30. Connect Collaborate Create Deliver faster Reduce bottlenecks Market IT capabilities be THE trusted advisor Simplify the footprint fewer apps, fewer layers 30 Make employees Tools that help the front line
- 32. Make employees happy Delivered development and test environment for the new Crew Swap application in 5 minutes.
- 33. Simplify! Training Management System Consolidating 5 systems Addressing EOSL Same tool for different work groups 33
- 34. Simplify! API’s: Employee Images / Notifications 34 Recognize opportunities to create services that can be reused by other applications: • Image service now used by 5 applications (&growing) • Notification service integrated with 3 channels and used by 4 applications (&growing)
- 35. “Quality in a service or product is not what you put into it. It is what the client or customer gets out of it.” – Peter Drucker 35
- 36. 36
- 37. Connect Collaborate Create A different approach was needed to meet merger integration milestones OT had to change – we couldn’t keep on throwing bodies at it… DevOps was our answer, we just didn’t know that is what it was called back in 2013. 37
- 38. 38 WHAT DO WE NEED FROM DOES? • How do you measure success? • DevOps in an MSP World • How has DevOps simplified your environment? • How to market IT capabilities?
- 39. 39
Editor's Notes
- BC – What I love about the video is that it speaks to our rich history, the many first from American Airlines and a new first, today, an airline speaking at the DevOps Enterprise Summit! SB - Good morning San Francisco! Ben and I are very excited to be here and share with you a little about our DevOps journey.
- SB At the end of 2013, US Airways and American Airlines merged – the financial transaction was fast but the IT integration not so much so. In fact, it will take between 5 to 7 years to complete all the work to consolidate 1400 systems. Because of the volume of work it was agreed that a line in the sand would be drawn , and no innovation (new business ideas) would be pursued until we could get integration completed Otherwise, the consolidation would go on ad naseum Integration became the priority for teams so that we could: Better serve our customers Better serve our employees And of course, realize the value of the merger itself
- SB However, drawing a line in the sand to get work started was easier said than done. Each legacy carrier had different headquarters, different technology stacks and different standards and approaches. Basically, very different cultures. Moving forward with work was an organizational change management effort. Mergers are the perfect vehicle to foster lack of trust and trust was just what we needed if we were going to get the work done.
- BC That’s true. We needed Trust! There were thousands of front end devices that had to be integrated: computers, mobile devices and kiosks The task at hand was: HUUUGGEEE very, very complicated AND planes have to keep flying! Crazy times…
- SB Our projects were not just technical, the changes in many cases greatly impacted the way our employees had to do their work. Despite all these challenges, the AA IT team has delivered, not just one project, but many highly complex efforts in a single year! Since the financial merger in late 2013, we have completed over 1000 projects and are xx% with integration efforts
- SB 2015 was a major year in terms of project delivery. In early 2015 , we consolidated Dividend Miles and AAdvantage, our customer loyalty programs In second quarter, we hit a major milestones by securing the FAA’s blessing to operate as a single carrier While this was all happening, other IT teams were partnering with our corporate real estate group to build a new operational center building and move our dispatchers and crew schedulers from PIT and DFW to single location If that was not enough, we 2015 culminated with the integration of our passenger reservation system. This included changes in airports, reservations , websites, and mobile. It was obviously the most highly customer visible change and it went off seamlessly, as you can see from the newspaper headlines at the time.
- BC You can see that there is no shortage of work with all that goes on in a merger. The success of these projects can be attributed to the fact that we’ve been hyper focused on critical efforts – we call these THE BIG ROCKS and from a resource perspective we added any and all necessary people
- BC However, as our CIO Maya says “We can’t keep throwing bodies at every problem” And, the largest integration item was on the horizon.
- SB For my organization, our epiphany came early on, as we were planning the largest integration project to date… The work on the consolidation of our flight and pilot management system started in late 2014 . This project was called Single FOS or SFP and was the largest, most complex projects to date. In fact, US Airways never completed the merger of the crew management systems. So, SFP really was merger two flight operating systems and three crew management systems. Basically, three airlines into one! We realized we had to take a different approach in the infrastructure arena, otherwise we would not be able to support the volume of work within the timelines prescribed. Historically, environment builds for new applications could take 6 months to get right, yet we had to deliver 26 new applications in that same time frame!
- BC Yes, we had a mountain of work to cover and our systems are critical for AA to fly. As we look back at the last 2 years, we realized that the merger and specifically Single FOS, created the momentum to work differently Connecting teams was foundational and we set the stage in 2014 Intense Collaboration was introduced with SFP in 2015 And Creation has been the culmination in 2016
- BC Connecting is obviously that foundation cultural piece that everyone talks about. In a merger, connecting is even more challenging because of inherent FUD (FEAR, UNCERTAINTY, DOUBT). People were afraid of the implications that the systems they know and love may not exist in the future and were worried for their jobs. The distributed locations of IT development also made connecting and creating trust a challenge.
- SB In order to build trust after merger, we had to rally around a shared vision which we established in 2014. Namely, that any team member could add value to any team from anywhere. Being able to realize this vision meant that leaders and teams had to work to establish common norms. As we say in the south, we had to sing from the same hymn book. We took a very intentional approach to get to know each other, build trust, share knowledge and learn to work together across distances.
- SB As new team members joined the groups, we were able to use our internal website (based on Jive) for social collaboration. This was the place where we could share architectures, team structures and miscellaneous tips. Information from staff meetings would be posted quickly to create transparency and ease the flow of information to all team members. BC You can also see that Susanna is informing everyone of some very important news Chick-fil-A is coming to our dining venue called ‘The Spot’. And if you know me, there’s no better way to connect than over food.
- BC Of course, as technologist, there are other ways to connect. We also leverage tools to communicate in real time. Here you can see my team working on a spike in compute for our Line of Flight Display System which tracks our planes. The team was able to quickly add capacity in our virtual environment and thank goodness we had no disruptions. Here you can see we are targeting LEAN Teams of Software Development – with a focus on “items to complete” And finally, we also use video conferencing and virtual white boards to facilitate exchange of ideas without losing focus and to foster those personal connections.
- SB Connecting is so important. We are social creatures and having shared experiences build bonds. Volunteering, Happy Hours, Team building and Outside activities all make a difference. Of course, these events are planned. You can also have the shared experiences by running through an airport to catch a plane or dancing a bollywood dance as part of a dewali celebration. At American we also make a really big deal out of Halloween which allows us to extend the relationship building to the families of our employees.
- SB 2014 was all about connecting and setting the stage for Collaboration where we focused on automation, synergizing and partnering across the enterprise.
- BC Speaking of automation, the DevOps tool chain is where all of us as technologist gravitate. Again, with our merger, we had different technology stacks and we wanted everyone to come together around a single set of tools; but, it’s best not to slow down work or impede progress in pursuit of this utopia. In the end it’ was about delivering business capabilities and value to our customers; so, we decided to have 2 DevOps Toolchains and allow the developers to work in familiar stacks. Here we have the java tool chain.
- BC Similarly, here’s the .Net toolchain We do share some components across the toolchains. Slack for collaboration. Puppet for configuration state management vRealize Automation for provisioning environments And testing tools
- BC Configuration automation is near and dear to my heart. How many times do we troubleshoot for half a day or days to find out that one of 4 nodes was configured differently than the others. Or, software works fine on a developers machine; but, not in a higher environment due to some configuration change which was not captured. Here we have an example of Redhat patching. Say you have an engineer that needs to patch Redhat. He or she needs to find the information for the server, connect to the server, download the software, apply the software, verify everything updated correctly and then logout and onto the next machine. Now, let’s say they are fast on a keyboard and can do that in 5 or 10 minutes. 5 or 10 minutes isn’t bad. It’s a small amount of time. Well, 5 minutes across 200 nodes and we are talking 1000 minutes or about 17 hours. However, with Puppet we were able to patch the nodes in 5 minutes and it’s not just a speed improvement. It’s a consistency improvement. Automation eliminated the risk of my fast typing engineer actually mistyping. Another example is when we learned about a security vulnerability with OpenSSH and my team was able to determine the systems with the impacted files and correct them in under 5 minutes. So, these are just a couple of examples of technology we are using. Susanna will talk about technology at American.
- SB We like to say that Technology is the engine that makes American Airlines run…
- SB Of course, American Airlines is a large company. To give you a quick introduction to other parts of American, we have: Corporate Technology Customer Technology * Enterprise Technology And, finally, Airline Technology which is where we are at. This area supports maintenance, crews, flight operations, catering, and onboard technologies. We like to think that we ARE AA: the area that makes planes FLY! The reality though is that each and every area is critical for American to thrive.
- SB Operations Technology, cannot operate in a silo. It’s important for us to connect and collaborate across the company. So, we want to make sure that we are proactive in engaging other groups and we want to learn from them and for them to learn from us. Educating on capabilities, and benefits is a first step. Integration has of course helped with adoption since we are all about getting projects done quickly and efficiently. For DevOps to extend beyond Airline Technology, we have actively worked with the enterprise groups: Infrastructure, Network and Security.
- BC Yes, in fact last month, we partnered with our enterprise friends and kicked off a series of ‘Automation Challenges’ focused on solving specific problems. The concept is one day a month we all get together to tackle stories to further our strategy, cut through bottlenecks and clean up technical debt. In this case we wanted to tackle slack and email notification for builds with java and .Net; as well as automate active directory, Siteminder and firewall access requests.
- BC Now at these events we always need lots of sugar. You can see these fabulous Halloween decorated cup cakes!
- BC Of course, we often have a discrepancy between design and implementation; so, while we had all these very ambitious plans we hit some rough waves. But, let’s remember we need to experiment, fail fast and keep improving.
- BC In the end we had 11 initiatives provide readouts at the close of the day. Here you can see Lucas giving a readout on the work and progress made with our enterprise security team with regards to automating siteminder configuration. They have an approach and I look forward to seeing them try it out in the lab.
- SB Moving on to what is probably the Achilles Heel of DevOps… Test Automation We recognize that this area is lagging and have worked to create some Lighthouse projects that we can use as a showcase within the organization. The application furthest ahead in this arena is our Flight Attendant Customer Experience Tool. At the end of last year, our automation coverage was 60% and by the end of this summer we were at 86%. The remaining 14% isn’t automated as it’s attributable to testing the physical peripherals such as the card reader.
- BC The FACET device is key for our Flight Attendants and Here you can see that we were able to reduce cost by 85% and execution time by 33%. We are targeting a 66% time reduction by end of year.
- SB A critical part of collaboration is to institutionalize what has been learned through automation or documentation. This allows for others to follow. Another concept that we have seen pay dividends is inserting experts into the delivery organizations to help gain consistency across the group. This is all taking place while still leveraging our Agile and ITIL methodologies
- SB Connecting and collaborating are essential for us to be able to create more for our business. We need to make sure we are delivering solutions that make our front line employees happy. We want to simplify our footprint because there’s still a cost to managing a large number of applications. And, we strive to be the trusted advisors and partners for our business.
- BC To be a trusted advisor we have to deliver faster starting with environments that developers use to build solutions Traditionally we allowed for 2 months to build out Dev and Test and now we can do this in a matter of minutes. We put into place a self –service portal to allow developers to spin up their own infrastructure through blueprints. This automation not only allows speed of delivery; it provides consistency and empowers the development teams to try new things, run experiments and start and stop what they need. The introduction of the standard build also ensures that configuration changes made by developers in lower environments gets identified and instrumented. So, it’s not just speed, it’s empowerment and consistency.
- SB Empowering our employees to be in the best possible position to help our customers is extremely important at American. 3 months prior to the SFP launch, a new critical application was requested to help flight attendants change their working position on the aircraft. The infrastructure for this new application was delivered within 3 minutes and the application within the month. The project manager over this effort, Ottis Wooten stated that “we were able to deploy the final Crew Position Swap application on the Windows 2012 production hardware 30 days before the due date”
- SB Another key SFP project was the new training management system which helped simplify the application architecture through the consolidation of 5 tools into one and by leveraging the same tool across the pilot and flight attendant work forces.
- BC Another example of simplification was the creation of services that could be leveraged across applications. Here are two that were most recently created by SFP. The new Image Service and the new Employee Notification Service. By having services like these, IT can deliver products faster.
- BC In the end, its about what the customer gets. And, we believe that not only delivering quality; but, delivering quickly on customer requests positions us to be the trusted advisor to the business.
- SB And with that….. Our largest integration project to-date completed successfully on October 1st YES!!!!
- SB A different approach was needed to meet merger integration milestones Operations Technology had to change – we couldn’t keep on throwing bodies at it… DevOps was our answer, we just didn’t know that is what it was called way back then… Merger at the end of the day was the catalyst for the changes in AA needed for DevOps to take hold.
- BC I’m glad we could share our story with you. We’re still very much in the infancy of our DevOps journey and I look forward to sharing & benefiting from the collective knowledge of this group. A few things we’d like to know. How do you measure success? How do you handle DevOps in a Managed Service Provider World? How has DevOps Simplified your environment? And, finally, how do you market your IT capabilities?
- Thank You all so much for the time! I hope we see you flying American!