Transcript of a BriefingsDirect podcast with Delta Air Lines executive on gaining visibility into application testing to improve customer experience.

1. Delta Air Lines Improves Customer Self-Service Apps Quickly
Using Quality Assurance Tools
Transcript of a BriefingsDirect podcast with Delta Air Lines executive on gaining visibility into
application testing to improve customer experience.
Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Sponsor:
HP.
Dana Gardner: Hello, and welcome to a special BriefingsDirect podcast series, coming to you
from the HP Software Universe 2010 Conference in Washington D.C. We're here
the week of June 14, 2010, to explore some major enterprise software and
solutions trends and innovations making news across HP’s ecosystem of
customers, partners, and developers.
I'm Dana Gardner, Principal Analyst at Interarbor Solutions and I'll be your host
throughout this series of HP sponsored Software Universe Live discussions.
Our customer case study today focuses on Delta Air Lines and the use of HP quality assurance
products for requirements management as well as mapping the test cases and moving into full
production. We are here with David Moses, Manager of Quality Assurance for Delta.com and its
self service. Thanks for joining us, David.
David Moses: Thank you very much. Glad to be here.
Gardner: We're also here with John Bell. He's a Senior Test Engineer at Delta. Welcome John.
John Bell: Thank you. Welcome.
Gardner: Guys, tell me about the market drivers. What is the problem set when it comes to
managing the development process around requirements and then quality and test out through
your production? What are the problems that you're generally facing these days?
Moses: Generally, the airline industry, along with the lot of other industries I'm sure, is highly
competitive. We have a very, very quick, fast-to-market type environment, where
we've got to get products out to our customers. We have a lot of innovation that's
being worked on in the industry and a lot of competing channels outside the
airline industry that would also like to get at the same customer set. So, it's very
important to be able to deliver the best products you can as quickly as possible.
"Speed Wins" is our motto.
Gardner: What is it about the use of some of the quality assurance products that helps you pull
off that dual trick of speed, but also reliability and high quality?

2. Moses: The one thing I really like about the Quality Center suite especially is that your entire
software development cycle can live within that tool. Whenever you're using different tools to do
different things, it becomes a little bit more difficult to get the data from one point to another. It
becomes a little bit more difficult to pull reports and figure out where you can improve.
Data in one place
What you really want to do is get all your data in one place and Quality Center allows you to do
that. We put our requirements in in the beginning. By having those in the system, we can then
map to those with our test cases, after we build those in the testing phase.
Not only do we have the QA engineers working on it in Quality Center, we also have the
business analysts working on it, whenever they're doing the requirements. That also helps the
two groups work together a bit more closely.
Gardner: Do you have anything to add to that, John?
Bell: The one thing that's been very helpful is the way that the Quality Center tabs are set up. It
allows us to follow a specific process, looking at the release level all the way down
to the actual cycles, and that allows us to manage it.
It's very nice that Quality Center has it all tied into one unit. So, as we go through
our processes, we're able to go from tab to tab and we know that all of that
information is interconnected. We can ultimately trace a defect back to a specific
cycle or a specific test case, all the way back to our requirement. So, the tool is very helpful in
keeping all of the information in one area, while still maintaining the consistent process.
Gardner: Can you give us a sense of how much activity you process or how many applications
there are -- the size of the workload you’ve got these days?
Bell: There is a lot. I look back to metrics we pulled for 2008. We were doing fewer than 70
projects. By 2009, after we had fully integrated Quality Center, we did over 129 projects. That
also included a lot of extra work, which you may have heard about us doing related to a merger.
Gardner: With that increase in the number of applications that you're managing and dealing
with, did you have any metrics in terms of the quality that you were able to manage, even though
that volume increased so dramatically?
Moses: We were able to do that. That's one of the nice things. You can use your dashboard in
Quality Center to pull those metrics up and see those reports. You can point out the projects that
were your most troublesome children and look at the projects where you did really well.

3. Best-case scenario
You can go back and do a best-case scenario, and see what you did great and what you could
improve. Having that view into it really helps. It’s also beneficial,
whenever you have another project similar to one that was such an
issue. You can have a heads up to say, "Okay, we need to treat this
one differently this time."
Gardner: It’s the visibility to have repeatability when things go well, and, I suppose, visibility to
avoid repeatability when things didn't go well.
Moses: Exactly.
Gardner: Let’s take a look at some of the innovation you've done. Tell me a bit about what
you've worked with in terms of Quality Center in some of your own integration or tweaking?
Bell: One thing that we've been able to do with Quality Center is connect it with Quick Test Pro,
and we do have Quality Center 10, as well as Quick Test Pro 10. We've been able to build our
automation and store those in the Test Plan tab of Quality Center.
This has really been beneficial for us, when we go into our test labs and build our test set. We're
able to take all of these automated pieces and combine them into test set. What this has allowed
us to do is run all of our automation as one test set. We've been able to run those on a remote
box. It's taken our regression test time from one person for five days, down to zero people and
approximately an hour and 45 minutes.
Also, with the Test Lab tab, we're able to schedule these test sets to run during off hours. A lot of
times our automation for things such as regression or sanity, can run on off hours. We schedule
those to run at perhaps 6 o'clock in the morning. Then, when we come in at 8 o'clock in the
morning, all of those tests would have already run.
That frees up our testers to be doing more of the manual functional testing and that allows us to
know that we have complete coverage with the automation, as well as our sanity pieces. So,
that's a unique way that we've used Quality Center to help manage that and to reduce our testing
times by over 50 percent.
Gardner: Thank you, John. David, there have been some ways in which your larger goals as a
business have been either improved upon or perhaps better aligned with the whole development
process. I guess I'm looking for whether there is some payback here in terms of your larger
business goals?
Moses: It definitely is. It goes back to speed to market with new functionality and making the
customer's experience better. In all of our self-service products, it's very important that we test
from the customers’ point of view.

4. We deliver those products that make it easier for them to use our services. That's one of the
things that always sticks in my mind, when I'm at an airport, and I'm watching people use the
kiosk. That's one of the things we do. We bring our people out to the airports and we watch our
customers use our products, so we get that inside view of what's going on with them.
A lot on the line
I'll see people hesitantly reaching out to hit a button. Their hand may be shaking. It could be an
elderly person. It could be a person with a lot on the line. Say it’s somebody
taking their family on vacation. It's the only vacation they can afford to go on,
and they’ve got a lot of investment into that flight to get there and also to get
back home. Really there's a lot on the line for them.
A lot of people don’t know a lot about the airline industry and they don’t
realize that it's okay if they hit the wrong button. It's really easy to start over. But, sometimes
they would be literally shaking, when they reach out to hit the button. We want to make sure that
they have a good comfort level. We want to make sure they have the best experience they could
possibly have. And, the faster we can deliver products to them, that make that experience real for
them, the better.
Gardner: I should think the whole notion of self service is usually important. It's important for
the customer to be able to move through and do things their way, and I suppose there are some
great cost savings and efficiencies on your end as well.
Dave, you could just highlight a little bit about how the whole notion of self service embedded
into applications. It's important how some of the quality assurance tools and processes have
helped there.
Moses: I go back to anytime you have to give up whenever you're having an issue with products,
while you're online. You're on a website, and you have to call customer service. I think most
people just sort of feel defeated at that point. People like to handle things themselves. You need a
channel there for the customer to go to, if they need additional help.
So many clients and customers these days are so tech savvy. They know the industry they are in,
and they know the tools they're working with, especially frequent flyers. I'd venture to say that
most frequent flyers can hit the airport, check-in, get through security, and get to their plane
really quickly. They just know their airports and they know everything they need to know about
their flight, because this is where they live part of their lives.
You don't want to make them wait in line. You don't want to make them wait on a phone tree,
when they make a phone call. You want them to be able to walk into the airport, hit a couple of
buttons, get through security, and get to their gate.

5. By offering these types of products to the customers, you give them the best of both worlds. You
give them a fast pass to check in. You give them a fast pass book. But, you can also give the less-
experienced customer an easy-to-understand path to do what they need as well.
Gardner: And, to get those business benefits, those customer loyalty benefits, is really a function
of good software development overall, isn't it?
Moses: Exactly. You have to give the customer the right tools that they want to get the job done
for them.
Gardner: For other enterprises that are perhaps are going to be working towards a higher degree
of quality in their software, but probably also interested in reducing the time to develop and time
to value, do you have any suggestions, now that you’ve gone through this, that you might offer to
them?
Interim approach
Bell: In using Quality Center, we've uses an interim approach. Initially, we just used the Defects
tab of Quality Center. Then, we slowly began to add the Requirements piece, and then Test
Cases, and ultimately the Releases and Cycles.
One thing that we've found to be very beneficial with Quality Center is that it shows the
development organization that this just isn't a QA tool that a QA team uses. What we've been
able to do by bringing the requirements piece into it and by bringing the defects and other parts
of it together, is bring the whole team on board to using a common tool.
In the past, a lot of people have always thought of Quality Centers as just a little tool that the QA
people use in the corner and nobody else needs to be aware of. Now, we have our business
analysts, project managers, and developers, as well as the QA team and even managers, because
each person can get a different view of different information.
From Dashboard, your managers can look at your trends and what type of overall development
lifecycle is coming through. Your project managers can be very involved in pulling the number
of defects and see which ones are still outstanding and what the criticality of that is. The
developers can be involved via entering information in on defects when those issues have been
resolved?
We've found that Quality Center is actually a tool that has drawn together all of the teams.
They're all using a common interface, and they all start to recognize the importance of tying all
of this together, so that everyone can get a view as to what's going on throughout the whole
lifecycle.
Moses: John hits on a really good point there. You have to realize the importance of it, and we
did a long time ago. We've realized the importance of automating and we've realized the
importance of having multiple groups using the same tool.

6. In all honesty, we were just miserable in our own history of trying to get those to work. You
really take certain shots at it. For the past eight years, if we can go back that far, we've been
using Quality Center tools for Test Director, just trying to get things automated, using the tools
we had at the time.
The one thing that we never actually did was dedicate the resources. It's not just a tool. There are
people there too. There are processes. There are concepts you're going to have to get in your head
to get this to work, but you have to be willing to buy-in by having the people resources dedicated
to building the test scripts. Then, you're not done. You've got to maintain them. That's where
most people fall short and that's where we fell short for quite some time.
Once we were able to finally dedicate the people to the maintenance of these scripts to keep them
active and running, that's where we got a win. If you look at a web site these days, it's following
one of two models. You either have a release schedule, that’s a more static site, or you have a
highly dynamic site that's always changing and always throwing out improvements.
We fit into that "Speed Wins," when we get the product out for the customers’ trading, and
improve the experience as often as possible. So, we’re a highly dynamic site. We'll break up to
20 percent of all of our test scripts, all of our automated test scripts, every week. That's a lot of
maintenance, even though we're using a lot of reusable code. You have to have those resources
dedicated to keep that going.
Gardner: Well, I appreciate your time. We've been talking about the quality assurance process
and the use of some HP tools. We've been learning about experiences from Delta Air Lines
development executives. I want to thank our guests today, David Moses, Manager of Quality
Assurance for Delta.com in the self-service function there. Thank you, David.
Moses: Thank you very much.
Gardner: We've also been joined by John Bell, Senior Test Engineer there at Delta Air Lines.
Thanks to you too, John.
Bell: It's been a pleasure.
Gardner: And, thanks to our audience for joining us for this special BriefingsDirect podcast
coming to you from the HP Software Universe 2010 conference in Washington DC.
Look for other podcasts from this HP event on the hp.com website, as well as via the
BriefingsDirect Network.
I'm Dana Gardner, Principal Analyst at Interarbor Solutions, your host for this series of Software
Universe Live Discussions. Thanks again for listening, and come back next time.
Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Sponsor:
Hewlett-Packard.

7. Transcript of a BriefingsDirect podcast with Delta Air Lines executive on gaining visibility into
application testing to improve customer experience. Copyright Interarbor Solutions, LLC,
2005-2010. All rights reserved.
You may also be interested in:
• HP's BTO Chief Ben Horowtiz on How Application Lifecycle Optimization Enhances
Next Generation Data Centers
• Winning the Quality War: HP Customers Offer Case Studies on Managing Application
Performance
• HP Software Marketing Head Anton Knolmar Delves into Creating New IT Economies of
Performance