SlideShare uma empresa Scribd logo
1 de 28
Baixar para ler offline
SHIFT LEFT

Quality Assurance
Best Practices in
Digital Marketing
You cannot inspect quality into a product. The quality is
there or it isn’t by the time it’s inspected.
Harold S. Dodge, via W. Edwards Deming

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

2
Quality Assurance: We Struck a Nerve
Last October in Chicago, AbelsonTaylor and GA Communication Group
co-hosted the first ever Quality Assurance Summit for digital marketing.
The original thought was to get 10 guys in a room to talk about the issues
that affect the quality of digital projects. Well, we certainly struck a nerve with
this topic. It turns out that everyone we spoke to has the same problems.

Presenters included Jim Spillson, Manager of Quality Assurance,
AbelsonTaylor, as well as Elizabeth Estes EVP, Chief Strategy Officer,
Ben Currie, Group Director, Digital Solutions, GA Communication Group,
and Shachar Schiff, Principal, Bad Testing. Michael Morowitz, Senior
Technology Director, R/GA Chicago was the keynote speaker.

Join the QA Conversation

So, we ended up with over 50 marketing directors, QA professionals
and digital directors from the leading healthcare companies and
advertising agencies meeting all day at the Ritz-Carlton to try to figure
out some best practices.

This white paper is the product of a lot of blood, sweat, tears and
laughter. But it’s clearly not the end of the conversation. Please join
us at stateofqa.com and through @StateofQA to help refine the best
practices in quality assurance.

The Goal: Take a Holistic View of QA
The QA Summit was moderated by Joe Shields, Global Director,
Campaign Management, AstraZeneca and included a presentation
on the state of QA, an expert panel discussion and a client roundtable,
as well as a group working session.

A Word of Thanks
Finally, we’d like to thank everyone who took time out of their busy
schedules to make this summit a reality. QA is in a much better place
due to your passion and honest insights.

Joe Shields

Jay Carter

Geoff Melick

Jim Spillson

Ben Currie

Michael Morowitz

Scott Hansen

Global Director,
Campaign Management,
AstraZeneca

SVP, Director of
Strategy Services,
AbelsonTaylor

EVP, Chief
Innovation Officer,
GA Communication Group

Manager of
Quality Assurance,
AbelsonTaylor

Group Director,
Digital Solutions,
GA Communication Group

Senior Technology
Director,
R/GA Chicago

VP, Executive Director,
Digital Creative,
AbelsonTaylor

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

3
Glitch Happens
Need proof that quality assurance in digital healthcare marketing has hit a
nerve? Just take a look back at HealthCare.gov—the digital marketplace
that launched, notably on schedule, only to become the most publicized
and preventable software failure.
The site is intended to improve healthcare access, efficiency and quality.
Instead, it’s become a public service announcement on the state of
quality assurance.

156 Weeks of Planning—And Only 2 Weeks
of Testing at the End
The scope of HealthCare.gov was massive. The launch was highly
anticipated. And as time to market approached, a disparity between
project requirements, documentation and expectations continued to
grow. Cross-functional teams spanning various government agencies and
55 contractors took 3 years (or 156 weeks) to concept, plan and develop
the software. But proper testing didn’t occur until 2 weeks prior to launch.
From a QA perspective, the project was dead on arrival.

We’re Not Just Testing Tech
Unfortunately, this scenario is all too familiar to digital healthcare marketers: the complex requirements,
the lack of documentation, the nebulous stakeholders and responsibilities, the silos of communication,
the late-breaking scope changes, the immovable deadline. Quality assurance starts way too late in the
process—like at the end.
To manage the risks inherent in digital projects, it’s critical that quality be an intrinsic part of the process.
It has to be an attribute of the process, not an afterthought. Quality is the means and the end. And every
team, every individual must have quality procedures built into what they do.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

Failing to plan is planning to fail.
Alan Lakein

4
Quality Assurance—
It’s Hard to Keep Up
Quality assurance is not just finding bugs in the code. It’s the impact of flaws in direction—
or misdirection if you will—that cause work to be scrapped or redone late in the process.

Testing is not assurance.

Quality assurance is keeping up with changes in hardware and software. In the last year,
over 30 different digital devices, platforms, operating systems and browsers
were released.

Michael Morowitz,
Senior Technology Director,
R/GA Chicago

Quality assurance is staying current with how your target consumes digital media. For
instance, a recent study of physician behavior by Epocrates found that there would be
more digital doctors than non-digital doctors in 2014. That means that in the 2-year period
between 2012 and 2014, there will be a complete flip in the ratio of connected clinicians
from 20:80 to 80:20.
At the same time, something called an E-patient has appeared on the horizon,
empowered by apps and social media, who is having his or her voice heard by
healthcare companies.
When this accelerating velocity of change crashes into the typical development life cycle of
digital projects—6 months to a year—one can clearly see why quality assurance struggles
to keep pace.

“Mobile Trends Report,” Epocrates, 2013.
Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

5
Looking for Problems
in the Right Places
Research has shown that 56% of all digital project
problems occur during the requirements phase of a project.
27% occurs during the design phase. Meaning that the
majority of bugs happen before programming even starts.
However, the typical QA process doesn’t even begin until
after the programming portion of the project is complete.

Where
problems
originate

That means that bugs that start early on in a project just
sit there, costing time and money.

That’s the trap
If you don’t know what you’re looking for, or even when or
where you should be looking, why bother with quality at
all? This leads to the common practice of just “looking at
it before it goes out the door.”
Without clearly defined quality assurance principles,
processes, goals and metrics up-front, it’s easy to see
why everyone falls in the trap of only testing at the end.

Requirements

Design

Code

Other

56%

27%

7%

10%

CrossTalk Magazine—The Journal of Defense Software Engineering

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

6
The Cost of Quality
Quality begins with the strategy. It includes customer considerations, on-target
messaging, intuitive interface design, product branding and airtight code.
The earlier you find a problem, the less it will cost to fix. If we find an issue in the
requirements phase, the cost to fix is measured in the time it takes to edit a
requirements or specifications document.

Problems found at the end cost 10 times more to fix than
problems found in the beginning.
Jim Spillson, Manager of Quality Assurance, AbelsonTaylor

If we find that same problem during the testing phase—requiring new content, new
functionality, new code, new compatibility testing—it can easily cost 10 times as much
to fix, maybe more.
The cost of quality isn’t the price of creating quality products or services. It’s the
cost of NOT creating quality products or services. Every time work is redone, the cost
of quality increases.

Time detected
Cost to fix a defect
Requirements

Development

Testing

Post-release

Requirements

1×

3×

5–10×

10×

10–100×

Doc/Creative

–

1×

10×

15×

25–100×

Development

Time
introduced

Doc/Creative

–

–

1×

10×

10–25×

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

7
QA Is More Than Tech and Code
In the end, quality assurance is more than checking for glitches in the code.
Quality assurance is protecting the value of your brand. Every mistake that goes live leads
to a decreased user experience by your target because it affects load time, how email
replies are handled and how easy or difficult forms are to fill out.
In other words, defects erode the relationship your target has with your brand.

Quality is not somebody’s department. It’s not
a made-up job. It’s the cost of entry in business.
Joe Shields, Global Director,
Campaign Management, AstraZeneca

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

8
Shift Left: Quality Assurance From the Start
The concept of Shift Left is more than an adjustment to a timeline.
It’s more than finding glitches in the code.

communications, better interface design, and projects that benefit both
our clients and customers.

Shift Left operationalizes the idea that quality assurance starts at the
beginning, during the planning stage of a project. And that quality
assurance is present during every subsequent step of the way.

The Benefits of Shift Left

The goal of Shift Left is to be more nimble, accurate and inventive
when creating digital projects. If we’re looking for defects up-front, we
can catch them and move on. That’s nimble. If we catch defects as they
occur, we improve accuracy. Finally, if we’re both nimble and accurate,
we allow ourselves the opportunity to be more inventive—creating better

Number of Defects

Job Opens

Requirements

Design

Development

The benefit of Shift Left is that we can now set and meet expectations
for successful quality assurance across every browser, every operating
system and every device. And for every brand.
And we can be efficient because doing it right, at the right time, is faster
than finding and fixing the problems just days before launch.

Test

Release

Shift Left

Shift Left
Life Cycle

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

Early Detection
of Defects
Project Life Cycle

9
5 Ways QA Challenges
Clients and Agencies
It all starts with the fact that we don’t have a consensusdriven definition of what quality means. Talk to an account
person, a creative, a technologist, a brand manager—they
all have different definitions of quality.
For the software engineer, quality can be measured by
defect density. For the creative, it’s that the interface looks
and functions properly. For the brand manager, quality
revolves around messaging, timeline and budget. For the
customer, it’s all about having a painless experience with
your digital asset.
These disparate views of quality assurance put the project
and participants at risk: budgetary risk, timeline risk, client
expectation risk and customer satisfaction risk.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

1

Creative Client Service Focus
Clients don’t approach agencies for technology. Clients
are looking for strategy, service and creative.
So, the first challenge is that the people we talk to on the
client side are not usually technologists. They are CMOs.
They are marketing directors. They are brand managers.
What does that mean to the concept of quality assurance?
It means that no agency is going to win business solely
based on QA. Conversely, there is a real chance that an
agency will lose business because of bad QA. Clients
expect perfect. When we miss perfect, we put the
business at risk.

10
5 Key QA Challenges

2

Custom Technology
As brand champions, agencies find themselves between a
rock and a hard place when it comes to custom technology.
Differentiation is key to the success of any brand. That
differentiation extends to technology. Custom technology
is literally in an agency’s charter and DNA.
So, we have to consider the cost and liability of built from
scratch. When it’s custom, we end up being the sole
defender of the technology. We’re responsible for ensuring
that each and every browser and OS update doesn’t break
our tech.

3

Varied Schedules
It’s challenging when you are trying to fit test planning,
process management, software testing and non-standard
deliverables into schedules that last anywhere from
2 weeks to 18 months.
The challenge is compounded when you add in sales
cycles, medical/regulatory/legal reviews and time-sensitive
FDA mandates.

As a result, custom technology places a huge burden on
the quality assurance process.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

11
5 Key QA Challenges

4

Non-Standard Deliverables
There are situations when we have no technology
deliverables. For instance, there are occasions when we
deliver creative—in the form of PSDs—to a client IT team,
who then builds the site.
So the question is, if technology is not involved, and QA is
a function of the technology team, is the client not getting
quality assurance?

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

5

Wide Range of Platforms
Companies like AstraZeneca, Capital One or Nike have
an infrastructure, and they have opinions, and they know
where they want the technology to go. Almost every
agency feels the pain of having to operate under and
across a broad range of platforms.

12
A clear definition of quality goals leads directly to a clear
definition of QA principles.
Michael Morowitz, Senior Technology Director,
R/GA Chicago

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

13
4 Core Principles of Quality Assurance

1

Complete Process Coverage
Testing is important, and testing at the end is really
important. But it’s also just one tool in our tool belt.
Requirements and standards analyses, creative reviews
and test planning are all critical-to-quality moments that
need to come into play as we Shift Left.

Testing

Test
Planning

Requirements
Analysis

Project Timeline

Testing

Typical testing paradigm

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

Standards
Analysis

Testing

Testing

Creative
Reviews

Testing

Testing

Testing

Project Timeline
How Shift Left affects testing

14
4 Core Principles of QA

2

Holistic Knowledge

Strategy &
Analytics

If you aren’t seeing the big picture, you aren’t providing
quality assurance.
That’s why each team member must be aware of the roles
and responsibilities of their teammates. Each team member
must understand how the team will work to bring all the
pieces together, and how they will monitor quality, in order
to ensure success.
Holistic knowledge of the project and collaboration is the
glue that prevents the types of mistakes that happen when
team members work in silos. That’s when problems get
sent downstream for someone else to figure out and when
costs start climbing.

Copy

Social &
Mobile

iXD

Interface

Tech

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

15
4 Core Principles of QA

3

Independent Accountability
This principle is specifically about the agency’s org chart.
It states that the QA lead should, ideally, report to a
management structure that is not shared with any other
production groups. More specifically, a QA analyst/tester
should NOT report to the Technology Director, as is
common practice in many agencies.
Independence ensures a greater “purity of practice”
of quality assurance. The primary team members who
are accountable for quality assurance are themselves
accountable to a team structure that has quality
assurance as its primary mission.

4

Team Focus
The idea of Team Focus is that quality assurance lives
everywhere. Everyone’s responsible. While Team Focus is
an extension of Independent Accountability, it’s different.
Team Focus states that even though there is an
independently accountable QA lead, ALL team members
are responsible for quality. QA is not fully contained within
the QA lead. Every functional and every non-functional area
should have a role within the QA process.

Therefore, it makes sense for QA analysts and testers
to report to an Executive Director of QA.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

16
The ‘cost of quality’ isn’t the price of creating a quality
product or service. It’s the cost of NOT creating a quality
product or service. Every time work is redone, the cost
of quality increases.
American Society for Quality, ASQ.org

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

17
The Shift Left Model—
Outcomes of the QA Summit
To make a healthcare analogy, the Shift Left model of quality assurance is a lot like preventive
care. You anticipate problems and head them off before any real damage is done.
Testing at the end of a project, on the other hand, is similar to going to the emergency
room. Aside from it costing you an arm and a leg, emergency room visits usually
happen in a shroud of panic. Judgment is replaced by reactivity.

Build things that aren’t broken in the first place.
Geoff Melick, EVP, Chief Innovation Officer,
GA Communication Group

Shift Left is More Than an Action Step. It’s a Mind-set.
There is a gap in understanding around the role of QA.
QA is often confused with functional testing.
However, QA is more. It’s getting detailed requirements, it’s clearly identifying stakeholders
and responsibilities, it’s having a system for effective communication, it’s a proper process
of scope change and it’s effective management of expectations.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

18
10 Practical Guiding Principles
of Shift Left
During the course of the conversation, it became clear that there are 3 key areas of
focus inside the idea of Shift Left: 1) Planning and Communication, 2) Concept Creation
and 3) Execution.
The following 10 Practical Steps are intended to foster a strong, from beginning-to-end
mind-set around QA that you can apply to your current set of processes.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

Shift Left allows you to be more nimble,
accurate and inventive.
Scott Hansen, VP, Executive Director,
Digital Creative, AbelsonTaylor

19
Planning and Communication
Here everyone involved must be made aware of the project plan and understand his or her role in making it happen.
Be certain to find and use common language. It will go a long way in assuring that stakeholders truly understand
what’s going to happen.

1

Identify Key Stakeholders
Everyone involved in the creation of a digital project—both client and
agency—is a stakeholder.

of introducing oneself changed the behavior and the dynamics of
responsibility in the room.

And nothing is more important than identifying those key decision
makers and their roles, both at the agency and at the client. Having
the right people in the right place at the right time is a key step in
creating an empowered and accountable team.

When someone stood up and said, “My name is Michael Smith and
I am the anesthesiologist,” or “My name is Betty Jones and I am
the head nurse,” it changed the way the team functioned. Lines of
responsibility became clear to the team members. In addition, team
members became empowered to speak up and identify issues.

It’s also critical that all of the stakeholders discuss what they’re doing
on the project.
In the book The Checklist Manifesto, How to Get Things Right,
one of the items on Dr. Atul Gawande’s pre-surgery checklist is for
everyone in the operating room to introduce themselves and their
role. What Dr. Gawande found was that the simple activity

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

Avoid the typical agency dynamic where you walk into a project
kickoff and everyone just starts talking. Roles are assumed. “Oh,
there’s the tech guy, he must be in charge of QA.” This type of
thought process has disaster written all over it.

20
Planning and Communication

2

Share the Plan
Everyone hates surprises.
That’s why it’s so important to share the plan—strategy,
messaging, media, business requirements documents,
functional requirements documents, design, metrics,
test plan—with everyone up-front.
And gain approval before moving on.
Each and every stakeholder must have a clear
understanding of where “true north” is for the project.
This includes agency account, creative and tech, as well
as client brand managers, IT and sales management.

QA includes understanding what the customer wants, their
channel preferences and delivering the right message.
Matt Barry, Associate Director,
HCP Multichannel Marketing, Astellas Pharma US

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

3

Address Barriers to Success
Take the time to course correct before you’ve spent a dime.
If corporate direction, scope, legal requirements or brand
imperatives change, now is the time to address any updates
to the plan.
It may not just be changes that are the barriers. The
timeline, an unknown platform or a team that has never
worked together could all be potential pitfalls.
Similarly, language barriers can betray project success.
A team may share the same native tongue, but few are
fluent in both the languages of marketing and technology.
Critical success factors means something to marketing
folks, whereas critical-to-quality moments means
something to the tech folks.
Establishing a common lexicon for strategic goals breaks
down the walls between team members, allowing them to
clearly define and effectively communicate shared goals
and commitment to success.
Project planning should come to a “dead stop” until you’ve
had time to discuss what steps the team will collectively
take to understand and address the barriers to success.

21
Concept Creation
It’s critical that stakeholders understand what the project is trying to accomplish and how. Beware of vague head nods
when you present wireframes. Take the extra steps necessary to actually show how your program will perform before you
invest a lot of time and money.

4

Don’t Reinvent the Wheel
Don’t fall into the custom-made trap. Not every project
requires a novel operating system or unique piece of
hardware. Nor should the client be forced to pay for it either.
This doesn’t mean you shouldn’t be inventive. Many times,
your client will require a custom solution to their problems.
It does mean, however, that if a platform solution such as
Adobe DPS or Drupal can solve your problem, then don’t
build your own platform. Using pre-existing resources can
reduce the liability of having to test both the platform and
the project.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

5

Proof of Concept
If a picture is worth a thousand words, how much is a
Proof of Concept worth?
Is it priceless?
A Proof of Concept does 2 things: 1) it gives you the ability
to show a client an early example of how a project will look
and perform. This always goes a long way to eliminating
confusion as to what the end product will actually be; and
2) it allows you to solve sticky tech issues before the entire
team starts burning hours.
Even if you need multiple Proofs of Concept before you
gain stakeholder alignment, you haven’t incurred the
expense of total front-end and back-end development only
to find out that what’s been built isn’t what the client wants
or expects. It also provides the opportunity to determine
if and how the product can be tested.

22
Concept Creation

6

Match Execution to Strategy
Due to the complexity of digital projects, oftentimes there
is such an intense focus on the technological aspects that
strategy ends up taking a back seat.
Decisions made during the project can dilute the
original intention. It’s as if the strategic mission has
been communicated via a game of telephone.
A key to quality assurance is to make certain that as
adjustments are made to the functional requirements,
they don’t disrupt the strategy of the overall project.

61% of users have better opinions of brands that offer
a good mobile experience. That makes mobile crucial
to Shift Left.
Shachar Schiff, Principal, Bad Testing

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

23
Execution
Here quality assurance is dependent on the discipline of the team to stick to, and not deviate from the plan.

7

Don’t Overengineer
Consider making a minimal viable product.
There’s nothing wrong with having a master plan for a
project. After all, it’s digital. This doesn’t mean you have
to start small. It does mean you have to start reasonable.
Make sure that what you’re building fits the available
timeline and budget.
Look to a second, third or fourth release to increase content
and functionality. This will go a long way toward keeping
your asset intuitive and relevant to your customers. Starting
with a minimally viable product allows you to be quicker to
market. Tighter release schedules provide tighter customer
feedback loops, offering you the ability to collect customer
intelligence that leads to actionable insights that can impact
strategy and content of future rollouts.

8

Don’t Build Until Everyone Is Onboard
Measure twice and cut once.
Consider making a “Digital Proof” of your project. A Digital
Proof is a fully functioning sliver of the full project. A good
rule of thumb is to build the home page and one vertical
of the content architecture.
As with picture lock in broadcast or a print proof, once the
Digital Proof is approved, it allows you to say, “Okay, now
we’re going to spend the big money finishing the project.”
By taking this step of sharing with stakeholders exactly
how the project will look and feel, you eliminate building
the complete piece without interim approvals.

Finally, if the ask is for an economy car, don’t build the
luxury version to satisfy some misguided sense of “doing
my best.”

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

24
Execution

9

Don’t Change Your Mind
Change is the only constant.
Changes in the competitive environment, shifts in customer
tastes and brand updates all need to be vetted by the
proper client and agency stakeholders. Everyone wants to
avoid shooting from the hip and agreeing to something that
can’t be accomplished.
To that end, it’s critical to have a change order process in
place. And to understand how change will impact timeline
and budget.
The concept of Shift Left provides ample opportunity
to fine-tune and adjust projects every step of the way.
Hopefully, once a project reaches the Execution phase,
direction is set and it’s all smooth sailing. While that may
be wishful thinking, it’s important to have a process in
place to articulate the impact of change.

10

Maintenance
Customers are asking for digital platforms that last for a
very long time. For instance, if you have type 1 diabetes,
you don’t have it for 18 months, you don’t have it for the
6 months that you are interested in diabetes—you have it
for your entire life.
This has led to a change in the agency-client relationship.
Where once an agency would hand over disks and walk
away from a project, now the 2 groups are much more
connected over a life cycle of browser and OS updates,
as well as the ever-changing behavior of our target
audiences in regard to technology.
Maintaining the integrity of a digital project over time is
critical-to-quality factor. Digital assets typically have a much
longer shelf life than broadcast or print. Digital also tends
to remain live long after the original client stakeholder has
been promoted to another job.
Therefore, it’s critical to create a maintenance plan that
offers a clear road map and direction to anyone who inherits
the project. This includes content updates and schedule,
as well as budget. It also includes periodic assessments
of newly released browsers, OSs and devices.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

25
Quality: The Cost of Entry
A final thought: quality is not somebody’s department, it’s not a made-up job and
it’s not another gatekeeper. Quality doesn’t happen at the end of a project.
In reality, quality is the cost of entry.
Shift Left means we have to have the mind-set to catch quality when we are writing a
project brief, when we are developing strategy, during the creative process and well
before the project goes to functional testing.

Shift Left is more than an action step.
It’s a mind-set.
Ben Currie, Group Director, Digital
Solutions, GA Communication Group

Handshakes, Not Handoffs
Therefore, it is incumbent upon agencies to include the development team from the
beginning. It’s incumbent upon clients to admit that, “We don’t know what we don’t
know.” That’s part of the issue: trying to admit and saying, “I don’t understand what you
are saying,” and asking, “Why is that a big deal?”
We need to be more honest with what we don’t know; we need to be more patient when
the team comes back with answers that make our heads want to blow off.
If we are able to shift quality assurance left in the timeline, we can save a lot of angst as
well as a lot of money.

Quality Is an Ongoing Conversation
While the 2013 QA Summit was the first, it clearly isn’t the end of the conversation.
There are so many aspects of quality that need further investigation and discussion.
Our goal is to continue the conversation on stateofqa.com and through @StateofQA.
And with your help, we’ll continue to refine the best practices in quality assurance.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

26
Participants:
Company

Contact name

Title

Company

Contact name

Title

AbelsonTaylor

Dale Taylor

President, CEO

GA Communication Group

Jody Cahill

SVP, Director of Agency Operations

AbelsonTaylor

Jay Carter

SVP, Director of Strategy Services

GA Communication Group

Leah Fergus

Project Manager

AbelsonTaylor

Jim Spillson

Manager of Quality Assurance

GA Communication Group

Kate Shudy

Senior Project Manager

AbelsonTaylor

Scott Hansen

VP, Executive Director Digital Creative

CloserLook

David Ormesher

CEO

AbelsonTaylor

Stephen Neale

SVP, Executive Creative Director

CloserLook

Patrick Conlon

Director of QA and BA

AbelsonTaylor

Brendan Turner

Director of Digital Production

Corbett Accel Healthcare Group Dennis Hoppe

Chief Creative Officer

AbelsonTaylor

Adam Clark

Associate Director of Interactive Technology

Corbett Accel Healthcare Group Chris Seda

Group Creative Director

AbelsonTaylor

Jose Andrade

VP, Director of Interactive Technology

Discovery USA

Jeff Pazen

SVP, Digital Strategy

AbelsonTaylor

Jeff Berg

SVP, Director of Client Services

Discovery USA

Colin Gildea

Director of Interactive Development

AbbVie

Nancy Replogle

IT Director, eBusiness, Finance & Marketing

DraftFCB

Carl Arndt

Lead Quality Assurance

AbbVie

Eruch Rustomji

Sr. Manager, Application Technology

Greater Than One

James Askew

Director of QA

AbbVie

Jaskarn Singh

Sr. Manager, CRM Database Strategy

Greater Than One

Caren Estesen

Senior QA

Astellas Pharmaceutical

Matt Barry

Associate Director, HCP Multichannel Marketing HY Connect

Larry Farkos

VP, Director of Operations

Astellas Pharmaceutical

Richard Grueter

Sr. Manager Strategic Sourcing

HY Connect

Ryan Duffy

Sr. Digital Producer

Astellas Pharmaceutical

Mia Patterson-Brennan Sr. Marketing Manager

HY Connect

Kate Loichinger

Digital Producer

AstraZeneca

Joe Shields

Global Director, Campaign Management

InTouch Solutions

Wendy Blackburn

EVP

Bad Testing

Shachar Schiff

Principal

InViVo Communications

Julie Saunders

QA Manager

GA Communication Group

Mark Goble

President/COO

Juice Pharma Worldwide

Sara Allen

Interactive Production Supervisor

GA Communication Group

Ben Currie

Group Director, Digital Solutions

Juice Pharma Worldwide

Ted Shimzu

Senior Digital Producer

GA Communication Group

Geoff Melick

EVP, Chief Innovation Officer

PACIFIC Communications

Patrick Macke

VP, Creative Director, Interactive

GA Communication Group

Elizabeth Estes

EVP, Chief Strategy Officer

Phenomblue

Kt McBratney

VP, General Manager

GA Communication Group

Nancy Finigan

EVP, Account Group Director

R/GA

Michael Morowitz

Senior Technology Director

GA Communication Group

Steve Buecking

Executive Producer

Valeritas

Tara Charvat

VP, Marketing and Marketing Operations

GA Communication Group

Barclay Missen

Creative Director/UX

Association for Software Testing Dee Ann Pizzica

Senior Business Analyst

GA Communication Group

Sarah Kaps

Director, Digital Solutions

Unlike Any Other Writer

Richard Verne

President

GA Communication Group

Christina Welsh

Executive Assistant

ClearBalance

Laurie Heavey

Vice President, Marketing

GA Communication Group

Joe Kuchta

CEO

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

27
Special thanks to Dale Taylor and Mark Goble for their sponsorship and support.

Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.

Mais conteúdo relacionado

Mais procurados

Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test AutomationNaveen Kumar Singh
 
What is Sanity Testing? Edureka
What is Sanity Testing? EdurekaWhat is Sanity Testing? Edureka
What is Sanity Testing? EdurekaEdureka!
 
Agile testing principles and practices - Anil Karade
Agile testing principles and practices - Anil KaradeAgile testing principles and practices - Anil Karade
Agile testing principles and practices - Anil KaradeIndicThreads
 
Test Strategy and Planning
Test Strategy and PlanningTest Strategy and Planning
Test Strategy and PlanningSachin-QA
 
Scrum Testing Methodology
Scrum Testing MethodologyScrum Testing Methodology
Scrum Testing MethodologyGaya1985
 
Risk-based Testing
Risk-based TestingRisk-based Testing
Risk-based TestingJohan Hoberg
 
Sap test center of excellence
Sap test center of excellenceSap test center of excellence
Sap test center of excellenceInfosys
 
Agile QA presentation
Agile QA presentationAgile QA presentation
Agile QA presentationCarl Bruiners
 
Shift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBMShift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBMIBM UrbanCode Products
 
Using the test process improvement models. Case study based on TPI Next model...
Using the test process improvement models. Case study based on TPI Next model...Using the test process improvement models. Case study based on TPI Next model...
Using the test process improvement models. Case study based on TPI Next model...Sigma Software
 
Shift left shift-right performance testing for superior end-user by Arun Dutta
Shift left shift-right performance testing for superior end-user by Arun DuttaShift left shift-right performance testing for superior end-user by Arun Dutta
Shift left shift-right performance testing for superior end-user by Arun DuttaSoftware Testing Board
 
Why would you want to improve your test process using TPI or TMMi?
Why would you want to improve your test process using TPI or TMMi?Why would you want to improve your test process using TPI or TMMi?
Why would you want to improve your test process using TPI or TMMi?Rik Marselis
 
UAT Kickoff Presentation 10 29 09
UAT Kickoff Presentation 10 29 09UAT Kickoff Presentation 10 29 09
UAT Kickoff Presentation 10 29 09goozer65
 
What is Shift Left Testing.pdf
What is Shift Left Testing.pdfWhat is Shift Left Testing.pdf
What is Shift Left Testing.pdfTestbytes
 
Agile Manifesto and Principles
Agile Manifesto and PrinciplesAgile Manifesto and Principles
Agile Manifesto and PrinciplesAryan Rajbhandari
 

Mais procurados (20)

Agile Testing and Test Automation
Agile Testing and Test AutomationAgile Testing and Test Automation
Agile Testing and Test Automation
 
QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)QA metrics in Agile (GUIDE)
QA metrics in Agile (GUIDE)
 
What is Sanity Testing? Edureka
What is Sanity Testing? EdurekaWhat is Sanity Testing? Edureka
What is Sanity Testing? Edureka
 
Agile testing principles and practices - Anil Karade
Agile testing principles and practices - Anil KaradeAgile testing principles and practices - Anil Karade
Agile testing principles and practices - Anil Karade
 
Agile Testing by Example
Agile Testing by ExampleAgile Testing by Example
Agile Testing by Example
 
Test Strategy and Planning
Test Strategy and PlanningTest Strategy and Planning
Test Strategy and Planning
 
Scrum Testing Methodology
Scrum Testing MethodologyScrum Testing Methodology
Scrum Testing Methodology
 
Risk-based Testing
Risk-based TestingRisk-based Testing
Risk-based Testing
 
Chapter 1 - Agile Methodology
Chapter 1 - Agile MethodologyChapter 1 - Agile Methodology
Chapter 1 - Agile Methodology
 
Sap test center of excellence
Sap test center of excellenceSap test center of excellence
Sap test center of excellence
 
Agile QA presentation
Agile QA presentationAgile QA presentation
Agile QA presentation
 
Shift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBMShift Left - Approach and practices with IBM
Shift Left - Approach and practices with IBM
 
Using the test process improvement models. Case study based on TPI Next model...
Using the test process improvement models. Case study based on TPI Next model...Using the test process improvement models. Case study based on TPI Next model...
Using the test process improvement models. Case study based on TPI Next model...
 
Shift left shift-right performance testing for superior end-user by Arun Dutta
Shift left shift-right performance testing for superior end-user by Arun DuttaShift left shift-right performance testing for superior end-user by Arun Dutta
Shift left shift-right performance testing for superior end-user by Arun Dutta
 
Why would you want to improve your test process using TPI or TMMi?
Why would you want to improve your test process using TPI or TMMi?Why would you want to improve your test process using TPI or TMMi?
Why would you want to improve your test process using TPI or TMMi?
 
UAT Kickoff Presentation 10 29 09
UAT Kickoff Presentation 10 29 09UAT Kickoff Presentation 10 29 09
UAT Kickoff Presentation 10 29 09
 
What is Shift Left Testing.pdf
What is Shift Left Testing.pdfWhat is Shift Left Testing.pdf
What is Shift Left Testing.pdf
 
Shift left-testing
Shift left-testingShift left-testing
Shift left-testing
 
TestOps and Shift Left
TestOps and Shift LeftTestOps and Shift Left
TestOps and Shift Left
 
Agile Manifesto and Principles
Agile Manifesto and PrinciplesAgile Manifesto and Principles
Agile Manifesto and Principles
 

Destaque

Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.Worksoft
 
10 Steps to Shift Left - Guiding Principles in QA
10 Steps to Shift Left - Guiding Principles in QA10 Steps to Shift Left - Guiding Principles in QA
10 Steps to Shift Left - Guiding Principles in QAJim Spillson
 
UID 2015 Presentation - Online Marketing - Where's it going? How to win.
UID 2015 Presentation - Online Marketing - Where's it going? How to win.UID 2015 Presentation - Online Marketing - Where's it going? How to win.
UID 2015 Presentation - Online Marketing - Where's it going? How to win.Brian Bluff
 
2013 digital trends for healthcare
2013 digital trends for healthcare2013 digital trends for healthcare
2013 digital trends for healthcareGSW
 
Shift Left Mobile Application Testing
Shift Left Mobile Application TestingShift Left Mobile Application Testing
Shift Left Mobile Application TestingJosiah Renaudin
 
Early Alert Powerpoint
Early Alert PowerpointEarly Alert Powerpoint
Early Alert PowerpointCyrus Helf
 
Quality Improvement Sepsis Recognition & Treatment
Quality Improvement Sepsis Recognition & TreatmentQuality Improvement Sepsis Recognition & Treatment
Quality Improvement Sepsis Recognition & TreatmentJoseph Di Genova
 
Shift left - find defects earlier through automated test and deployment
Shift left - find defects earlier through automated test and deploymentShift left - find defects earlier through automated test and deployment
Shift left - find defects earlier through automated test and deploymentClaudia Ring
 
Model Risk Management : Best Practices
Model Risk Management : Best PracticesModel Risk Management : Best Practices
Model Risk Management : Best PracticesQuantUniversity
 
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...QuantUniversity
 
3 Digital Healthcare Trends That We Can Expect to See More of in 2017
3 Digital Healthcare Trends That We Can Expect to See More of in 20173 Digital Healthcare Trends That We Can Expect to See More of in 2017
3 Digital Healthcare Trends That We Can Expect to See More of in 2017Jeremy Barbera
 
Risk Management
Risk ManagementRisk Management
Risk Managementcgeorgeo
 
2012 CNO Investor Day
2012 CNO Investor Day2012 CNO Investor Day
2012 CNO Investor DayCNOServices
 
IS20G New York Mike Udell Day 1 A Powerful Metaphor
IS20G New York Mike Udell Day 1 A Powerful MetaphorIS20G New York Mike Udell Day 1 A Powerful Metaphor
IS20G New York Mike Udell Day 1 A Powerful MetaphorSean Bradley
 
Ocb solutions -_investor_slides_final
Ocb solutions -_investor_slides_finalOcb solutions -_investor_slides_final
Ocb solutions -_investor_slides_finalCNOServices
 
The world at your fingertips
The world at your fingertipsThe world at your fingertips
The world at your fingertipsGraceAng23
 

Destaque (20)

Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.Shift Left Quality Assurance: How to do it. Why it matters.
Shift Left Quality Assurance: How to do it. Why it matters.
 
10 Steps to Shift Left - Guiding Principles in QA
10 Steps to Shift Left - Guiding Principles in QA10 Steps to Shift Left - Guiding Principles in QA
10 Steps to Shift Left - Guiding Principles in QA
 
UID 2015 Presentation - Online Marketing - Where's it going? How to win.
UID 2015 Presentation - Online Marketing - Where's it going? How to win.UID 2015 Presentation - Online Marketing - Where's it going? How to win.
UID 2015 Presentation - Online Marketing - Where's it going? How to win.
 
2013 digital trends for healthcare
2013 digital trends for healthcare2013 digital trends for healthcare
2013 digital trends for healthcare
 
Shift Left Mobile Application Testing
Shift Left Mobile Application TestingShift Left Mobile Application Testing
Shift Left Mobile Application Testing
 
Early Alert Powerpoint
Early Alert PowerpointEarly Alert Powerpoint
Early Alert Powerpoint
 
Quality Improvement Sepsis Recognition & Treatment
Quality Improvement Sepsis Recognition & TreatmentQuality Improvement Sepsis Recognition & Treatment
Quality Improvement Sepsis Recognition & Treatment
 
Shifting Left Webinar Slideshow
Shifting Left Webinar SlideshowShifting Left Webinar Slideshow
Shifting Left Webinar Slideshow
 
Shift left - find defects earlier through automated test and deployment
Shift left - find defects earlier through automated test and deploymentShift left - find defects earlier through automated test and deployment
Shift left - find defects earlier through automated test and deployment
 
Model Risk Management : Best Practices
Model Risk Management : Best PracticesModel Risk Management : Best Practices
Model Risk Management : Best Practices
 
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...
 
Capa
CapaCapa
Capa
 
Risk Management Training
Risk Management TrainingRisk Management Training
Risk Management Training
 
3 Digital Healthcare Trends That We Can Expect to See More of in 2017
3 Digital Healthcare Trends That We Can Expect to See More of in 20173 Digital Healthcare Trends That We Can Expect to See More of in 2017
3 Digital Healthcare Trends That We Can Expect to See More of in 2017
 
Risk Management
Risk ManagementRisk Management
Risk Management
 
CAPA: A Risk Mitigating Quality System
CAPA: A Risk Mitigating Quality SystemCAPA: A Risk Mitigating Quality System
CAPA: A Risk Mitigating Quality System
 
2012 CNO Investor Day
2012 CNO Investor Day2012 CNO Investor Day
2012 CNO Investor Day
 
IS20G New York Mike Udell Day 1 A Powerful Metaphor
IS20G New York Mike Udell Day 1 A Powerful MetaphorIS20G New York Mike Udell Day 1 A Powerful Metaphor
IS20G New York Mike Udell Day 1 A Powerful Metaphor
 
Ocb solutions -_investor_slides_final
Ocb solutions -_investor_slides_finalOcb solutions -_investor_slides_final
Ocb solutions -_investor_slides_final
 
The world at your fingertips
The world at your fingertipsThe world at your fingertips
The world at your fingertips
 

Semelhante a QA Best Practices in Digital Marketing [whitepaper]

InfinityQS_Quality Metrics Survival Handbook final_042413
InfinityQS_Quality Metrics Survival Handbook final_042413InfinityQS_Quality Metrics Survival Handbook final_042413
InfinityQS_Quality Metrics Survival Handbook final_042413Vanessa Stirling
 
The Value PMLC Process Capability
The Value PMLC Process CapabilityThe Value PMLC Process Capability
The Value PMLC Process CapabilityBill Monroe
 
3. introduction to software testing
3. introduction to software testing3. introduction to software testing
3. introduction to software testingChandra Maddigapu
 
Top CEO Challenges 2021
Top CEO Challenges 2021Top CEO Challenges 2021
Top CEO Challenges 2021Jellyfish.tech
 
Principles of effective software quality management
Principles of effective software quality managementPrinciples of effective software quality management
Principles of effective software quality managementNeeraj Tripathi
 
How Project Management Tools Enable Improved Customer Experiences
How Project Management Tools Enable Improved Customer ExperiencesHow Project Management Tools Enable Improved Customer Experiences
How Project Management Tools Enable Improved Customer ExperiencesOrangescrum
 
How to Solve Top Project Management Challenges
How to Solve Top Project Management ChallengesHow to Solve Top Project Management Challenges
How to Solve Top Project Management ChallengesOrangescrum
 
InfinityQS_7 Habits of Quality Obsessed Manufacturers_Print final
InfinityQS_7 Habits of Quality Obsessed Manufacturers_Print finalInfinityQS_7 Habits of Quality Obsessed Manufacturers_Print final
InfinityQS_7 Habits of Quality Obsessed Manufacturers_Print finalVanessa Stirling
 
Building a Strategic Partnership - From Software Developers to Solution Innov...
Building a Strategic Partnership - From Software Developers to Solution Innov...Building a Strategic Partnership - From Software Developers to Solution Innov...
Building a Strategic Partnership - From Software Developers to Solution Innov...Mukesh Lagadhir
 
How Can Quality Assurance Ensure Effective Application Development.
How Can Quality Assurance Ensure Effective Application Development.How Can Quality Assurance Ensure Effective Application Development.
How Can Quality Assurance Ensure Effective Application Development.Techugo
 
Does Your Online Experiences Meet Your Customer Expectations
Does Your Online Experiences Meet Your Customer ExpectationsDoes Your Online Experiences Meet Your Customer Expectations
Does Your Online Experiences Meet Your Customer Expectationsindeuppal
 
The Future of Project Management
The Future of Project ManagementThe Future of Project Management
The Future of Project ManagementOrangescrum
 
Digital products development: going behind the scene of product development -...
Digital products development: going behind the scene of product development -...Digital products development: going behind the scene of product development -...
Digital products development: going behind the scene of product development -...OgilvyOne Worldwide
 
Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010
Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010
Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010TEST Huddle
 
Using Conversation Analytics to better understand Net Promoter Scores
Using Conversation Analytics to better understand Net Promoter Scores Using Conversation Analytics to better understand Net Promoter Scores
Using Conversation Analytics to better understand Net Promoter Scores Call Journey
 
Lehigh Valley Business_Digital Transformation
Lehigh Valley Business_Digital TransformationLehigh Valley Business_Digital Transformation
Lehigh Valley Business_Digital TransformationGene Ferro
 

Semelhante a QA Best Practices in Digital Marketing [whitepaper] (20)

InfinityQS_Quality Metrics Survival Handbook final_042413
InfinityQS_Quality Metrics Survival Handbook final_042413InfinityQS_Quality Metrics Survival Handbook final_042413
InfinityQS_Quality Metrics Survival Handbook final_042413
 
The Value PMLC Process Capability
The Value PMLC Process CapabilityThe Value PMLC Process Capability
The Value PMLC Process Capability
 
3. introduction to software testing
3. introduction to software testing3. introduction to software testing
3. introduction to software testing
 
Top CEO Challenges 2021
Top CEO Challenges 2021Top CEO Challenges 2021
Top CEO Challenges 2021
 
Principles of effective software quality management
Principles of effective software quality managementPrinciples of effective software quality management
Principles of effective software quality management
 
How Project Management Tools Enable Improved Customer Experiences
How Project Management Tools Enable Improved Customer ExperiencesHow Project Management Tools Enable Improved Customer Experiences
How Project Management Tools Enable Improved Customer Experiences
 
How to Solve Top Project Management Challenges
How to Solve Top Project Management ChallengesHow to Solve Top Project Management Challenges
How to Solve Top Project Management Challenges
 
InfinityQS_7 Habits of Quality Obsessed Manufacturers_Print final
InfinityQS_7 Habits of Quality Obsessed Manufacturers_Print finalInfinityQS_7 Habits of Quality Obsessed Manufacturers_Print final
InfinityQS_7 Habits of Quality Obsessed Manufacturers_Print final
 
Agile challenges
Agile challengesAgile challenges
Agile challenges
 
Building a Strategic Partnership - From Software Developers to Solution Innov...
Building a Strategic Partnership - From Software Developers to Solution Innov...Building a Strategic Partnership - From Software Developers to Solution Innov...
Building a Strategic Partnership - From Software Developers to Solution Innov...
 
07.pdf
07.pdf07.pdf
07.pdf
 
How Can Quality Assurance Ensure Effective Application Development.
How Can Quality Assurance Ensure Effective Application Development.How Can Quality Assurance Ensure Effective Application Development.
How Can Quality Assurance Ensure Effective Application Development.
 
Does Your Online Experiences Meet Your Customer Expectations
Does Your Online Experiences Meet Your Customer ExpectationsDoes Your Online Experiences Meet Your Customer Expectations
Does Your Online Experiences Meet Your Customer Expectations
 
The Future of Project Management
The Future of Project ManagementThe Future of Project Management
The Future of Project Management
 
Digital products development: going behind the scene of product development -...
Digital products development: going behind the scene of product development -...Digital products development: going behind the scene of product development -...
Digital products development: going behind the scene of product development -...
 
Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010
Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010
Bob Galen - Differentiating Ourselves & Demonstrating Value - EuroSTAR 2010
 
Using Conversation Analytics to better understand Net Promoter Scores
Using Conversation Analytics to better understand Net Promoter Scores Using Conversation Analytics to better understand Net Promoter Scores
Using Conversation Analytics to better understand Net Promoter Scores
 
Lehigh Valley Business_Digital Transformation
Lehigh Valley Business_Digital TransformationLehigh Valley Business_Digital Transformation
Lehigh Valley Business_Digital Transformation
 
2014_HR_Technology_Trends
2014_HR_Technology_Trends2014_HR_Technology_Trends
2014_HR_Technology_Trends
 
Seo+for+agencies+2014
Seo+for+agencies+2014Seo+for+agencies+2014
Seo+for+agencies+2014
 

Último

Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CVKhem
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024SynarionITSolutions
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdflior mazor
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 

Último (20)

Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024Top 10 Most Downloaded Games on Play Store in 2024
Top 10 Most Downloaded Games on Play Store in 2024
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 

QA Best Practices in Digital Marketing [whitepaper]

  • 1. SHIFT LEFT Quality Assurance Best Practices in Digital Marketing
  • 2. You cannot inspect quality into a product. The quality is there or it isn’t by the time it’s inspected. Harold S. Dodge, via W. Edwards Deming Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 2
  • 3. Quality Assurance: We Struck a Nerve Last October in Chicago, AbelsonTaylor and GA Communication Group co-hosted the first ever Quality Assurance Summit for digital marketing. The original thought was to get 10 guys in a room to talk about the issues that affect the quality of digital projects. Well, we certainly struck a nerve with this topic. It turns out that everyone we spoke to has the same problems. Presenters included Jim Spillson, Manager of Quality Assurance, AbelsonTaylor, as well as Elizabeth Estes EVP, Chief Strategy Officer, Ben Currie, Group Director, Digital Solutions, GA Communication Group, and Shachar Schiff, Principal, Bad Testing. Michael Morowitz, Senior Technology Director, R/GA Chicago was the keynote speaker. Join the QA Conversation So, we ended up with over 50 marketing directors, QA professionals and digital directors from the leading healthcare companies and advertising agencies meeting all day at the Ritz-Carlton to try to figure out some best practices. This white paper is the product of a lot of blood, sweat, tears and laughter. But it’s clearly not the end of the conversation. Please join us at stateofqa.com and through @StateofQA to help refine the best practices in quality assurance. The Goal: Take a Holistic View of QA The QA Summit was moderated by Joe Shields, Global Director, Campaign Management, AstraZeneca and included a presentation on the state of QA, an expert panel discussion and a client roundtable, as well as a group working session. A Word of Thanks Finally, we’d like to thank everyone who took time out of their busy schedules to make this summit a reality. QA is in a much better place due to your passion and honest insights. Joe Shields Jay Carter Geoff Melick Jim Spillson Ben Currie Michael Morowitz Scott Hansen Global Director, Campaign Management, AstraZeneca SVP, Director of Strategy Services, AbelsonTaylor EVP, Chief Innovation Officer, GA Communication Group Manager of Quality Assurance, AbelsonTaylor Group Director, Digital Solutions, GA Communication Group Senior Technology Director, R/GA Chicago VP, Executive Director, Digital Creative, AbelsonTaylor Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 3
  • 4. Glitch Happens Need proof that quality assurance in digital healthcare marketing has hit a nerve? Just take a look back at HealthCare.gov—the digital marketplace that launched, notably on schedule, only to become the most publicized and preventable software failure. The site is intended to improve healthcare access, efficiency and quality. Instead, it’s become a public service announcement on the state of quality assurance. 156 Weeks of Planning—And Only 2 Weeks of Testing at the End The scope of HealthCare.gov was massive. The launch was highly anticipated. And as time to market approached, a disparity between project requirements, documentation and expectations continued to grow. Cross-functional teams spanning various government agencies and 55 contractors took 3 years (or 156 weeks) to concept, plan and develop the software. But proper testing didn’t occur until 2 weeks prior to launch. From a QA perspective, the project was dead on arrival. We’re Not Just Testing Tech Unfortunately, this scenario is all too familiar to digital healthcare marketers: the complex requirements, the lack of documentation, the nebulous stakeholders and responsibilities, the silos of communication, the late-breaking scope changes, the immovable deadline. Quality assurance starts way too late in the process—like at the end. To manage the risks inherent in digital projects, it’s critical that quality be an intrinsic part of the process. It has to be an attribute of the process, not an afterthought. Quality is the means and the end. And every team, every individual must have quality procedures built into what they do. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. Failing to plan is planning to fail. Alan Lakein 4
  • 5. Quality Assurance— It’s Hard to Keep Up Quality assurance is not just finding bugs in the code. It’s the impact of flaws in direction— or misdirection if you will—that cause work to be scrapped or redone late in the process. Testing is not assurance. Quality assurance is keeping up with changes in hardware and software. In the last year, over 30 different digital devices, platforms, operating systems and browsers were released. Michael Morowitz, Senior Technology Director, R/GA Chicago Quality assurance is staying current with how your target consumes digital media. For instance, a recent study of physician behavior by Epocrates found that there would be more digital doctors than non-digital doctors in 2014. That means that in the 2-year period between 2012 and 2014, there will be a complete flip in the ratio of connected clinicians from 20:80 to 80:20. At the same time, something called an E-patient has appeared on the horizon, empowered by apps and social media, who is having his or her voice heard by healthcare companies. When this accelerating velocity of change crashes into the typical development life cycle of digital projects—6 months to a year—one can clearly see why quality assurance struggles to keep pace. “Mobile Trends Report,” Epocrates, 2013. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 5
  • 6. Looking for Problems in the Right Places Research has shown that 56% of all digital project problems occur during the requirements phase of a project. 27% occurs during the design phase. Meaning that the majority of bugs happen before programming even starts. However, the typical QA process doesn’t even begin until after the programming portion of the project is complete. Where problems originate That means that bugs that start early on in a project just sit there, costing time and money. That’s the trap If you don’t know what you’re looking for, or even when or where you should be looking, why bother with quality at all? This leads to the common practice of just “looking at it before it goes out the door.” Without clearly defined quality assurance principles, processes, goals and metrics up-front, it’s easy to see why everyone falls in the trap of only testing at the end. Requirements Design Code Other 56% 27% 7% 10% CrossTalk Magazine—The Journal of Defense Software Engineering Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 6
  • 7. The Cost of Quality Quality begins with the strategy. It includes customer considerations, on-target messaging, intuitive interface design, product branding and airtight code. The earlier you find a problem, the less it will cost to fix. If we find an issue in the requirements phase, the cost to fix is measured in the time it takes to edit a requirements or specifications document. Problems found at the end cost 10 times more to fix than problems found in the beginning. Jim Spillson, Manager of Quality Assurance, AbelsonTaylor If we find that same problem during the testing phase—requiring new content, new functionality, new code, new compatibility testing—it can easily cost 10 times as much to fix, maybe more. The cost of quality isn’t the price of creating quality products or services. It’s the cost of NOT creating quality products or services. Every time work is redone, the cost of quality increases. Time detected Cost to fix a defect Requirements Development Testing Post-release Requirements 1× 3× 5–10× 10× 10–100× Doc/Creative – 1× 10× 15× 25–100× Development Time introduced Doc/Creative – – 1× 10× 10–25× Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 7
  • 8. QA Is More Than Tech and Code In the end, quality assurance is more than checking for glitches in the code. Quality assurance is protecting the value of your brand. Every mistake that goes live leads to a decreased user experience by your target because it affects load time, how email replies are handled and how easy or difficult forms are to fill out. In other words, defects erode the relationship your target has with your brand. Quality is not somebody’s department. It’s not a made-up job. It’s the cost of entry in business. Joe Shields, Global Director, Campaign Management, AstraZeneca Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 8
  • 9. Shift Left: Quality Assurance From the Start The concept of Shift Left is more than an adjustment to a timeline. It’s more than finding glitches in the code. communications, better interface design, and projects that benefit both our clients and customers. Shift Left operationalizes the idea that quality assurance starts at the beginning, during the planning stage of a project. And that quality assurance is present during every subsequent step of the way. The Benefits of Shift Left The goal of Shift Left is to be more nimble, accurate and inventive when creating digital projects. If we’re looking for defects up-front, we can catch them and move on. That’s nimble. If we catch defects as they occur, we improve accuracy. Finally, if we’re both nimble and accurate, we allow ourselves the opportunity to be more inventive—creating better Number of Defects Job Opens Requirements Design Development The benefit of Shift Left is that we can now set and meet expectations for successful quality assurance across every browser, every operating system and every device. And for every brand. And we can be efficient because doing it right, at the right time, is faster than finding and fixing the problems just days before launch. Test Release Shift Left Shift Left Life Cycle Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. Early Detection of Defects Project Life Cycle 9
  • 10. 5 Ways QA Challenges Clients and Agencies It all starts with the fact that we don’t have a consensusdriven definition of what quality means. Talk to an account person, a creative, a technologist, a brand manager—they all have different definitions of quality. For the software engineer, quality can be measured by defect density. For the creative, it’s that the interface looks and functions properly. For the brand manager, quality revolves around messaging, timeline and budget. For the customer, it’s all about having a painless experience with your digital asset. These disparate views of quality assurance put the project and participants at risk: budgetary risk, timeline risk, client expectation risk and customer satisfaction risk. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 1 Creative Client Service Focus Clients don’t approach agencies for technology. Clients are looking for strategy, service and creative. So, the first challenge is that the people we talk to on the client side are not usually technologists. They are CMOs. They are marketing directors. They are brand managers. What does that mean to the concept of quality assurance? It means that no agency is going to win business solely based on QA. Conversely, there is a real chance that an agency will lose business because of bad QA. Clients expect perfect. When we miss perfect, we put the business at risk. 10
  • 11. 5 Key QA Challenges 2 Custom Technology As brand champions, agencies find themselves between a rock and a hard place when it comes to custom technology. Differentiation is key to the success of any brand. That differentiation extends to technology. Custom technology is literally in an agency’s charter and DNA. So, we have to consider the cost and liability of built from scratch. When it’s custom, we end up being the sole defender of the technology. We’re responsible for ensuring that each and every browser and OS update doesn’t break our tech. 3 Varied Schedules It’s challenging when you are trying to fit test planning, process management, software testing and non-standard deliverables into schedules that last anywhere from 2 weeks to 18 months. The challenge is compounded when you add in sales cycles, medical/regulatory/legal reviews and time-sensitive FDA mandates. As a result, custom technology places a huge burden on the quality assurance process. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 11
  • 12. 5 Key QA Challenges 4 Non-Standard Deliverables There are situations when we have no technology deliverables. For instance, there are occasions when we deliver creative—in the form of PSDs—to a client IT team, who then builds the site. So the question is, if technology is not involved, and QA is a function of the technology team, is the client not getting quality assurance? Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 5 Wide Range of Platforms Companies like AstraZeneca, Capital One or Nike have an infrastructure, and they have opinions, and they know where they want the technology to go. Almost every agency feels the pain of having to operate under and across a broad range of platforms. 12
  • 13. A clear definition of quality goals leads directly to a clear definition of QA principles. Michael Morowitz, Senior Technology Director, R/GA Chicago Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 13
  • 14. 4 Core Principles of Quality Assurance 1 Complete Process Coverage Testing is important, and testing at the end is really important. But it’s also just one tool in our tool belt. Requirements and standards analyses, creative reviews and test planning are all critical-to-quality moments that need to come into play as we Shift Left. Testing Test Planning Requirements Analysis Project Timeline Testing Typical testing paradigm Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. Standards Analysis Testing Testing Creative Reviews Testing Testing Testing Project Timeline How Shift Left affects testing 14
  • 15. 4 Core Principles of QA 2 Holistic Knowledge Strategy & Analytics If you aren’t seeing the big picture, you aren’t providing quality assurance. That’s why each team member must be aware of the roles and responsibilities of their teammates. Each team member must understand how the team will work to bring all the pieces together, and how they will monitor quality, in order to ensure success. Holistic knowledge of the project and collaboration is the glue that prevents the types of mistakes that happen when team members work in silos. That’s when problems get sent downstream for someone else to figure out and when costs start climbing. Copy Social & Mobile iXD Interface Tech Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 15
  • 16. 4 Core Principles of QA 3 Independent Accountability This principle is specifically about the agency’s org chart. It states that the QA lead should, ideally, report to a management structure that is not shared with any other production groups. More specifically, a QA analyst/tester should NOT report to the Technology Director, as is common practice in many agencies. Independence ensures a greater “purity of practice” of quality assurance. The primary team members who are accountable for quality assurance are themselves accountable to a team structure that has quality assurance as its primary mission. 4 Team Focus The idea of Team Focus is that quality assurance lives everywhere. Everyone’s responsible. While Team Focus is an extension of Independent Accountability, it’s different. Team Focus states that even though there is an independently accountable QA lead, ALL team members are responsible for quality. QA is not fully contained within the QA lead. Every functional and every non-functional area should have a role within the QA process. Therefore, it makes sense for QA analysts and testers to report to an Executive Director of QA. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 16
  • 17. The ‘cost of quality’ isn’t the price of creating a quality product or service. It’s the cost of NOT creating a quality product or service. Every time work is redone, the cost of quality increases. American Society for Quality, ASQ.org Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 17
  • 18. The Shift Left Model— Outcomes of the QA Summit To make a healthcare analogy, the Shift Left model of quality assurance is a lot like preventive care. You anticipate problems and head them off before any real damage is done. Testing at the end of a project, on the other hand, is similar to going to the emergency room. Aside from it costing you an arm and a leg, emergency room visits usually happen in a shroud of panic. Judgment is replaced by reactivity. Build things that aren’t broken in the first place. Geoff Melick, EVP, Chief Innovation Officer, GA Communication Group Shift Left is More Than an Action Step. It’s a Mind-set. There is a gap in understanding around the role of QA. QA is often confused with functional testing. However, QA is more. It’s getting detailed requirements, it’s clearly identifying stakeholders and responsibilities, it’s having a system for effective communication, it’s a proper process of scope change and it’s effective management of expectations. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 18
  • 19. 10 Practical Guiding Principles of Shift Left During the course of the conversation, it became clear that there are 3 key areas of focus inside the idea of Shift Left: 1) Planning and Communication, 2) Concept Creation and 3) Execution. The following 10 Practical Steps are intended to foster a strong, from beginning-to-end mind-set around QA that you can apply to your current set of processes. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. Shift Left allows you to be more nimble, accurate and inventive. Scott Hansen, VP, Executive Director, Digital Creative, AbelsonTaylor 19
  • 20. Planning and Communication Here everyone involved must be made aware of the project plan and understand his or her role in making it happen. Be certain to find and use common language. It will go a long way in assuring that stakeholders truly understand what’s going to happen. 1 Identify Key Stakeholders Everyone involved in the creation of a digital project—both client and agency—is a stakeholder. of introducing oneself changed the behavior and the dynamics of responsibility in the room. And nothing is more important than identifying those key decision makers and their roles, both at the agency and at the client. Having the right people in the right place at the right time is a key step in creating an empowered and accountable team. When someone stood up and said, “My name is Michael Smith and I am the anesthesiologist,” or “My name is Betty Jones and I am the head nurse,” it changed the way the team functioned. Lines of responsibility became clear to the team members. In addition, team members became empowered to speak up and identify issues. It’s also critical that all of the stakeholders discuss what they’re doing on the project. In the book The Checklist Manifesto, How to Get Things Right, one of the items on Dr. Atul Gawande’s pre-surgery checklist is for everyone in the operating room to introduce themselves and their role. What Dr. Gawande found was that the simple activity Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. Avoid the typical agency dynamic where you walk into a project kickoff and everyone just starts talking. Roles are assumed. “Oh, there’s the tech guy, he must be in charge of QA.” This type of thought process has disaster written all over it. 20
  • 21. Planning and Communication 2 Share the Plan Everyone hates surprises. That’s why it’s so important to share the plan—strategy, messaging, media, business requirements documents, functional requirements documents, design, metrics, test plan—with everyone up-front. And gain approval before moving on. Each and every stakeholder must have a clear understanding of where “true north” is for the project. This includes agency account, creative and tech, as well as client brand managers, IT and sales management. QA includes understanding what the customer wants, their channel preferences and delivering the right message. Matt Barry, Associate Director, HCP Multichannel Marketing, Astellas Pharma US Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 3 Address Barriers to Success Take the time to course correct before you’ve spent a dime. If corporate direction, scope, legal requirements or brand imperatives change, now is the time to address any updates to the plan. It may not just be changes that are the barriers. The timeline, an unknown platform or a team that has never worked together could all be potential pitfalls. Similarly, language barriers can betray project success. A team may share the same native tongue, but few are fluent in both the languages of marketing and technology. Critical success factors means something to marketing folks, whereas critical-to-quality moments means something to the tech folks. Establishing a common lexicon for strategic goals breaks down the walls between team members, allowing them to clearly define and effectively communicate shared goals and commitment to success. Project planning should come to a “dead stop” until you’ve had time to discuss what steps the team will collectively take to understand and address the barriers to success. 21
  • 22. Concept Creation It’s critical that stakeholders understand what the project is trying to accomplish and how. Beware of vague head nods when you present wireframes. Take the extra steps necessary to actually show how your program will perform before you invest a lot of time and money. 4 Don’t Reinvent the Wheel Don’t fall into the custom-made trap. Not every project requires a novel operating system or unique piece of hardware. Nor should the client be forced to pay for it either. This doesn’t mean you shouldn’t be inventive. Many times, your client will require a custom solution to their problems. It does mean, however, that if a platform solution such as Adobe DPS or Drupal can solve your problem, then don’t build your own platform. Using pre-existing resources can reduce the liability of having to test both the platform and the project. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 5 Proof of Concept If a picture is worth a thousand words, how much is a Proof of Concept worth? Is it priceless? A Proof of Concept does 2 things: 1) it gives you the ability to show a client an early example of how a project will look and perform. This always goes a long way to eliminating confusion as to what the end product will actually be; and 2) it allows you to solve sticky tech issues before the entire team starts burning hours. Even if you need multiple Proofs of Concept before you gain stakeholder alignment, you haven’t incurred the expense of total front-end and back-end development only to find out that what’s been built isn’t what the client wants or expects. It also provides the opportunity to determine if and how the product can be tested. 22
  • 23. Concept Creation 6 Match Execution to Strategy Due to the complexity of digital projects, oftentimes there is such an intense focus on the technological aspects that strategy ends up taking a back seat. Decisions made during the project can dilute the original intention. It’s as if the strategic mission has been communicated via a game of telephone. A key to quality assurance is to make certain that as adjustments are made to the functional requirements, they don’t disrupt the strategy of the overall project. 61% of users have better opinions of brands that offer a good mobile experience. That makes mobile crucial to Shift Left. Shachar Schiff, Principal, Bad Testing Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 23
  • 24. Execution Here quality assurance is dependent on the discipline of the team to stick to, and not deviate from the plan. 7 Don’t Overengineer Consider making a minimal viable product. There’s nothing wrong with having a master plan for a project. After all, it’s digital. This doesn’t mean you have to start small. It does mean you have to start reasonable. Make sure that what you’re building fits the available timeline and budget. Look to a second, third or fourth release to increase content and functionality. This will go a long way toward keeping your asset intuitive and relevant to your customers. Starting with a minimally viable product allows you to be quicker to market. Tighter release schedules provide tighter customer feedback loops, offering you the ability to collect customer intelligence that leads to actionable insights that can impact strategy and content of future rollouts. 8 Don’t Build Until Everyone Is Onboard Measure twice and cut once. Consider making a “Digital Proof” of your project. A Digital Proof is a fully functioning sliver of the full project. A good rule of thumb is to build the home page and one vertical of the content architecture. As with picture lock in broadcast or a print proof, once the Digital Proof is approved, it allows you to say, “Okay, now we’re going to spend the big money finishing the project.” By taking this step of sharing with stakeholders exactly how the project will look and feel, you eliminate building the complete piece without interim approvals. Finally, if the ask is for an economy car, don’t build the luxury version to satisfy some misguided sense of “doing my best.” Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 24
  • 25. Execution 9 Don’t Change Your Mind Change is the only constant. Changes in the competitive environment, shifts in customer tastes and brand updates all need to be vetted by the proper client and agency stakeholders. Everyone wants to avoid shooting from the hip and agreeing to something that can’t be accomplished. To that end, it’s critical to have a change order process in place. And to understand how change will impact timeline and budget. The concept of Shift Left provides ample opportunity to fine-tune and adjust projects every step of the way. Hopefully, once a project reaches the Execution phase, direction is set and it’s all smooth sailing. While that may be wishful thinking, it’s important to have a process in place to articulate the impact of change. 10 Maintenance Customers are asking for digital platforms that last for a very long time. For instance, if you have type 1 diabetes, you don’t have it for 18 months, you don’t have it for the 6 months that you are interested in diabetes—you have it for your entire life. This has led to a change in the agency-client relationship. Where once an agency would hand over disks and walk away from a project, now the 2 groups are much more connected over a life cycle of browser and OS updates, as well as the ever-changing behavior of our target audiences in regard to technology. Maintaining the integrity of a digital project over time is critical-to-quality factor. Digital assets typically have a much longer shelf life than broadcast or print. Digital also tends to remain live long after the original client stakeholder has been promoted to another job. Therefore, it’s critical to create a maintenance plan that offers a clear road map and direction to anyone who inherits the project. This includes content updates and schedule, as well as budget. It also includes periodic assessments of newly released browsers, OSs and devices. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 25
  • 26. Quality: The Cost of Entry A final thought: quality is not somebody’s department, it’s not a made-up job and it’s not another gatekeeper. Quality doesn’t happen at the end of a project. In reality, quality is the cost of entry. Shift Left means we have to have the mind-set to catch quality when we are writing a project brief, when we are developing strategy, during the creative process and well before the project goes to functional testing. Shift Left is more than an action step. It’s a mind-set. Ben Currie, Group Director, Digital Solutions, GA Communication Group Handshakes, Not Handoffs Therefore, it is incumbent upon agencies to include the development team from the beginning. It’s incumbent upon clients to admit that, “We don’t know what we don’t know.” That’s part of the issue: trying to admit and saying, “I don’t understand what you are saying,” and asking, “Why is that a big deal?” We need to be more honest with what we don’t know; we need to be more patient when the team comes back with answers that make our heads want to blow off. If we are able to shift quality assurance left in the timeline, we can save a lot of angst as well as a lot of money. Quality Is an Ongoing Conversation While the 2013 QA Summit was the first, it clearly isn’t the end of the conversation. There are so many aspects of quality that need further investigation and discussion. Our goal is to continue the conversation on stateofqa.com and through @StateofQA. And with your help, we’ll continue to refine the best practices in quality assurance. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 26
  • 27. Participants: Company Contact name Title Company Contact name Title AbelsonTaylor Dale Taylor President, CEO GA Communication Group Jody Cahill SVP, Director of Agency Operations AbelsonTaylor Jay Carter SVP, Director of Strategy Services GA Communication Group Leah Fergus Project Manager AbelsonTaylor Jim Spillson Manager of Quality Assurance GA Communication Group Kate Shudy Senior Project Manager AbelsonTaylor Scott Hansen VP, Executive Director Digital Creative CloserLook David Ormesher CEO AbelsonTaylor Stephen Neale SVP, Executive Creative Director CloserLook Patrick Conlon Director of QA and BA AbelsonTaylor Brendan Turner Director of Digital Production Corbett Accel Healthcare Group Dennis Hoppe Chief Creative Officer AbelsonTaylor Adam Clark Associate Director of Interactive Technology Corbett Accel Healthcare Group Chris Seda Group Creative Director AbelsonTaylor Jose Andrade VP, Director of Interactive Technology Discovery USA Jeff Pazen SVP, Digital Strategy AbelsonTaylor Jeff Berg SVP, Director of Client Services Discovery USA Colin Gildea Director of Interactive Development AbbVie Nancy Replogle IT Director, eBusiness, Finance & Marketing DraftFCB Carl Arndt Lead Quality Assurance AbbVie Eruch Rustomji Sr. Manager, Application Technology Greater Than One James Askew Director of QA AbbVie Jaskarn Singh Sr. Manager, CRM Database Strategy Greater Than One Caren Estesen Senior QA Astellas Pharmaceutical Matt Barry Associate Director, HCP Multichannel Marketing HY Connect Larry Farkos VP, Director of Operations Astellas Pharmaceutical Richard Grueter Sr. Manager Strategic Sourcing HY Connect Ryan Duffy Sr. Digital Producer Astellas Pharmaceutical Mia Patterson-Brennan Sr. Marketing Manager HY Connect Kate Loichinger Digital Producer AstraZeneca Joe Shields Global Director, Campaign Management InTouch Solutions Wendy Blackburn EVP Bad Testing Shachar Schiff Principal InViVo Communications Julie Saunders QA Manager GA Communication Group Mark Goble President/COO Juice Pharma Worldwide Sara Allen Interactive Production Supervisor GA Communication Group Ben Currie Group Director, Digital Solutions Juice Pharma Worldwide Ted Shimzu Senior Digital Producer GA Communication Group Geoff Melick EVP, Chief Innovation Officer PACIFIC Communications Patrick Macke VP, Creative Director, Interactive GA Communication Group Elizabeth Estes EVP, Chief Strategy Officer Phenomblue Kt McBratney VP, General Manager GA Communication Group Nancy Finigan EVP, Account Group Director R/GA Michael Morowitz Senior Technology Director GA Communication Group Steve Buecking Executive Producer Valeritas Tara Charvat VP, Marketing and Marketing Operations GA Communication Group Barclay Missen Creative Director/UX Association for Software Testing Dee Ann Pizzica Senior Business Analyst GA Communication Group Sarah Kaps Director, Digital Solutions Unlike Any Other Writer Richard Verne President GA Communication Group Christina Welsh Executive Assistant ClearBalance Laurie Heavey Vice President, Marketing GA Communication Group Joe Kuchta CEO Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved. 27
  • 28. Special thanks to Dale Taylor and Mark Goble for their sponsorship and support. Copyright © 2014 by AbelsonTaylor and GA Communication Group. All rights reserved.