KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
UR on the Cheap December 2012
1. User Research on the
Cheap
The views Jen McGinn Chris LaRoche,
expressed in this
User Researcher Usability Consultant
presentation are
my own and do Oracle MIT
not necessarily
reflect the views
of Oracle
2. Agenda
๏ Recruiting & incentives on the cheap
๏ Inexpensive methods
๏ Agile testing on the cheap
๏ Tools and technologies for under $100
๏ Mobile testing on the cheap
๏ Q &A
3. Camera
Mirror
Test system
Microphone Connections to
control room
Speaker
phone
4. PiP
Monitor
PiP
Monitor Recording Sound
system mixer
Microphone Camera
controller
5. What other costs are there?
๏ Recruiting
- Recruiters charge ~$150 per participant + an
administrative fee for each study
๏ Incentives
- $100 - $150 per participant
๏ Time
- As an employee
- As a consultant
- As a function of whether or not the
stakeholders can take action on your results
7. Recruiting
๏ Your existing customers
- Get contacts from sales or support
- Licensee databases
๏ Friends & family recruit
- Send an email to your company alias
- Include dates, general recruiting criteria,
incentives, and description of the activity
๏ Employees
- Same process as friends & family
8. Recruiting: Creating a DB
๏ User group meetings or conferences
- By industry or product
- Set up a table
- Get permission, have them announce you
- Hand out chocolates or branded swag
- Gather contact info or business cards from
people who are willing to opt into future
research
๏ Link on the company home page, in the
forums, or at the end of the installer flow
- Have them complete a screening survey
9. Considerations for Recruiting
๏ Conflict with other departments or orgs
๏ Do not contact lists
๏ Legal responsibilities
- Massachusetts 201 CMR 17.00
- Protection & storage
- Maintaining and updating personal
information
10. Incentives
๏ Customers
- Often not allowed to offer $$
- Company-branded pens, caps, water bottles,
etc. under $20 if not government employees
- $5 coffee cards
๏ Employees
- Will often opt in for no incentive
- Company-branded swag
- Lunch on us
- $25 gift cards
11. Incentives
๏ Depends on the (in)convenience of the
research
๏ Ways to drive down the inconvenience
(and therefore cost of the incentive)
- Go to them (eg. Engineers at work)
- Go to them (eg. Social site users at home)
- Turn it into a phone call
- Remote meeting/test/feedback session using
screen sharing software
12. Methods: Competitive Analysis
๏ Compare features or tasks across
products
- Great for background data
- No users required
- Uncovers mental models, current experience
- Great if competitors exist
- Great if those products are obtainable
- Great if the competitors are going after the
same customers or business space
13. Methods: Data Mining
๏ Customer support data (bugs filed) &
helpdesk (live)
- Great if you can get access to it
- Great if it’s being collected & you know
where to look for it
๏ Usage (click stream) data
- Great if you are allowed to gather usage data
- Great if you can install code on the website
- Great if data is complete enough that you
can draw conclusions from it
14. Methods: Surveys
๏ Data Collection
- Profile: when you want to get a snapshot of
the composition of your current user
population
- Satisfaction: when you want to find out what
works well and what doesn’t
- Value: when you want to know what people
find important
๏ Click stream analysis
- Adding hidden code to the survey can turn it
into an unmoderated testing tool: Demo
http://www.slideshare.net/Andrea_Evans/andrea-evans-upa-
2012-ignite
15. Methods: Interviews/Site Visit
๏ In person
- Great if you are within driving distance
- Gives you rich, contextual data
- Can view them in their environment, see their
performance support tools
- Recommendation: pair up with a colleague
๏ On the phone
- Great for quick scheduling and execution
- Great for geographically distributed
participants
16. Methods: Remote Testing
๏ Moderated remote testing
- Just like testing in the lab
- Using an online meeting and conference call
- Any length
๏ Unmoderated remote testing
- No facilitator
- Participants screen themselves
- Participants can be instructed to think aloud
- Usually tasks under 30 minutes total
- Fast! Same day results
17. Methods: Agile Testing
๏ RITE-Krug combines 2 “discount” test methods
- RITE = Rapid Iterative Test Environment
- Krug = Steve Krug’s approach in his books
๏ Formative: testing designs or code about to
change
๏ Representatives from PM, Dev & Design are
required to attend all sessions + debrief
๏ 3 – 4 participants within 24 hours
๏ Prototypes can change between participants
๏ Debrief happens ASAP after last participant
๏ No formal report, just observations & action
items that stakeholders will take as a result
Submitted for publication, Copyright 2012 Oracle
18. Usability Testing Lab Costs
๏ Facilities (participant room, control room,
giant wall of mirror in between)
๏ Recording hardware (camera and mic)
๏ Web-conferencing software
๏ Recording software
๏ Observation system (computer, optional
second monitor)
๏ Conference call
19. Portable Lab
Speaker
Microphone Test system phone
Camera
20. Conference room /observation room
Web conferencing
software
Recording
software
Conference
call
Second monitor
for observers
39. The views
expressed in this
Thank you!
presentation are
Jen McGinn
my own and do
not necessarily jen dot mcginn at oracle dot com
reflect the views Chris LaRoche
of Oracle
c dot laroche at neu dot edu
Notas do Editor
So here we are in the typical tricked out usability lab. We’ve got a dedicated room with a mirrored window, a dedicated test system, speakerphone, camera, and microphone all hard-wired to an observation room next door.
As an employee, faster studies & results mean that you can conduct more studies. As a consultant, more studies can == more $$. As a f() of stakeholders’ abilities to implement your recommendations, that comes down to how effective you are as a practitioner – what value are you providing? So sometimes cheap == fast.
In the 1990’s, the business joke was you could “pick two” Good and Fast (but not cheap), cheap and fast (but not good). We’ll show you tools and methods so you can have all three.
If you only need to share your screen with the one system in conference room where your observers will be sitting, you can use Skype. It ‘s free, includes VoIP calling, and works on Mac, PC and mobile platforms.
And if you need to share your screen with multiple observers, you can try join.me. Like Skype, it can provide a VoIP phone number, but allows you to share your screen with up to 250 people. Free. And it works on PC, Mac, and Mobile platforms.
But for the Skype call to be free, everyone has to have a Skype-enabled device, like a computer or smart phone. What if they just have phones? Not a problem. You can set up a reservationless conference with free conference call dot com.
The observers can hear over the phone , but I recommend adding a webcam to your test system, to improve the sound quality of your recordings. You can get one for around $30. Because it’s likely your laptop mic isn’t great.
Now what about a note-taker? If you can’t afford to hire one, and you can’t afford one of those super-cool pens that records conversations while you write on special paper, there are options.
Say you already have an iPad – you can purchase a stylus for between $2 and $30. Live Note is an app that records while you write for $6. The notes play back as the same time as the audio, so you can hear what was being said when you wrote them.
Evernote is free, works on the Mac, PC and mobile platforms, and your notes get stored in the cloud. It will record the audio as you write – it just won’t be synchronized with the text as it plays back. That’s what the $6 for LiveNote gets you.
Rather than having a dedicated lab, you can get the same high quality results of a dedicated lab at a much lower cost and greater portability, so you can take the lab to where your participants work and live, or even conduct research from the comfort of your home.
http://www.reflectorapp.com/
Now, I have to be honest -- I do love the ease and convenience of having a permanent usability lab. But if you’re an independent or if your company doesn’t have the budget for a traditional usability lab, you probably have budget for these options.
Now, I have to be honest -- I do love the ease and convenience of having a permanent usability lab. But if you’re an independent or if your company doesn’t have the budget for a traditional usability lab, you probably have budget for these options.