1. An innovative use for Kinect as tracking
consumer
New idea creates a different utility proposition
Nikkei Business Online
January 18, 2012
Satoe Kuwahara, Nobuyuki Yajima
Satoe Kuwahara
Sapporo Sparkle k.k.

2. An innovative use for Kinect as tracking consumer —- New idea creates a different utility proposition
An innovative use for Kinect as tracking consumer
New idea creates a different utility proposition
Nobuyuki Yajima (Research associate of Nikkei Visionary Management Institute):
G
" reat grand design"is a project that looks at the grand design with Ms. Kuwahara who has been de-
signing information systems for long time(main purpose of this project is discussed in " aking money
m
with great idea". So what is today's topic?
)
Satoe Kuwahara(System coordinator in Sapporo Sparkle K.K.):
Today, we will consider the idea that makes the best use of technology through an example that con-
sumer IT products create added value.
First of all I pick up an application that tracks the is not a big deal you can put it wherever you like),
shopper's location in the store and analyzes the this recognition software tracks customer's move-
shopping traffic lines of individual consumer by ment and behavior and allows you to acquire data
using Microsoft's Kinect(reference article: Shopper you need.
Tracker's Kinect hack is like Google analytics for This application analyzes the data acquired by Ki-
retail store shelves). nect and indicates which shelves in the store are
attracting shoppers and which products they touch
or take.
Motion sensing input device for the
Yajima:
Xbox 360 video game console analyses So Kinect can be used for a purpose other than a
the shopper's movement. game controller.
Yajima: Kuwahara:
Shopper Tracker is a new product developed by Yes, it's true and Microsoft has already released
Argentinean developer Agile Route built off of Mi- Kinect software development kit for developers. It
crosoft's Kinect. Kinect is famously known for be- will allows the software developers to create new
ing a completely controller-less experience. uses for motion-based system beyond gaming such
as industrial robots.
Kuwahara:
In order to analyze customer's movement, Shopper Yajima:
Tracker uses pattern recognition technology includ- User can control and interact with the game con-
ing spatial recognition software, heat sensor and sole without the need to touch a game controller,
propriety algorithm. The Kinect measures the dis- through a natural user interface using gesture and
tance to a shopper to figure out the physical attrib- spoken command. Using these characteristics, de-
ute of the shopper. The data reception and interpre- velopers try to create applications for various mar-
tation occurs in almost real-time as the the system ket fields such as healthcare.
receives 200 frames per second.
This application using Kinect's technology is Kuwahara:
placed on the display shelf of the store and analyz- Yes, you right. Recently, the doctors at a hospital
es customer's movements to provide the traffic in Toronto, Canada have attached Kinect to their
flow analysis and heat maps indicating which medical imaging computer that lets them to manip-
shelves are attracting shoppers and which products ulate the 3D rendered CT image by waving their
they touch or take. hands, to watch the video during an operation. Ac-
tually we could find similar examples in Japan.
Yajima: For example, Kinect is used as a virtual fitting
Can we use this application right away? room that is an augmented reality tool that users
allow to try on clothes simply by standing in front
Kuwahara: of a mirror. The users are even able to swap their
Yes, you can use it with default setting. After ap- outfits with a wave of your hand.
propriate implementation(the location you place it
2 Sapporo Sparkle k.k.

3. An innovative use for Kinect as tracking consumer —- New idea creates a different utility proposition
the virtual world, but it's quit difficult in the real
world.
The idea is great and it works really
well. Yajima:
Retailers have tried to collect these data but they
As you all know, Kinect has been used other than never really succeeded.
the game console such as virtual reality, robotics
and other great and new ideas. Shopper Tracker, Kuwahara:
one of the latest Kinect application that allows For the retailers, t is very important to understand
marketers and store owners track the way consum- demand forecasting. There are multiple factors that
ers shop also applies a new idea to particular issue play crucial part to maximize the profit. The retail-
of need. ers should pay attention to many things including
What I'm interested in this application is the idea of consumer preference, competitor pricing and mar-
trying to analyze customer movement with Kinect. ket trend and so on.
It is a totally different way of using Kinect than
what was originally intended as a game controller. Yajima:
In addition to this, it is used to gather data that is What is the point of using Kinect in this applica-
essential to the operation of the business. I actually tion?
thought that I was done for, because the distribu-
tion is one of my specialty and I involved this kind Kuwahara:
of project before. I knew that Kinect has been used I think the good point about this is that this Kinect-
in variety of ways, but I couldn't think of using Ki- based application does not bother customers unnec-
nect for analyzing customer's movement and be- essarily because it tracks and analyzes customer
havior. behavior remotely.
It is a completely new way of using Kinect and In the past researches, surveillance cameras placed
fully utilizes its characteristics features. I really am closely to the store shelves somehow interrupted
impressed by the idea and efforts of a developer. I customer's natural behavior. Naturally, the credibil-
think we can use this application in many areas of ity of the search was not very high. Some retailers
business. used consumer monitors to collect data, but it actu-
ally did not cooperate well with this research.
Yajima:
How did the developer create this application? Yajima:
I think that using Kinect application makes the
Kuwahara: analysis of the research results faster.
I think they had an idea to improve store operation,
but they didn't find a way how to realize it. They Kuwahara:
took some time and then finally they found the way Yes that's true. Kinect-based application only re-
to their idea into a reality, and their answer was trieves the data of the shoppers in the store. On the
using Kinect. other hand, usual surveillance camera retrieves the
The developer of this application said that they will data of whole image, consequently it takes long
realize "Google Analytics in the real world", and time to analyze the data.
they titled the demo video file "Shopper Tracker's The unique and advanced technology used in Ki-
Kinect hack is like Google Analytics for retail store nect inspires the developers. Since the release of
shelves". the Kinect software development kit, the develop-
A tool like Google Analytics can track website us- ers have created thousands of uses for Kinect, and
er's behavior and help you analyze where your traf- the potential for additional innovation is literally
fic comes and how long your visitor stays on your limitless. Shopper Tracker that analyzes customer
site and so on. However in the real world, it is al- movements to provide traffic flow analysis is just
most impossible to track and analyze consumer's one of these, however it is changing the whole pro-
behavior in the store. Trying to change the whole cess of customer analysis.
picture, they developed the Kinect application. And now the customer analysis is becoming a part
Retailer collects the buying history of their custom- of the business process that maintain and manages
ers in order to improve sales and store operation. store, and is of growing importance in the retailing
However these data do not show the relationship industry. The substantial changes in the position
between the sales operation and the customer's be- and the process of the customer analysis affect the
havior and are not sufficient to understand the cus- way of selling a business.
tomer's behavior. To collect these data is easy in
3 Sapporo Sparkle k.k.

4. An innovative use for Kinect as tracking consumer —- New idea creates a different utility proposition
Yajima: you can control a stream of customer as you desire.
Fresh and innovative idea helps to create a new Moreover, Kinect can track customer's bone struc-
application that maximizes the new technology. ture and recognize each customer as an individual.
And the new application creates a new usage occa- Using these information, it is possible to track one
sion. particular customer's shopping frequency and be-
havior etc.
Kuwahara: And now, we look at a way of using the result of
Yes, I think that is actually a power of design. De- the store research and analysis on the sales promo-
signing the application with state-of-art technology tion activity. Using Kinect and the real-timeness of
changes in the design of retailing. the application, it is possible to give messages in
However, this structure, cycle of change has not accordance with the customer's movements.
been completed. Shopper Tracker demonstrates For concrete example, commercial message for the
great potential for use of the new technology as product in customer's hand on POP display or dis-
analytical tool. From here, we go into the next count service by scanning bar-code coupon shown
stage and should expand the idea of the technology on the display with customer's mobile device.
to create more valuable applications. We also can use the data accumulated by this re-
Using Shopper Tracker, Kinect-based application search and analysis system. For example, showing
will inspire you and help you to expand you imagi- a detailed information of a product that is frequent-
nation. As many people have said Kinect can be ly taken from a shelf but never purchased on the
used in very many different application with little display. In order to promote the product, the sys-
modification and POS system feature a combina- tem automatically chooses what the information to
tion of Kinect technology is heading to laptops provide to the customer.
soon. There is another way of utilization of this research
and analysis system. Using best available data,
Yajima: combination of the POS system and the results of
So we can develop various applications based on research and analysis, to determine appropriate dis-
the need, demand and innovation. And they can be count price rate.
used in various usage scenarios.
Kuwahara: An exciting and usable product
That's correct. Kinect itself and the idea used in
this application is very attractive. However, the Yajima: When I surf web pages, I often see the ad-
structure of the application is quite simple. So we vertisement of sales promotion, and it is actually
can use it in various ways. annoying. I think what you have suggested are
This application is made of four parts, including a pretty similar to this advertisement.
gesture recognition, retrieve and store of the data,
analysis of the data, and display of the analysis re- Kuwahara:
sults. The usage of the application will be deter- The important point is to make customers satisfied
mined by the data source type selected and the op- or to give them help they need. In order to achieve
eration process will be changed by of displaying this goal, we should design application from the
the result of the analysis. viewpoint of customer.
There is a wide variety of store research and analy- The usage scenario or use case of application will
sis for a retail company. For example, if you want decide its effectiveness. However this is not only
to research consumer's behavior at the testing for the application but the any sales promotion ac-
event, you should track the customer's movements tivities that we propose. So the usage scenario is
and how many people are tried of tasting. If you very important for retailing business and we have
want to research the relationship between store studied effective and efficient ways to use the ap-
shelf space and unit sale in a store, you should plication for long time.
track the customer's movements and shopping hab-
its. Yajima:
By placing a number of Kinect all over the store, I think this study can be used in other area of busi-
you can check all area at all times. And it makes ness.
you to analyze the information such as a level of
crowdedness in the store, customer's shopping be- Kuwahara:
havior(its purchasing pattern and merchandize se- The operation is led by knowing about the move-
lection), where shoppers stop in a store and what ment of worker, and is guided by connecting the
they really do and so on. Using these information,
4 Sapporo Sparkle k.k.

5. An innovative use for Kinect as tracking consumer —- New idea creates a different utility proposition
worker's movements and the product. The relation-
ship between work flow and work time can be ana-
lyzed.
The idea of store research and analysis is devel-
oped by "Kinect + data utilization". However if
you focus more on the data, the range of applica-
tion will be expanded. Therefore we should focus
on the currency of the data.
There is a production system in which the industri-
al robot automatically works by knowing about the
movements of the worker. So we can use Kinect in
the field of the industrial robot manipulators.
The use of Kinect is now expanding and is getting
better and better. Some consumer devices are very
usable and may even inspire your imagination. The
application that fully utilizes those usable and ad-
vanced devices shows us that the idea realizes its
full potential.
There is a surprise that "we are able to do some-
thing like this" and it inspires our imagination. The
usage scenario of the use case will be expanded by
the ideas and innovations of people. I think this is a
great pleasure of developing a new device.
5 Sapporo Sparkle k.k.

6. Publisher Sapporo Sparkle k.k.
Date of Issue January 31, 2012
No. 2012013101