1. Staying together:
Understanding People and Media in
Synchronous Connected Systems.
david @ayman shamma
microeconomics & social system

2. Internet Experiences Group
(David) Ayman
Shamma Lyndon Kennedy Jude Yew Elizabeth Churchill

3. Methods as Verbs
OPTIONS GET HEAD
DELETE PUT POST
TRACE CONNECT PATCH

5. Legacy Video

6. Traditional Comments and Tags
Left in Whole, Unattached.

7. Social Conversations Happen Around Media
Dolores Park, San Francisco, 2006

8. Social Conversations Happen Around Media
Dolores Park, San Francisco, 2006

9. Social Conversations happen around videos
Well – actually people join in a session and converse afterwards.

11. What to Collect to measure
• Type of event
(Zync player command or a normal chat message)
• Anonymous hash
(uniquely identifies the sender and the receiver, without
exposing personal account data)
• URL to the shared video
• Timestamp for the event
• The player time (with respect to the specific video) at the
point the event occurred
• The number of characters and the number words typed
(for chat messages)
• Emoticons used in the chat message

12. A Short Movie

13. Percent of actions over time.

14. Chat follows the video!
CHAT

15. Reciprocity
• 43.6% of the sessions the invitee played at
least one video back to the session’s initiator.
• 77.7% sharing reciprocation
• Pairs of people often exchanged more than
one set of videos in a session.
• In the categories of Nonprofit, Technology
and Shows, the invitees shared more videos

16. How do we know what people are watching?
How can we give them better things to watch?
CLASSIFICATION

17. Social Multimedia, no really this time...

18. 5 star ratings has been the golden egg for recommendation systems
so far; implicit human cooperative sharing activity works better.
CLASSIFICATION BASED ON
IMPLICIT CONNECTED SOCIAL

20. Used and Unused Data
You Tube Zync
Duration (video) Duration (session)*
Views (video)
Duration # of Play/Pause*
Duration (session)*
Rating*
Views # of Scrubs*
# of Play/Pause*
Rating* # of Chats*
# of Scrubs*
You Tube (not used) Zync (not used)
Tags Emoticons
Comments User ID data
Favorites # of Sessions
# of Loads

21. Phone in your favorite ML technique.
FIRST ORDER DATA WASN’T
PRETTY

22. Naïve Bayes Classification
Type Accuracy
Random Chance 23.0%
You Tube Features 14.6%
You Tube Top 5 Categories 32.4%
Zync Features 53.9%
Humans 60.9%

23. What about these three videos? Which one you like?
Nominal Factorization

24. Ratings doen’t particularly specify order.
Nominal Factorization

25. Classification with Factoring
Type Accuracy
Random Chance 23.0%
You Tube Features 14.6%
You Tube Top 5 Categories 32.4%
YT Top 5 Factoring Duration 51.8%
Humans 60.9%
YT Top 5 Factoring Views 66.9%
YT Top 5 Factoring Ratings 75.5%
YT Top 5 Factoring All Features 75.9%
psst, yes we know that more training will do the same thing eventually,
I just don’t like waiting.

26. Classification w/ Zync features
Type Accuracy
Random Chance 23.0%
You Tube Features 14.6%
You Tube Top 5 Categories 32.4%
YT Top 5 Factoring Duration 51.8%
Humans 60.9%
YT Top 5 Factoring Views 66.9%
YT Top 5 Factoring Ratings 75.5%
YT Top 5 Factoring All Features 75.9%
Zync Factored All Features 87.8%
psst, yes we know that more training will do the same thing eventually,
I just don’t like waiting.

27. Triangulation?
Classifier
Survey Data Interviews

28. Social Conversations Happen Around Media
Dolores Park, San Francisco, 2006

29. Social Conversations Happen Around Media
Dolores Park, San Francisco, 2006

30. People Tweet While They Watch

32. Sept 26, 2009 18:23 EST
RT: @jowyang If you are watching the debate you’re
invited to participate in #tweetdebate Here is the 411
http://tinyurl.com/3jdy67
INDIRECT ANNOTATION

33. Repeated (retweet) content starts with
RT
Address other users with an @
Rich Media embeds via links
Tags start with #
RT: @jowyang If you are watching the debate you’re
invited to participate in #tweetdebate Here is the 411
http://tinyurl.com/3jdy67
ANATOMY OF A TWEET

34. Tweet Crawl circa 2008
• Three hashtags: #current #debate08 #tweetdebate
• 97 mins debate + 53 mins following = 2.5 hours total.
• 3,238 tweets from 1,160 people.
– 1,824 tweets from 647 people during the debate.
– 1,414 tweets from 738 people post debate.
• 577 @ mentions (reciprocity!)
– 266 mentions during the debate
– 311 afterwards.
• Low RT: 24 retweets in total
– 6 during
– 18 afterwards.

35. Volume of Tweets by Minute
Crawled from the Twitter RESTful search API.

36. Tweets During and After the Debates
Conversation swells after the debate.

37. Post debate
Volume of Conversation Follows the Debate

38. Social Conversations happen around videos
Well – actually people join in a session and converse afterwards.

39. http://www.flickr.com/photos/wvs/3833148925/

40. Post Segment?
Does Conversation follow After a Segment
Think of Isaac Newton

41. Will the roots of fʼ(x) find segmentation?

42. Automatic Segment Detection
We use Newton’s Method to find extrema outside μ±σ to find
candidate markers. Any marker that follows from the a marker on the
previous minute is ignored.

43. Automatic Segment Detection with 92%
When compared to CSPAN’s editorialized Debate Summary ± 1
minute.

44. Directed Communication via @mentions
John Tweets: “Hey @mary, my person is winning!” Makes a directed
graph from John to Mary.

45. Barack, NewsHour, & McCain automatically discovered.
High Eigenvector Centrality Figures on Twitter from the First US
Presidential Debate of 2008.

46. Tweets to Terms
Common stems in bold-italic.

47. Tweets are Reaction not Content

48. Sen$ment/Affect judgements from the debate.
[1]
Diakopoulos, N. A., and Shamma, D. A. Characterizing debate performance via
aggregated twitter sentiment. In CHI ’10: Proceedings of the 28th international
conference on Human factors in computing systems (New York, NY, USA, 2010),
ACM, pp. 1195–1198.
48

49. Sen$ment/Affect judgements by candidate.
[1]
Diakopoulos, N. A., and Shamma, D. A. Characterizing debate performance via
aggregated twitter sentiment. In CHI ’10: Proceedings of the 28th international
conference on Human factors in computing systems (New York, NY, USA, 2010),
ACM, pp. 1195–1198.
49

50. Inauguration 2009
http://www.flickr.com/photos/twistedart/3212723019/

51. Tweet Stream circa 2009
• Data Mining Feed
• 600 Tweets per minute
• 90 Minutes
• 54,000 Tweets from 1.5 hours
• Constant data rate means the volume
method doesn’t work.

52. Data Mining Feed
53,000 Tweets @ 600 per minutes

53. Drop in @conversation as onset

54. Less @ means less chars

55. Terms as topic points
Using a TF/IDF window of 5 mins

56. No significant
occurrence of
Terms as topics points “remaking”
Using a TF/IDF window of 5 mins, find terms that are only relevant to
that slice, subtract out salient, non-stop listed terms like: Obama,
president, and speech.

57. Peak occurrence
of “remaking”.
Contains an
occurrence of
“remaking” less
No significant significant than
occurrence of peak.
“remaking”.
Terms as Sustained Interest
Using a TF/IDF window of 5 mins, find terms that are only relevant to
that slice, subtract out salient, non-stop listed terms like: Obama,
president, and speech.

58. Sustained Interest & Background Whispers.
Some topics continue over time with a higher conversational context.

59. 0.35
0.25
Sustained Interest & Background Whispers.
These terms are not sailent by any standard term/document model.

60. People Announce
(12:05) Bastille71: OMG - Obama just messed
up the oath - AWESOME! he’s human!
(12:07) ryantherobot: LOL Obama messed up
his inaugural oath twice! regardless, Obama is
the president today! whoooo!
(12:46) mattycus: RT @deelah: it wasn’t Obama
that messed the oath, it was Chief Justice
Roberts: http://is.gd/gAVo
(12:53) dawngoldberg: @therichbrooks He
flubbed the oath because Chief Justice

61. People Reply
(12:05) Bastille71: OMG - Obama just messed
up the oath - AWESOME! he’s human!
(12:07) ryantherobot: LOL Obama messed up
his inaugural oath twice! regardless, Obama is
the president today! whoooo!
(12:46) mattycus: RT @deelah: it wasn’t Obama
that messed the oath, it was Chief Justice
Roberts: http://is.gd/gAVo
(12:53) dawngoldberg: @therichbrooks He
flubbed the oath because Chief Justice

62. Two Metrics

63. Two Metrics

64. Statler
http://bit.ly/statler

65. 2.4 Million Tweets on the VMA

66. 2010 VMAs TOC

67. Top 2 whisper terms during the

68. kanye & a**--le
BLUE Line RED Line

69. Earlier you said
synchronous,
what’s that all
You may ask yourself….

70. Understanding Engagement
Better recommendations.
Better understanding of the relationship between users and the sharing/
consumption of media content.
Better organization and classification of media for efficient navigation and content
retrieval.
Better advertising.

71. Remember these Verbs?
Which one was added?
OPTIONS GET HEAD POST
CONNECT TRACE DELETE PUT
PATCH MONITOR

72. More and more sync…
Tweeting Together
• External Associated Media
• HTTP Long Poll
Watching Together
• Embedded & Manipulated Media
• Connected Sharing Session
Creating Together
• Appropriated Collaborative Media
• Connected Action & Motion

73. Mapping Cursor Movement to Physical

77. Dancers & People Acting Together
They thought we were controlling the
images, once they learned that they were
controlling it was interesting to see their
delight in that and how it brought them to a
new place of play with the phones and then
they got a little bit more engaged and
excited. (D3)

78. Body Moving, Body Moving
“I want to just start moving my
body so much even though I
know it doesn’t make a
difference.” (A3)

79. Recapitulation
Human conversation is Human conversation…start there before you go all “Big Data”.
Connected action provides better signals. (Implicit Synchronous Sharing activity is richer
than Asynchronous Annotations)
Metrics and instrumentation should account for social interactions & engagement (which may
be measured by the lack of signal).
Nominal Factorization Assists Classification (enough with the Big Data thing already)
The Verbs are in need of updating. How we build the next generation of tools and appliances
shouldnʼt be limited by them.

80. +
Thanks round one…
To my fellow arVsts (Renata & Jürgen).
Also to our amazing dancers Christy
Funsch, Nol Simonse, and Erin Mei‐Ling
Stuart; their contribuVon, advise, and
paVence during many a rehearsal secVon.

81. Fin.
Thanks to N. Diakopoulos, E. Churchill, L. Kennedy, J.Yew, S. Pentland, A.
Brooks, J. Antin, J. Dunning, Chloe S., Ben C., Marc S., & M. Cameron J.
Human-to-Dancer Interaction Designing for Embodied Performances in a Participatory Installation. David A. Shamma,
Renata Sheppard, Jürgen Schible, DIS 2010.
Tweet the Debates: Understanding Community Annotation of Uncollected Sources David A. Shamma; Lyndon Kennedy;
Elizabeth F. Churchill, ACM Multimedia, ACM, 2009
Understanding the Creative Conversation: Modeling to Engagement David A. Shamma; Dan Perkel; Kurt Luther,
Creativity and Cognition, ACM, 2009
Spinning Online: A Case Study of Internet Broadcasting by DJs David A. Shamma; Elizabeth Churchill; Nikhil Bobb; Matt
Fukuda, Communities & Technology, ACM, 2009
Zync with Me: Synchronized Sharing of Video through Instant Messaging David A. Shamma; Yiming Liu; Pablo Cesar,
David Geerts, Konstantinos Chorianopoulos, Social Interactive Television: Immersive Shared Experiences and
Perspectives, Information Science Reference, IGI Global, 2009
Enhancing online personal connections through the synchronized sharing of online video Shamma, D. A.; Bastéa-Forte,
M.; Joubert, N.; Liu, Y., Human Factors in Computing Systems (CHI), ACM, 2008
Supporting creative acts beyond dissemination David A. Shamma; Ryan Shaw, Creativity and Cognition, ACM, 2007
Watch what I watch: using community activity to understand content David A. Shamma; Ryan Shaw; Peter Shafton;
Yiming Liu, ACM Multimedia Workshop on Multimedia Information Retrival (MIR), ACM, 2007
Zync: the design of synchronized video sharing Yiming Liu; David A. Shamma; Peter Shafton; Jeannie Yang, Designing
for User eXperiences, ACM, 2007