AI art talk with generative poetry and image recognition for KIKK Festival 2017.

1. COCO’s Memory Palace:
A Strange Fantasia
Lynn Cherny
@arnicas

3. Images

4. Images
Poetry

5. Images
Poetry
Memory

6. Project 1: Color
Poems
Color analysis & Google Vision API, Unsplash API

7. Martin Krzywinski’s color summarizer API
http://mkweb.bcgsc.ca/colorsummarizer/

8. Martin Krzywinski’s color summarizer API
http://mkweb.bcgsc.ca/colorsummarizer/

9. “My list uses the following sources: bang-v2, beer SRM chart,
colorhexa, crayola, n3, ntc, pantone, raveling, resene, rgb.txt
(X11), steve.hollasch, wikipedia, and xkcd.”
9284 Named Colors
http://mkweb.bcgsc.ca/colornames/

15. Some More Design Details
• BigText.js for text layout (space-filling width/size)
• Complicated drop shadows to make text stand
out against the image background
• CSS image effects to grayscale, blur, opacity
• (Flexbox saved my ass)

18. –John Berger, Understanding a Photograph
“I have decided that seeing this is worth
recording. … Photography is the process of
rendering observation self-conscious.”

19. https://www.nyu.edu/classes/bkg/methods/harper.pdf
“Talking about pictures: a case for photo elicitation” (Harper, 2002)

20. https://www.ncbi.nlm.nih.gov/pubmed/28889976

21. Fludd’s
Memory Palace
of Music

22. –Frances Yates, The Art of Memory
“There are two kinds of memory… one natural,
the other artificial. The natural memory is that
which is engrafted in our minds, born
simultaneously with thought. The artificial
memory is a memory strengthened or
confirmed by training.”

24. Romberch Memory,
1553

25. –Frances Yates, The Art of Memory
“The artificial memory is established from
places and images…. A locus is a place easily
grasped by the memory, such as a house, an
intercolumnar space, a corner, an arch, or the
like. Images are forms, marks or simulacra.. of
what we wish to remember. For instance if we
wish to recall the genus of a horse, or a lion, of
an eagle, we must place their images on
definite loci.”

26. @TheGlenlivet

27. amazon rekognition says…

28. amazon rekognition says…

29. Recognition works on either an S3 bucket url,
or an image blob.

30. Using Google Vision API is similar but
can take any url

31. Project 2: COCO’s
poems, & 3 voices

32. Fourteen
Mary Doria Russell
All bone, no meat
Growing, not grown.
Desire, dreams,
Nothing to show.
Diving, but still dry.
Photo courtesy of Accidental Mysteries, collection of John Foster.
“Poems About Photography”

33. MS COCO Dataset
“Common Objects in Context” image training dataset:
“Our dataset contains photos of [80 objects]* types that would
be easily recognizable by a 4 year old.”
http://cocodataset.org/#home *2014 was 80, not 91

34. Useful articles
How to Freeze a Model and Serve It With A Python API
Dat Tran’s post: How to Train Your Own Object Detector With
Tensorflow

35. Useful articles
How to Freeze a Model and Serve It With A Python API
Dat Tran’s post: How to Train Your Own Object Detector With
Tensorflow

37. Undergraduates said…. “people, event”

39. What is the model trained on
==

40. What is the model trained on
==
what can it talk about?

41. COCO’s world of 80 things.
"person"
"bicycle"
"car"
"motorcycle"
"airplane"
"bus"
"train"
"truck"
"boat"
"traffic light"
"fire hydrant"
"stop sign"
"parking meter"
"bench"

42. COCO’s world of 80 things.
"person"
"bicycle"
"car"
"motorcycle"
"airplane"
"bus"
"train"
"truck"
"boat"
"traffic light"
"fire hydrant"
"stop sign"
"parking meter"
"bench"
"bird"
"cat"
"dog"
"horse"
"sheep"
"cow"
"elephant"
"bear"
"zebra"
"giraffe"

43. "backpack"
"umbrella"
"handbag"
"tie"
"suitcase"
"frisbee"
"skis"
"snowboard"
"sports ball"
"kite"
"baseball bat"
"baseball glove"
"skateboard"
"surfboard"
"tennis racket"

44. "backpack"
"umbrella"
"handbag"
"tie"
"suitcase"
"frisbee"
"skis"
"snowboard"
"sports ball"
"kite"
"baseball bat"
"baseball glove"
"skateboard"
"surfboard"
"tennis racket"
"bottle"
"wine glass"
"cup"
"fork"
"knife"
"spoon"
"bowl"
"banana"
“apple”
"orange"
"broccoli"
"carrot"
"hot dog"
"pizza"
"donut"
“cake"
"sandwich"

45. "chair"
"couch"
"potted plant"
"bed"
"dining table"
"toilet"
"tv"
"laptop"
"mouse"
"remote"
"keyboard"
"cell phone"
"microwave"
“oven"
"toaster"
"sink"
"refrigerator"

46. "chair"
"couch"
"potted plant"
"bed"
"dining table"
"toilet"
"tv"
"laptop"
"mouse"
"remote"
"keyboard"
"cell phone"
"microwave"
“oven"
"toaster"
"sink"
"refrigerator"
"book"
"clock"
"vase"
"scissors"
"teddy bear"
"hair drier"
"toothbrush"

47. “potted plant” but not “tree.”

48. “potted plant” but not “tree.”
COCO knows about computers, but
there is no sky, no grass, no mountains.

51. There is a Jupyter notebook there that will run the
tensorflow demo for you.

52. How poetic is COCO?

54. eh, whoops. Not legit: “traffic light” not “light”, “fire hydrant”
not “fire.”

56. “couch” > “cat”
“toilet” > “umbrella”
“traffic” > “bus”
“spoon” == “fork” but “knife” beats both

57. > grep sandwich poetry-json

58. “in microwave fashion”
“donut-sized scales”
“even if ice-cream and
pizza attach dead-pan
humor”
> grep sandwich poetry-json

59. COCO’s grammar*
• Includes words related to her degree of
uncertainty at the object recognition
• nouns mixed in that are NOT in her vocabulary,
from a word2vec lookup of related terms in the
Gigaword corpus served by API (part-of-speech
tagged)
• Language style reflecting naivety and youth
*tracery.js from Kate Compton

60. https://github.com/zonination/perceptions

62. Part-of-speech tagged Gigaword corpus word2vec model
Words similar to “castle_NOUN”:

63. –John Berger
“A photograph, whilst recording what has been
seen, always and by its nature refers to what is
not seen. … the quantum of truth in an
'impersonal' photograph must still depend upon
the general categories already in the
spectator's mind.”

65. Naho
Matsudo
http://everythingeverytime.net/about.html

68. Adding more voices
• Google Vision API — more “objective” and
scientific language
• Amazon Rekognition — more colorful, more
poetic

70. Google Vision API
& Amazon Rekognition
(the nouns)

74. Using COCO, Amazon, and Google API’s all together…

77. WH Auden on “The Fall of Icarus”
by Peter Brueghel

80. (Google said “FU” to the URL)

81. “the old guitarist,” Picasso
“The Man with the Blue Guitar”
by Wallace Stevens

82. “The man bent over his guitar,
A shearsman of sorts. The day was green.
They said, “You have a blue guitar,
You do not play things as they are.”
The man replied, ‘Things as they are
Are changed upon the blue guitar.’”

84. https://www.amazon.com/PERSONAL-Recordable-Plush-Talking-Teddy/dp/B001RZE0E4

85. COCO says, man with a teddy bear.
https://www.amazon.com/PERSONAL-Recordable-Plush-Talking-Teddy/dp/B001RZE0E4

86. COCO says, man with a teddy bear.
https://www.amazon.com/PERSONAL-Recordable-Plush-Talking-Teddy/dp/B001RZE0E4

87. https://www.eff.org/ai/metrics

88. –Wallace Stevens, Man With the Blue Guitar
“We shall forget by day, except
The moments when we choose to play
The imagined pine, the imagined jay.”

89. –Yates, the Art of Memory
“an artificial memory gone out of control into
wild imaginative indulgence…”

90. –Yates, the Art of Memory
“an artificial memory gone out of control into
wild imaginative indulgence…”
“the dream archaeology of a humanist mingles with dream
memory systems to form the strange fantasia.”

91. Thank you.
-@arnicas
Slides will go on slideshare and a blog post.