This document discusses new media and mixed media art. It covers several topics: - Lev Manovich's five levels of analyzing new media, including numerical representation, modularity, automation, variability, and transcoding. - Examples of new media like digital images, which are discrete, modular, compressed, and variable. Images can also function as interfaces and instruments. - Hyperlinked images and databases of images as cultural units, which can reveal patterns of interaction and define niches. Collaborative storytelling is possible in databases. - The shift to digital film, which combines various techniques and creates an "elastic reality." Future directions for digital film include non-linear and non-narrative forms. - Games

1. Introduc)on to New Media
Lecture 4
Mixed media art and comprehension
What perhaps has been
most surprising and
consequen4al about new
technologies has been not so
much what such innova4ons
were originally designed to
do, as what users have
chosen to use them to
accomplish.
Harrison & Barthel, 2009

2. Topics
• The levels of visual media and examples
• New media, crea)on and art
• Comprehension of new media

3. Ques)ons
• Draw a concept map of comprehension and
interac)on with new media (with mixed media
types that allow par)cipatory interac)ons)
• For essay: Mixed media percep)on – how we
may perceive one new media applica)on?

4. The levels for visual media
• Lev Manovich The Language of New Media
(2001)
• Manovich is looking new media at five levels:
– Numerical representa)on & programmability
– Modularity at code level
– Automa)za)on of opera)ons in crea)on,
manipula)on and access
– Variability and branching type of interac)vity
– Transcoding at the cultural interface level
Lev Manovich The Language of New Media (2001)

5. Digital images
• Computer‐based image is discrete, since it is
broken into pixels. This makes it more like a
human language (but not in the semio)c
sense of having dis)nct units of meaning).
• Computer‐based image is modular, since it
typically consists from a number of layers
whose contents oSen correspond to
meaningful parts of the image.
Lev Manovich The Language of New Media (2001)

6. Digital images
• Computer‐based images are typically compressed
using lossy compression techniques, such as
JPEG. Therefore, presence of noise (in a sense of
undesirable ar)facts and loss of original
informa)on) is its essen6al, rather than
accidental, quality.
Lev Manovich The Language of New Media (2001)

7. Digital images
• Variability and
automa6on, these general
principles of new media,
also apply to images.
• For example, using a
computer program a
designer can automa)cally
generate infinite versions
of the same image which
can vary in size, resolu)on,
colors, composi)on and so
on.
Lev Manovich The Language of New Media (2001)
Technologies have disappeared behind
the metaphor of pain)ng, enabling
others to use them within the familiar
frame of tradi)onal ar)s)c prac)ce
(from Lehmann, 2009).

8. Digital images
• Computer‐based image consists from two levels, a
surface appearance and the underlying code (which
may be the pixel values, a mathema)cal func)on or
HTML code).
• In terms of its “surface,” an image par6cipates in the
dialog with other cultural objects. The surface‐code
pain can be related to signifier — signified, base —
superstructure, unconscious — conscious pairs. So, just
as a signifier exists in a structure with other signifiers
of a language, a “surface” of an image, i.e. its
“contents” enters in dialog with all other images in a
culture.
• In terms of its code, an image exist on the same
conceptual plane as other computer objects.
Lev Manovich The Language of New Media (2001)

9. Digital images: image interfaces
Lev Manovich The Language of New Media (2001)
• An image acquires the new role of an interface
‐ image becomes image‐interface.
• In this role it func6ons as a portal into
another world
• Rather than staying on its
surface, we expect to go
“into” the image.
• Image can func)on as an
interface because it can be “wired” to
programming code; thus clicking on the image
ac)vates a computer program (or its part).

10. An image‐interface may be
interac)vely manipulated

11. image as image‐interface
• The new role of an image as image‐interface
competes with is older role as representa6on.
• Therefore, conceptually, a computer image is
situated between two opposing poles: an
illusionis6c window into a fic6onal universe and
a tool for computer control.
• The task of new media design and art is learn
how to combine these two compe6ng roles of
an image.
Lev Manovich The Language of New Media (2001)

12. image‐instruments
• computer images also
func6ons as image‐
instruments.
• If image‐interface
controls a computer, an
image‐instrument
allows the user to
remotely affect physical
reality in real 6me.
Lev Manovich The Language of New Media (2001)

13. Ini)a)ng
Social art
ac)vity

14. Mixed body parts
• I came up with the idea of new narra)ve using parts of
bodies in urban environment and trace the dimension
of urban hybrid being, thus research how different
people perceive and par)cipate.
• For this experiment common tag besides
#narra)veecology is mixedbodies and then for
par)cular images ‐ head, foot, torso, arm.
• In flickr you can easily organise photos in a batch
(rotate, add tags, geo loca)on to all needed pictures at
once and send them to the group sets ;).

15. Mixed body parts
It is not easy to trigger par)cipa)on
The flickr group is created
and you are welcome to
add your body parts.

16. Hyperlinked images
• A computer image is frequently hyperlinked to
other images, texts, and other media
elements.
• Rather than being a self‐enclosed en)ty it
points, leads to, directs the user outside of
itself towards something else.
• Hyperlinked image, and hypermedia in
general, “externalizes” Pierce’s idea of infinite
semiosis
Lev Manovich The Language of New Media (2001)

17. Nelson: ”hyperfilm” and “hypermedia”
• Nelson introduced the terms hyperfilm and
hypermedia:
• Films, sound recordings, and video recordings
are also linear strings, basically for mechanical
reasons. But these, too, can now be arranged as
non‐linear systems – for instance, laFces – for
educa6onal purposes, or for display with
different emphasis…
• The hyperfilm – a browsable or vari‐sequenced
movie – is only one of the possible hypermedia.
Lev Manovich The Language of New Media (2001)

18. A database of images as a cultural unit
• From a single image which represented the
“cultural unit” of a previous period we move
to a database of images as “cultural units” .
• Today the problem is no longer how to create
the right image, but how to find already
exis)ng one.
Lev Manovich The Language of New Media (2001)

19. hhp://mayron.net/liam/pub/mayron_disserta)on.pdf

20. A database of images as a cultural unit
Sunsets by )me: )me graphs (Photographs mapped by )me.)
hhp://www.flickr.com/photos/krazydad/4992355/in/set‐140323/
The deepest "dip" in the wave formed by the images is the Summer Sols)ce.

21. A database of images reveals paherns
of interac)on with space
hhp://liSlab.com/think/fabien/2006/12/13/tracing‐the‐visitors‐eye/
A trace consists in an
ordered set of geotagged
images taken by one
person in one day and
downloaded to Flickr.
There might be a
poten)al to define and
confirm paherns of how
tourists navigate the
urban space

22. A database of images as a cultural unit
reveals niches
• Invisible Ci)es maps
geocoded ac)vity
from online services
such as Twiher and
Flickr, both in real‐
)me and in
aggregate to
another: an
immersive, three
dimensional space
crea)ng hills and
valleys represen)ng
areas with high and
low densi)es of
data.
hhp://www.chris)anmarcschmidt.com/invisibleci)es/

23. Storytelling in databases is
collabora)ve
Iden)fying narra)ve paherns have always been one of our main
strategies for understanding the world
hhp://jilltxt.net/txt/Walker‐AoIR‐3500words.pdf

24. Storytelling in databases reveals niches

25. Film
• Cinema erases any traces of its own
produc6on process, including any indica6on
that the images which we see could have
been constructed rather than recorded
• It denies that the reality it shows oSen does
not exist outside of the film image
• It pretends to be a simple recording of an
already exis6ng reality
Lev Manovich The Language of New Media (2001)

26. the shiS to digital film
• digital film = live ac6on material + pain6ng +
image processing + composi6ng + 2D computer
anima6on + 3D computer anima6on
• film obtains the plas6city which was previously
only possible in pain6ng or anima6on ‐ "elas6c
reality.”
• The result: a new kind of realism, which can be
described as "something which looks is intended
to look exactly as if it could have happened,
although it really could not."
Lev Manovich The Language of New Media (2001)

27. Future for digital films
• cinema)c forms which are not linear narra6ves: music
video, CD‐ROM‐based games, computer mul)media
• non‐narra6ve films
• New Temporality: Loop as a Narra)ve Engine (Lola,
run) (adopted from peep show), arranging the
trajectory of every character through space as a loop
• The loop gave birth not only to cinema‐ computer
programming involves altering the linear flow of data
through control structures, such as "if/then" and
"repeat/while"; the loop is the most elementary of
these control structures.
Lev Manovich The Language of New Media (2001)

28. A game
• A game is an ar6s6cally simplified representa6on of a
phenomenon.
• The game designer simplifies deliberately in order to focus
the player's aRen6on on those factors the designer judges
to be important.
• A game is collec)on of parts which interact with each
other, oSen in complex ways. It is a system in which
interac6ve element is a crucial factor
• The subjec6ve reality and objec6ve reality are intertwined
– it is a subset of reality. A game is a safe way to experience
reality
• The player's fantasy is the key agent in making the game
psychologically real.
The Art of Computer Game Design by Chris Crawford

29. A game
• While a story is a vehicle for represen)ng
reality, not through its facts per se, but through
the cause and effect rela)onships suggested by
the sequence of facts,
• the game presents a
branching tree of
sequences and allows
the player to create his
own story by making
choices at each branch
point,
• a game is dynamic.
The Art of Computer Game Design by
Chris Crawford

30. A game
• Game‐playing requires two components: a game
and a player.
• A game acknowledges the player's existence and
reacts to the player's personality and transforms
the nature of the challenge from a passive
challenge (defined by someone) to an ac)ve
challenge (defined by me)
• Interac)on injects a social or interpersonal
element into the event and transforms the
challenge of the game from a technical one to an
interpersonal one.
The Art of Computer Game Design by Chris Crawford

31. A game
• Conflict is fundamental to all games
• If an intelligent agent from the game ac)vely
blocks the player's ahempts to reach his goals,
and purposefully responds to the player’s
ahempts, the conflict between the player and the
agent is inevitable.
• Without ac)ve response, there can be no
interac)on. Thus, expunging conflict from a
game inevitably destroys the game.
• It is possible to include coopera)ve elements by
shiSing the conflict
The Art of Computer Game Design by Chris Crawford

32. spa)al narra)ve
• In general, spa)al montage would involve a
number of images, poten6ally of different sizes
and propor6ons, appearing on the screen at the
same 6me.
• Its up to the filmmaker to construct a logic which
drives which images appear together, when they
appear and what kind of rela)onships they enter
with each other
• The same principle made computer programming
possible: a computer program breaks a tasks into
a series of elemental opera)ons to be executed
one at a )me.
Lev Manovich The Language of New Media (2001)

33. Spaces of cultural par)cipa)on
• Television programs or website interfaces:
– User prac)ces are structured by pre‐exis)ng
socially and ideologically defined spaces within
which ac)ons are performed
– User ac)ons may nego)ate and transform the
very conven)ons and limits of a given mediated
space
– The social power to construct such spaces and to
define the frameworks for ac)on is not shared
equally within a society
Lev Manovich The Language of New Media (2001)

34. Example: Interac)ve television
• Loops controlled by
sms
• Akvaario program and
Sheep TV in Finland.
• Repe))ve ac)on can
be basis for new
rou)ne
A story between stories: algorithmic and audience control of video segments in an
experimental interac)ve television programme. Chris Hales; Teijo Pellinen; Markus
CastrénDigital Crea)vity 2006, Vol. 17, No. 4, pp. 234–242

35. Interac)ve television
• Online video‐sharing sites such as YouTube, are
examples of how the corporate media industry invests
in digital technologies to redefine the rela6onship
between the spheres of produc6on and consump6on:
– by seducing the audience to contribute to a website, new
sources of revenue are created;
– users gain access to encrypted live streaming;
– users contribute their own content without monetary
compensa)on;
– and whenever they do go online and interact, users
produce data valuable for targeted marke)ng and mass
customiza)on.

36. Example: Youtube
• YouTube and Google Video create a space of
par)cipa)on in which users maintain more control over
the space
– Liberty: any one who owns the technological means can
record and upload a video film and share it online
– Restric6ons: ‘protocols’ and copyright regula)ons
– Commercial: markets for customized adver6sing; the
adver)sers with data reveal any individual user’s online
ac)vi)es, his or her cultural preferences, and the
communi)es in which he or she par)cipates; ‘consumers’ of
a ‘tube of plenty’; viral videos
– Interac6vity: ‘snippets’: ‘prerecorded, rerecorded, )nkered,
and self‐produced audiovisual content’ that is not finished
invites users to appropriate, 6nker with and respond to it;
‘channel’ to which other users can subscribe

37. Example: Youtube
• Nine prototypes of users (YouTube 4 You, Miller
2007):
– the ‘recorder/sharer’,
– the ‘historian/enthusiast’,
– the ‘home movie maker’,
– the ‘video blogger’,
– the ‘instructor’,
– the ‘reporter’,
– The ‘performer’,
– the ‘aspiring film director’
– the ‘online business’
Find one example for each type from Youtube videos

38. You know your meme: Drama)c Chipmunk
and other varia)ons in Youtube
hhp://knowyourmeme.com/
‘FormaFng’ in reference to the adapta)on of interna)onally
circula)ng television programs, in which the format details how a
program should be produced but at the same )me allows
producers to adapt it to the local culture.

39. Loca)ve interac)ve videos
• Can be focusing on:
– Geographical loca)ons ‐ places with loca6ve film
content might bring forth interac6ons and ac6vi6es
with the film or between people
(see Michael Naimark projects – Aspen moviemap
hhp://www.naimark.net/projects/aspen.html)
– Narra)ves – story from place, anecdote; user‐
configured perspec)ves can be chosen from the pool
of thumbnails of videos as a pool of the space
(see projects from Medialab Europe
hhp://medialabeurope.org/ )
– Aesthe)c, landscape tradi)ons
– Social use of space
In search of Oldton
hhp://www.oldton.com/our_oldton.html

40. Chris Hales
• Structure for the interac)ve videomaking process:
– Building projects of the place.
One method: walk around and find a place of hanging around, make
some basic films, interviews of the place, which other places you
hang around – go to the second place and repeat the process.
– Constraints will be needed:
‐ broad range of subject to fixed loca)ons; fix theme and film it in
wider range of loca)ons (eg. movement etc.).
‐ geographical architecture of city (add visitors’ viewpoints or
interviewing local residents, this could work with the older
genera)on: memory‐based approach)
‐ social ac)vi)es related to the place
‐ all the factors involved in the place
– Examples: hhp://csw.art.pl/new/99/7e_heldl.html
– hhp://www.youtube.com/watch?v=4va2dwRl6dU
– See Smartlab projects
hhp://smartlab.uel.ac.uk/2projects/index.htm

41. Criteria defining par)cipatory culture
• ‘Low barriers to ar)s)c expression and civic
engagement’
• ‘Strong support for crea)ng and sharing one’s
crea)ons with others’
• ‘Informal mentorship whereby what is known by
the most experienced is passed along to novices’
• The belief ‘that their contribu)ons maher’
• The belief ‘members feel some degree of social
connec)on with one another’ (Jenkins et al.
2006)
Jenkins, Henry, Ka)e Clinton, Ravi Purushotma, Alice J. Robison, and Margaret Weigel.
2006. Confron)ng the challenges of par)cipatory culture: Media educa)on for the 21st
century. MacArthur Founda)on. hhp://www.digitallearning.macfound.org/ay/cf/%
7B7E45C7E0‐A3E0‐4B89‐AC9C‐E807E1B0AE4E%7D/JENKINS_WHITE_PAPER.PDF.

42. Concept: Produsage
• Bruns (2006) has coined the term ‘produsage’ to
describe a new form of produc6on arising in the
transi)on of media users from audience and
consumers to ‘produsers’ enabled by new
technologies.
• ‘Produsers tend to collaborate’ (Bruns, 2006)
because they are simultaneously consumers of
content produced by others which they then use
as raw material for their own content
construc6on to be consumed and subsequently
acted upon by others.
Bruns, A. (2006) ‘Towards Produsage: Futures for User‐led Content Produc)on’, in
F. Sudweeks, H. Hrachovec and C. Ess (eds) Proceedings: Cultural AItudes towards
Communica4on and Technology 2006, pp. 275–84. Perth: Murdoch University.

43. New media for crea)on and art
Art explores and expresses our aesthe6c
rela6on to our environment and ourselves.
• Ar)s)c opportuni)es are expanding into:
individual expressiveness and collec6ve
ac6vism (Harrison & Barthel, 2009).
• Using social soSware everybody can
simultaneously be an ar6st and the audience,
and this is the ground where both may be
mutually inspiring for par6cipa6on.
Wielding new media in Web 2.0: exploring the history of engagement with the
collabora)ve construc)on of media products
Teresa M. Harrison; Brea Barthel (2009)

44. Community’s par)cipa)on in art
• Five modes of crea)ve control are (Brown,
2005):
– inven6veness
– interpre6ve acts of self‐expression
– curatorial acts of selec)ng organizing and
collec)ng
– observa6onal arts experience that one chooses
– ambient arts experiences that are encountered
unconsciously
Brown, A. (2005) Presenta)on of Neighborhood Survey Results for Philadelphia–
Camden Cultural Par)cipa)on Benchmarking Project

45. Par)cipa)on inside? User ac)vi)es between design and appropria)on
Mirko Tobias Schäfer
Implementa)on of crea)ve user
ac)vi)es into commercial applica)ons

46. Accumula)ng and aggrega)ng

47. Mixing and remixing media for art
• Portwiture” –
the applica)on
that aggregate
Twiher and
Flickr accounts
of user, making
visual
representa)on
of last Twiher
posts of the
user.

48. Par)cipa)ng for art: archiving and
accumula)on
Mikrogalleri.es – the project of ZKM Media Museum in
Karlsruhe. “Visitors to the museum or anyone with an
Internet connec)on can upload images which are
manipulated and published with other user‐supplied images”.
The project goes further than Flickr: it does not only network
the data of digital cameras as par)cipatory images, but also
the cameras itself and with them their physical and ar)s)c
context, place or space. The museum’s website supplies
direc)ons for crea)ng a display of Internet images that can be
projected at remote loca)ons and on the Internet. As ZKM
proclaims, this network of par)cipatory produc)on is a
further step from ‘user generated content’ to ‘user created/
shared media,’ a preview to the new genera)on of the web:
the Internet of Things” (Yang & Roch, 2007).

50. Cosntruc)on and sharing: Rhizome.org

51. Swarming for art
• Twee6ng Colors by Brian Piana ‐ a public webpage
consisted of “ver)cal color bars created by special tweets
from Twiher users” (Piana, 2009). Any Twiher user can add
bars by pos)ng special tweet as shown in the direc)ons.
• Mobotag, by Marta Lwin, “reveals the hidden layers of a
city through an ac)ve exchange of loca)on based media
and text messages via the cellphone” (Lwin, 2009). It is a
collabora)ve phone tagging of the city. Part virtual graffi),
part walking tour, “mobotag” creates a spontaneous and
easy way for tagging a neighborhood via the cellphone.
Par)cipants may respond with their mobile media “in the
crea)ve expression and mapping of their neighborhood” by
sending and view messages, images, videos and sounds. A
unique geocoding feature of Mobotag is uncovering other
messages exis)ng in people’s local area.

52. Swarming for art

53. Hybrid & augmented reality art
The object is oversized 3‐
dimensional computer
cursor (pointer). Placed in
a public space and
embedded with GPS
device it could be touched
and moved even further
the special “screen based”
territory.
Coordinates were mapped in
Google Maps.
Par)cipants could also
upload photos of the
cursor at the website.
(Campion, 2009).
hhp://www.urbancursor.com/
Urban Cursor is a project facilita)ng “social
interac)on and play”.

54. Cogni)ve informa)on‐processing
model
• We need to consider, how humans process
informa)on in order to predict the effect of new
media and develop new media applica)ons

55. Percep)on
• Inputs is received from perceptual organs:
‐ Eyes: Visual input (graphics, text) (visual
percep)on)
‐ Ears: Auditory input (sound, speech)
‐ Skin: Somato‐sensory input (hap)c percep)on)
‐ Tongue: Gustatory input
‐ Nose: Olfactory input
hhp://www.medical‐look.com/human_anatomy/systems/Sensory_organs.html

56. Technology can extend human senses
• McLuhan: any technology can extend human
senses and bodies
• In which way technology can extend the sense
organ? How does using such technology change
our behavior?
– Sensor technology
– Wearable displays
– Seamless interfaces
– Ergonomics
• Robots to simulate human senses

57. Percep)on
Perceptual informa)on is kept 1 sec in perceptual
registers of the sensory memory, this is an
unconscious process
Memory, stored schemata, and past experience
play an important role in percep)on.
Interface and interac)on

58. Percep)on
• Percep)on is a media)onal process ‐ “mental
processes” (cogni)ve ac)va)on from memory) are
needed to enable the pick up of the relevant informa)on
• Cultural interface and cogni6ve processing in working
memory influence what we perceive when using new
media.
• Selec6ve aRen6on requires us to select those items
which will enable us to succeed in our task from within
an informa)on‐rich environment.
• Divided aRen6on refers to the parallel or concurrent
informa)on processing opera)ons mobilized
simultaneously in order to accomplish the task.

59. Seven records of eye movements
by the same subject. Each record
lasted 3 minutes.
1) Free examina)on. Before
subsequent recordings, the
subject was asked to:
2) es)mate the material
circumstances of the family;
3) give the ages of the people;
4) surmise what the family had
been doing before the arrival
of the "unexpected visitor;”
5) remember the clothes worn
by the people;
6) remember the posi)on of
the people and objects in the
room;
7) es)mate how long the
"unexpected visitor" had
been away from the family
Yarbus, A. L. (1967).Eye movements during percep)on of complex objects, in L. A. Riggs,
ed., `Eye Movements and Vision', Plenum Press, New York, chapter VII, pp. 171‐196.

60. Working memory
• Working memory is based on the classic defini)on of
this memory structure (Baddeley, 1986; Paivio, 1986)
which the authors believe to be:
– (a) composed of two different systems for short‐term
informa6on processing (one for graphic informa)on and
one for verbal informa)on) and an aRen6onal component
which selects the informa6on to be processed,
– (b) endowed with a limited capacity to process new,
complex informa)on,
– (c) capable of having parts of its capacity freed up by
certain specific configura)ons of the mul)media
presenta)on

61. Components of working memory

62. Informa)on processing theory
explana)ons
– Cogni)ve theory of mul)media learning (Clark &
Paivio, 1991; Mayer, 2001)
– Cogni)ve load theory (Sweller, 2003)
– Model of text and graphics comprehension (1993;
Schnotz & Bannert, 2003);

63. Clark & Paivio (1991): Dual‐coding
theory
• Informa)on is processed through one of two generally
independent channels.
• One channel processes verbal informa)on such as text
or audio. The representa)ons of informa)on processed
by this system are known as logogens.
• The other channel processes nonverbal images such as
illustra)ons and sounds in the environment. The
representa)ons of informa)on processed by this
system are known as imagens.
• Referen6al processing is ac6va6on that goes across
the two representa6onal systems. Informa)on
processed through both channels has an addi6ve
effect on recall.

64. Clark & Paivio (1991): Dual‐coding
theory
Clark, J.M., & Paivio, A. (1991). Dual Coding Theory and Educa)on. Educa)onal
Psychology Review, 3, 149‐210.

65. Cogni)ve theory of mul)media
learning (R.Mayer)
• Subjects learning in a mul)media se•ng
mobilize three successive cogni6ve processes:
– selec6on (permi•ng the construc)on of the text
and graphic base),
– organiza6on (permi•ng the crea)on of models
based on the textual and graphic material)
– integra6on (permi•ng the establishment of
connec)ons between the corresponding events in
the models based on the verbal and graphic
material).
How do these cogni)ve processes influence new media ac)vi)es such as remixing/
percep)on of remixes or percep)on of aggregated and interac)ve new media?

66. Cogni)ve theory of mul)media
learning (R.Mayer)

67. A benefit can be gained from the dual coding of the informa)on

68. Cogni)ve load (Sweller)
• Cogni)ve load is the total amount of cogni)ve
ac)vity imposed on working memory at an
instance in )me.
On the table is plate, in the leS of plate there is a cup. On the right of the plate are
spoon and fork.
Chunking informa)on in working memory: Low and high interac)vity

69. Processing in working memory:
unconscious and conscious
• Joint ac)on of two dis)nct processes in
informa)on processing (Sweller, 2003):
– an automa6c process of "similarity processing"
which permits the automa)c retrieval of
informa)on from long‐term memory,
– and a controlled process for the processing of
differences, the role of which is to process new or
contradictory informa)on.

70. Cogni)ve cost of informa)on
processing
• Sweller (2003) believes that split aRen6on
occurs when subjects have to process mul)ple
sources of informa)on which they have to
integrate at the mental level in order to infer
the meaning from the presented material.
• The cogni)ve load theory has been used to
suggest that the processes of spliFng
aRen6on and mentally integra6ng the
material are cogni6vely costly

71. Principles derived from Cogni)ve
theory of mul)media learning
• The modality effect: elimina)ng the split ahen)on
effect improves learning by reducing the working
memory load (Jeung, Chandler & Sweller, 1997)
• The redundancy effect: The addi)on of redundant
informa)on impairs learning. The modality effect
should be used only if it is necessary to process
disparate informa)on in order to reduce working
memory load.
• The element interac6vity effect: the effects of
modality and redundancy are observed if and only if
the material for processing is cogni)vely demanding, if
it possesses a large number of interac)ng elements.

72. Principles derived from Cogni)ve
theory of mul)media learning
• The isolated interac6ng elements effect:
when a learner has to process a very complex
situa)on, processing can be facilitated by
ini)ally isola)ng the elements of which it
consists of
• The exper6se reversal effect: when dealing
with advanced learners, almost all the effects
described above cease to operate.

73. Does interface output influence how
users interact with it?
• When the system uses text or speech, users
tend to use speech.
• When the system uses graphics then users
tend to use poin6ng gestures on the touch
screen
• It seems that each modality has a kind of
influence power and when the modali6es are
combined their influence powers are
combined too.
hhp://www.limsi.fr/Individu/jps/research/daS/doc/09.A_INTERACT‐bellik.
23august09.pdf

74. Does interface output influence how
users interact with it?
hhp://www.limsi.fr/Individu/jps/research/daS/doc/09.A_INTERACT‐bellik.23august09.pdf
If the output is mixed media,
How do users interact with the systems?

75. Essay ques)ons
• Mixed media percep)on – how we may
perceive one new media applica)on?
– For this task choose one new media applica)on
that is largely based on mixed media informa)on.
– How can you explain the effect of new media on
people, using this new media applica)on as an
example?