Structured Interactive Scores: From a formal specification to a real-time capable implementation

1. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
“If I cannot draw it, I do not understand it”
–Anonymous.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 1

2. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Structured interactive scores: From a
structural description of a multimedia
scenario to a real-time capable
implementation with formal semantics
Mauricio TORO – LaBRI, Universit´e de Bordeaux.
Ph.D defense
Supervised by Myriam Desainte-Catherine and Camilo Rueda
September 25th 2012
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 2

3. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Some domains shaped by the arose of technology
Tape
Music
Electroacoustic
Music
Installations
Interactive
Multimedia
Experimental
Music
Non-linear
Multimedia
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 3

4. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. Examples of multimedia interaction
2. Problems with most existing tools
3. Solution: interactive scores
4. History of interactive scores
5. Tools for interactive scores
6. Related formalisms to interactive scores
7. Related tools to interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 4

5. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Contemporary Dance
./videos/theatre.mov
Minchoul synchronizes human gestes, video and sound in
real-time in cleaning (2009).
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 5

6. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Interactive multimedia installations
./videos/installation.mov
Torres uses the participation of the audience to co-author the
meaning of the installation in pollen (2008).
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 6

7. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Meta-instruments for physically disabled people
./videos/baopao.mov
The Bao-Pao1
is a meta-instrument. As an example, this
meta-instrument can control the start and end of notes.
1
http://www.bao-pao.com/index.php
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 7

8. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. Examples of multimedia interaction
2. Problems with most existing tools
3. Solution: interactive scores
4. History of interactive scores
5. Tools for interactive scores
6. Related formalisms to interactive scores
7. Related tools to interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 8

9. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Problems with most existing tools
1. No formal semantics [TDCR12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 9

10. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Problems with most existing tools
1. No formal semantics [TDCR12]
2. Time models are unrelated [DCAA12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 10

11. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Cue-Lists model
As an example, consider theater cue-list manager Qlab 2
.
2
http://figure53.com/qlab/
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 11

12. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Timeline model
As an example, consider sequencer Pro Tools 3
.
3
http://www.avid.com/US/resources/digi-orientation
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 12

13. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Time models are unrelated:
Cue Lists Vs. Timeline
Vs.
It is argued in [DCAA12] that having both time models
temporally related will bring new possibilities for the creation
of interactive multimedia.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 13

14. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Problems with most existing tools
1. No formal semantics [TDCR12]
2. Time models are unrelated [DCAA12]
3. Schedulers are not appropriate for real-time [TDCC12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 14

15. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Scheduler problem:
aggravated under high cpu load
• Most tools do not allow the users to interact with them
without letting them experience noticeable delays
• Karplus-Strong is a well-known digital signal processing
algorithm to simulate the sound of metallic strings
• A Karplus-Strong arpeggio implementation in Pure Data4
.
./sounds/barry-loaded.mov
• A Karplus-Strong arpeggio implementation using Faust
([OFL04]) and interactive scores [TDCC12].
./sounds/faust-loaded.mov
Results from [TDCC12].
4
Colin Barry’s implementation found at www.loomer.co.uk
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 15

16. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Problems with most existing tools
1. No formal semantics [TDCR12]
2. Time models are unrelated [DCAA12]
3. Schedulers are not appropriate for real-time [TDCC12]
4. No hierarchy (e.g., movements, parts, measures, motives
and notes) [Vic04]
5. No unified model for conditional branching and temporal
relations [Vic03, Ran09, All09, Vic11]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 16

17. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. Examples of multimedia interaction
2. Problems with most existing tools
3. Solution: interactive scores
4. History of interactive scores
5. Tools for interactive scores
6. Related formalisms to interactive scores
7. Related tools to interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 17

18. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Solution: Interactive Scores
Green (g)
Red (r)
Sound (u)
Lights (l)
a
b
Scenario (c)
d
overlaps
starts
starts
ends ends
ends
time
A formalism for interactive multimedia developed at LaBRI:
[BDC01, DCB03, ADC05, AADCR06, AADC07, ADCLA08,
AADC08, Ran09, All09, OR09, BAM+
09, Sar08, ABM+
10],
[TDCB10, TDC10, Tor10], [MADC11],
[ADCT11, TDCC12, TDCR12, Tor12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 18

19. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. Examples of multimedia interaction
2. Problems with most existing tools
3. Solution: interactive scores
4. History of interactive scores
5. Tools for interactive scores
6. Related formalisms to interactive scores
7. Related tools to interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 19

20. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
History of interactive scores
• Boxes: A software to model a hierarchy and temporal
constraints [BDC01]
• A model of interactive scores [ADC05]
• Hierarchical Time Stream Petri nets (hts petri nets)
semantics for interactive scores [AADC07]
• An extension of interactive scores with temporal
reductions [ADCLA08]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 20

21. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
History of interactive scores
• First non-deterministic timed concurrent constraint (ntcc)
model of interactive scores [AADCR06]
• A model change the hierarchy of the temporal objects
during execution [OR09]
• A model to represent conditional branching and temporal
relations separately [Ran09, All09]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 21

22. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. Examples of multimedia interaction
2. Problems with most existing tools
3. Solution: interactive scores
4. History of interactive scores
5. Tools for interactive scores
6. Related formalisms to interactive scores
7. Related tools to interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 22

23. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Existing tools for interactive scores
• Boxes [BDC01]
• Iscore developed for OpenMusic [AADC08]
• Acousmouscribe
• Stand-alone i-score (currently maintained by LaBRI)
• Prototypes using Ntccrt, a real-time interpreter for ntcc
[TDCB10, TDC10, TDCC12]
• Virage [BAM+
09, ABM+
10, MADC11]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 23

24. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Virage: An implementation of interactive scores
./videos/virage.mov
Software Virage developed under Virage project [BAM+
09].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 24

25. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. Examples of multimedia interaction
2. Problems with most existing tools
3. Solution: interactive scores
4. History of interactive scores
5. Tools for interactive scores
6. Related formalisms to interactive scores
7. Related tools to interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 25

26. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Related formalisms to interactive scores
• Globally Asynchronous, Locally Synchronous (gals)
[TGL07]
• Hierarchical time stream Petri nets (hts petri nets)
[SSSW95]
• Temporal constraints: qualitative [All83], quantitative
[Gen98] and combination of both [Mei96]
• Concurrent constraint programming (ccp) calculi
[Sar92, SJG94, Sar08, OV08].
• Non-deterministic timed concurrent constraint
programming (ntcc) [NPV02]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 26

27. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Non-deterministic Timed Concurrent Constraint
Programming (ntcc)
• Process calculus to model reactive systems with discrete
time, non-determinism, partial information and
asynchrony [NPV02, Val02]
• Includes a temporal logic to verify properties [NPV02]
• A logic formula can be translated into a process and the
strongest post-condition5
of a process into a B¨uchi
automaton6
[Val05]
5
Sequences that the process can output under any environment.
6
A B¨uchi automaton recognizes sequences of infinite length.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 27

28. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Non-deterministic Timed Concurrent Constraint
Programming (ntcc)
• Has been used for the verification and simulation of other
interactive multimedia systems [RV02, RV05, RV04]
• Has been used for interactive scores
[AADCR06, Sar08, OR09]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 28

29. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. Examples of multimedia interaction
2. Problems with most existing tools
3. Solution: Interactive scores
4. History of interactive scores
5. Tools for interactive scores
6. Related formalism to interactive scores
7. Related tools to interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 29

30. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Related tools to interactive scores
• Score following (e.g., Antescofo [Con08, ECGJ11])
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 30

31. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
An example of score following
./videos/antescofo.mov
Example of the score following system Antescofo.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 31

32. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Related tools to interactive scores
• Score Following (e.g., Antescofo [Con08, ECGJ11])
• Asynchronous Dataflow Languages (e.g., Max/msp
[PAZ98] and Pure Data[Puc96])
• Synchronous Dataflow Languages (e.g., CSound, Faust
[OFL04], Esterel, Signal [GLGB87] and Lustre)
• Machine Improvisation (e.g., OMax [ABC+
06] and
Continuator [Pac02])
• Computer Assisted Composition (e.g., OpenMusic’s
Maquettes [BAA11])
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 32

33. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Introduction
1. We saw some examples of multimedia interaction
2. There are problems with most existing tools
3. Our solution is interactive scores (is)
4. is is a mature formalism
5. There are several tools for is
6. Score following and Maquettes are related to is
7. gals, htspn and ntcc are related to is
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 33

34. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Outline of the presentation
1. Hierarchical Model
2. Time Conditional Model
3. Signal Processing Model
4. Simulation
5. Verification
6. Summary and Conclusions
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 34

35. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Hierarchical model of interactive scores
1. Structural definition [TDCR12]
2. Abstract semantics [TDCR12]
3. Operational semantics [TDCR12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 35

36. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Structural definition: Temporal objects
Green (g)
Red (r)
Sound (u)
time
Temporal objects are aligned on a timeline, thus they have
nominal start time and nominal duration.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 36

37. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Structural definition: Hierarchy of temporal objects
Green (g)
Red (r)
Sound (u)
Lights (l)
Scenario (c)
A temporal object can contain other objects to constraint the
execution start and end times of its children. The hierarchical
model has been previously discussed in [BDC01, All09]. We
presented a new formalization in [TDCR12].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 37

38. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Structural definition: Interactive objects
Green (g)
Red (r)
Sound (u)
Lights (l)
a
b
Scenario (c)
d
Interactive objects (gray circles) allow to control the start or
end of other temporal objects.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 38

39. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Structural definition: Temporal relations
Green (g)
Red (r)
Sound (u)
Lights (l)
a
b
Scenario (c)
d
overlaps
starts
starts
ends ends
ends
time
Temporal relations constraint the set of possible execution
times and durations of the temporal objects.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 39

40. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Structural definition
Green (g)
Red (r)
Sound (u)
Lights (l)
a
b
Scenario (c)
d
∆r
∆g
∆u
∆l
∆c
{0}
{0}
{0}
{0}
{0}
[0, ∞)
[0, ∞) [0, ∞)
Durations and temporal relations can be represented by
point-to-point temporal relations labeled by integer intervals
[Mei96].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 40

41. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Hierarchical model of interactive scores
1. Structural definition
2. Abstract semantics
3. Operational semantics
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 41

42. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Timed Event Structures (tes)
• Langerak’s timed event structures is a mathematical
model to represent systems with non-determinism,
real-time and concurrency [BKL98].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 42

43. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Timed Event Structures (tes)
a
b
c d
e
Figure : Circles are events.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 43

44. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Timed Event Structures (tes)
a
b
c d
e
[0,7]
[0,5]
[4,4]
Figure : Circles are events. Full-line arrows represent event delays.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 44

45. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Timed Event Structures (tes)
a
b
c d
e
[0,7]
[0,5]
[4,4]
Figure : Circles are events. Full-line arrows represent event delays.
Pointed-line arrows represent conflicts; when they have no
direction, the conflict is symmetrical.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 45

46. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: Temporal objects
sc
su
sb
sl
sr
sg
sa
sd
ec
eu
er
eg
el
STRUCTURAL DEFINITION
tes semantics for interactive scores [TDCR12].
Colored circles represent events of static temporal objects.
Gray circles represent events of interactive objects.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 46

47. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: Duration of temporal objects
∆r
∆g
∆u
∆c
∆l
sc
su
sb
sl
sr
sg
sa
sd
ec
eu
er
eg
el
STRUCTURAL DEFINITION
Durations of the temporal objects are represent by event
delays.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 47

48. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: Temporal relations
0 0
0
0
(0,∞
)(0,∞
)
(0,∞
)
sc
su
sb
sl
sr
sg 0
sa
sd
ec
eu
er
eg
STRUCTURAL DEFINITION
Red arrows are event delays of the form ei
[0,∞)
→ ej , derived
from the hierarchy. Black arrows are event delays derived from
the explicit temporal relations.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 48

49. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: Interactive scores
0 0
0
0
(0,∞
)(0,∞
)
(0,∞
)
∆r
∆g
∆u
∆c
∆l
sc
su
sb
sl
sr
sg 0
sa
sd
ec
eu
er
eg
STRUCTURAL DEFINITION
tes semantics is obtained by combining the events, the event
delays from the duration of temporal objects, and the event
delays from the temporal relations.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 49

50. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Hierarchical model of interactive scores
1. Structural definition
2. Abstract semantics
3. Operational semantics
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 50

51. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Operational semantics: Intuition
sc
su
sb
sl
sr
sg
sa
sd
ec
eu
er
eg
0 0
0
0
(0
,∞
)(0,∞)
(0
,
∞
)
∆r
∆g
∆u
∆c
∆l
sc
su
sb
sl
sr
sg 0
sa
sd
ec
eu
er
eg
CONSTRAINT STORESTRUCTURAL DEFINITION
x
Agent that represents
the behavior of point x
• Ntcc semantics for interactive scores were introduced in
[AADCR06] and later formalized in [TDCR12].
• Operational semantics respects the event structures
semantics
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 51

52. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Hierarchical model of interactive scores
1. The structural definition is composed by temporal objects
and point-to-point temporal relations
2. Abstract semantics are defined in timed event structures
3. Operational semantics are defined in ntcc
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 52

53. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Outline of the presentation
1. Hierarchical Model
2. Time Conditional Model
3. Signal Processing Model
4. Simulation
5. Verification
6. Summary and Conclusions
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 53

54. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Time conditional model of interactive scores
1. Structural definition [TDCB10, TDC10, ADCT11]
2. Abstract semantics
3. Operational semantics [TDCB10, TDC10, ADCT11]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 54

55. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Time conditional model: Structural definition
Verse 1 (v)
Chorus (ch)
Verse 3 (x)
Verse 2 (w)
Verse 4 (y)
Chorus (c)
∆v
∆w∆ch
∆c
∆ = {0}
∆
=
{0}
∆ = {0}
∆ = [2, 5]
∆ = [1, 4]
ϕ > 0 ∨ ψ ≤ 3
φ
>
0 ∧
ψ
≤
5
d
∆x ∆y
∆a
∆b
V = {φ, ψ, ϕ}
main (m)
∆ = {0}
ϕ > 0 ∨ ψ ≤ 3
V = {φ, ψ, ϕ}
Timed conditional relation
duration
condition
Variables
Interactive object Choice
The model was introduced in [TDCB10, TDC10, ADCT11].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 55

56. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Time conditional model of interactive scores
1. Structural definition
2. Abstract semantics
3. Operational semantics
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 56

57. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: “Visible”-action events
sa
sb
ea ecscsv ev
d
Structural Definition
Colored circles are the “visible”-action events.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 57

58. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: “Invisible”-action events
sa
sb
eb
ea ecscsv ev
d s
a
s
b
Structural Definition
Black circles are the “invisible”-action events.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 58

59. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: Temporal relations
0
0
sa
sb
eb
ea ecscsv ev
d s
a
s
b
Structural Definition
Temporal relations are translated to event delays. Event delays
are represented as arrows. For simplicity, the arrows have no
labels.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 59

60. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: Conflicts
0
0
sa
sb
eb
ea ecscsv ev
d s
a
s
b
Structural Definition
Dashed lines represent symmetrical conflicts: It means that
two events are mutually exclusive.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 60

61. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
tes semantics: More conflicts
0
0
sa
sb
eb
ea ecscsv ev
d s
a
s
b
Structural Definition
Dashed-line arrows represent asymmetrical conflicts: It means
that one event must happen before another or it will not
happen.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 61

62. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Why visible and invisible events?
• Because it is a simple way to represent conditional
branching in a single bi-dimensional plane.
A
b
s
t
r
a
c
t
i
o
n
Tim
e
(Im
plication) Values
Paralelism (and)
Alternative (or)
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 62

63. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
David Janin’s dimensions
A
b
s
t
r
a
c
t
i
o
n
Tim
e
(Im
plication)
Values
Paralelism (and)
Alternative (or)
There are several dimensions in multimedia interaction7
.
7
http://www.labri.fr/perso/janin/
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 63

64. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Time conditional model of interactive scores
1. Structural definition
2. Abstract semantics
3. Operational semantics
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 64

65. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Time conditional model: Operational semantics
b
a
c
when
finish
unless finish
∆b = 3 ∆c = 2
d ∆d = 1
∆ = 0
∆ = 1
∆ = 1
∆ = 1
∆ = 1
∆ = 0
V = {finish}
Operational semantics for the conditional branching extension
with loops were introduced in [TDCB10, TDC10, ADCT11].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 65

66. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Time conditional model of interactive scores
1. Structural definition is given for scores with and without
loops
2. Abstract semantics are defined in timed event structures
for scores without loops
3. Operational semantics are defined in ntcc, support loops,
but are not related to the abstract semantics, as it was
the case in the hierarchical model
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 66

67. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Outline of the presentation
1. Hierarchical Model
2. Time Conditional Model
3. Signal Processing Model
4. Simulation
5. Verification
6. Summary and Conclusions
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 67

68. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Interactive scores with signal processing
1. Structural definition [TDCC12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 68

69. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Interactive Scores with signal processing
Karplus (k1)
Karplus (k2)
Karplus (k3)
Output
An extension in which ntcc and the synchronous programming
language with formal semantics Faust [OFL04] interact.
Introduced in [TDCC12] and it is inspired from [All09].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 69

70. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Interactive Scores: Temporal objects
Karplus (k1)
Karplus (k2)
Karplus (k3)
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
Output
Temporal objects may contain a Faust audio process, before
they used to control external audio processes. Input8
and
Output streams are also temporal objects.
8
In this score there is no audio input.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 70

71. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Interactive Scores: Temporal relations
Karplus (k1)
Karplus (k2)
Karplus (k3)
100smp
[2s, 4s]
[0s, 0s]
[0s, 0s]
Output
In addition to the usual temporal relations, dashed-line arrows
represent high-precision temporal relations.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 71

72. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Interactive Scores: Dataflow relations
Karplus (k1)
Karplus (k2)
Karplus (k3)
Output
White-headed arrows represent dataflow relations: To describe
how the sound is transferred from one object to another.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 72

73. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Interactive Scores: Interactive objects
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b
Output
Interactive objects are represented (and behave) as usual.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 73

74. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Extension of Interactive Scores
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b
[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Temporal object
Temporal relation
High-precision
temporal relation
Dataflow relation
Interactive object
Interactive score
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 74

75. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Outline of the presentation
1. Hierarchical Model
2. Time Conditional Model
3. Signal Processing Model
4. Simulation
5. Verification
6. Summary and Conclusions
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 75

76. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Simulation of Interactive Scores
1. Ntccrt [TAAR09]
2. Results on the conditional branching model [TDC10]
3. Results on the signal processing extension [TDCC12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 76

77. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Simulation: Interpreter for ntcc
OpenMusic
interface
Ntccrt
compiler
Pure Data
external
Max/Msp
external
Common Lisp
interface
C++
interface Stand-alone
program
User
Programmer
Ntccrt9
is a simulation tool for ntcc introduced in [TAAR09].
Experience on the usage of Ntccrt is described in [ORS+
11].
9
http://sourceforge.net/projects/ntccrt/
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 77

78. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Simulation: Example using Pure Data
Structural Definition
“Bang” objects are controlled by the Ntccrt plugin [TDC10].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 78

79. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Simulation
1. Ntccrt
2. Results on the conditional branching model
3. Results on the signal processing extension
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 79

80. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Results on the conditional branching extension
0
5
10
15
20
25
30
35
40
45
50
0 200 400 600 800 1000 1200 1400 1600 1800
Average dura*on of a *me unit (ms)
Number of points in the Interac*ve Score
Stress test to compute the average duration of a discrete time
unit of the simulation of a conditional branching score using
Ntccrt [TDC10].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 80

81. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Simulation
1. Ntccrt
2. Results on the conditional branching model
3. Results on the signal processing extension
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 81

82. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Results on the signal processing extension
Average relative jitter: The average time difference between
the expected starting time of each string, with respect to the
first string of an arpeggio, and the time obtained during
execution.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 82

83. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Results on the signal processing extension
Implementation 85% cpu load 3% cpu load
Using object block˜ 1 in Pd 7991 ms 1 ms
Using object z˜ 1 in Pd 9231 ms 2 ms
Using Faust object in Pd 0.5 ms 0.5 ms
Table : Average relative jitter of each string of an arpeggio with
respect to the beep. All implementations use the Karplus-Strong
algorithm to simulate the sound of the strings [TDCC12].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 83

84. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Results on the signal processing extension
In some cases –such as programs with feedback loops– our
interactive scores implementation using Faust outperforms
Pure Data under high-cpu load. Pure data’s performance is
unacceptable, in such cases, for real-time interaction.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 84

85. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Simulation
1. Our models are simulated using Ntccrt, a real-time
capable interpreter for ntcc
2. Results on the conditional branching model show that we
can execute up to 400 temporal objects
3. Results on the signal processing extension show that
interactive scores outperforms Pure Data under high-cpu
load
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 85

86. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Outline of the presentation
1. Hierarchical Model
2. Time Conditional Model
3. Signal Processing Model
4. Simulation
5. Verification
6. Summary and Conclusions
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 86

87. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Verification
1. Motivation
2. Constraint linear-time temporal logic (cltl) [NPV02]
3. Model checking algorithm
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 87

88. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Verification of interactive scores
• Existing semantics for interactive scores were given in
hts petri nets.
• Existing verification tools for time Petri nets, such as
Rom´eo [GLMR05] and Tapaal [BJS09], cannot be used to
verify hts petri nets [SSSW95] because hts petri nets
cannot be translated to time Petri nets [BD99].
• Existing verification tools for linear-time logic, such as :
Uppaal [BDL+
01], Kronos [BDM+
98] and Spin [Hol97],
cannot be used directly for ntcc.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 88

89. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Verification of interactive scores
• Results on bisimilarity for ccp [Ari10, ABPV12] are quite
promising, but not yet applicable to ntcc.
• Strongest post-condition of a process can be translated
into a B¨uchi automaton [Val05], but using classic
Schimpf’s algorithm [SMS09] is intractable because the
encoding heavily relies on the complement of such
automata [Saf88].
• We proposed a bounded-time finite-state-automata-based
model checker for ntcc, ntccmc10
.
10
http://sourceforge.net/projects/ntccmc/
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 89

90. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Verification
1. Motivation
2. Constraint linear-time temporal logic (cltl)
3. Model checking algorithm
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 90

91. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
cltl properties
Examples of constraint linear-time logic (cltl) [NPV02]
formulae are
• “always the constraint pitch = 60 can be deduced from
the output store”, namely (pitch = 60); and
• “eventually, both object a and object b are launched at
the same time”, namely ♦(launcha ˙∧launchb).
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 91

92. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Verification
1. Motivation
2. Constraint linear-time temporal logic (cltl)
3. Model checking algorithm
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 92

93. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
YES
NO
Is the
language
empty?
Bounded-time automata-based model checking algorithm for ntcc
A. intersection
P → Pt
F → Ft Ft
Pt
Pt → A
Pt → A
Ft → Pt Qt A
A
BF
P
t A B
B
Does a a ntcc process P
satisfy a CLTL formula F
for a bounded time t ?
Adaptation of classic model checking algorithm for linear-time
logic [SMS09] to ntcc, based on the encoding of ntcc to B¨uchi
automata given by Valencia et al. [Val05].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 93

94. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
Bounded-time automata-based model checking algorithm for ntcc
ntcc Process
CLTL formulaF
P
t Number of discrete time units
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 94

95. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
YES
NO
Does a ntcc process P
satisfy a CLTL formula F
for a bounded time t ?
Bounded-time automata-based model checking algorithm for ntcc
P
F
ntcc Process
CLTL formula
P → Pt
P → Pt
P to bounded-time P
F to bounded-time FF → Ft
F → FtF
P
t
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 95

96. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
YES
NO
Bounded-time automata-based model checking algorithm for ntcc
P
F
ntcc Process
CLTL formula
P → Pt
P → Pt
P to bounded-time P
F to bounded-time FF → Ft
F → Ft Ft
Pt
Ft → Pt
Ft → Pt Qt
Bounded-time formula to
bounded-time process
F
P
t
Does a a ntcc process P
satisfy a CLTL formula F
for a bounded time t ?
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 96

97. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
YES
NO
Bounded-time automata-based model checking algorithm for ntcc
P
F
ntcc Process
CLTL formula
P → Pt
P → Pt
P to bounded-time P
F to bounded-time FF → Ft
F → Ft Ft
Pt
Bounded-time process to
deterministic automaton (A)
Pt → A
Pt → A
Pt → A
Ft → Pt
Ft → Pt Qt
Bounded-time formula to
bounded-time process
A
BF
P
t
Does a a ntcc process P
satisfy a CLTL formula F
for a bounded time t ?
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 97

98. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
YES
NO
Bounded-time automata-based model checking algorithm for ntcc
P
F
ntcc Process
CLTL formula
P → Pt
P → Pt
P to bounded-time P
F to bounded-time FF → Ft
F → Ft Ft
Pt
Bounded-time process to
deterministic automaton (A)
Pt → A
Pt → A
Pt → A
Ft → Pt
Ft → Pt Qt
Bounded-time formula to
bounded-time process
A
A
A. complement
A
BF
P
t
B
A Automaton
Does a a ntcc process P
satisfy a CLTL formula F
for a bounded time t ?
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 98

99. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
YES
NO
Bounded-time automata-based model checking algorithm for ntcc
A. intersection
P
F
ntcc Process
CLTL formula
P → Pt
P → Pt
P to bounded-time P
F to bounded-time FF → Ft
F → Ft Ft
Pt
Bounded-time process to
deterministic automaton (A)
Pt → A
Pt → A
Pt → A
Ft → Pt
Ft → Pt Qt
Bounded-time formula to
bounded-time process
A
A
A. complement
A
BF
P
t A B
B
A Automaton
Does a a ntcc process P
satisfy a CLTL formula F
for a bounded time t ?
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 99

100. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Automatic verification of ntcc
YES
NO
Is the
language
empty?
Bounded-time automata-based model checking algorithm for ntcc
A. intersection
P
F
ntcc Process
CLTL formula
P → Pt
P → Pt
P to bounded-time P
F to bounded-time FF → Ft
F → Ft Ft
Pt
Bounded-time process to
deterministic automaton (A)
Pt → A
Pt → A
Pt → A
Ft → Pt
Ft → Pt Qt
Bounded-time formula to
bounded-time process
A
A
A. complement
A
BF
P
t A B
B
A Automaton
Does a a ntcc process P
satisfy a CLTL formula F
for a bounded time t ?
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 100

101. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Verification
1. Existing tools cannot be used directly to verify interactive
scores and we want to prove properties of interactive
scores automatically
2. In particular, constraint linear-time temporal logic (cltl)
properties
3. We presented our bounded-time automata-based model
checking algorithm for ntcc, ntccMC, for cltl properties
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 101

102. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Outline of the presentation
1. Hierarchical Model
2. Time Conditional Model
3. Signal Processing Model
4. Simulation
5. Verification
6. Conclusions
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 102

103. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conclusions
1. Summary
2. Solutions to the problem statements [TDCC12, TDCR12]
3. Comparison w.r.t previous models of interactive scores
[TDCR12]
4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab
and Protools [TDCC12]
5. Future work directions
6. Ongoing projects
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 103

104. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Summary
• A Formal model for multimedia interaction that combines
• timeline model
• cue-list model [TDCR12]
• An extension for conditional branching
[TDCB10, TDC10, ADCT11]
• An extension for audio processing [Tor10, TDCC12]
• A simulation tool [TAAR09, ORS+
11]
• A verification tool [Tor10]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 104

105. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Contributions
• 2 book chapters in “Constraint Programming in Music”
[ADCT11, ORS+
11]
• 1 journal article in “Mathematics and Music” [TDCR12]
• 2 papers in “Sound and Music Computing” conference
[TDC10, TDCC12]
• 1 extended abstract in “International Conference on Logic
Programming” [Tor10]
• 1 paper in “Journ´ees d’Informatique Musicale” [TDCB10]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 105

106. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conclusions
1. Summary
2. Solutions to the problem statements
[TDCC12, TDCR12]
3. Comparison w.r.t previous models of interactive scores
[TDCR12]
4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab
and Protools [TDCC12]
5. Future work directions
6. Ongoing projects
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 106

107. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Solutions to the problem statements
• Time models are unrelated
• In interactive scores, time models are temporally related
Green (g)
Red (r)
Sound (u)
Lights (l)
a
b
Scenario (c)
d
overlaps
starts
starts
ends ends
ends
time
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 107

108. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Solutions to the problem statements
• Schedulers are not appropriate for real-time
• Using interactive scores and Faust, it is possible to have
real-time interaction even under high cpu load
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 108

109. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Solutions to the problem statements
• No hierarchy
• Interactive scores provide a hierarchy of temporal objects
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 109

110. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Solutions to the problem statements
• No formal semantics
• We defined abstract and operational semantics for
interactive scores
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 110

111. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Solutions to the problem statements
• No unified model for conditional branching and temporal
relations
• We provide a unified model for conditional branching and
temporal relations in which all branches have the same
set of possible durations
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 111

112. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conclusions
1. Summary
2. Solutions to the problem statements [TDCC12, TDCR12]
3. Comparison w.r.t previous models of interactive
scores [TDCR12]
4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab
and Protools [TDCC12]
5. Future work directions
6. Ongoing projects
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 112

113. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Comparison w.r.t previous models of interactive
scores
• Model to change the hierarchy during performance [OR09]
• Previous model developed at LaBRI [All09]
• Our models [TDC10, TDCB10, TDCR12, TDCC12]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 113

114. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Model to change the hierarchy during performance
• Provides an elegant solution to change the structure of
the hierarchy during performance
• Has not been implemented
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 114

115. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Previous model developed at LaBRI
• Supports different mechanisms of temporal reductions
• Average duration of a discrete time unit outperforms our
simulations
• Durations are intervals are limited to {0}, (0, ∞] and
[0, ∞]
• Conditional branching and temporal relations are
presented separately
• Processes associated are meant to be handled outside the
formalism; in the implementation as external programs
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 115

116. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Our models
• Abstract semantics to describe the execution traces
• Arbitrary integer intervals in the operational semantics
• Unified model for conditional branching and temporal
relations
• Automatic verification
• Does not allow temporal reductions
• Simulation tool is a prototype; it is not user friendly
• Average duration of time units is slower than in the
implementation of previous model
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 116

117. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conclusions
1. Summary
2. Solutions to the problem statements [TDCC12, TDCR12]
3. Comparison w.r.t previous models of interactive scores
[TDCR12]
4. Comparison w.r.t Csound, Pure Data, Max, Live,
QLab and Protools [TDCC12]
5. Future work directions
6. Ongoing projects
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 117

118. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Comparison w.r.t Csound, Pure Data, Max, Live,
QLab and Protools
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
• Qlab and Ableton Live do not allow to model delays of
100 samples
• Protools does not allow interaction
In all the programs mentioned above, it is very hard to
synchronize processes whose durations are integer intervals
such as duration ∈ [2, 4].
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 118

119. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conclusions
1. Summary
2. Solutions to the problem statements [TDCC12, TDCR12]
3. Comparison w.r.t previous models of interactive scores
[TDCR12]
4. Comparison w.r.t Csound, Pure Data Max, Live, QLab
and Protools [TDCC12]
5. Future work directions
6. Ongoing projects
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 119

120. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Future work directions
• Event structures semantics for scores with loops
• Model interactive theater performances
• Model interactive museum exhibitions
• Verify scores in which ntcc and Faust interact
• Model music pieces combining the timeline and cue-list
model
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 120

121. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Music pieces combining both time models
Baltazar et al. used i-score to compose miniature # 1 (2012).
I-score controls the smoke, fan, sound and lights.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 121

122. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Music pieces combining both time models
./videos/pascal.mp4
Baltazar et al. needed to have objects with flexible duration
because the behavior of the smoke is unpredictable.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 122

123. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conclusions
1. Summary
2. Solutions to the problem statements [TDCC12, TDCR12]
3. Comparison w.r.t previous models of interactive scores
[TDCR12]
4. Comparison w.r.t Csound, Pure Data, Max, Live, QLab
and Protools [TDCC12]
5. Future work directions
6. Ongoing projects
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 123

124. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Ongoing projects
• Project Interactivity in the Writing of Time and
Interactions (inedit) (2012-2015) founded by the french
national research agency (anr).
• Project Robust theories for Emerging Applications in
Concurrency Theory: Processes and Logic Used in
emergent Systems(react+) (2011-2013) founded by
the colombian national research agency (Colciencias).
• Project ossia (2012-2015) founded by the anr.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124

125. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Ongoing projects
• Project Interactivity in the Writing of Time and
Interactions (inedit) (2012-2015) founded by the french
national research agency (anr).
• Project Robust theories for Emerging Applications in
Concurrency Theory: Processes and Logic Used in
emergent Systems(react+) (2011-2013) founded by
the colombian national research agency (Colciencias).
• Project ossia (2012-2015) founded by the anr.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124

126. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Ongoing projects
• Project Interactivity in the Writing of Time and
Interactions (inedit) (2012-2015) founded by the french
national research agency (anr).
• Project Robust theories for Emerging Applications in
Concurrency Theory: Processes and Logic Used in
emergent Systems(react+) (2011-2013) founded by
the colombian national research agency (Colciencias).
• Project ossia (2012-2015) founded by the anr.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 124

127. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
M E R C I.
Avez-vous des questions?
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 125

128. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Outline of the presentation
1. Hierarchical Model
2. Time Conditional Model
3. Signal Processing Model
4. Simulation
5. Verification
6. Summary and Conclusions
7. Extra
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

129. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Antoine Allombert, G´erard Assayag, and Myriam
Desainte-Catherine.
A system of interactive scores based on petri nets.
In Proc. of SMC ’07, Athens, Greece, 2007.
Antoine Allombert, G´erard Assayag, and Myriam
Desainte-Catherine.
Iscore: a system for writing interaction.
In Proc. of 3rd International Conference on Digital
Interactive Media in Entertainment and Arts (DIMEA)
’08, pages 360–367, New York, NY, USA, 2008. ACM.
Antoine Allombert, G´erard Assayag,
M. Desainte-Catherine, and Camilo Rueda.
Concurrent constraint models for interactive scores.
In Proc. of Sound and Music Computing (SMC) ’06,
Marseille, France, 2006.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

130. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
G´erard Assayag, Georges Bloch, Marc Chemillier, Arshia
Cont, and Shlomo Dubnov.
Omax brothers: a dynamic topology of agents for
improvization learning.
In Proc. of the 1st ACM workshop on Audio and music
computing multimedia (AMCMM) ’06, pages 125–132,
New York, NY, USA, 2006. ACM.
Antoine Allombert, Pascal Baltazar, Rapha¨el Marczak,
Myriam Desainte-Catherine, and Laurent Garnier.
Designing an interactive intermedia sequencer from users
requirements and theoretical background.
In Proc. of International Computer Music Conference
(ICMC) ’10, 2010.
A. Aristizabal, F. Bonchi, L. Pino, and F. Valencia.
Partition refinement for bisimilarity in ccp.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

131. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
In Proc. of the 27th ACM Symposium On Applied
Computing (SAC 2012), 2012.
Antoine Allombert and Myriam Desainte-Catherine.
Interactive scores: A model for specifying temporal
relations between interactive and static events.
In Journal of New Music Research, 2005.
Antoine Allombert, Myriam Desainte-Catherine,
J. Larralde, and G´erard Assayag.
A system of interactive scores based on qualitative and
quantitative temporal constraints.
In Proc. of 4th International Conference on Digital Arts
(Artech) ’08, Porto, Portugal, 2008. The Artech
International Association.
Antoine Allombert, Myriam Desainte-Catherine, and
Mauricio Toro.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

132. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Modeling temporal constrains for a system of interactive
score.
In G´erard Assayag and Charlotte Truchet, editors,
Constraint Programming in Music, chapter 1, pages 1–23.
Wiley, 2011.
James F. Allen.
Maintaining knowledge about temporal intervals.
Communication of ACM, 26, 1983.
Antoine Allombert.
Aspects temporels d’un syst`eme de partitions num`eriques
interactives pour la composition et l’interpr´etation.
PhD thesis, Universit´e de Bordeaux, November 2009.
Andr´es Aristiz´abal.
Bisimilarity in concurrent constraint programming.
In 26th International Conference on Logic Programming
(ICLP) 2010, 2010.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

133. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Jean Bresson, Carlos Agon, and G´erard Assayag.
Openmusic: visual programming environment for music
composition, analysis and research.
In Proceedings of the 19th ACM international conference
on Multimedia, MM ’11, pages 743–746, New York, NY,
USA, 2011. ACM.
P. Baltazar, A. Allombert, R. Marczak, J.M. Couturier,
M. Roy, A. S`edes, and M. Desainte-Catherine.
Virage : Une reflexion pluridisciplinaire autour du temps
dans la creation numerique.
In in Proc. of JIM, 2009.
Marc Boyer and Michel Diaz.
Non equivalence between time petri nets and time stream
petri nets.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

134. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
In Proc. of the The 8th International Workshop on Petri
Nets and Performance Models, number 98 in 1,
Washington, DC, USA, 1999. IEEE Computer Society.
Anthony Beuriv´e and Myriam Desainte-Catherine.
Representing musical hierarchies with constraints.
In 7th International Conference on Principles and Practice
of Constraint Programming, Musical Constraints
Workshop, Paphos, 2001.
Gerd Behrmann, Alexandre David, Kim G. Larsen, Oliver
M¨oller, Paul Pettersson, and Wang Yi.
Uppaal - present and future.
In Proc. of 40th IEEE Conference on Decision and
Control. IEEE Computer Society Press, 2001.
Marius Bozga, Conrado Daws, Oded Maler, Alfredo
Olivero, Stavros Tripakis, and Sergio Yovine.
Kronos: A model-checking tool for real-time systems.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

135. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
In Anders Ravn and Hans Rischel, editors, Formal
Techniques in Real-Time and Fault-Tolerant Systems,
volume 1486 of Lecture Notes in Computer Science, pages
298–302. Springer Berlin / Heidelberg, 1998.
Joakim Byg, Kenneth Yrke Jørgensen, and Jiˇr´ı Srba.
Tapaal: Editor, simulator and verifier of timed-arc petri
nets.
In Proceedings of the 7th International Symposium on
Automated Technology for Verification and Analysis
(ATVA) ’09, pages 84–89, Berlin, Heidelberg, 2009.
Springer-Verlag.
Christel Baier, Joost-Pieter Katoen, and Diego Latella.
Metric semantics for true concurrent real time.
In Proc. of International Conference on Automata,
Languages and Programming (ICALP) ’98, Berlin,
Germany, 1998. Springer.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

136. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Arshia Cont.
Antescofo: Anticipatory synchronization and control of
interactive parameters in computer music.
In Proc. of ICMC ’08, 2008.
Myriam Desainte-Catherine, Antoine Allombert, and
G´erard Assayag.
Towards a hybrid temporal paradigm for musical
composition and performance: The case of musical
interpretation.
Computer Music Journal, To appear in fall, 2012.
M. Desainte-Catherine and N. Brousse.
Towards a specification of musical interactive pieces.
In Proc. of the XIX Colloquium on Musical Informatics
(CIM), Firenze, Italy, 2003.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

137. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Jos´e Echeveste, Arshia Cont, Jean-Louis Giavitto, and
Florent Jacquemard.
Formalisation des relations temporelles entre une partition
et une performance musicale dans un contexte
d’accompagnement automatique : Accompagnement
musical automatique.
In Journal Europ´een des Syst`emes Automatis´es, 2011.
Rosella Gennari.
Temporal resoning and constraint programming - a survey.
CWI Quaterly, 11:3–163, 1998.
Thierry Gautier, Paul Le Guernic, and L¨oic Besnard.
Signal: A declarative language for synchronous
programming of real-time systems.
In Proc. of FPCA ’87, 1987.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

138. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Guillaume Gardey, Didier Lime, Morgan Magnin, and
Olivier (H.) Roux.
Rom´eo: A tool for analyzing time Petri nets.
In 17th International Conference on Computer Aided
Verification (CAV’05), volume 3576 of Lecture Notes in
Computer Science, pages 418–423, Edinburgh, Scotland,
UK, 2005. Springer.
Gerard J. Holzmann.
The model checker spin.
Software Engineering, 23(5):279–295, 1997.
Rapha¨el Marczak, Antoine Allombert, and Myriam
Desainte-Catherine.
Real-time temporal control of musical processes.
In Proc. of the International Conferences on Advances in
Multimedia (MMEDIA) ’11, 2011.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

139. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Itay Meiri.
Combining qualitative and quantitative constraints in
temporal reasoning.
Artificial Inteligence, 87(1-2):343–385, 1996.
M. Nielsen, C. Palamidessi, and F. Valencia.
Temporal concurrent constraint programming:
Denotation, logic and applications.
Nordic Journal of Computing, 1(9):145–188, 2002.
Yann Orlarey, Dominique Fober, and Stephane Letz.
Syntactical and semantical aspects of faust.
Soft Comput., 8(9):623–632, 2004.
Carlos Olarte and Camilo Rueda.
A Declarative Language for Dynamic Multimedia
Interaction Systems.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

140. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
In Proc. of Mathematics and Computation in Music,
volume 38, Berlin, Germany, july 2009. Springer.
Carlos Olarte, Camilo Rueda, Gerardo Sarria, Mauricio
Toro, and Frank Valencia.
Concurrent constraints models of music interaction.
In G´erard Assayag and Charlotte Truchet, editors,
Constraint Programming in Music, chapter 6, pages
133–153. Wiley, Hoboken, NJ, USA., 2011.
Carlos Olarte and Frank D. Valencia.
The expressivity of universal timed ccp: undecidability of
monoadic fltl and closure operators for security.
In PPDP ’08: Proceedings of the 10th international ACM
SIGPLAN conference on Principles and practice of
declarative programming, pages 8–19. ACM, 2008.
Fran¸cois Pachet.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

141. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Playing with virtual musicians: the continuator in practice.
IEEE Multimedia, 9:77–82, 2002.
Miller Puckette, Theodore Apel, and David Zicarelli.
Real-time audio analysis tools for pd and max/msp.
In Proc. of ICMC ’98, Ann Arbor, USA, 1998.
M. Puckette.
Pure data.
In Proceedings of the International Computer Music
Conference. San Francisco 1996, 1996.
Nirina Ranaivoson.
R´eflexion sur la mise en place des structures logiques dans
un logiciel de partitions musicales interactives.
Master’s thesis, Universit´e de Bordeaux, 2009.
Camilo Rueda and Frank Valencia.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

142. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Proving musical properties using a temporal concurrent
constraint calculus.
In Proc. of the 28th International Computer Music
Conference (ICMC) ’02, 2002.
C. Rueda and F. Valencia.
On validity in modelization of musical problems by ccp.
Soft Computing, 8(9):641–648, 2004.
Camilo Rueda and Frank Valencia.
A temporal concurrent constraint calculus as an audio
processing framework.
In SMC ’05, 2005.
S. Safra.
On the complexity of omega-automata.
In Proceedings of the 29th Annual Symposium on
Foundations of Computer Science (SFCS) ’88, pages
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

143. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
319–327, Washington, DC, USA, 1988. IEEE Computer
Society.
Vijay A. Saraswat.
Concurrent Constraint Programming.
MIT Press, Cambridge, MA, 1992.
Gerardo Sarria.
Formal Models of Timed Musical Processes.
PhD thesis, Universidad del Valle, Colombia, 2008.
Vijay A. Saraswat, Radha Jagadeesan, and Vineet Gupta.
Foundations of timed concurrent constraint programming.
In Proceedings of the Ninth Annual IEEE Symposium on
Logic in Computer Science, pages 71–80. IEEE Computer
Press, 1994.
Alexander Schimpf, Stephan Merz, and Jan-Georg Smaus.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

144. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Construction of b¨uchi automata for ltl model checking
verified in isabelle/hol.
In Proceedings of the 22nd International Conference on
Theorem Proving in Higher Order Logics, TPHOLs ’09,
pages 424–439, Berlin, Heidelberg, 2009. Springer-Verlag.
Patrick S´enac, Pierre de Saqui-Sannes, and Roberto
Willrich.
Hierarchical time stream petri net: A model for
hypermedia systems.
In Proc. of the 16th International Conference on
Application and Theory of Petri Nets, pages 451–470,
London, UK, 1995. Springer-Verlag.
Mauricio Toro, Carlos Ag´on, G´erard Assayag, and Camilo
Rueda.
Ntccrt: A concurrent constraint framework for real-time
interaction.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

145. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
In Proc. of ICMC ’09, Montreal, Canada, 2009.
Mauricio Toro and Myriam Desainte-Catherine.
Concurrent constraint conditional branching interactive
scores.
In Proc. of SMC ’10, Barcelona, Spain, 2010.
Mauricio Toro, Myriam Desainte-Catherine, and
P. Baltazar.
A model for interactive scores with temporal constraints
and conditional branching.
In Proc. of Journ´ees d’Informatique Musical (JIM) ’10,
May 2010.
Mauricio Toro, Myriam Desainte-Catherine, and Julien
Castet.
An extension of interactive scores for multimedia scenarios
with temporal relations for micro and macro controls.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

146. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
In Proc. of Sound and Music Computing (SMC) ’12,
Copenhagen, Denmark, July 2012.
Mauricio Toro, Myriam Desainte-Catherine, and Camilo
Rueda.
Formal semantics for interactive music scores: A
framework to design, specify properties and execute
interactive scenarios.
Journal of Mathematics and Music, To be published.,
2012.
Paul Teehan, Mark R. Greenstreet, and Guy G. Lemieux.
A survey and taxonomy of gals design styles.
IEEE Design Test of Computers, 24(5):418–428, 2007.
Mauricio Toro.
Structured musical interactive scores (short).
In Proc. of the doctoral consortium in ICLP ’10,
Edinburgh, Scotland, UK, 2010.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

147. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Mauricio Toro.
Structured Interactive Scores: From a simple structural
definition to a real-time capable implementation.
PhD thesis, Universit´e de Bordeaux, Bordeaux, September
2012.
Frank D. Valencia.
Temporal Concurrent Constraint Programming.
PhD thesis, University of Aarhus, 2002.
Frank D. Valencia.
Decidability of infinite-state timed ccp processes and
first-order ltl.
Journal of Theoretical Computer Science - Expressiveness
in concurrency, 330(3):557–607, 2005.
Lindsay Vickery.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 126

148. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Non-linear structures for real-time interactive musical
works.
In Proceedings of the Australasian Computer Music
Conference (ACMC) ’03, 2003.
Lindsay Vickery.
Interactive control of higher order musical structures.
In Proc. of ACMC ’04, Victoria University, New Zealand,
July 2004. ACMA.
Lindsay Vickery.
The possibilities of novel format structures through
computer controlled live performance.
In Proc. of ACMC ’11, 2011.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127

149. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
More Future Work
Score with
conditions
Event Structures
semantics
Block Diagram
semantics
Abstract
semantics
Structural
Definition
Ntcc model
Ntcc and Faust
interconnected
Model
checker
Temporal
Property
Operational
semantics
Pure Data
Program
Score with
audio proc.
Score with
interval
durations
Score with
arbitrary
durations
Event Structures
semantics
Event Structures
semantics
Stand Alone
Programs
Programs Music
Properties
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 127

150. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Limitation of time conditional branching
b
e
Choose either
d or e after aa
c
d
∆c = {3}
∆e = {2}
∆d = {1}
∆b = [1, 2]
=
=
=
=
=
=
=
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 128

151. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conditional branching semantics in detail
Red (r)
∆r
∆r
sr er
∆r
s
r e
r
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 129

152. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conditional branching semantics in detail
φ
∆
∆
∆
∆
∆
∆a ∆b
∆a ∆b
∆a
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 130

153. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Conditional branching semantics in detail
Green (g)
Red (r)
Lights (l)
a
∆l
∆r
∆g
∆a = {0}
∆r
∆r
∆g
∆g
sr
sr
er
er
sg
sg
eg
eg
a
sl
el
∆l
∆l
Video (e)
ee
e
e
e
l
s
l
φ ≥ 10
∆ = {0}
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 131

154. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Equivalence between interactive score constraints
and its event traces
Theorem
The temporal constraints of a score are equivalent to the
temporal constraints of the event structures semantics.
Proof.
This holds by replacing points in the temporal constraints of
the score by its corresponding events.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 132

155. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Correctness of the operational semantics with
respect to the event structures semantics
Theorem
The possible time units in which a point process can be
launched in the operational semantics are included in the
possible time units in which an event can appear in an event
trace in the tes semantics [TDCR12].
Proof.
By induction on the structure of the timed event structures
(tes) semantics of a score.
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 133

156. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Correctness of the operational semantics with
respect to the event structures semantics
Theorem
Let s be a score, ε∗
= des(normal(es(s))) = E, l∗
, R its
event structures semantics, tc(ε∗
) its temporal constraints,
P = ScoreS,I,Pr,R the ntcc process that represents the score,
[[P]] the denotation of the ntcc process, and Ti is the set of
indexes j such that [[P]]j+1 entails launchi . It holds for all
sequences in [[P]], n = card(E), that
T1 × T2... × Tn ⊆ Solutions(tc(ε∗
)).
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 134

157. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Correctness of the operational semantics with
respect to the event structures semantics
ε∗
Values of Ti for process Scoreε∗ tc(ε∗
)
((i)) Ti ⊆ [1, n∞] ti ∈ N
(i)
∆
→ (j) Ti = {1} ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti +
((i))
∆
→ (j) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + min(∆) ti ∈ N ∧ tj ∈ ti +
((i))
∆
→ ((j)) Ti ⊆ [1, n∞] ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti +
(i)
∆
→ ((j)) Ti = {1} ∧ Tj ∈ Ti + ∆ ti ∈ N ∧ tj ∈ ti +
Table : Temporal constraints to launch the events in the ntcc
model Vs. temporal constraints of the event structures semantics.
We denote an interactive object as ((i)) and a static one as (i).
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 135

158. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Extension of Interactive Scores
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b
[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Temporal object
Temporal relation
High-precision
temporal relation
Dataflow relation
Interactive object
Interactive score
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 136

159. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
The novelty of our extension:
Faust and ntcc interconnection
1. A model based on the Non-deterministic Timed
Concurrent Constraint (ntcc) calculus for concurrency,
user interactions and temporal relations, and
2. a model based on the Faust programming language for
sound processing and micro controls.
buton
Label
mouse
down
mouse
up
0
1
user
ntcc
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 137

160. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Extension of Interactive Scores
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b
[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Temporal object
Temporal relation
High-precision
temporal relation
Dataflow relation
Interactive object
Interactive score
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 138

161. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Example: An arpeggio with three strings
Karplus (k1)
Karplus (k2)
Karplus (k3)a
b[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
ThreeStrings(f)
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 139

162. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Example: An arpeggio with three strings
a
b
[2s, 4s]
∆k1 = [10s, 10s]
∆k2 = [5s, 10s]
∆k3 = [4s, 4s]
Figure : The constraint graph
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 140

163. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Example: An arpeggio with three strings
Karplus (k1)
Karplus (k2)
Karplus (k3)
@100
threeCords(f)
output
sk1
ek1
sk1
ek2
sk2
Figure : Faust’s Block diagram
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 141

164. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Example: An arpeggio with three strings
Figure : Implementation
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 142

165. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Application: Three user controlled arpeggios
∆ = 10 ∆ = 10 ∆ = 10
Figure : The double-headed arrow represents an inequality (≤) and
a white-headed arrow represents an equality relation (=).
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 143

166. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Application: An arpeggio without “clicks”
a
b
[100smp, 100smp]
[2s, 4s]
[0s, 0s]
[0s, 0s]
∆k1 = [10s, 10s] Anti Click Three Karplus (f)Karplus (k1)
Karplus' AC[9.5s, 9.5s] [0.5s, 0.5s]
Karplus (k2)
Karplus' AC [0.5s, 0.5s]
∆k2 = [5s, 10s]
[4.5s, 9.5s]
Karplus (k3)
Karplus' AC [0.5s, 0.5s]
∆k3 = [4s, 4s]
[3.5s, 3.5s]
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 144

167. Introduction Hierarchical model Extensions Implementation Conclusions Thanks Extra
Application: Sound source perception
Karplus (k1)
L Output (o1)
[0s, 0s]
∆k1 = [10s, 10s]
∆o1 = [10s, 10s]
R Output (o2) ∆o2 = [10s, 10s]
Karplus (k1)
L Output (o1)
∆k1 = [10s, 10s]
∆o1 = [10s, 10s]
R Output (o2) ∆o2 = [10s, 10s]
500µs
Time Time
Mauricio TORO – LaBRI, Universit´e de Bordeaux. , Ph.D defense 145