1. SYNCHRONIZATION OF MULTIMEDIA OB.IECTS
D. Verdejo. A. Kannoüch, N. D. Georganâs
Telecommunicâiions Rcsearch Insailutc ol Onlano
Universitÿ of oltâwâ Medicâl Communicâlions Rcsearch Centcr
16l Louis Pas@ur, Onawâ Ontario, Cânâdâ KIN 6N5
e.maiI: GRDEJO@UOTTAWA.BITNET
FAX: + I'613'5&l-7681
Absùacl
Uso of muldmediâ documenB involves presenhlion o[ data
lrom dilfcrcnl succm. rn lhe mosr gcneral ürse. (ùmrng
from distributed dalâbâses. A mcdium rciers to â particular
kind of s[eâm such as vidco, audio, and datâ streams likc
grâphics, text ând displây mânâgemenl.
In this issue we deal wilh â cenralizcd synchronlzation
schemâ for media objects (MO) considered âs separate
cniities which âre intendcd to cooperatc in a common
cnvironment 1o constilutc thc MulliMedia Document
(MMD).
Thc synchrcnizaiion Droblem is râised a! two dislincl lc'cls.
Firsl il appears at lhe document specificalion level when thc
MMD is lo be dcfined, lhen al ùe plâybâck ol dre MMD
when the data involved in its consritution arc !o be gâûercd
àgain. We use here a relâtionâl dâtâbâse model scenârio às
tho basic represeû1âdon of thc synchronizarion needs, Wc
âlso present an implementation of the Masler Controilcr
Synchronizer (MCS). the centralizcd cntity managing mcdiâ
l.lnEoduction
the cvolution ol mullimedia lcchnologies (vidco, optical
srorâge)[1] ând broadbând rclccommunicadon ncrÿorks hils
significândy coûribuæd to thc cmergencc ol ncÿ
mullimediâ âpplicâlions. In mLrltiplc liclds. like compurcr
l.reü tmining. rclc.hoppin! )rd bu'rnc.r nrcscnrrriôn
Audiovisual Inleracdve appticâtions (AVI) lurn out to bc
porertrll communicaiions toois 21. SûDultaneously. Lh.
needs for massive inlcrconnccûon ol severâl usùrs lbr
Leieconlèrenciûg grows in application ilelds like Cornputcr
supporrcd Collrborauve vÿork ànù datrnt . :.ininll rc,turrinr
reâl time video and sound communication In lhls ncw area.
ùere is a great need for homogeneily. |o increase thc
Jeclopmcnr ol qVl Jpphcrlionc Jnü Ihelr (ornmunrcJln,n..
However, in our âpplicalion. the documens manipulalcd
mây contâin vârious and large amouo6 of dâla, Thcrclbrc â
dislributed ltrchitecturc wherc dillèrcnt portions ol lhc
documentare dispÂrched on differcnl servcrs is required. Onc
of ûe mos! importânt problems ùal wc must dcal witlr in
such an archilecturc is ürc synchronization of dilfcrcnl
componen§ of the documen! during thc prcsenlation ol thc
multimediâ documenl on ùe displây workÿation (DWS).
Indeed we have to retrieve the ditlcrcnt rcsoùrces associâtcd
wllh fte mediâ before displaying thcm. These dâtâ. undcr lhc
form of files havc to be ûânsmilted over ûc nc!$'ork, lrom
the seûers to ûe DWS. Basically, those synchronization
constrâinB come from the archiicclure of ûc MlilD. As wc
ujc lools uch i mullmclir cJiro'. rnrl :ltrlr, rrn! ') l(rn'
f4l ro crcarc rhe MMD, wc spccify timing consraints for the
Jiplâ) of cvcry mcdium Jnd lhese conslrxints move to
synchronizadon problèrhs for the phyback oflhe documert.
wc disringuish the se)f synchronialion which considcrs the
yârious (communicalion) pârameters introduced by ûe
nersork lo evirlu]lc Lhc dclJy nccessary ior Scning  given
rcsource ând üe intcrmcdia synchronization thal describes the
mcrhods to plâyback the diffcrenl mcdia logether on the
DWS when ùeir resou.ces ârcâvailâblc.
Among rhe diflerenl approaches tor ùe crcaiion ol a MMD
ÿe can cile rhc slorc and forward mcahod whcrc no æmporal
dcpcndency urisls bctwcen thc time of crcation of the
documenr ând thc momenl it is plîyed back, ând the real
timc method whcre both authors ând rcadcrs ol lhc document
ârc intcrconnectcd viâ a nelwork whiie the MMD is being
crcâlcd. lvlosr rccenr work ât lhis lcvcl indicatcs thât thc
ynchronizâ1ion requircmcnls could be fuliillcd ar thc OSI
rransport lclcl l3l.
In this prpcr wc chose ro locus on the store ând forward
cnvironmcnr as previous reseârch hàs beefl done in this field
in our lâborâtory [:1] and wc studied the synchronizâtion
conccpr âpplicd !o MMDs. Wc must there distinguish two
directrons in thc MMD .rpp.oâch. One is focuscd on lhe
document tcchnology. thc d.rumenl architecùre, represented
hy rhe ODA/ODIF s!ând.rrd which deals wiû thc structure of
thc documcnt ând lhc rrclhods neccssery lbr iLs layout. This
modcl is bcing cxtcndcd to cope with rhc synchronizatioû
consûtlinrs ol lvllt4Ds.
The olher diroction is uken by the Frcnch CCETT wiû its
proposàl oi standud RAVI (RcpresenÉtion of AVI) which
dcscribcs rll thc Ircets ol rhc proccssing ol ân AVI
.rpplicarion 8ll9l providing r hngurge to describe lhe
applications. This approâch is more onented towards thc
rnulrjmcdiâ rppiic.rtion rn Ènns ol computer program than
rhc archirecrurc ol Lhc N,llvlD mânipulâtod. 1n RAVI a MMD
is â scqucnce of compuler progrâm struclured âs â trcc ol
operàors wùh their parâmetcrs, including ÿrucrural ând
presenrâlion opcrâlors, Thc p.ogram wrilEn wilh ûese
opentors conslilutc the sccnârio of ùe MMD which, when
inrerprclcd consûtutes rhe Mullimcdia Applicâtion. Hcre, the
§cenario rcplâccs and exrends the nolion ol document
archirecture, allowing references to any kind ofresource ând
pcnnirling to hândlc intcracliviry with thc uscr ùrough a set
ol dcdicaled op(rârcrs.
Whichcver di.ccrion is Eken lo dcal with MMD. scenârio or
documenr architeclurc. tho playback ol the MMD râises
problenrs ol si7e and loürxon ol ùc components: the display
frn.,c..;cncrJll crnnor rcr rhc wloic MMD at oncc in ir
local mcmory and needs rime to retrievc thc.cmorc
inlbrmalion. This âppears c1cârly in thc râdiologicâl
JttrrtrLr i (l(ÿ. "inc,l in rh. 'lLlrIncJiJ C.,rnrnunicflion
L.rbora()ry lbr jnÿancc, whcrc prcrurcs consliLuling thc

2. 2
râdiological examinâ1ion âre 1.2 Mcgabyles in sizc (10
Mbirs) and are localed on dedicâled servcrs. Altemauvely. üe
C'D ROM tcchnology l5l solves thcse problems by
compiling on ûe sâme physical rcsourc'e (opdcal discs) all
ûe MMD. The plâyback is then simply the sequenliâl
reading of the disc wiù a ùroughput thal allows all ûe
medra to be drspâtched ând plâycd on tJleir own deviccs.
Io our system. the media ârc kep! separâte in disributed tilc
systems unlil thc documenl has to be playcd bâck on â
Displây Vr'orksÉ1ion (DWS). We üen nccd âspeciâl process
ro ordcr the rcuieval ol the difierenl rcsources from the
remole servcrs. 11 is the Master Controllcr Synchronizer
(MCS) which accesses the remporâl and sructurâl
information âbout thc MMD and builds â rcsource planning
in order 1o provide dala for üe ou1pu1 dcviccs associaied to
fte media, âccording to the memory spâce avaiiable ând thc
dâla lrânsmission rales requircd. Thus thc MCS acls às a
plaoner for the communications bctwccn ùe network âIrd thc
media obje.r.s (MO).
This paper dcscribes lhc rcsearch in the mulLimedia
communicadons laboralory of ùe unilersily of Ollawâ. Wc
describe the conlexl of this rescarch and rhe prololype
deveiopcd to implcmcnl o0r ideâs ând deiine our approach to
rhe probiem ol synchronizarion in this pârticulâr conrexr and
ûe meùods we usc to solvc it.
2. Ivlul mediâ Slorc ând Foôÿâd Environnrcnt
The Uni!crsit) of Otl.rwJ lcil.Jl CommLrnicr'.ôn.
Research Ccnrc has developed a Multimcdia Radiology
Informâlion ând Communicâtions system i4l. Thc systcrrr
providcs radiologists and physicians with a sel ol'
rvorkslalions liom lvhich râdidogy iniormalion cân bc
elccEonically caprured. rransmitled, updatcd ând coosulte(1.
Our âpproach is bâsed on thc usc ol thc Rclâlional DBùIS
(Dala Baso Managemcnl Sysrcm) to dcscribcs mullimcdir
obiects and dreir limc and lunctionnal dcpcndcncics, ('ilh
some external cxtensions to suppo( unlormaLlcd dâtn
(imâges ând voice). Our prolot)pc has bcen implcmcntcd
using ORACLE on SUN] as a Drtabase Scncr. Thc Scnù
is connecrcd viâ a 10 'Ibib/s ELhùrnct L,N kr scÿùrll i1116
based worksLrtions- using TCP/lP end approprrrt.
Application prorcool§. Two iill6 workstâtions rrc uscd x
serÿers lor 1cxr, ÿoicc and imâse.
In this environmenr wc dcline lwo levels oi synchronizati{rn
ùâl crn be âssociâtcd 1o dre phases oi crcalion and ph) brck
oi the MMD:
Firsr is the coarse Srain synchronizatron [6] delincd a! thc
uscr lcvcl in order to specif-v the links berween thc mcdia
Juring dr,phy: hcrc lh( qucilùn r. s hJr rrLJ.r :dc .:uir': 1,,
be tied in Lhe documcnl and !hen Ifig-11.
For cxamplc wc câo cile lip-synching on a vidco document
whcrc voicc ând film are on sepuatc resourccs, Wc thcrcb)'
fâce dreproblem ofspecilying the way wc nced ûc mcdia to
bc mixcd at thc moment they will be displâycd. This is the
purpose of rhe crcâ1iofl ol rhc sccnlrio (scrip, ol thc
documenr. Duflng Ine plly birck ul lhc,locum.nl. ll I., nc
of the duties oi lhc Synchronizer lo manâlc such
synchronizâtion whjch e !vill cùll intcr mcdjx
synchronizâ1ion,
Frg.l Snccrli(rtiunùl.rnrullimu(li;! s(enùio
CIock
Fig.: Drt:l RcquircmenL! Chârt lbrâ Multimcdia
On ùc olher hrnd, the lvlMD s playback lcads to ân ordered
list ol rctrieÿ.rls ol poftir)ns ol documenCs conæn! so as to
rcspcct thc tirning requirements during lhc display.We
âssume rhâr eâch mcdium must bc proccssed scparaleiy to
mcct its own drt, ând dmc requiremcnts, and ûat the role ol
MCS is to takc into accounr lhc schcdulc consûain§ o1'
clerÿ media as dclined in the scenârio. Consc4ucnlly we will
câll lhis kind ol synchronization sclËsynchronizarion-. The
synchronizrlion ol two mcdià is though solcly symboiized
hy rhc raLr lhrl lhcy aivc dxrJ rr.luircmcnt. cr r sarmc lime.
ft seems conlcnicnt lo represcnt drc sct ofconsuâinls by lhc
chândcpictcd in Jlig.2l whcrc ùc dashcd surfaces indica@ùc
tolal ûmounts ol'mcnrcry requircd by eâch medium during
rl'c lr..cnrirlr"n ul rhc N,l4D Thc .)r)chrollr/Jrron nroce§s
cân rhcn bc dcpictcd u a lwo stcps proccss, first is seli-
picture i Tcirt T.xr2
GraPhic, I
End ol

3. 3
synchronization ùat resolves lbe extemal constraints thcn is
ûe inter-mediâ synchronizadon àl the local level.
Synchronizâtion is a sub-problem of lhe presenlation oi û
MMD. By presentation we mean herc'execuaion of üc
applicâtion ahal will play bâck ùe scenario describing lhc
document Maûy efforts havc been mâde to formalizc Lhe
represenuiion of multimedia audio visuâl inleractic
applicarions (AVI)[71. The French CCITT s proposal ol
slandard, RAVI, introduces a sel of operators to design drc
slrucure of âny AVI like â compuler program. Some ol dre
operâtors arc used 10 reference and process ûc mediâ ând
more generâily afly resource. othcrs are solely sutemen§ lbr
slrucluring the application. Among üc opcrators are lhc
syûchronizalion operâtors. which âllow the deiinitiûr ol'
relâuons of sequence ând pârallelism betwcen the mcdia
As a cenlrâlized cndly, lhe syûchronizer should drcrclorr
Jicrrbule lhe ordcrs ol prelJrrllun. srrl inri Ic'lilJli,,lr .'
thc differen! MOs and dccide on Ûeir lilèlime according 1o
thc scenà o [fig.4].We designed a spccilicâ1ion lânguagc ol
ihc synchronization rcquirements in our mul!ime(i1r
documents, which permils rhe crcâiion ol simplc sccnânos
In rhis lânguage, a first sel of commands wiil albw thc
dcfinilion of the dillcrent objeds involvcd in the multimcdi,
applicatron. defining the tlpc of the medium ând thc nr c ol
the file'rhich contains its resource. Accordin! lo ihe t)pc.
ùe resource will thcn be reaievcd liom a pârliculâr scrvcr. ln
a second sct arc the commânds lbr spcciiying thc
synchronizalioo condrtions ând lhe rcLions synchronizc(I.
mostly thc playbacks of ûe MOs-
.1. Funoliontl Principlcs
As mentioncd in the preceding scctions, lhe nolion ol
p.cparalion of lhe resources is cssenrial to insurc thât thc
mcdia will be ready ior lurlher synchroniTâlion.
Prcparation includes rcscrvalion of memory spacc. âllocation
of lhe playback chânncls, downloâding ol resourccs and
burering.
Thc prcpÏcuon mu.t be ôrdcr..r I rhc r) n r1n 'rrr/cr u r. { r'
lhc execution ol any mediutr. thc I'ICS rnâtâgcs urÿ
,rmhcr ol mrdiâ inolved rn r docurucnt. Thù inLcrllr.
prccess cân then bc seen as â mcdium consuntcr 'hllc th.
producers would bc the diu sêncrs.
Therctorc managrng a mcdiu consisl ol scülng up ll
dialogùe wirh bolh lhc scrver rel:rted lo this lrrcdiurlr a d lhc
consumer âppiicadon. The dialogue !vith thc âpplicâtion is
direcÉd by the limes vr'hen dâtâ should bc âyàilàblc {ApD§)
and is achieved locâlly on ùe DwS, whilc ùc diâloguc §'ith
ûe scrver is dlected by thc times when lhc loading requests
for dâtâ should be made (Lpips) ând is âchiclcd ûrough â
communicâlion channcl (CCH). One ol thc pu.poses ol thc
MCS is thc compulation of thc serial ol Lpips in iunclion
of thc scrjâl ol Apips draL is prcvided by thc interpr.lâliorl o!'
thc scenârio. Such mânagement should lcad to lhc
establishmenl of links belween servers ând clients rith an
averâge throùghpur corresponding to the media needs and
rimês dcfined in thc scenârio. Our aim is to cnabie the IICS
|o managc the trânslers of dal.â belwccn scrvers ând mcdlil
objects wilh ùe finest granularily possible. For ûât purposc
wc dcfine a protocol which allows ùc MCS to gel
inlbrmation about all the mcdià objccts înd scqucnce tlcir
downloading lrccly, compuling
fequesEd on eâch scrvcr-
Fig.,l: Mânagemcnt ol mcdia Obtcct! by
-Ihc
Synchronizer
One could rrguc ûa! tolal doÿnbâding on ûe DwS of ùe
resourccs necessâry for MMD plâybâok would bc sulficient
ro ensurc good rclricval delays. uûfortunâlely we musl
.onsidcr rhrt- il thc hcâl hârd disk ol thc DVr'S has the
capâciry «) storc thc cnlire documcn( il will ccrtâinly not
providc cnough ûroughput lo play back all ùc media
logelhcr. Thc rhroughput ol thc disk musl be split in as
mânÿ pûrls âs ùe nunrber o[ mcdia. considcriûg thâl ils
clliciency dccrcrscs with this ûumbcr. Wc made a seriâl oi
lù§rs on our SUN3 Workst:r!rcn. involving lwo mediâ
imul:llcd rcquircmlnts- ÿ(lco ând lorcc wiû respccuve bil
rurcs ol l50 and Il Kbytcs/s.This sLudy has shown hrt a
inglc hàd disk is unâblc to providc suiiicicnl ûrorghpul
ror mulriplc prcccsscs rn this contcxL. so unlcss 1l mul1i
disk.rrchircclurc rs aÿai1âblc, wc must usc thc throughput
nnr r.)Jl h) rhcnct$.rÀ Elhcrncl ,roürùr.J.§hich.llow.
Jhour ôr' Kh) rcv' Iur.J! h lnnn(JlcJ soclcl I l0l.
As â ùalrcr oi làcl. we deconrposcd thc problcm of
synchronizalion in lwo sub problenrs: ânlicipalion and
communicàlion. Since lhc ânlicipalion hrs to do with the
rirning consuaints and thc sccnârio drc MCS dcâls with it-
On lhc oùcr lrand thc communications responsibility is
gilcn t) thc COI'IM (Commonicîtion managco to frec thc
I'ICS liom lhcsc lunctions. Thc MCS slâys ûc mânagcr of
rhc whole process by imposing â synchronizâlion cycling
rime i.c. by slicing the timc in slots oi lixed sizc whoro all
cvcnr occurring should lakc plûcc. In a given cyclc the MCS
ûchieves ùrce tâsks ol cquâl importancei it ânlicipâtes by
ordering re COMM to conneci objccls !o CCH Lo fill lhcir
resourccs, it displâys ready obiects whcn their display lime
has comc, and cvcntuâlly i! ordcrs the COMM to poll the
tho amounts of memory

4. 4
differcnt open CCH âllributed to connected obie.§ 10 retrieve
dam on them. The COMM mâin duly is ùen to âchieve this
polhng ano di.conncct tilc ohrects when ùcy hrve comPlclcÙ
rh.ir frllins
ii. Conclù§ion
Wo treâÉd thc subiect of synchronizâtion of Muldmedia
Objecrs in a particulâr environmenl ûat is lhe store ând
forward docùmcnls, on ât Etheme! locâl âreâ network. Such
link is unforlunatety asynchronous and thc prolocoi does nor
gu.rans any quaLly ùl §cnicc. For our purlose. whrch i ru
Lnk rogeLher a fc* .enerj ând a drsplry $ork'!âlron rn orJer
10 play bâck â multimedia document. oûer kinds ofphysical
archireclurcs ând dcdicatcd prorccols likc ISDN oTFDDI Jl l l
would bc more suiÈble. Rcalistic rcnderings are mrdc
po'!rble by inlcrlrcrnt our moriel ol mJllrnrcdir dôcur.,.nl
prc.enLrlion ùrlh 3n X Window icrcr ll Jnd clft(rc].
mock-ups are being rcalized in the Mullimc(lia
Communication Lrb. of thc Univcrsity ol Ottawr.
Althoùgh, the mclhods wc proposcd ibr achicving !h.
synchronizalion arc closeiy dcpcndânt on the tcchnology
crrenlly available the functional model oi the synchronizcr
ând thc dilfcrenl enlitics dclincd shoLrld âcluâlly rcmàin
rÿhatcver ùc meâns cen bc,
6. Acknowledsment
This rcsearch wâs supported in pùt by ùe Cânâdiân InstituLc
lor Tclccommunicâtions Rcsearch (CITR) and iû parl bl'lhc
Tcle.ommunications Rcscârch Inÿitute ol On1âr jo ('fRIO).
I1l
tll
tll
i6l
T.Relerenccs
Karen A. Frcnkcl, "The ncx! gcneration ol inteructilc
technologics', Communicalioos oi Thc ACM, luly
1989, Volumc 32 Nùmber 7, pp. 872-8U1.
Edsard A. FOX. 'Thc -omrnÊ rc",,lurion rr
inleractive digi!âl video . Communicalions Oi The
ACM. July 1989. Volumc 3: -wumber 7. pp. 791-
u01.
lvl. Sâlmony ilnd D. Sh.phcfu. Elcndrng OSI 1()
rupporr synchroorrâ!roIr rcquircd h)' ntultlmodrr
applicâtions . IBM ENC Tcch. Rcp no 1i.390'1.
Apr, 1989.
A. Kârmouch. L. Orozco Bâ-rhosa. N.D. Gùorgânr'
M. Goldberg, "A Mullimedia Medic:rl
Communicalion Syÿem , IEEE JournÀl on Sclectc(l
Areâs in Communicalions. Volumc il Numbor 3.
April 1990, pp. 325-339.
compr(t Di(k_lnlerrclrc. J ilcjiSncr "vcrv
c§
ediled by Philips, Kluwcr Technicâl Books, Dovcntcr.
Anlwerpcn, 1987.243 pages.
Rrll Srcinmcl/. Sln.hronru rrrr,n Ptnfcrlr(' 1
Multimcdiâ Syslems", 1EEE Jou.nrl on Sele.:rùl
Areas in Communications. Volumc 8 Numbor.l,
April 1990, pp. 401-112.
Final Report of the AFNOR Z60R - Commitlcô on
lhc srândàrdizalion ol lhe Representatiorl and
frùl^ùol. lor Intcrchrngc ol AudlovisuJl Inlerâctive
Applicalions (AVIs), " ISO JTCl N424, March 1989
89 pîges.
F. Oguet, C. Schwârtz, F. Krciz, M. Quere, "RAVI,
A Proposcd Sundârd ior the Inlerchânge of
Audio/Visuâl Inieracdve Applicâdons". IEEE Journal
on Sciecred Arcas in Communicâtions. Volume 8
Number 1. AnIl I9o0. pp.428 416.
RAVI, spccification version V1 30/4/1987, revised
vcrsion Vl.2 30/9/88. Brochures 1.2.3,4. CCETT,
Dpt PAA/MGA/DCT, 38-:10 rue du General Lcclerc,
92I3I ISSY Lcs MoLrlineaux, France.
I.lrcmcru,,rking $ lr TaP-lP. D Cnmcr. Prcnuce-
Hrll Pub.. 1989.
t7l
l8t
I9l
I10l
L11l F.E. Ross, An O!crÿicw ot FDDI: The Fiber
Dirrihurc(l Dâtâ Intorlàcc'. IEEE Journalon Sclectcd
Areas in Communicalions, volùmc 7 Number 7,
Scptcmbcr l9li9. pp. 1043- 1051.
ll2l Win(io§, S)§tcms ProgrÀm,ning and Appiications
wi!h Xr. Douglas A. Young. Prcnlicc Hall Ed.. 1989.
I2l
i5l