ZING: the Next Generation of Z39.50

2. ZING: the Next Generation ofZING: the Next Generation of
Z39.50Z39.50
ADLUG Users Group MeetingADLUG Users Group Meeting
San Sebastian, 8-10 September, 2004San Sebastian, 8-10 September, 2004
ÁdámÁdám HorváthHorváth
NSZLNSZL

3. 3 ZING: the Next
ContentsContents
Why does Z39.50 need renewalWhy does Z39.50 need renewal
What is ZINGWhat is ZING
Technical aspectsTechnical aspects
Social contextSocial context
Amicus/LibriVision and ZINGAmicus/LibriVision and ZING
Experimental SRW client of NSZLExperimental SRW client of NSZL

4. 4 ZING: the Next
Why does Z39.50 need renewalWhy does Z39.50 need renewal
There are problems with Z39.50There are problems with Z39.50
Not popular with the web communityNot popular with the web community
– Connection-based sessionsConnection-based sessions
– Binary encodingBinary encoding
– Transmitted directly over TCP/IPTransmitted directly over TCP/IP
ComplicatedComplicated

5. 5 ZING: the Next
The complexity of Z39.50The complexity of Z39.50
11 Native Services11 Native Services
– Init, Search, Present, DeleteResultSet,Init, Search, Present, DeleteResultSet,
AccessControl, ResourceControl,AccessControl, ResourceControl,
TriggerResourceControl, ResourceReport,TriggerResourceControl, ResourceReport,
Scan, Sort, Segment.Scan, Sort, Segment.
7 Extended Services7 Extended Services
– PersistentResultSet, PersistentQuery,PersistentResultSet, PersistentQuery,
PeriodicQuerySchedule, ItemOrder,PeriodicQuerySchedule, ItemOrder,
DatabaseUpdate, ExportSpecification,DatabaseUpdate, ExportSpecification,
ExportInvocationExportInvocation

6. 6 ZING: the Next
The complexity of Z39.50The complexity of Z39.50
A 156 page standardA 156 page standard
Vocabulary from the OSI communityVocabulary from the OSI community
– Target, Origin, APDU, A-association, Z-Target, Origin, APDU, A-association, Z-
association...association...
Records described with ASN.1Records described with ASN.1
Attribute SetsAttribute Sets

7. 7 ZING: the Next
Strengths of Z39.50Strengths of Z39.50
Result SetsResult Sets
AbstractionAbstraction
– Abstract Access Points (Attribute Sets)Abstract Access Points (Attribute Sets)
– Abstract Record SchemasAbstract Record Schemas
ExplainExplain

8. 8 ZING: the Next
ContentsContents
Why does Z39.50 need renewalWhy does Z39.50 need renewal
What is ZINGWhat is ZING
Technical aspectsTechnical aspects
Social contextSocial context
Amicus/LibriVision and ZINGAmicus/LibriVision and ZING
Experimental SRW client of NSZLExperimental SRW client of NSZL

9. 9 ZING: the Next
What is ZING?What is ZING?
ZING:ZING:
ZZ39.5039.50 IInternational:nternational: NNextext GGenerationeneration
New web based Search and RetrieveNew web based Search and Retrieve
protocol familyprotocol family
Keeps the strengths of Z39.50Keeps the strengths of Z39.50
Eliminates the complexity of Z39.50Eliminates the complexity of Z39.50

10. 10 ZING: the Next
What is ZING?What is ZING?
Protocol familyProtocol family
– SRW - Search/Retrieve Web ServiceSRW - Search/Retrieve Web Service
– SRU - Search and Retrieve URL ServiceSRU - Search and Retrieve URL Service
– CQL - Common Query LanguageCQL - Common Query Language
– ZeeRex - Z39.50 Explain, Explained and Re-ZeeRex - Z39.50 Explain, Explained and Re-
Engineered in XMLEngineered in XML
– ZOOM : the Z39.50 Object Orientation ModelZOOM : the Z39.50 Object Orientation Model
– ez3950 : simple implementation over SOAPez3950 : simple implementation over SOAP
using XERusing XER
Protocol familyProtocol family
– SRW - Search/Retrieve Web ServiceSRW - Search/Retrieve Web Service
– SRU - Search and Retrieve URL ServiceSRU - Search and Retrieve URL Service
– CQL - Common Query LanguageCQL - Common Query Language
– ZeeRex - Z39.50 Explain, Explained and Re-ZeeRex - Z39.50 Explain, Explained and Re-
Engineered in XMLEngineered in XML
– ZOOM : the Z39.50 Object Orientation ModelZOOM : the Z39.50 Object Orientation Model
– ez3950 : simple implementation over SOAPez3950 : simple implementation over SOAP
using XERusing XER

11. 11 ZING: the Next
SRWSRW Search/Retrieve Web ServiceSearch/Retrieve Web Service
What is SRWWhat is SRW
– XML oriented protocolXML oriented protocol
– Performs searches and other informationPerforms searches and other information
retrieval operationsretrieval operations

12. 12 ZING: the Next
SRW and Z39.50SRW and Z39.50
Z39.50 concepts retained in SRWZ39.50 concepts retained in SRW
– Result setsResult sets
– Abstract access pointsAbstract access points
– Abstract record schemasAbstract record schemas
– ExplainExplain
– DiagnosticsDiagnostics

13. 13 ZING: the Next
SRW and Z39.50SRW and Z39.50
SRW differences from Z39.50SRW differences from Z39.50
– Result sets named by serverResult sets named by server
– No explicit concept of connection, sessionNo explicit concept of connection, session
or stateor state
– No distinction between server and databaseNo distinction between server and database
– Single record syntax: XMLSingle record syntax: XML
• Schemas are XML schemas: DC, MODS,Schemas are XML schemas: DC, MODS,
MarcXMLMarcXML
– Simplified explainSimplified explain

14. 14 ZING: the Next
SRW and Z39.50SRW and Z39.50
SRW differences from Z39.50 (cont’d)SRW differences from Z39.50 (cont’d)
– String queryString query
• SRW querySRW query
– <SRW:query>(<SRW:query>(titletitle = information= information ANDAND titletitle = retrieval)= retrieval)
</SRW:query></SRW:query>
• Z39.50 queryZ39.50 query
– @and@and @attr 6=1 @attr 3=3 @attr 2=3@attr 6=1 @attr 3=3 @attr 2=3 @attr 1=4@attr 1=4
”information" @attr 6=1 @attr 3=3 @attr 2=3”information" @attr 6=1 @attr 3=3 @attr 2=3 @attr 1=4@attr 1=4
”retrieval"”retrieval"

15. 15 ZING: the Next
SRW operationsSRW operations
searchRetrieve OperationsearchRetrieve Operation
– searchRetrieveRequestsearchRetrieveRequest
– searchRetrieveResponsesearchRetrieveResponse
scan Operationscan Operation
– scanRequestscanRequest
– scanResponsescanResponse
explain Operationexplain Operation
– explainRequestexplainRequest
– explainResponseexplainResponse

16. 16 ZING: the Next
SRW sample search requestSRW sample search request
<SOAP:Envelope<SOAP:Envelope
xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP:Body><SOAP:Body>
<SRW:searchRetrieveRequest<SRW:searchRetrieveRequest
xmlns:SRW="http://www.loc.gov/zing/srw/">xmlns:SRW="http://www.loc.gov/zing/srw/">
<SRW:version>1.1</SRW:version><SRW:version>1.1</SRW:version>
<SRW:query>(dc.author exact "jones" and dc.title >=<SRW:query>(dc.author exact "jones" and dc.title >=
"smith")</SRW:query>"smith")</SRW:query>
<SRW:startRecord>1</SRW:startRecord><SRW:startRecord>1</SRW:startRecord>
<SRW:maximumRecords>10</SRW:maximumRecords><SRW:maximumRecords>10</SRW:maximumRecords>
<SRW:recordSchema>info:srw/schema/1/mods-<SRW:recordSchema>info:srw/schema/1/mods-
v3.0</SRW:recordsSchema>v3.0</SRW:recordsSchema>
</SRW:searchRetrieveRequest></SRW:searchRetrieveRequest>
</SOAP:Body></SOAP:Body>
</SOAP:Envelope></SOAP:Envelope>

17. 17 ZING: the Next
SRUSRU Search/Retrieve URL ServiceSearch/Retrieve URL Service
Search requestSearch request
– Coded in the URL and sent as HTTP GETCoded in the URL and sent as HTTP GET
ResponseResponse
– SRW without the SOAP wrapperSRW without the SOAP wrapper
Example for a search requestExample for a search request
http://www.loc.gov/sru.cgi?http://www.loc.gov/sru.cgi?
operation=searchRetrieve&query=dc.title=cat&maximumRecordsoperation=searchRetrieve&query=dc.title=cat&maximumRecords
=10&recordSchema=DC&sortKeys=title,dc&startRecord=1=10&recordSchema=DC&sortKeys=title,dc&startRecord=1

18. 18 ZING: the Next
CQL - Common Query LanguageCQL - Common Query Language
Loosely based on CCL searchLoosely based on CCL search
Can be simple for every day useCan be simple for every day use
– This is a valid query:This is a valid query: programming languageprogramming language
Can be very complexCan be very complex
– dc.author=(kern* or ritchie) and (bath.titledc.author=(kern* or ritchie) and (bath.title
exact "the c programming language" orexact "the c programming language" or
dc.title=elements prox///4dc.title=elements prox///4
dc.title=programming) and subjectdc.title=programming) and subject
any/relevant "style design analysis"any/relevant "style design analysis"

19. 19 ZING: the Next
CQL - Common Query LanguageCQL - Common Query Language
Many shortcutsMany shortcuts
– Index and relation can be omittedIndex and relation can be omitted
• E.g.:E.g.: “bird dog”“bird dog”
• Full version:Full version: cql.serverChoice scr “bird dog”cql.serverChoice scr “bird dog”
– Relationship typesRelationship types
• allall
– E.g.:E.g.: dc.title all “bird dog”dc.title all “bird dog”
– Full version:Full version: dc.title = “bird”dc.title = “bird” andand dc.title = “dog”dc.title = “dog”
• anyany
– E.g.:E.g.: dc.title any “bird dog”dc.title any “bird dog”
– Full version:Full version: dc.title = “bird”dc.title = “bird” oror dc.title = “dog”dc.title = “dog”

20. 20 ZING: the Next
CQL - Common Query LanguageCQL - Common Query Language
Context setsContext sets
– Contexts may be applied to indexes,Contexts may be applied to indexes,
relations, relation modifiers and booleanrelations, relation modifiers and boolean
modifiers. They are of the formmodifiers. They are of the form prefix.valueprefix.value,,
where prefix is a short name for a uniquewhere prefix is a short name for a unique
context set identifier, defined either in thecontext set identifier, defined either in the
query or by the server.query or by the server.
– PurposePurpose
• Make a distinction between two identical indexMake a distinction between two identical index
names:names: dc.titledc.title,, foo.titlefoo.title

21. 21 ZING: the Next
CQL - Common Query LanguageCQL - Common Query Language
RelationsRelations
– <, >, <=,<, >, <=, andand >=>=
retain their regular meaningsretain their regular meanings
– ==
• For word adjacency, when the term is a list ofFor word adjacency, when the term is a list of
words.words.
• Otherwise, for exact equality of valueOtherwise, for exact equality of value
– allall,, anyany
• ““all of these items" or "any of these items”all of these items" or "any of these items”
• E.g.:E.g.: dc.title all “bird dog” dc.title any “bird dog”dc.title all “bird dog” dc.title any “bird dog”

22. 22 ZING: the Next
CQL - Common Query LanguageCQL - Common Query Language
Relations (cont’d)Relations (cont’d)
– exactexact
• is used for exact string matchingis used for exact string matching
– scrscr
• is used to mean "server choice relation”is used to mean "server choice relation”
– withinwithin
• may be used with a search term that has multiplemay be used with a search term that has multiple
dimensionsdimensions
– enclosesencloses
• database's term fully encloses the search termdatabase's term fully encloses the search term

23. 23 ZING: the Next
CQL - Common Query LanguageCQL - Common Query Language
Pattern matchingPattern matching
– ?? matches any single charactermatches any single character
– ** matches any sequence of charactersmatches any sequence of characters
E.g.:E.g.: ?in?s*r?in?s*r
– ^^ word anchoringword anchoring
• ^bird^bird must be the first in fieldmust be the first in field
• bird^bird^ must be the last in fieldmust be the last in field
• ^bird^^bird^ must be the only word in fieldmust be the only word in field

24. 24 ZING: the Next
ContentsContents
Why does Z39.50 need renewalWhy does Z39.50 need renewal
What is ZINGWhat is ZING
Technical aspectsTechnical aspects
Social contextSocial context
Amicus/LibriVision and ZINGAmicus/LibriVision and ZING
Experimental SRW client of NSZLExperimental SRW client of NSZL

25. 25 ZING: the Next
Bath profile in context setBath profile in context set
Defines indexesDefines indexes
– It uses the DC context set when appropriateIt uses the DC context set when appropriate
– Creates some bath indexes: bath.issnCreates some bath indexes: bath.issn
Defines relationsDefines relations
– exact, =,exact, =, <, <=, >=, >,<, <=, >=, >, withinwithin
Defines usage of wildcards and anchorsDefines usage of wildcards and anchors
Defines schemasDefines schemas
– MarcXML, DCMarcXML, DC

26. 26 ZING: the Next
DevelopersDevelopers
Principal Editor of the SRW 1.1Principal Editor of the SRW 1.1
SpecificationSpecification
– Rob SandersonRob Sanderson
WSDL/Schema EditorWSDL/Schema Editor
– Matthew DoveyMatthew Dovey

27. 27 ZING: the Next
DevelopersDevelopers
Additional Contributors to Version 1.1:Additional Contributors to Version 1.1:
– Eliot ChristianEliot Christian
– Ray DenenbergRay Denenberg
– Adam DickmeissAdam Dickmeiss
– Janifer GatenbyJanifer Gatenby
– Ralph LeVanRalph LeVan
– Mike TaylorMike Taylor
– Theo Van VeenTheo Van Veen

28. 28 ZING: the Next
DevelopersDevelopers
Additional Contributors to theAdditional Contributors to the
Intellectual Development of SRW:Intellectual Development of SRW:
– Larry DixsonLarry Dixson
– Sebastian HammerSebastian Hammer
– Poul Henrik JorgensenPoul Henrik Jorgensen
– Alan KentAlan Kent
– Bill MoenBill Moen
– Pat StevensPat Stevens

29. 29 ZING: the Next
Early implementersEarly implementers
Library of CongressLibrary of Congress
Koninklijke BibliotheekKoninklijke Bibliotheek
OCLC/PICAOCLC/PICA
OCLC ResearchOCLC Research
Index DataIndex Data
The European Library projectThe European Library project
National Széchényi LibraryNational Széchényi Library

30. 30 ZING: the Next
ToolsTools
CQL-JavaCQL-Java
– CQL parser for JavaCQL parser for Java
Python CQL ParserPython CQL Parser
– A Python Library for parsing CQL andA Python Library for parsing CQL and
XCQL. Includes CQL to RPN transformationXCQL. Includes CQL to RPN transformation
for PyZ3950for PyZ3950
YAZ Proxy from IndexdataYAZ Proxy from Indexdata
– SRU/SRW server to allow any Z39.50 serverSRU/SRW server to allow any Z39.50 server
to also support the ZING protocolsto also support the ZING protocols

31. 31 ZING: the Next
InformationInformation
http://www.loc.gov/zinghttp://www.loc.gov/zing

32. 32 ZING: the Next
ContentsContents
Why does Z39.50 need renewalWhy does Z39.50 need renewal
What is ZINGWhat is ZING
Technical aspectsTechnical aspects
Social contextSocial context
Amicus/LibriVision and ZINGAmicus/LibriVision and ZING
Experimental SRW client of NSZLExperimental SRW client of NSZL

33. 33 ZING: the Next
ZING compliant AmicusZING compliant Amicus
Developing a ZING server for AmicusDeveloping a ZING server for Amicus
– It could be fully ZING compliantIt could be fully ZING compliant
The use of a YAZ ProxyThe use of a YAZ Proxy
– The proxy translates SRW/SRU into Z39.50The proxy translates SRW/SRU into Z39.50
– LicenseLicense
• Amicus users can use it freely (Amicus users can use it freely (GPL license)GPL license)
• LibriCore would need special agreementLibriCore would need special agreement
– Problems in matching CQL and attributeProblems in matching CQL and attribute
setssets

34. 34 ZING: the Next
CQL and attribute setsCQL and attribute sets
RelationsRelations
– ““all” and “any” have no equivalent relationall” and “any” have no equivalent relation
attributeattribute
Relation modifiersRelation modifiers
– fuzzy has no equivalent relation attributefuzzy has no equivalent relation attribute

35. 35 ZING: the Next
CQL and attribute setsCQL and attribute sets
IndexesIndexes
– CQL is based on DC indexesCQL is based on DC indexes
– Not all of them have USE attribute equivalentsNot all of them have USE attribute equivalents
• E.g.: dc.coverage, dc.rights, etc.E.g.: dc.coverage, dc.rights, etc.
Pattern matchingPattern matching
– AnchorsAnchors
• ‘‘Position last’ has no equivalent position attributePosition last’ has no equivalent position attribute
– WildcardWildcard **
• CQL search terms likeCQL search terms like b*db*d cannot be expressed bycannot be expressed by
truncation attributestruncation attributes

36. 36 ZING: the Next
LibriVision as a ZING clientLibriVision as a ZING client
LibriVision was designed as aLibriVision was designed as a
multiprotocol productmultiprotocol product
It might be easy to incorporate the SRWIt might be easy to incorporate the SRW
or SRU protocolor SRU protocol
Tools are availableTools are available
It could be developed by LibriCoreIt could be developed by LibriCore

37. 37 ZING: the Next
ContentsContents
Why does Z39.50 need renewalWhy does Z39.50 need renewal
What is ZINGWhat is ZING
Technical aspectsTechnical aspects
Social contextSocial context
Amicus/LibriVision and ZINGAmicus/LibriVision and ZING
Experimental SRW client of NSZLExperimental SRW client of NSZL

38. 38 ZING: the Next
Experimental SRW client of NSZLExperimental SRW client of NSZL
Web based clientWeb based client
Tools usedTools used
– YAZ Proxy, PHP, Sablotron, Expat, Java,YAZ Proxy, PHP, Sablotron, Expat, Java,
SaxonSaxon
PlatformsPlatforms
– WindowsWindows
– LinuxLinux

39. 39 ZING: the Next
Experimental SRW client of NSZLExperimental SRW client of NSZL
Flow of informationFlow of information
WebclientWebclient
YAZProxyYAZProxy
Z39.50serverZ39.50server
SRW SearchRetrieve RequestSRW SearchRetrieve Request Z39.50 Search RequestZ39.50 Search Request
SRW SearchRetrieve ResponseSRW SearchRetrieve Response
Z39.50 Present RequestZ39.50 Present Request
Z39.50 Search ResponseZ39.50 Search Response
Z39.50 Present ResponseZ39.50 Present Response

40. 40 ZING: the Next
Experimental SRW client of NSZLExperimental SRW client of NSZL
The target databaseThe target database
– AmicusAmicus
• The Z server used to be ZEDLIB made byThe Z server used to be ZEDLIB made by
CrossnetCrossnet
– Anonymous login (solved)Anonymous login (solved)
– Present problem (solved)Present problem (solved)

41. 41 ZING: the Next
Experimental SRW client of NSZLExperimental SRW client of NSZL
Extra featureExtra feature
– FRBR display is integrated into the clientFRBR display is integrated into the client
Main purposeMain purpose
– To learn about this new protocol in depthTo learn about this new protocol in depth
Live demoLive demo
– http://w3.oszk.hu/zingamicus/index.phphttp://w3.oszk.hu/zingamicus/index.php

42. 42 ZING: the Next
adam@oszk.huadam@oszk.hu
Thank you for your attentionThank you for your attention