Evidence from Metadata
LBSC 796/INFM 718R
Session 9: November 5, 2007
Douglas W. Oard
Problems with “Free Text” Search
•	Homonymy
–	Terms may have many unrelated meanings
–	Polysemy (related meanings) is less of a problem

•	Synonymy
–	Many ways of saying (nearly) the same thing

•	Anaphora
–	Alternate ways of referring to the same thing
Behavior Helps, But not Enough
•	Privacy limits access to observations

•	Queries based on behavior are hard to craft
–	Explicit queries are rarely used
–	Query by example requires behavior history

•	“Cold start” problem limits applicability
A “Solution:” Concept Retrieval
•	Develop a concept inventory
–	Uniquely identify concepts using “descriptors”
–	Concept labels form a “controlled vocabulary”
–	Organize concepts using a “thesaurus”

•	Assign concept descriptors to documents
–	Known as “indexing”

•	Craft queries using the controlled vocabulary

Two Ways of Searching
Boolean Search Example
•	Canine AND Fox
–	Doc 1
•	Canine AND Political action
–	Empty
•	Canine OR Political action
–	Doc 1, Doc 2
Applications
•	When implied concepts must be captured
–	Political action, volunteerism, …
•	When terminology selection is impractical
–	Searching foreign language materials
•	When no words are present
–	Photos w/o captions, videos w/o transcripts, …
•	When user needs are easily anticipated
–	Weather reports, yellow pages, …
Agenda
•	Designing metadata

•	Generating metadata

•	Putting the pieces together
Aspects of Metadata
•	What kinds of objects can we describe?
–	MARC, Dublin Core, FRBR, …

•	How can we convey it?
–	MODS, RDF, OAI-PMH, METS

•	What can we say?
–	LCSH, MeSH, PREMIS, …

•	What can we do with it?
–	Discovery, description, reasoning
Functional Requirements for Bibliographic Records (FRBR)
•	Work (e.g., a specific play)
–	Expression (e.g., a specific performance)
•	Manifestation (e.g., a specific publisher’s DVD)
–	Item (e.g., a specific DVD)

•	Responsible Entities (person, corporate body)

•	Subject (concept, object, event, place)
FRBR in OCLC’s FictionFinder
Dublin Core
•	Goals: 
–	Easily understood, implemented and used
–	Broadly applicable to many applications
•	Approach:
–	Intersect several  standards (e.g., MARC)
–	Suggest only “best practices” for element content
•	Implementation:
–	Initially 15 optional and repeatable “elements”
•	Refined using a growing set of “qualifiers”
–	Now extended to 22 elements
Dublin Core Elements (version 1.1)
Content
•	Title
•	Subject [LCSH, MeSH, …]
•	Description
•	Type
•	Coverage [spatial, temporal, …]
•	Related resource
•	Rights
Instantiation
•	Date [Created, Modified, Copyright, …]
•	Format
•	Language
•	Identifier [URI, Citation, …]

Responsibility
•	Creator
•	Contributor
•	Source
•	Publisher
Resource Description Framework
•	XML schema for describing resources

•	Can integrate multiple metadata standards 
–	Dublin Core, P3P, PICS, vCARD, …

•	Dublin Core provides a XML “namespace”
–	DC Elements are XML “properties
•	DC Refinements are RDF “subproperties”
–	Values are XML “content”
A Rose By Any Other Name …
Open Archives Initiative-
Protocol for Metadata Harvesting
(OAI-PMH)
Metadata Encoding and Transmission Standard (METS)
•	Descriptive metadata (e.g., subject, author)
•	Administrative metadata (e.g., rights, provenance)
•	Technical metadata (e.g., resolution, color space)
•	Behavior (which program can render this?)
•	Structural map (e.g., page order)
–	Structural links (e.g., Web site navigation links)
•	Files (the raw data)
•	Root (meta-metadata!)
Open Archival Information System
(OAIS) Reference Model
Agenda
•	Designing metadata

•	Generating metadata

•	Putting the pieces together
Thesaurus Design
•	Thesaurus must match the document collection
–	Literary warrant

•	Thesaurus must match the information needs
–	User-centered indexing

•	Thesaurus can help to guide the searcher
–	Broader term (“is-a”), narrower term, used for, …

Challenges
•	Changing concept inventories
–	Literary warrant and user needs are hard to predict

•	Accurate concept indexing is expensive
–	Machines are inaccurate, humans are inconsistent

•	Users and indexers may think differently
–	Diverse user populations add to the complexity

•	Using thesauri effectively requires training
–	Meta-knowledge and thesaurus-specific expertise
Machine-Assisted Indexing
•	Goal:  Automatically suggest descriptors
–	Better consistency with lower cost

•	Approach:  Rule-based expert system
–	Design thesaurus by hand in the usual way
–	Design an expert system to process text
•	String matching, proximity operators, …
–	Write rules for each thesaurus/collection/language
–	Try it out and fine tune the rules by hand
 Machine-Assisted Indexing Example
Machine Learning: kNN Classifier
“Folksonomies”
“Named Entity” Tagging
•	Machine learning techniques can find:
–	Location
–	Extent
–	Type

•	Two types of features are useful
–	Orthography
•	e.g., Paired or non-initial capitalization
–	Trigger words
•	e.g., Mr., Professor, said, …

Normalization
•	Variant forms of names (“name authority”)
–	Pseudonyms, partial names, citation styles

•	Acronyms and abbreviations
–	Organizations, political entities, projects, …

•	Co-reference resolution
–	References to roles or objects rather than names
–	Anaphoric pronouns for an antecedent name
Example: Bibliographic References
Agenda
•	Designing metadata

•	Generating metadata

•	Putting the pieces together
Supporting the Search Process
Putting It All Together
Before You Go!
   On a sheet of paper, please briefly answer the following question (no names):

   What was the muddiest point in today’s class?