Speech and Music Retrieval
LBSC 796/INFM 718R
Session 12, November 18, 2007
Douglas W. Oard
Agenda
•	Questions

•	Speech retrieval

•	Music retrieval
Spoken Word Collections
•	Broadcast programming
–	News, interview, talk radio, sports, entertainment
•	Scripted stories
–	Books on tape, poetry reading, theater
•	Spontaneous storytelling
–	Oral history, folklore
•	Incidental recording
–	Speeches, oral arguments, meetings, phone calls
Some Statistics
~250,000 hours of oral history in British Library

>4,000 radio stations webcasting
~35 million hours per year

~100 billion hours of phone calls each year
Economics of the Web

Speech Retrieval Approaches
•	Controlled vocabulary indexing

•	Ranked retrieval based on associated text

•	Automatic feature-based indexing

•	Social filtering based on other users’ ratings

Supporting Information Access
Description Strategies
•	Transcription
–	Manual transcription (with optional post-editing)

•	Annotation
–	Manually assign descriptors to points in a recording
–	Recommender systems (ratings, link analysis, …)

•	Associated materials
–	Interviewer’s notes, speech scripts, producer’s logs

•	Automatic
–	Create access points with automatic speech processing
Detectable Speech Features
•	Content 
–	Phonemes, one-best word recognition, n-best
•	Identity 
–	Speaker identification, speaker segmentation
•	Language
–	Language, dialect, accent
•	Other measurable parameters
–	Time, duration, channel, environment
How Speech Recognition Works
•	Three stages
–	What sounds were made?
•	Convert from waveform to subword units (phonemes)
–	How could the sounds be grouped into words?
•	Identify the most probable word segmentation points
–	Which of the possible words were spoken?
•	Based on likelihood of possible multiword sequences

•	These can be learned from “training data”
–	Hill-climbing sequence models
•	(a “Hidden Markov Model”)
Using Speech Recognition
Phone Lattice
Phoneme Trigrams
•	Manage -> m ae n ih jh
–	Dictionaries provide accurate transcriptions
•	But valid only for a single accent and dialect
–	Rule-base transcription handles unknown words
•	Index every overlapping 3-phoneme sequence
–	m ae n
–	ae n ih
–	n ih jh
Cambridge Video Mail Retrieval
•	Translate queries to phonemes with dictionary
–	Skip stopwords and words with  3 phonemes
•	Find no-overlap matches in the lattice
–	Queries take about 4 seconds per hour of material 
•	Vector space exact word match
–	No morphological variations checked
–	Normalize using most probable phoneme sequence
•	Select from a ranked list of subject lines

Key Results from TREC/TDT
•	Recognition and retrieval can be decomposed
–	Word recognition/retrieval works well in English

•	Retrieval is robust with recognition errors
–	Up to 40% word error rate is tolerable

•	Retrieval is robust with segmentation errors
–	Vocabulary shift/pauses provide strong cues
Competing Demands on the Interface
•	Query must result in a manageable set
–	But users prefer simple query interfaces

•	Selection interface must show several segments
–	Representations must be compact, but informative

•	Rapid examination should be possible
–	But complete access to the recordings is desirable
BBN Radio News Retrieval
AT&T Radio News Retrieval 
MIT “Speech Skimmer”
Comparison with Text Retrieval
•	Detection is harder
–	Speech recognition errors
•	Selection is harder
–	Date and time are not very informative
•	Examination is harder
–	Linear medium is hard to browse
–	Arbitrary segments produce unnatural breaks
A Richer View of Speech
•	Speaker identification
–	Known speaker and “more like this” searches
–	Gender detection for search and browsing

•	Topic segmentation
–	Vocabulary shift, cue words
–	More natural breakpoints for browsing

•	Speaker segmentation
–	Visualize turn-taking behavior for browsing
–	Classify turn-taking patterns for searching
Speaker Identification
•	Gender
–	Classify speakers as male or female
•	Identity
–	Detect speech samples from same speaker
–	To assign a name, need a known training sample
•	Speaker segmentation
–	 Identify speaker changes
–	Count number of speakers
Visualizing Turn-Taking
Broadcast News Retrieval Study
•	NPR Online
•	Manually prepared transcripts
•	Human cataloging

•	SpeechBot
•	Automatic Speech Recognition
•	Automatic indexing

Study Design
•	Seminar on visual and sound materials 
–	Recruited 5 students
•	After training, we provided 2 topics
–	3 searched NPR Online, 2 searched SpeechBot
•	All then tried both systems with a 3rd topic
–	Each choosing their own topic
•	Rich data collection
–	Observation, think aloud, semi-structured interview
•	Model-guided inductive analysis
–	Coded to the model with QSR NVivo

Lessons Learned from the MALACH Project
Douglas W. Oard

College of Information Studies and
Institute for Advanced Computer Studies

Outline
•	The MALACH Project

•	Learning from our users

•	Evaluating new access technologies

•	What’s left to do?
Indexing Options
•	Transcript-based (e.g., NASA)
–	Manual transcription, editing by interviewee

•	Thesaurus-based (e.g., Shoah Foundation)
–	Manually assign descriptors to points in interview

•	Catalog-based (e.g., British Library)
–	Catalog record created from interviewer’s notes

•	Speech-based (MALACH)
–	Create access points with speech processing
Supporting the Search Process
The MALACH Project
The MALACH/CLEF Team
•	USC (Shoah Foundation)
–	Sam Gustman
•	IBM TJ Watson
–	Bhuvana Ramabhadran
–	Martin Franz
•	U. Maryland
–	Doug Oard
–	Dagobert Soergel
•	Johns Hopkins
–	Zak Schafran
•	U. Cambridge (UK)
–	Bill Byrne
•	Charles University (CZ)
–	Jan Hajic
–	Pavel Pecina
•	U. West Bohemia (CZ)
–	Josef Psutka
–	Pavel Ircing
•	Dublin City University (IE)
–	Gareth Jones
•	Budapest U Tech+Econ (HU)
–	Tibor Fegyo
–	Peter Mihajlik

•	Spontaneous conversational speech

•	Digitized

•	Large
–	116,000 hours;  52,000 people;  32 languages

•	Full-description segment indexing
–	10,000 hours, ~150,000 topical segments

•	Real users
Interview Excerpt
•	Content characteristics
–	Domain-specific terms
–	Named entities

•	Audio characteristics
–	Accented (this one is unusually clear)
–	Two channels (interviewer / interviewee)

•	Dialog structure
–	Interviewers have different styles
Outline
•	The MALACH Project

•	Learning from our users

•	Evaluating new access technologies

•	What’s left to do?
Who Uses the Collection?
•	History
•	Linguistics
•	Journalism
•	Material culture
•	Education
•	Psychology
•	Political science
•	Law enforcement
•	Book
•	Documentary film
•	Research paper
•	CDROM
•	Study guide
•	Obituary
•	Evidence
•	Personal use
2002-2003 Observational Studies
   8 independent searchers
–	Holocaust studies (2)
–	German Studies
–	History/Political Science
–	Ethnography
–	Sociology
–	Documentary producer
–	High school teacher

   8 teamed searchers
–	All high school teachers

   Thesaurus-based search

   Rich data collection
–	Intermediary interaction
–	Semi-structured interviews
–	Observer’s notes
–	Participant notes
–	Screen capture

   Qualitative analysis
–	Theory-guided coding
–	Abductive reasoning

Scholars: Topic Types
Basis for Query Reformulation
•	Searcher’s prior knowledge (31%)
•	Viewing an interview or segment (20%)
•	Help from an intermediary (15%)
•	Thesaurus (13%)
•	Assigned descriptors (12%)
•	Pre-interview questionnaire (6%)
•	Result set size (4%)
Teachers: Search Strategies
•	Scenario-guided searching
–	Look for experiences that would reflect themes

•	Proxy searching
–	Descriptors & interview questions

•	Browsing
–	Listened to testimonies for inspiration
–	Used thesaurus for ideas and orientation
Iterating Searching & Planning
“Yesterday in my search, I just felt like I was kind of going around in the dark. But that productive writing session really directed my search.”
Teachers: Relevance Criteria
•	Relevant to teaching content/ method
–	A: Relationship to theme
–	B: As part of broader curriculum
–	C: Characteristics of the story
–	D: Relationship of story to student
–	E: Represents different populations
–	F: Characteristics of Oral History

•	Appropriateness
–	Developmental
–	Acceptability to stakeholders

•	Length-to-contribution ratio

•	Technical production quality
•	B: Relates to other schoolwork
•	B: Variety for the classroom
•	B: Vocabulary

•	C: Positive Message for Students
•	C: Role of Interviewee in Holocaust Events

•	D: Students connect with passage
•	D: Students identify with interviewee
•	D: Radical difference from students’ reality

•	E: Age of interviewee at time of events
•	E: Race

•	F: Expressive power
•	F: Language & verbal expression
•	F: Nonverbal Communication
•	F: Diction
•	F: Flow of interview
Tie Into Curriculum 
•	Relates to other school work

•	Suggests:
–	 Query by published example
–	Using vocabulary lists to prime ASR
Linking Students to a Story 
•	Students identify with interviewee

•	Suggests
–	 Demographic search constraints
–	 Indexing characteristics, not just words
2006 Observational Study (Teachers)

User Study Limitations
•	Generalization
-	Exploratory rather than comparative
-	Results influenced by system and domain

•	Limited number of study participants
-	Insufficient to observe group differences

•	Lack of prior experience with the system
-	Experienced users may behave differently
Outline
•	The MALACH Project

•	Learning from our users

•	Evaluating new access technologies

•	What’s left to do?
English Transcription Accuracy
Speech Retrieval Evaluations
•	1996-1998: TREC SDR
–	EN broadcast news / EN queries

•	1997-2004: TDT
–	EN+CN+AR broadcast news / Query by example

•	2003-2004: CLEF CL-SDR
–	EN broadcast news / Many query languages

•	2005-2007: CLEF CL-SR
–	EN+CZ interviews / Many query languages
English Test Collection Design
English Test Collection Design
2006/07 CL-SR English Collection
•	8,104 topically-coherent segments
–	297 English interviews
–	Known-boundary condition (“Segments”)
–	Average 503 words/segment
•	96 topics
–	Title / Description / Narrative
–	63 training + 33 evaluation
–	6 topic languages (CZ, DE, EN, FR, NL, SP)
•	Relevance judgments
–	Search-guided + “highly ranked” (pooled) 
•	Distributed to track participants by ELDA

Supplementary Resources
•	Thesaurus (Included in the test collection)
~3,000 core concepts
•	Plus alternate vocabulary + standard combinations
~30,000 location-time pairs, with lat/long
Is-a, part-whole, “entry vocabulary”

•	Digitized speech
.mp2 or .mp3

•	In-domain expansion collection (MALACH)
186,000 scratchpad + 3-sentence summaries

Sequence-Based Classification
Sequence Modeling Results
An English Topic
The 118 CL-SR Topics
5-level Relevance Judgments
•	“Classic” relevance (to “food in Auschwitz”)
•	Direct		Knew food was sometimes withheld
•	Indirect	Saw undernourished people

•	Additional relevance types
•	Context	Intensity of manual labor
•	Comparison	Food situation in a different camp
•	Pointer	Mention of a study on the subject
English Assessment Process
•	Search-guided
–	Iterate topic research/query formulation/judging 
•	Grad students in History, trained by librarians
–	Essential for collection reuse with future ASR
–	Done between topic release and submission

•	Highly-ranked (=“Pooled”)
–	Same assessors (usually same individual)
–	Pools formed from two systems per team
•	Top-ranked documents (typically 50)
•	Chosen in order recommended by each team
•	Omitting segments with search-guided judgments
Quality Assurance
•	14 topics independently assessed
–	0.63 topic-averaged kappa (over all judgments)
–	44% topic-averaged overlap (relevant judgments)
–	Assessors later met to adjudicate

•	14 topics assessed and then reviewed
–	Decisions of the reviewer were final

Assessor Agreement (2004)
English Relevance Judgments
CLEF-2006 CL-SR Overview
•	2 tasks: English segments, Czech start times
–	Max of 5 “official” runs per team per task
–	Baseline English run: ASR / English TD topics
•	7 teams / 6 countries
–	Canada: Ottawa (EN, CZ)
–	Czech Republic: West Bohemia (CZ)
–	Ireland: DCU (EN)
–	Netherlands: Twente (EN)
–	Spain: Alicante (EN), UNED (EN)
–	USA: Maryland (EN, CZ)
2006 English Results
Contributions by Assessment Type 
Search-Guided Rankings (2005)
Highly-Ranked Rankings (2005)
Comparing Index Terms (2003)
Maryland English Results (2005)
Comparing ASR with Metadata (2005)

2004 Error Analysis
Correcting Relevant Segments
Other Languages
2006: Czech Test Collection
•	Two sets of speech recognition transcripts
–	Each with both channels recognized

•	Queries in Czech or (for CLIR) English

•	Unknown-boundary evaluation measure
–	Based only on passage start times
–	Start and end time assessments available

Standard Resources for Czech
•	Brain-dead automatic segmentation
–	400-word passages, with 67% overlap
–	Start time used as DOCNO
•	Prague Dependency Treebank (LDC)
–	Czech/English translation lexicon
•	Includes precompiled morphology and probabilities
–	Czech morphology
•	For both formal and colloquial
•	Partially translated thesaurus
–	Most commonly used terms, single translation
Interview Usage
Czech Relevance Assessment
•	Czech subject experts assessors
–	New assessment system for unsegmented audio
–	2 assessor training topics available for debugging

•	Search-guided assessment run in advance
–	Search based on onset marks + ASR results
–	Summaries from translated thesaurus + ASR

•	Highly-ranked after run submission failed
–	Time-base misalignment made this less useful
Czech Assessment System
Czech Relevant Passages
Czech Judgment Rate
Topic-Specific Passages (CZ)
Topic-Independent Passages (EN)
Generalized Average Precision
UWB Czech ASR-Only Results
CLEF CL-SR Legacy
•	Test collection available from ELDA
–	IR
–	CLIR
–	Topic classification
–	Topic segmentation
•	Baseline results
–	Evaluation measures
–	System descriptions

English Information Extraction
English Information Extraction
•	Entity detection
–	In hand-built transcripts: 		F = 66%
–	In speech recognition:		F = 11%

•	Relationship detection
–	In hand-built transcripts:		F = 28%
–	In speech recognition:		F = 0% (!)
Extractive Summarization
Outline
•	The MALACH Project

•	Learning from our users

•	Evaluating new access technologies

•	What’s left to do?
What We’ve Learned
•	ASR-based search works
–	Breakeven for hand transcription: ~1,000 hours
–	Breakeven for thesaurus tagging: ~5,000 hours
–	Inaccessible content clusters by speaker

•	Supervised classification works
–	Global+local sequence provide useful evidence

•	Entity extraction is weak, relationships fail

•	Segmentation should be topic-specific
Things MALACH Won’t Do
•	New guidelines for oral history collection
•	Transcript-based access to oral history audio
•	Less expensive domain tuning for ASR
•	Faster (=cheaper) ASR for large collections
•	Improved handling of out of vocabulary terms
•	Strategies for ASR-based searching
•	Access paradigms other than ranked lists
For More Information

•	CLEF Cross-Language Speech Retrieval track
–	http://clef-clsr.umiacs.umd.edu/

•	The MALACH project
–	http://malach.umiacs.umd.edu/

•	NSF/DELOS Spoken Word Access Group
–	http://www.dcs.shef.ac.uk/spandh/projects/swag

Other Possibly Useful Features
•	Channel characteristics
–	Cell phone, landline, studio mike, ...
•	Accent
–	Another way of grouping speakers
•	Prosody
–	Detecting emphasis could help search or browsing
•	Non-speech audio
–	Background sounds, audio cues
New Zealand Melody Index
•	Index musical tunes as contour patterns
–	Rising, descending, and repeated pitch
–	Note duration as a measure of rhythm
•	Users sing queries using words or la, da, …
–	Pitch tracking accommodates off-key queries
•	Rank order using approximate string match
–	Insert, delete, substitute, consolidate, fragment
•	Display title, sheet music, and audio
Contour Matching Example
•	“Three Blind Mice” is indexed as:
–	*DDUDDUDRDUDRD
•	* represents the first note
•	D represents a descending pitch (U is ascending)
•	R represents a repetition (detectable split, same pitch)
•	My singing produces:
–	*DDUDDUDRRUDRR
•	Approximate string match finds 2 substitutions

Muscle Fish Audio Retrieval
•	Compute 4 acoustic features for each time slice
–	Pitch, amplitude, brightness, bandwidth
•	Segment at major discontinuities
–	Find average, variance, and smoothness of segments
•	Store pointers to segments in 13 sorted lists
–	Use a commercial database for proximity matching
•	4 features, 3 parameters for each, plus duration
–	Then rank order using statistical classification
•	Display file name and audio
Muscle Fish Audio Retrieval
Summary
•	Limited audio indexing is practical now
–	Audio feature matching, answering machine detection

•	Present interfaces focus on a single technology
–	Speech recognition, audio feature matching
–	Matching technology is outpacing interface design
Human History