Indexing and Retrieving 
Images of Documents
LBSC 796/INFM 718R

David Doermann, UMIACS

October 29th, 2007

Agenda
•	Questions
•	Definitions - Document, Image, Retrieval
•	Document Image Analysis
–	Page decomposition
–	Optical character recognition
•	Traditional Indexing with Conversion
–	Confusion matrix
–	Shape codes
•	Doing things Without Conversion
–	Duplicate Detection, Classification, Summarization, Abstracting
–	Keyword spotting, etc

Goals of this Class
•	Expand your definition of what is a “DOCUMENT”

•	To get an appreciation of the issues in document image analysis and their effects on indexing

•	To look at different ways of solving the same problems with different media

•	Your job: compare/contrast with other media
Quiz
•	What is a document?
Document
•	Basic Medium for Recording Information
•	Transient
–	Space
–	Time 
•	Multiple Forms
–	Hardcopy (paper, stone, ..) / Electronic (CDROM, Internet, …)
–	Written/Auditory/Visual (symbolic, scenic)
•	Access Requirements 
–	Search
–	Browse
–	“Read” 
Sources of Document Images
•	The Web
–	Some PDF files come from scanned documents
–	Arabic news stories are often GIF images
•	Digital copiers
–	Produce “corporate memory” as a byproduct
•	Digitization projects
–	Provide improved access to hardcopy documents
Some Definitions
•	Modality
–	A means of expression
•	Linguistic modalities
–	Electronic text, printed, handwritten, spoken, signed
•	Nonlinguistic modalities
–	Music, drawings, paintings, photographs, video
•	Media
–	The means by which the expression reaches you
•	Internet, videotape, paper, canvas, …
Quiz
•	What is a document?

•	What is an image?
Images
DOCUMENT
Document Images
DOCUMENT
Document Image 
Database
DOCUMENT

Other “Documents”

Quiz
•	What is a document?
•	What is an image?

•	How can we index and retrieve document images?

Indexing Page Images
(Traditional)
Managing Document Image Databases
•	Document Image Databases are often influenced by traditional DB indexing and retrieval philosophies
–	We are comfortable with them
–	They work
•	Problem: Requires content to be accessible
•	Techniques:
–	Content based retrieval (keywords, natural language)
–	Query by structure (logical/physical)
–	Query by Functional attributes (titles, bold, …)
•	Requirements:
–	Ability to Browse, search and read

Document Image Analysis
•	General Flow:
–	Obtain Image - Digitize
–	Preprocessing
–	Feature Extraction
–	Classification
•	General Tasks
–	Logical and Physical Page Structure Analysis
–	Zone Classification
–	Language ID
–	Zone Specific Processing
•	Recognition
•	Vectorization

Quiz
•	What is a document?
•	What is an image?
•	How can we index and retrieve document images?

•	Why is document analysis difficult?

Page Layer Segmentation
Page Analysis
•	Skew correction
–	Based on finding the primary orientation of lines
•	Image and text region detection
–	Based on texture and dominant orientation
•	Structural classification
–	Infer logical structure from physical layout
•	Text region classification
–	Title, author, letterhead, signature block, etc.
Image Detection
	Text Region Detection
More Complex Example
Application to Page Segmentation
Language Identification
•	Language-independent skew detection
–	Accommodate horizontal and vertical writing
•	Script class recognition
–	Asian script have blocky characters
–	Connected scripts can’t be segmented easily
•	Language identification
–	Shape statistics work well for western languages
–	Competing classifiers work for Asian languages

What about handwriting?

Optical Character Recognition
•	Pattern-matching approach
–	Standard approach in commercial systems
–	Segment individual characters
–	Recognize using a neural network classifier
•	Hidden Markov model approach
–	Experimental approach developed at BBN
–	Segment into sub-character slices
–	Limited lookahead to find best character choice
–	Useful for connected scripts (e.g., Arabic)
Quiz
•	What is a document?
•	What is an image?
•	How can we index and retrieve document images?
•	Why is document analysis difficult?

•	Is the (Doc Image IR) problem solved?  Why or Why not?

OCR Accuracy Problems
•	Character segmentation errors
–	In English, segmentation often changes “m” to “rn”
•	Character confusion
–	Characters with similar shapes often confounded
•	OCR on copies is much worse than on originals
–	Pixel bloom, character splitting, binding bend
•	Uncommon fonts can cause problems
–	If not used to train a neural network
Improving OCR Accuracy
•	Image preprocessing
–	Mathematical morphology for bloom and splitting
–	Particularly important for degraded images
•	“Voting” between several OCR engines helps
–	Individual systems depend on specific training data
•	Linguistic analysis can correct some errors
–	Use confusion statistics, word lists, syntax, …
–	But more harmful errors might be introduced
OCR Speed
•	Neural networks take about 10 seconds a page
–	Hidden Markov models are slower

•	Voting can improve accuracy
–	But at a substantial speed penalty

•	Easy to speed things up with several machines
–	For example, by batch processing - using desktop computers at night
Problem: Logical Page Analysis 
(Reading Order)
•	Can be hard to guess in some cases
–	Newspaper columns, figure captions, appendices, …
•	Sometimes there are explicit guides
–	“Continued on page 4” (but page 4 may be big!)
•	Structural cues can help
–	Column 1 might continue to column 2
•	Content analysis is also useful
–	Word co-occurrence statistics, syntax analysis
Processing Converted Text
Typical Document Image Indexing

•	Convert hardcopy to an “electronic” document
–	OCR
–	Page Layout Analysis
–	Graphics Recognition
•	Use structure to add metadata
•	Manually supplement with keywords 

Use traditional text indexing and retrieval techniques?

Information Retrieval on OCR
•	Requires robust ways of indexing
•	Statistical methods with large documents work best
•	Key Evaluations
–	Success for high quality OCR (Croft et al 1994, Taghva 1994)

–	Limited success for poor quality OCR (1996 TREC, UNLV)	
–	Clustering successful for > 85% accuracy  (Tsuda et al, 1995)	

Proposed Solutions
•	Improve OCR
•	Automatic Correction 
–	Taghva et al, 1994
•	Enhance IR techniques 
–	Lopresti and Zhou, 1996
4	NGrams 

Applications 
–	Cornell CS TR Collection (Lagoze et al, 1995)
–	Degraded Text Simulator (Doermann and Yao, 1995)	
N-Grams
•	Powerful, Inexpensive statistical method for characterizing populations
•	Approach 
–	Split up document into n-character pairs fails
–	Use traditional indexing representations to perform analysis
–	“DOCUMENT” -> DOC, OCU, CUM, UME, MEN, ENT
•	Advantages
–	Statistically robust to small numbers of errors
–	Rapid indexing and retrieval
–	Works from 70%-85% character accuracy where traditional IR fails
Matching with OCR Errors
•	Above 80% character accuracy, use words
–	With linguistic correction

•	Between 75% and 80%, use n-grams
–	With n somewhat shorter than usual
–	And perhaps with character confusion statistics

•	Below 75%, use word-length shape codes
Handwriting Recognition
•	With stroke information, can be automated
–	Basis for input pads

•	Simple things can be read without strokes
–	Postal addresses, filled-in forms

•	Free text requires human interpretation
–	But repeated recognition is then possible
Conversion?
•	Full Conversion often required
•	Conversion is difficult!
–	Noisy data
–	Complex Layouts
–	Non-text components

Outline	
•	Processing Converted Text			
•	Manipulating Images of Text	
–	Title Extraction
–	Named Entity Extraction
–	Keyword Spotting
–	Abstracting and Summarization
•	Indexing based on Structure
•	Graphics and Drawings
•	Related Work and Applications
Processing Images of Text
•	Characteristics
–	Does not require expensive OCR/Conversion	
–	Applicable to filtering applications
–	May be more robust to noise

•	Possible Disadvantages
–	Application domain may be very limited
–	Processing time may be an issue if indexing is otherwise required
Proper Noun Detection 
(DeSilva and Hull, 1994)
•	Problem: Filter proper nouns in images of text
–	People, Places, Things 
•	Advantages of the Image Domain: 
–	Saves converting all of the text	
–	Allows application of word recognition approaches
–	Limits post-processing to a subset of words
–	Able to use features which are not available in the text
•	Approach: 
–	Identify Word Features
•	Capitalization, location, length, and  syntactic categories
–	Classify using rule-set
–	Achieve 75-85% accuracy without conversion
 Keyword Spotting	
Techniques:
–	Work Shape/HMM  - (Chen et al, 1995)
–	Word Image Matching - (Trenkle and Vogt, 1993; Hull et al)
–	Character Stroke Features - (Decurtins and Chen, 1995)
4	Shape Coding - (Tanaka and Torii; Spitz 1995; Kia, 1996)

Applications:
–	Filing System (Spitz - SPAM, 1996)
–	Numerous IR
–	Processing handwritten documents
Formal Evaluation :
–	Scribble vs. OCR (DeCurtins, SDIUT 1997) 	 
Shape Coding
•	Approach
–	Use of Generic Character Descriptors
–	Make Use of Power of Language to resolve ambiguity
–	Map Character based on Shape features including ascenders, descenders, punctuation and character with holes

Shape Codes
•	Group all characters that have similar shapes
–	{A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z, 2, 3, 4, 5, 6, 7, 8, 9, 0}
–	{a, c, e, n, o, r, s, u, v, x, z} 
–	{b, d, h, k, }
–	{f, t}
–	{g, p, q, y}
–	{i, j, l, 1}
–	{m, w} 
Why Use Shape Codes?
•	Can recognize shapes faster than characters
–	Seconds per page, and very accurate

•	Preserves recall, but with lower precision
–	Useful as a first pass in any system

•	Easily extracted from JPEG-2 images
–	Because JPEG-2 uses object-based compression
Additional Applications
•	Handwritten Archival Manuscripts 
–	(Manmatha, 1997)
•	Page Classification 
–	(Decurtins and Chen, 1995)
•	Matching Handwritten Records 
–	(Ganzberger et al, 1994)
•	Headline Extraction
•	Document Image Compression (UMD, 1996-1998)	
Evaluation
•	The usual approach: Model-based evaluation
–	Apply confusion statistics to an existing collection

•	A bit better: Print-scan evaluation
–	Scanning is slow, but availability is no problem

•	Best: Scan-only evaluation
–	Few existing IR collections have printed materials
Summary
•	Many applications benefit from image based indexing
–	Less discriminatory features
–	Features may therefore be easier to compute
–	More robust to noise
–	Often computationally more efficient
•	Many classical IR techniques have application for DIR
•	Structure as well as content are important for indexing
•	Preservation of structure is essential for in-depth understanding
Closing thoughts….
•	What else is useful?
–	Document Metadata? – Logos? Signatures?

•	Where is research heading?
–	Cameras to capture Documents?

•	What massive collections are out there?
–	Tobacco Litigation Documents
•	49 million page images
–	Google Books
–	Other Digital Libraries

Additional Reading
•	A. Balasubramanian, et al.  Retrieval from Document Image Collections, Document Analysis Systems VII, pages 1-12, 2006.

•	D. Doermann. The Indexing and Retrieval of Document Images: A Survey. Computer Vision and Image Understanding, 70(3), pages 287-298, 1998.