Web Characterization
Week 11
LBSC 690
Information Technology
The Why of the Web (in 1995)
•	Affordable storage
–	300,000 words/$
•	Adequate backbone capacity
–	25,000 simultaneous transfers
•	Adequate “last mile” bandwidth
–	1 second/screen
•	Display capability
–	10% of US population
•	Effective search capabilities
–	Lycos, Yahoo
Defining the Web
•	HTTP, HTML, or URL?

•	Static, dynamic or streaming?

•	Public, protected, or internal?

•	Content or behavior?

Number of Web Sites
Discussion Topic:
What’s a Web “Site”?
•	OCLC counted any server at port 80
–	Misses many servers at other ports

•	Some servers host unrelated content
–	Geocities

•	Some content requires specialized servers
–	rtsp
Crawling the Web
Link Structure of the Web
Web Crawl Challenges
•	Discovering “islands” and “peninsulas”

•	Duplicate and near-duplicate content
–	30-40% of total content

•	Server and network loads 

•	Dynamic content generation

•	Link rot
–	Changes at 1% per week

•	Temporary server interruptions
Duplicate Detection
•	Structural
–	Identical directory structure (e.g., mirrors, aliases)

•	Syntactic
–	Identical bytes
–	Identical markup (HTML, XML, …)

•	Semantic
–	Identical content
–	Similar content (e.g., with a different banner ad)
–	Related content (e.g., translated)
Robots Exclusion Protocol
•	Requires voluntary compliance by crawlers

•	Exclusion by site
–	Create a robots.txt file at the server’s top level
–	Indicate which directories not to crawl

•	Exclusion by document (in HTML head)
–	Not implemented by all crawlers
	<meta name="robots“ content="noindex,nofollow"> 
Hands on:
The Internet Archive
•	alexa.com Web crawls since 1997
–	http://archive.org

•	Check out the CLIS Web site from 1998!

•	Check out the history of your favorite site
Discussion Point
•	Can we save everything?

•	Should we?

•	Do people have a right to remove things?
The “Deep Web”
•	Dynamic pages, generated from databases

•	Not easily discovered using crawling 

•	Perhaps 400-500 times larger than surface Web

•	Fastest growing source of new information

Content of the Deep Web
Deep Web
•	60 Deep Sites Exceed Surface Web by 40 Times

Global Internet Users
Global Internet Users
World Trade in 2001
European Web Content
Blogs

A Web of Speech?
Rethinking the Spoken Word
•	Speech is better for some things than writing 

•	Spoken bits are as persistent as written bits

•	Storage costs is 80 times more than text 
–	Disk cost falls by a factor of 80 in ~16 years

•	If speech is searchable, we will keep lots of it
A Little Math
•	Collectable spoken words ≈ 10 Tw/day
–	1 billion users * 100 words/min * 200 min/day / 2
•	Compressed speech ≈ 2 words/kiloByte
–	(100/60 w/sec) * (6.5 kb/sec / 8 b/B)
•	Required storage ≈ 5 PetaBytes/day

A Little Math
•	Collectable spoken words ≈ 10 Tw/day
–	1 billion users * 100 words/min * 200 min/day / 2
•	Compressed speech ≈ 2 words/kiloByte
–	(100/60 w/sec) * (6.5 kb/sec / 8 b/B)
•	Required storage ≈ 5 PetaBytes/day

•	Storage array sales > 5 PB/day
–	457 PB in 2Q 2005 (increasing 59% per year)

•	$22/person/year (decreasing at 31%/year)
Human History
Hands On: Speech on the Web
•	audio.search.yahoo.com

•	blinkx.com

•	ocw.mit.edu

•	podcasts.net

Estimating Authority from Links
Collecting Click Streams
•	Browsing histories are easily captured
–	Make all links initially point to a central site
•	Encode the desired URL as a parameter
–	Build a time-annotated transition graph for each user
•	Cookies identify users (when they use the same machine)
–	Redirect the browser to the desired page

•	Reading time is correlated with interest
–	Can be used to build individual profiles
–	Used to target advertising by doubleclick.com
Search Engine
Query Logs
Search Engine Query Logs
http://hannu.biz/aolsearch/
AOL User 4417749
Gaining Access to Observations
•	Observe public behavior
–	Hypertext linking, publication, citing, …

•	Policy protection
–	EU: Privacy laws
–	US: Privacy policies + FTC enforcement

•	Statistical assurance of privacy
–	Distributed architecture
–	Model and mitigate privacy risks

More Complete Observations
•	User selects an article
–	Interpretation: Summary was interesting
•	User quickly prints the article
–	Interpretation: They want to read it
•	User selects a second article
–	Interpretation: another interesting summary
•	User scrolls around in the article
–	Interpretation: Parts with high dwell time and/or repeated revisits are interesting
•	User stops scrolling for an extended period
–	Interpretation: User was interrupted

Critical Issues
•	Protecting privacy
–	What absolute assurances can we provide?
–	How can we make remaining risks understood?

•	Scalable rating servers
–	Is a fully distributed architecture practical?

•	Non-cooperative users
–	How can the effect of spamming be limited?