W17: A N A NA LY S I S O F I N F E R E N C E W I T H T H E U N I V E R S U M
FA B I A N S I N Z , O L I V I E R C H A P E L L E , A L E K H AG A RWA L & B E R N H A R D S C H Ö L KO P F
Data-dependent regularization with the Universum U = {z 1 , ..., zq } :
min
w ,b

X X 1 2 ||w || + CL H+1,-1 [ w , x i + b] + CU I[ w , z j + b ] 2 i =1 j =1

m

q

with H =

and I =
minimize

(U-SVM) or H =

and I =

(Uls -SVM)

Results for U-SVM
For CU =  the solution w  lives on the or thogonal complement of the affine subspacespanned by U

w

µU

For CU <  the U-SVM minimizes an upper bound on the "L1 -variance" P P CU q=1 | w  , z j + b |  CU minb q=1 | w  , z j + b| j j

Results for Uls -SVM
minimize

The Uls -SVM is equivalent (up to a linear constraint) to a hybird of oriented PCA and Fisher Disriminant Analysis where U specifies the "noise directions" in oriented PCA

Results
Both, the absolute position as well as the main principal axes influence the solution w 

The Universum set U allows to specify implicit invariance directions but must be carefully chosen
F, S I N Z , O . C H A P E L L E , A . AG A RWA L & B . S C H Ö L KO P F
M A X P L A N C K I N S T I T U T E T Ü B I N G E N / YA H O O R E S E A R C H / U C B E R K E LY