A comparison of regularization techniques in the classification of handwritten digits

If dataset is relatively small (e.g. number of samples is less than number of features) or samples are distorted by noise, regularized models built on that dataset often give better results than unregularized models. When problem is ill-conditioned, regularizaton is necessary in order to find solution. For data where neighbouring values are correlated (like in images or time series), not only individual weights, but also differences between them may be penalized in the model. This paper presents results of the experiment, in which several types of regularization (l2, l1, penalized differences) and their combinations were used in fitting logistic regression model (trained using one-vs.-rest strategy) to find which one of them works the best for various sizes of training set. Data used in the experiment came from MNIST dataset, which is publicly available.