Ridge Regression is the estimator used in this example. Each color in the left plot represents one different dimension of the coefficient vector, and this is displayed as a function of the regularization parameter. The right plot shows how exact the solution is. This example illustrates how a well defined solution is found by Ridge regression and how regularization affects the coefficients and their values. The plot on the right shows how the difference of the coefficients from the estimator changes as a function of regularization.
In this example the dependent variable Y is set as a function of the input features: y = X*w + c. The coefficient vector w is randomly sampled from a normal distribution, whereas the bias term c is set to a constant. As alpha tends toward zero the coefficients found by Ridge regression stabilize towards the randomly sampled vector w. For big alpha (strong regularisation) the coefficients are smaller (eventually converging at 0) leading to a simpler and biased solution. These dependencies can be observed on the left plot.
The right plot shows the mean squared error between the coefficients found by the model and the chosen vector w. Less regularised models retrieve the exact coefficients (error is equal to 0), stronger regularised models increase the error.
Please note that in this example the data is non-noisy, hence it is possible to extract the exact coefficients.
import sklearn sklearn.__version__
import plotly.plotly as py import plotly.graph_objs as go from plotly import tools import numpy as np from sklearn.datasets import make_regression from sklearn.linear_model import Ridge from sklearn.metrics import mean_squared_error
clf = Ridge() X, y, w = make_regression(n_samples=10, n_features=10, coef=True, random_state=1, bias=3.5) coefs =  errors =  alphas = np.logspace(-6, 6, 200) # Train the model with different regularisation strengths for a in alphas: clf.set_params(alpha=a) clf.fit(X, y) coefs.append(clf.coef_) errors.append(mean_squared_error(clf.coef_, w))
fig = tools.make_subplots(rows=1, cols=2, print_grid=False, subplot_titles=('Ridge coefficients as a function of the regularization', 'Coefficient error as a function of the regularization') )
Ridge coefficients as a function of the regularization
y_ =  for col in range(0, len(coefs)): y_.append([ ]) for row in range(0, len(coefs)): y_[col].append(coefs[row][col]) for i in range(0, len(y_)): trace = go.Scatter(x=alphas, y=y_[i], showlegend=False) fig.append_trace(trace, 1, 1) fig['layout']['xaxis1'].update(title='alpha', type='log') fig['layout']['yaxis1'].update(title='weights')
Coefficient error as a function of the regularization
trace = go.Scatter(x=alphas, y=errors, showlegend=False) fig.append_trace(trace, 1, 2) fig['layout']['xaxis2'].update(title='alpha', type='log') fig['layout']['yaxis2'].update(title='errors')
Kornel Kielczewski -- <email@example.com>