Show Sidebar Hide Sidebar

Ridge Coefficients as a Function of the L2 Regularization in Scikit-learn

Ridge Regression is the estimator used in this example. Each color in the left plot represents one different dimension of the coefficient vector, and this is displayed as a function of the regularization parameter. The right plot shows how exact the solution is. This example illustrates how a well defined solution is found by Ridge regression and how regularization affects the coefficients and their values. The plot on the right shows how the difference of the coefficients from the estimator changes as a function of regularization.

In this example the dependent variable Y is set as a function of the input features: y = X*w + c. The coefficient vector w is randomly sampled from a normal distribution, whereas the bias term c is set to a constant. As alpha tends toward zero the coefficients found by Ridge regression stabilize towards the randomly sampled vector w. For big alpha (strong regularisation) the coefficients are smaller (eventually converging at 0) leading to a simpler and biased solution. These dependencies can be observed on the left plot.

The right plot shows the mean squared error between the coefficients found by the model and the chosen vector w. Less regularised models retrieve the exact coefficients (error is equal to 0), stronger regularised models increase the error.

Please note that in this example the data is non-noisy, hence it is possible to extract the exact coefficients.

New to Plotly?

Plotly's Python library is free and open source! Get started by downloading the client and reading the primer.
You can set up Plotly to work in online or offline mode, or in jupyter notebooks.
We also have a quick-reference cheatsheet (new!) to help you get started!

Version

In [1]:
import sklearn
sklearn.__version__
Out[1]:
'0.18.1'

Imports

This tutorial imports make_regression, Ridge and mean_squared_error.

In [2]:
import plotly.plotly as py
import plotly.graph_objs as go
from plotly import tools

import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error

Calculations

In [3]:
clf = Ridge()

X, y, w = make_regression(n_samples=10, n_features=10, coef=True,
                          random_state=1, bias=3.5)

coefs = []
errors = []

alphas = np.logspace(-6, 6, 200)

# Train the model with different regularisation strengths
for a in alphas:
    clf.set_params(alpha=a)
    clf.fit(X, y)
    coefs.append(clf.coef_)
    errors.append(mean_squared_error(clf.coef_, w))

Plot Results

In [4]:
fig = tools.make_subplots(rows=1, cols=2,
                          print_grid=False,
                          subplot_titles=('Ridge coefficients as a function of the regularization',
                                          'Coefficient error as a function of the regularization')
                          )

Ridge coefficients as a function of the regularization

In [5]:
y_ = []

for col in range(0, len(coefs[0])):
    y_.append([ ])
    for row in range(0, len(coefs)):
        y_[col].append(coefs[row][col])
        
for i in range(0, len(y_)):
    trace = go.Scatter(x=alphas, y=y_[i],
               showlegend=False)
    fig.append_trace(trace, 1, 1)
    
fig['layout']['xaxis1'].update(title='alpha', type='log')
fig['layout']['yaxis1'].update(title='weights')

Coefficient error as a function of the regularization

In [6]:
trace = go.Scatter(x=alphas, y=errors,
                   showlegend=False)
fig.append_trace(trace, 1, 2)

fig['layout']['xaxis2'].update(title='alpha', type='log')
fig['layout']['yaxis2'].update(title='errors')
In [7]:
py.iplot(fig)
Out[7]:

License

Author:

    Kornel Kielczewski -- <kornel.k@plusnet.pl>
Still need help?
Contact Us

For guaranteed 24 hour response turnarounds, upgrade to a Developer Support Plan.