Show Sidebar Hide Sidebar

Ridge Coefficients as a Function of the Regularization in Scikit-learn

Shows the effect of collinearity in the coefficients of an estimator.

Ridge Regression is the estimator used in this example. Each color represents a different feature of the coefficient vector, and this is displayed as a function of the regularization parameter.

This example also shows the usefulness of applying Ridge regression to highly ill-conditioned matrices. For such matrices, a slight change in the target variable can cause huge variances in the calculated weights. In such cases, it is useful to set a certain regularization (alpha) to reduce this variation (noise).

When alpha is very large, the regularization effect dominates the squared loss function and the coefficients tend to zero. At the end of the path, as alpha tends toward zero and the solution tends towards the ordinary least squares, coefficients exhibit big oscillations. In practise it is necessary to tune alpha in such a way that a balance is maintained between both.

New to Plotly?

Plotly's Python library is free and open source! Get started by downloading the client and reading the primer.
You can set up Plotly to work in online or offline mode, or in jupyter notebooks.
We also have a quick-reference cheatsheet (new!) to help you get started!

Version

In [1]:
import sklearn
sklearn.__version__
Out[1]:
'0.18.1'

Imports

In [2]:
import plotly.plotly as py
import plotly.graph_objs as go


import numpy as np
from sklearn import linear_model

Calculations

In [3]:
# X is the 10x10 Hilbert matrix
X = 1. / (np.arange(1, 11) + np.arange(0, 10)[:, np.newaxis])
y = np.ones(10)

Compute paths

In [4]:
n_alphas = 200
alphas = np.logspace(-10, -2, n_alphas)
clf = linear_model.Ridge(fit_intercept=False)

coefs = []
for a in alphas:
    clf.set_params(alpha=a)
    clf.fit(X, y)
    coefs.append(clf.coef_)

Plot Results

In [5]:
data = [ ]
y_ = []

for col in range(0, len(coefs[0])):
    y_.append([ ])
    for row in range(0, len(coefs)):
        y_[col].append(coefs[row][col])
        
for i in range(0, len(y_)):
    trace = go.Scatter(y=y_[i], x=alphas,
                       mode='lines', showlegend=False)
    data.append(trace)
    


layout = go.Layout(title='Ridge coefficients as a function of the regularization',
                   hovermode='closest',
                   xaxis=dict(title='alpha', type='log',
                             autorange='reversed'),
                   yaxis=dict(title='weights'))
fig = go.Figure(data=data, layout=layout)
In [6]:
py.iplot(fig)
Out[6]:

License

Author:

    Fabian Pedregosa -- <fabian.pedregosa@inria.fr>

License:

    BSD 3 clause
Still need help?
Contact Us

For guaranteed 24 hour response turnarounds, upgrade to a Developer Support Plan.