Show Sidebar Hide Sidebar

Learning Curves in Scikit-learn

On the left side the learning curve of a naive Bayes classifier is shown for the digits dataset. Note that the training score and the cross-validation score are both not very good at the end. However, the shape of the curve can be found in more complex datasets very often: the training score is very high at the beginning and decreases and the cross-validation score is very low at the beginning and increases. On the right side we see the learning curve of an SVM with RBF kernel. We can see clearly that the training score is still around the maximum and the validation score could be increased with more training samples.

New to Plotly?

Plotly's Python library is free and open source! Get started by downloading the client and reading the primer.
You can set up Plotly to work in online or offline mode, or in jupyter notebooks.
We also have a quick-reference cheatsheet (new!) to help you get started!

Version

In [1]:
import sklearn
sklearn.__version__
Out[1]:
'0.18.1'

Imports

In [2]:
print(__doc__)

import plotly.plotly as py
import plotly.graph_objs as go
from plotly import tools

import numpy as np
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.datasets import load_digits
from sklearn.model_selection import learning_curve
from sklearn.model_selection import ShuffleSplit
Automatically created module for IPython interactive environment

Calculations

In [3]:
fig = tools.make_subplots(rows=1, cols=2,
                          subplot_titles=("Learning Curves (Naive Bayes)",
                                          "Learning Curves (SVM, RBF kernel, gamma=0.001)"))

def plot_learning_curve(estimator, X, y, colnum, cv=None,
                        n_jobs=1, train_sizes=np.linspace(.1, 1.0, 5),
                        ):
    """
    Generate a simple plot of the test and training learning curve.

    Parameters
    ----------
    estimator : object type that implements the "fit" and "predict" methods
        An object of that type which is cloned for each validation.

    title : string
        Title for the chart.

    X : array-like, shape (n_samples, n_features)
        Training vector, where n_samples is the number of samples and
        n_features is the number of features.

    y : array-like, shape (n_samples) or (n_samples, n_features), optional
        Target relative to X for classification or regression;
        None for unsupervised learning.

    ylim : tuple, shape (ymin, ymax), optional
        Defines minimum and maximum yvalues plotted.

    cv : int, cross-validation generator or an iterable, optional
        Determines the cross-validation splitting strategy.
        Possible inputs for cv are:
          - None, to use the default 3-fold cross-validation,
          - integer, to specify the number of folds.
          - An object to be used as a cross-validation generator.
          - An iterable yielding train/test splits.

        For integer/None inputs, if ``y`` is binary or multiclass,
        :class:`StratifiedKFold` used. If the estimator is not a classifier
        or if ``y`` is neither binary nor multiclass, :class:`KFold` is used.

        Refer :ref:`User Guide <cross_validation>` for the various
        cross-validators that can be used here.

    n_jobs : integer, optional
        Number of jobs to run in parallel (default 1).
    """
    
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)
    
    if(colnum==1):
        leg=True
    else:
        leg=False
        
    p1 = go.Scatter(x=train_sizes, y=test_scores_mean + test_scores_std,
                    mode='lines',
                    line=dict(color="green", width=1),
                    showlegend=False,  )
    fig.append_trace(p1, 1, colnum)
    
    p2 = go.Scatter(x=train_sizes, y=test_scores_mean - test_scores_std,
                    mode='lines',
                    line=dict(color="green", width=1),
                    showlegend=False, fill='tonexty')
    fig.append_trace(p2, 1, colnum)
    
    p3 = go.Scatter(x=train_sizes, y=train_scores_mean + train_scores_std,
                    mode='lines',
                    line=dict(color="red", width=1),
                    showlegend=False)
    fig.append_trace(p3, 1, colnum)
    
    p4 = go.Scatter(x=train_sizes, y=train_scores_mean - train_scores_std,
                    mode='lines',
                    line=dict(color="red", width=1),
                    showlegend=False, fill='tonexty')
    fig.append_trace(p4, 1, colnum)
    
    p5 = go.Scatter(x=train_sizes, y=train_scores_mean, 
                    marker=dict(color='red'),
                    name="Training score", showlegend=leg)
    fig.append_trace(p5, 1, colnum)
    
    p6 = go.Scatter(x=train_sizes, y=test_scores_mean, 
                    marker=dict(color='green'),
                    name="Cross-validation score", showlegend=leg)
    fig.append_trace(p6, 1, colnum)
This is the format of your plot grid:
[ (1,1) x1,y1 ]  [ (1,2) x2,y2 ]

Plot Results

In [4]:
digits = load_digits()
X, y = digits.data, digits.target
 
# Cross validation with 100 iterations to get smoother mean test and train
# score curves, each time with 20% data randomly selected as a validation set.
cv = ShuffleSplit(n_splits=100, test_size=0.2, random_state=0)

estimator = GaussianNB()
plot_learning_curve(estimator,  X, y, 1, cv=cv, n_jobs=4)


# SVC is more expensive so we do a lower number of CV iterations:
cv = ShuffleSplit(n_splits=10, test_size=0.2, random_state=0)
estimator = SVC(gamma=0.001)
plot_learning_curve(estimator,  X, y, 2, cv=cv, n_jobs=4)

fig['layout'].update(hovermode='closest')
In [5]:
py.iplot(fig)
Out[5]:
Still need help?
Contact Us

For guaranteed 24 hour response turnarounds, upgrade to a Developer Support Plan.