![\"Neo4j](\"pics/tokenization-spacy.svg\")
\n",
" Image source: Spacy.io \n", "
SC 4125: Developing Data Products
\n", "Module-6: Statistical tools for natural language processing and machine learning
\n",
" ![\"modelcycle\"](\"pics/modelcycle.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Model training, tuning and evaluation \n",
"\n",
"> How well does the model do with data not seen during fitting?\n",
"> - for data which has already been seen, a memory based model would be always \"perfect\"\n",
" \n",
"![\"bigpic\"](\"pics/traintest.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Cross validation\n",
"Variants: \n",
"- K-fold\n",
"- Stratified: Each fold maintains proportional representation of classes\n",
" * sklearn.model_selection.StratifiedKFold\n",
"- Leave one out\n",
"\n",
"![\"bigpic\"](\"pics/crosstrain.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Validation curve\n",
"\n",
"![\"validation](\"pics/validationcurve.png\")
\n",
"\n",
"> In this plot (see source for details and source-code) we see the training scores and validation scores of an SVM for different values of the kernel parameter $\\gamma$, from which we can idenitfy the regions of under-fitting and over-fitting. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"#### Data leakage\n",
"\n",
"> Leakage is essentially the introduction of information about the data mining target, which should not be legitimately available to mine from. \n",
"> - This may lead to very good performance during the model tuning and evaluation, and yet, lead to poor performance in production environment\n",
"> * Recommeded reading: Leakage in Data Mining:\n",
"Formulation, Detection, and Avoidance by Kaufman et al. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### Holdout data for evaluation\n",
"Separate from the data used in cross-validation\n",
"\n",
"![\"bigpic\"](\"pics/holdouteval.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Accuracy\n",
"Limitations of accuracy as a metric\n",
"\n",
"![\"classifier](\"pics/classifierchoices.png\")
![\"bigpic\"](\"pics/precisionrecall1.png\")
| | ![\"Sampling\"](\"pics/precisionrecall2.png\")
\n",
"Image adapted from https://en.wikipedia.org/wiki/Precision_and_recall"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"- When trying to detect malignant tumors, will you prioritize precision or recall?\n",
"\n",
"- When trying to detect fraudulent credit-card transactions in real-time, will you prioritize precision or recall?\n",
"\n",
"- When making recommendations for jobs, what will give a person a better user experience?\n",
" * Recall earlier discussions\n",
" * Data Jujitsu"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Precision recall curve"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"precision: [0.4805 0.45368421 0.42444444 0.39235294 0.35875 0.32133333\n",
" 0.28357143 0.23769231 0.18583333 0.14454545 0.108 0.08333333\n",
" 0.05875 0.05142857 0.04166667 0.032 0.0275 0.01666667\n",
" 0.02 0.02 1. ] \n",
"\n",
" recall: [1. 0.89698231 0.7950052 0.69406868 0.59729448 0.50156087\n",
" 0.41311134 0.32154006 0.23204995 0.16545265 0.11238293 0.0780437\n",
" 0.04890739 0.03746098 0.02601457 0.01664932 0.01144641 0.00520291\n",
" 0.00416233 0.00208117 0. ]\n"
]
}
],
"source": [
"# Precision recall curve\n",
"from sklearn.metrics import precision_recall_curve\n",
"from sklearn.metrics import PrecisionRecallDisplay\n",
"predictions_prob=[x[0] for x in model3.predict_proba(cv3.transform(X_test))]\n",
"precision, recall, _ = precision_recall_curve(y_test, predictions_prob)\n",
"print(\"precision:\",precision[::100],\"\\n\\n\", \"recall:\",recall[::100])"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"![\"ROC/AUC\"](\"pics/Roc_curve.png\")
![\"OVR\"](\"pics/oneversusrest.png\")
| ![\"OVO\"](\"pics/oneversusone.png\")
"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"{False: 5, True: 145}"
]
},
"execution_count": 78,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Scikit-learn allows to choose, e.g., SVC uses decision_function_shape{‘ovo’, ‘ovr’}, default=’ovr’\n",
"# An alternative is to explicitly use the multiclass library \n",
"from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.svm import LinearSVC\n",
"from sklearn import datasets\n",
"iris_X, iris_y = datasets.load_iris(return_X_y=True)\n",
"iris_X = StandardScaler().fit_transform(iris_X)\n",
"# Caution: I have not used train/test splits in this example\n",
"OvR=OneVsRestClassifier(LinearSVC(random_state=0)).fit(iris_X, iris_y).predict(iris_X)\n",
"OvO=OneVsOneClassifier(LinearSVC(random_state=0)).fit(iris_X, iris_y).predict(iris_X)\n",
"unique, counts = np.unique(OvR==OvO, return_counts=True)\n",
"dict(zip(unique, counts))"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Grid search\n",
"Exploring the parameter space to identify good/optimal parameters\n",
"- Parameter choices depend on the algorithm being used \n",
"- See Scikit-learn examples for more details and sophisticated use cases. "
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"GridSearchCV(estimator=SVC(),\n",
" param_grid={'C': range(1, 11, 2), 'kernel': ('linear', 'rbf')})"
]
},
"execution_count": 79,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# A very basic example of Grid search \n",
"from sklearn.ensemble import RandomForestClassifier \n",
"from sklearn.model_selection import GridSearchCV\n",
"iris = datasets.load_iris()\n",
"iris_X, iris_y = datasets.load_iris(return_X_y=True)\n",
"parameters = {'kernel':('linear', 'rbf'), 'C':range(1,11,2)}\n",
"# Remember: Parameter choices depend on the algorithm being used\n",
"svc = SVC()\n",
"clf = GridSearchCV(estimator=svc, param_grid=parameters)\n",
"clf.fit(iris_X, iris_y)"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time', 'param_C', 'param_kernel', 'params', 'split0_test_score', 'split1_test_score', 'split2_test_score', 'split3_test_score', 'split4_test_score', 'mean_test_score', 'std_test_score', 'rank_test_score']\n"
]
}
],
"source": [
"results=clf.cv_results_\n",
"# What all is available in the results?\n",
"print(list(results.keys()))"
]
},
{
"cell_type": "code",
"execution_count": 81,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([0.98 , 0.96666667, 0.97333333, 0.98 , 0.98 ,\n",
" 0.98666667, 0.97333333, 0.98666667, 0.97333333, 0.98 ])"
]
},
"execution_count": 81,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results['mean_test_score']"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 3, 10, 7, 3, 3, 1, 7, 1, 7, 3])"
]
},
"execution_count": 82,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results['rank_test_score']"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"#### Model tuning and evaluation wrap-up\n",
"\n",
"- Our discussion was anchored around supervised learning, and for Classification task in particular\n",
" * Metrics for Regression are simpler, e.g., r2 score, mean absolute error, etc\n",
" * For unsupervised learning, e.g., Clustering, some of the measures we saw here, e.g., F-score can be repurposed, if (a big IF!) the \"ground truth\" is known. \n",
"- While many measures exist and several/all of them can be used to understand the general behaviour of the models, final evaluation and optimization may need to be use-case driven.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Transformations\n",
"- Revisting representation phase of the ML-lifecycle\n",
" - We have already discussed some basics: Data cleaning, removal of samples with missing values, syntactic conversion of information so that the algorithms can \"consume\" the data (e.g., CountVectorizer) \n",
" - Standardization: Normalization, scaling, etc.\n",
" * See sklearn.preprocessing\n",
" * StandardScaler: Standardize features by removing the mean and scaling to unit variance.\n",
" * MinMaxScaler: Transform features by scaling each feature to a given range."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"## Transformations\n",
"- Revisting representation\n",
" - Data embedding/dimensionality reduction: Principal Component Analysis (PCA), Linear discriminant analysis (LDA), manifold, etc.\n",
" \n",
"Recommended reading: Blog showcasing multiple dimensionality reduction techniques. "
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"### Philosophical tangent: PCA vs LDA \n",
"PCA | LDA\n",
":-------------------------:|:-------------------------:\n",
"![\"bigpic\"](\"pics/pca.png\")
| ![\"Sampling\"](\"pics/lda.png\")
\n",
"Image source https://sebastianraschka.com/. See also scikit-learn PCA & LDA usage example."
]
},
{
"cell_type": "code",
"execution_count": 83,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# The following example of using PCA, Standardization and make_pipeline is adapted from \n",
"# https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html#sphx-glr-auto-examples-preprocessing-plot-scaling-importance-py\n",
"# It has been made available under License: BSD 3 clause by\n",
"# Tyler Lanigan ![\"Do](\"pics/electronicsheep2.png\")
\n",
"