595 lines
15 KiB
Plaintext
595 lines
15 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "dffa60e8",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%matplotlib inline"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3340eabf",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"Inclass-Lab\n",
|
|
"\n",
|
|
"Automatically generated by Colab.\n",
|
|
"\n",
|
|
"Original file is located at\n",
|
|
" https://colab.research.google.com/drive/13N7lQHvv4_LxKgcerumJ-18e5GfdBggm"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "34271feb",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9491847c",
|
|
"metadata": {
|
|
"cell_marker": "\"\"\""
|
|
},
|
|
"source": [
|
|
"In-Class Lab: Comparing Classification Models (Breast Cancer Dataset)\n",
|
|
"\n",
|
|
"Learning Objectives\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"\n",
|
|
"By the end of this lab, you should be able to:\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"* Train multiple models (KNN, Logistic Regression, Decision Tree)\n",
|
|
"* Compare performance across various scenarios\n",
|
|
"* Understand the effect of scaling and evaluation methods\n",
|
|
"* Interpret confusion matrix & classification report\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cceb43dc",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 1:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "818bfdad",
|
|
"metadata": {
|
|
"cell_marker": "'''",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"Load dataset\n",
|
|
"Explain features/target\n",
|
|
"Do train_test_split"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6de94b5d",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "a8899ca0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.datasets import load_breast_cancer\n",
|
|
"data = load_breast_cancer()\n",
|
|
"X = data.data\n",
|
|
"y = data.target\n",
|
|
"\n",
|
|
"print(data.target_names)\n",
|
|
"print(X.shape)\n",
|
|
"#0 - malignant\n",
|
|
"#1 - benigh"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ae1b0d61",
|
|
"metadata": {
|
|
"cell_marker": "###############################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 2: IMPORT LIBRARIES FOR CLASSIFIERS\n",
|
|
"##############################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8db6dfd6",
|
|
"metadata": {
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.neighbors import KNeighborsClassifier\n",
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
|
"from sklearn.tree import DecisionTreeClassifier\n",
|
|
"import numpy as np"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a2301b40",
|
|
"metadata": {
|
|
"cell_marker": "#########################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 3: IMPORT MODUL FOR DATA SPLIT\n",
|
|
"########################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "75fe2532",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.model_selection import train_test_split"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fb69921d",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 4: SPLIT THE DATASET\n",
|
|
"###################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6eb66fc0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b182e930",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 5\n",
|
|
"TASK 1: Train 3 models (no scaling)\n",
|
|
"###################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f31f0262",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"knn = KNeighborsClassifier(n_neighbors=5)\n",
|
|
"logreg = LogisticRegression(max_iter=5000)\n",
|
|
"tree = DecisionTreeClassifier()\n",
|
|
"\n",
|
|
"knn.fit(X_train, y_train)\n",
|
|
"logreg.fit(X_train, y_train)\n",
|
|
"tree.fit(X_train, y_train)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d3e0088e",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 6\n",
|
|
"Task 2: Evaluate accuracy"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3a8845a1",
|
|
"metadata": {
|
|
"cell_marker": "'''",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"* Which model performs best?\n",
|
|
"* Are the results very different?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d49b62b3",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "395d887c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(\"KNN:\", knn.score(X_test, y_test))\n",
|
|
"print(\"LogReg:\", logreg.score(X_test, y_test))\n",
|
|
"print(\"Tree:\", tree.score(X_test, y_test))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9de74c34",
|
|
"metadata": {
|
|
"cell_marker": "####################################"
|
|
},
|
|
"source": [
|
|
"BLOCK 7\n",
|
|
"Part 2: Evaluation Metrics\n",
|
|
"Task 3: Confusion Matrix"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "deec64c6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.metrics import confusion_matrix\n",
|
|
"print(confusion_matrix(y_test, knn.predict(X_test)))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "18bc0c0e",
|
|
"metadata": {
|
|
"cell_marker": "#####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 8: CLASSIFICATION FREPORT"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a2e8747f",
|
|
"metadata": {
|
|
"cell_marker": "'''",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"* Which model has better recall?\n",
|
|
"* Which is better for detecting cancer?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "88303af7",
|
|
"metadata": {
|
|
"cell_marker": "#####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "e9ef7bea",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.metrics import classification_report\n",
|
|
"print(classification_report(y_test, knn.predict(X_test)))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c1d8978a",
|
|
"metadata": {
|
|
"cell_marker": "########################################"
|
|
},
|
|
"source": [
|
|
"BLOCK 9:\n",
|
|
"Part 3: Scaling Effect\n",
|
|
"Task 5: Apply scaling using pipeline\n",
|
|
"#######################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "84e1f5d4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.pipeline import Pipeline\n",
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
|
"from sklearn.metrics import accuracy_score, confusion_matrix, classification_report\n",
|
|
"\n",
|
|
"knn_scaled = Pipeline([\n",
|
|
" ('scaler', StandardScaler()),\n",
|
|
" ('knn', KNeighborsClassifier(n_neighbors=5))\n",
|
|
"])\n",
|
|
"\n",
|
|
"logreg_scaled = Pipeline([\n",
|
|
" ('scaler', StandardScaler()),\n",
|
|
" ('logreg', LogisticRegression(max_iter=5000))\n",
|
|
"])\n",
|
|
"\n",
|
|
"tree_scaled = Pipeline([\n",
|
|
" ('scaler', StandardScaler()),\n",
|
|
" ('tree', DecisionTreeClassifier(random_state=42))\n",
|
|
"])\n",
|
|
"\n",
|
|
"knn_scaled.fit(X_train, y_train)\n",
|
|
"logreg_scaled.fit(X_train, y_train)\n",
|
|
"tree_scaled.fit(X_train, y_train)\n",
|
|
"\n",
|
|
"knn_pred_scaled = knn_scaled.predict(X_test)\n",
|
|
"logreg_pred_scaled = logreg_scaled.predict(X_test)\n",
|
|
"tree_pred_scaled = tree_scaled.predict(X_test)\n",
|
|
"\n",
|
|
"print(\"\\n=== RESULTS WITH SCALING ===\")\n",
|
|
"print(\"KNN Accuracy:\", accuracy_score(y_test, knn_pred_scaled))\n",
|
|
"print(\"Logistic Regression Accuracy:\", accuracy_score(y_test, logreg_pred_scaled))\n",
|
|
"print(\"Decision Tree Accuracy:\", accuracy_score(y_test, tree_pred_scaled))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d5caa8ec",
|
|
"metadata": {
|
|
"cell_marker": "#####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 10: CLASSIFICATION REPORT\n",
|
|
"####################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6643f0af",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(\"\\n=== CLASSIFICATION REPORTS (WITH SCALING) ===\")\n",
|
|
"\n",
|
|
"print(\"\\nKNN Report:\")\n",
|
|
"print(classification_report(y_test, knn_pred_scaled))\n",
|
|
"\n",
|
|
"print(\"\\nLogistic Regression Report:\")\n",
|
|
"print(classification_report(y_test, logreg_pred_scaled))\n",
|
|
"\n",
|
|
"print(\"\\nDecision Tree Report:\")\n",
|
|
"print(classification_report(y_test, tree_pred_scaled))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c38c51d0",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 11: CONFUSION MATRIX\n",
|
|
"###################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "444fbb13",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(\"\\n=== CONFUSION MATRICES (WITH SCALING) ===\")\n",
|
|
"\n",
|
|
"print(\"\\nKNN Confusion Matrix:\")\n",
|
|
"print(confusion_matrix(y_test, knn_pred_scaled))\n",
|
|
"\n",
|
|
"print(\"\\nLogistic Regression Confusion Matrix:\")\n",
|
|
"print(confusion_matrix(y_test, logreg_pred_scaled))\n",
|
|
"\n",
|
|
"print(\"\\nDecision Tree Confusion Matrix:\")\n",
|
|
"print(confusion_matrix(y_test, tree_pred_scaled))\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c5d82a08",
|
|
"metadata": {
|
|
"cell_marker": "###############################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 12: MODWLS WITH CROSS-VALIDATION\n",
|
|
"##############################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "17493cc3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.model_selection import cross_val_score\n",
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
|
"\n",
|
|
"knn_pipeline = Pipeline([\n",
|
|
" ('scaler', StandardScaler()),\n",
|
|
" ('knn', KNeighborsClassifier(n_neighbors=5))\n",
|
|
"])\n",
|
|
"\n",
|
|
"logreg_pipeline = Pipeline([\n",
|
|
" ('scaler', StandardScaler()),\n",
|
|
" ('logreg', LogisticRegression(max_iter=5000))\n",
|
|
"])\n",
|
|
"\n",
|
|
"tree_model = DecisionTreeClassifier(random_state=42)\n",
|
|
"\n",
|
|
"knn_cv_scores = cross_val_score(knn_pipeline, X, y, cv=5, scoring='accuracy')\n",
|
|
"logreg_cv_scores = cross_val_score(logreg_pipeline, X, y, cv=5, scoring='accuracy')\n",
|
|
"tree_cv_scores = cross_val_score(tree_model, X, y, cv=5, scoring='accuracy')\n",
|
|
"\n",
|
|
"print(\"=== Cross-Validation Results ===\")\n",
|
|
"\n",
|
|
"print(\"\\nKNN\")\n",
|
|
"print(\"CV Scores:\", knn_cv_scores)\n",
|
|
"print(\"Mean Accuracy:\", np.mean(knn_cv_scores))\n",
|
|
"print(\"Standard Deviation:\", np.std(knn_cv_scores))\n",
|
|
"\n",
|
|
"print(\"\\nLogistic Regression\")\n",
|
|
"print(\"CV Scores:\", logreg_cv_scores)\n",
|
|
"print(\"Mean Accuracy:\", np.mean(logreg_cv_scores))\n",
|
|
"print(\"Standard Deviation:\", np.std(logreg_cv_scores))\n",
|
|
"\n",
|
|
"print(\"\\nDecision Tree\")\n",
|
|
"print(\"CV Scores:\", tree_cv_scores)\n",
|
|
"print(\"Mean Accuracy:\", np.mean(tree_cv_scores))\n",
|
|
"print(\"Standard Deviation:\", np.std(tree_cv_scores))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94d94dd2",
|
|
"metadata": {
|
|
"cell_marker": "####################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 13: CONFUSION MATRIX\n",
|
|
"###################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c706abc3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.model_selection import cross_val_predict\n",
|
|
"from sklearn.metrics import confusion_matrix, classification_report\n",
|
|
"\n",
|
|
"knn_pred = cross_val_predict(knn_pipeline, X, y, cv=5)\n",
|
|
"logreg_pred = cross_val_predict(logreg_pipeline, X, y, cv=5)\n",
|
|
"tree_pred = cross_val_predict(tree_model, X, y, cv=5)\n",
|
|
"\n",
|
|
"print(\"\\n=== Confusion Matrices ===\")\n",
|
|
"print(\"\\nKNN\")\n",
|
|
"print(confusion_matrix(y, knn_pred))\n",
|
|
"\n",
|
|
"print(\"\\nLogistic Regression\")\n",
|
|
"print(confusion_matrix(y, logreg_pred))\n",
|
|
"\n",
|
|
"print(\"\\nDecision Tree\")\n",
|
|
"print(confusion_matrix(y, tree_pred))\n",
|
|
"\n",
|
|
"print(\"\\n=== Classification Reports ===\")\n",
|
|
"print(\"\\nKNN\")\n",
|
|
"print(classification_report(y, knn_pred))\n",
|
|
"\n",
|
|
"print(\"\\nLogistic Regression\")\n",
|
|
"print(classification_report(y, logreg_pred))\n",
|
|
"\n",
|
|
"print(\"\\nDecision Tree\")\n",
|
|
"print(classification_report(y, tree_pred))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "34354ec9",
|
|
"metadata": {
|
|
"cell_marker": "###########################################",
|
|
"lines_to_next_cell": 0
|
|
},
|
|
"source": [
|
|
"BLOCK 14: LETS GET THE BEST VALUE FOR K\n",
|
|
"task: use the k value to recompute the previous KNN model and evaluate the performance\n",
|
|
"#########################################"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "93fa45e5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sklearn.model_selection import cross_val_score\n",
|
|
"import numpy as np\n",
|
|
"\n",
|
|
"k_values = range(1, 21)\n",
|
|
"scores = []\n",
|
|
"\n",
|
|
"for k in k_values:\n",
|
|
" knn = KNeighborsClassifier(n_neighbors=k)\n",
|
|
" cv_scores = cross_val_score(knn, X, y, cv=5)\n",
|
|
" scores.append(np.mean(cv_scores))\n",
|
|
"\n",
|
|
"best_k = k_values[np.argmax(scores)]\n",
|
|
"\n",
|
|
"print(\"Best k:\", best_k)\n",
|
|
"print(\"Best CV Accuracy:\", max(scores))\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"jupytext": {
|
|
"cell_metadata_filter": "-all",
|
|
"encoding": "# -*- coding: utf-8 -*-",
|
|
"main_language": "python",
|
|
"notebook_metadata_filter": "-all"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|