{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# \ud83d\udccb Day 5: Assignment \u2014 Model Card + Manager Recommendation (Churn \u00d7 Value)\n", "\n", "## Overview\n", "\n", "You will deliver a manager-ready artifact based on the Telco churn dataset:\n", "\n", "1. A **churn classifier** ($p(\\text{churn})$)\n", "2. A **MonthlyCharges regressor** ($\\widehat{\\text{MonthlyCharges}}$)\n", "3. A **Revenue-at-Risk** targeting list\n", "4. A short **manager memo** with a deployment/monitoring plan\n", "\n", "You must also include your **Independent Lab extension track** (A/B/C/D)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 0: Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install -q -U pandas numpy scikit-learn shap google-genai\n", "# Optional (Track C): !pip install -q -U flaml" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## \ud83d\udd27 GenAI Copilot Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import json\n", "import warnings\n", "from datetime import datetime, timezone\n", "\n", "import google.genai as genai\n", "\n", "# ── GenAI API Setup ────────────────────────────────────────────\n", "try:\n", " from google.colab import userdata\n", " os.environ[\"GEMINI_API_KEY\"] = userdata.get(\"GEMINI_API_KEY\")\n", "except Exception:\n", " pass\n", "\n", "API_KEY = os.environ.get(\"GEMINI_API_KEY\")\n", "if not API_KEY:\n", " import getpass\n", " API_KEY = getpass.getpass(\"Paste your GEMINI_API_KEY: \")\n", " os.environ[\"GEMINI_API_KEY\"] = API_KEY\n", "\n", "client = genai.Client(api_key=API_KEY)\n", "MODEL_ID = \"gemini-2.5-flash-lite\"\n", "\n", "# ── Logging Infrastructure ────────────────────────────────────\n", "PROMPT_LOG = []\n", "\n", "def _now():\n", " \"\"\"Return current UTC timestamp in ISO format.\"\"\"\n", " return datetime.now(timezone.utc).isoformat(timespec=\"seconds\").replace(\"+00:00\", \"Z\")\n", "\n", "def log_interaction(role, content, label=None):\n", " \"\"\"Log a GenAI interaction (prompt or response).\"\"\"\n", " entry = {\n", " \"ts\": _now(),\n", " \"role\": role,\n", " \"content\": content if isinstance(content, str) else json.dumps(content),\n", " \"label\": label or \"\",\n", " }\n", " PROMPT_LOG.append(entry)\n", " return entry\n", "\n", "print(\"✅ GenAI + logging ready.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🧾 Required: GenAI Copilot Log\n", "\n", "Include **at least 3** unique uses of GenAI while working on this assignment:\n", "\n", "- **Use #1:** …\n", " - Prompt summary: …\n", " - Change I made: …\n", " - What I verified manually: …\n", "\n", "- **Use #2:** …\n", " - Prompt summary: …\n", " - Change I made: …\n", " - What I verified manually: …\n", "\n", "- **Use #3:** …\n", " - Prompt summary: …\n", " - Change I made: …\n", " - What I verified manually: …" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ✅ Verification Checklist (required)\n", "\n", "- [ ] Metrics computed on a **holdout test set**\n", "- [ ] Preprocessing inside the **pipeline** (no train/test contamination)\n", "- [ ] Basic **leakage scan** completed and documented\n", "- [ ] SHAP plots generated from the final model (global + local)\n", "- [ ] Manager memo references **actual metrics/plots** from this notebook\n", "- [ ] Required artifacts saved (CSV, JSON, memo)\n", "- [ ] GenAI copilot log includes **at least 3 substantive uses**" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n", "from sklearn.impute import SimpleImputer\n", "\n", "from sklearn.linear_model import LogisticRegression, Ridge\n", "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n", "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n", "\n", "from sklearn.metrics import (\n", " roc_auc_score, average_precision_score,\n", " precision_score, recall_score, accuracy_score,\n", " mean_absolute_error, mean_squared_error, r2_score\n", ")\n", "\n", "import shap\n", "shap.initjs()\n", "\n", "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n", "\n", "RANDOM_STATE = 42\n", "\n", "print(\"✅ All imports ready.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 1: Data, leakage checks, and split (Deliverable 1)\n", "\n", "✅ **Deliverable:** document what you removed/changed and why." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "url = \"https://raw.githubusercontent.com/blastchar/telco-customer-churn/master/WA_Fn-UseC_-Telco-Customer-Churn.csv\"\n", "df = pd.read_csv(url)\n", "df[\"TotalCharges\"] = pd.to_numeric(df[\"TotalCharges\"].astype(str).str.strip().replace(\"\", np.nan), errors=\"coerce\")\n", "customer_ids = df[\"customerID\"].copy()\n", "\n", "y_clf = (df[\"Churn\"] == \"Yes\").astype(int)\n", "y_reg = df[\"MonthlyCharges\"].astype(float)\n", "X = df.drop(columns=[\"customerID\", \"Churn\"])\n", "\n", "print(f\"✅ Loaded: {df.shape[0]:,} rows × {df.shape[1]} columns\")\n", "print(f\" Churn rate: {y_clf.mean():.1%}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ── TODO: Leakage check notes ───────────────────────\n", "\n", "Write a few sentences:\n", "- What would count as leakage in a churn setting?\n", "- Which columns did you exclude (if any) and why?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# TODO: Complete leakage analysis\n", "# Example considerations:\n", "# - Do NOT use 'Churn' itself as a feature\n", "# - Avoid features only known AFTER churn occurs (e.g., end date)\n", "# - Be careful with variables that might change during preprocessing\n", "# - Check for data contamination between train/test (handled by pipeline)\n", "\n", "leakage_notes = \"\"\"\n", "TODO: Document leakage checks:\n", "1. Columns excluded: ...\n", "2. Reason: ...\n", "3. Pipeline ensures no train/test contamination: ...\n", "\"\"\"\n", "\n", "print(leakage_notes)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Train/test split ──────────────────────────────────\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y_clf,\n", " test_size=0.25,\n", " random_state=RANDOM_STATE,\n", " stratify=y_clf\n", ")\n", "yreg_train = y_reg.loc[X_train.index]\n", "yreg_test = y_reg.loc[X_test.index]\n", "\n", "print(f\"Train: {X_train.shape[0]:,} rows (churn rate: {y_train.mean():.1%})\")\n", "print(f\"Test: {X_test.shape[0]:,} rows (churn rate: {y_test.mean():.1%})\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 2: Model comparison — churn classification (Deliverable 2)\n", "\n", "✅ **Required:** logistic regression, random forest, gradient boosting." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Preprocessing Pipeline ────────────────────────────\n", "numeric_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n", "categorical_cols = X_train.select_dtypes(exclude=[\"number\"]).columns.tolist()\n", "\n", "# Handle sklearn API variations\n", "try:\n", " ohe = OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False)\n", "except TypeError:\n", " ohe = OneHotEncoder(handle_unknown=\"ignore\", sparse=False)\n", "\n", "preprocess = ColumnTransformer(\n", " transformers=[\n", " (\"num\", Pipeline([(\"impute\", SimpleImputer(strategy=\"median\")), (\"scale\", StandardScaler())]), numeric_cols),\n", " (\"cat\", Pipeline([(\"impute\", SimpleImputer(strategy=\"most_frequent\")), (\"onehot\", ohe)]), categorical_cols),\n", " ]\n", ")\n", "\n", "print(f\"✅ Preprocessing pipeline ready.\")\n", "print(f\" Numeric columns: {len(numeric_cols)}\")\n", "print(f\" Categorical columns: {len(categorical_cols)}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Helper functions ──────────────────────────────────\n", "\n", "def evaluate_classifier(name, model, X_te, y_te, threshold=0.5):\n", " \"\"\"Evaluate a classifier and return metrics + probabilities.\"\"\"\n", " proba = model.predict_proba(X_te)[:, 1]\n", " pred = (proba >= threshold).astype(int)\n", " out = {\n", " \"model\": name,\n", " \"roc_auc\": roc_auc_score(y_te, proba),\n", " \"pr_auc\": average_precision_score(y_te, proba),\n", " \"accuracy\": accuracy_score(y_te, pred),\n", " \"precision\": precision_score(y_te, pred, zero_division=0),\n", " \"recall\": recall_score(y_te, pred, zero_division=0),\n", " }\n", " return out, proba, pred\n", "\n", "\n", "def lift_by_decile(y_true, y_score, n_bins=10):\n", " \"\"\"Compute lift by decile (manager-friendly ranking metric).\"\"\"\n", " tmp = pd.DataFrame({\"y\": y_true, \"score\": y_score}).copy()\n", " tmp[\"decile\"] = pd.qcut(tmp[\"score\"].rank(method=\"first\"), q=n_bins, labels=False) + 1\n", " overall = tmp[\"y\"].mean()\n", " table = (\n", " tmp.groupby(\"decile\")\n", " .agg(n=(\"y\", \"size\"), churn_rate=(\"y\", \"mean\"), avg_score=(\"score\", \"mean\"))\n", " .sort_index(ascending=False)\n", " .reset_index()\n", " )\n", " table[\"lift\"] = table[\"churn_rate\"] / overall\n", " return table, overall\n", "\n", "\n", "def eval_regression(name, model, X_te, y_te):\n", " \"\"\"Evaluate a regression model.\"\"\"\n", " pred = model.predict(X_te)\n", " out = {\n", " \"model\": name,\n", " \"mae\": mean_absolute_error(y_te, pred),\n", " \"rmse\": mean_squared_error(y_te, pred, squared=False),\n", " \"r2\": r2_score(y_te, pred)\n", " }\n", " return out, pred\n", "\n", "print(\"✅ Helper functions defined.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Train three classification models ──────────────────\n", "logit = Pipeline([(\"prep\", preprocess), (\"clf\", LogisticRegression(max_iter=2000, class_weight=\"balanced\"))])\n", "rf = Pipeline([(\"prep\", preprocess), (\"clf\", RandomForestClassifier(n_estimators=600, n_jobs=-1, random_state=RANDOM_STATE, class_weight=\"balanced_subsample\"))])\n", "gb = Pipeline([(\"prep\", preprocess), (\"clf\", GradientBoostingClassifier(random_state=RANDOM_STATE))])\n", "\n", "for m in [logit, rf, gb]:\n", " m.fit(X_train, y_train)\n", "\n", "m1, p1, pred1 = evaluate_classifier(\"logit\", logit, X_test, y_test)\n", "m2, p2, pred2 = evaluate_classifier(\"rf\", rf, X_test, y_test)\n", "m3, p3, pred3 = evaluate_classifier(\"gb\", gb, X_test, y_test)\n", "\n", "clf_results = pd.DataFrame([m1, m2, m3]).sort_values([\"pr_auc\", \"roc_auc\"], ascending=False)\n", "\n", "print(\"\\n📊 Classification Model Results:\")\n", "print(clf_results.to_string(index=False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ── TODO: Choose your final churn model ──────────────\n", "\n", "Explain briefly:\n", "- which model you chose and why (metric + interpretability)\n", "- which threshold rule you will use and why" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Pick the best model by PR-AUC\n", "final_clf_name = clf_results.iloc[0][\"model\"]\n", "final_clf = {\"logit\": logit, \"rf\": rf, \"gb\": gb}[final_clf_name]\n", "p_final = {\"logit\": p1, \"rf\": p2, \"gb\": p3}[final_clf_name]\n", "\n", "print(f\"Selected model: {final_clf_name}\")\n", "print(f\"\\nModel justification:\")\n", "print(f\" PR-AUC: {clf_results.iloc[0]['pr_auc']:.3f} (best among three models)\")\n", "print(f\" ROC-AUC: {clf_results.iloc[0]['roc_auc']:.3f}\")\n", "print(f\"\\nThreshold rule (TODO): Describe your threshold decision...\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ── GenAI: Interpret Model Comparison ────────────────" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── GenAI: Generate model interpretation ────────────────\n", "metrics_summary = clf_results.to_string(index=False)\n", "\n", "interpret_prompt = f\"\"\"Here are churn classification model results on a holdout test set:\n", "\n", "{metrics_summary}\n", "\n", "Context: We are building a churn retention campaign where the cost of missing a churner (false negative)\n", "is roughly 5× the cost of contacting a non-churner (false positive).\n", "\n", "In 3–4 sentences, recommend which model to use and why. Explain the precision-recall tradeoff and\n", "how it impacts our retention targeting strategy.\"\"\"\n", "\n", "log_interaction(\"user\", interpret_prompt, label=\"model_interpretation\")\n", "\n", "response = client.models.generate_content(model=MODEL_ID, contents=interpret_prompt)\n", "interpretation = response.text\n", "log_interaction(\"assistant\", interpretation, label=\"model_interpretation\")\n", "\n", "print(\"🤖 GenAI Model Interpretation:\")\n", "print(\"=\" * 70)\n", "print(interpretation)\n", "print()\n", "print(\"TODO: Review the interpretation above and note any changes you made below.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 3: Model comparison — MonthlyCharges regression (Deliverable 2)\n", "\n", "✅ **Required:** ridge, random forest regressor, gradient boosting regressor." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Regression features (exclude MonthlyCharges itself) ───\n", "Xr_train = X_train.drop(columns=[\"MonthlyCharges\"])\n", "Xr_test = X_test.drop(columns=[\"MonthlyCharges\"])\n", "\n", "num_r = Xr_train.select_dtypes(include=[\"number\"]).columns.tolist()\n", "cat_r = Xr_train.select_dtypes(exclude=[\"number\"]).columns.tolist()\n", "\n", "preprocess_r = ColumnTransformer(\n", " transformers=[\n", " (\"num\", Pipeline([(\"impute\", SimpleImputer(strategy=\"median\")), (\"scale\", StandardScaler())]), num_r),\n", " (\"cat\", Pipeline([(\"impute\", SimpleImputer(strategy=\"most_frequent\")), (\"onehot\", ohe)]), cat_r),\n", " ]\n", ")\n", "\n", "ridge = Pipeline([(\"prep\", preprocess_r), (\"reg\", Ridge(alpha=1.0))])\n", "rfr = Pipeline([(\"prep\", preprocess_r), (\"reg\", RandomForestRegressor(n_estimators=600, n_jobs=-1, random_state=RANDOM_STATE))])\n", "gbr = Pipeline([(\"prep\", preprocess_r), (\"reg\", GradientBoostingRegressor(random_state=RANDOM_STATE))])\n", "\n", "for m in [ridge, rfr, gbr]:\n", " m.fit(Xr_train, yreg_train)\n", "\n", "r1, v1 = eval_regression(\"ridge\", ridge, Xr_test, yreg_test)\n", "r2m, v2 = eval_regression(\"rfr\", rfr, Xr_test, yreg_test)\n", "r3, v3 = eval_regression(\"gbr\", gbr, Xr_test, yreg_test)\n", "\n", "reg_results = pd.DataFrame([r1, r2m, r3]).sort_values([\"mae\", \"rmse\"], ascending=True)\n", "\n", "print(\"\\n📊 Regression Model Results:\")\n", "print(reg_results.to_string(index=False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 4: Explainability with SHAP (Deliverable 4)\n", "\n", "✅ **Required:** 1 global plot + 1 local explanation." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── SHAP Setup ────────────────────────────────────────────\n", "prep_fitted = final_clf.named_steps[\"prep\"]\n", "clf_fitted = final_clf.named_steps[\"clf\"]\n", "X_train_enc = prep_fitted.transform(X_train)\n", "feature_names = prep_fitted.get_feature_names_out()\n", "\n", "# Sample for SHAP speed\n", "sample_idx = np.random.RandomState(RANDOM_STATE).choice(X_train_enc.shape[0], size=min(800, X_train_enc.shape[0]), replace=False)\n", "X_shap = X_train_enc[sample_idx]\n", "\n", "# Create explainer with fallback\n", "explainer = None\n", "try:\n", " explainer = shap.TreeExplainer(clf_fitted)\n", " shap_values = explainer.shap_values(X_shap)\n", "except Exception as e:\n", " print(f\"TreeExplainer failed, falling back: {e}\")\n", " explainer = shap.Explainer(clf_fitted, X_shap)\n", " shap_values = explainer(X_shap)\n", "\n", "print(f\"✅ SHAP explainer ready. Shap values shape: {np.array(shap_values).shape}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── SHAP Global Summary Plot ─────────────────────────────\n", "plt.figure(figsize=(10, 6))\n", "try:\n", " if isinstance(shap_values, list):\n", " shap.summary_plot(shap_values[1], X_shap, feature_names=feature_names, max_display=15, show=False)\n", " else:\n", " shap.summary_plot(shap_values, X_shap, feature_names=feature_names, max_display=15, show=False)\n", " plt.title(f\"SHAP Summary Plot — {final_clf_name}\")\n", " plt.tight_layout()\n", " plt.show()\nexcept Exception as e:\n", " print(f\"Could not render summary plot: {e}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── SHAP Local Explanation (one customer) ────────────────\n", "row_i = 0\n", "x_one = X_shap[row_i:row_i+1]\n", "\n", "try:\n", " if hasattr(explainer, \"__call__\") and not isinstance(shap_values, list):\n", " exp = explainer(x_one)\n", " shap.plots.waterfall(exp[0])\n", " else:\n", " # TreeExplainer legacy API\n", " sv = shap_values[1][row_i] if isinstance(shap_values, list) else shap_values[row_i]\n", " base = explainer.expected_value[1] if isinstance(explainer.expected_value, (list, np.ndarray)) else explainer.expected_value\n", " shap.plots._waterfall.waterfall_legacy(base, sv, feature_names=feature_names)\n", "except Exception as e:\n", " print(f\"Local explanation failed: {e}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ── GenAI: SHAP Narrative for Individual Customer ────────" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── GenAI: Generate customer narrative from SHAP ──────────\n", "if isinstance(shap_values, list):\n", " sv_row = shap_values[1][row_i]\n", "else:\n", " sv_row = shap_values[row_i]\n", "\n", "shap_df = pd.DataFrame({\"feature\": feature_names, \"shap_value\": sv_row})\n", "shap_df[\"abs_shap\"] = shap_df[\"shap_value\"].abs()\n", "top_features = shap_df.nlargest(6, \"abs_shap\")\n", "\n", "shap_summary = \"\\n\".join(\n", " f\" - {row['feature']}: SHAP={row['shap_value']:+.3f} ({'increases' if row['shap_value'] > 0 else 'decreases'} churn risk)\"\n", " for _, row in top_features.iterrows()\n", ")\n", "\n", "narrative_prompt = f\"\"\"A customer has been flagged by our churn prediction model.\n", "The baseline churn rate is {y_test.mean():.0%}.\n", "\n", "Top factors driving this prediction (SHAP values):\n", "{shap_summary}\n", "\n", "Write a 3-sentence explanation for a retention manager who needs to decide\n", "whether to call this customer. Use plain business language, not technical jargon.\"\"\"\n", "\n", "log_interaction(\"user\", narrative_prompt, label=\"shap_narrative\")\n", "\n", "response = client.models.generate_content(model=MODEL_ID, contents=narrative_prompt)\n", "narrative = response.text\n", "log_interaction(\"assistant\", narrative, label=\"shap_narrative\")\n", "\n", "print(\"🤖 GenAI Narrative for Retention Manager:\")\n", "print(\"=\" * 70)\n", "print(narrative)\n", "print()\n", "print(\"TODO: Review the narrative and note any changes below.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 5: Revenue-at-Risk call list (Deliverable 5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Pick final regression model ───────────────────────────\n", "final_reg_name = reg_results.iloc[0][\"model\"]\n", "final_reg = {\"ridge\": ridge, \"rfr\": rfr, \"gbr\": gbr}[final_reg_name]\n", "v_final = final_reg.predict(Xr_test)\n", "v_final = np.clip(v_final, 0, None)\n", "\n", "# ── Compute Revenue-at-Risk ──────────────────────────────\n", "call_list = pd.DataFrame({\n", " \"customerID\": customer_ids.loc[X_test.index].values,\n", " \"p_churn\": p_final,\n", " \"pred_monthly_charges\": v_final,\n", "})\n", "call_list[\"revenue_at_risk\"] = call_list[\"p_churn\"] * call_list[\"pred_monthly_charges\"]\n", "\n", "call_list = call_list.sort_values(\"revenue_at_risk\", ascending=False)\n", "\n", "print(f\"\\n📊 Top 10 Customers by Revenue-at-Risk:\")\n", "print(call_list.head(10).to_string(index=False))\n", "print(f\"\\nTotal Revenue-at-Risk (test set): ${call_list['revenue_at_risk'].sum():,.0f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ── TODO: Add simple reason codes (top SHAP drivers) ────\n", "\n", "You can do this approximately:\n", "- take the top 2–3 absolute SHAP features for each selected customer\n", "- store them as strings (e.g., `\"Contract=Month-to-month; tenure(low); TechSupport=No\"`)\n", "\n", "Tip: For grading, it's fine to compute reason codes for the **top N** only." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# TODO: Compute SHAP-based reason codes\n", "# For the top N customers in the call list, extract their top 2-3 SHAP drivers\n", "# and create a simple text summary.\n", "\n", "# Placeholder code structure:\n", "# reason_codes = []\n", "# for idx in call_list.head(100).index:\n", "# # Get SHAP values for this customer\n", "# shap_row = shap_values[1][idx] if isinstance(shap_values, list) else shap_values[idx]\n", "# # Extract top 3 features\n", "# top_k = 3\n", "# top_idx = np.argsort(np.abs(shap_row))[-top_k:][::-1]\n", "# reasons = \"; \".join([feature_names[i] for i in top_idx])\n", "# reason_codes.append(reasons)\n", "#\n", "# call_list['reason_codes'] = reason_codes\n", "\n", "print(\"TODO: Add reason_codes column to call_list\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ── GenAI: Generate Action Recommendations ────────────" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── GenAI: Action recommendations for top-at-risk ────────\n", "top_10_stats = call_list.head(10)[[\"p_churn\", \"pred_monthly_charges\", \"revenue_at_risk\"]].describe().to_string()\n", "\n", "action_prompt = f\"\"\"Based on our revenue-at-risk analysis for the top 10 at-risk customers, recommend 3 concrete business actions.\n", "\n", "Top 10 customer statistics:\n", "{top_10_stats}\n", "\n", "Consider retention strategy, offer design, and outreach timing. Keep it practical.\"\"\"\n", "\n", "log_interaction(\"user\", action_prompt, label=\"action_recommendations\")\n", "\n", "response = client.models.generate_content(model=MODEL_ID, contents=action_prompt)\n", "actions = response.text\n", "log_interaction(\"assistant\", actions, label=\"action_recommendations\")\n", "\n", "print(\"🤖 GenAI Action Recommendations:\")\n", "print(\"=\" * 70)\n", "print(actions)\n", "print()\n", "print(\"TODO: Review recommendations and incorporate into your manager memo.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 6: Save Artifacts" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Save call list ───────────────────────────────────────\n", "call_list_out = \"day5_assignment_call_list.csv\"\n", "call_list.to_csv(call_list_out, index=False)\n", "print(f\"✅ Saved call list to {call_list_out}\")\n", "\n", "# ── Save metrics ─────────────────────────────────────────\n", "metrics = {\n", " \"final_churn_model\": final_clf_name,\n", " \"final_value_model\": final_reg_name,\n", " \"churn_metrics\": clf_results.to_dict(orient=\"records\"),\n", " \"value_metrics\": reg_results.to_dict(orient=\"records\"),\n", " \"extension_track\": \"TODO: A/B/C/D\",\n", " \"threshold_rule\": \"TODO: describe threshold/top-N rule\",\n", " \"verification\": {\n", " \"holdout_test\": True,\n", " \"pipeline_no_leakage\": True,\n", " \"leakage_scan_done\": False, # TODO: set to True when complete\n", " \"memo_references_real_outputs\": False, # TODO: set to True when memo is done\n", " },\n", "}\n", "\n", "metrics_out = \"day5_assignment_metrics.json\"\n", "with open(metrics_out, \"w\") as f:\n", " json.dump(metrics, f, indent=2)\n", "print(f\"✅ Saved metrics to {metrics_out}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 7: Manager Memo (Deliverable 3)\n", "\n", "✅ Create a 1-page memo as a separate markdown file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ── GenAI: Draft Manager Memo ────────────────────────" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── GenAI: Generate manager memo ──────────────────────────\n", "memo_context = f\"\"\"Please write a 1-page executive memo for the VP of Customer Success on our churn targeting initiative.\n", "\n", "Key facts:\n", "- Best churn model: {final_clf_name} (PR-AUC: {clf_results.iloc[0]['pr_auc']:.3f}, ROC-AUC: {clf_results.iloc[0]['roc_auc']:.3f})\n", "- Best value model: {final_reg_name} (MAE: ${reg_results.iloc[0]['mae']:.2f})\n", "- Total revenue at risk: ${call_list['revenue_at_risk'].sum():,.0f}\n", "- Test set size: {len(X_test)} customers\n", "- Baseline churn rate: {y_test.mean():.1%}\n", "\n", "The memo should include:\n", "1. Executive summary (2-3 sentences on business impact)\n", "2. Model performance (key metrics in plain language)\n", "3. Targeting strategy (how to prioritize customers)\n", "4. Top risk drivers (3-4 main churn factors from SHAP)\n", "5. Recommended actions (3-4 next steps)\n", "6. Implementation risks and monitoring plan\n", "\n", "Keep it professional, actionable, and free of technical jargon.\"\"\"\n", "\n", "log_interaction(\"user\", memo_context, label=\"manager_memo\")\n", "\n", "response = client.models.generate_content(model=MODEL_ID, contents=memo_context)\n", "memo_draft = response.text\n", "log_interaction(\"assistant\", memo_draft, label=\"manager_memo\")\n", "\n", "print(\"🤖 GenAI Draft Manager Memo:\")\n", "print(\"=\" * 70)\n", "print(memo_draft)\n", "print()\n", "print(\"TODO: Review, edit, and save the final memo below.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Save manager memo ────────────────────────────────────\n", "# TODO: Copy the final memo text from the GenAI draft above, make edits, paste below\n", "\n", "memo = \"\"\"# TODO: Day 5 Manager Memo — Churn × Value Targeting\n", "\n", "[Copy the final memo from the GenAI draft above and make any edits here.]\n", "\n", "## Executive summary\n", "- TODO\n", "\n", "## What decision are we supporting?\n", "- TODO\n", "\n", "## Model performance\n", "- Churn model: TODO (metric + interpretation)\n", "- Value model: TODO (MAE, business meaning)\n", "\n", "## Targeting plan\n", "- Capacity: TODO\n", "- Rule: TODO (top-N by revenue-at-risk or threshold)\n", "- Expected impact: TODO\n", "\n", "## Top drivers (from SHAP)\n", "- TODO (3–5 drivers + business meaning)\n", "\n", "## Risks & monitoring\n", "- Data drift: TODO\n", "- Segment risk: TODO\n", "- Calibration drift: TODO\n", "\n", "## Next steps\n", "- TODO (A/B test, rollout plan, etc.)\n", "\"\"\"\n", "\n", "memo_path = \"day5_assignment_manager_memo.md\"\n", "with open(memo_path, \"w\") as f:\n", " f.write(memo)\n", "print(f\"✅ Saved manager memo to {memo_path}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 8: GenAI Copilot Log Summary" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# ── Export Copilot Log ───────────────────────────────────\n", "print(f\"📋 Copilot interactions logged: {len(PROMPT_LOG)}\")\n", "print()\n", "\n", "for i, entry in enumerate(PROMPT_LOG, 1):\n", " print(f\"--- Log Entry {i} ---\")\n", " print(f\"Time: {entry['ts']}\")\n", " print(f\"Role: {entry['role']}\")\n", " print(f\"Label: {entry['label']}\")\n", " if len(entry['content']) > 200:\n", " print(f\"Content (first 200 chars): {entry['content'][:200]}...\")\n", " else:\n", " print(f\"Content: {entry['content']}\")\n", " print()\n", "\n", "# Save log as JSON\n", "log_path = \"day5_assignment_copilot_log.json\"\n", "with open(log_path, \"w\") as f:\n", " json.dump(PROMPT_LOG, f, indent=2)\n", "print(f\"✅ Saved copilot log to {log_path}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## ✅ Final Verification Checklist\n", "\n", "Before submission, verify all items are complete:\n", "\n", "- [ ] Metrics computed on a **holdout test set** (25% split)\n", "- [ ] Preprocessing inside a **pipeline** (no train/test leakage)\n", "- [ ] **Leakage scan** documented in Part 1\n", "- [ ] **Three churn models** trained (logit, RF, GB) with metrics\n", "- [ ] **Three regression models** trained (ridge, RF, GB) with MAE/RMSE/R2\n", "- [ ] **SHAP summary plot** (global feature importance)\n", "- [ ] **SHAP waterfall** for at least 1 customer (local explanation)\n", "- [ ] **Revenue-at-Risk** ranked call list created and saved as CSV\n", "- [ ] **Manager memo** saved as markdown file, with real metrics referenced\n", "- [ ] **Metrics JSON** saved with model names, scores, threshold rule, extension track\n", "- [ ] **Copilot log** includes at least 3 GenAI uses with clear labels (model interpretation, SHAP narrative, action recommendations, memo draft)\n", "- [ ] **Extension track** from Independent Lab documented in metrics JSON\n", "- [ ] All output files saved to working directory\n", "- [ ] Notebook runs end-to-end without errors\n", "\n", "### Files to submit:\n", "1. `Day5_Assignment5.ipynb` (this notebook)\n", "2. `day5_assignment_call_list.csv` (ranked customers)\n", "3. `day5_assignment_metrics.json` (model scores + metadata)\n", "4. `day5_assignment_manager_memo.md` (executive summary)\n", "5. `day5_assignment_copilot_log.json` (GenAI interaction log)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 4 }