{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# ๐Ÿง  Day 5: Independent Lab โ€” From Predictions to Decisions\n", "\n", "## Your mission\n", "\n", "You already know how to build churn + value models. Now you will make the solution **decision-ready**.\n", "\n", "### Minimum deliverables\n", "1. Improved churn report (ROC-AUC, PR-AUC, lift@10%)\n", "2. Improved value model report (MAE/RMSE)\n", "3. Final Revenue-at-Risk call list (CSV)\n", "4. A short manager recommendation (bullets)\n", "\n", "### Choose ONE extension track (A/B/C/D)\n", "- **A:** cost & capacity targeting (recommended)\n", "- **B:** calibration & probability quality\n", "- **C:** AutoML benchmark (FLAML)\n", "- **D:** segment stress test" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 0: Setup (5 min)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install -q -U pandas numpy scikit-learn shap google-genai\n", "# Track C AutoML (optional)\n", "# !pip install -q -U flaml" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import json\n", "import warnings\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "from datetime import datetime, timezone\n", "\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n", "from sklearn.impute import SimpleImputer\n", "\n", "from sklearn.linear_model import LogisticRegression, Ridge\n", "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n", "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n", "\n", "from sklearn.metrics import (\n", " roc_auc_score, average_precision_score,\n", " precision_score, recall_score, accuracy_score,\n", " mean_absolute_error, mean_squared_error, r2_score,\n", " brier_score_loss\n", ")\n", "\n", "import shap\n", "shap.initjs()\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "RANDOM_STATE = 42" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### GenAI Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ Google GenAI Setup โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "import google.generativeai as genai\n", "\n", "try:\n", " from google.colab import userdata\n", " API_KEY = userdata.get(\"GOOGLE_API_KEY\")\n", "except ImportError:\n", " import getpass\n", " API_KEY = getpass.getpass(\"Enter your Google API key: \")\n", "\n", "genai.configure(api_key=API_KEY)\n", "\n", "# Model configuration\n", "MODEL_ID = \"gemini-2.5-flash-lite\"\n", "client = genai.client\n", "\n", "print(f\"โœ… GenAI configured with model: {MODEL_ID}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ Prompt Logging Infrastructure โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "PROMPT_LOG = []\n", "\n", "def log_interaction(prompt, response_text, model=MODEL_ID):\n", " \"\"\"Log a GenAI interaction to PROMPT_LOG.\"\"\"\n", " entry = {\n", " \"timestamp\": datetime.now(timezone.utc).isoformat(timespec=\"seconds\"),\n", " \"model\": model,\n", " \"prompt\": prompt[:200] + \"...\" if len(prompt) > 200 else prompt,\n", " \"response_snippet\": response_text[:300] + \"...\" if len(response_text) > 300 else response_text,\n", " }\n", " PROMPT_LOG.append(entry)\n", " return entry\n", "\n", "def show_log(limit=None):\n", " \"\"\"Display the prompt log.\"\"\"\n", " log_df = pd.DataFrame(PROMPT_LOG)\n", " if limit:\n", " log_df = log_df.tail(limit)\n", " return log_df\n", "\n", "print(\"โœ… Logging infrastructure ready\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 1: Load data + build a strong baseline (15โ€“20 min)\n", "\n", "โœ… **Task:** Copy your working solution from the Guided Lab (or re-run it here).\n", "\n", "**You should end this part with:**\n", "- best churn model (manual)\n", "- best regression model (manual)\n", "- baseline Revenue-at-Risk call list" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TODO: Load Telco dataset and rebuild your baseline models\n", "\n", "Hints:\n", "- Use the same URL as in the guided lab\n", "- Clean `TotalCharges`\n", "- Create `y_clf` and `y_reg`\n", "- Build preprocessing pipelines\n", "- Fit logit/RF/GB for churn; ridge/RFR/GBR for value" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO: Load data โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "url = \"https://raw.githubusercontent.com/blastchar/telco-customer-churn/master/WA_Fn-UseC_-Telco-Customer-Churn.csv\"\n", "df = pd.read_csv(url)\n", "df[\"TotalCharges\"] = pd.to_numeric(df[\"TotalCharges\"].astype(str).str.strip().replace(\"\", np.nan), errors=\"coerce\")\n", "customer_ids = df[\"customerID\"].copy()\n", "\n", "y_clf = (df[\"Churn\"] == \"Yes\").astype(int)\n", "y_reg = df[\"MonthlyCharges\"].astype(float)\n", "X = df.drop(columns=[\"customerID\", \"Churn\"])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO: Train/test split (stratify for churn) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y_clf,\n", " test_size=0.25,\n", " random_state=RANDOM_STATE,\n", " stratify=y_clf\n", ")\n", "yreg_train = y_reg.loc[X_train.index]\n", "yreg_test = y_reg.loc[X_test.index]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TODO: Define preprocessing + fit 3 churn models\n", "\n", "You can start with the same defaults as the guided lab and then improve them." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO: Preprocessing โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "numeric_cols = X_train.select_dtypes(include=[\"number\"]).columns.tolist()\n", "categorical_cols = X_train.select_dtypes(exclude=[\"number\"]).columns.tolist()\n", "\n", "try:\n", " ohe = OneHotEncoder(handle_unknown=\"ignore\", sparse_output=False)\n", "except TypeError:\n", " ohe = OneHotEncoder(handle_unknown=\"ignore\", sparse=False)\n", "\n", "preprocess = ColumnTransformer(\n", " transformers=[\n", " (\"num\", Pipeline([( \"impute\", SimpleImputer(strategy=\"median\")), (\"scale\", StandardScaler()) ]), numeric_cols),\n", " (\"cat\", Pipeline([( \"impute\", SimpleImputer(strategy=\"most_frequent\")), (\"onehot\", ohe) ]), categorical_cols),\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def evaluate_classifier(model, X_te, y_te, threshold=0.5):\n", " \"\"\"Evaluate a classifier and return metrics + probabilities.\"\"\"\n", " proba = model.predict_proba(X_te)[:, 1]\n", " pred = (proba >= threshold).astype(int)\n", " return {\n", " \"roc_auc\": roc_auc_score(y_te, proba),\n", " \"pr_auc\": average_precision_score(y_te, proba),\n", " \"precision\": precision_score(y_te, pred, zero_division=0),\n", " \"recall\": recall_score(y_te, pred, zero_division=0),\n", " \"accuracy\": accuracy_score(y_te, pred),\n", " }, proba\n", "\n", "\n", "def eval_regression(model, X_te, y_te):\n", " \"\"\"Evaluate a regression model.\"\"\"\n", " pred = model.predict(X_te)\n", " return {\n", " \"mae\": mean_absolute_error(y_te, pred),\n", " \"rmse\": mean_squared_error(y_te, pred, squared=False),\n", " \"r2\": r2_score(y_te, pred)\n", " }, pred\n", "\n", "\n", "def lift_by_decile(y_true, y_score, n_bins=10):\n", " \"\"\"Compute lift by decile (manager-friendly ranking metric).\"\"\"\n", " tmp = pd.DataFrame({\"y\": y_true, \"score\": y_score}).copy()\n", " tmp[\"decile\"] = pd.qcut(tmp[\"score\"].rank(method=\"first\"), q=n_bins, labels=False) + 1\n", " overall = tmp[\"y\"].mean()\n", " table = (\n", " tmp.groupby(\"decile\")\n", " .agg(n=(\"y\", \"size\"), churn_rate=(\"y\", \"mean\"), avg_score=(\"score\", \"mean\"))\n", " .sort_index(ascending=False)\n", " .reset_index()\n", " )\n", " table[\"lift\"] = table[\"churn_rate\"] / overall\n", " return table\n", "\n", "print(\"โœ… Helper functions defined.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO: Fit churn models โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "logit = Pipeline([( \"prep\", preprocess), (\"clf\", LogisticRegression(max_iter=2000, class_weight=\"balanced\")) ])\n", "rf = Pipeline([( \"prep\", preprocess), (\"clf\", RandomForestClassifier(n_estimators=500, n_jobs=-1, random_state=RANDOM_STATE, class_weight=\"balanced_subsample\")) ])\n", "gb = Pipeline([( \"prep\", preprocess), (\"clf\", GradientBoostingClassifier(random_state=RANDOM_STATE)) ])\n", "\n", "for m in [logit, rf, gb]:\n", " m.fit(X_train, y_train)\n", "\n", "m_logit, p_logit = evaluate_classifier(logit, X_test, y_test)\n", "m_rf, p_rf = evaluate_classifier(rf, X_test, y_test)\n", "m_gb, p_gb = evaluate_classifier(gb, X_test, y_test)\n", "\n", "pd.DataFrame([\n", " {\"model\": \"logit\", **m_logit},\n", " {\"model\": \"rf\", **m_rf},\n", " {\"model\": \"gb\", **m_gb},\n", "]).sort_values([\"pr_auc\", \"roc_auc\"], ascending=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### GenAI: Ask for feature engineering suggestions" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (GenAI): Ask for feature engineering ideas โ”€โ”€โ”€โ”€\n", "# Uncomment and customize this section to ask GenAI for feature suggestions\n", "\n", "# feature_prompt = f\"\"\"\n", "# I have trained churn models on a telco dataset with these features:\n", "# {list(X_train.columns)}\n", "# \n", "# Current best model: Random Forest with PR-AUC={m_rf['pr_auc']:.3f}\n", "# \n", "# Suggest 3-5 NEW features I could engineer to improve churn prediction.\n", "# For each feature, provide:\n", "# 1. Feature name\n", "# 2. Business intuition (why it matters for churn)\n", "# 3. Pandas code to create it\n", "# \"\"\"\n", "\n", "# response = client.models.generate_content(model=MODEL_ID, contents=feature_prompt)\n", "# print(response.text)\n", "# log_interaction(feature_prompt, response.text)\n", "\n", "print(\"โณ TODO (optional): Send feature engineering prompt to GenAI\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 2: Choose ONE extension track (60โ€“80 min)\n", "\n", "Pick your track and complete the TODO sections below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ•”โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•—\n", "# โ•‘ SELECT YOUR TRACK โ€” change the letter below โ•‘\n", "# โ•šโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•\n", "SELECTED_TRACK = \"\" # Change to \"A\", \"B\", \"C\", or \"D\"\n", "\n", "assert SELECTED_TRACK in (\"A\", \"B\", \"C\", \"D\"), \"โš ๏ธ Set SELECTED_TRACK to 'A', 'B', 'C', or 'D'\"\n", "print(f\"โœ… Selected Track: {SELECTED_TRACK}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Track A โ€” Cost & Capacity Targeting (recommended)\n", "\n", "โœ… **Goal:** pick a threshold and/or top-N rule that reflects your business reality.\n", "\n", "**Inputs (choose your own numbers):**\n", "- offer cost $c$ (e.g., 10โ‚ฌ)\n", "- churn loss $L$ (e.g., 150โ‚ฌ)\n", "- capacity: top N calls/week (e.g., 500)\n", "\n", "**Deliverables:**\n", "- chosen rule (threshold or top-N)\n", "- expected value calculation (simple is fine)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (Track A): define costs and compute a cost-based threshold\n", "if SELECTED_TRACK == \"A\":\n", " c = 10 # offer cost\n", " L = 150 # churn loss\n", " threshold_cost = c / L\n", " print(f\"Cost-based threshold: {threshold_cost:.3f}\")\n", "else:\n", " print(\"โญ๏ธ Skipping Track A\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (Track A): compare outcomes at different thresholds\n", "if SELECTED_TRACK == \"A\":\n", " def expected_value(y_true, p, threshold, c, L):\n", " pred = (p >= threshold).astype(int)\n", " # contact cost for predicted positives\n", " cost = pred.sum() * c\n", " # churn loss for missed churners (false negatives)\n", " fn = ((y_true == 1) & (pred == 0)).sum()\n", " loss = fn * L\n", " return -(cost + loss), pred.sum(), fn\n", "\n", " p_use = p_rf # โ”€โ”€ TODO: set to your chosen model probabilities โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", " for t in [0.05, 0.10, 0.20, threshold_cost, 0.50]:\n", " val, n_contact, fn = expected_value(y_test.values, p_use, t, c, L)\n", " print(f\"t={t:.3f} | contacts={n_contact:4d} | FN={fn:4d} | -cost-loss={val:,.0f}\")\n", "else:\n", " print(\"โญ๏ธ Skipping Track A\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Track B โ€” Calibration & Probability Quality\n", "\n", "โœ… **Goal:** check whether predicted probabilities mean what they claim.\n", "\n", "**Deliverables:**\n", "- calibration curve\n", "- Brier score\n", "- calibrated model (sigmoid or isotonic)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (Track B): Calibration โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "if SELECTED_TRACK == \"B\":\n", " from sklearn.calibration import calibration_curve, CalibratedClassifierCV\n", "\n", " # โ”€โ”€ TODO: Choose a base model โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", " base_model = rf # Change to logit or gb if you prefer\n", "\n", " cal = CalibratedClassifierCV(base_model, method=\"sigmoid\", cv=3)\n", " cal.fit(X_train, y_train)\n", "\n", " p_cal = cal.predict_proba(X_test)[:, 1]\n", " brier = brier_score_loss(y_test, p_cal)\n", " print(f\"Brier score (calibrated): {brier:.4f}\")\n", "else:\n", " print(\"โญ๏ธ Skipping Track B\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (Track B): Plot calibration curve โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "if SELECTED_TRACK == \"B\":\n", " prob_true, prob_pred = calibration_curve(y_test, p_cal, n_bins=10)\n", " plt.figure(figsize=(5, 5))\n", " plt.plot(prob_pred, prob_true, marker=\"o\")\n", " plt.plot([0, 1], [0, 1], linestyle=\"--\")\n", " plt.xlabel(\"Predicted probability\")\n", " plt.ylabel(\"Observed churn rate\")\n", " plt.title(\"Calibration curve\")\n", " plt.show()\n", "else:\n", " print(\"โญ๏ธ Skipping Track B\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Track C โ€” AutoML Benchmark (FLAML)\n", "\n", "โœ… **Goal:** run AutoML as a benchmark and compare on the same holdout set.\n", "\n", "**Deliverables:**\n", "- churn: ROC-AUC + PR-AUC vs. your manual best model\n", "- regression: MAE/RMSE vs. your manual best model\n", "- recommendation: would you ship AutoML? why/why not?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (Track C): install and run AutoML โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "if SELECTED_TRACK == \"C\":\n", " # !pip install -q -U flaml\n", " # from flaml import AutoML\n", " #\n", " # automl = AutoML()\n", " # automl.fit(X_train, y_train, task=\"classification\", metric=\"auc\", time_budget=60)\n", " # p_automl = automl.predict_proba(X_test)[:, 1]\n", " # print(\"AutoML model:\", automl.model)\n", " # print(\"ROC-AUC:\", roc_auc_score(y_test, p_automl))\n", " # print(\"PR-AUC:\", average_precision_score(y_test, p_automl))\n", " print(\"โณ TODO (Track C): Uncomment and run FLAML AutoML\")\nelse:\n", " print(\"โญ๏ธ Skipping Track C\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Track D โ€” Segment Stress Test\n", "\n", "โœ… **Goal:** check whether performance is consistent across key segments.\n", "\n", "**Deliverables:**\n", "- at least 2 segments (e.g., Contract, tenure band)\n", "- metric table by segment (PR-AUC or lift@10%)\n", "- risk notes (where might the model fail?)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (Track D): segment evaluation helper โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "if SELECTED_TRACK == \"D\":\n", " def segment_report(df_features, y_true, p, segment_col):\n", " tmp = df_features[[segment_col]].copy()\n", " tmp[\"y\"] = y_true.values\n", " tmp[\"p\"] = p\n", " out = []\n", " for g, part in tmp.groupby(segment_col):\n", " if part[\"y\"].nunique() < 2:\n", " continue\n", " out.append({\n", " segment_col: g,\n", " \"n\": len(part),\n", " \"churn_rate\": part[\"y\"].mean(),\n", " \"roc_auc\": roc_auc_score(part[\"y\"], part[\"p\"]),\n", " \"pr_auc\": average_precision_score(part[\"y\"], part[\"p\"]),\n", " })\n", " return pd.DataFrame(out).sort_values(\"pr_auc\")\n", "\n", " print(segment_report(X_test, y_test, p_rf, \"Contract\"))\nelse:\n", " print(\"โญ๏ธ Skipping Track D\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 3: Final Revenue-at-Risk call list (15โ€“20 min)\n", "\n", "โœ… **Task:** rebuild your MonthlyCharges model and create a final call list.\n", "\n", "Deliverable: `day5_independent_call_list.csv`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO: Regression pipeline (drop MonthlyCharges from features) โ”€\n", "Xr_train = X_train.drop(columns=[\"MonthlyCharges\"])\n", "Xr_test = X_test.drop(columns=[\"MonthlyCharges\"])\n", "\n", "num_r = Xr_train.select_dtypes(include=[\"number\"]).columns.tolist()\n", "cat_r = Xr_train.select_dtypes(exclude=[\"number\"]).columns.tolist()\n", "\n", "preprocess_r = ColumnTransformer(\n", " transformers=[\n", " (\"num\", Pipeline([( \"impute\", SimpleImputer(strategy=\"median\")), (\"scale\", StandardScaler()) ]), num_r),\n", " (\"cat\", Pipeline([( \"impute\", SimpleImputer(strategy=\"most_frequent\")), (\"onehot\", ohe) ]), cat_r),\n", " ]\n", ")\n", "\n", "ridge = Pipeline([( \"prep\", preprocess_r), (\"reg\", Ridge(alpha=1.0)) ])\n", "gbr = Pipeline([( \"prep\", preprocess_r), (\"reg\", GradientBoostingRegressor(random_state=RANDOM_STATE)) ])\n", "ridge.fit(Xr_train, yreg_train)\n", "gbr.fit(Xr_train, yreg_train)\n", "\n", "pred_ridge = ridge.predict(Xr_test)\n", "pred_gbr = gbr.predict(Xr_test)\n", "\n", "print(\"Ridge:\", eval_regression(ridge, Xr_test, yreg_test)[0])\n", "print(\"GBR:\", eval_regression(gbr, Xr_test, yreg_test)[0])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO: pick churn probabilities and value predictions โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "p_final = p_rf # โ”€โ”€ TODO: set to your chosen churn model (or calibrated probs) โ”€โ”€โ”€\n", "v_final = pred_gbr # โ”€โ”€ TODO: set to your chosen regression model โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "v_final = np.clip(v_final, 0, None)\n", "\n", "call_list = pd.DataFrame({\n", " \"customerID\": customer_ids.loc[X_test.index].values,\n", " \"p_churn\": p_final,\n", " \"pred_monthly_charges\": v_final,\n", "})\n", "call_list[\"revenue_at_risk\"] = call_list[\"p_churn\"] * call_list[\"pred_monthly_charges\"]\n", "\n", "call_list = call_list.sort_values(\"revenue_at_risk\", ascending=False)\n", "call_list.head(10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ Save call list โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "out_path = \"day5_independent_call_list.csv\"\n", "call_list.to_csv(out_path, index=False)\n", "print(f\"โœ… Saved to {out_path}\")\n", "out_path" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### GenAI: Interpret model comparison results" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (GenAI): Ask GenAI to interpret model results โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "# Uncomment and customize this section to ask for business interpretation\n", "\n", "# model_comp = {\n", "# \"Logistic\": m_logit,\n", "# \"RandomForest\": m_rf,\n", "# \"GradientBoosting\": m_gb\n", "# }\n", "# \n", "# interp_prompt = f\"\"\"\n", "# I've trained 3 churn models with these results:\n", "# {json.dumps(model_comp, indent=2)}\n", "# \n", "# Business context: False negatives (missed churners) cost 5x more than false positives (unnecessary calls).\n", "# \n", "# In 3-4 sentences, recommend which model to deploy and explain your reasoning based on PR-AUC and business impact.\n", "# \"\"\"\n", "# \n", "# response = client.models.generate_content(model=MODEL_ID, contents=interp_prompt)\n", "# print(response.text)\n", "# log_interaction(interp_prompt, response.text)\n", "\n", "print(\"โณ TODO (optional): Send model comparison to GenAI for interpretation\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 4: Manager recommendation (10 min)\n", "\n", "โœ… **Task:** write a short recommendation.\n", "\n", "Include:\n", "- which customers to contact (rule + capacity)\n", "- what metric you optimized and why\n", "- top 3 drivers (from SHAP or feature importance)\n", "- biggest risks (leakage, drift, segment instability)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### TODO: Manager recommendation (write 5โ€“10 bullets)\n", "\n", "- **Decision & action:** โ€ฆ\n", "- **Targeting rule:** โ€ฆ\n", "- **Expected impact:** โ€ฆ\n", "- **Top drivers:** โ€ฆ\n", "- **Risks & monitoring:** โ€ฆ" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### SHAP Explanation (optional enhancement)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ TODO (GenAI): Ask for SHAP narrative explanations โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "# Use SHAP feature importance to explain why the model makes certain predictions\n", "\n", "# Extract transformed features for SHAP\n", "# X_test_transformed = preprocess.transform(X_test)\n", "# feature_names = preprocess.get_feature_names_out()\n", "#\n", "# # Pick a high-risk customer\n", "# high_risk_idx = p_rf.argmax()\n", "# shap_prompt = f\"\"\"\n", "# Customer {customer_ids.iloc[X_test.index[high_risk_idx]]} has a predicted churn probability of {p_rf[high_risk_idx]:.1%}.\n", "# \n", "# Top SHAP drivers (features that increase churn risk):\n", "# - Contract=Month-to-month (SHAP=+0.35)\n", "# - MonthlyCharges=120 (SHAP=+0.12)\n", "# - tenure=6 months (SHAP=+0.08)\n", "# \n", "# Write a short narrative (2-3 sentences) for a retention manager explaining why this customer is at risk.\n", "# \"\"\"\n", "# \n", "# response = client.models.generate_content(model=MODEL_ID, contents=shap_prompt)\n", "# print(response.text)\n", "# log_interaction(shap_prompt, response.text)\n", "\n", "print(\"โณ TODO (optional): Send SHAP insights to GenAI for narrative explanations\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Part 5: GenAI Copilot Log\n", "\n", "Record **at least 3** GenAI interactions you used today." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## ๐Ÿงพ Required: GenAI Copilot Log\n", "\n", "Record **at least 3** GenAI interactions you used today.\n", "\n", "- **Prompt #1:**\n", " - What I asked: `...`\n", " - What changed in my work: `...`\n", " - What I verified manually: `...`\n", "\n", "- **Prompt #2:**\n", " - What I asked: `...`\n", " - What changed in my work: `...`\n", " - What I verified manually: `...`\n", "\n", "- **Prompt #3:**\n", " - What I asked: `...`\n", " - What changed in my work: `...`\n", " - What I verified manually: `...`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# โ”€โ”€ Export Prompt Log โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€\n", "if PROMPT_LOG:\n", " log_df = pd.DataFrame(PROMPT_LOG)\n", " log_df.to_csv(\"day5_lab2_prompt_log.csv\", index=False)\n", " print(f\"โœ… Exported {len(log_df)} prompt log entries to day5_lab2_prompt_log.csv\")\n", " print(log_df)\nelse:\n", " print(\"โ„น๏ธ Prompt log is empty (no GenAI calls yet)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## โœ… Verification Checklist (complete before you finish)\n", "\n", "- [ ] Metrics computed on a **holdout test set**\n", "- [ ] Preprocessing inside a **pipeline** (no leakage)\n", "- [ ] One manager-friendly ranking view included (e.g., **lift@10%** or deciles)\n", "- [ ] Revenue-at-Risk call list saved as CSV\n", "- [ ] Your recommendation references **real outputs** (numbers/plots)\n", "- [ ] At least one extension track (A/B/C/D) completed\n", "- [ ] GenAI copilot log has **at least 3** entries\n", "- [ ] Prompt log exported to CSV" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.0" } }, "nbformat": 4, "nbformat_minor": 4 }