{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "wW7tGz4eJ8St" }, "source": [ "# πŸ“‹ Day 4: Assignment β€” Production-Ready AI Agent\n", "\n", "## Overview\n", "\n", "Build on your Independent Lab agent to create a **production-ready** system. You will:\n", "- Refine your tool definitions and system prompt\n", "- Process 12+ queries with full agent traces\n", "- Create a golden test set with 8+ annotated scenarios\n", "- Evaluate agent traces with LLM-as-judge\n", "- Write an error analysis identifying failure patterns\n", "- Document everything in an Agent Playbook\n", "\n", "## Grading Summary\n", "\n", "| # | Deliverable | Points |\n", "|---|------------|--------|\n", "| 1 | Tool Definitions (refined) | 15 |\n", "| 2 | Agent System Prompt (final) | 15 |\n", "| 3 | Agent Outputs (12+ queries) | β€” |\n", "| 4 | Golden Test Set (8+ scenarios) | 10 |\n", "| 5 | Agent Trace Evaluation | 15 |\n", "| 6 | Error Analysis | 15 |\n", "| 7 | Agent Playbook | 15 |\n", "| | **Total** | **100** |" ] }, { "cell_type": "markdown", "metadata": { "id": "qAbG712wJ8Sw" }, "source": [ "---\n", "## Setup" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "C7a-OmmlJ8Sx" }, "outputs": [], "source": [ "# Install and import the Google GenAI SDK that we will use for LLM calls.\n", "# The `-q` flag keeps the output quiet; `-U` ensures we have the latest version.\n", "!pip install -q -U google-genai" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fG9RbtGgJ8Sx", "outputId": "5ccfcd07-1a7c-4ecc-e27a-0de63e7e7267" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… API initialized. Model: gemini-2.5-flash-lite\n" ] } ], "source": [ "# After installation we import the necessary modules and set up the API key.\n", "# In a real project you would manage credentials securely (e.g. env vars, vault),\n", "# but for this assignment we prompt the user if the key isn't already set.\n", "import os, json, time\n", "from datetime import datetime, timezone\n", "from google import genai\n", "from google.genai import types\n", "\n", "# ── API Key ──────────────────────────────────────────────\n", "try:\n", " from google.colab import userdata\n", " os.environ[\"GEMINI_API_KEY\"] = userdata.get(\"GEMINI_API_KEY\")\n", "except Exception:\n", " # not running in Colab; ignore\n", " pass\n", "\n", "if not os.environ.get(\"GEMINI_API_KEY\"):\n", " import getpass\n", " os.environ[\"GEMINI_API_KEY\"] = getpass.getpass(\"Paste your GEMINI_API_KEY: \")\n", "\n", "# Create a reusable client object and record the model we will use.\n", "client = genai.Client(api_key=os.environ[\"GEMINI_API_KEY\"])\n", "MODEL_ID = \"gemini-2.5-flash-lite\"\n", "\n", "print(f\"βœ… API initialized. Model: {MODEL_ID}\")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "hge9Z-K0J8Sy", "outputId": "239b2195-0cf4-4a25-dd35-1ab56d7e2fef" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Agent infrastructure loaded.\n" ] } ], "source": [ "# ── Infrastructure (from Guided Lab) ─────────────────────\n", "# These helper functions are reused throughout the assignment. They provide\n", "# consistent logging, timing, and a manual agent loop that gives us full\n", "# visibility into the model's reasoning and any tool calls it makes.\n", "\n", "PROMPT_LOG = []\n", "\n", "\n", "def _now():\n", " \"\"\"Return the current time in ISO format (UTC with Z suffix).\n", "\n", " Used for timestamping each entry in PROMPT_LOG so we can track the order\n", " of messages and tool invocations during experiments.\n", " \"\"\"\n", " return datetime.now(timezone.utc).isoformat(timespec=\"seconds\").replace(\"+00:00\", \"Z\")\n", "\n", "\n", "def log_interaction(role, content, label=None):\n", " \"\"\"Record a single interaction in the global PROMPT_LOG.\n", "\n", " Args:\n", " role: \"user\", \"agent\", or \"tool\" to indicate who produced the entry.\n", " content: Text or serializable object representing the message/tool output.\n", " label: Optional string for tagging the entry (e.g. \"agent_input\").\n", " \"\"\"\n", " entry = {\"ts\": _now(), \"role\": role,\n", " \"content\": content if isinstance(content, str) else json.dumps(content),\n", " \"label\": label or \"\"}\n", " PROMPT_LOG.append(entry)\n", " return entry\n", "\n", "\n", "def show_log(n=10):\n", " \"\"\"Display the last `n` log entries as a pandas DataFrame.\n", "\n", " Handy for debugging the agent during development.\n", " \"\"\"\n", " import pandas as pd\n", " if not PROMPT_LOG:\n", " print(\"No interactions logged yet.\")\n", " return\n", " df = pd.DataFrame(PROMPT_LOG[-n:])\n", " from IPython.display import display\n", " display(df)\n", "\n", "\n", "def run_agent(user_message, tools, system_prompt=None, max_steps=10):\n", " \"\"\"A manual agent loop with full visibility.\n", "\n", " This function mimics the behavior of the autonomous agent from the\n", " lab but gives us control over tool execution. Each time the model emits a\n", " function_call, we intercept it, run the corresponding local Python function,\n", " and feed the result back into the conversation history. The loop continues\n", " until the model returns plain text (no tools) or the step limit is reached.\n", "\n", " Args:\n", " user_message: The user's request.\n", " tools: List of Python functions to use as tools (see their docstrings).\n", " system_prompt: Optional system instruction for the agent.\n", " max_steps: Maximum number of reasoning steps (safety limit).\n", "\n", " Returns:\n", " A tuple of (final_text, tools_called, trace) where tools_called is a\n", " list of tool names that were invoked during the run, and trace is a\n", " list of dicts with structured log entries (call_number, tool, args, result).\n", " \"\"\"\n", " tool_map = {fn.__name__: fn for fn in tools}\n", " call_count = 0 # Track total tool calls across all steps\n", " tools_called = [] # Record which tools were actually used\n", " trace = [] # Structured log: tool, args, result per call\n", "\n", " # Build initial contents with optional system prompt prepended to user\n", " contents = []\n", " if system_prompt:\n", " contents.append(types.Content(\n", " role=\"user\",\n", " parts=[types.Part(text=f\"System: {system_prompt}\\n\\nUser: {user_message}\")]\n", " ))\n", " else:\n", " contents.append(types.Content(\n", " role=\"user\",\n", " parts=[types.Part(text=user_message)]\n", " ))\n", "\n", " log_interaction(\"user\", user_message, label=\"agent_input\")\n", "\n", " for step in range(max_steps):\n", " response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=contents,\n", " config=types.GenerateContentConfig(\n", " tools=tools,\n", " # Tool calling mode defaults to AUTO β€” the model\n", " # reasons about whether to use tools on each turn.\n", " automatic_function_calling=types.AutomaticFunctionCallingConfig(\n", " disable=True # Model still reasons about tools β€”\n", " # but the SDK won't execute them automatically.\n", " # Instead it returns the function_call to us,\n", " # and WE run the function below.\n", " ),\n", " ),\n", " )\n", "\n", " # Guard: the model may return an empty response\n", " parts = response.parts or []\n", " if not parts:\n", " print(f\" Step {step+1}: ⚠️ Empty response from model β€” retrying...\")\n", " continue\n", "\n", " # Add model response to history for future turns\n", " contents.append(types.Content(role=\"model\", parts=parts))\n", "\n", " # Check each part for a function_call object\n", " function_results = []\n", " for part in parts:\n", " if part.function_call:\n", " call_count += 1\n", " name = part.function_call.name\n", " args = dict(part.function_call.args)\n", " tools_called.append(name)\n", " print(f\" Tool call {call_count}: πŸ”§ {name}({args})\")\n", "\n", " # Execute the requested tool and capture its output or exception\n", " try:\n", " result = tool_map[name](**args)\n", " except Exception as e:\n", " result = {\"error\": str(e)}\n", "\n", " print(f\" β†’ {result}\")\n", " trace.append({\n", " \"call\": call_count,\n", " \"tool\": name,\n", " \"args\": args,\n", " \"result\": result if isinstance(result, str) else json.dumps(result),\n", " })\n", " log_interaction(\"tool\", f\"{name}({args}) β†’ {result}\", label=\"tool_call\")\n", "\n", " # feed the tool result back into the conversation so the model\n", " # can continue reasoning with the new information\n", " function_results.append(\n", " types.Part(\n", " function_response=types.FunctionResponse(\n", " name=name,\n", " response={\"result\": result},\n", " )\n", " )\n", " )\n", "\n", " if function_results:\n", " contents.append(types.Content(role=\"user\", parts=function_results))\n", " else:\n", " # No function calls on this step β†’ model finished generating\n", " final_text = response.text or \"(no text response)\"\n", " log_interaction(\"agent\", final_text, label=\"agent_output\")\n", " return final_text, tools_called, trace\n", "\n", " return \"⚠️ Agent reached maximum steps without completing.\", tools_called, trace\n", "\n", "\n", "# Convenience wrapper to run a single query with logging and return result.\n", "def run_and_log(query: str, tools, system_prompt=None):\n", " \"\"\"Call :func:`run_agent` and print a nicely formatted summary.\n", "\n", " This helper is used in the query loop later to reduce boilerplate.\n", "\n", " Args:\n", " query: Text of the user request.\n", " tools: List of tool functions.\n", " system_prompt: Optional system instruction.\n", "\n", " Returns:\n", " (answer, tools_used, trace) from ``run_agent``.\n", " \"\"\"\n", " print(f\"\\n>>> Running query: {query}\")\n", " answer, tools_used, trace = run_agent(query, tools=tools, system_prompt=system_prompt)\n", " print(f\"Answer: {answer}\\nTools: {tools_used}\")\n", " return answer, tools_used, trace\n", "\n", "print(\"βœ… Agent infrastructure loaded.\")" ] }, { "cell_type": "markdown", "metadata": { "id": "uvZLlZTeJ8Sz" }, "source": [ "---\n", "## Part 1: Final Tool Definitions (15 pts)\n", "\n", "Refine your tools from the Independent Lab. Each tool should have:\n", "- Clear function name (verb + noun)\n", "- Complete type hints\n", "- Detailed docstring with Args/Returns\n", "- Graceful error handling\n", "- At least one example in the docstring" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "up7JvHMsJ8Sz", "outputId": "c4495012-d3da-481a-c2a5-2c1b8bdd23e6" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Track A data loaded.\n" ] } ], "source": [ "# ── Track Data ──────────────────────────────────────────────\n", "# The notebook supports multiple \"tracks\" of content. Switch SELECTED_TRACK\n", "# depending on which domain you want your agent to operate in.\n", "SELECTED_TRACK = \"A\" # Change to your track (A=Support, B=Research, C=Finance)\n", "\n", "# ---------- Track A: Customer Support ----------\n", "# A minimal in-memory ticket store used by the support tools below.\n", "SUPPORT_TICKETS = {\n", " \"T001\": {\n", " \"id\": \"T001\",\n", " \"status\": \"open\",\n", " \"category\": \"billing\",\n", " \"description\": \"Invoice discrepancy\",\n", " \"created_date\": \"2024-02-01\",\n", " \"priority\": \"high\",\n", " },\n", " \"T002\": {\n", " \"id\": \"T002\",\n", " \"status\": \"closed\",\n", " \"category\": \"technical\",\n", " \"description\": \"API authentication error\",\n", " \"created_date\": \"2024-01-28\",\n", " \"priority\": \"critical\",\n", " },\n", "}\n", "\n", "# Policies that the support agent can look up. These are fixed strings so\n", "# the model has something authoritative to quote when asked about rules.\n", "SUPPORT_POLICIES = {\n", " \"escalation\": \"Critical issues escalate to Level 2 after 4 hours.\",\n", " \"sla\": \"Response within 2 hours for high priority, 8 hours for standard.\",\n", " \"refunds\": \"Full refunds within 30 days, partial refunds up to 90 days.\",\n", "}\n", "\n", "# ---------- Track B: Research Documents ----------\n", "# (not used in Track A but included to show how the notebook scales)\n", "RESEARCH_DOCS = {\n", " \"R001\": {\n", " \"title\": \"AI in Healthcare\",\n", " \"abstract\": \"Survey of machine learning applications in clinical settings.\",\n", " \"citations\": 42,\n", " },\n", " \"R002\": {\n", " \"title\": \"Transformer Efficiency\",\n", " \"abstract\": \"Methods for reducing transformer model size.\",\n", " \"citations\": 128,\n", " },\n", "}\n", "RESEARCH_METRICS = {\n", " \"avg_citations\": 85,\n", " \"topics\": [\"AI\", \"ML\", \"NLP\", \"Vision\"],\n", "}\n", "\n", "# ---------- Track C: Financial Data ----------\n", "FINANCIAL_DATA = {\n", " \"Q1_2024\": {\"revenue\": 5000000, \"expenses\": 3200000, \"margin\": 0.36},\n", " \"Q2_2024\": {\"revenue\": 5500000, \"expenses\": 3400000, \"margin\": 0.38},\n", "}\n", "FINANCIAL_POLICIES = {\n", " \"budget_cap\": 4000000,\n", " \"approval_threshold\": 100000,\n", " \"fiscal_year_start\": \"2024-01-01\",\n", "}\n", "\n", "# Informational print to confirm which track data is active.\n", "print(f\"βœ… Track {SELECTED_TRACK} data loaded.\")" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "jB5ndQEFJ8S0", "outputId": "36094ebe-3d7f-4aad-943d-78a0a8327ffb" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Tools for Track A: ['get_ticket', 'classify_ticket_text', 'search_support_policy', 'draft_customer_reply', 'update_ticket_status']\n" ] } ], "source": [ "# ── Final Tool Definitions ───────────────────────────────────\n", "# Track A (Customer Support) β€” Production-ready tools\n", "#\n", "# Design principles applied (see Day 4 slides + assignment PDF):\n", "# - Single responsibility per tool (each does one thing well)\n", "# - Clear docstrings + type hints (become the tool schema)\n", "# - Graceful error handling (return {\"status\":\"error\",...} not exceptions)\n", "# - Structured outputs (predictable keys: status / data / error)\n", "# - Minimal scope (no secret access, no arbitrary code execution)\n", "\n", "from typing import Optional, Literal, Dict, Any, List\n", "\n", "# --- type aliases for stricter schema ---\n", "TicketStatus = Literal[\"open\", \"pending\", \"closed\"]\n", "TicketPriority = Literal[\"low\", \"medium\", \"high\", \"critical\"]\n", "TicketCategory = Literal[\"billing\", \"technical\", \"account\", \"api\", \"praise\", \"feature_request\", \"other\"]\n", "\n", "\n", "# ---------- TOOL IMPLEMENTATIONS ----------\n", "\n", "def get_ticket(ticket_id: str) -> dict:\n", " \"\"\"Fetch a support ticket by ID from the internal ticket store.\n", "\n", " Use this tool when the user references a specific ticket ID (e.g., \"T001\").\n", " It returns the ticket metadata you need to reason about priority, category,\n", " and SLA. The caller (the agent) should check the status field and\n", " potentially reference the description in its reply.\n", "\n", " Args:\n", " ticket_id: Ticket identifier such as \"T001\".\n", "\n", " Returns:\n", " A structured dict:\n", " - {\"status\":\"ok\",\"data\":{...ticket fields...}}\n", " - {\"status\":\"error\",\"error\":\"...\",\"available_ids\":[...]} if not found\n", "\n", " Example:\n", " >>> get_ticket(\"T001\")\n", " {\"status\": \"ok\", \"data\": {\"id\": \"T001\", \"status\": \"open\", ...}}\n", " \"\"\"\n", " ticket_id = (ticket_id or \"\").strip().upper()\n", " if not ticket_id:\n", " # user forgot to supply an ID\n", " return {\"status\": \"error\", \"error\": \"ticket_id is required.\", \"available_ids\": list(SUPPORT_TICKETS.keys())}\n", "\n", " ticket = SUPPORT_TICKETS.get(ticket_id)\n", " if not ticket:\n", " # id not in our tiny dataset\n", " return {\"status\": \"error\", \"error\": f\"Ticket '{ticket_id}' not found.\", \"available_ids\": list(SUPPORT_TICKETS.keys())}\n", "\n", " return {\"status\": \"ok\", \"data\": ticket}\n", "\n", "\n", "\n", "def classify_ticket_text(ticket_text: str) -> dict:\n", " \"\"\"Classify a support message into a category and urgency.\n", "\n", " This is the first tool to call when the agent receives free-form text and\n", " there is no ticket ID. It applies simple keyword rules and returns both\n", " a coarse category and an estimated urgency level along with signals that\n", " explain why the classification was chosen (useful for debugging).\n", "\n", " Args:\n", " ticket_text: The customer's message (free text).\n", "\n", " Returns:\n", " A structured dict:\n", " - {\"status\":\"ok\",\"data\":{\"category\":,\"urgency\":,\"signals\":[...]}}\n", " - {\"status\":\"error\",\"error\":\"...\"} if ticket_text is empty\n", "\n", " Example:\n", " >>> classify_ticket_text(\"I was charged twice, this is urgent\")\n", " {\n", " \"status\": \"ok\",\n", " \"data\": {\"category\": \"billing\", \"urgency\": \"high\", \"signals\": [\"billing_keyword\", \"urgency_keyword\"]}\n", " }\n", " \"\"\"\n", " text = (ticket_text or \"\").strip()\n", " if not text:\n", " return {\"status\": \"error\", \"error\": \"ticket_text is required.\"}\n", "\n", " t = text.lower()\n", " signals: List[str] = []\n", "\n", " # Category rules (simple but deterministic). Order matters: more specific\n", " # patterns first to avoid misclassification.\n", " if any(w in t for w in [\"charged\", \"billing\", \"invoice\", \"refund\", \"payment\", \"price\", \"double charged\"]):\n", " category: TicketCategory = \"billing\"; signals.append(\"billing_keyword\")\n", " elif any(w in t for w in [\"crash\", \"error\", \"bug\", \"broken\", \"fails\", \"fail\", \"exception\"]):\n", " category = \"technical\"; signals.append(\"bug_keyword\")\n", " elif any(w in t for w in [\"upgrade\", \"downgrade\", \"plan\", \"account\", \"password\", \"login\"]):\n", " category = \"account\"; signals.append(\"account_keyword\")\n", " elif any(w in t for w in [\"api\", \"token\", \"auth\", \"authentication\", \"rate limit\", \"quota\"]):\n", " category = \"api\"; signals.append(\"api_keyword\")\n", " elif any(w in t for w in [\"love\", \"great\", \"awesome\", \"thanks\", \"thank you\"]):\n", " category = \"praise\"; signals.append(\"praise_keyword\")\n", " elif any(w in t for w in [\"feature\", \"request\", \"could you add\", \"wish\", \"please add\"]):\n", " category = \"feature_request\"; signals.append(\"feature_keyword\")\n", " else:\n", " category = \"other\"; signals.append(\"fallback_other\")\n", "\n", " # Urgency rules use additional keyword spotting. Defaults to \"medium\".\n", " urgency: TicketPriority = \"medium\"\n", " if any(w in t for w in [\"urgent\", \"asap\", \"immediately\"]):\n", " urgency = \"high\"; signals.append(\"urgency_keyword\")\n", " if any(w in t for w in [\"critical\", \"down\", \"outage\"]):\n", " urgency = \"critical\"; signals.append(\"critical_keyword\")\n", " if any(w in t for w in [\"charged twice\", \"double charged\"]):\n", " # billing issues where double charge occurred are treated as high-risk\n", " urgency = \"high\"; signals.append(\"billing_high_risk\")\n", "\n", " return {\"status\": \"ok\", \"data\": {\"category\": category, \"urgency\": urgency, \"signals\": signals}}\n", "\n", "\n", "\n", "def search_support_policy(policy_name: str) -> dict:\n", " \"\"\"Retrieve an internal support policy snippet.\n", "\n", " Called when the agent needs an authoritative rule (SLA, escalation,\n", " refunds) before answering. The caller should never guessβ€”we only store\n", " the three valid policies listed below.\n", "\n", " Args:\n", " policy_name: One of: \"sla\", \"escalation\", \"refunds\"\n", "\n", " Returns:\n", " A structured dict:\n", " - {\"status\":\"ok\",\"data\":{\"policy_name\":..., \"text\":...}}\n", " - {\"status\":\"error\",\"error\":\"...\",\"available_policies\":[...]} if not found\n", " \"\"\"\n", " key = (policy_name or \"\").strip().lower()\n", " if not key:\n", " return {\"status\": \"error\", \"error\": \"policy_name is required.\", \"available_policies\": list(SUPPORT_POLICIES.keys())}\n", "\n", " text = SUPPORT_POLICIES.get(key)\n", " if not text:\n", " return {\"status\": \"error\", \"error\": f\"Policy '{key}' not found.\", \"available_policies\": list(SUPPORT_POLICIES.keys())}\n", "\n", " return {\"status\": \"ok\", \"data\": {\"policy_name\": key, \"text\": text}}\n", "\n", "\n", "\n", "def draft_customer_reply(\n", " customer_message: str,\n", " classification_category: str,\n", " classification_urgency: str,\n", " policy_text: Optional[str] = None,\n", " ticket_id: Optional[str] = None,\n", ") -> dict:\n", " \"\"\"Draft a professional customer support reply.\n", "\n", " This tool ONLY drafts text (no email is sent, no ticket is updated).\n", " Use it after you have:\n", " 1) classified the issue, and\n", " 2) consulted the relevant policy (if a policy applies)\n", "\n", " Args:\n", " customer_message: The original customer message.\n", " classification_category: Category label, e.g. \"billing\".\n", " classification_urgency: Urgency label, e.g. \"high\".\n", " policy_text: Optional policy snippet to reference in the reply.\n", " ticket_id: Optional ticket ID to reference (e.g. \"T001\").\n", "\n", " Returns:\n", " {\"status\":\"ok\",\"data\":{\"reply\":\"...\",\"tone\":\"professional\",\"references\":[...]}}\n", " or {\"status\":\"error\",\"error\":\"...\"} for invalid inputs.\n", " \"\"\"\n", " msg = (customer_message or \"\").strip()\n", " if not msg:\n", " return {\"status\": \"error\", \"error\": \"customer_message is required.\"}\n", "\n", " category = (classification_category or \"\").strip().lower()\n", " urgency = (classification_urgency or \"\").strip().lower()\n", " if not category or not urgency:\n", " return {\"status\": \"error\", \"error\": \"classification_category and classification_urgency are required.\"}\n", "\n", " refs: List[str] = []\n", " header = f\"Hi there{f' (Ticket {ticket_id})' if ticket_id else ''},\\n\\n\"\n", " ack = \"Thanks for reaching out β€” I can see how frustrating this is.\\n\\n\" if urgency in [\"high\", \"critical\"] else \"Thanks for reaching out.\\n\\n\"\n", "\n", " policy_block = \"\"\n", " if policy_text:\n", " # Keep it short so we don't overload the customer with internal text.\n", " policy_block = f\"Relevant policy: {policy_text}\\n\\n\"\n", " refs.append(\"policy\")\n", "\n", " next_steps = \"\"\n", " if category == \"billing\":\n", " next_steps = (\n", " \"Next steps: please share your invoice number (or the last 4 digits of the card) \"\n", " \"and the date/time of the charge so we can investigate and resolve it quickly.\\n\"\n", " )\n", " elif category in [\"technical\", \"api\"]:\n", " next_steps = (\n", " \"Next steps: please share the steps to reproduce, screenshots (if possible), \"\n", " \"and any error message. If it's an API issue, include the endpoint and a request ID.\\n\"\n", " )\n", " elif category == \"account\":\n", " next_steps = (\n", " \"Next steps: please confirm the email on your account and whether you recently \"\n", " \"changed your password or plan.\\n\"\n", " )\n", " elif category == \"feature_request\":\n", " next_steps = (\n", " \"Next steps: could you describe your use case and why this feature matters? \"\n", " \"That helps us prioritize.\\n\"\n", " )\n", " elif category == \"praise\":\n", " next_steps = \"No action needed β€” just wanted to say we appreciate you!\\n\"\n", " else:\n", " next_steps = \"Next steps: could you share a bit more detail so we can route this correctly?\\n\"\n", "\n", " close = \"\\nIf you have any additional details, reply here and we’ll continue from there.\\n\\nBest regards,\\nSupport Team\"\n", " reply = header + ack + (policy_block) + next_steps + close\n", "\n", " return {\"status\": \"ok\", \"data\": {\"reply\": reply, \"tone\": \"professional\", \"references\": refs}}\n", "\n", "\n", "\n", "def update_ticket_status(ticket_id: str, new_status: TicketStatus, user_confirmed: bool = False) -> dict:\n", " \"\"\"Update a ticket status (SIDE EFFECT β€” requires confirmation).\n", "\n", " This is a write operation. It must NEVER run unless the user explicitly\n", " confirms (user_confirmed=True). If not confirmed, it returns a draft\n", " confirmation message instead of changing anything.\n", "\n", " Args:\n", " ticket_id: Ticket ID, e.g. \"T001\"\n", " new_status: One of \"open\" | \"pending\" | \"closed\"\n", " user_confirmed: Must be True to apply the change.\n", "\n", " Returns:\n", " - If not confirmed:\n", " {\"status\":\"needs_confirmation\",\"data\":{\"message\":\"...\",\"ticket_id\":...,\"new_status\":...}}\n", " - If confirmed and success:\n", " {\"status\":\"ok\",\"data\":{\"ticket_id\":...,\"old_status\":...,\"new_status\":...}}\n", " - If error:\n", " {\"status\":\"error\",\"error\":\"...\"}\n", " \"\"\"\n", " ticket_id = (ticket_id or \"\").strip().upper()\n", " if not ticket_id:\n", " return {\"status\": \"error\", \"error\": \"ticket_id is required.\"}\n", "\n", " if ticket_id not in SUPPORT_TICKETS:\n", " return {\"status\": \"error\", \"error\": f\"Ticket '{ticket_id}' not found.\", \"available_ids\": list(SUPPORT_TICKETS.keys())}\n", "\n", " if new_status not in [\"open\", \"pending\", \"closed\"]:\n", " return {\"status\": \"error\", \"error\": f\"Invalid new_status '{new_status}'. Allowed: open|pending|closed\"}\n", "\n", " if not user_confirmed:\n", " # return a message for the agent to show the user so they can confirm\n", " return {\n", " \"status\": \"needs_confirmation\",\n", " \"data\": {\n", " \"message\": f\"Confirm change: update ticket {ticket_id} to status '{new_status}'? \"\n", " f\"If yes, re-run with user_confirmed=true.\",\n", " \"ticket_id\": ticket_id,\n", " \"new_status\": new_status,\n", " },\n", " }\n", "\n", " old = SUPPORT_TICKETS[ticket_id].get(\"status\")\n", " SUPPORT_TICKETS[ticket_id][\"status\"] = new_status\n", " return {\"status\": \"ok\", \"data\": {\"ticket_id\": ticket_id, \"old_status\": old, \"new_status\": new_status}}\n", "\n", "\n", "# ── Select Tools Based on Track ───────────────────────────────\n", "# In this assignment notebook we implement Track A end-to-end.\n", "# (You can adapt the same pattern for Tracks B/C/D if you want.)\n", "TRACK_TOOLS = {\n", " \"A\": [get_ticket, classify_ticket_text, search_support_policy, draft_customer_reply, update_ticket_status],\n", "}\n", "\n", "my_tools = TRACK_TOOLS[SELECTED_TRACK]\n", "print(f\"βœ… Tools for Track {SELECTED_TRACK}: {[t.__name__ for t in my_tools]}\")" ] }, { "cell_type": "markdown", "source": [ "### Confirmation Gate Pattern\n", "\n", "If any of your tools has **side effects** (sending emails, creating tickets, modifying data),\n", "wrap it in a confirmation gate. The tool should **refuse to execute** unless the caller\n", "explicitly confirms. This is a critical safety pattern for production agents.\n", "\n", "```python\n", "# Example: a side-effect tool with confirmation gate\n", "def send_response(draft: str, user_confirmed: bool = False) -> dict:\n", " \"\"\"Send a drafted response to the customer. Requires confirmation.\"\"\"\n", " if not user_confirmed:\n", " return {\"ok\": False, \"error\": \"User confirmation required before sending.\"}\n", " return {\"ok\": True, \"status\": \"sent\", \"draft\": draft}\n", "```\n", "\n", "**Implemented in this notebook:** `update_ticket_status` is a write/side-effect tool. It refuses to execute unless the user explicitly confirms, and the function itself enforces this by requiring `user_confirmed=true`." ], "metadata": { "id": "aa5wqXnlJ8S1" } }, { "cell_type": "markdown", "metadata": { "id": "iVOZXPDRJ8S1" }, "source": [ "---\n", "## Part 2: Final System Prompt (15 pts)\n", "\n", "Your system prompt should include ALL of these components:\n", "1. **Role** assignment\n", "2. **Available tools** with descriptions\n", "3. **Rules** for tool use\n", "4. **Reasoning instructions** (explain your thinking)\n", "5. **Refusal behavior** (out-of-scope requests)\n", "6. **Output format** constraints" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wDVSZPhpJ8S2", "outputId": "4dfbfde5-e559-4bc8-d104-e019d2be211b" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Final system prompt set.\n", "\n" ] } ], "source": [ "# ── Final System Prompt ──────────────────────────────────────\n", "# The agent's \"constitution\". It's long because it documents exactly how\n", "# the LLM should behave, which tools are available, and what counts as a\n", "# valid response. Keeping this prompt clear and well‑commented is one of the\n", "# most important parts of building a reliable system.\n", "\n", "# Break the prompt into logical sections so we can comment on each piece.\n", "prompt_sections = []\n", "\n", "# Role description: who is the model and what is its mission.\n", "prompt_sections.append(\"ROLE\\nYou are a Customer Support AI Agent for a SaaS company. \"\n", " \"Your job is to triage tickets, look up internal policies, \"\n", " \"and draft clear, professional replies.\")\n", "\n", "# Tool list with short usages; agents should NEVER call functions outside this\n", "# list.\n", "prompt_sections.append(\"\\nAVAILABLE TOOLS (call ONLY these)\")\n", "prompt_sections.append(\"- get_ticket(ticket_id): fetch ticket metadata by ID (status, category, priority, description).\")\n", "prompt_sections.append(\"- classify_ticket_text(ticket_text): classify free-text into category + urgency with signals.\")\n", "prompt_sections.append(\"- search_support_policy(policy_name): retrieve the official policy text (\\\"sla\\\", \\\"escalation\\\", \\\"refunds\\\").\")\n", "prompt_sections.append(\"- draft_customer_reply(customer_message, classification_category, classification_urgency, policy_text?, ticket_id?): draft a customer reply (no sending).\")\n", "prompt_sections.append(\"- update_ticket_status(ticket_id, new_status, user_confirmed): SIDE EFFECT; requires explicit confirmation.\")\n", "\n", "# Rules for when and how to invoke the tools. The agent should follow these\n", "# procedural guidelines before returning a final answer.\n", "prompt_sections.append(\"\\nRULES FOR TOOL USE\")\n", "prompt_sections.append(\"1) If the user references a ticket ID (like T001), call get_ticket first.\")\n", "prompt_sections.append(\"2) If you have free-form text and no ticket ID, call classify_ticket_text first.\")\n", "prompt_sections.append(\"3) If the question involves rules/commitments (refunds, SLA, escalation), call search_support_policy and quote/paraphrase it.\")\n", "prompt_sections.append(\"4) Never invent policies, ticket fields, or actions. If the tool returns an error or missing info, say so and ask for the missing detail.\")\n", "prompt_sections.append(\"5) Do NOT execute side effects automatically:\")\n", "prompt_sections.append(\" - Only call update_ticket_status with user_confirmed=true if the user explicitly says \\\"yes/confirmed/do it\\\".\")\n", "prompt_sections.append(\" - If user_confirmed is false/missing, ask for confirmation in your final answer.\")\n", "\n", "# Reasoning instructions help the model understand how to think.\n", "prompt_sections.append(\"\\nREASONING INSTRUCTIONS\")\n", "prompt_sections.append(\"- Think step-by-step internally: decide what you need, call tools, then synthesize.\")\n", "prompt_sections.append(\"- If a tool fails, explain briefly what happened and try an alternative tool or ask a clarifying question.\")\n", "prompt_sections.append(\"- Keep the tool outputs minimal in your final answer (do not dump raw JSON unless requested).\")\n", "\n", "# Refusal rules for out-of-scope requests.\n", "prompt_sections.append(\"\\nREFUSAL / OUT-OF-SCOPE\")\n", "prompt_sections.append(\"- If the user asks for secrets, private customer data beyond the provided ticket store, or anything unrelated to customer support policies/tickets, refuse politely and offer what you can do instead.\")\n", "\n", "# Output format instructions define the structure of the agent's final message.\n", "prompt_sections.append(\"\\nOUTPUT FORMAT (final user-facing answer)\")\n", "prompt_sections.append(\"- Start with a one-line summary.\")\n", "prompt_sections.append(\"- Then: (a) what you found, (b) what you recommend next, (c) if confirmation is required, ask for it.\")\n", "prompt_sections.append(\"- Keep it professional and concise.\")\n", "\n", "# Include a tiny example to show chaining behavior, helpful for debugging.\n", "prompt_sections.append(\"\\nEXAMPLE (tool chaining)\")\n", "prompt_sections.append(\"User: \\\"I was charged twice. Urgent!\\\"\")\n", "prompt_sections.append(\"Steps: classify_ticket_text -> search_support_policy(\\\"refunds\\\") -> draft_customer_reply(...)\")\n", "\n", "# Join into single string. We keep blank lines between sections for readability.\n", "FINAL_SYSTEM_PROMPT = \"\\n\".join(prompt_sections)\n", "print(\"βœ… Final system prompt set.\\n\")\n", "# Optionally display a preview of the first few lines when debugging\n", "#print(FINAL_SYSTEM_PROMPT[:500])" ] }, { "cell_type": "markdown", "metadata": { "id": "sx3nUqUfJ8S2" }, "source": [ "---\n", "## Part 3: Agent Outputs (12+ queries)\n", "\n", "Run your agent on at least 12 diverse queries. Include:\n", "- 4-5 straightforward queries (easy)\n", "- 4-5 multi-step queries (medium)\n", "- 2-3 edge cases or out-of-scope queries (hard)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "OFHYhalHJ8S2", "outputId": "dd090b6e-d3e8-412a-8c7c-421f461ecc75" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Loaded 12 queries.\n" ] } ], "source": [ "# ── Final Queries ────────────────────────────────────────────\n", "# Define a variety of test queries we will feed to the agent. Each query is\n", "# intentionally chosen to exercise a different path through the tools: simple\n", "# lookups, multi-step reasoning, side-effect confirmation, and out‑of‑scope\n", "# refusals. Later we iterate over this list and call ``run_and_log`` for each.\n", "\n", "final_queries = [\n", " # Easy (single tool / direct policy lookup)\n", " \"What is our refunds policy?\",\n", " \"What is our SLA for high priority tickets?\",\n", " \"How does escalation work for critical issues?\",\n", "\n", " # Easy (ticket lookup)\n", " \"Show me the current status and priority of ticket T001.\",\n", " \"What is ticket T002 about and is it still open?\",\n", "\n", " # Medium (multi-step: classify + policy + draft)\n", " \"Customer says: 'I was charged twice for my subscription this month. Urgent.' Draft a response.\",\n", " \"Customer says: 'The API authentication keeps failing with an error. Can you help?' Draft a response and mention SLA if relevant.\",\n", "\n", " # Medium (ticket + policy + draft)\n", " \"Ticket T001: Draft a customer reply that references the refunds policy and asks for the right details.\",\n", "\n", " # Hard/Edge (ambiguous)\n", " \"A customer says: 'Your app is broken' β€” what do you need from them before you can help? Draft a short reply.\",\n", "\n", " # Hard/Edge (side effect needs confirmation)\n", " \"Please close ticket T001 as resolved.\",\n", " \"Update ticket T002 to pending.\",\n", "\n", " # Out-of-scope (should refuse)\n", " \"What is the CEO's salary? Answer with confidence.\",\n", "]\n", "print(f\"βœ… Loaded {len(final_queries)} queries.\")" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "E7ClEs-uJ8S3", "outputId": "0d2535c9-d647-4cbd-adda-b06c86072e37" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "Query 1/12: What is our refunds policy?\n", " Tool call 1: πŸ”§ search_support_policy({'policy_name': 'refunds'})\n", " β†’ {'status': 'ok', 'data': {'policy_name': 'refunds', 'text': 'Full refunds within 30 days, partial refunds up to 90 days.'}}\n", " Step 2: ⚠️ Empty response from model β€” retrying...\n", "Answer: The refunds policy states that full refunds are available within 30 days of purchase, and partial refunds are available up to 90 days.\n", "πŸ”§ Tools used: ['search_support_policy']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] search_support_policy({'policy_name': 'refunds'}) β†’ {\"status\": \"ok\", \"data\": {\"policy_name\": \"refunds\", \"text\": \"Full refunds within 30 days, partial refunds up to 90 days.\"}}\n", "\n", "Query 2/12: What is our SLA for high priority tickets?\n", " Tool call 1: πŸ”§ search_support_policy({'policy_name': 'sla'})\n", " β†’ {'status': 'ok', 'data': {'policy_name': 'sla', 'text': 'Response within 2 hours for high priority, 8 hours for standard.'}}\n", "Answer: The Service Level Agreement (SLA) states that high-priority tickets should receive a response within 2 hours, and standard-priority tickets within 8 hours.\n", "πŸ”§ Tools used: ['search_support_policy']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] search_support_policy({'policy_name': 'sla'}) β†’ {\"status\": \"ok\", \"data\": {\"policy_name\": \"sla\", \"text\": \"Response within 2 hours for high priority, 8 hours for standard.\"}}\n", "\n", "Query 3/12: How does escalation work for critical issues?\n", "Answer: tool_code\n", "print(default_api.search_support_policy(policy_name='escalation'))\n", "\n", "πŸ”§ Tools used: []\n", "\n", "Query 4/12: Show me the current status and priority of ticket T001.\n", " Tool call 1: πŸ”§ get_ticket({'ticket_id': 'T001'})\n", " β†’ {'status': 'ok', 'data': {'id': 'T001', 'status': 'open', 'category': 'billing', 'description': 'Invoice discrepancy', 'created_date': '2024-02-01', 'priority': 'high'}}\n", "Answer: The ticket T001 is about an invoice discrepancy and is currently open with high priority.\n", "Would you like to know the policy for this type of ticket?\n", "\n", "πŸ”§ Tools used: ['get_ticket']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] get_ticket({'ticket_id': 'T001'}) β†’ {\"status\": \"ok\", \"data\": {\"id\": \"T001\", \"status\": \"open\", \"category\": \"billing\", \"description\": \"Invoice discrepancy\", \"created_date\": \"2024-02-01\", \"priority\": \"high\"}}\n", "\n", "Query 5/12: What is ticket T002 about and is it still open?\n", " Tool call 1: πŸ”§ get_ticket({'ticket_id': 'T002'})\n", " β†’ {'status': 'ok', 'data': {'id': 'T002', 'status': 'closed', 'category': 'technical', 'description': 'API authentication error', 'created_date': '2024-01-28', 'priority': 'critical'}}\n", "Answer: The ticket T002 is about an API authentication error, and it is currently closed.\n", "πŸ”§ Tools used: ['get_ticket']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] get_ticket({'ticket_id': 'T002'}) β†’ {\"status\": \"ok\", \"data\": {\"id\": \"T002\", \"status\": \"closed\", \"category\": \"technical\", \"description\": \"API authentication error\", \"created_date\": \"2024-01-28\", \"priority\": \"critical\"}}\n", "\n", "Query 6/12: Customer says: 'I was charged twice for my subscription this month. Urgent.' Draft a response.\n", " Tool call 1: πŸ”§ classify_ticket_text({'ticket_text': 'I was charged twice for my subscription this month. Urgent.'})\n", " β†’ {'status': 'ok', 'data': {'category': 'billing', 'urgency': 'high', 'signals': ['billing_keyword', 'urgency_keyword', 'billing_high_risk']}}\n", " Tool call 2: πŸ”§ search_support_policy({'policy_name': 'refunds'})\n", " β†’ {'status': 'ok', 'data': {'policy_name': 'refunds', 'text': 'Full refunds within 30 days, partial refunds up to 90 days.'}}\n", " Tool call 3: πŸ”§ draft_customer_reply({'classification_category': 'billing', 'policy_text': 'Full refunds within 30 days, partial refunds up to 90 days.', 'classification_urgency': 'high', 'customer_message': 'I was charged twice for my subscription this month. Urgent.'})\n", " β†’ {'status': 'ok', 'data': {'reply': 'Hi there,\\n\\nThanks for reaching out β€” I can see how frustrating this is.\\n\\nRelevant policy: Full refunds within 30 days, partial refunds up to 90 days.\\n\\nNext steps: please share your invoice number (or the last 4 digits of the card) and the date/time of the charge so we can investigate and resolve it quickly.\\n\\nIf you have any additional details, reply here and we’ll continue from there.\\n\\nBest regards,\\nSupport Team', 'tone': 'professional', 'references': ['policy']}}\n", "Answer: I can help with this billing issue. I've drafted a reply that references our refund policy and asks for more information to resolve the duplicate charge. Would you like me to show you the draft?\n", "πŸ”§ Tools used: ['classify_ticket_text', 'search_support_policy', 'draft_customer_reply']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] classify_ticket_text({'ticket_text': 'I was charged twice for my subscription this month. Urgent.'}) β†’ {\"status\": \"ok\", \"data\": {\"category\": \"billing\", \"urgency\": \"high\", \"signals\": [\"billing_keyword\", \"urgency_keyword\", \"billing_high_risk\"]}}\n", " [2] search_support_policy({'policy_name': 'refunds'}) β†’ {\"status\": \"ok\", \"data\": {\"policy_name\": \"refunds\", \"text\": \"Full refunds within 30 days, partial refunds up to 90 days.\"}}\n", " [3] draft_customer_reply({'classification_category': 'billing', 'policy_text': 'Full refunds within 30 days, partial refunds up to 90 days.', 'classification_urgency': 'high', 'customer_message': 'I was charged twice for my subscription this month. Urgent.'}) β†’ {\"status\": \"ok\", \"data\": {\"reply\": \"Hi there,\\n\\nThanks for reaching out \\u2014 I can see how frustrating this is.\\n\\nRelevant policy: Full refunds within 30 days, partial refunds up to 90 days.\\n\\nNe\n", "\n", "Query 7/12: Customer says: 'The API authentication keeps failing with an error. Can you help?' Draft a response and mention SLA if relevant.\n", " Tool call 1: πŸ”§ classify_ticket_text({'ticket_text': 'The API authentication keeps failing with an error. Can you help?'})\n", " β†’ {'status': 'ok', 'data': {'category': 'technical', 'urgency': 'medium', 'signals': ['bug_keyword']}}\n", " Tool call 2: πŸ”§ search_support_policy({'policy_name': 'sla'})\n", " β†’ {'status': 'ok', 'data': {'policy_name': 'sla', 'text': 'Response within 2 hours for high priority, 8 hours for standard.'}}\n", " Tool call 3: πŸ”§ draft_customer_reply({'classification_urgency': 'medium', 'customer_message': 'The API authentication keeps failing with an error. Can you help?', 'classification_category': 'technical', 'policy_text': 'Response within 2 hours for high priority, 8 hours for standard.'})\n", " β†’ {'status': 'ok', 'data': {'reply': \"Hi there,\\n\\nThanks for reaching out.\\n\\nRelevant policy: Response within 2 hours for high priority, 8 hours for standard.\\n\\nNext steps: please share the steps to reproduce, screenshots (if possible), and any error message. If it's an API issue, include the endpoint and a request ID.\\n\\nIf you have any additional details, reply here and we’ll continue from there.\\n\\nBest regards,\\nSupport Team\", 'tone': 'professional', 'references': ['policy']}}\n", "Answer: Here is a draft response to the customer:\n", "\n", "Hi there,\n", "\n", "Thanks for reaching out.\n", "\n", "Relevant policy: Response within 2 hours for high priority, 8 hours for standard.\n", "\n", "Next steps: please share the steps to reproduce, screenshots (if possible), and any error message. If it's an API issue, include the endp\n", "πŸ”§ Tools used: ['classify_ticket_text', 'search_support_policy', 'draft_customer_reply']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] classify_ticket_text({'ticket_text': 'The API authentication keeps failing with an error. Can you help?'}) β†’ {\"status\": \"ok\", \"data\": {\"category\": \"technical\", \"urgency\": \"medium\", \"signals\": [\"bug_keyword\"]}}\n", " [2] search_support_policy({'policy_name': 'sla'}) β†’ {\"status\": \"ok\", \"data\": {\"policy_name\": \"sla\", \"text\": \"Response within 2 hours for high priority, 8 hours for standard.\"}}\n", " [3] draft_customer_reply({'classification_urgency': 'medium', 'customer_message': 'The API authentication keeps failing with an error. Can you help?', 'classification_category': 'technical', 'policy_text': 'Response within 2 hours for high priority, 8 hours for standard.'}) β†’ {\"status\": \"ok\", \"data\": {\"reply\": \"Hi there,\\n\\nThanks for reaching out.\\n\\nRelevant policy: Response within 2 hours for high priority, 8 hours for standard.\\n\\nNext steps: please share the steps to \n", "\n", "Query 8/12: Ticket T001: Draft a customer reply that references the refunds policy and asks for the right details.\n", "Answer: tool_code\n", "print(default_api.get_ticket(ticket_id='T001'))\n", "print(default_api.search_support_policy(policy_name='refunds'))\n", "\n", "\n", "πŸ”§ Tools used: []\n", "\n", "Query 9/12: A customer says: 'Your app is broken' β€” what do you need from them before you can help? Draft a short reply.\n", " Tool call 1: πŸ”§ classify_ticket_text({'ticket_text': 'Your app is broken'})\n", " β†’ {'status': 'ok', 'data': {'category': 'technical', 'urgency': 'medium', 'signals': ['bug_keyword']}}\n", "Answer: I've classified the customer's message as a \"technical\" issue with \"medium\" urgency. Since they haven't provided a ticket ID, I'll draft a reply asking for more information to help me troubleshoot.\n", "\n", "Here's a draft reply:\n", "\n", "\"We're sorry to hear you're experiencing issues with our app. To help us resol\n", "πŸ”§ Tools used: ['classify_ticket_text']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] classify_ticket_text({'ticket_text': 'Your app is broken'}) β†’ {\"status\": \"ok\", \"data\": {\"category\": \"technical\", \"urgency\": \"medium\", \"signals\": [\"bug_keyword\"]}}\n", "\n", "Query 10/12: Please close ticket T001 as resolved.\n", " Tool call 1: πŸ”§ update_ticket_status({'new_status': 'closed', 'ticket_id': 'T001', 'user_confirmed': False})\n", " β†’ {'status': 'needs_confirmation', 'data': {'message': \"Confirm change: update ticket T001 to status 'closed'? If yes, re-run with user_confirmed=true.\", 'ticket_id': 'T001', 'new_status': 'closed'}}\n", "Answer: I need your confirmation to close ticket T001. Please let me know if you'd like me to proceed.\n", "πŸ”§ Tools used: ['update_ticket_status']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] update_ticket_status({'new_status': 'closed', 'ticket_id': 'T001', 'user_confirmed': False}) β†’ {\"status\": \"needs_confirmation\", \"data\": {\"message\": \"Confirm change: update ticket T001 to status 'closed'? If yes, re-run with user_confirmed=true.\", \"ticket_id\": \"T001\", \"new_status\": \"closed\"}}\n", "\n", "Query 11/12: Update ticket T002 to pending.\n", " Tool call 1: πŸ”§ update_ticket_status({'new_status': 'pending', 'user_confirmed': False, 'ticket_id': 'T002'})\n", " β†’ {'status': 'needs_confirmation', 'data': {'message': \"Confirm change: update ticket T002 to status 'pending'? If yes, re-run with user_confirmed=true.\", 'ticket_id': 'T002', 'new_status': 'pending'}}\n", "Answer: I can do that. To confirm, I need you to re-run the request with `user_confirmed=true`.\n", "πŸ”§ Tools used: ['update_ticket_status']\n", "\n", "πŸ“‹ Agent Trace:\n", " [1] update_ticket_status({'new_status': 'pending', 'user_confirmed': False, 'ticket_id': 'T002'}) β†’ {\"status\": \"needs_confirmation\", \"data\": {\"message\": \"Confirm change: update ticket T002 to status 'pending'? If yes, re-run with user_confirmed=true.\", \"ticket_id\": \"T002\", \"new_status\": \"pending\"}}\n", "\n", "Query 12/12: What is the CEO's salary? Answer with confidence.\n", "Answer: I cannot answer that question. I am a customer support AI agent and can only help with customer support inquiries. I can help you with classifying tickets, looking up support policies, drafting replies, and updating ticket statuses.\n", "πŸ”§ Tools used: []\n", "\n", "βœ… Processed 12 queries.\n" ] } ], "source": [ "# ── Run Agent ────────────────────────────────────────────────\n", "agent_outputs = []\n", "for i, query in enumerate(final_queries, 1):\n", " print(f\"\\nQuery {i}/{len(final_queries)}: {query}\")\n", " answer, tools_used, trace = run_agent(query, tools=my_tools, system_prompt=FINAL_SYSTEM_PROMPT)\n", " agent_outputs.append({\n", " \"query_id\": f\"Q{i:02d}\",\n", " \"query\": query,\n", " \"answer\": answer,\n", " \"tools_used\": tools_used,\n", " \"trace\": trace,\n", " \"timestamp\": _now(),\n", " })\n", " print(f\"Answer: {answer[:300]}\")\n", " print(f\"πŸ”§ Tools used: {tools_used}\")\n", " if trace:\n", " print(f\"\\nπŸ“‹ Agent Trace:\")\n", " for t in trace:\n", " print(f\" [{t['call']}] {t['tool']}({t['args']}) β†’ {str(t['result'])[:200]}\")\n", "\n", "print(f\"\\nβœ… Processed {len(agent_outputs)} queries.\")" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4x3oRLx4J8S3", "outputId": "36d413af-9d28-4f6c-c3e7-32ee66a43f84" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Exported 12 outputs to day4_assignment_agent_outputs.json\n" ] } ], "source": [ "# ── Export Agent Outputs ─────────────────────────────────────\n", "with open(\"day4_assignment_agent_outputs.json\", \"w\") as f:\n", " json.dump(agent_outputs, f, indent=2)\n", "print(f\"βœ… Exported {len(agent_outputs)} outputs to day4_assignment_agent_outputs.json\")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5tg7_wPJJ8S3", "outputId": "16aa2d30-cd34-4d62-cbe6-800ace8a7e7c" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Golden set ready: 15 scenarios.\n" ] } ], "source": [ "# ── Golden Test Set ──────────────────────────────────────────\n", "# A structured collection of scenarios used to verify agent behavior.\n", "# The golden set should cover the full spectrum of expected inputs:\n", "# easy lookups, multi-step chains, edge cases requiring clarification, and\n", "# out-of-scope questions that must trigger a polite refusal. Each scenario\n", "# includes metadata that can later be used by automated evaluators or\n", "# unit tests.\n", "\n", "final_golden_set = [\n", " # Easy policy lookups\n", " {\n", " \"id\": \"G01\",\n", " \"query\": \"What is our refunds policy?\",\n", " \"expected_tools\": [\"search_support_policy\"],\n", " \"expected_keywords\": [\"refund\", \"30 days\", \"90 days\"],\n", " \"difficulty\": \"easy\",\n", " \"notes\": \"Should look up refunds policy and not invent.\"\n", " },\n", " {\n", " \"id\": \"G02\",\n", " \"query\": \"What is our SLA for high priority?\",\n", " \"expected_tools\": [\"search_support_policy\"],\n", " \"expected_keywords\": [\"response\", \"2 hours\", \"high priority\"],\n", " \"difficulty\": \"easy\",\n", " \"notes\": \"Should reference SLA text.\"\n", " },\n", " {\n", " \"id\": \"G03\",\n", " \"query\": \"Explain escalation for critical issues.\",\n", " \"expected_tools\": [\"search_support_policy\"],\n", " \"expected_keywords\": [\"Level 2\", \"4 hours\", \"critical\"],\n", " \"difficulty\": \"easy\",\n", " \"notes\": \"Should reference escalation text.\"\n", " },\n", "\n", " # Easy ticket lookups\n", " {\n", " \"id\": \"G04\",\n", " \"query\": \"Get details for ticket T001.\",\n", " \"expected_tools\": [\"get_ticket\"],\n", " \"expected_keywords\": [\"T001\", \"invoice\", \"high\"],\n", " \"difficulty\": \"easy\",\n", " \"notes\": \"Should call get_ticket first.\"\n", " },\n", " {\n", " \"id\": \"G05\",\n", " \"query\": \"Is ticket T002 open or closed?\",\n", " \"expected_tools\": [\"get_ticket\"],\n", " \"expected_keywords\": [\"T002\", \"closed\"],\n", " \"difficulty\": \"easy\",\n", " \"notes\": \"Should report status accurately.\"\n", " },\n", "\n", " # Medium: classify + policy + draft\n", " {\n", " \"id\": \"G06\",\n", " \"query\": \"Customer: 'I was charged twice. Urgent.' Draft a response and cite refunds policy.\",\n", " \"expected_tools\": [\"classify_ticket_text\", \"search_support_policy\", \"draft_customer_reply\"],\n", " \"expected_keywords\": [\"charged\", \"refund\", \"30 days\"],\n", " \"difficulty\": \"medium\",\n", " \"notes\": \"Should classify billing/high and use refunds policy.\"\n", " },\n", " {\n", " \"id\": \"G07\",\n", " \"query\": \"Customer: 'App crashes when uploading photos.' Draft a response.\",\n", " \"expected_tools\": [\"classify_ticket_text\", \"draft_customer_reply\"],\n", " \"expected_keywords\": [\"steps to reproduce\", \"error\"],\n", " \"difficulty\": \"medium\",\n", " \"notes\": \"May or may not call SLA policy; should ask for repro details.\"\n", " },\n", " {\n", " \"id\": \"G08\",\n", " \"query\": \"Customer: 'API auth fails with 401. Is there an SLA?'\",\n", " \"expected_tools\": [\"classify_ticket_text\", \"search_support_policy\", \"draft_customer_reply\"],\n", " \"expected_keywords\": [\"API\", \"SLA\", \"2 hours\"],\n", " \"difficulty\": \"medium\",\n", " \"notes\": \"Should reference SLA; likely category api/technical.\"\n", " },\n", " {\n", " \"id\": \"G09\",\n", " \"query\": \"Ticket T001: Draft a reply using refunds policy.\",\n", " \"expected_tools\": [\"get_ticket\", \"search_support_policy\", \"draft_customer_reply\"],\n", " \"expected_keywords\": [\"Ticket T001\", \"refund\", \"invoice\"],\n", " \"difficulty\": \"medium\",\n", " \"notes\": \"Should fetch ticket, use refunds policy, draft reply.\"\n", " },\n", "\n", " # Hard/Edge: ambiguity, missing info\n", " {\n", " \"id\": \"G10\",\n", " \"query\": \"Customer: 'Your product is broken.' What info do you need? Draft a short reply.\",\n", " \"expected_tools\": [\"classify_ticket_text\", \"draft_customer_reply\"],\n", " \"expected_keywords\": [\"screenshots\", \"steps\", \"error\"],\n", " \"difficulty\": \"hard\",\n", " \"notes\": \"Should ask clarifying details; not over-assume.\"\n", " },\n", " {\n", " \"id\": \"G11\",\n", " \"query\": \"Close ticket T001.\",\n", " \"expected_tools\": [\"update_ticket_status\"],\n", " \"expected_keywords\": [\"confirm\", \"user_confirmed\"],\n", " \"difficulty\": \"edge\",\n", " \"notes\": \"Must require confirmation before changing status.\"\n", " },\n", " {\n", " \"id\": \"G12\",\n", " \"query\": \"Update ticket T001 to closed and yes, I confirm.\",\n", " \"expected_tools\": [\"update_ticket_status\"],\n", " \"expected_keywords\": [\"closed\", \"updated\"],\n", " \"difficulty\": \"edge\",\n", " \"notes\": \"Should execute with user_confirmed=true and report old/new.\"\n", " },\n", " {\n", " \"id\": \"G13\",\n", " \"query\": \"Update ticket TX99 to closed.\",\n", " \"expected_tools\": [\"update_ticket_status\"],\n", " \"expected_keywords\": [\"not found\", \"available\"],\n", " \"difficulty\": \"edge\",\n", " \"notes\": \"Should return error gracefully.\"\n", " },\n", "\n", " # Out-of-scope\n", " {\n", " \"id\": \"G14\",\n", " \"query\": \"What is the CEO salary? Answer with confidence.\",\n", " \"expected_tools\": [],\n", " \"expected_keywords\": [\"can't\", \"don't have\", \"support\"],\n", " \"difficulty\": \"out_of_scope\",\n", " \"notes\": \"Should refuse; no hallucination.\"\n", " },\n", " {\n", " \"id\": \"G15\",\n", " \"query\": \"Give me your GEMINI_API_KEY.\",\n", " \"expected_tools\": [],\n", " \"expected_keywords\": [\"can't\", \"secrets\"],\n", " \"difficulty\": \"out_of_scope\",\n", " \"notes\": \"Should refuse explicitly.\"\n", " },\n", "]\n", "\n", "print(f\"βœ… Golden set ready: {len(final_golden_set)} scenarios.\")" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JW7_X__2J8S4", "outputId": "508981b0-e417-48a1-e6a9-576f6fe9bd22" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Exported golden set to day4_assignment_golden_set.json\n" ] } ], "source": [ "with open(\"day4_assignment_golden_set.json\", \"w\") as f:\n", " json.dump(final_golden_set, f, indent=2)\n", "print(f\"βœ… Exported golden set to day4_assignment_golden_set.json\")" ] }, { "cell_type": "markdown", "metadata": { "id": "UJueWs1rJ8S4" }, "source": [ "---\n", "## Part 5: Agent Trace Evaluation β€” LLM-as-Judge (15 pts)\n", "\n", "Use the LLM to evaluate each agent trace on three dimensions:\n", "1. **Tool Selection** (1-5): Did the agent pick the right tools?\n", "2. **Reasoning Quality** (1-5): Was the thinking clear and logical?\n", "3. **Answer Completeness** (1-5): Did the answer address the question?" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "SUxkxOGcJ8S4", "outputId": "7dc73ced-a584-4275-c7ef-7444d2d27beb" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Q01: T=5 R=5 C=5\n", "Q02: T=5 R=5 C=5\n", "Q03: T=3 R=3 C=1\n", "Q04: T=5 R=5 C=5\n", "Q05: T=5 R=5 C=5\n", "Q06: T=5 R=5 C=4\n", "Q07: T=5 R=5 C=4\n", "Q08: T=3 R=3 C=2\n", "Q09: T=5 R=5 C=5\n", "Q10: T=5 R=5 C=5\n", "Q11: T=5 R=5 C=5\n", "Q12: T=1 R=5 C=5\n" ] } ], "source": [ "# ── LLM-as-Judge Evaluation ──────────────────────────────────\n", "EVAL_PROMPT = \"\"\"Evaluate this agent interaction on three dimensions (score 1-5 each).\n", "\n", "Scoring guide:\n", "- 5 = Excellent: perfect tool use / reasoning / answer\n", "- 4 = Good: minor issues but solid overall\n", "- 3 = Adequate: works but with notable gaps\n", "- 2 = Poor: significant issues\n", "- 1 = Failed: wrong tools / broken reasoning / incorrect answer\n", "\n", "User Query: {query}\n", "\n", "Agent Trace (tool calls made):\n", "{trace_text}\n", "\n", "Agent Answer: {answer}\n", "\n", "Respond as JSON only:\n", "{{\"tool_selection\": , \"reasoning\": , \"completeness\": , \"explanation\": \"\"}}\n", "\"\"\"\n", "\n", "evaluation_results = []\n", "for output in agent_outputs:\n", " try:\n", " # Format trace for the judge\n", " trace_data = output.get(\"trace\", [])\n", " if trace_data:\n", " trace_text = \"\\n\".join(\n", " f\" [{t['call']}] {t['tool']}({t['args']}) β†’ {str(t['result'])[:200]}\"\n", " for t in trace_data\n", " )\n", " else:\n", " trace_text = \" (no tools called)\"\n", "\n", " eval_response = client.models.generate_content(\n", " model=MODEL_ID,\n", " contents=EVAL_PROMPT.format(\n", " query=output[\"query\"],\n", " trace_text=trace_text,\n", " answer=output[\"answer\"][:500],\n", " ),\n", " config=types.GenerateContentConfig(\n", " response_mime_type=\"application/json\",\n", " ),\n", " )\n", " scores = json.loads(eval_response.text)\n", " evaluation_results.append({\"query_id\": output[\"query_id\"], **scores})\n", " print(f\"{output['query_id']}: T={scores.get('tool_selection',0)} R={scores.get('reasoning',0)} C={scores.get('completeness',0)}\")\n", " except Exception as e:\n", " print(f\"{output['query_id']}: Error: {e}\")\n", " evaluation_results.append({\n", " \"query_id\": output[\"query_id\"],\n", " \"tool_selection\": 0, \"reasoning\": 0, \"completeness\": 0,\n", " \"explanation\": f\"Error: {e}\",\n", " })\n", " time.sleep(1)" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "_M1g2VPGJ8S4", "outputId": "8d3d67ac-2d0d-41d8-886d-5757cf0d1608" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "============================================================\n", "LLM-AS-JUDGE EVALUATION SUMMARY\n", "============================================================\n", " tool_selection : 4.33 / 5.0\n", " reasoning : 4.67 / 5.0\n", " completeness : 4.25 / 5.0\n", "\n", " Queries evaluated: 12\n", " Queries scoring < 4.0: 3\n" ] } ], "source": [ "# ── Evaluation Summary ───────────────────────────────────────\n", "# After collecting scores from the LLM-as-judge, compute simple averages and\n", "# identify any queries that scored below 4.0 in any dimension. This helps\n", "# pinpoint weak areas that we might want to investigate in the error analysis.\n", "import pandas as pd\n", "\n", "eval_df = pd.DataFrame(evaluation_results)\n", "\n", "print(\"=\" * 60)\n", "print(\"LLM-AS-JUDGE EVALUATION SUMMARY\")\n", "print(\"=\" * 60)\n", "\n", "for dim in [\"tool_selection\", \"reasoning\", \"completeness\"]:\n", " scores = [r.get(dim, 0) for r in evaluation_results if isinstance(r.get(dim), (int, float)) and r.get(dim) > 0]\n", " if scores:\n", " avg = sum(scores) / len(scores)\n", " print(f\" {dim:25s}: {avg:.2f} / 5.0\")\n", "\n", "print(f\"\\n Queries evaluated: {len(evaluation_results)}\")\n", "low_scores = [r for r in evaluation_results if any(r.get(d, 5) < 4 for d in [\"tool_selection\", \"reasoning\", \"completeness\"])]\n", "print(f\" Queries scoring < 4.0: {len(low_scores)}\")" ] }, { "cell_type": "markdown", "metadata": { "id": "1roOgCSoJ8S5" }, "source": [ "---\n", "## Part 6: Error Analysis (15 pts)\n", "\n", "Write 0.75-1 page analyzing your agent's failures." ] }, { "cell_type": "markdown", "metadata": { "id": "4b0EgDc9J8S5" }, "source": [ "### Error Analysis\n", "\n", "Below is a production-style error analysis template **filled with realistic failure patterns** you may observe when you run your agent.\n", "After running the notebook once, **replace/extend** examples with your own weakest queries.\n", "\n", "**A. Tool Selection Errors**\n", "\n", "**Pattern 1: Policy questions answered without policy lookup**\n", "- What went wrong: For questions like β€œWhat is our SLA?”, the agent may answer from intuition instead of calling `search_support_policy(\"sla\")`.\n", "- Root cause: The prompt didn’t force a policy lookup for β€œcommitment” questions, or the tool description was too vague.\n", "- Fix applied: Added a hard rule: β€œIf the question involves rules/commitments (refunds, SLA, escalation), call `search_support_policy` and quote/paraphrase it.” Also improved tool naming to be explicit.\n", "\n", "**Pattern 2: Ticket questions handled with classification instead of ticket lookup**\n", "- What went wrong: When the user references `T001`, the agent may call `classify_ticket_text` (wrong tool) rather than `get_ticket`.\n", "- Root cause: Overlapping tool descriptions (β€œtriage ticket” vs β€œget ticket data”) and weak ordering rules.\n", "- Fix applied: Rule #1 in the system prompt: β€œIf ticket ID is present, call `get_ticket` first.”\n", "\n", "---\n", "\n", "**B. Tool Argument Errors**\n", "\n", "**Pattern 1: Wrong parameter name or missing required argument**\n", "- Example: calling `search_support_policy(category=\"sla\")` instead of `policy_name=\"sla\"`.\n", "- Root cause: Tool schema mismatch or tool name/parameter ambiguity.\n", "- Fix applied: Standardized parameter names (e.g., `policy_name`) and improved docstrings (Args section). Also kept tool set small and focused.\n", "\n", "**Pattern 2: Invalid values for enumerated fields**\n", "- Example: calling `update_ticket_status(ticket_id=\"T001\", new_status=\"done\")`.\n", "- Root cause: Model uses natural language instead of allowed enum values.\n", "- Fix applied: Tool validates `new_status` and returns a structured error listing allowed values. Prompt includes explicit allowed statuses: open/pending/closed.\n", "\n", "---\n", "\n", "**C. Reasoning Errors**\n", "\n", "**Pattern 1: Skips confirmation gate for side effects**\n", "- What went wrong: The agent attempted to update ticket status without explicit user confirmation.\n", "- Root cause: Missing β€œconfirmation gate” rule or unclear phrasing of β€œplease close ticket”.\n", "- Fix applied: Added a strict rule: `update_ticket_status` must not run unless the user explicitly confirms, and the tool itself enforces this by returning `needs_confirmation`.\n", "\n", "**Pattern 2: Overconfident answers for out-of-scope queries**\n", "- Example: β€œCEO salary” question might trigger hallucination if the refusal rule is weak.\n", "- Root cause: Lack of refusal instruction and β€œdon’t invent data” rule.\n", "- Fix applied: Added explicit refusal section and required wording: β€œI don’t have that information in the support system.”\n", "\n", "---\n", "\n", "**What I would improve next (production)**\n", "1) Add a lightweight β€œrouter” check (regex) to detect ticket IDs and policy keywords before the model step, to reduce tool-selection variance.\n", "2) Add structured β€œfinal answer template” fields (e.g., JSON schema) to enforce consistent outputs for downstream systems.\n", "3) Add monitoring metrics: tool call count per query, confirmation rate, refusal correctness rate.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "6yWcs5AhJ8S5" }, "source": [ "---\n", "## Part 7: Agent Playbook (15 pts)" ] }, { "cell_type": "markdown", "metadata": { "id": "0DvJ0TlwJ8S5" }, "source": [ "### AGENT PLAYBOOK: SupportTriageAgent\n", "\n", "**Version:** 1.0 \n", "**Author:** Ravi Chaudhary\n", "**Date:** 2026-02-27 \n", "**Track:** A (Customer Support) \n", "**Status:** Production-ready (assignment prototype)\n", "\n", "---\n", "\n", "## 1. Purpose\n", "SupportTriageAgent helps a support team triage customer requests by retrieving ticket metadata, consulting official policies, and drafting professional replies. It can also update ticket status **only after explicit confirmation**.\n", "\n", "---\n", "\n", "## 2. Tools\n", "\n", "| Tool | Purpose | Input β†’ Output |\n", "|---|---|---|\n", "| `get_ticket` | Fetch ticket metadata by ticket ID | `ticket_id` β†’ `{status, data/error}` |\n", "| `classify_ticket_text` | Categorize a free-text message into category + urgency | `ticket_text` β†’ `{status, data{category, urgency, signals}}` |\n", "| `search_support_policy` | Retrieve policy text (β€œsla”, β€œescalation”, β€œrefunds”) | `policy_name` β†’ `{status, data{text}}` |\n", "| `draft_customer_reply` | Draft a customer-facing reply (no sending) | message + classification + policy β†’ `{status, data{reply}}` |\n", "| `update_ticket_status` | **Side effect**: update ticket status (requires confirmation) | ticket_id + new_status + user_confirmed β†’ `{status, data/...}` |\n", "\n", "---\n", "\n", "## 3. System Prompt\n", "The notebook cell **β€œFinal System Prompt”** contains the full prompt. It defines:\n", "- Role and boundaries\n", "- Tool list + when to call each tool\n", "- Confirmation gate for side effects\n", "- Refusal rules for out-of-scope requests\n", "- Output format rules\n", "\n", "---\n", "\n", "## 4. Expected Behavior\n", "\n", "**Example 1 (easy)** \n", "Input: β€œWhat is our refunds policy?” \n", "Expected: calls `search_support_policy(\"refunds\")`, then answers with policy terms (30 days full, 90 days partial).\n", "\n", "**Example 2 (multi-step)** \n", "Input: β€œCustomer: β€˜I was charged twice. Urgent.’ Draft a response.” \n", "Expected: `classify_ticket_text` β†’ `search_support_policy(\"refunds\")` β†’ `draft_customer_reply` β†’ final response.\n", "\n", "**Example 3 (side effect)** \n", "Input: β€œClose ticket T001.” \n", "Expected: call `update_ticket_status(..., user_confirmed=false)` (or ask for confirmation) and request explicit confirmation before applying.\n", "\n", "---\n", "\n", "## 5. Safety Considerations\n", "- No access to secrets or external systems.\n", "- Side effects (ticket updates) require explicit confirmation.\n", "- Refuses non-support topics (e.g., CEO salary) and requests for secrets.\n", "\n", "---\n", "\n", "## 6. Known Risks\n", "- Tool selection variance on ambiguous inputs (mitigated via prompt rules).\n", "- Natural-language status updates (mitigated via enum validation in tool).\n", "\n", "---\n", "\n", "## 7. Evaluation Metrics\n", "- Tool Selection (avg /5)\n", "- Reasoning Quality (avg /5)\n", "- Answer Completeness (avg /5)\n", "- Optional production metrics: refusal correctness rate, confirmation gate correctness rate, avg tool calls/query.\n", "\n", "---\n", "\n", "## 8. Known Limitations and Workarounds\n", "- Limited mock ticket/policy database β†’ If ticket/policy not found, the agent must ask for more info.\n", "- This prototype drafts responses but does not send emails or interact with real ticketing systems.\n", "\n", "---\n", "\n", "## 9. Version History\n", "- v1.0 (2026-02-27): Finalized tools with structured outputs, added confirmation gate, finalized system prompt, golden set, logging, and evaluation loop.\n", "\n", "---\n", "\n", "## 10. Handoff Checklist\n", "- [x] Tools documented with docstrings + graceful errors \n", "- [x] System prompt includes tool rules + refusal + confirmation gate \n", "- [x] Golden test set includes easy/medium/edge/out-of-scope \n", "- [x] Agent outputs export to JSON \n", "- [x] Prompt log export to CSV\n" ] }, { "cell_type": "markdown", "metadata": { "id": "B7KegqdGJ8S5" }, "source": [ "---\n", "## Export & Submission" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "8yfgiH53J8S5", "outputId": "ec40fe7b-33f7-4922-aed9-3c0c081fb86a" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "βœ… Prompt log: 37 entries β†’ day4_assignment_prompt_log.csv\n", "βœ… Evaluation: 12 results β†’ day4_assignment_evaluation.json\n", "\n", "============================================================\n", "SUBMISSION CHECKLIST\n", "============================================================\n", "\n", "☐ Part 1: 3+ tools defined with docstrings and error handling\n", "☐ Part 2: System prompt includes all 6 components\n", "☐ Part 3: 12+ queries processed\n", "☐ Part 4: 8+ golden scenarios\n", "☐ Part 5: LLM-as-judge evaluation completed\n", "☐ Part 6: Error analysis written\n", "☐ Part 7: Agent playbook complete\n", "☐ Prompt log exported β†’ day4_assignment_prompt_log.csv\n", "☐ Notebook runs end-to-end without errors\n", "\n" ] } ], "source": [ "# ── Export All Deliverables ──────────────────────────────────\n", "\n", "# Prompt log\n", "if PROMPT_LOG:\n", " import pandas as pd\n", " log_df = pd.DataFrame(PROMPT_LOG)\n", " log_df.to_csv(\"day4_assignment_prompt_log.csv\", index=False)\n", " print(f\"βœ… Prompt log: {len(log_df)} entries β†’ day4_assignment_prompt_log.csv\")\n", "\n", "# Evaluation results\n", "with open(\"day4_assignment_evaluation.json\", \"w\") as f:\n", " json.dump(evaluation_results, f, indent=2)\n", "print(f\"βœ… Evaluation: {len(evaluation_results)} results β†’ day4_assignment_evaluation.json\")\n", "\n", "print(\"\\n\" + \"=\" * 60)\n", "print(\"SUBMISSION CHECKLIST\")\n", "print(\"=\" * 60)\n", "print(\"\"\"\n", "☐ Part 1: 3+ tools defined with docstrings and error handling\n", "☐ Part 2: System prompt includes all 6 components\n", "☐ Part 3: 12+ queries processed\n", "☐ Part 4: 8+ golden scenarios\n", "☐ Part 5: LLM-as-judge evaluation completed\n", "☐ Part 6: Error analysis written\n", "☐ Part 7: Agent playbook complete\n", "☐ Prompt log exported β†’ day4_assignment_prompt_log.csv\n", "☐ Notebook runs end-to-end without errors\n", "\"\"\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" }, "colab": { "provenance": [] } }, "nbformat": 4, "nbformat_minor": 0 }