chatkit-actions

Original source / Raw SKILL.md

---
name: chatkit-actions
description: Implements interactive widget actions and bidirectional communication patterns for ChatKit. This skill should be used when building AI-driven interactive UIs with buttons, forms, entity tagging (@mentions), composer tools, and server-handled widget actions. Covers the full widget lifecycle from creation to replacement.
---

# ChatKit Actions Skill

## Overview

This skill unlocks the full power of ChatKit's agentic UI capabilities - where AI can render interactive widgets, users can click buttons that trigger both client and server actions, and the conversation becomes a two-way interactive experience.

## Core Concepts

### Action Handler Types

Widgets can specify where actions are handled:

| Handler | Defined In | Processed By | Use Case |
|---------|------------|--------------|----------|
| `"client"` | Widget template | Frontend `onAction` | Navigation, local state, send follow-up |
| `"server"` | Widget template | Backend `action()` method | Data mutation, widget replacement |

### Widget Lifecycle

```
1. Agent tool generates widget → yield WidgetItem
2. Widget renders in chat with action buttons
3. User clicks action → action dispatched
4. Handler processes action:
   - client: onAction callback in frontend
   - server: action() method in ChatKitServer
5. Optional: Widget replaced with updated state
```

## Implementation Patterns

### Pattern 1: Widget Templates (.widget files)

**When**: Define reusable widget layouts with dynamic data

**Widget Template Format**:
```json
{
  "version": "1.0",
  "name": "task_list",
  "template": "{\"type\":\"ListView\",\"children\":[...jinja template...]}",
  "jsonSchema": {
    "type": "object",
    "properties": {
      "tasks": { "type": "array", "items": {...} }
    }
  }
}
```

**Widget Components Available**:
- Layout: `ListView`, `ListViewItem`, `Row`, `Col`, `Box`
- Content: `Text`, `Title`, `Image`, `Icon`
- Interactive: `Button` (with `onClickAction`)
- Styling: `gap`, `padding`, `background`, `border`, `radius`

**Example - Task List Widget**:
```json
{
  "type": "ListView",
  "children": [
    {
      "type": "ListViewItem",
      "key": "task-1",
      "onClickAction": {
        "type": "task.select",
        "handler": "client",
        "payload": { "taskId": "task-1" }
      },
      "children": [
        {
          "type": "Row",
          "gap": 3,
          "children": [
            { "type": "Icon", "name": "check", "color": "success" },
            { "type": "Text", "value": "Complete review", "weight": "semibold" }
          ]
        }
      ]
    }
  ]
}
```

**Python - Loading Templates**:
```python
from chatkit.widgets import WidgetTemplate, WidgetRoot

# Load template from file
task_list_template = WidgetTemplate.from_file("task_list.widget")

def build_task_list_widget(tasks: list[Task]) -> WidgetRoot:
    return task_list_template.build(
        data={
            "tasks": [task.model_dump() for task in tasks],
            "selected": None,
        }
    )
```

**Evidence**: `cat-lounge/backend/app/widgets/cat_name_suggestions.widget`

### Pattern 2: Client-Handled Actions

**When**: Actions that update local state, navigate, or send follow-up messages

**Widget Definition (handler: "client")**:
```json
{
  "type": "Button",
  "label": "View Article",
  "onClickAction": {
    "type": "open_article",
    "handler": "client",
    "payload": { "id": "article-123" }
  }
}
```

**Frontend Handler**:
```typescript
import { useChatKit, type Widgets } from "@openai/chatkit-react";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (
      action: { type: string; payload?: Record<string, unknown> },
      widgetItem: { id: string; widget: Widgets.Card | Widgets.ListView }
    ) => {
      switch (action.type) {
        case "open_article":
          // Navigate to article
          navigate(`/article/${action.payload?.id}`);
          break;

        case "more_suggestions":
          // Send follow-up message
          await chatkit.sendUserMessage({ text: "More suggestions, please" });
          break;

        case "select_option":
          // Update local state
          setSelectedOption(action.payload?.optionId);
          break;
      }
    },
  },
});
```

**Evidence**: `news-guide/frontend/src/components/ChatKitPanel.tsx:55-79`

### Pattern 3: Server-Handled Actions

**When**: Actions that mutate data, update widgets, or require backend processing

**Widget Definition (handler: "server")**:
```json
{
  "type": "ListViewItem",
  "onClickAction": {
    "type": "line.select",
    "handler": "server",
    "payload": { "id": "blue-line" }
  }
}
```

**Backend Handler**:
```python
from chatkit.server import ChatKitServer
from chatkit.types import (
    Action,
    WidgetItem,
    ThreadItemReplacedEvent,
    ThreadItemDoneEvent,
    AssistantMessageItem,
    HiddenContextItem,
    ClientEffectEvent,
)

class MyServer(ChatKitServer[dict]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:

        if action.type == "line.select":
            line_id = action.payload["id"]

            # 1. Update widget with selection
            updated_widget = build_line_selector_widget(
                lines=self.lines,
                selected=line_id,
            )
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Add hidden context for future agent input
            await self.store.add_thread_item(
                thread.id,
                HiddenContextItem(
                    id=self.store.generate_item_id("ctx", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=f"<LINE_SELECTED>{line_id}</LINE_SELECTED>",
                ),
                context=context,
            )

            # 3. Stream assistant message
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {line_id}. Where to add station?"}],
                )
            )

            # 4. Trigger client effect
            yield ClientEffectEvent(
                name="location_select_mode",
                data={"lineId": line_id},
            )
```

**Evidence**: `metro-map/backend/app/server.py`

### Pattern 4: Client-to-Server Action Forwarding

**When**: Client handles action locally, then notifies server for persistence/widget update

**Frontend - Send Custom Action**:
```typescript
widgets: {
  onAction: async (action, widgetItem) => {
    if (action.type === "select_name") {
      // 1. Forward to server for processing
      await chatkit.sendCustomAction(action, widgetItem.id);

      // 2. Optionally refresh local state after server processes
      const data = await refreshCatStatus();
      if (data) {
        handleStatusUpdate(data, `Now called ${data.name}`);
      }
    }
  },
}
```

**Evidence**: `cat-lounge/frontend/src/components/ChatKitPanel.tsx:68-80`

### Pattern 5: Entity Tagging (@mentions)

**When**: Allow users to @mention entities (users, articles, tasks) in messages

**Frontend - Entity Configuration**:
```typescript
import { useChatKit, type Entity } from "@openai/chatkit-react";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    // Search for entities as user types @...
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await fetch(`/api/search?q=${query}`).then(r => r.json());

      return results.map((item) => ({
        id: item.id,
        title: item.name,
        icon: item.type === "person" ? "profile" : "document",
        group: item.type === "People" ? "People" : "Articles",
        interactive: true,
        data: {
          type: item.type,
          article_id: item.id,
        },
      }));
    },

    // Handle entity click (e.g., navigate)
    onClick: (entity: Entity) => {
      if (entity.data?.article_id) {
        navigate(`/article/${entity.data.article_id}`);
      }
    },

    // Render entity preview on hover
    onRequestPreview: async (entity: Entity) => {
      const details = await fetch(`/api/entity/${entity.id}`).then(r => r.json());

      return {
        preview: {
          type: "Card",
          children: [
            { type: "Text", value: entity.title, weight: "bold" },
            { type: "Text", value: details.description, color: "tertiary" },
          ],
        },
      };
    },
  },
});
```

**Backend - Converting Entity Tags**:
```python
# thread_item_converter.py
class EntityAwareConverter(BasicThreadItemConverter):
    """Convert entity tags to model-readable markers."""

    async def to_agent_input(self, items: list[ThreadItem]) -> list:
        result = []
        for item in items:
            if isinstance(item, UserMessageItem):
                content = item.content
                # Convert entity tags to XML markers
                for entity in item.entities or []:
                    if entity.type == "article":
                        content = content.replace(
                            f"@{entity.title}",
                            f"<ARTICLE_REFERENCE id='{entity.id}'>{entity.title}</ARTICLE_REFERENCE>"
                        )
                result.append({"role": "user", "content": content})
        return result
```

**Evidence**:
- `news-guide/frontend/src/components/ChatKitPanel.tsx:122-126`
- `news-guide/backend/app/thread_item_converter.py`
- `metro-map/frontend/src/components/ChatKitPanel.tsx:73-117`

### Pattern 6: Composer Tools (Mode Selection)

**When**: Let users select different AI modes/tools from the composer

**Frontend - Tool Configuration**:
```typescript
const TOOL_CHOICES = [
  {
    id: "general",
    label: "Chat",
    shortLabel: "Chat",
    icon: "sparkle",
    placeholderOverride: "Ask anything...",
    pinned: true,
  },
  {
    id: "event_finder",
    label: "Find Events",
    shortLabel: "Events",
    icon: "calendar",
    placeholderOverride: "What events are you looking for?",
    pinned: true,
  },
  {
    id: "puzzle",
    label: "Word Puzzle",
    shortLabel: "Puzzle",
    icon: "bolt",
    placeholderOverride: "Ready for today's puzzle?",
    pinned: false,
  },
];

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },
  composer: {
    placeholder: "What would you like to do?",
    tools: TOOL_CHOICES,
  },
});
```

**Backend - Routing by Tool Choice**:
```python
# server.py
async def respond(self, thread, item, context):
    tool_choice = context.get("tool_choice")

    if tool_choice == "event_finder":
        agent = self.event_finder_agent
    elif tool_choice == "puzzle":
        agent = self.puzzle_agent
    else:
        agent = self.general_agent

    # Run selected agent
    result = Runner.run_streamed(agent, input_items, context=agent_context)
    async for event in stream_agent_response(agent_context, result):
        yield event
```

**Evidence**: `news-guide/frontend/src/lib/config.ts`

### Pattern 7: Thread Item Actions (Feedback/Retry/Share)

**When**: Enable built-in actions on AI messages

**Frontend Configuration**:
```typescript
const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  threadItemActions: {
    feedback: true,   // Thumbs up/down
    retry: true,      // Regenerate response
    share: true,      // Share message
  },

  onLog: ({ name, data }) => {
    if (name === "message.feedback") {
      // Track feedback analytics
      fetch("/api/analytics/feedback", {
        method: "POST",
        body: JSON.stringify(data),
      });
    }
    if (name === "message.share") {
      // Track share events
      fetch("/api/analytics/share", {
        method: "POST",
        body: JSON.stringify(data),
      });
    }
  },
});
```

### Pattern 8: Widget Streaming from Tools

**When**: Agent tool generates a widget as part of response

**Backend - Tool with Widget Output**:
```python
from chatkit.types import WidgetItem
from agents import function_tool

@function_tool
async def show_article_list(ctx: AgentContext, query: str) -> str:
    """Show a list of articles matching the query."""

    articles = await article_store.search(query)

    # Build widget
    widget = build_article_list_widget(articles)

    # Yield widget item
    widget_item = WidgetItem(
        id=ctx.store.generate_item_id("widget", ctx.thread, ctx.request_context),
        thread_id=ctx.thread.id,
        created_at=datetime.now(),
        widget=widget,
    )

    # Save to store
    await ctx.store.add_thread_item(ctx.thread.id, widget_item, ctx.request_context)

    # Yield as event
    yield ThreadItemDoneEvent(item=widget_item)

    return f"Showing {len(articles)} articles"
```

**Evidence**: `news-guide/backend/app/agents/news_agent.py`

## Widget Component Reference

### Layout Components

| Component | Props | Description |
|-----------|-------|-------------|
| `ListView` | `children` | Scrollable list container |
| `ListViewItem` | `key`, `onClickAction`, `children` | Clickable list item |
| `Row` | `gap`, `align`, `justify`, `children` | Horizontal flex |
| `Col` | `gap`, `align`, `justify`, `flex`, `padding`, `children` | Vertical flex |
| `Box` | `size`, `radius`, `background`, `border`, `padding` | Container with styling |

### Content Components

| Component | Props | Description |
|-----------|-------|-------------|
| `Text` | `value`, `size`, `weight`, `color`, `maxLines` | Text display |
| `Title` | `value`, `size`, `weight` | Heading text |
| `Image` | `src`, `alt`, `width`, `height`, `fit`, `radius` | Image display |
| `Icon` | `name`, `size`, `color` | Icon from icon set |

### Interactive Components

| Component | Props | Description |
|-----------|-------|-------------|
| `Button` | `label`, `variant`, `color`, `size`, `pill`, `block`, `iconStart`, `iconEnd`, `onClickAction`, `disabled` | Clickable button |

### Action Structure

```typescript
interface Action {
  type: string;           // Action identifier
  handler: "client" | "server";
  payload?: Record<string, unknown>;
}
```

## Common Patterns Summary

| Pattern | Frontend | Backend | Use Case |
|---------|----------|---------|----------|
| Navigation | `onAction` → navigate | - | Open details page |
| Follow-up | `onAction` → sendUserMessage | - | "More suggestions" |
| Selection | `sendCustomAction` | `action()` → `ThreadItemReplacedEvent` | Select from list |
| Data mutation | `sendCustomAction` | `action()` → update DB | Approve/reject |
| @mentions | `entities.onTagSearch` | `ThreadItemConverter` | Reference entities |
| Mode switch | `composer.tools` | Route by tool_choice | Different agents |

## Critical Implementation Details

### Action Object Structure

**IMPORTANT**: The `Action` object uses `payload`, NOT `arguments`:

```python
# ❌ WRONG - Will cause AttributeError
action.arguments  # 'Action' object has no attribute 'arguments'

# ✅ CORRECT
action.payload    # Access action data via .payload
```

**Action Type Definition**:
```python
from chatkit.types import Action

# Action[str, Any] has these fields:
action.type      # str - action identifier (e.g., "task.start")
action.payload   # dict[str, Any] - action data
action.handler   # "client" | "server" - where action is processed
```

### Server Action Handler Signature

**CRITICAL**: The `context` parameter is `RequestContext`, NOT `dict[str, Any]`

```python
# Type annotation vs runtime reality mismatch
async def action(
    self,
    thread: ThreadMetadata,
    action: Action[str, Any],
    sender: WidgetItem | None,
    context: dict[str, Any],  # ⚠️ Type hint says dict, but runtime is RequestContext!
) -> AsyncIterator[ThreadStreamEvent]:

    # ❌ WRONG - Tries to wrap RequestContext inside RequestContext
    request_context = RequestContext(metadata=context)

    # ✅ CORRECT - Use context directly, it's already RequestContext
    user_id = context.user_id
    metadata = context.metadata
```

**Why this happens**: ChatKit SDK passes `RequestContext` object at runtime, despite type annotations suggesting `dict`. Always use `context` directly without wrapping.

### UserMessageItem Required Fields

When creating synthetic user messages from actions, **ALL** these fields are required:

```python
from chatkit.types import UserMessageItem, UserMessageTextContent
from datetime import datetime

# ❌ WRONG - Missing required fields causes ValidationError
synthetic_message = UserMessageItem(
    content=[UserMessageTextContent(type="text", text=message_text)]
)

# ✅ CORRECT - Include all required fields
synthetic_message = UserMessageItem(
    id=self.store.generate_item_id("message", thread, context),
    thread_id=thread.id,
    created_at=datetime.now(),
    content=[UserMessageTextContent(type="input_text", text=message_text)],
    inference_options={},
)
```

**Required fields**:
- `id`: Generate via `store.generate_item_id("message", thread, context)`
- `thread_id`: From `thread.id` parameter
- `created_at`: Current timestamp via `datetime.now()`
- `content`: List of content blocks (UserMessageTextContent)
- `inference_options`: Empty dict `{}` if no special options

**UserMessageTextContent type values**:
- ✅ `type="input_text"` - User text input (correct)
- ❌ `type="text"` - Invalid for UserMessageTextContent (causes ValidationError)

### Local Tool Wrappers for Widget Streaming

**Problem**: Agent calls MCP tool successfully, but widget doesn't appear in UI.

**Root Cause**: Widgets stream via `RunHooks` pattern. MCP tools alone don't trigger widget rendering - you need **local tool wrappers**.

**Solution Pattern**:

```python
# 1. Create local tool wrapper
from agents import function_tool

@function_tool
async def show_task_form(
    ctx: RunContextWrapper[TaskFlowAgentContext],
) -> str:
    """Show interactive task creation form widget."""

    agent_ctx = ctx.context
    mcp_url = agent_ctx.mcp_server_url

    # Call MCP tool via HTTP
    result = await _call_mcp_tool(
        mcp_url,
        "taskflow_show_task_form",
        arguments={"params": {"user_id": agent_ctx.user_id}},
        access_token=agent_ctx.access_token,
    )

    # Return result - RunHooks will intercept and stream widget
    return json.dumps(result)

# 2. Register local wrapper with agent
agent = Agent(
    name="TaskFlow Assistant",
    tools=[
        show_task_form,  # Local wrapper - triggers RunHooks
        # ... other local wrappers
    ],
)

# 3. In RunHooks.on_tool_end() - Stream widget
async def on_tool_end(self, output: str | None, tool_name: str) -> None:
    if tool_name == "show_task_form":
        result = json.loads(output)
        if result.get("action") == "show_form":
            widget = build_task_form_widget()
            yield WidgetItem(...)
```

**Key insight**: Direct MCP tools → no widgets. Local wrappers → RunHooks → widgets streamed.

## Common Pydantic Validation Errors

### Error 1: 'Action' object has no attribute 'arguments'

```
AttributeError: 'Action[str, Any]' object has no attribute 'arguments'
```

**Fix**: Use `action.payload` instead of `action.arguments`

### Error 2: UserMessageTextContent type mismatch

```
ValidationError: Input should be 'input_text' [type=literal_error, input_value='text']
```

**Fix**: Use `type="input_text"` for user input, not `type="text"`

### Error 3: UserMessageItem missing required fields

```
4 validation errors for UserMessageItem
- id: Field required
- thread_id: Field required
- created_at: Field required
- inference_options: Field required
```

**Fix**: Include all required fields when creating UserMessageItem (see pattern above)

### Error 4: RequestContext wrapping issue

```
2 validation errors for RequestContext
user_id: Field required
metadata: Input should be a valid dictionary [input_value=RequestContext(...)]
```

**Fix**: Don't wrap `context` - it's already a RequestContext object

## Widget Action Testing Checklist

Before claiming widget actions are complete, test:

- [ ] Widget renders with correct data
- [ ] All buttons have clear labels (not just icons)
- [ ] Client actions navigate/update UI correctly
- [ ] Server actions call backend successfully
- [ ] Action payload contains all required data
- [ ] Widget updates after server action completes
- [ ] No AttributeError on action.payload access
- [ ] No ValidationError on UserMessageItem creation
- [ ] Local tool wrappers trigger widget streaming
- [ ] All status transitions have appropriate buttons
- [ ] Test with real user session (not mock data)
- [ ] Check browser console for errors
- [ ] Verify backend logs show action processing
- [ ] Test error cases (network failure, invalid data)

## Anti-Patterns to Avoid

1. **Mixing handlers** - Don't handle same action in both client and server
2. **Missing payload** - Always include necessary data in action payload
3. **Forgetting widget ID** - `sendCustomAction` requires widget ID for updates
4. **Not updating widget** - Server actions should yield `ThreadItemReplacedEvent`
5. **Blocking in onAction** - Keep client handlers fast, offload to server
6. **Using action.arguments** - Use `action.payload` (arguments doesn't exist)
7. **Wrapping RequestContext** - Context is already RequestContext, don't wrap it
8. **Missing UserMessageItem fields** - Include id, thread_id, created_at, inference_options
9. **Wrong content type** - Use `type="input_text"` for user messages
10. **No local tool wrappers** - MCP tools alone don't stream widgets
11. **Not testing thoroughly** - Test all actions with real data before claiming done
12. **Assuming type hints are correct** - ChatKit has type annotation vs runtime mismatches

## References

### Documentation
- `references/widget-templates.md` - Widget template syntax
- `references/client-vs-server-actions.md` - Action routing guide
- `references/entity-tagging.md` - @mention implementation
- `references/composer-tools.md` - Tool choice patterns
- `references/server-action-handler.py` - Complete backend action handler pattern

### Widget Template Assets
- `assets/line-select.widget` - Server action selection list (metro-map pattern)
- `assets/name-suggestions.widget` - Client action with "more" button (cat-lounge pattern)
- `assets/article-list.widget` - Rich card layout with images (news-guide pattern)

## Evidence Sources

All patterns derived from OpenAI ChatKit advanced samples:
- `blueprints/openai-chatkit-advanced-samples-main/examples/cat-lounge/`
- `blueprints/openai-chatkit-advanced-samples-main/examples/metro-map/`
- `blueprints/openai-chatkit-advanced-samples-main/examples/news-guide/`


---

## Referenced Files

> The following files are referenced in this skill and included for context.

### references/widget-templates.md

```markdown
# Widget Templates Reference

## Overview

Widget templates (`.widget` files) define reusable, data-driven UI components that can be rendered in chat responses.

## Template Structure

```json
{
  "version": "1.0",
  "name": "widget_name",
  "template": "<jinja-templated JSON string>",
  "jsonSchema": {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": { ... },
    "required": [ ... ]
  }
}
```

## Template Syntax

Templates use Jinja2 syntax embedded in JSON:

```jinja
{
  "type": "ListView",
  "children": [
    {%- for item in items -%}
    {
      "type": "ListViewItem",
      "key": {{ item.id | tojson }},
      "children": [
        {
          "type": "Text",
          "value": {{ item.name | tojson }}
        }
      ]
    }{% if not loop.last %},{% endif %}
    {%- endfor -%}
  ]
}
```

## Component Reference

### ListView

```json
{
  "type": "ListView",
  "children": [ ... ListViewItem components ... ]
}
```

### ListViewItem

```json
{
  "type": "ListViewItem",
  "key": "unique-key",
  "gap": 3,
  "onClickAction": {
    "type": "action.type",
    "handler": "client" | "server",
    "payload": { ... }
  },
  "children": [ ... ]
}
```

### Row

```json
{
  "type": "Row",
  "gap": 3,
  "align": "center" | "start" | "end" | "stretch",
  "justify": "start" | "end" | "center" | "between" | "around",
  "children": [ ... ]
}
```

### Col

```json
{
  "type": "Col",
  "gap": 2,
  "flex": 1,
  "align": "stretch",
  "justify": "between",
  "padding": { "x": 4, "y": 3 },
  "children": [ ... ]
}
```

### Box

```json
{
  "type": "Box",
  "size": 25,
  "radius": "full" | "none" | "soft" | "round",
  "background": "#06b6d4",
  "border": { "color": "gray-900", "size": 1 },
  "padding": 0,
  "maxWidth": 450,
  "minWidth": 300,
  "children": [ ... ]
}
```

### Text

```json
{
  "type": "Text",
  "value": "Display text",
  "size": "xs" | "sm" | "md" | "lg" | "xl",
  "weight": "normal" | "semibold" | "bold",
  "color": "tertiary" | "emphasis" | "gray-300" | "gray-900",
  "maxLines": 4
}
```

### Image

```json
{
  "type": "Image",
  "src": "https://example.com/image.jpg",
  "alt": "Description",
  "width": 160,
  "height": 200,
  "fit": "cover" | "contain",
  "position": "top" | "center",
  "radius": "none" | "soft" | "round",
  "frame": true
}
```

### Icon

```json
{
  "type": "Icon",
  "name": "check" | "dot" | "chevron-right" | "sparkle" | ...,
  "size": "sm" | "md" | "lg" | "xl",
  "color": "success" | "gray-200" | "gray-300"
}
```

### Button

```json
{
  "type": "Button",
  "label": "Click me",
  "variant": "solid" | "outline" | "ghost",
  "color": "discovery" | "warning" | "success",
  "size": "sm" | "md" | "lg",
  "pill": true,
  "block": true,
  "iconStart": "icon-name",
  "iconEnd": "sparkle",
  "disabled": false,
  "onClickAction": {
    "type": "action.type",
    "handler": "client",
    "payload": { ... }
  }
}
```

## Python Usage

```python
from chatkit.widgets import WidgetTemplate, WidgetRoot

# Load template
template = WidgetTemplate.from_file("my_widget.widget")

# Build with data
widget: WidgetRoot = template.build(
    data={
        "items": [{"id": "1", "name": "First"}],
        "selected": None,
    }
)

# Yield as widget item
yield ThreadItemDoneEvent(
    item=WidgetItem(
        id=generate_id(),
        thread_id=thread.id,
        created_at=datetime.now(),
        widget=widget,
    )
)
```

## Conditional Rendering

```jinja
{%- if selected == item.id -%}
  { "type": "Icon", "name": "check", "color": "success" }
{%- else -%}
  { "type": "Icon", "name": "dot", "color": "gray-300" }
{%- endif -%}
```

## Disabling Actions

```jinja
"onClickAction": {% if selected %}null{% else %}{
  "type": "select",
  "handler": "client",
  "payload": { "id": {{ item.id | tojson }} }
}{% endif %}
```

```

### references/client-vs-server-actions.md

```markdown
# Client vs Server Actions Guide

## Decision Framework

| Question | Client Action | Server Action |
|----------|--------------|---------------|
| Mutates backend data? | No | **Yes** |
| Needs widget update? | No | **Yes** |
| Navigation only? | **Yes** | No |
| Sends follow-up message? | **Yes** | No |
| Local state change? | **Yes** | No |
| Requires auth context? | No | **Yes** |

## Client Action Flow

```
User clicks → onAction fires → Local processing only
```

```typescript
// Frontend
onAction: async (action, widgetItem) => {
  if (action.type === "open_details") {
    navigate(`/details/${action.payload?.id}`);
  }
  if (action.type === "more") {
    await chatkit.sendUserMessage({ text: "More please" });
  }
}
```

## Server Action Flow

```
User clicks → onAction fires → sendCustomAction → Backend action() → Widget update/message
```

```typescript
// Frontend - forward to server
onAction: async (action, widgetItem) => {
  if (action.type === "approve") {
    await chatkit.sendCustomAction(action, widgetItem.id);
  }
}
```

```python
# Backend - handle and respond
async def action(self, thread, action, sender, context):
    if action.type == "approve":
        # 1. Mutate data
        await self.db.approve(action.payload["id"])

        # 2. Update widget
        updated_widget = build_widget(approved=True)
        yield ThreadItemReplacedEvent(
            item=sender.model_copy(update={"widget": updated_widget})
        )

        # 3. Send confirmation message
        yield ThreadItemDoneEvent(
            item=AssistantMessageItem(content=[{"text": "Approved!"}])
        )
```

## Hybrid Pattern

When action needs both local and server handling:

```typescript
onAction: async (action, widgetItem) => {
  if (action.type === "select_name") {
    // 1. Forward to server for data persistence
    await chatkit.sendCustomAction(action, widgetItem.id);

    // 2. Local state refresh after server processes
    const data = await refreshLocalState();
    updateUI(data);
  }
}
```

## Widget Template Handler Declaration

```json
// Client-handled
"onClickAction": {
  "type": "open_link",
  "handler": "client",
  "payload": { "url": "/page" }
}

// Server-handled
"onClickAction": {
  "type": "submit_form",
  "handler": "server",
  "payload": { "formId": "123" }
}
```

## Common Patterns

### Navigation (Client)
```json
{ "type": "view_details", "handler": "client", "payload": { "id": "123" } }
```

### Selection with State Update (Server)
```json
{ "type": "select_option", "handler": "server", "payload": { "optionId": "abc" } }
```

### Follow-up Message (Client)
```typescript
// In onAction handler
await chatkit.sendUserMessage({ text: "Tell me more about this" });
```

### Data Mutation + Widget Update (Server)
```python
# In action() handler
await db.update(...)
yield ThreadItemReplacedEvent(item=updated_widget)
yield ThreadItemDoneEvent(item=confirmation_message)
```

```

### references/entity-tagging.md

```markdown
# Entity Tagging (@mentions) Reference

## Overview

Entity tagging enables users to @mention entities (users, articles, tasks, stations) directly in the chat composer. The AI receives these as structured references.

## Frontend Configuration

```typescript
import { useChatKit, type Entity } from "@openai/chatkit-react";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    // Called as user types @...
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await searchEntities(query);
      return results.map(item => ({
        id: item.id,
        title: item.name,
        icon: getIcon(item.type),      // "profile", "document", etc.
        group: getGroup(item.type),    // "People", "Articles"
        interactive: true,
        data: {                        // Custom data passed to handlers
          type: item.type,
          url: item.url,
        },
      }));
    },

    // Called when user clicks an entity tag
    onClick: (entity: Entity) => {
      if (entity.data?.url) {
        navigate(entity.data.url);
      }
    },

    // Called on hover to show preview card
    onRequestPreview: async (entity: Entity) => {
      const details = await fetchEntityDetails(entity.id);
      return {
        preview: {
          type: "Card",
          children: [
            { type: "Text", value: entity.title, weight: "bold" },
            { type: "Text", value: details.description, color: "tertiary" },
            { type: "Text", value: `Type: ${entity.data?.type}`, size: "sm" },
          ],
        },
      };
    },
  },
});
```

## Entity Interface

```typescript
interface Entity {
  id: string;           // Unique identifier
  title: string;        // Display name
  icon?: string;        // Icon name (profile, document, etc.)
  group?: string;       // Grouping label
  interactive?: boolean; // Clickable?
  data?: Record<string, unknown>; // Custom data
}
```

## Backend Conversion

Convert entity tags to model-readable markers:

```python
# thread_item_converter.py
from chatkit.types import UserMessageItem, EntityTag

class EntityAwareConverter:
    """Convert entity tags to XML markers for agent."""

    async def to_agent_input(self, items: list) -> list:
        result = []
        for item in items:
            if isinstance(item, UserMessageItem):
                content = self._process_entities(item)
                result.append({"role": "user", "content": content})
        return result

    def _process_entities(self, item: UserMessageItem) -> str:
        content = item.content
        for entity in item.entities or []:
            marker = self._entity_to_marker(entity)
            content = content.replace(f"@{entity.title}", marker)
        return content

    def _entity_to_marker(self, entity: EntityTag) -> str:
        if entity.data.get("type") == "article":
            return f"<ARTICLE_REFERENCE id='{entity.id}'>{entity.title}</ARTICLE_REFERENCE>"
        if entity.data.get("type") == "user":
            return f"<USER_REFERENCE id='{entity.id}'>{entity.title}</USER_REFERENCE>"
        if entity.data.get("type") == "station":
            return f"<STATION_TAG id='{entity.id}'>{entity.title}</STATION_TAG>"
        return f"<ENTITY id='{entity.id}'>{entity.title}</ENTITY>"
```

## Agent Instructions

Include in system prompt:

```
When you see entity references like <ARTICLE_REFERENCE id='...'>Title</ARTICLE_REFERENCE>,
use the get_article_by_id tool to fetch the full content before answering questions about it.
```

## Preview Widget Types

```typescript
// Simple text preview
{
  preview: {
    type: "Card",
    children: [
      { type: "Text", value: "Title", weight: "bold" },
      { type: "Text", value: "Description", color: "tertiary" },
    ],
  },
}

// Preview with image
{
  preview: {
    type: "Card",
    children: [
      { type: "Image", src: imageUrl, height: 100, fit: "cover" },
      { type: "Text", value: "Title", weight: "bold" },
    ],
  },
}
```

## Common Use Cases

### User Mentions
```typescript
onTagSearch: async (query) => {
  const users = await searchUsers(query);
  return users.map(u => ({
    id: u.id,
    title: u.name,
    icon: "profile",
    group: "Team Members",
    data: { type: "user", email: u.email },
  }));
}
```

### Task References
```typescript
onTagSearch: async (query) => {
  const tasks = await searchTasks(query);
  return tasks.map(t => ({
    id: t.id,
    title: t.title,
    icon: t.completed ? "check" : "dot",
    group: "Tasks",
    data: { type: "task", status: t.status },
  }));
}
```

### Article/Document Mentions
```typescript
onTagSearch: async (query) => {
  const articles = await searchArticles(query);
  return articles.map(a => ({
    id: a.id,
    title: a.title,
    icon: "document",
    group: "Articles",
    data: { type: "article", author: a.author },
  }));
}
```

## Evidence

- `news-guide/frontend/src/components/ChatKitPanel.tsx` - Article/author tagging
- `news-guide/backend/app/thread_item_converter.py` - Entity conversion
- `metro-map/frontend/src/components/ChatKitPanel.tsx:73-117` - Station tagging

```

### references/composer-tools.md

```markdown
# Composer Tools Reference

## Overview

Composer tools add mode-switching buttons in the chat input area, allowing users to change the AI's behavior or focus.

## Frontend Configuration

```typescript
// lib/config.ts
import { ToolOption } from "@openai/chatkit";

export const TOOL_CHOICES: ToolOption[] = [
  {
    id: "event_finder",
    label: "Event finder",
    icon: "calendar",
    placeholderOverride: "Anything happening this weekend?",
    persistent: true,  // Stays selected after sending
  },
  {
    id: "puzzle",
    label: "Coffee break puzzle",
    shortLabel: "Puzzle",  // Shown when selected (compact)
    icon: "atom",
    placeholderOverride: "Give me a puzzle to solve",
    persistent: true,
  },
  {
    id: "search",
    label: "Deep search",
    icon: "search",
    placeholderOverride: "What would you like me to research?",
    persistent: false,  // Resets after sending
  },
];
```

## ChatKit Integration

```typescript
import { useChatKit } from "@openai/chatkit-react";
import { TOOL_CHOICES } from "../lib/config";

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  composer: {
    tools: TOOL_CHOICES,
    placeholder: "Ask me anything...",
  },
});
```

## Backend Handling

The selected tool is passed as `tool_choice` in the request metadata:

```python
# server.py
class NewsChatKitServer(ChatKitServer):
    async def respond(
        self,
        thread: ThreadMetadata,
        item: UserMessageItem | None,
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:
        # Get selected tool from context
        tool_choice = context.get("tool_choice")

        if tool_choice == "event_finder":
            # Use event-finding agent or specialized prompt
            agent = self.event_finder_agent
        elif tool_choice == "puzzle":
            # Use puzzle agent
            agent = self.puzzle_agent
        else:
            # Default agent
            agent = self.default_agent

        result = Runner.run_streamed(agent, input_items, context=agent_context)
        async for event in stream_agent_response(agent_context, result):
            yield event
```

## Tool Option Properties

```typescript
interface ToolOption {
  id: string;           // Unique identifier (sent to backend)
  label: string;        // Full display text
  shortLabel?: string;  // Compact text when selected
  icon?: string;        // Icon name (calendar, search, etc.)
  placeholderOverride?: string;  // Replace composer placeholder when selected
  persistent?: boolean; // Stay selected after message sent (default: false)
}
```

## Common Icons

- `calendar` - Events, scheduling
- `search` - Search, research
- `atom` - Science, puzzles
- `sparkle` - AI, magic
- `globe` - Web, world
- `document` - Documents, files
- `lightbulb` - Ideas, tips

## Use Cases

### Mode Switching
```typescript
const MODES: ToolOption[] = [
  { id: "quick", label: "Quick answer", icon: "zap" },
  { id: "detailed", label: "Detailed analysis", icon: "document" },
];
```

### Feature Selection
```typescript
const FEATURES: ToolOption[] = [
  { id: "code", label: "Write code", icon: "code" },
  { id: "explain", label: "Explain concept", icon: "lightbulb" },
  { id: "review", label: "Review code", icon: "check" },
];
```

### Agent Routing
```typescript
const AGENTS: ToolOption[] = [
  { id: "support", label: "Customer support", icon: "headset" },
  { id: "sales", label: "Sales inquiry", icon: "dollar" },
  { id: "technical", label: "Technical help", icon: "wrench" },
];
```

## Evidence

- `news-guide/frontend/src/lib/config.ts:51-70` - Tool choices definition
- `news-guide/frontend/src/components/ChatKitPanel.tsx` - Composer integration
- `news-guide/backend/app/server.py` - Backend tool_choice routing

```

### references/server-action-handler.py

```python
"""
Complete Server Action Handler Pattern
From: blueprints/openai-chatkit-advanced-samples-main/examples/metro-map/backend/app/server.py

This file shows the complete pattern for handling server-side widget actions in ChatKit.
"""

from __future__ import annotations

from datetime import datetime
from typing import Any, AsyncIterator

from agents import Runner
from chatkit.agents import stream_agent_response
from chatkit.server import ChatKitServer
from chatkit.types import (
    Action,
    AssistantMessageContent,
    AssistantMessageItem,
    Attachment,
    ClientEffectEvent,
    HiddenContextItem,
    ThreadItemDoneEvent,
    ThreadItemReplacedEvent,
    ThreadMetadata,
    ThreadStreamEvent,
    UserMessageItem,
    WidgetItem,
)
from openai.types.responses import ResponseInputContentParam


class ExampleChatKitServer(ChatKitServer[dict[str, Any]]):
    """Example ChatKit server with action handling."""

    def __init__(self) -> None:
        from .memory_store import MemoryStore
        self.store = MemoryStore()
        super().__init__(self.store)

    # =========================================================================
    # REQUIRED: respond() - Handle user messages
    # =========================================================================
    async def respond(
        self,
        thread: ThreadMetadata,
        item: UserMessageItem | None,
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:
        """Handle user message and generate response."""

        # Load conversation history
        items_page = await self.store.load_thread_items(
            thread.id,
            after=None,
            limit=20,
            order="desc",
            context=context,
        )
        items = list(reversed(items_page.data))

        # Convert to agent input format
        input_items = await self.thread_item_converter.to_agent_input(items)

        # Create agent context (available in tool calls)
        agent_context = MyAgentContext(
            thread=thread,
            store=self.store,
            request_context=context,
        )

        # Run agent with streaming
        result = Runner.run_streamed(my_agent, input_items, context=agent_context)

        # Yield events to client
        async for event in stream_agent_response(agent_context, result):
            yield event

    # =========================================================================
    # REQUIRED: action() - Handle widget actions with handler="server"
    # =========================================================================
    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,  # The widget that triggered the action
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:
        """Handle server-side widget actions."""

        # Route by action type
        if action.type == "line.select":
            async for event in self._handle_line_select(
                thread, action.payload, sender, context
            ):
                yield event
            return

        if action.type == "task.approve":
            async for event in self._handle_task_approve(
                thread, action.payload, sender, context
            ):
                yield event
            return

        # Unknown action - do nothing
        return

    # =========================================================================
    # PATTERN: Complete action handler with all response types
    # =========================================================================
    async def _handle_line_select(
        self,
        thread: ThreadMetadata,
        payload: dict[str, Any],
        sender: WidgetItem | None,
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:
        """
        Complete action handler showing all response patterns:
        1. Update the widget that triggered the action
        2. Add hidden context for future agent input
        3. Send assistant message
        4. Trigger client effect for UI update
        """
        line_id = payload["id"]

        # -----------------------------------------------------------------
        # 1. UPDATE WIDGET - Replace with new state (e.g., show selection)
        # -----------------------------------------------------------------
        updated_widget = build_widget(
            items=self.get_items(),
            selected=line_id,  # Now shows as selected
        )

        if sender:
            updated_widget_item = sender.model_copy(update={"widget": updated_widget})
            yield ThreadItemReplacedEvent(item=updated_widget_item)

        # -----------------------------------------------------------------
        # 2. ADD HIDDEN CONTEXT - Agent sees this on next message
        # -----------------------------------------------------------------
        await self.store.add_thread_item(
            thread.id,
            HiddenContextItem(
                id=self.store.generate_item_id("ctx", thread, context),
                thread_id=thread.id,
                created_at=datetime.now(),
                content=f"<LINE_SELECTED>{line_id}</LINE_SELECTED>",
            ),
            context=context,
        )

        # -----------------------------------------------------------------
        # 3. SEND ASSISTANT MESSAGE - Acknowledge the action
        # -----------------------------------------------------------------
        yield ThreadItemDoneEvent(
            item=AssistantMessageItem(
                thread_id=thread.id,
                id=self.store.generate_item_id("msg", thread, context),
                created_at=datetime.now(),
                content=[
                    AssistantMessageContent(
                        text="Would you like to add the station to the beginning or end?"
                    )
                ],
            ),
        )

        # -----------------------------------------------------------------
        # 4. TRIGGER CLIENT EFFECT - Update frontend state
        # -----------------------------------------------------------------
        yield ClientEffectEvent(
            name="location_select_mode",
            data={"lineId": line_id},
        )

    # =========================================================================
    # PATTERN: Simple action - just update widget
    # =========================================================================
    async def _handle_task_approve(
        self,
        thread: ThreadMetadata,
        payload: dict[str, Any],
        sender: WidgetItem | None,
        context: dict[str, Any],
    ) -> AsyncIterator[ThreadStreamEvent]:
        """Simple action that just updates the widget."""
        task_id = payload["task_id"]

        # Update database
        await self.db.approve_task(task_id)

        # Update widget to show approved state
        updated_widget = build_task_widget(
            task=await self.db.get_task(task_id),
            status="approved",
        )

        if sender:
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

    # =========================================================================
    # REQUIRED: Handle attachments (can raise if not supported)
    # =========================================================================
    async def to_message_content(self, _input: Attachment) -> ResponseInputContentParam:
        raise RuntimeError("File attachments are not supported.")


# =========================================================================
# Factory function
# =========================================================================
def create_chatkit_server() -> ExampleChatKitServer | None:
    """Create server instance, return None if dependencies missing."""
    try:
        return ExampleChatKitServer()
    except ImportError:
        return None

```