building-chat-widgets

Original source / Raw SKILL.md

---
name: building-chat-widgets
description: Build interactive AI chat widgets with buttons, forms, and bidirectional actions. Use when creating agentic UIs with clickable widgets, entity tagging (@mentions), composer tools, or server-handled widget actions. Covers full widget lifecycle. NOT when building simple text-only chat without interactive elements.
---

# Building Chat Widgets

Create interactive widgets for AI chat with actions and entity tagging.

## Quick Start

```typescript
const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      if (action.type === "view_details") {
        navigate(`/details/${action.payload.id}`);
      }
    },
  },
});
```

---

## Action Handler Types

| Handler | Defined In | Processed By | Use Case |
|---------|------------|--------------|----------|
| `"client"` | Widget template | Frontend `onAction` | Navigation, local state |
| `"server"` | Widget template | Backend `action()` | Data mutation, widget replacement |

---

## Widget Lifecycle

```
1. Agent tool generates widget → yield WidgetItem
2. Widget renders in chat with action buttons
3. User clicks action → action dispatched
4. Handler processes action:
   - client: onAction callback in frontend
   - server: action() method in ChatKitServer
5. Optional: Widget replaced with updated state
```

---

## Core Patterns

### 1. Widget Templates

Define reusable widget layouts with dynamic data:

```json
{
  "type": "ListView",
  "children": [
    {
      "type": "ListViewItem",
      "key": "item-1",
      "onClickAction": {
        "type": "item.select",
        "handler": "client",
        "payload": { "itemId": "item-1" }
      },
      "children": [
        {
          "type": "Row",
          "gap": 3,
          "children": [
            { "type": "Icon", "name": "check", "color": "success" },
            { "type": "Text", "value": "Item title", "weight": "semibold" }
          ]
        }
      ]
    }
  ]
}
```

### 2. Client-Handled Actions

Actions that update local state, navigate, or send follow-up messages:

**Widget Definition:**

```json
{
  "type": "Button",
  "label": "View Article",
  "onClickAction": {
    "type": "open_article",
    "handler": "client",
    "payload": { "id": "article-123" }
  }
}
```

**Frontend Handler:**

```typescript
const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  widgets: {
    onAction: async (action, widgetItem) => {
      switch (action.type) {
        case "open_article":
          navigate(`/article/${action.payload?.id}`);
          break;

        case "more_suggestions":
          await chatkit.sendUserMessage({ text: "More suggestions, please" });
          break;

        case "select_option":
          setSelectedOption(action.payload?.optionId);
          break;
      }
    },
  },
});
```

### 3. Server-Handled Actions

Actions that mutate data, update widgets, or require backend processing:

**Widget Definition:**

```json
{
  "type": "ListViewItem",
  "onClickAction": {
    "type": "line.select",
    "handler": "server",
    "payload": { "id": "blue-line" }
  }
}
```

**Backend Handler:**

```python
from chatkit.types import (
    Action, WidgetItem, ThreadItemReplacedEvent,
    ThreadItemDoneEvent, AssistantMessageItem, ClientEffectEvent,
)

class MyServer(ChatKitServer[dict]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: RequestContext,  # Note: Already RequestContext, not dict
    ) -> AsyncIterator[ThreadStreamEvent]:

        if action.type == "line.select":
            line_id = action.payload["id"]  # Use .payload, not .arguments

            # 1. Update widget with selection
            updated_widget = build_selector_widget(selected=line_id)
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Stream assistant message
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {line_id}"}],
                )
            )

            # 3. Trigger client effect
            yield ClientEffectEvent(
                name="selection_changed",
                data={"lineId": line_id},
            )
```

### 4. Entity Tagging (@mentions)

Allow users to @mention entities in messages:

```typescript
const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  entities: {
    onTagSearch: async (query: string): Promise<Entity[]> => {
      const results = await fetch(`/api/search?q=${query}`).then(r => r.json());

      return results.map((item) => ({
        id: item.id,
        title: item.name,
        icon: item.type === "person" ? "profile" : "document",
        group: item.type === "People" ? "People" : "Articles",
        interactive: true,
        data: { type: item.type, article_id: item.id },
      }));
    },

    onClick: (entity: Entity) => {
      if (entity.data?.article_id) {
        navigate(`/article/${entity.data.article_id}`);
      }
    },
  },
});
```

### 5. Composer Tools (Mode Selection)

Let users select different AI modes from the composer:

```typescript
const TOOL_CHOICES = [
  {
    id: "general",
    label: "Chat",
    icon: "sparkle",
    placeholderOverride: "Ask anything...",
    pinned: true,
  },
  {
    id: "event_finder",
    label: "Find Events",
    icon: "calendar",
    placeholderOverride: "What events are you looking for?",
    pinned: true,
  },
];

const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },
  composer: {
    placeholder: "What would you like to do?",
    tools: TOOL_CHOICES,
  },
});
```

**Backend Routing:**

```python
async def respond(self, thread, item, context):
    tool_choice = context.metadata.get("tool_choice")

    if tool_choice == "event_finder":
        agent = self.event_finder_agent
    else:
        agent = self.general_agent

    result = Runner.run_streamed(agent, input_items)
    async for event in stream_agent_response(context, result):
        yield event
```

---

## Widget Component Reference

### Layout Components

| Component | Props | Description |
|-----------|-------|-------------|
| `ListView` | `children` | Scrollable list container |
| `ListViewItem` | `key`, `onClickAction`, `children` | Clickable list item |
| `Row` | `gap`, `align`, `justify`, `children` | Horizontal flex |
| `Col` | `gap`, `padding`, `children` | Vertical flex |
| `Box` | `size`, `radius`, `background`, `padding` | Styled container |

### Content Components

| Component | Props | Description |
|-----------|-------|-------------|
| `Text` | `value`, `size`, `weight`, `color` | Text display |
| `Title` | `value`, `size`, `weight` | Heading text |
| `Image` | `src`, `alt`, `width`, `height` | Image display |
| `Icon` | `name`, `size`, `color` | Icon from set |

### Interactive Components

| Component | Props | Description |
|-----------|-------|-------------|
| `Button` | `label`, `variant`, `onClickAction` | Clickable button |

---

## Critical Implementation Details

### Action Object Structure

**IMPORTANT**: Use `action.payload`, NOT `action.arguments`:

```python
# WRONG - Will cause AttributeError
action.arguments

# CORRECT
action.payload
```

### Context Parameter

The `context` parameter is `RequestContext`, not `dict`:

```python
# WRONG - Tries to wrap RequestContext
request_context = RequestContext(metadata=context)

# CORRECT - Use directly
user_id = context.user_id
```

### UserMessageItem Required Fields

When creating synthetic user messages:

```python
from chatkit.types import UserMessageItem, UserMessageTextContent

# Include ALL required fields
synthetic_message = UserMessageItem(
    id=self.store.generate_item_id("message", thread, context),
    thread_id=thread.id,
    created_at=datetime.now(),
    content=[UserMessageTextContent(type="input_text", text=message_text)],
    inference_options={},
)
```

---

## Anti-Patterns

1. **Mixing handlers** - Don't handle same action in both client and server
2. **Missing payload** - Always include data in action payload
3. **Using action.arguments** - Use `action.payload`
4. **Wrapping RequestContext** - Context is already RequestContext
5. **Missing UserMessageItem fields** - Include id, thread_id, created_at
6. **Wrong content type** - Use `type="input_text"` for user messages

---

## Verification

Run: `python3 scripts/verify.py`

Expected: `✓ building-chat-widgets skill ready`

## If Verification Fails

1. Check: references/ folder has widget-patterns.md
2. **Stop and report** if still failing

## References

- [references/widget-patterns.md](references/widget-patterns.md) - Complete widget patterns
- [references/server-action-handler.md](references/server-action-handler.md) - Backend action handling


---

## Referenced Files

> The following files are referenced in this skill and included for context.

### references/widget-patterns.md

```markdown
# Widget Patterns Reference

Complete patterns for ChatKit widget implementation.

## Widget Template Format

```json
{
  "version": "1.0",
  "name": "task_list",
  "template": "{\"type\":\"ListView\",\"children\":[...jinja template...]}",
  "jsonSchema": {
    "type": "object",
    "properties": {
      "tasks": { "type": "array", "items": {...} }
    }
  }
}
```

## Loading Templates in Python

```python
from chatkit.widgets import WidgetTemplate, WidgetRoot

# Load template from file
task_list_template = WidgetTemplate.from_file("task_list.widget")

def build_task_list_widget(tasks: list[Task]) -> WidgetRoot:
    return task_list_template.build(
        data={
            "tasks": [task.model_dump() for task in tasks],
            "selected": None,
        }
    )
```

## Client-to-Server Action Forwarding

When client handles action locally then notifies server:

```typescript
widgets: {
  onAction: async (action, widgetItem) => {
    if (action.type === "select_name") {
      // Forward to server for processing
      await chatkit.sendCustomAction(action, widgetItem.id);

      // Refresh local state after server processes
      const data = await refreshStatus();
      if (data) handleStatusUpdate(data);
    }
  },
}
```

## Thread Item Actions (Feedback/Retry/Share)

```typescript
const chatkit = useChatKit({
  api: { url: API_URL, domainKey: DOMAIN_KEY },

  threadItemActions: {
    feedback: true,   // Thumbs up/down
    retry: true,      // Regenerate response
    share: true,      // Share message
  },

  onLog: ({ name, data }) => {
    if (name === "message.feedback") {
      trackFeedback(data);
    }
  },
});
```

## Widget Streaming from Tools

When agent tool generates a widget:

```python
from chatkit.types import WidgetItem, ThreadItemDoneEvent
from agents import function_tool

@function_tool
async def show_article_list(ctx: AgentContext, query: str) -> str:
    """Show a list of articles matching the query."""

    articles = await article_store.search(query)

    # Build widget
    widget = build_article_list_widget(articles)

    # Create widget item
    widget_item = WidgetItem(
        id=ctx.store.generate_item_id("widget", ctx.thread, ctx.request_context),
        thread_id=ctx.thread.id,
        created_at=datetime.now(),
        widget=widget,
    )

    # Save to store
    await ctx.store.add_thread_item(ctx.thread.id, widget_item, ctx.request_context)

    # Yield as event
    yield ThreadItemDoneEvent(item=widget_item)

    return f"Showing {len(articles)} articles"
```

## Local Tool Wrappers for Widget Streaming

**Problem**: MCP tools alone don't stream widgets - need local wrappers.

```python
from agents import function_tool

@function_tool
async def show_task_form(
    ctx: RunContextWrapper[AgentContext],
) -> str:
    """Show interactive task creation form widget."""

    agent_ctx = ctx.context
    mcp_url = agent_ctx.mcp_server_url

    # Call MCP tool via HTTP
    result = await _call_mcp_tool(
        mcp_url,
        "show_task_form",
        arguments={"params": {"user_id": agent_ctx.user_id}},
        access_token=agent_ctx.access_token,
    )

    # Return result - RunHooks will stream widget
    return json.dumps(result)

# In RunHooks.on_tool_end()
async def on_tool_end(self, output: str | None, tool_name: str) -> None:
    if tool_name == "show_task_form":
        result = json.loads(output)
        if result.get("action") == "show_form":
            widget = build_task_form_widget()
            yield WidgetItem(...)
```

## Entity Conversion in Backend

Convert @mentions to model-readable markers:

```python
class EntityAwareConverter(BasicThreadItemConverter):
    async def to_agent_input(self, items: list[ThreadItem]) -> list:
        result = []
        for item in items:
            if isinstance(item, UserMessageItem):
                content = item.content
                for entity in item.entities or []:
                    if entity.type == "article":
                        content = content.replace(
                            f"@{entity.title}",
                            f"<ARTICLE id='{entity.id}'>{entity.title}</ARTICLE>"
                        )
                result.append({"role": "user", "content": content})
        return result
```

## Common Validation Errors

### Error 1: 'Action' object has no attribute 'arguments'

```
AttributeError: 'Action[str, Any]' object has no attribute 'arguments'
```

**Fix**: Use `action.payload` instead of `action.arguments`

### Error 2: UserMessageTextContent type mismatch

```
ValidationError: Input should be 'input_text' [type=literal_error, input_value='text']
```

**Fix**: Use `type="input_text"` for user input, not `type="text"`

### Error 3: UserMessageItem missing required fields

```
4 validation errors for UserMessageItem
- id: Field required
- thread_id: Field required
- created_at: Field required
- inference_options: Field required
```

**Fix**: Include all required fields when creating UserMessageItem

### Error 4: RequestContext wrapping issue

```
2 validation errors for RequestContext
metadata: Input should be a valid dictionary [input_value=RequestContext(...)]
```

**Fix**: Don't wrap `context` - it's already a RequestContext object

## Widget Action Testing Checklist

- [ ] Widget renders with correct data
- [ ] All buttons have clear labels
- [ ] Client actions navigate/update UI correctly
- [ ] Server actions call backend successfully
- [ ] Action payload contains all required data
- [ ] Widget updates after server action completes
- [ ] No AttributeError on action.payload access
- [ ] No ValidationError on UserMessageItem creation
- [ ] Local tool wrappers trigger widget streaming

## Evidence Sources

Patterns derived from:
- `cat-lounge/backend/app/widgets/cat_name_suggestions.widget`
- `cat-lounge/frontend/src/components/ChatKitPanel.tsx`
- `metro-map/backend/app/server.py`
- `metro-map/frontend/src/components/ChatKitPanel.tsx`
- `news-guide/frontend/src/components/ChatKitPanel.tsx`
- `news-guide/backend/app/agents/news_agent.py`

```

### references/server-action-handler.md

```markdown
# Server Action Handler Reference

Complete backend pattern for handling widget actions.

## Full Action Handler Implementation

```python
from chatkit.server import ChatKitServer
from chatkit.types import (
    Action,
    WidgetItem,
    ThreadMetadata,
    ThreadItemReplacedEvent,
    ThreadItemDoneEvent,
    AssistantMessageItem,
    HiddenContextItem,
    ClientEffectEvent,
    UserMessageItem,
    UserMessageTextContent,
)
from datetime import datetime
from typing import Any, AsyncIterator

class MyServer(ChatKitServer[RequestContext]):

    async def action(
        self,
        thread: ThreadMetadata,
        action: Action[str, Any],
        sender: WidgetItem | None,
        context: RequestContext,  # Already RequestContext, not dict!
    ) -> AsyncIterator[ThreadStreamEvent]:
        """Handle widget actions.

        CRITICAL NOTES:
        - context is RequestContext, not dict (type hint is misleading)
        - Use action.payload, not action.arguments
        - Include all required fields in UserMessageItem
        """

        if action.type == "item.select":
            item_id = action.payload["id"]

            # 1. Update widget with selection state
            updated_widget = build_selector_widget(
                items=self.items,
                selected=item_id,
            )
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": updated_widget})
            )

            # 2. Add hidden context for future agent input
            await self.store.add_thread_item(
                thread.id,
                HiddenContextItem(
                    id=self.store.generate_item_id("ctx", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=f"<SELECTED>{item_id}</SELECTED>",
                ),
                context=context,
            )

            # 3. Stream assistant response
            yield ThreadItemDoneEvent(
                item=AssistantMessageItem(
                    id=self.store.generate_item_id("msg", thread, context),
                    thread_id=thread.id,
                    created_at=datetime.now(),
                    content=[{"text": f"Selected {item_id}. What next?"}],
                )
            )

            # 4. Trigger client effect
            yield ClientEffectEvent(
                name="selection_mode",
                data={"itemId": item_id},
            )

        elif action.type == "form.submit":
            form_data = action.payload

            # Process form submission
            result = await process_form(form_data)

            # Create synthetic user message to trigger agent
            synthetic_message = UserMessageItem(
                id=self.store.generate_item_id("message", thread, context),
                thread_id=thread.id,
                created_at=datetime.now(),
                content=[
                    UserMessageTextContent(
                        type="input_text",  # Must be "input_text", not "text"
                        text=f"Form submitted: {result.summary}"
                    )
                ],
                inference_options={},  # Required field
            )

            # Run agent with synthetic message
            async for event in self.respond(thread, synthetic_message, context):
                yield event

        elif action.type == "confirm.accept":
            # Handle confirmation action
            item_id = action.payload["item_id"]

            # Update database
            await self.db.confirm_item(item_id)

            # Replace widget with success state
            success_widget = build_success_widget(item_id)
            yield ThreadItemReplacedEvent(
                item=sender.model_copy(update={"widget": success_widget})
            )

            # Notify client
            yield ClientEffectEvent(
                name="item_confirmed",
                data={"itemId": item_id},
            )
```

## Action Type Definition

```python
from chatkit.types import Action

# Action[str, Any] has these fields:
action.type      # str - action identifier (e.g., "task.start")
action.payload   # dict[str, Any] - action data
action.handler   # "client" | "server" - where processed
```

## Common Patterns

### Selection Actions

```python
if action.type.endswith(".select"):
    entity_id = action.payload["id"]

    # Update widget
    yield ThreadItemReplacedEvent(
        item=sender.model_copy(update={
            "widget": build_widget(selected=entity_id)
        })
    )
```

### Form Actions

```python
if action.type == "form.submit":
    # Create synthetic message
    message = UserMessageItem(
        id=self.store.generate_item_id("message", thread, context),
        thread_id=thread.id,
        created_at=datetime.now(),
        content=[UserMessageTextContent(type="input_text", text="...")],
        inference_options={},
    )

    # Run agent
    async for event in self.respond(thread, message, context):
        yield event
```

### Confirmation Actions

```python
if action.type == "confirm":
    # Perform action
    await self.do_action(action.payload)

    # Replace widget
    yield ThreadItemReplacedEvent(
        item=sender.model_copy(update={
            "widget": build_success_widget()
        })
    )
```

## Error Handling

```python
async def action(self, thread, action, sender, context):
    try:
        if action.type == "dangerous.action":
            # Validate
            if not self.can_perform(action, context):
                yield ThreadItemDoneEvent(
                    item=AssistantMessageItem(
                        id=self.store.generate_item_id("msg", thread, context),
                        thread_id=thread.id,
                        created_at=datetime.now(),
                        content=[{"text": "Permission denied."}],
                    )
                )
                return

            # Proceed
            ...

    except Exception as e:
        # Log error
        logger.error(f"Action failed: {e}")

        # Notify user
        yield ThreadItemDoneEvent(
            item=AssistantMessageItem(
                id=self.store.generate_item_id("msg", thread, context),
                thread_id=thread.id,
                created_at=datetime.now(),
                content=[{"text": f"Action failed: {str(e)}"}],
            )
        )
```

```

### scripts/verify.py

```python
#!/usr/bin/env python3
"""Verify building-chat-widgets skill has required references."""
import os
import sys

def main():
    skill_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
    refs_dir = os.path.join(skill_dir, "references")

    required = ["widget-patterns.md", "server-action-handler.md"]
    missing = [r for r in required if not os.path.isfile(os.path.join(refs_dir, r))]

    if not missing:
        print("✓ building-chat-widgets skill ready")
        sys.exit(0)
    else:
        print(f"✗ Missing: {', '.join(missing)}")
        sys.exit(1)

if __name__ == "__main__":
    main()

```