local-deep-research

Original source / Raw SKILL.md

---
name: local-deep-research
description: Multi-cycle deep research using locally-hosted LDR (Local Deep Research) service. Use when user asks for comprehensive research with citations, literature reviews, competitive intelligence, or any research requiring exhaustive web search with iterative question generation. Triggers on: "deep research", "research this topic", "comprehensive analysis with sources", "literature review", "investigate [topic]", "quick summary on [topic]", "detailed report on [topic]", "research in Spanish/French/etc", or when academic-deep-research is requested but using local LDR instance.
version: 1.0.2
homepage: https://github.com/eplt/local-deep-research-skill
metadata:
  openclaw:
    emoji: "🔬"
    requires:
      bins:
        - curl
        - jq
      env:
        - LDR_BASE_URL
        - LDR_SERVICE_USER
        - LDR_SERVICE_PASSWORD
    primaryEnv: LDR_BASE_URL
    files:
      - "scripts/*"
      - "references/*"
---

# Local Deep Research Skill

This skill interfaces with a locally-hosted LDR (Local Deep Research) service to perform multi-cycle, iterative research with full citations and source tracking.

## What to consider before installing

- **LDR service**: The script talks only to the URL in `LDR_BASE_URL` (default `http://127.0.0.1:5000`). Only point it at an LDR instance you control. Do not set it to an unknown or untrusted remote host.
- **Required binaries**: Ensure `curl` and `jq` are installed on the host where the skill runs.
- **Credentials**: If your LDR instance requires login, set `LDR_SERVICE_USER` and `LDR_SERVICE_PASSWORD` (or `LDR_USERNAME`/`LDR_PASSWORD`) via environment variables or a local `.env` file only. Use a dedicated, low-privilege LDR account (e.g. `openclaw_service`). Do not store secrets in committed config or in the skill directory.
- **Sourced .env**: The script optionally sources `~/.config/local_deep_research/config/.env` if that file exists. That file may expose any variables it contains to the script. Verify the contents of that path before use; do not place unrelated secrets there.
- **Review the script**: The script performs form-based session+CSRF login and uses an ephemeral cookie jar. It does not send data to any endpoint other than the configured LDR service. You can review `scripts/ldr-research.sh` before use. For higher assurance, run it in an isolated environment (e.g. container or VM) with network restricted to your LDR host.

## Configuration

### Credentials (local-only, never transmitted)

LDR uses **session-cookie auth with CSRF protection** (not HTTP Basic Auth). The skill script performs a proper login flow: GET login page → obtain session cookie and CSRF token → POST credentials + CSRF → reuse session cookie (and CSRF for POSTs) for all API calls. Username and password are used **only** to create a session with your **local** LDR instance; they are never sent to ClawHub, GitHub, or any other server.

**Do not** put credentials in skill config or committed files. Use **environment variables or a local `.env` file** only (e.g. `LDR_SERVICE_USER`, `LDR_SERVICE_PASSWORD`, or `LDR_USERNAME`/`LDR_PASSWORD`). Optional: LDR’s `~/.config/local_deep_research/config/.env` is sourced by the script if present. Use a dedicated LDR user (e.g. `openclaw_service`) for this skill.

### All configuration options

- `LDR_BASE_URL` — LDR service URL (default: `http://127.0.0.1:5000`)
- `LDR_LOGIN_URL` — Login page URL for session + CSRF (default: `$LDR_BASE_URL/auth/login`)
- `LDR_SERVICE_USER` or `LDR_USERNAME` — LDR account username (local auth only)
- `LDR_SERVICE_PASSWORD` or `LDR_PASSWORD` — LDR account password (local auth only)
- `LDR_DEFAULT_MODE` — Default research mode: `quick` (Quick Summary) or `detailed` (Detailed Report) (default: `detailed`)
- `LDR_DEFAULT_LANGUAGE` — Default output language code for report/summary (e.g. `en`, `es`, `fr`, `de`, `zh`, `ja`); empty = LDR default
- `LDR_DEFAULT_SEARCH_TOOL` — Default search tool: `searxng`, `auto`, `local_all` (default: `auto`)

## Research modes (Quick Summary vs Detailed Report)

- **`quick`** — **Quick Summary**: fewer cycles, shorter output, faster. Use when the user wants a concise summary or a quick overview.
- **`detailed`** — **Detailed Report**: full multi-cycle research, full markdown report, full citations and sources. Use when the user wants comprehensive analysis, literature review, or in-depth coverage.

## Actions

### start_research

Fire-and-forget: submit a query to LDR and return a research ID immediately.

**Inputs:**
- `query` (required) — The research question or topic
- `mode` (optional) — `quick` (Quick Summary) or `detailed` (Detailed Report) (default from config)
- `language` (optional) — Output language for the report/summary, e.g. `en`, `es`, `fr`, `de`, `zh`, `ja` (default from config or LDR default)
- `search_tool` (optional) — `searxng`, `auto`, `local_all` (default from config)
- `iterations` (optional) — Number of research cycles (default: LDR's default)
- `questions_per_iteration` (optional) — Questions to generate per cycle

**Returns:**
```json
{
  "research_id": "uuid-string",
  "mode": "detailed",
  "search_tool": "auto",
  "submitted_at": "2026-03-10T08:00:00Z",
  "status": "queued"
}
```

**Usage:**
```bash
# Quick Summary (faster, shorter)
scripts/ldr-research.sh start_research --query "Solid-state battery advances" --mode quick

# Detailed Report with output in Spanish
scripts/ldr-research.sh start_research \
  --query "What are the latest developments in solid-state batteries?" \
  --mode detailed \
  --language es \
  --search_tool searxng
```

### get_status

Check the status of a research job.

**Inputs:**
- `research_id` (required) — The research job ID from start_research

**Returns:**
```json
{
  "research_id": "uuid-string",
  "state": "pending|running|completed|failed|timeout",
  "progress": 45,
  "message": "Synthesizing sources from iteration 2...",
  "last_milestone": "Generated 12 questions from 8 sources"
}
```

**Usage:**
```bash
scripts/ldr-research.sh get_status --research_id <uuid>
```

### get_result

Fetch the complete research report once finished.

**Inputs:**
- `research_id` (required) — The research job ID

**Returns:**
```json
{
  "research_id": "uuid-string",
  "query": "original query",
  "mode": "detailed",
  "summary": "executive summary text",
  "report_markdown": "full markdown report",
  "sources": [
    {
      "id": 1,
      "title": "Source Title",
      "url": "https://example.com",
      "snippet": "relevant excerpt",
      "type": "web|local_doc"
    }
  ],
  "iterations": 3,
  "created_at": "2026-03-10T08:00:00Z",
  "completed_at": "2026-03-10T08:15:00Z"
}
```

**Usage:**
```bash
scripts/ldr-research.sh get_result --research_id <uuid>
```

## Orchestration Pattern

### One-shot (wait for completion)

For interactive sessions where the user can wait:

1. Call `start_research`
2. Poll `get_status` every 10-30 seconds
3. When `state == "completed"`, call `get_result`
4. Present the report to the user

### Async (fire-and-forget with follow-up)

For background processing:

1. Call `start_research`, return the `research_id` to the user
2. User can check status later with `get_status --research_id <id>`
3. When ready, call `get_result` to fetch the complete report

### Chained workflows

After research completes:

1. Call `get_result` to get sources
2. Pass sources to other skills (e.g., `markdown-converter`, `summarize`)
3. Build RAG indexes or knowledge bases from the sources

## Error Handling

### start_research failures

- **HTTP/network errors** — Retry with exponential backoff (3 attempts)
- **LDR validation errors** — Return error to user (bad query, invalid params)
- **Auth failures** — Check credentials, return clear error

### get_status / get_result failures

- **Temporarily unavailable** — Retry 2-3 times before surfacing error
- **Research not found** — Return "unknown research_id" error
- **Timeout** — Return state with timeout reason

## Timeouts

- **Per HTTP request** — 30-60 seconds (configurable)
- **Total research duration** — No client-side limit (LDR manages this)
- **Status polling interval** — 10-30 seconds recommended

## Example Session

```
User: "Research the latest developments in quantum computing"

Assistant: Starting deep research with LDR...
→ start_research(query="latest developments in quantum computing", mode="detailed")
→ Returns: research_id="abc-123", status="queued"

Assistant: Research started (ID: abc-123). This will take ~5-10 minutes.
I'll check the progress and let you know when it's complete.

[After polling...]

Assistant: Research complete! Here's what I found:

## Summary
[summary from get_result]

## Full Report
[report_markdown from get_result]

## Sources (12 found)
1. [Source 1 title](url)
2. [Source 2 title](url)
...
```

## Related Skills

- **academic-deep-research** — Alternative for academic-focused research with APA 7th citations
- **deep-research-pro** — Web-based deep research (no local LDR required)
- **tavily** / **searxng** — Simple web search for quick lookups
- **summarize** — Process LDR output for additional summarization

## Troubleshooting

### LDR service not responding

1. Check `LDR_BASE_URL` is correct
2. Verify LDR service is running: `curl http://127.0.0.1:5000/health`
3. Check LDR logs for errors

### Authentication failures

1. Ensure credentials are set via **env or local .env only** (e.g. `LDR_SERVICE_USER`, `LDR_SERVICE_PASSWORD`), not in committed config.
2. LDR uses **session + CSRF** (not Basic Auth). The script GETs the login page, extracts the CSRF token, then POSTs the login form. If LDR uses a different login path or field names, set `LDR_LOGIN_URL` or see the script’s login section.
3. Test: run the script with credentials set and check for "Login successful"; or open `LDR_LOGIN_URL` in a browser and sign in there to verify LDR is up.

### Research stuck in "running" state

1. Check LDR service health
2. Review LDR logs for stuck jobs
3. Consider timeout and restart if >30 minutes with no progress


---

## Referenced Files

> The following files are referenced in this skill and included for context.

### scripts/ldr-research.sh

```bash
#!/bin/bash
#
# LDR Research CLI - Interface with Local Deep Research service
#
# REQUIRED BINARIES: curl, jq (declared in skill manifest requires.bins)
# EXPECTED ENV (declared in skill manifest requires.env / primaryEnv):
#   LDR_BASE_URL    - LDR service URL (default http://127.0.0.1:5000); only use a host you control
#   LDR_SERVICE_USER, LDR_SERVICE_PASSWORD (or LDR_USERNAME/LDR_PASSWORD) - when LDR auth is enabled
#
# Actions: start_research, get_status, get_result, poll_until_complete
#
# Authentication: LDR uses session-cookie auth with CSRF. This script:
#   1. GETs the login page to obtain session cookie and CSRF token
#   2. POSTs login form (username, password, CSRF) to establish session
#   3. Reuses the session cookie (and CSRF token for POSTs) for API calls
# Credentials are for local LDR only; never transmitted elsewhere.
#
# Optional: ~/.config/local_deep_research/config/.env is sourced if present (verify its contents).
#

set -e

# Load LDR's local .env if present (secrets stay local)
LDR_ENV="${LDR_CONFIG_DIR:-$HOME/.config/local_deep_research/config}/.env"
if [[ -f "$LDR_ENV" ]]; then
    set -a
    # shellcheck source=/dev/null
    source "$LDR_ENV"
    set +a
fi

# Config
LDR_BASE_URL="${LDR_BASE_URL:-http://127.0.0.1:5000}"
LDR_USERNAME="${LDR_SERVICE_USER:-${LDR_USERNAME:-}}"
LDR_PASSWORD="${LDR_SERVICE_PASSWORD:-${LDR_PASSWORD:-}}"
LDR_DEFAULT_MODE="${LDR_DEFAULT_MODE:-detailed}"
LDR_DEFAULT_SEARCH_TOOL="${LDR_DEFAULT_SEARCH_TOOL:-auto}"
LDR_DEFAULT_LANGUAGE="${LDR_DEFAULT_LANGUAGE:-}"
# Login page URL (GET for form + CSRF; POST for submit). Adjust if LDR uses different path.
LDR_LOGIN_URL="${LDR_LOGIN_URL:-${LDR_BASE_URL}/auth/login}"

HTTP_TIMEOUT="${HTTP_TIMEOUT:-60}"
RETRY_COUNT="${RETRY_COUNT:-3}"
RETRY_DELAY="${RETRY_DELAY:-2}"

# Cookie jar for session; CSRF token stored after login
COOKIE_JAR=$(mktemp)
trap 'rm -f "$COOKIE_JAR"' EXIT
CSRF_TOKEN=""

RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
log_info() { echo -e "${GREEN}[INFO]${NC} $1" >&2; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1" >&2; }
log_error() { echo -e "${RED}[ERROR]${NC} $1" >&2; }

# Fetch login page to get session cookie and CSRF token; then POST credentials.
# LDR uses form-based login + CSRF (no HTTP Basic Auth).
login() {
    if [[ -z "$LDR_USERNAME" || -z "$LDR_PASSWORD" ]]; then
        return 0
    fi

    log_info "Logging in to LDR (session + CSRF)..."
    local login_page
    login_page=$(curl -s -c "$COOKIE_JAR" -b "$COOKIE_JAR" -L --max-time "$HTTP_TIMEOUT" "$LDR_LOGIN_URL" 2>/dev/null) || true
    if [[ -z "$login_page" ]]; then
        log_error "Could not fetch login page at $LDR_LOGIN_URL"
        return 1
    fi

    # Extract CSRF token from form (common: name="csrf_token" or "_csrf" or "csrf_token")
    CSRF_TOKEN=$(echo "$login_page" | sed -n 's/.*name="csrf_token"[^>]*value="\([^"]*\)".*/\1/p' | head -1)
    [[ -z "$CSRF_TOKEN" ]] && CSRF_TOKEN=$(echo "$login_page" | sed -n 's/.*name="_csrf"[^>]*value="\([^"]*\)".*/\1/p' | head -1)
    [[ -z "$CSRF_TOKEN" ]] && CSRF_TOKEN=$(echo "$login_page" | sed -n 's/.*value="\([^"]*\)"[^>]*name="csrf_token".*/\1/p' | head -1)
    if [[ -z "$CSRF_TOKEN" ]]; then
        log_warn "No CSRF token found on login page; POST may still work if LDR uses cookie-only CSRF"
    fi

    # POST login form (application/x-www-form-urlencoded)
    local post_data
    post_data="username=$(printf '%s' "$LDR_USERNAME" | jq -sRr @uri)&password=$(printf '%s' "$LDR_PASSWORD" | jq -sRr @uri)"
    [[ -n "$CSRF_TOKEN" ]] && post_data="csrf_token=$(printf '%s' "$CSRF_TOKEN" | jq -sRr @uri)&$post_data"

    local http_code
    http_code=$(curl -s -o /dev/null -w "%{http_code}" -c "$COOKIE_JAR" -b "$COOKIE_JAR" -L --max-time "$HTTP_TIMEOUT" \
        -X POST \
        -H "Content-Type: application/x-www-form-urlencoded" \
        -d "$post_data" \
        "$LDR_LOGIN_URL" 2>/dev/null) || true

    if [[ "$http_code" =~ ^2[0-9][0-9]$ ]] || [[ "$http_code" == "302" ]]; then
        log_info "Login successful (HTTP $http_code)"
        return 0
    fi
    log_error "Login failed (HTTP $http_code). Check URL and credentials."
    return 1
}

# Make authenticated request. Uses session cookie; for POST, sends CSRF if we have it.
http_request() {
    local method="$1"
    local endpoint="$2"
    local data="$3"
    local url="${LDR_BASE_URL}${endpoint}"

    local attempt=0
    while [[ $attempt -lt $RETRY_COUNT ]]; do
        local response http_code
        if [[ "$method" == "GET" ]]; then
            response=$(curl -s -w "\n%{http_code}" -b "$COOKIE_JAR" --max-time "$HTTP_TIMEOUT" "$url" 2>/dev/null)
        else
            if [[ -n "$CSRF_TOKEN" ]]; then
                response=$(curl -s -w "\n%{http_code}" -b "$COOKIE_JAR" -H "X-CSRFToken: $CSRF_TOKEN" --max-time "$HTTP_TIMEOUT" \
                    -H "Content-Type: application/json" -d "$data" -X POST "$url" 2>/dev/null)
            else
                response=$(curl -s -w "\n%{http_code}" -b "$COOKIE_JAR" --max-time "$HTTP_TIMEOUT" \
                    -H "Content-Type: application/json" -d "$data" -X POST "$url" 2>/dev/null)
            fi
        fi

        http_code=$(echo "$response" | tail -n1)
        local body=$(echo "$response" | sed '$d')

        if [[ "$http_code" == "302" ]] || [[ "$http_code" == "401" ]]; then
            log_warn "Session expired or unauthorized (HTTP $http_code), re-login..."
            login || { log_error "Re-login failed"; return 1; }
            attempt=$((attempt + 1))
            continue
        fi

        if [[ "$http_code" =~ ^2[0-9][0-9]$ ]]; then
            echo "$body"
            return 0
        fi
        if [[ "$http_code" =~ ^5[0-9][0-9]$ ]]; then
            attempt=$((attempt + 1))
            if [[ $attempt -lt $RETRY_COUNT ]]; then
                log_warn "Server error (HTTP $http_code), retrying in ${RETRY_DELAY}s..."
                sleep "$RETRY_DELAY"
                continue
            fi
        fi

        log_error "HTTP request failed with status $http_code"
        echo "$body"
        return 1
    done
    return 1
}

start_research() {
    local query="" mode="$LDR_DEFAULT_MODE" search_tool="$LDR_DEFAULT_SEARCH_TOOL" language="$LDR_DEFAULT_LANGUAGE" iterations="" questions_per_iteration=""
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --query) query="$2"; shift 2 ;;
            --mode) mode="$2"; shift 2 ;;
            --search_tool) search_tool="$2"; shift 2 ;;
            --language) language="$2"; shift 2 ;;
            --iterations) iterations="$2"; shift 2 ;;
            --questions_per_iteration) questions_per_iteration="$2"; shift 2 ;;
            *) log_error "Unknown argument: $1"; exit 1 ;;
        esac
    done

    if [[ -z "$query" ]]; then
        log_error "Missing required argument: --query"
        exit 1
    fi

    login || { log_error "Authentication failed"; exit 1; }

    local payload="{\"query\": \"$(echo "$query" | jq -sRr .)\", \"mode\": \"$mode\", \"search_tool\": \"$search_tool\""
    [[ -n "$language" ]] && payload="$payload, \"language\": \"$(echo "$language" | jq -sRr .)\""
    [[ -n "$iterations" ]] && payload="$payload, \"iterations\": $iterations"
    [[ -n "$questions_per_iteration" ]] && payload="$payload, \"questions_per_iteration\": $questions_per_iteration"
    payload="$payload}"

    log_info "Starting research: $query (mode: $mode, search_tool: $search_tool${language:+, language: $language})"
    local response
    response=$(http_request "POST" "/research/api/start" "$payload")
    if [[ $? -eq 0 ]]; then
        log_info "Research started successfully"
        echo "$response" | jq .
    else
        log_error "Failed to start research"
        echo "$response"
        exit 1
    fi
}

get_status() {
    local research_id=""
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --research_id) research_id="$2"; shift 2 ;;
            *) log_error "Unknown argument: $1"; exit 1 ;;
        esac
    done

    if [[ -z "$research_id" ]]; then
        log_error "Missing required argument: --research_id"
        exit 1
    fi

    log_info "Checking status for research: $research_id"
    login || true
    local response
    response=$(http_request "GET" "/research/api/status/${research_id}" "")
    if [[ $? -eq 0 ]]; then
        echo "$response" | jq .
    else
        log_error "Failed to get status"
        echo "$response"
        exit 1
    fi
}

get_result() {
    local research_id=""
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --research_id) research_id="$2"; shift 2 ;;
            *) log_error "Unknown argument: $1"; exit 1 ;;
        esac
    done

    if [[ -z "$research_id" ]]; then
        log_error "Missing required argument: --research_id"
        exit 1
    fi

    log_info "Fetching result for research: $research_id"
    login || true
    local response
    response=$(http_request "GET" "/research/api/report/${research_id}" "")
    if [[ $? -eq 0 ]]; then
        echo "$response" | jq .
    else
        log_error "Failed to get result"
        echo "$response"
        exit 1
    fi
}

poll_until_complete() {
    local research_id="" interval="${INTERVAL:-15}" max_wait="${MAX_WAIT:-1800}"
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --research_id) research_id="$2"; shift 2 ;;
            --interval) interval="$2"; shift 2 ;;
            --max_wait) max_wait="$2"; shift 2 ;;
            *) log_error "Unknown argument: $1"; exit 1 ;;
        esac
    done

    if [[ -z "$research_id" ]]; then
        log_error "Missing required argument: --research_id"
        exit 1
    fi

    log_info "Polling research $research_id (interval: ${interval}s, max_wait: ${max_wait}s)"
    login || true
    local elapsed=0
    while [[ $elapsed -lt $max_wait ]]; do
        local status_response
        status_response=$(http_request "GET" "/research/api/status/${research_id}" "" 2>/dev/null)
        [[ $? -ne 0 ]] && { log_error "Failed to get status"; exit 1; }

        local state message progress
        state=$(echo "$status_response" | jq -r '.state // "unknown"')
        message=$(echo "$status_response" | jq -r '.message // ""')
        progress=$(echo "$status_response" | jq -r '.progress // "N/A"')

        log_info "Status: $state - $message (Progress: $progress%)"

        case "$state" in
            completed) log_info "Research completed!"; echo "$status_response"; return 0 ;;
            failed|timeout) log_error "Research $state"; echo "$status_response"; return 1 ;;
            pending|running) sleep "$interval"; elapsed=$((elapsed + interval)) ;;
            *) log_warn "Unknown state: $state"; sleep "$interval"; elapsed=$((elapsed + interval)) ;;
        esac
    done

    log_error "Max wait time ($max_wait seconds) exceeded"
    echo '{"state": "timeout", "message": "Max wait time exceeded"}' | jq .
    return 1
}

show_usage() {
    cat << EOF
LDR Research CLI - Local Deep Research service (session + CSRF auth)

Usage: $0 <action> [options]

Actions:
  start_research      Start a new research job
  get_status          Check status of a research job
  get_result          Fetch completed research report
  poll_until_complete Poll until research completes (blocking)

Options for start_research:
  --query <text>              Research query (required)
  --mode <quick|detailed>     quick = Quick Summary, detailed = Detailed Report (default: $LDR_DEFAULT_MODE)
  --search_tool <tool>        Search tool (default: $LDR_DEFAULT_SEARCH_TOOL)
  --language <code>           Output language, e.g. en, es, fr, de, zh, ja (default: $LDR_DEFAULT_LANGUAGE)
  --iterations <n>            Number of research cycles
  --questions_per_iteration <n> Questions per cycle

Options for get_status/get_result:
  --research_id <uuid>       Research job ID (required)

Options for poll_until_complete:
  --research_id <uuid>       Research job ID (required)
  --interval <seconds>        Polling interval (default: 15)
  --max_wait <seconds>       Maximum wait time (default: 1800)

Environment (credentials for local LDR only; session cookie + CSRF):
  LDR_BASE_URL              LDR service URL (default: http://127.0.0.1:5000)
  LDR_LOGIN_URL             Login page URL (default: \$LDR_BASE_URL/auth/login)
  LDR_SERVICE_USER          LDR account username (or LDR_USERNAME)
  LDR_SERVICE_PASSWORD      LDR account password (or LDR_PASSWORD)
  LDR_CONFIG_DIR            Dir containing .env (default: ~/.config/local_deep_research/config)
  LDR_DEFAULT_MODE          Default mode: quick | detailed
  LDR_DEFAULT_LANGUAGE       Default output language code (e.g. en, es, fr)

Examples:
  $0 start_research --query "quantum computing" --mode quick
  $0 start_research --query "AI trends" --mode detailed --language es
  $0 get_status --research_id abc-123
  $0 poll_until_complete --research_id abc-123 --interval 10
EOF
}

main() {
    [[ $# -lt 1 ]] && { show_usage; exit 1; }
    local action="$1"; shift
    case "$action" in
        start_research) start_research "$@" ;;
        get_status) get_status "$@" ;;
        get_result) get_result "$@" ;;
        poll_until_complete) poll_until_complete "$@" ;;
        help|--help|-h) show_usage ;;
        *) log_error "Unknown action: $action"; show_usage; exit 1 ;;
    esac
}

main "$@"

```



---

## Skill Companion Files

> Additional files collected from the skill directory layout.

### README.md

```markdown
# Local Deep Research (LDR) — OpenClaw Skill

> Multi-cycle deep research using a **locally-hosted** Local Deep Research (LDR) service for OpenClaw / ClawHub. Use this skill when you need comprehensive research with citations, literature reviews, competitive intelligence, or exhaustive web search with iterative question generation.

**Triggers:** "deep research", "research this topic", "comprehensive analysis with sources", "literature review", "investigate [topic]", or when local LDR is preferred over cloud research.

---

## Before installing

- **Required**: `curl` and `jq` must be installed. The skill declares these in its manifest (`requires.bins`).
- **Environment / credentials**: The skill expects `LDR_BASE_URL` (default `http://127.0.0.1:5000`) and, if your LDR uses auth, `LDR_SERVICE_USER` and `LDR_SERVICE_PASSWORD` (or `LDR_USERNAME`/`LDR_PASSWORD`). These are declared in the skill manifest (`requires.env` / `primaryEnv`). Set them via your environment or a local `.env` file only; do not commit secrets.
- **Trust**: Only point `LDR_BASE_URL` at an LDR instance you control (e.g. localhost). The script sources `~/.config/local_deep_research/config/.env` if present—verify that file’s contents and do not put unrelated secrets there.
- **Optional**: Review `scripts/ldr-research.sh`; for higher assurance, run the skill in an isolated environment with network restricted to your LDR host.

---

## Requirements

- A running **LDR** instance (e.g. `http://127.0.0.1:5000`)
- OpenClaw with network and shell permissions
- `curl` and `jq` on the host
- (If LDR has auth enabled) LDR account credentials — **stored only locally**, see below

---

## Quick start

1. **Install the skill** (e.g. via ClawHub or copy into your OpenClaw skills directory).
2. **Set LDR URL** (optional if using default):
   ```bash
   export LDR_BASE_URL="http://127.0.0.1:5000"
   ```
3. **If your LDR instance requires login**, set credentials via **environment variables or a local `.env` file only** (see [Credentials](#credentials-local-only)):
   ```bash
   export LDR_SERVICE_USER="your-ldr-username"
   export LDR_SERVICE_PASSWORD="your-ldr-password"
   ```
4. Run research via the skill (e.g. "Research the latest developments in quantum computing").

---

## Credentials (local-only)

LDR uses **session-cookie authentication with CSRF protection** (not HTTP Basic Auth). The script performs a proper login flow: fetches the login page for a session cookie and CSRF token, then POSTs the login form. Your username and password are used **only** to create that session with **your local** LDR instance (and for LDR’s per-user encrypted results).

- **They are not transmitted** to any third party or to ClawHub/GitHub — only to your own LDR instance (e.g. on localhost).
- **Do not** put credentials in the skill config file or commit them to git.

### Recommended: environment variables or LDR’s `.env`

- **Option A — Environment variables**  
  In your shell or process:
  ```bash
  export LDR_SERVICE_USER="openclaw_service"
  export LDR_SERVICE_PASSWORD="a-strong-password"
  ```
- **Option B — LDR config directory**  
  LDR can load `~/.config/local_deep_research/config/.env`. This script will source that file if it exists, so you can keep one local file for LDR and this skill. Ensure that path is **not** in git (e.g. in `.gitignore` if you ever copy it into a project).
- **Option C — Project `.env`**  
  If you use a project-level `.env`, add it to `.gitignore` and never commit it.

Use a **dedicated LDR user** (e.g. `openclaw_service`) for this skill so you can rotate that account without affecting your personal LDR login.

See `env.example` for variable names. Do **not** commit real credentials.

---

## Configuration

| Variable | Description | Default |
|----------|-------------|---------|
| `LDR_BASE_URL` | LDR service URL | `http://127.0.0.1:5000` |
| `LDR_LOGIN_URL` | Login page URL (session + CSRF) | `$LDR_BASE_URL/auth/login` |
| `LDR_SERVICE_USER` | LDR account username (local auth only) | — |
| `LDR_SERVICE_PASSWORD` | LDR account password (local auth only) | — |
| `LDR_CONFIG_DIR` | Directory containing `.env` to source | `~/.config/local_deep_research/config` |
| `LDR_DEFAULT_MODE` | Research mode: `quick` (Quick Summary) or `detailed` (Detailed Report) | `detailed` |
| `LDR_DEFAULT_LANGUAGE` | Default output language for report/summary (e.g. `en`, `es`, `fr`, `de`, `zh`, `ja`) | — |
| `LDR_DEFAULT_SEARCH_TOOL` | Search tool: `searxng`, `auto`, `local_all` | `auto` |

**Research modes:** Use `quick` for a short summary (faster); use `detailed` for a full report with full citations. You can override the output language per run with `--language` (e.g. `--language es` for Spanish).

`LDR_USERNAME` / `LDR_PASSWORD` are supported as fallbacks.

---

## Actions

The skill exposes these actions via `scripts/ldr-research.sh`:

| Action | Description |
|--------|-------------|
| `start_research` | Submit a query; returns `research_id`. |
| `get_status` | Check job status by `research_id`. |
| `get_result` | Fetch the full report when completed. |
| `poll_until_complete` | Block until the job completes or times out. |

See **SKILL.md** for input/output schemas and orchestration patterns.

---

## Project layout

```
local-deep-research/
├── README.md           # This file
├── SKILL.md            # OpenClaw skill instructions (entry point)
├── env.example         # Example env vars (no secrets)
├── references/
│   ├── api.md          # LDR API reference
│   └── examples.md     # Usage examples
└── scripts/
    └── ldr-research.sh # CLI for start_research, get_status, get_result, poll_until_complete
```

---

## Troubleshooting

- **LDR not responding**  
  Check `LDR_BASE_URL` and that LDR is running: `curl "$LDR_BASE_URL/health"`.

- **Authentication failures**  
  LDR uses **session + CSRF**, not Basic Auth. Ensure `LDR_SERVICE_USER` and `LDR_SERVICE_PASSWORD` (or `LDR_USERNAME`/`LDR_PASSWORD`) are set via env or a local `.env` that is **not** committed. Run the script and look for "Login successful"; or open `LDR_LOGIN_URL` in a browser to confirm LDR’s login page works.

- **Research stuck**  
  Check LDR logs and service health; consider timeout/restart if there’s no progress for a long time.

---

## Publishing (ClawHub / GitHub)

- **ClawHub:** From the skill root, run  
  `clawhub publish . --slug local-deep-research --name "Local Deep Research" --version 1.0.0 --tags latest`  
  (Requires `clawhub login` first.)
- **GitHub:** Ensure `.gitignore` is in place so `.env` and secrets are never committed. Clone or fork the repo as usual.

## License

MIT — see [LICENSE](LICENSE).

## Links

- [ClawHub](https://clawhub.ai/) — OpenClaw skill registry  
- [OpenClaw](https://docs.openclaw.ai/) — Documentation

## Author

[Edward Tsang](https://github.com/eplt) — blockchain & AI engineer. Open to consulting → [Email](mailto:[email protected]) · [LinkedIn](https://www.linkedin.com/in/edwardtsang/)

```

### _meta.json

```json
{
  "owner": "eplt",
  "slug": "local-deep-research",
  "displayName": "Local Deep Research",
  "latest": {
    "version": "1.0.2",
    "publishedAt": 1773120952381,
    "commit": "https://github.com/openclaw/skills/commit/db3de812aff241818464f24aec56771ea9bdff60"
  },
  "history": []
}

```

### references/api.md

```markdown
# LDR API Reference

This document describes the Local Deep Research (LDR) API endpoints used by this skill.

## Base URL

Default: `http://127.0.0.1:5000`  
Configurable via `LDR_BASE_URL` environment variable.

## Authentication (session + CSRF)

LDR uses **session-cookie authentication with CSRF protection**; it does **not** use HTTP Basic Auth. A proper login flow is required before calling the research API:

1. **GET** the login page (e.g. `/auth/login`) to obtain the session cookie and CSRF token (e.g. from a hidden form field `csrf_token` or `_csrf`).
2. **POST** the login form with `username`, `password`, and the CSRF token (typically `application/x-www-form-urlencoded`).
3. Use the session cookie for all subsequent requests. For state-changing **POST** requests (e.g. `/research/api/start`), send the CSRF token in a header (e.g. `X-CSRFToken`) if the server expects it.

Credentials are for **local LDR only** and must be provided via environment variables or a local `.env` file (never in committed config). The skill script implements this flow automatically.

| Env var | Purpose |
|--------|---------|
| `LDR_SERVICE_USER` / `LDR_USERNAME` | LDR account username |
| `LDR_SERVICE_PASSWORD` / `LDR_PASSWORD` | LDR account password |
| `LDR_LOGIN_URL` | Login page URL (default: `$LDR_BASE_URL/auth/login`) |

---

## Endpoints

### POST /research/api/start

Start a new research job. Requires an established session (and CSRF token for POST).

**Request Body:**
```json
{
  "query": "string (required)",
  "mode": "quick|detailed (optional, default: detailed)",
  "language": "string (optional)",
  "search_tool": "searxng|auto|local_all (optional, default: auto)",
  "iterations": "number (optional)",
  "questions_per_iteration": "number (optional)",
  "max_results": "number (optional)"
}
```

**`mode`** — Research output type:
- **`quick`** — Quick Summary: fewer cycles, shorter output, faster run.
- **`detailed`** — Detailed Report: full multi-cycle research, full markdown report and citations.

**`language`** — Output language for the report/summary. ISO 639-1 code (e.g. `en`, `es`, `fr`, `de`, `zh`, `ja`). Omit or leave empty for LDR default (often English).

**Response (200 OK):**
```json
{
  "research_id": "uuid-string",
  "mode": "detailed",
  "search_tool": "auto",
  "submitted_at": "2026-03-10T08:00:00Z",
  "status": "queued",
  "estimated_duration_seconds": 300
}
```

**Error Responses:**  
`400` Bad Request, `401` Unauthorized / session required, `500` Internal Server Error.

---

### GET /research/api/status/{research_id}

Check the status of a research job. Send session cookie.

**Path Parameters:** `research_id` — research job UUID.

**Response (200 OK):**
```json
{
  "research_id": "uuid-string",
  "state": "pending|running|completed|failed|timeout",
  "progress": 45,
  "message": "Synthesizing sources from iteration 2...",
  "last_milestone": "...",
  "current_iteration": 2,
  "total_iterations": 3,
  "sources_found": 24,
  "questions_generated": 12,
  "started_at": "...",
  "updated_at": "..."
}
```

**Error Responses:** `401` Unauthorized, `404` Not Found.

---

### GET /research/api/report/{research_id}

Fetch the complete research report. Send session cookie.

**Path Parameters:** `research_id` — research job UUID.

**Response (200 OK):**
```json
{
  "research_id": "uuid-string",
  "query": "original research query",
  "mode": "detailed",
  "summary": "Executive summary...",
  "report_markdown": "# Full Report\n\n...",
  "sources": [{"id": 1, "title": "...", "url": "...", "snippet": "...", "type": "web|local_doc", ...}],
  "iterations": 3,
  "total_questions": 24,
  "total_sources": 48,
  "created_at": "...",
  "completed_at": "...",
  "execution_time_seconds": 900
}
```

**Error Responses:** `400` Not yet completed, `401` Unauthorized, `404` Not Found.

---

## Health Check

**GET /health** — Check if the LDR service is running. No auth required for typical setups.

```json
{"status": "healthy", "version": "1.0.0", "uptime_seconds": 86400, "active_jobs": 3, "queued_jobs": 1}
```

---

## Error Codes

| Code | Meaning | Handling |
|------|---------|----------|
| 400 | Bad Request | Fix request parameters |
| 401 | Unauthorized | Re-establish session (login again) |
| 404 | Not Found | Verify research_id or path |
| 429 | Too Many Requests | Retry after delay |
| 500 | Internal Error | Retry with backoff |

---

## Testing auth (session + CSRF)

Do **not** use `curl -u user:pass` (Basic Auth). Instead:

1. Set `LDR_SERVICE_USER` and `LDR_SERVICE_PASSWORD` (or `LDR_USERNAME`/`LDR_PASSWORD`).
2. Run the skill script: `scripts/ldr-research.sh start_research --query "test"` and look for "Login successful".
3. Or open `LDR_LOGIN_URL` in a browser and sign in to confirm LDR’s login page and credentials.

```

### references/examples.md

```markdown
# LDR Skill Usage Examples

## Modes and language

- **Quick Summary** (`--mode quick`): faster, shorter output; good for a brief overview.
- **Detailed Report** (`--mode detailed`): full multi-cycle report with full citations.
- **Output language** (`--language <code>`): e.g. `en`, `es`, `fr`, `de`, `zh`, `ja`. Set default via `LDR_DEFAULT_LANGUAGE`.

## Quick Start

### One-shot Research (wait for completion)

```bash
# Ensure LDR_SERVICE_USER and LDR_SERVICE_PASSWORD are set (env or local .env)
research_id=$(scripts/ldr-research.sh start_research \
  --query "latest developments in solid-state batteries" \
  --mode detailed | jq -r '.research_id')

echo "Started research: $research_id"
scripts/ldr-research.sh poll_until_complete --research_id "$research_id" --interval 15
scripts/ldr-research.sh get_result --research_id "$research_id" | jq -r '.report_markdown'
```

### Quick Summary in Spanish

```bash
scripts/ldr-research.sh start_research --query "tendencias en IA 2026" --mode quick --language es
```

### Fire-and-forget (async)

```bash
response=$(scripts/ldr-research.sh start_research \
  --query "quantum computing advances 2026" \
  --mode detailed \
  --language en \
  --search_tool searxng)

research_id=$(echo "$response" | jq -r '.research_id')
echo "Research ID: $research_id"
echo "Check later: scripts/ldr-research.sh get_status --research_id $research_id"

# When ready:
status=$(scripts/ldr-research.sh get_status --research_id "$research_id")
state=$(echo "$status" | jq -r '.state')
[[ "$state" == "completed" ]] && scripts/ldr-research.sh get_result --research_id "$research_id" > report.json
```

## OpenClaw integration

Credentials: set `LDR_SERVICE_USER` and `LDR_SERVICE_PASSWORD` (or `LDR_USERNAME`/`LDR_PASSWORD`) via env or `~/.config/local_deep_research/config/.env`. The script performs session + CSRF login automatically.

```bash
LDR_SCRIPT="$(pwd)/scripts/ldr-research.sh"   # when run from skill root

query="$1"
result=$("$LDR_SCRIPT" start_research --query "$query" --mode detailed)
research_id=$(echo "$result" | jq -r '.research_id')

echo "Started deep research: $query (ID: $research_id)"

while true; do
  status=$("$LDR_SCRIPT" get_status --research_id "$research_id")
  state=$(echo "$status" | jq -r '.state')
  echo "Progress: $(echo "$status" | jq -r '.progress')% - $(echo "$status" | jq -r '.message')"

  case "$state" in
    completed) "$LDR_SCRIPT" get_result --research_id "$research_id" | jq -r '.report_markdown'; break ;;
    failed|timeout) echo "Research $state"; exit 1 ;;
  esac
  sleep 20
done
```

## Configuration (env only for secrets)

Store credentials in env or a **private** .env (add to .gitignore). LDR uses session + CSRF; do not put username/password in OpenClaw config.

```bash
# In shell or private .env
export LDR_BASE_URL="http://127.0.0.1:5000"
export LDR_SERVICE_USER="openclaw_service"
export LDR_SERVICE_PASSWORD="your-strong-password"
export LDR_DEFAULT_MODE="detailed"
export LDR_DEFAULT_LANGUAGE="en"
export LDR_DEFAULT_SEARCH_TOOL="searxng"
```

If LDR uses a different login path:

```bash
export LDR_LOGIN_URL="http://127.0.0.1:5000/login"
```

## Troubleshooting

### Authentication failures

LDR uses **session-cookie + CSRF**, not Basic Auth. Ensure credentials are set (env or local .env) and run the script; look for "Login successful". If LDR’s login form uses different field names (e.g. `email` instead of `username`), you may need to adjust the script’s login POST or set `LDR_LOGIN_URL` to the correct path.

### Connection refused

```bash
curl -s "$LDR_BASE_URL/health"
# Start LDR service if not running
```

### Research stuck

Check LDR service health and logs; consider timeout/restart if there’s no progress for a long time.

```