yves-web-search

Original source / Raw SKILL.md

---
name: web-search
description: Search the web and fetch web pages. Use when user wants to look up information, find answers, or search for anything online. Supports multiple free methods: Jina AI reader, DuckDuckGo Lite, and Python ddgs fallback. No API keys required for basic use.
metadata:
  {
    "openclaw": {
      "emoji": "🔍",
      "homepage": "https://github.com/openclaw/openclaw",
      "requires": {}
    }
  }
---

# Web Search Skill

Comprehensive web search and content extraction — free, no API keys required.

## Quick Start

**Just want to search?** Use DuckDuckGo Lite:
```
web_fetch url="https://lite.duckduckgo.com/lite/?q=YOUR+QUERY"
```

**Want full page content?** Use Jina Reader:
```
web_fetch url="https://r.jina.ai/http://TARGET_URL"
```

---

## Method 1: Jina AI Reader (Free - Recommended for Content)

Extract full page content using Jina's free reader API.

### Read a URL
```
web_fetch url="https://r.jina.ai/http://example.com"
```

### Search the Web
```
web_fetch url="https://r.jina.ai/http://duckduckgo.com/?q=YOUR+QUERY"
```

**Examples:**
| Task | Command |
|------|---------|
| Get Next.js docs | `web_fetch url="https://r.jina.ai/http://nextjs.org"` |
| Get React docs | `web_fetch url="https://r.jina.ai/http://react.dev"` |
| Get Python docs | `web_fetch url="https://r.jina.ai/http://docs.python.org"` |

### Advanced Jina Reader (with API Key)

For advanced features, get a free API key from [jina.ai/reader](https://jina.ai/reader):

```bash
export JINA_API_KEY="jina_..."
```

Then use the bundled script:
```
{baseDir}/scripts/reader.sh "https://example.com"
{baseDir}/scripts/reader.sh --mode search "AI news 2025"
{baseDir}/scripts/reader.sh --mode ground "OpenAI founded 2015"
```

Options: `--mode`, `--selector`, `--remove`, `--format`, `--json`

---

## Method 2: DuckDuckGo Lite (Free - Recommended for Search)

Search without any API key using DuckDuckGo Lite.

### Basic Search
```
web_fetch url="https://lite.duckduckgo.com/lite/?q=YOUR+QUERY"
```

### Regional Search
```
web_fetch url="https://lite.duckduckgo.com/lite/?q=QUERY&kl=us-en"
```

Regions: `au-en`, `us-en`, `uk-en`, `de-de`, `fr-fr`

### Search Tips
- Use `+` for spaces: `python+tutorial`
- Use quotes for exact phrases: `%22exact+phrase%22`
- Skip first 1-2 results (ads)

---

## Method 3: Python ddgs (Fallback)

If web_fetch is blocked, use Python's `ddgs` package:

```bash
pip install ddgs
```

```python
from ddgs import DDGS
ddgs = DDGS()
results = ddgs.text("search query", max_results=5)
for r in results:
    print(f"{r['title']}: {r['url']}")
```

---

## Workflow: Search then Fetch

1. **Search** → DDG Lite to find relevant URLs
2. **Pick** → Identify best result(s)
3. **Fetch** → Jina Reader to extract full content

Example:
```
# Step 1: Find info about Next.js auth
web_fetch url="https://lite.duckduckgo.com/lite/?q=nextjs+authentication+docs"

# Step 2: Fetch the official docs
web_fetch url="https://r.jina.ai/http://nextjs.org/docs/app/..."
```

---

## Quick Reference

| Need | Method | Command |
|------|--------|---------|
| Find URLs | DDG Lite | `?q=search+terms` |
| Get page content | Jina Reader | `r.jina.ai/http://URL` |
| Advanced extraction | Jina API | `--mode search --json` |
| Python fallback | ddgs | `ddgs.text()` |
| Browser (if available) | Headless | `browser action=navigate` |

---

## Limitations

- Google search blocked (captcha)
- No date filtering via DDG Lite
- Jina free tier has rate limits


---

## Skill Companion Files

> Additional files collected from the skill directory layout.

### _meta.json

```json
{
  "owner": "flashery",
  "slug": "yves-web-search",
  "displayName": "Yves Web Search",
  "latest": {
    "version": "1.0.1",
    "publishedAt": 1771607794652,
    "commit": "https://github.com/openclaw/skills/commit/5d56e6410767923122b151a0347e24da6d74019e"
  },
  "history": []
}

```

### scripts/reader.sh

```bash
#!/bin/bash
# Jina Reader CLI wrapper
# Usage: ./reader.sh [--mode read|search|ground] [--format markdown|html|text|json] [--json] [--nocache] <URL|query>
# 
# Modes:
#   read    - Extract content from URL (default)
#   search  - Search web and get top results with full content
#   ground  - Fact-check a statement
#
# Examples:
#   ./reader.sh "https://example.com"
#   ./reader.sh --mode search "latest AI news"
#   ./reader.sh --mode ground "OpenAI was founded in 2015"

set -e

MODE="read"
FORMAT="markdown"
JSON=false
NOCACHE=""
SELECTOR=""
REMOVE=""
PROXY=""

show_help() {
    cat << EOF
Jina Reader - Extract web content via Jina AI

Usage: $0 [options] <URL or query>

Options:
    --mode MODE     read, search, or ground (default: read)
    --format FMT    markdown, html, text, or screenshot (default: markdown)
    --json          Output raw JSON
    --nocache       Force fresh content
    --selector CSS  Extract specific element
    --remove SEL   Remove elements (comma-separated)
    --proxy CC     Country code for geo-proxy (br, us, etc.)
    -h, --help     Show this help

Examples:
    $0 "https://example.com"
    $0 --mode search "AI trends 2025"
    $0 --mode ground "Tesla is worth more than Toyota"
EOF
}

# Parse arguments
while [[ $# -gt 0 ]]; do
    case $1 in
        --mode)
            MODE="$2"
            shift 2
            ;;
        --format)
            FORMAT="$2"
            shift 2
            ;;
        --json)
            JSON=true
            shift
            ;;
        --nocache)
            NOCACHE="?nocache=true"
            shift
            ;;
        --selector)
            SELECTOR="&selector=$(echo $2 | sed 's/ /%20/g')"
            shift 2
            ;;
        --remove)
            REMOVE="&remove=$(echo $2 | sed 's/,/%2C/g')"
            shift 2
            ;;
        --proxy)
            PROXY="?country=$2"
            shift 2
            ;;
        -h|--help)
            show_help
            exit 0
            ;;
        *)
            TARGET="$1"
            shift
            ;;
    esac
done

if [ -z "$TARGET" ]; then
    echo "Error: URL or query required"
    show_help
    exit 1
fi

# Get API key
API_KEY="${JINA_API_KEY:-}"

# Build URL based on mode
if [ "$MODE" = "read" ]; then
    BASE_URL="https://r.jina.ai/url/$TARGET"
elif [ "$MODE" = "search" ]; then
    BASE_URL="https://r.jina.ai/search$PROXY?q=$(echo "$TARGET" | sed 's/ /%20/g')"
elif [ "$MODE" = "ground" ]; then
    BASE_URL="https://r.jina.ai/ground"$PROXY
    # For ground mode, send as POST would be better, but use q param
    TARGET="statement=$(echo "$TARGET" | sed 's/ /%20/g')"
else
    echo "Error: Unknown mode: $MODE"
    exit 1
fi

# Add format if not default
if [ "$FORMAT" != "markdown" ]; then
    BASE_URL="$BASE_URL&format=$FORMAT"
fi

# Add selectors
BASE_URL="$BASE_URL$SELECTOR$REMOVE$NOCACHE"

# Make request
if [ -n "$API_KEY" ]; then
    # Use API key for full features
    curl -s -H "Authorization: Bearer $API_KEY" \
         -H "Accept: application/json" \
         "$BASE_URL" | if [ "$JSON" = true ]; then
             cat
         else
             # Extract content field if JSON
             grep -o '"content":"[^"]*"' | sed 's/"content":"//;s/"$//' || cat
         fi
else
    # Free tier
    curl -s "$BASE_URL"
fi

```

### scripts/search.py

```python
#!/usr/bin/env python3
"""
Web search using ddgs (DuckDuckGo) Python package.
Usage: python search.py "search query" [max_results]
"""
import sys
import argparse

def main():
    parser = argparse.ArgumentParser(description='DuckDuckGo Search')
    parser.add_argument('query', help='Search query')
    parser.add_argument('--max', '-n', type=int, default=5, help='Max results')
    parser.add_argument('--json', action='store_true', help='JSON output')
    
    args = parser.parse_args()
    
    try:
        from ddgs import DDGS
    except ImportError:
        print("Error: ddgs not installed. Run: pip install ddgs", file=sys.stderr)
        sys.exit(1)
    
    ddgs = DDGS()
    results = ddgs.text(args.query, max_results=args.max)
    
    if args.json:
        import json
        print(json.dumps(list(results), indent=2))
    else:
        for i, r in enumerate(results, 1):
            print(f"{i}. {r.get('title', 'No title')}")
            print(f"   {r.get('url', 'No URL')}")
            if r.get('body'):
                print(f"   {r.get('body', '')[:200]}...")
            print()

if __name__ == '__main__':
    main()

```