marker

Original source / Raw SKILL.md

---
name: marker
description: Convert PDF documents to Markdown using marker_single. Use when Claude needs to extract text content from PDFs while preserving LaTeX formulas, equations, and document structure. Ideal for academic papers and technical documents containing mathematical notation.
---

# Marker PDF-to-Markdown Converter

Convert PDFs to Markdown while preserving LaTeX formulas and document structure. Uses the `marker_single` CLI from the marker-pdf package.

## Dependencies
- `marker_single` on PATH (`pip install marker-pdf` if missing)
- Python 3.10+ (available in the task image)

## Quick Start

```python
from scripts.marker_to_markdown import pdf_to_markdown

markdown_text = pdf_to_markdown("paper.pdf")
print(markdown_text)
```

## Python API

- `pdf_to_markdown(pdf_path, *, timeout=600, cleanup=True) -> str`
  - Runs `marker_single --output_format markdown --disable_image_extraction`
  - `cleanup=True`: use a temp directory and delete after reading the Markdown
  - `cleanup=False`: keep outputs in `<pdf_stem>_marker/` next to the PDF
  - Exceptions: `FileNotFoundError` if the PDF is missing, `RuntimeError` for marker failures, `TimeoutError` if it exceeds the timeout
- Tips: bump `timeout` for large PDFs; set `cleanup=False` to inspect intermediate files

## Command-Line Usage

```bash
# Basic conversion (prints markdown to stdout)
python scripts/marker_to_markdown.py paper.pdf

# Keep temporary files
python scripts/marker_to_markdown.py paper.pdf --keep-temp

# Custom timeout
python scripts/marker_to_markdown.py paper.pdf --timeout 600
```

## Output Locations
- `cleanup=True`: outputs stored in a temporary directory and removed automatically
- `cleanup=False`: outputs saved to `<pdf_stem>_marker/`; markdown lives at `<pdf_stem>_marker/<pdf_stem>/<pdf_stem>.md` when present (otherwise the first `.md` file is used)

## Troubleshooting
- `marker_single` not found: install `marker-pdf` or ensure the CLI is on PATH
- No Markdown output: re-run with `--keep-temp`/`cleanup=False` and check `stdout`/`stderr` saved in the output folder


---

## Referenced Files

> The following files are referenced in this skill and included for context.

### scripts/marker_to_markdown.py

```python
"""Utility to convert a PDF to Markdown using marker_single and return the text."""

from __future__ import annotations

import argparse
import shutil
import subprocess
import tempfile
from pathlib import Path


def _find_markdown_files(folder: Path) -> list[Path]:
    """Return all markdown files beneath folder, sorted for deterministic selection."""
    return sorted(path for path in folder.rglob("*.md") if path.is_file())


def _run_marker(source: Path, output_dir: Path, timeout: int) -> subprocess.CompletedProcess[str]:
    cmd = [
        "marker_single",
        str(source),
        "--output_format",
        "markdown",
        "--disable_image_extraction",
        "--output_dir",
        str(output_dir),
    ]

    try:
        return subprocess.run(
            cmd,
            check=True,
            capture_output=True,
            text=True,
            timeout=timeout,
        )
    except subprocess.CalledProcessError as exc:
        raise RuntimeError(f"marker_single failed with code {exc.returncode}\n" f"stdout:\n{exc.stdout}\n\nstderr:\n{exc.stderr}") from exc
    except subprocess.TimeoutExpired as exc:
        raise TimeoutError(f"marker_single timed out after {timeout}s") from exc


def _select_markdown(output_dir: Path, pdf_stem: str, stdout: str) -> Path:
    """Pick the primary markdown output; prefer file matching the PDF stem."""
    preferred = output_dir / pdf_stem / f"{pdf_stem}.md"
    if preferred.exists():
        return preferred

    candidates = _find_markdown_files(output_dir)
    if candidates:
        return candidates[0]

    raise FileNotFoundError(f"No markdown output found in {output_dir}; marker_single stdout:\n{stdout}")


def pdf_to_markdown(pdf_path: str, *, timeout: int = 120, cleanup: bool = True) -> str:
    """
    Run marker_single on the given PDF and return the generated Markdown as a string.

    The output directory is created in a temporary location when cleanup is True (default)
    and removed after reading the Markdown. When cleanup is False, outputs are written next
    to the PDF under a subfolder named "<pdf_stem>_marker" and left on disk.
    timeout is in seconds; marker_single typically finishes within ~1 minute.

    Usage examples:
      >>> pdf_to_markdown("paper.pdf")  # returns markdown string and removes temp files
      >>> pdf_to_markdown("paper.pdf", cleanup=False)  # keep outputs in ./paper_marker/
      >>> pdf_to_markdown("paper.pdf", timeout=300)  # allow longer runtime if needed
    """

    source = Path(pdf_path).expanduser().resolve()
    if not source.exists():
        raise FileNotFoundError(f"PDF not found: {source}")

    if shutil.which("marker_single") is None:
        raise RuntimeError("marker_single CLI not found. Install marker-pdf to proceed.")

    def convert(output_dir: Path) -> str:
        result = _run_marker(source, output_dir, timeout)
        md_path = _select_markdown(output_dir, source.stem, result.stdout)
        return md_path.read_text(encoding="utf-8")

    if cleanup:
        with tempfile.TemporaryDirectory(prefix="marker_single_") as tmpdir:
            return convert(Path(tmpdir))

    output_dir = source.parent / f"{source.stem}_marker"
    output_dir.mkdir(exist_ok=True)
    return convert(output_dir)


def main() -> None:
    """
    Keep temp files: python marker_to_markdown.py 2405.10501v1.pdf --keep-temp
    Longer timeout: python marker_to_markdown.py 2405.10501v1.pdf --timeout 300 --keep-temp
    """
    parser = argparse.ArgumentParser(description="Convert a PDF to Markdown using marker_single.")
    parser.add_argument("pdf", help="Path to the PDF file.")
    parser.add_argument(
        "--timeout",
        type=int,
        default=120,
        help="Timeout in seconds (default: 120).",
    )
    parser.add_argument(
        "--keep-temp",
        action="store_true",
        help="Keep the temporary output directory instead of deleting it.",
    )
    args = parser.parse_args()

    markdown = pdf_to_markdown(
        args.pdf,
        timeout=args.timeout,
        cleanup=not args.keep_temp,
    )
    print(markdown)


if __name__ == "__main__":
    main()

```