arena-cli
CLI tools for Are.na: export blocks, enrich with vision AI, generate views. Use when: (1) exporting Are.na blocks incrementally, (2) enriching images with AI-generated titles/tags/patterns, (3) generating browsable HTML views, (4) searching blocks by UI patterns or tags, (5) visual search results when terminal output is insufficient. Triggers: "export arena", "enrich arena", "sync arena", "arena view", "search arena for [pattern]", "show me [pattern]".
Packaged view
This page reorganizes the original catalog entry around fit, installability, and workflow context first. The original raw source lives below.
Install command
npx @skill-hub/cli install rohunvora-cool-claude-skills-arena-cli
Repository
Skill path: skills/arena-cli
CLI tools for Are.na: export blocks, enrich with vision AI, generate views. Use when: (1) exporting Are.na blocks incrementally, (2) enriching images with AI-generated titles/tags/patterns, (3) generating browsable HTML views, (4) searching blocks by UI patterns or tags, (5) visual search results when terminal output is insufficient. Triggers: "export arena", "enrich arena", "sync arena", "arena view", "search arena for [pattern]", "show me [pattern]".
Open repositoryBest for
Primary workflow: Design Product.
Technical facets: Full Stack, Frontend, Data / AI, Designer.
Target audience: everyone.
License: Unknown.
Original source
Catalog source: SkillHub Club.
Repository owner: rohunvora.
This is still a mirrored public skill entry. Review the repository before installing into production workflows.
What it helps with
- Install arena-cli into Claude Code, Codex CLI, Gemini CLI, or OpenCode workflows
- Review https://github.com/rohunvora/cool-claude-skills before adding arena-cli to shared team environments
- Use arena-cli for development workflows
Works across
Favorites: 0.
Sub-skills: 0.
Aggregator: No.
Original source / Raw SKILL.md
---
name: arena-cli
description: >
CLI tools for Are.na: export blocks, enrich with vision AI, generate views.
Use when: (1) exporting Are.na blocks incrementally, (2) enriching images with
AI-generated titles/tags/patterns, (3) generating browsable HTML views,
(4) searching blocks by UI patterns or tags, (5) visual search results when
terminal output is insufficient. Triggers: "export arena", "enrich arena",
"sync arena", "arena view", "search arena for [pattern]", "show me [pattern]".
---
# arena-cli
CLI tools to export, enrich, and browse Are.na blocks locally.
## Setup
First-time setup requires:
```bash
# In your project directory
echo "ARENA_TOKEN=your_token_here" >> .env
echo "ARENA_USER_SLUG=your_username" >> .env
echo "GEMINI_API_KEY=your_key_here" >> .env
```
Get your Are.na token from: https://dev.are.na/oauth/applications
## Workflow
```
1. Export ā 2. Enrich ā 3. View/Search
(blocks) (vision AI) (HTML + grep)
```
### 1. Export Blocks
```bash
# First run - exports all channels
npx ts-node scripts/export-blocks.ts
# Incremental update (only new blocks)
npx ts-node scripts/export-blocks.ts
# Specific channel
npx ts-node scripts/export-blocks.ts --channel=ui-ux-abc123
# With local image download
npx ts-node scripts/export-blocks.ts --images
```
Output: `arena-export/blocks/{id}.json`
### 2. Enrich with Vision AI
```bash
# Enrich all image blocks
npx ts-node scripts/enrich-blocks.ts
# Specific channel
npx ts-node scripts/enrich-blocks.ts --channel=ui-ux-abc123
# Preview without saving
npx ts-node scripts/enrich-blocks.ts --dry-run
# Re-enrich already processed
npx ts-node scripts/enrich-blocks.ts --force
```
Adds to each block:
- `vision.suggested_title` - Clean title
- `vision.description` - What's notable
- `vision.tags` - Searchable tags
- `vision.ui_patterns` - UI component patterns
### 3. Generate View
```bash
# Generate HTML
node scripts/gen-view.cjs
# Generate and open
node scripts/gen-view.cjs --open
```
Output: `arena-export/view.html`
### 4. Visual Search Results
When terminal output is insufficient (need to see actual images):
```bash
# Ad-hoc search - comma-separated patterns
node scripts/gen-search-view.cjs "dashboard,metric-cards" --open
# Multiple search groups
node scripts/gen-search-view.cjs "avatar,profile" "chart,graph" --open
# From config file
node scripts/gen-search-view.cjs --config=searches.json --open
```
Config file format (`searches.json`):
```json
[
{ "name": "Dashboards", "patterns": ["dashboard", "metric-cards", "kpi"] },
{ "name": "Charts", "patterns": ["chart", "graph", "visualization"] }
]
```
Output: `arena-export/search-results.html` with image grid, grouped by category.
## Searching
```bash
# Search by UI pattern
grep -l "inline-stats" arena-export/blocks/*.json
# Search by tag
grep -l '"dashboard"' arena-export/blocks/*.json
# Search with context
grep -B2 -A2 "leaderboard" arena-export/blocks/*.json
```
## Block Schema
```json
{
"id": 12345,
"title": "original-filename.png",
"class": "Image",
"image_url": "https://...",
"channels": ["ui-ux-abc"],
"vision": {
"suggested_title": "Dark Trading Dashboard",
"description": "Crypto dashboard with real-time charts",
"tags": ["dashboard", "dark-mode", "trading"],
"ui_patterns": ["metric-cards", "time-series-chart"]
}
}
```
## Environment Variables
| Variable | Required | Description |
|----------|----------|-------------|
| `ARENA_TOKEN` | Yes | Are.na API token |
| `ARENA_USER_SLUG` | Yes | Your Are.na username |
| `GEMINI_API_KEY` | Yes | Google AI API key |
| `ARENA_EXPORT_DIR` | No | Export path (default: ./arena-export) |
## Customization
To customize the vision enrichment prompt, see [references/enrichment-prompt.md](references/enrichment-prompt.md).
---
## Referenced Files
> The following files are referenced in this skill and included for context.
### references/enrichment-prompt.md
```markdown
# Enrichment Prompt
This is the prompt sent to Gemini Vision for each image block. Customize this to change what metadata is extracted.
## Default Prompt
```
Analyze this image for a design reference library (Are.na).
Generate:
1. **suggested_title**: A clean, descriptive title (3-8 words). NOT a filename. Describe what this IS, not what it shows.
- Good: "Dark Mode Trading Dashboard", "Mobile Onboarding Flow", "Stat Card with Progress Bars"
- Bad: "Screenshot", "UI Design", "Image of interface"
2. **description**: 1-2 sentences describing what makes this notable as a design reference. What could someone learn from this?
3. **tags**: 5-15 lowercase tags for searching. Include:
- Layout type: table, card, list, grid, dashboard, modal, form, nav
- Components: avatar, button, input, stat-bar, progress, badge, tag, icon
- Patterns: leaderboard, onboarding, settings, profile, feed, search
- Style: dark-mode, light-mode, minimal, dense, colorful, monochrome
- Platform: mobile, desktop, web, ios, android
4. **ui_patterns**: Specific UI patterns visible (for styling reference):
- Examples: "inline-stats", "avatar-with-name", "tiered-ranking", "progress-percentage", "sortable-headers", "status-pills", "metric-cards"
Respond in JSON format only:
{
"suggested_title": "...",
"description": "...",
"tags": ["...", "..."],
"ui_patterns": ["...", "..."]
}
```
## Customization
To customize for different use cases:
### For code/technical references
Add to tags section:
```
- Language: python, javascript, rust, go
- Concepts: algorithm, data-structure, architecture, api
```
### For marketing/copy references
Add to tags section:
```
- Type: headline, cta, testimonial, pricing
- Tone: playful, professional, urgent, minimal
```
### For illustration/art references
Replace ui_patterns with:
```
4. **art_patterns**: Style and technique patterns:
- Examples: "flat-illustration", "3d-render", "hand-drawn", "gradient-mesh"
```
## Output Schema
The enrichment adds a `vision` field to each block:
```json
{
"id": 12345,
"title": "original-filename.png",
"vision": {
"suggested_title": "Dark Mode Trading Dashboard",
"description": "A crypto trading interface with real-time charts and whale activity tracking.",
"tags": ["dashboard", "crypto", "dark-mode", "trading", "charts"],
"ui_patterns": ["metric-cards", "time-series-chart", "status-pills"],
"enriched_at": "2025-01-01T00:00:00.000Z",
"model": "gemini-2.0-flash"
}
}
```
```
### scripts/export-blocks.ts
```typescript
#!/usr/bin/env npx ts-node
/**
* Are.na Incremental Block Exporter
*
* Exports all blocks from all channels with:
* - Incremental fetching (never re-downloads existing data)
* - Local image storage
* - Resume support (interruption-safe)
* - Backfill mode for historical data
*
* Usage:
* npx ts-node export-blocks.ts # Incremental update
* npx ts-node export-blocks.ts --backfill # Fetch older blocks
* npx ts-node export-blocks.ts --full # Full re-export (skip existing)
* npx ts-node export-blocks.ts --channel=slug # Export specific channel
* npx ts-node export-blocks.ts --images # Also download images
*
* Environment:
* ARENA_TOKEN - Are.na API token (required)
* ARENA_USER_SLUG - Your Are.na username (required)
* ARENA_EXPORT_DIR - Export directory (default: ./arena-export)
*/
import 'dotenv/config';
import * as fs from 'fs';
import * as path from 'path';
// ============================================================================
// CONFIGURATION
// ============================================================================
const ARENA_TOKEN = process.env.ARENA_TOKEN!;
const ARENA_USER_SLUG = process.env.ARENA_USER_SLUG!;
const ARENA_API_BASE = 'https://api.are.na/v2';
const EXPORT_DIR = process.env.ARENA_EXPORT_DIR || path.join(process.cwd(), 'arena-export');
const STATE_FILE = path.join(EXPORT_DIR, 'state.json');
const BLOCKS_DIR = path.join(EXPORT_DIR, 'blocks');
const IMAGES_DIR = path.join(EXPORT_DIR, 'images');
const CHANNELS_FILE = path.join(EXPORT_DIR, 'channels.json');
const PER_PAGE = 100;
const RATE_LIMIT_MS = 200; // Be nice to Are.na API
// ============================================================================
// TYPES
// ============================================================================
interface ArenaBlock {
id: number;
title: string | null;
updated_at: string;
created_at: string;
class: string;
base_class: string;
content: string | null;
content_html: string | null;
description: string | null;
description_html: string | null;
source: { url: string } | null;
image: {
filename: string;
content_type: string;
original: { url: string };
display: { url: string };
thumb: { url: string };
} | null;
attachment: {
url: string;
file_name: string;
content_type: string;
} | null;
embed: {
url: string;
type: string;
html: string;
} | null;
}
interface ArenaChannel {
id: number;
title: string;
slug: string;
length: number;
status: string;
updated_at: string;
created_at: string;
}
interface ChannelState {
slug: string;
title: string;
// Two watermarks pattern
newest_id: number | null; // For fetching new blocks
oldest_id: number | null; // For backfilling old blocks
// Progress tracking
total_exported: number;
last_updated: string;
fully_backfilled: boolean;
}
interface ExportState {
user_slug: string;
channels: Record<string, ChannelState>;
last_channels_fetch: string | null;
created_at: string;
updated_at: string;
}
interface ExportedBlock {
id: number;
title: string | null;
class: string;
created_at: string;
updated_at: string;
content: string | null;
description: string | null;
source_url: string | null;
image_url: string | null;
image_local: string | null;
attachment_url: string | null;
embed_url: string | null;
channels: string[]; // Channel slugs this block appears in
exported_at: string;
}
// ============================================================================
// STATE MANAGEMENT
// ============================================================================
function loadState(): ExportState {
if (fs.existsSync(STATE_FILE)) {
return JSON.parse(fs.readFileSync(STATE_FILE, 'utf-8'));
}
return {
user_slug: ARENA_USER_SLUG,
channels: {},
last_channels_fetch: null,
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
};
}
function saveState(state: ExportState): void {
state.updated_at = new Date().toISOString();
fs.writeFileSync(STATE_FILE, JSON.stringify(state, null, 2));
}
function loadBlock(blockId: number): ExportedBlock | null {
const blockFile = path.join(BLOCKS_DIR, `${blockId}.json`);
if (fs.existsSync(blockFile)) {
return JSON.parse(fs.readFileSync(blockFile, 'utf-8'));
}
return null;
}
function saveBlock(block: ExportedBlock): void {
const blockFile = path.join(BLOCKS_DIR, `${block.id}.json`);
fs.writeFileSync(blockFile, JSON.stringify(block, null, 2));
}
// ============================================================================
// API HELPERS
// ============================================================================
async function arenaFetch<T>(endpoint: string): Promise<T> {
const res = await fetch(`${ARENA_API_BASE}${endpoint}`, {
headers: {
'Authorization': `Bearer ${ARENA_TOKEN}`,
'Content-Type': 'application/json',
},
});
if (res.status === 429) {
// Rate limited - wait and retry
const retryAfter = parseInt(res.headers.get('retry-after') || '60', 10);
console.log(` ā³ Rate limited, waiting ${retryAfter}s...`);
await sleep(retryAfter * 1000);
return arenaFetch(endpoint);
}
if (!res.ok) {
throw new Error(`Are.na API error: ${res.status} ${res.statusText}`);
}
return res.json();
}
async function fetchChannels(): Promise<ArenaChannel[]> {
const allChannels: ArenaChannel[] = [];
let page = 1;
while (true) {
const data = await arenaFetch<{ channels: ArenaChannel[] }>(
`/users/${ARENA_USER_SLUG}/channels?per=${PER_PAGE}&page=${page}`
);
if (!data.channels || data.channels.length === 0) break;
allChannels.push(...data.channels);
console.log(` š Fetched ${allChannels.length} channels...`);
if (data.channels.length < PER_PAGE) break;
page++;
await sleep(RATE_LIMIT_MS);
}
return allChannels;
}
async function fetchChannelBlocks(
slug: string,
page: number = 1
): Promise<{ blocks: ArenaBlock[]; hasMore: boolean }> {
const data = await arenaFetch<{ contents: ArenaBlock[]; length: number }>(
`/channels/${slug}/contents?per=${PER_PAGE}&page=${page}`
);
const blocks = (data.contents || []).filter(b => b.class !== 'Channel');
const hasMore = blocks.length === PER_PAGE;
return { blocks, hasMore };
}
async function downloadImage(url: string, blockId: number): Promise<string | null> {
try {
const res = await fetch(url);
if (!res.ok) return null;
const contentType = res.headers.get('content-type') || 'image/jpeg';
const ext = contentType.includes('png') ? 'png'
: contentType.includes('gif') ? 'gif'
: contentType.includes('webp') ? 'webp'
: 'jpg';
const buffer = Buffer.from(await res.arrayBuffer());
const filename = `${blockId}.${ext}`;
const filepath = path.join(IMAGES_DIR, filename);
fs.writeFileSync(filepath, buffer);
return filename;
} catch (err) {
return null;
}
}
function sleep(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}
// ============================================================================
// EXPORT LOGIC
// ============================================================================
function convertBlock(block: ArenaBlock, channelSlug: string): ExportedBlock {
const existing = loadBlock(block.id);
const channels = existing?.channels || [];
if (!channels.includes(channelSlug)) {
channels.push(channelSlug);
}
return {
id: block.id,
title: block.title,
class: block.class,
created_at: block.created_at,
updated_at: block.updated_at,
content: block.content,
description: block.description,
source_url: block.source?.url || null,
image_url: block.image?.original?.url || block.image?.display?.url || null,
image_local: existing?.image_local || null,
attachment_url: block.attachment?.url || null,
embed_url: block.embed?.url || null,
channels,
exported_at: new Date().toISOString(),
};
}
async function exportChannel(
channel: ArenaChannel,
state: ExportState,
options: { backfill?: boolean; downloadImages?: boolean }
): Promise<{ newBlocks: number; updatedBlocks: number }> {
const channelState = state.channels[channel.slug] || {
slug: channel.slug,
title: channel.title,
newest_id: null,
oldest_id: null,
total_exported: 0,
last_updated: new Date().toISOString(),
fully_backfilled: false,
};
let newBlocks = 0;
let updatedBlocks = 0;
let page = 1;
// Track watermarks for this run
let runNewestId: number | null = null;
let runOldestId: number | null = null;
console.log(`\nš ${channel.title} (${channel.slug})`);
console.log(` Length: ${channel.length} blocks`);
if (options.backfill && channelState.fully_backfilled) {
console.log(` ā
Already fully backfilled, skipping`);
return { newBlocks: 0, updatedBlocks: 0 };
}
while (true) {
const { blocks, hasMore } = await fetchChannelBlocks(channel.slug, page);
if (blocks.length === 0) break;
// Process each block immediately (save after each page for resilience)
for (const block of blocks) {
// Track watermarks
if (runNewestId === null || block.id > runNewestId) {
runNewestId = block.id;
}
if (runOldestId === null || block.id < runOldestId) {
runOldestId = block.id;
}
// Skip if already exported (unless updating channels list)
const existing = loadBlock(block.id);
const exported = convertBlock(block, channel.slug);
if (existing) {
// Update channels list if block exists in new channel
if (!existing.channels.includes(channel.slug)) {
saveBlock(exported);
updatedBlocks++;
}
} else {
// New block - download image if requested
if (options.downloadImages && exported.image_url) {
const localImage = await downloadImage(exported.image_url, block.id);
exported.image_local = localImage;
}
saveBlock(exported);
newBlocks++;
}
}
console.log(` Page ${page}: +${blocks.length} blocks (${newBlocks} new, ${updatedBlocks} updated)`);
if (!hasMore) {
// Reached end of channel - mark as fully backfilled
channelState.fully_backfilled = true;
break;
}
// In backfill mode, stop if we've reached already-exported blocks
if (options.backfill && channelState.oldest_id !== null) {
const reachedExisting = blocks.some(b => b.id <= channelState.oldest_id!);
if (reachedExisting) {
console.log(` ā
Reached existing data, stopping backfill`);
break;
}
}
page++;
await sleep(RATE_LIMIT_MS);
}
// Update watermarks ONLY after successful completion
if (runNewestId !== null) {
if (channelState.newest_id === null || runNewestId > channelState.newest_id) {
channelState.newest_id = runNewestId;
}
}
if (runOldestId !== null) {
if (channelState.oldest_id === null || runOldestId < channelState.oldest_id) {
channelState.oldest_id = runOldestId;
}
}
channelState.total_exported += newBlocks;
channelState.last_updated = new Date().toISOString();
state.channels[channel.slug] = channelState;
return { newBlocks, updatedBlocks };
}
// ============================================================================
// MAIN
// ============================================================================
async function main() {
console.log('āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā');
console.log(' ARE.NA INCREMENTAL BLOCK EXPORTER ');
console.log('āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā\n');
// Parse args
const args = process.argv.slice(2);
const backfill = args.includes('--backfill');
const full = args.includes('--full');
const downloadImages = args.includes('--images');
const channelArg = args.find(a => a.startsWith('--channel='));
const targetChannel = channelArg?.split('=')[1];
console.log(`Mode: ${backfill ? 'BACKFILL' : full ? 'FULL' : 'INCREMENTAL'}`);
console.log(`Images: ${downloadImages ? 'YES' : 'NO'}`);
if (targetChannel) console.log(`Channel: ${targetChannel}`);
// Validate env
if (!ARENA_TOKEN || !ARENA_USER_SLUG) {
console.error('\nā Missing ARENA_TOKEN or ARENA_USER_SLUG in .env');
process.exit(1);
}
// Ensure directories exist
fs.mkdirSync(EXPORT_DIR, { recursive: true });
fs.mkdirSync(BLOCKS_DIR, { recursive: true });
if (downloadImages) {
fs.mkdirSync(IMAGES_DIR, { recursive: true });
}
// Load state
const state = loadState();
console.log(`\nš Export directory: ${EXPORT_DIR}`);
console.log(` Existing channels tracked: ${Object.keys(state.channels).length}`);
// Fetch channels
console.log('\nš Fetching channels...');
const channels = await fetchChannels();
console.log(` Found ${channels.length} channels`);
// Save channels list
fs.writeFileSync(CHANNELS_FILE, JSON.stringify(channels, null, 2));
state.last_channels_fetch = new Date().toISOString();
// Filter to target channel if specified
const channelsToExport = targetChannel
? channels.filter(c => c.slug === targetChannel)
: channels.filter(c => c.length > 0);
if (targetChannel && channelsToExport.length === 0) {
console.error(`\nā Channel not found: ${targetChannel}`);
process.exit(1);
}
console.log(`\nš Exporting ${channelsToExport.length} channels...\n`);
// Export each channel
let totalNew = 0;
let totalUpdated = 0;
for (const channel of channelsToExport) {
try {
const { newBlocks, updatedBlocks } = await exportChannel(
channel,
state,
{ backfill, downloadImages }
);
totalNew += newBlocks;
totalUpdated += updatedBlocks;
// Save state after each channel (resilience)
saveState(state);
} catch (err: any) {
console.error(` ā Error: ${err.message}`);
// Save state even on error
saveState(state);
}
}
// Final summary
console.log('\nāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā');
console.log(' COMPLETE ');
console.log('āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā');
console.log(`\nš Summary:`);
console.log(` New blocks exported: ${totalNew}`);
console.log(` Blocks updated: ${totalUpdated}`);
console.log(` Total blocks on disk: ${fs.readdirSync(BLOCKS_DIR).length}`);
console.log(` Channels tracked: ${Object.keys(state.channels).length}`);
console.log(`\nš Export location: ${EXPORT_DIR}`);
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});
```
### scripts/enrich-blocks.ts
```typescript
#!/usr/bin/env npx ts-node
/**
* Are.na Block Enrichment via Vision AI
*
* Analyzes block images with Gemini Vision to generate:
* - Clean, descriptive titles (replacing broken filenames)
* - Descriptions suitable for Are.na
* - Structured searchable tags
* - UI pattern classification
*
* Usage:
* npx ts-node enrich-blocks.ts # First channel with images
* npx ts-node enrich-blocks.ts --channel=slug # Specific channel
* npx ts-node enrich-blocks.ts --all # All channels
* npx ts-node enrich-blocks.ts --force # Re-enrich already processed
* npx ts-node enrich-blocks.ts --dry-run # Preview without saving
*
* Environment:
* GEMINI_API_KEY or GOOGLE_API_KEY - Required for vision analysis
* ARENA_EXPORT_DIR - Export directory (default: ./arena-export)
*/
import 'dotenv/config';
import * as fs from 'fs';
import * as path from 'path';
import * as readline from 'readline';
import { GoogleGenerativeAI } from '@google/generative-ai';
// ============================================================================
// CONFIGURATION
// ============================================================================
const GEMINI_API_KEY = process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY;
const EXPORT_DIR = process.env.ARENA_EXPORT_DIR || path.join(process.cwd(), 'arena-export');
const BLOCKS_DIR = path.join(EXPORT_DIR, 'blocks');
const STATE_FILE = path.join(EXPORT_DIR, 'state.json');
const RATE_LIMIT_MS = 1000; // 1 request per second
// Gemini 3 Pro Image Preview - latest vision model
// Source: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-pro-image
const GEMINI_MODEL = 'gemini-3-pro-image-preview';
// ============================================================================
// TYPES
// ============================================================================
interface VisionEnrichment {
suggested_title: string;
description: string;
tags: string[];
ui_patterns: string[];
enriched_at: string;
model: string;
}
interface ExportedBlock {
id: number;
title: string | null;
class: string;
created_at: string;
updated_at: string;
content: string | null;
description: string | null;
source_url: string | null;
image_url: string | null;
image_local: string | null;
attachment_url: string | null;
embed_url: string | null;
channels: string[];
exported_at: string;
vision?: VisionEnrichment;
}
interface EnrichmentState {
total_enriched: number;
last_enriched_at: string | null;
channels_enriched: Record<string, {
total: number;
enriched: number;
last_block_id: number | null;
}>;
}
interface ExportState {
user_slug: string;
channels: Record<string, any>;
last_channels_fetch: string | null;
created_at: string;
updated_at: string;
enrichment?: EnrichmentState;
}
// ============================================================================
// GEMINI CLIENT
// ============================================================================
function createGeminiClient() {
if (!GEMINI_API_KEY) {
throw new Error('Missing GEMINI_API_KEY or GOOGLE_API_KEY in .env');
}
return new GoogleGenerativeAI(GEMINI_API_KEY);
}
async function analyzeImage(
genAI: GoogleGenerativeAI,
imageUrl: string,
existingContent: string | null
): Promise<VisionEnrichment> {
const model = genAI.getGenerativeModel({ model: GEMINI_MODEL });
const prompt = `Analyze this image for a design reference library (Are.na).
Generate:
1. **suggested_title**: A clean, descriptive title (3-8 words). NOT a filename. Describe what this IS, not what it shows.
- Good: "Dark Mode Trading Dashboard", "Mobile Onboarding Flow", "Stat Card with Progress Bars"
- Bad: "Screenshot", "UI Design", "Image of interface"
2. **description**: 1-2 sentences describing what makes this notable as a design reference. What could someone learn from this?
3. **tags**: 5-15 lowercase tags for searching. Include:
- Layout type: table, card, list, grid, dashboard, modal, form, nav
- Components: avatar, button, input, stat-bar, progress, badge, tag, icon
- Patterns: leaderboard, onboarding, settings, profile, feed, search
- Style: dark-mode, light-mode, minimal, dense, colorful, monochrome
- Platform: mobile, desktop, web, ios, android
4. **ui_patterns**: Specific UI patterns visible (for styling reference):
- Examples: "inline-stats", "avatar-with-name", "tiered-ranking", "progress-percentage", "sortable-headers", "status-pills", "metric-cards"
${existingContent ? `\nExisting content/OCR for context: "${existingContent.slice(0, 500)}"` : ''}
Respond in JSON format only:
{
"suggested_title": "...",
"description": "...",
"tags": ["...", "..."],
"ui_patterns": ["...", "..."]
}`;
try {
// Fetch image and convert to base64
const imageResponse = await fetch(imageUrl);
if (!imageResponse.ok) {
throw new Error(`Failed to fetch image: ${imageResponse.status}`);
}
const imageBuffer = await imageResponse.arrayBuffer();
const base64Image = Buffer.from(imageBuffer).toString('base64');
const mimeType = imageResponse.headers.get('content-type') || 'image/jpeg';
const result = await model.generateContent([
{
inlineData: {
mimeType,
data: base64Image,
},
},
prompt,
]);
const response = result.response.text();
// Extract JSON from response (handle markdown code blocks)
const jsonMatch = response.match(/\{[\s\S]*\}/);
if (!jsonMatch) {
throw new Error('No JSON found in response');
}
const parsed = JSON.parse(jsonMatch[0]);
return {
suggested_title: parsed.suggested_title || 'Untitled',
description: parsed.description || '',
tags: Array.isArray(parsed.tags) ? parsed.tags : [],
ui_patterns: Array.isArray(parsed.ui_patterns) ? parsed.ui_patterns : [],
enriched_at: new Date().toISOString(),
model: GEMINI_MODEL,
};
} catch (error: any) {
// Return minimal enrichment on error
console.error(` ā ļø Vision error: ${error.message}`);
return {
suggested_title: 'Analysis Failed',
description: `Error: ${error.message}`,
tags: ['error'],
ui_patterns: [],
enriched_at: new Date().toISOString(),
model: GEMINI_MODEL,
};
}
}
// ============================================================================
// STATE MANAGEMENT
// ============================================================================
function loadState(): ExportState {
if (fs.existsSync(STATE_FILE)) {
return JSON.parse(fs.readFileSync(STATE_FILE, 'utf-8'));
}
throw new Error('No state.json found. Run export first.');
}
function saveState(state: ExportState): void {
state.updated_at = new Date().toISOString();
fs.writeFileSync(STATE_FILE, JSON.stringify(state, null, 2));
}
function loadBlock(blockId: number): ExportedBlock | null {
const blockFile = path.join(BLOCKS_DIR, `${blockId}.json`);
if (fs.existsSync(blockFile)) {
return JSON.parse(fs.readFileSync(blockFile, 'utf-8'));
}
return null;
}
function saveBlock(block: ExportedBlock): void {
const blockFile = path.join(BLOCKS_DIR, `${block.id}.json`);
fs.writeFileSync(blockFile, JSON.stringify(block, null, 2));
}
function getBlocksForChannel(channelSlug: string): number[] {
const blockIds: number[] = [];
const files = fs.readdirSync(BLOCKS_DIR);
for (const file of files) {
if (!file.endsWith('.json')) continue;
const blockId = parseInt(file.replace('.json', ''), 10);
const block = loadBlock(blockId);
if (block && block.channels.includes(channelSlug)) {
blockIds.push(blockId);
}
}
return blockIds;
}
function getAllImageBlocks(): number[] {
const blockIds: number[] = [];
const files = fs.readdirSync(BLOCKS_DIR);
for (const file of files) {
if (!file.endsWith('.json')) continue;
const blockId = parseInt(file.replace('.json', ''), 10);
const block = loadBlock(blockId);
if (block && block.class === 'Image' && block.image_url) {
blockIds.push(blockId);
}
}
return blockIds;
}
function sleep(ms: number): Promise<void> {
return new Promise(resolve => setTimeout(resolve, ms));
}
// ============================================================================
// MAIN
// ============================================================================
async function main() {
console.log('āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā');
console.log(' ARE.NA BLOCK ENRICHMENT (VISION AI) ');
console.log('āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā\n');
// Parse args
const args = process.argv.slice(2);
const forceReenrich = args.includes('--force');
const dryRun = args.includes('--dry-run');
const allChannels = args.includes('--all');
const channelArg = args.find(a => a.startsWith('--channel='));
// If no channel specified and not --all, process all (user should specify)
const targetChannel = channelArg?.split('=')[1] || null;
console.log(`Mode: ${dryRun ? 'DRY RUN' : 'ENRICH'}`);
console.log(`Force re-enrich: ${forceReenrich ? 'YES' : 'NO'}`);
console.log(`Target: ${targetChannel || 'ALL CHANNELS'}`);
// Initialize Gemini
const genAI = createGeminiClient();
console.log(`\nā
Gemini client initialized`);
// Load state
const state = loadState();
// Initialize enrichment state if needed
if (!state.enrichment) {
state.enrichment = {
total_enriched: 0,
last_enriched_at: null,
channels_enriched: {},
};
}
// Get blocks to process
let blockIds: number[];
if (targetChannel) {
console.log(`\nš Loading blocks from channel: ${targetChannel}`);
blockIds = getBlocksForChannel(targetChannel);
} else {
console.log(`\nš Loading all image blocks`);
blockIds = getAllImageBlocks();
}
// Filter to Image blocks with image_url
const imageBlocks = blockIds
.map(id => loadBlock(id))
.filter((b): b is ExportedBlock => b !== null && b.class === 'Image' && !!b.image_url);
console.log(` Found ${imageBlocks.length} image blocks to process`);
// Filter already enriched (unless force)
const blocksToProcess = forceReenrich
? imageBlocks
: imageBlocks.filter(b => !b.vision);
console.log(` To enrich: ${blocksToProcess.length} blocks`);
if (blocksToProcess.length === 0) {
console.log('\nā
All blocks already enriched. Use --force to re-enrich.');
return;
}
// Process blocks
console.log(`\nš Starting enrichment...\n`);
let enriched = 0;
let errors = 0;
for (let i = 0; i < blocksToProcess.length; i++) {
const block = blocksToProcess[i];
const progress = `[${i + 1}/${blocksToProcess.length}]`;
console.log(`${progress} Block ${block.id}: ${block.title?.slice(0, 40) || '(no title)'}...`);
if (dryRun) {
console.log(` š Would analyze: ${block.image_url?.slice(0, 60)}...`);
continue;
}
try {
// Analyze with Gemini
const vision = await analyzeImage(genAI, block.image_url!, block.content);
// Update block
block.vision = vision;
saveBlock(block);
console.log(` ā
"${vision.suggested_title}"`);
console.log(` Tags: ${vision.tags.slice(0, 5).join(', ')}${vision.tags.length > 5 ? '...' : ''}`);
enriched++;
// Update state
state.enrichment!.total_enriched++;
state.enrichment!.last_enriched_at = new Date().toISOString();
// Track per-channel progress
for (const ch of block.channels) {
if (!state.enrichment!.channels_enriched[ch]) {
state.enrichment!.channels_enriched[ch] = {
total: 0,
enriched: 0,
last_block_id: null,
};
}
state.enrichment!.channels_enriched[ch].enriched++;
state.enrichment!.channels_enriched[ch].last_block_id = block.id;
}
// Save state after each block (resilience)
saveState(state);
} catch (error: any) {
console.log(` ā Error: ${error.message}`);
errors++;
}
// Rate limit
if (i < blocksToProcess.length - 1) {
await sleep(RATE_LIMIT_MS);
}
}
// Summary
console.log('\nāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā');
console.log(' COMPLETE ');
console.log('āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā');
console.log(`\nš Summary:`);
console.log(` Enriched: ${enriched}`);
console.log(` Errors: ${errors}`);
console.log(` Total enriched (all time): ${state.enrichment!.total_enriched}`);
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});
```