Skip to content
AI Learning Playground Blog
X
English

Level 7 Practice: Auto-Collect Blog Ideas from Trending Articles

About This Tutorial

Section titled “About This Tutorial”

If you want to understand the concepts and mechanics first, see the Level 7 concept guide.

I’ll add Puppeteer MCP (Model Context Protocol server) to Claude Code, use browser automation to collect trending articles, and auto-generate blog posts from them. This brings the time spent searching for blog ideas close to zero.

Who this is for: Anyone who has completed up to Level 6 with the /blog slash command working. Familiarity with basic Node.js operations makes this easier to follow.

Estimated time: 10 min read + 40 min hands-on

Carrying Over from the Previous Level

Section titled “Carrying Over from the Previous Level”

PR auto-review from Level 6 is complete. I can produce blog articles with /blog, but finding ideas takes time — automation is the next goal.

Step 1: Configure Puppeteer MCP

Section titled “Step 1: Configure Puppeteer MCP”

Add the Puppeteer MCP server to .claude/settings.json. The complete configuration including the existing GitHub MCP is shown below:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
      }
    },
    "puppeteer": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-puppeteer"],
      "env": {
        "PUPPETEER_LAUNCH_OPTIONS": "{\"headless\": true, \"args\": [\"--no-sandbox\"]}"
      }
    }
  },
  "permissions": {
    "allow": [
      "mcp:github:*",
      "mcp:puppeteer:*"
    ]
  }
}

@modelcontextprotocol/server-puppeteer is downloaded automatically when run with npx — no pre-installation needed.

Restart Claude Code to confirm the MCP server loads:

# Exit and restart
exit

claude

If you see something like puppeteer MCP server started in the startup log, you’re good.

Section titled “Step 2: Collect Trends via Browser Automation”

Enter the following prompts sequentially in Claude Code’s interactive mode.

Step 2-1: Collect trending articles

> Open dev.to/t/typescript and get the top 10 trending article titles from this week.

Claude uses Puppeteer MCP to launch a browser and parse the page content. Example output:

Opened dev.to/t/typescript.

This week's trending articles (10 items):
1. "TypeScript 5.5's Inferred Type Predicates Explained"
2. "Stop Using `any`: A Practical Guide to Unknown"
3. "Zod v4 Released: What's New for Runtime Validation"
4. "Branded Types in TypeScript for Domain Modeling"
5. "TypeScript Decorators Are Finally Stable"
6. "React 19 and TypeScript: New Hooks Typing Patterns"
7. "Building Type-Safe APIs with tRPC and Next.js 14"
8. "Why I Switched from Prisma to Drizzle ORM"
9. "TypeScript Performance Tips for Large Codebases"
10. "Satisfies Operator: When to Use It Over As"

Step 2-2: Prioritize for my-portfolio

> Select the top 3 that my-portfolio's blog (focused on Next.js and TypeScript) should cover.

Example output from Claude:

Selected based on my-portfolio's blog direction (Next.js + TypeScript focused):

## High Priority (top 3)

1. **"TypeScript 5.5's Inferred Type Predicates Explained"**
   Reason: New TypeScript feature that can be applied directly in a Next.js project.
   Suggested title: "TypeScript 5.5 Inferred Type Predicates Change How You Write if Statements"

2. **"React 19 and TypeScript: New Hooks Typing Patterns"**
   Reason: The React + TypeScript combination is the core topic of my-portfolio.
   Suggested title: "Using React 19's New Hooks Type-Safely with TypeScript"

3. **"Building Type-Safe APIs with tRPC and Next.js 14"**
   Reason: Directly tied to my-portfolio's tech stack of Next.js 14 + TypeScript.
   Suggested title: "Type-Safe Full-Stack Development with tRPC + Next.js 14"

## Lower Priority (skipped this time)

- "Zod v4 Released": Useful, but may overlap with existing articles on my-portfolio
- "Why I Switched from Prisma to Drizzle ORM": Database-focused, outside the current blog direction

Step 3: Create the Auto-Collection and Post-Generation Script

Section titled “Step 3: Create the Auto-Collection and Post-Generation Script”

Create scripts/collect_and_blog.sh to run the full pipeline from collection to blog post generation in one step.

#!/usr/bin/env bash
# scripts/collect_and_blog.sh
# Collect trends from dev.to and auto-generate blog posts with the /blog command

set -euo pipefail

BLOG_DIR="src/content/posts"
LOG_FILE="logs/collect_and_blog_$(date +%Y%m%d_%H%M%S).log"
mkdir -p logs

echo "=== Auto Blog Idea Collection and Generation ===" | tee "${LOG_FILE}"
echo "Started: $(date)" | tee -a "${LOG_FILE}"
echo "" | tee -a "${LOG_FILE}"

# Step 1: Use Puppeteer MCP to collect trends and output as JSON
echo "Step 1: Collecting trends from dev.to..." | tee -a "${LOG_FILE}"

TREND_PROMPT="Open dev.to/t/typescript and get the top 10 trending articles from this week.
Select the top 3 suitable for my-portfolio (a Next.js + TypeScript focused blog).
Return the result only as the following JSON format (no explanatory text):

{
  \"articles\": [
    {\"rank\": 1, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"},
    {\"rank\": 2, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"},
    {\"rank\": 3, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"}
  ]
}"

TREND_JSON=$(echo "${TREND_PROMPT}" | claude --print --dangerously-skip-permissions)
echo "Collection results:" | tee -a "${LOG_FILE}"
echo "${TREND_JSON}" | tee -a "${LOG_FILE}"
echo "" | tee -a "${LOG_FILE}"

# Step 2: Parse the JSON and generate each article
echo "Step 2: Generating blog posts..." | tee -a "${LOG_FILE}"

# Extract article titles with jq and loop
TITLES=$(echo "${TREND_JSON}" | jq -r '.articles[].blog_title' 2>/dev/null || echo "")

if [ -z "${TITLES}" ]; then
  echo "ERROR: Failed to parse JSON. Check the log." | tee -a "${LOG_FILE}"
  exit 1
fi

COUNT=0
while IFS= read -r TITLE; do
  COUNT=$((COUNT + 1))
  echo "  Article ${COUNT}: ${TITLE}" | tee -a "${LOG_FILE}"

  BLOG_PROMPT="/blog ${TITLE}"
  echo "${BLOG_PROMPT}" | claude --print --dangerously-skip-permissions >> "${LOG_FILE}" 2>&1

  echo "  Done" | tee -a "${LOG_FILE}"
  echo "" | tee -a "${LOG_FILE}"

  # Brief pause to avoid API rate limits
  sleep 3
done <<< "${TITLES}"

echo "=== Complete ===" | tee -a "${LOG_FILE}"
echo "Posts generated: ${COUNT}" | tee -a "${LOG_FILE}"
echo "Log: ${LOG_FILE}" | tee -a "${LOG_FILE}"

Grant execute permission and run:

chmod +x scripts/collect_and_blog.sh
./scripts/collect_and_blog.sh

After it completes, confirm that new article files have been generated in src/content/posts/:

ls -lt src/content/posts/ | head -5

Step 4: Verify the Generated Output

Section titled “Step 4: Verify the Generated Output”
# Check the latest article
LATEST=$(ls -t src/content/posts/*.md 2>/dev/null | head -1)
if [ -n "${LATEST}" ]; then
  head -20 "${LATEST}"
fi

Preview locally:

npm run dev
# Check the blog list at http://localhost:4321/blog

Notes on Scraping

Section titled “Notes on Scraping”

Before scraping, confirm the following:

CheckDetails
robots.txtCheck dev.to/robots.txt to confirm which paths allow crawling
Terms of ServiceCheck whether the service’s ToS permits scraping
Request frequencyDon’t send large volumes of requests in a short time (the sleep 3 in the script is a minimal courtesy)
CopyrightUse collected article titles only as inspiration; write original content

Puppeteer MCP operates the browser directly, which technically bypasses robots.txt restrictions — but complying with the service’s terms of service is required.

The my-portfolio at This Point

Section titled “The my-portfolio at This Point”
  • Puppeteer MCP server added to .claude/settings.json
  • scripts/collect_and_blog.sh added — runs the full pipeline from trend collection to post generation
  • logs/ directory stores execution logs
  • Trend-based blog articles auto-generated in src/content/posts/
my-portfolio/
├── .claude/
│   └── settings.json         # ← Puppeteer MCP added
├── scripts/
│   ├── review_pr.sh
│   └── collect_and_blog.sh   # ← newly added
├── logs/                     # ← newly added
│   └── collect_and_blog_*.log
├── src/
│   └── content/
│       └── posts/
│           ├── typescript-type-inference.md   # existing
│           ├── nextjs-app-router.md           # existing
│           ├── react-useeffect.md             # existing
│           └── typescript-55-inferred-*.md   # ← newly generated
└── ...

Next Level

Section titled “Next Level”

Level 8 Practice: Implement 3 Features in Parallel with Git Worktree