Level 7 Practice: Auto-Collect Blog Ideas from Trending Articles

About This Tutorial

This tutorial focuses on the practice steps and completion checks for this level.

---

I’ll add Puppeteer MCP (Model Context Protocol server) to Claude Code, use browser automation to collect trending articles, and auto-generate blog posts from them. This brings the time spent searching for blog ideas close to zero.

---

Carrying Over from the Previous Level

PR auto-review from Level 6 is complete. I can produce blog articles with /blog, but finding ideas takes time — automation is the next goal.

---

Step 1: Configure Puppeteer MCP

Add the Puppeteer MCP server to .claude/settings.json. The complete configuration including the existing GitHub MCP is shown below:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
      }
    },
    "puppeteer": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-puppeteer"],
      "env": {
        "PUPPETEER_LAUNCH_OPTIONS": "{\"headless\": true, \"args\": [\"--no-sandbox\"]}"
      }
    }
  },
  "permissions": {
    "allow": [
      "mcp:github:*",
      "mcp:puppeteer:*"
    ]
  }
}

@modelcontextprotocol/server-puppeteer is downloaded automatically when run with npx — no pre-installation needed.

Restart Claude Code to confirm the MCP server loads:

# Exit and restart
exit

claude

If you see something like puppeteer MCP server started in the startup log, you’re good.

---

Step 2: Collect Trends via Browser Automation

Enter the following prompts sequentially in Claude Code’s interactive mode.

Step 2-1: Collect trending articles

> Open dev.to/t/typescript and get the top 10 trending article titles from this week.

Claude uses Puppeteer MCP to launch a browser and parse the page content. Example output:

Opened dev.to/t/typescript.

This week's trending articles (10 items):
1. "TypeScript 5.5's Inferred Type Predicates Explained"
2. "Stop Using `any`: A Practical Guide to Unknown"
3. "Zod v4 Released: What's New for Runtime Validation"
4. "Branded Types in TypeScript for Domain Modeling"
5. "TypeScript Decorators Are Finally Stable"
6. "React 19 and TypeScript: New Hooks Typing Patterns"
7. "Building Type-Safe APIs with tRPC and Next.js 14"
8. "Why I Switched from Prisma to Drizzle ORM"
9. "TypeScript Performance Tips for Large Codebases"
10. "Satisfies Operator: When to Use It Over As"

Step 2-2: Prioritize for my-portfolio

> Select the top 3 that my-portfolio's blog (focused on Next.js and TypeScript) should cover.

Example output from Claude:

Selected based on my-portfolio's blog direction (Next.js + TypeScript focused):

## High Priority (top 3)

1. **"TypeScript 5.5's Inferred Type Predicates Explained"**
   Reason: New TypeScript feature that can be applied directly in a Next.js project.
   Suggested title: "TypeScript 5.5 Inferred Type Predicates Change How You Write if Statements"

2. **"React 19 and TypeScript: New Hooks Typing Patterns"**
   Reason: The React + TypeScript combination is the core topic of my-portfolio.
   Suggested title: "Using React 19's New Hooks Type-Safely with TypeScript"

3. **"Building Type-Safe APIs with tRPC and Next.js 14"**
   Reason: Directly tied to my-portfolio's tech stack of Next.js 14 + TypeScript.
   Suggested title: "Type-Safe Full-Stack Development with tRPC + Next.js 14"

## Lower Priority (skipped this time)

- "Zod v4 Released": Useful, but may overlap with existing articles on my-portfolio
- "Why I Switched from Prisma to Drizzle ORM": Database-focused, outside the current blog direction

---

Step 3: Create the Auto-Collection and Post-Generation Script

Create scripts/collect_and_blog.sh to run the full pipeline from collection to blog post generation in one step.

#!/usr/bin/env bash
# scripts/collect_and_blog.sh
# Collect trends from dev.to and auto-generate blog posts with the /blog command

set -euo pipefail

BLOG_DIR="src/content/posts"
LOG_FILE="logs/collect_and_blog_$(date +%Y%m%d_%H%M%S).log"
mkdir -p logs

echo "=== Auto Blog Idea Collection and Generation ===" | tee "${LOG_FILE}"
echo "Started: $(date)" | tee -a "${LOG_FILE}"
echo "" | tee -a "${LOG_FILE}"

# Step 1: Use Puppeteer MCP to collect trends and output as JSON
echo "Step 1: Collecting trends from dev.to..." | tee -a "${LOG_FILE}"

TREND_PROMPT="Open dev.to/t/typescript and get the top 10 trending articles from this week.
Select the top 3 suitable for my-portfolio (a Next.js + TypeScript focused blog).
Return the result only as the following JSON format (no explanatory text):

{
  \"articles\": [
    {\"rank\": 1, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"},
    {\"rank\": 2, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"},
    {\"rank\": 3, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"}
  ]
}"

TREND_JSON=$(echo "${TREND_PROMPT}" | claude --print --dangerously-skip-permissions)
echo "Collection results:" | tee -a "${LOG_FILE}"
echo "${TREND_JSON}" | tee -a "${LOG_FILE}"
echo "" | tee -a "${LOG_FILE}"

# Step 2: Parse the JSON and generate each article
echo "Step 2: Generating blog posts..." | tee -a "${LOG_FILE}"

# Extract article titles with jq and loop
TITLES=$(echo "${TREND_JSON}" | jq -r '.articles[].blog_title' 2>/dev/null || echo "")

if [ -z "${TITLES}" ]; then
  echo "ERROR: Failed to parse JSON. Check the log." | tee -a "${LOG_FILE}"
  exit 1
fi

COUNT=0
while IFS= read -r TITLE; do
  COUNT=$((COUNT + 1))
  echo "  Article ${COUNT}: ${TITLE}" | tee -a "${LOG_FILE}"

  BLOG_PROMPT="/blog ${TITLE}"
  echo "${BLOG_PROMPT}" | claude --print --dangerously-skip-permissions >> "${LOG_FILE}" 2>&1

  echo "  Done" | tee -a "${LOG_FILE}"
  echo "" | tee -a "${LOG_FILE}"

  # Brief pause to avoid API rate limits
  sleep 3
done <<< "${TITLES}"

echo "=== Complete ===" | tee -a "${LOG_FILE}"
echo "Posts generated: ${COUNT}" | tee -a "${LOG_FILE}"
echo "Log: ${LOG_FILE}" | tee -a "${LOG_FILE}"

Grant execute permission and run:

chmod +x scripts/collect_and_blog.sh
./scripts/collect_and_blog.sh

After it completes, confirm that new article files have been generated in src/content/posts/:

ls -lt src/content/posts/ | head -5

---

Step 4: Verify the Generated Output

# Check the latest article
LATEST=$(ls -t src/content/posts/*.md 2>/dev/null | head -1)
if [ -n "${LATEST}" ]; then
  head -20 "${LATEST}"
fi

Preview locally:

npm run dev
# Check the blog list at http://localhost:4321/blog

---

Notes on Scraping

Before scraping, confirm the following:

Scroll horizontally→

Check	Details
robots.txt	Check dev.to/robots.txt to confirm which paths allow crawling
Terms of Service	Check whether the service’s ToS permits scraping
Request frequency	Don’t send large volumes of requests in a short time (the sleep 3 in the script is a minimal courtesy)
Copyright	Use collected article titles only as inspiration; write original content

This table scrolls horizontally. Keyboard users can focus the table and use the left and right arrow keys.

Puppeteer MCP operates the browser directly, which technically bypasses robots.txt restrictions — but complying with the service’s terms of service is required.

---

The my-portfolio at This Point

• Puppeteer MCP server added to .claude/settings.json
• scripts/collect_and_blog.sh added — runs the full pipeline from trend collection to post generation
• logs/ directory stores execution logs
• Trend-based blog articles auto-generated in src/content/posts/

my-portfolio/
├── .claude/
│   └── settings.json         # ← Puppeteer MCP added
├── scripts/
│   ├── review_pr.sh
│   └── collect_and_blog.sh   # ← newly added
├── logs/                     # ← newly added
│   └── collect_and_blog_*.log
├── src/
│   └── content/
│       └── posts/
│           ├── typescript-type-inference.md   # existing
│           ├── nextjs-app-router.md           # existing
│           ├── react-useeffect.md             # existing
│           └── typescript-55-inferred-*.md   # ← newly generated
└── ...

---

Next Level

Level 8 Practice: Implement 3 Features in Parallel with Git Worktree

The automation pattern follows Anthropic’s Claude Code documentation, and the browser automation examples refer to Puppeteer documentation.[1][2]

References

1. Anthropic, Claude Code documentation
2. Anthropic, Claude API documentation