Skip to content
AI Learning Playground
LinkedInX
English
English

Level 7 Practice: Auto-Collect Blog Ideas from Trending Articles

5 min read + 40 min hands-on

Target audience: Anyone who has completed up to Level 6 with the `/blog` slash command working. Familiarity with basic Node.js operations makes this easier to follow.
Prerequisites: Read Level 6

About This Tutorial

Section titled “About This Tutorial”

If you want to understand the concepts and mechanics first, see the Level 7 concept guide.

I’ll add Puppeteer MCP (Model Context Protocol server) to Claude Code, use browser automation to collect trending articles, and auto-generate blog posts from them. This brings the time spent searching for blog ideas close to zero.

Carrying Over from the Previous Level

Section titled “Carrying Over from the Previous Level”

PR auto-review from Level 6 is complete. I can produce blog articles with /blog, but finding ideas takes time — automation is the next goal.

Step 1: Configure Puppeteer MCP

Section titled “Step 1: Configure Puppeteer MCP”

Add the Puppeteer MCP server to .claude/settings.json. The complete configuration including the existing GitHub MCP is shown below:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
      }
    },
    "puppeteer": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-puppeteer"],
      "env": {
        "PUPPETEER_LAUNCH_OPTIONS": "{\"headless\": true, \"args\": [\"--no-sandbox\"]}"
      }
    }
  },
  "permissions": {
    "allow": [
      "mcp:github:*",
      "mcp:puppeteer:*"
    ]
  }
}

@modelcontextprotocol/server-puppeteer is downloaded automatically when run with npx — no pre-installation needed.

Restart Claude Code to confirm the MCP server loads:

# Exit and restart
exit

claude

If you see something like puppeteer MCP server started in the startup log, you’re good.

Section titled “Step 2: Collect Trends via Browser Automation”

Enter the following prompts sequentially in Claude Code’s interactive mode.

Step 2-1: Collect trending articles

> Open dev.to/t/typescript and get the top 10 trending article titles from this week.

Claude uses Puppeteer MCP to launch a browser and parse the page content. Example output:

Opened dev.to/t/typescript.

This week's trending articles (10 items):
1. "TypeScript 5.5's Inferred Type Predicates Explained"
2. "Stop Using `any`: A Practical Guide to Unknown"
3. "Zod v4 Released: What's New for Runtime Validation"
4. "Branded Types in TypeScript for Domain Modeling"
5. "TypeScript Decorators Are Finally Stable"
6. "React 19 and TypeScript: New Hooks Typing Patterns"
7. "Building Type-Safe APIs with tRPC and Next.js 14"
8. "Why I Switched from Prisma to Drizzle ORM"
9. "TypeScript Performance Tips for Large Codebases"
10. "Satisfies Operator: When to Use It Over As"

Step 2-2: Prioritize for my-portfolio

> Select the top 3 that my-portfolio's blog (focused on Next.js and TypeScript) should cover.

Example output from Claude:

Selected based on my-portfolio's blog direction (Next.js + TypeScript focused):

## High Priority (top 3)

1. **"TypeScript 5.5's Inferred Type Predicates Explained"**
   Reason: New TypeScript feature that can be applied directly in a Next.js project.
   Suggested title: "TypeScript 5.5 Inferred Type Predicates Change How You Write if Statements"

2. **"React 19 and TypeScript: New Hooks Typing Patterns"**
   Reason: The React + TypeScript combination is the core topic of my-portfolio.
   Suggested title: "Using React 19's New Hooks Type-Safely with TypeScript"

3. **"Building Type-Safe APIs with tRPC and Next.js 14"**
   Reason: Directly tied to my-portfolio's tech stack of Next.js 14 + TypeScript.
   Suggested title: "Type-Safe Full-Stack Development with tRPC + Next.js 14"

## Lower Priority (skipped this time)

- "Zod v4 Released": Useful, but may overlap with existing articles on my-portfolio
- "Why I Switched from Prisma to Drizzle ORM": Database-focused, outside the current blog direction

Step 3: Create the Auto-Collection and Post-Generation Script

Section titled “Step 3: Create the Auto-Collection and Post-Generation Script”

Create scripts/collect_and_blog.sh to run the full pipeline from collection to blog post generation in one step.

#!/usr/bin/env bash
# scripts/collect_and_blog.sh
# Collect trends from dev.to and auto-generate blog posts with the /blog command

set -euo pipefail

BLOG_DIR="src/content/posts"
LOG_FILE="logs/collect_and_blog_$(date +%Y%m%d_%H%M%S).log"
mkdir -p logs

echo "=== Auto Blog Idea Collection and Generation ===" | tee "${LOG_FILE}"
echo "Started: $(date)" | tee -a "${LOG_FILE}"
echo "" | tee -a "${LOG_FILE}"

# Step 1: Use Puppeteer MCP to collect trends and output as JSON
echo "Step 1: Collecting trends from dev.to..." | tee -a "${LOG_FILE}"

TREND_PROMPT="Open dev.to/t/typescript and get the top 10 trending articles from this week.
Select the top 3 suitable for my-portfolio (a Next.js + TypeScript focused blog).
Return the result only as the following JSON format (no explanatory text):

{
  \"articles\": [
    {\"rank\": 1, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"},
    {\"rank\": 2, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"},
    {\"rank\": 3, \"title\": \"original article title\", \"blog_title\": \"suggested English article title\", \"reason\": \"reason for selection\"}
  ]
}"

TREND_JSON=$(echo "${TREND_PROMPT}" | claude --print --dangerously-skip-permissions)
echo "Collection results:" | tee -a "${LOG_FILE}"
echo "${TREND_JSON}" | tee -a "${LOG_FILE}"
echo "" | tee -a "${LOG_FILE}"

# Step 2: Parse the JSON and generate each article
echo "Step 2: Generating blog posts..." | tee -a "${LOG_FILE}"

# Extract article titles with jq and loop
TITLES=$(echo "${TREND_JSON}" | jq -r '.articles[].blog_title' 2>/dev/null || echo "")

if [ -z "${TITLES}" ]; then
  echo "ERROR: Failed to parse JSON. Check the log." | tee -a "${LOG_FILE}"
  exit 1
fi

COUNT=0
while IFS= read -r TITLE; do
  COUNT=$((COUNT + 1))
  echo "  Article ${COUNT}: ${TITLE}" | tee -a "${LOG_FILE}"

  BLOG_PROMPT="/blog ${TITLE}"
  echo "${BLOG_PROMPT}" | claude --print --dangerously-skip-permissions >> "${LOG_FILE}" 2>&1

  echo "  Done" | tee -a "${LOG_FILE}"
  echo "" | tee -a "${LOG_FILE}"

  # Brief pause to avoid API rate limits
  sleep 3
done <<< "${TITLES}"

echo "=== Complete ===" | tee -a "${LOG_FILE}"
echo "Posts generated: ${COUNT}" | tee -a "${LOG_FILE}"
echo "Log: ${LOG_FILE}" | tee -a "${LOG_FILE}"

Grant execute permission and run:

chmod +x scripts/collect_and_blog.sh
./scripts/collect_and_blog.sh

After it completes, confirm that new article files have been generated in src/content/posts/:

ls -lt src/content/posts/ | head -5

Step 4: Verify the Generated Output

Section titled “Step 4: Verify the Generated Output”
# Check the latest article
LATEST=$(ls -t src/content/posts/*.md 2>/dev/null | head -1)
if [ -n "${LATEST}" ]; then
  head -20 "${LATEST}"
fi

Preview locally:

npm run dev
# Check the blog list at http://localhost:4321/blog

Notes on Scraping

Section titled “Notes on Scraping”

Before scraping, confirm the following:

CheckDetails
robots.txtCheck dev.to/robots.txt to confirm which paths allow crawling
Terms of ServiceCheck whether the service’s ToS permits scraping
Request frequencyDon’t send large volumes of requests in a short time (the sleep 3 in the script is a minimal courtesy)
CopyrightUse collected article titles only as inspiration; write original content

Puppeteer MCP operates the browser directly, which technically bypasses robots.txt restrictions — but complying with the service’s terms of service is required.

The my-portfolio at This Point

Section titled “The my-portfolio at This Point”
  • Puppeteer MCP server added to .claude/settings.json
  • scripts/collect_and_blog.sh added — runs the full pipeline from trend collection to post generation
  • logs/ directory stores execution logs
  • Trend-based blog articles auto-generated in src/content/posts/
my-portfolio/
├── .claude/
│   └── settings.json         # ← Puppeteer MCP added
├── scripts/
│   ├── review_pr.sh
│   └── collect_and_blog.sh   # ← newly added
├── logs/                     # ← newly added
│   └── collect_and_blog_*.log
├── src/
│   └── content/
│       └── posts/
│           ├── typescript-type-inference.md   # existing
│           ├── nextjs-app-router.md           # existing
│           ├── react-useeffect.md             # existing
│           └── typescript-55-inferred-*.md   # ← newly generated
└── ...

Next Level

Section titled “Next Level”

Level 8 Practice: Implement 3 Features in Parallel with Git Worktree

See the references for the external specifications and background sources used on this page.[1][2]

References

Section titled “References”
  1. Anthropic, Claude Code documentation
  2. Anthropic, Claude API documentation