Automate and Scale

The weekly loop is powerful, but if you have to remember to run it, you eventually won't. The last move is making the system refresh itself — so the data is fresh before you even sit down. There's a ladder here; climb it as far as you need.

The automation ladder

Rung 1 — /loop (open session). The simplest step up from running things by hand. Inside an open Claude Code session, /loop re-runs a prompt or command on an interval. Great for a hands-on afternoon where you want the scrape to refresh every few minutes while you work.

Rung 2 — cron + claude -p --bare (local). Schedule Claude to run headless. A cron job fires claude -p --bare with your scrape/pull prompt on a schedule — say, every morning at 7am. No session open, no you clicking anything. The catch: it only runs while your machine is on.

Rung 3 — Routines or GitHub Actions (cloud). This is real automation — laptop closed, still running.

• Routines (/schedule) run your agent on a cron in the cloud. Set it once, it fires whether your machine is on or off.
• GitHub Actions with a schedule: trigger does the same for free if you're comfortable with a repo: a scheduled workflow runs the scrape and pull, commits the fresh JSON, and regenerates the dashboard.

Most people land on Rung 3 and never touch it again. The scrape runs nightly, your data/refs and data/mine stay current, and Monday's loop always has fresh numbers waiting.

Scaling the scrape

When you're tracking a handful of accounts, a simple scheduled scrape is plenty. When you scale to hundreds of creators and thousands of posts, you hit a wall: Instagram's rate limit. Burst too many requests and Meta blocks you.

The fix is a durable job queue — I use pg-boss over Postgres (no Redis needed). Instead of firing all the scrape jobs at once, you enqueue them and drain them throttled: one at a time, at a polite pace, with retries, exponential backoff, and a dead-letter queue for the ones that keep failing. The Instagram lane stays serial by design so you never burst past the limit. It's the difference between a scraper that gets you blocked in an hour and one that quietly works through 5,000 posts overnight.

You don't need this on day one. Add it the moment the simple scrape starts choking — you'll know, because the 429 errors show up.

Automation never means auto-posting

This is worth repeating because it's the line that keeps the system trustworthy. You can schedule every piece of the research pipeline to run without you. You never schedule the decision. The machine hands you fresh evidence every morning; you still choose what to make and when to ship it.

Key takeaway

Climb the automation ladder: /loop for hands-on runs, cron + claude -p --bare for local schedules, and cloud Routines or a GitHub Actions schedule: for real hands-off automation. At scale, put a durable queue (pg-boss over Postgres) in front of the scraper so you throttle past Instagram's rate limit. Automate the data — never the decision. Nothing auto-posts.