Build a Free Local AI Website Scraper with n8n

Overview & Architecture

Tracking competitor pages or news by hand is repetitive, and paid scraping tools add up. This blueprint loads a page in a headless Chrome browser you run yourself (Browserless), cleans the text in an n8n Code node, and asks a local Ollama model for a short summary and a few bullet findings. The result is saved in your own Postgres table with the source URL and a timestamp.

The workflow runs on a daily schedule, or on demand with a manual trigger. Only the page you scrape is fetched from the internet. The summarizing and storing stay on your own hardware.

The test run scraped the Hacker News front page with qwen2.5:14b-instruct-q5_K_M and inserted one row with a generated title, summary and bullet findings.

The Pipeline

1. Triggers: A Daily Schedule (07:00) and a Manual Demo Run trigger both feed the same flow.
2. Scrape Settings: One Set node holds the target URL, the scraping goal and the table name, so retargeting means editing three values.
3. Create Intel Table: A Postgres Execute Query node runs CREATE TABLE IF NOT EXISTS for the table (id, source URL, scraped time, title, summary, findings).
4. Fetch Page with Browserless: An HTTP Request posts a small script to the Browserless /function endpoint. The script opens the page, waits for the network to go idle and returns the rendered HTML.
5. Clean Page Text: A Code node removes script, style, nav, footer and header blocks, decodes common HTML entities, stops with a clear error if almost no text is left, and caps the text at 12,000 characters.
6. Summarize with Ollama: A Basic LLM Chain with an Ollama Chat Model (temperature 0.2, 8192 context) returns JSON with a title, a summary and bullets.
7. Format Row: A Code node parses the JSON (with a raw-text fallback) and builds one row.
8. Insert Intel Row: A parameterized Postgres insert that returns the new row id.
9. Insert Succeeded?: An If node sends successful inserts to a Success node and failed inserts to a Debug node that records the error message and time.

What You Need

• Docker with n8n, Browserless, Ollama and Postgres running on a shared network (built and tested on n8n 2.43.5)
• A local Ollama text model such as qwen2.5:14b-instruct-q5_K_M, or a lighter one such as gemma4:12b-it-qat
• A Postgres database and an n8n Postgres credential for it
• A Browserless token, set up as an n8n credential

Key Deployment Hurdles & Solutions

• Use Browserless’s function endpoint: The workflow posts a script to /function and reads the HTML from the data field of the response. The assistant prompt shows the exact request body.
• Context limits: Large pages are cut to 12,000 characters in the Code node before they reach the model, and the model runs with an 8192-token context.
• Blocked or empty pages: Some sites block headless browsers or render content late. The cleanup code throws a clear error when almost no text comes back, so you do not store an empty summary.
• Model output is a draft: The model is asked for JSON and the format step falls back to the raw text if it cannot be parsed. Treat summaries as drafts.
• Error branch coverage: Only Postgres insert failures reach the debug branch. A page that returns no text stops the run with a visible error in Executions.
• Public pages only: Scraping logged-in or paywalled content is out of scope and may break a site’s terms.

Download the Workflow (.json)

Import-ready n8n workflow with all 18 nodes (13 working nodes plus stage labels). In n8n choose Import from file, attach your own Postgres, Browserless and Ollama credentials, then set the target URL, goal and table name in the Scrape Settings node.

Download the n8n workflow (.json)