SiteIntel dossier

Product summary

Fact: SiteIntel is a domain-intelligence application that turns a domain into a progressively built report using web collection, DNS/HTTP/TLS checks, optional APIs, stored artefacts, and operational summaries.

Fact: The project dossier describes SiteIntel as in-development and in private-alpha.

Intended users

• Small business owners who need plain-English website health guidance.
• Agencies and SEO/marketing consultants monitoring client and competitor domains.
• Web developers catching deployment regressions.
• SaaS founders watching their own site and competitor public changes.

Problem being solved

Site owners and marketers often have scattered signals: DNS, TLS, redirects, headers, robots, sitemaps, metadata, performance, analytics tags, competitor changes, and external reference feeds. SiteIntel aims to collect those signals, retain evidence, and explain what matters without becoming a full crawler, security scanner, or generic SEO suite.

Current status

Fact: The current architecture docs describe a Cloudflare-hosted public web app, Cloudflare-hosted admin app, Cloudflare docs site, Google Cloud Run source/browser/BigQuery/domain-import workers, a Cloudflare Durable Objects progress worker, Supabase control-plane data, Cloudflare R2 raw artefact storage, and Google Cloud Tasks for queueing.

Fact: Older README text still mentions Firestore and Cloud Storage as starting assumptions, while the current architecture page says Supabase and R2 are current with Firestore/GCS-compatible contracts retained.

Inference: The implementation has evolved from the original Firestore/GCS shape toward a provider-abstracted runtime. AI assistants should prefer the newer architecture and deployable-unit docs when the README conflicts.

Built functionality

• Domain validation and canonical domain identity.
• Idempotent /api/domain/ensure flow that validates and queues work.
• Task registry with cost tier, TTL, queue, worker target, dependencies, retry policy, and enabled flags.
• Source tasks for DNS, HTTP headers, robots, homepage extraction, social metadata, schema, security headers, sitemap parsing, PageSpeed, and other free/paid sources.
• Worker lock checking through queued lockId payloads.
• Raw artefact and source-summary separation.
• Admin app for operational inspection.
• BigQuery worker responsibilities for source-task export, domain registry, public suffix list sync, bulk ingest, and BigQuery catalogue discovery.
• Durable Objects progress worker for live report state.

Key apps, workers, packages, and services

Surface	Role
apps/websites/web	Public domain report app and validation handoff.
apps/websites/admin	Internal operations app.
apps/websites/docs	Private documentation site.
apps/workers/worker	Main source collection worker on Cloud Run.
apps/workers/browser-worker	Browser-heavy collection worker on Cloud Run.
apps/workers/bigquery-worker	Export, domain registry, bulk ingest, and catalogue worker.
apps/workers/domain-import-worker	Domain import and batch queueing worker.
apps/workers/progress-worker	Cloudflare Durable Objects progress state.
packages/works/core	Cloud-agnostic domain contracts, task decisions, schemas, helpers.
packages/works/gcp and packages/runtime	Provider/runtime adapters and orchestration helpers.

Data sources and integrations

Fact: Free/no-key sources include DNS, RDAP, TLS, HTTP headers, robots.txt, sitemap parsing, auxiliary files, homepage extraction, social metadata, schema extraction, security headers, tech-stack detection, and limited crawling.

Fact: Planned or optional sources include Google PageSpeed Insights, Internet Archive CDX, Cloudflare Radar, Cloudflare Intel Domains, Mozilla Observatory, SSL Labs, OpenPageRank, DataForSEO, AI summaries, and derived change-detection tasks.

Fact: Bulk ingest sources include domain ranking, security, and reference datasets such as Tranco, Majestic Million, Cisco Umbrella, Cloudflare Radar, PhishTank, URLHaus, Public Suffix List, IANA root database, and OpenPageRank template sources.

Deployment and runtime assumptions

• Public/admin/docs surfaces run on Cloudflare Workers through OpenNext or Docusaurus Worker wrappers.
• Heavy source and browser tasks run on Google Cloud Run.
• Google Cloud Tasks is the bounded queue.
• Supabase/Postgres is the current control plane in newer docs.
• R2 is the current raw artefact store in newer docs.
• BigQuery is used for exports, domain registry, public/reference bulk ingest, and catalogue discovery.
• Paid tasks require explicit enablement to control spend.

Reusable capabilities

Capability	Reuse potential	Notes
Task registry and dependency-aware queueing	High	Could become a shared pattern for any product that has small source tasks and dependent expensive work.
Worker lock and retry policy	High	Also appears conceptually in Bollocks Radar and DataStitcher.
Domain normalisation and Public Suffix List registry	High	Useful for SiteIntel, Trackpal agency domains, public audit tools, and any domain intelligence product.
Story report and findings model	Medium to high	Product-specific wording, but finding shape and evidence pointers are reusable.
Bulk ingest ops schema	Medium	Strong reuse, but currently tied to SiteIntel source classes and BigQuery outputs.
Admin source health views	High	Similar operational needs appear across DataStitcher, Trackpal, BigQuery Atlas, and Bollocks Radar.

Notable design patterns

• Public routes validate and queue only.
• Private workers own source collection and raw payload writes.
• Raw payloads stay in object storage; control-plane data stores summaries, state, and pointers.
• Every task type is separate; there is no one giant scan task.
• Multiple monitors can share collection work when freshness requirements overlap.
• Competitor expansion is capped to prevent runaway queueing and paid API spend.

Dependencies on other products or Meridian tooling

Fact: SiteIntel uses Meridian deployment/config conventions and generated operations docs.

Inference: SiteIntel should be a proving ground for a future shared source-task orchestration pattern, while Meridian core remains responsible only for generic config, deployment, and validation tooling.

Risks and unknowns

• Conflicting README and current architecture docs around Firestore/GCS versus Supabase/R2.
• Paid API tasks need strict default-off behaviour, quotas, and operator visibility.
• Public domain input requires SSRF, redirect, DNS rebinding, size, timeout, and private-network guardrails.
• AI summaries must not invent evidence or inspect raw payloads outside deterministic findings.
• Full-site crawling is explicitly out of scope for early SaaS and should not creep in by accident.

Near-term next steps

• Update project metadata so the generated dossier reflects the current Supabase/R2/provider-abstracted state.
• Keep the findings library deterministic before adding AI rewriting.
• Harden paid-source enablement and cost visibility before broad beta.
• Identify whether domain normalisation and task orchestration should stay local, become a documented pattern, or move toward a shared Works package.

Longer-term product potential

Opportunity: SiteIntel is a strong SaaS candidate because it has a focused wedge: website smoke alarms, owner-friendly reports, agency portfolio monitoring, and competitor change digests.

Opportunity: A free one-off domain checker can be a lead magnet for paid monitoring, agency workflows, or developer regression checks.

AI notes

• Read current architecture docs before README when provider choices matter.
• Never add broad crawls, authenticated crawling, form submission, or vulnerability scanning unless explicitly requested.
• Keep public report copy evidence-backed and cautious.
• Preserve raw-payload boundaries.
• Treat paid providers as disabled unless config and product docs say otherwise.

Source coverage

The dossier is well-supported by current docs, but the generated metadata should be refreshed because some repository facts have changed since the dossier was generated.

Sources consulted

• projects/site-intel/README.md
• projects/site-intel/apps/websites/docs/metadata/project-dossier.json
• projects/site-intel/docs/overview/product/product-overview.mdx
• projects/site-intel/docs/overview/product/saas-product-strategy.mdx
• projects/site-intel/docs/architecture/architecture-overview.mdx
• projects/site-intel/docs/architecture/task-orchestration.mdx
• projects/site-intel/docs/architecture/data-source-registry.mdx
• projects/site-intel/docs/codebase/apps-workers-packages.mdx
• projects/site-intel/docs/operations/deployable-units.mdx
• projects/site-intel/.ideas/siteintel-domain-registry/README.md
• projects/site-intel/.ideas/siteintel-site-map-analyser/README.md
• projects/site-intel/.ideas/siteintel-bulk-ingestor-resumable/README.md
• projects/site-intel/docs/reference/technical/bigquery-domain-registry-and-bulk-ingest.mdx