BigQuery Atlas dossier

Product summary

Fact: BigQuery Atlas discovers evidence-backed public BigQuery project IDs, cautiously validates which projects are publicly readable, catalogues datasets/tables/schemas, and publishes the catalogue through Cloudflare-hosted public/admin/catalogue surfaces.

Fact: The project dossier marks BigQuery Atlas as in-development and in catalogue-build.

Intended users

• Data engineers and analysts looking for public BigQuery datasets.
• Researchers monitoring public dataset availability and schema changes.
• Scott's future data products that need source discovery and catalogue inputs.
• AI assistants needing a reliable inventory of public BigQuery metadata.

Problem being solved

Public BigQuery datasets are useful, but project discovery is noisy and unsafe if implemented as brute-force scanning. BigQuery Atlas separates candidate discovery, evidence scoring, cautious validation, schema cataloguing, and read-side publication.

Current status

Fact: The repository has public, admin, catalogue-docs, crawler, cataloguer, scheduler, D1 read-cache, and optional Postgres backend surfaces documented.

Fact: The public catalogue site is Cloudflare-hosted; crawler and cataloguer work run as bounded Google Cloud Run services; D1 is the default read-side store; R2 stores full table column JSON payloads for D1; Postgres/Neon remains selectable for heavier analysis.

Known aims and strategic intent

• Position as change intelligence for the BigQuery public data ecosystem, not just a directory.
• Never brute-force project IDs.
• Require evidence before validation.
• Track dataset freshness, schema changes, provider/category digests, and related dataset discovery.
• Keep D1 cheap for public/runtime reads while retaining Postgres for richer write-side or analytical work.

Built functionality

Fact: Source docs describe:

• Candidate crawler that rotates search strategies, extracts BigQuery-looking identifiers, scores evidence, dedupes, and exports candidate tables.
• Public cataloguer that uses BigQuery metadata APIs to validate datasets/tables/schemas for evidence-backed candidates.
• D1 current-state catalogue backend with R2-backed column payloads.
• Postgres backend support for heavier analysis.
• Cloudflare public/admin/catalogue-docs surfaces.
• Bounded Cloud Run cataloguer endpoint for admin-triggered schema checks.
• Rate-limit policy and cooldown rules.

Key apps, workers, packages, and services

Surface	Role
apps/websites/public	Public BigQuery catalogue website.
apps/websites/admin	Cloudflare Access-protected operational dashboard.
apps/websites/catalogue-docs	Generated catalogue docs site.
services/candidate-crawler	Evidence-backed candidate discovery.
services/public-cataloguer	BigQuery metadata validation and catalogue refresh.
packages/bigquery-identifiers	Planned pure identifier extraction/scoring package.
packages/db-contract	Planned database/table contract package.
packages/types	Domain contracts.
packages/postgres-state	Postgres state support.
db/d1	Cloudflare D1 read cache schema/migrations.

Data sources and integrations

Fact: Candidate discovery providers include GitHub Code Search, Brave Search API, Google Programmable Search JSON API, DataForSEO Google/Bing/Dataset Search, Common Crawl CDX, Stack Exchange API, OpenAlex, and manual seeds.

Fact: Validation uses BigQuery metadata APIs such as datasets.list, datasets.get, tables.list, and tables.get; INFORMATION_SCHEMA is optional/future because it can incur query charges and is region-scoped.

Fact: Runtime providers include Cloudflare Pages/Workers/D1/R2, Google Cloud Run, Google Cloud Scheduler, BigQuery, GCP Secret Manager, and optional Postgres/Neon.

Deployment and runtime assumptions

• Cloudflare D1 is the default read cache for public/admin runtime reads.
• R2 stores full table column JSON for D1-backed tables.
• Postgres remains selectable for large or exploratory runs.
• Cloud Run schema checks are bounded and should run at low concurrency until stronger distributed leasing exists.
• Admin-triggered rechecks need confirmation, batch caps, cooldown override reasons, audit events, and rate-limit checks.

Reusable capabilities

Capability	Reuse potential	Notes
Evidence-backed identifier discovery	High	Useful beyond BigQuery wherever candidate IDs should come from public evidence.
BigQuery metadata cataloguing	High	Reusable for DataStitcher, SiteIntel, and developer tools.
D1 read-cache plus R2 large payload pattern	High	Strong low-cost pattern for Cloudflare-hosted catalogues.
Rate-limit and cooldown model	High	General worker safety pattern.
BigQuery identifier parsing package	Medium to high	Already identified as a pure package candidate.
D1/Postgres backend selector	Medium	Useful pattern, but implementation details differ by product.

Notable design patterns

• Candidate discovery owns evidence.
• Validation owns public readability checks.
• Publication reads verified catalogue data and does not decide validity.
• D1 stores compact current-state rows; R2 stores heavy schema payloads.
• Large projects should be refreshed dataset-by-dataset where possible.
• The admin surface is operational and protected by Cloudflare Access.

Dependencies on other products or Meridian tooling

Fact: BigQuery Atlas follows Meridian deployment/config and generated operations docs.

Inference: BigQuery Atlas can feed DataStitcher with candidate public datasets and can share BigQuery catalogue learnings with SiteIntel's BigQuery worker.

Risks and unknowns

• BigQuery metadata calls can hit permissions, quota, or dataset-size limits.
• Search providers have API costs, credentials, and terms.
• D1 suitability for larger catalogues needs live measurements.
• Admin-triggered schema checks can become expensive if unbounded.
• Current placeholder packages should not become packages until implementation stabilises.

Near-term next steps

• Keep live D1 suitability evidence in docs before making D1 the unqualified production default.
• Promote identifier extraction only when both crawler and cataloguer need the stable API.
• Add generated capability metadata once crawler/cataloguer capabilities are stable.
• Expose public dataset change/freshness intelligence rather than only search/catalogue browsing.

Longer-term product potential

Opportunity: BigQuery Atlas can become a public-data monitoring SaaS for new dataset alerts, schema change alerts, provider/category digests, freshness checks, and related dataset discovery.

Opportunity: A lightweight free public catalogue can drive awareness for paid change monitoring or API access.

AI notes

• Never implement random project-ID generation or brute-force validation.
• Preserve evidence fields and cooldown/backoff decisions.
• Keep D1 schema payloads out of hot rows; use R2 references as designed.
• Prefer dataset-scoped checks for large public projects.
• Treat INFORMATION_SCHEMA as cost-bearing and opt-in.

Source coverage

Coverage is strong for architecture, crawler/cataloguer behaviour, backend decisions, and rate limits. Product Compass did not inspect implementation source files beyond README/package docs in this pass.

Sources consulted

• projects/bq-atlas/README.md
• projects/bq-atlas/apps/websites/docs/metadata/project-dossier.json
• projects/bq-atlas/docs/overview/product-positioning.md
• projects/bq-atlas/docs/architecture/target-monorepo-architecture.md
• projects/bq-atlas/docs/architecture/shared-code-extraction-plan.md
• projects/bq-atlas/docs/operations/catalogue-backend-decision.md
• projects/bq-atlas/docs/operations/rate-limit-policy.md
• projects/bq-atlas/docs/codebase/apps-workers-packages.md
• projects/bq-atlas/docs/data/index.md
• projects/bq-atlas/docs/operations/deployable-units.mdx
• projects/bq-atlas/services/candidate-crawler/README.md
• projects/bq-atlas/services/public-cataloguer/README.md
• projects/bq-atlas/packages/bigquery-identifiers/README.md
• projects/bq-atlas/packages/db-contract/README.md