Provider and data-source map
This map lists external providers, APIs, datasets, and platforms used or planned across the inspected product sources. It intentionally excludes secret values, account IDs, tokens, and credentials.
Cloud and platform providers
| Provider/API | Used by | Purpose | Auth/secrets implications | Reuse opportunity | Duplication risk |
|---|---|---|---|---|---|
| Cloudflare Workers | SiteIntel, Trackpal, DataStitcher, BigQuery Atlas, Bollocks Radar, Product Compass, Meridian Hub | Public/admin apps, docs sites, workers, progress worker | Worker secrets and Access policies need status-only handling | Shared deployable-unit docs, admin app patterns, secret-status components | Repeated OpenNext/Worker setup and Access rules |
| Cloudflare Pages | BigQuery Atlas | Public catalogue hosting | Build/deploy config, no browser secrets | Low-cost static/public catalogue pattern | Separate Pages/Workers conventions |
| Cloudflare D1 | BigQuery Atlas, Meridian Hub | Read-side cache and Hub database | Binding IDs should not be treated as secrets but should stay in config | D1 read-cache pattern | Schema drift across apps |
| Cloudflare R2 | BigQuery Atlas, SiteIntel, Meridian Hub | Large JSON/schema/raw artefact storage | Bucket bindings and access controls | D1 compact rows plus R2 full payload pattern | Raw artefact leakage if pointers are exposed badly |
| Cloudflare Access | Trackpal, SiteIntel, DataStitcher, BigQuery Atlas, Bollocks Radar, Meridian Hub | Protect admin/docs surfaces | Access app config and policy status only | Shared protected-admin blueprint | Inconsistent admin auth assumptions |
| Cloudflare Queues/Cron | Trackpal | Sync jobs and private-beta scheduled work | Worker secrets for provider sync | Small sync-worker pattern | Premature use where Cloud Tasks is already chosen |
| Google Cloud Run | SiteIntel, DataStitcher, BigQuery Atlas, Bollocks Radar | Heavy/private workers and jobs | Runtime service accounts and Secret Manager | Bounded worker/job pattern | One-off worker deployment scripts |
| Google Cloud Tasks | SiteIntel, Bollocks Radar | Bounded async task dispatch | Queue invoker/auth config | Shared lock/retry queue pattern | Duplicate queue state models |
| Google Cloud Scheduler/Workflows | DataStitcher, BigQuery Atlas, Bollocks Radar | Scheduled and orchestrated ingestion | Scheduler/workflow invoker accounts | Generic scheduled ingestion pattern | Source-specific workflows |
| Google Cloud Storage | DataStitcher, Bollocks Radar, SiteIntel ideas/current compatible contracts | Raw, parsed, manifest, and profile artefacts | Bucket IAM and regional policy | Raw artefact boundary | Region/policy mistakes |
| Google BigQuery | DataStitcher, BigQuery Atlas, SiteIntel, Meridian packages | Warehouse, public metadata, domain registry, marts | Google identity, dataset IAM, query cost controls | BigQuery catalogue package/pattern | Unbounded queries or duplicated schema code |
| Supabase/Postgres/Neon | SiteIntel, Trackpal, DataStitcher, Bollocks Radar, BigQuery Atlas optional | Control-plane, registry, state, metadata | Direct DB URLs and service roles must stay server-side | Shared control-plane run/source schemas | Exposing private ops schemas through public APIs |
Product data sources and APIs
| Provider/API | Used by | Purpose | Auth/secrets implications | Reuse opportunity | Notes |
|---|---|---|---|---|---|
| Google Fact Check Tools API | Bollocks Radar | Claim and fact-check review ingestion | API key server-side only | Source adapter pattern | Search API, not complete feed |
| GDELT DOC 2.0 | Bollocks Radar planned | Trend/context enrichment | API/key terms to verify | Enrichment adapter | Not MVP source |
| YouTube Data API | Bollocks Radar planned | Media/video context | API key and quota | Enrichment adapter | Requires abuse controls |
| Wikidata SPARQL | Bollocks Radar planned | Entity/context enrichment | Usually no secret, rate limits apply | Public reference adapter | Watch query limits |
| Google Cloud Vision Web Detection | Bollocks Radar later | Media verification | GCP credentials, cost | Media-source pattern | Requires careful confidence wording |
| GitHub Code Search | BigQuery Atlas | BigQuery identifier discovery | PAT or GitHub token | Evidence source adapter | Provider-specific rate limits |
| Brave Search API | BigQuery Atlas | Candidate discovery | API key | Search provider adapter | Paid/limited provider |
| Google Programmable Search JSON API | BigQuery Atlas | Candidate discovery | API key and CX | Search provider adapter | Source docs note new customer limitation |
| DataForSEO | BigQuery Atlas, SiteIntel, Bollocks Radar planned | SERP, dataset, trends, SEO/competitor data | Login/password/API credentials | Paid provider adapter and cost controls | Must be disabled by default unless configured |
| Common Crawl CDX | BigQuery Atlas | Discover URLs likely to mention BigQuery docs/pages | No secret | Public web evidence adapter | Need politeness/caps |
| Stack Exchange API | BigQuery Atlas | SQL/error examples with BigQuery IDs | Optional key | Evidence adapter | Rate limits |
| OpenAlex | BigQuery Atlas | Research/publication pages mentioning datasets | No secret | Evidence adapter | Public API terms |
| BigQuery metadata APIs | BigQuery Atlas, SiteIntel, DataStitcher | Dataset/table/schema metadata | Google identity and quota | Strong candidate for shared helper | Metadata-only still needs bounds |
data.gov.au CKAN | DataStitcher | Public dataset discovery | Usually no secret | CKAN connector | Source-specific pagination and formats |
| ABR/ATO/ABS/ASIC/ACNC and other official registers | DataStitcher | Business/public audit datasets | Varies; often public bulk feeds | Source registry plus adapters | Wording and publication policy matter |
| DNS/RDAP/TLS/HTTP/robots/sitemap/homepage | SiteIntel | Domain report source tasks | Mostly no secrets, but SSRF/abuse controls | Task adapters | Must cap time/size/redirects |
| Google PageSpeed Insights | SiteIntel | Performance checks | API key may be useful/required for quota | Source task adapter | Cost/quota controls |
| Cloudflare Radar/Intel APIs | SiteIntel | Domain rank/category/risk signals | Cloudflare token/account config | Provider adapter | Requires careful result wording |
| Mozilla Observatory / SSL Labs / OpenPageRank | SiteIntel | Security/authority signals | OpenPageRank requires key | Source adapters | Rate limits and data caveats |
| Google Ads | Trackpal | Read-only spend/performance metrics | OAuth, developer token, approval | Connector interface | Verify current docs/scopes before work |
| Meta Ads | Trackpal planned/stubbed | Read-only spend/performance metrics | App review and OAuth | Connector interface | Guarded stub |
| Shopify | Trackpal | Store pulse metrics | OAuth/admin scopes | Connector interface | Revenue definitions need caveats |
| Klaviyo | Trackpal | Email revenue/share metrics | API credentials/scopes | Connector interface | Attribution windows matter |
| Xero | Trackpal planned/stubbed | P&L/cost mapping | OAuth/accounting scopes | Connector interface | Later phase |
| models.dev pricing API | Codex Sessions Analyser | Optional model pricing estimates | Network fetch or saved file; no raw sessions | Pricing cache helper | Time-sensitive external data |
Shared adapter recommendations
- Build shared contracts before shared provider code.
- Keep provider-specific auth, pagination, quotas, terms, and billing visible in docs.
- Use status-only secret/config surfaces in admin apps.
- Do not expose ops schemas, service roles, or raw provider payloads to browser-readable routes.
- Add current official-doc verification tasks before implementing live ad/ecommerce/accounting connectors.
Sources consulted
- Product dossiers under
docs/projects/*/overview.md projects/site-intel/docs/architecture/data-source-registry.mdxprojects/trackpal/docs/product/12-integrations-strategy.mdprojects/bq-atlas/services/candidate-crawler/README.mdprojects/data-stitcher/docs/public-audit-sources.mdprojects/bollocks-radar/docs/reference/technical/google-fact-check-ingestion.mdx