DataStitcher dossier
Product summary
Fact: DataStitcher is a reusable ingestion, registry, documentation, and analytics foundation for turning messy public/open datasets into governed, searchable, dashboard-ready data products.
Fact: The first target data product is Australian business data, with an environment data product sample to keep the architecture domain-neutral.
Intended users
- Scott as the operator building reusable public-data products.
- Future users of public dashboards built from curated government/open datasets.
- Analysts, researchers, small businesses, and agencies needing interpreted public data.
- AI assistants and developers extending data ingestion pipelines.
Problem being solved
Public datasets are often scattered, inconsistently shaped, poorly documented, and difficult to turn into reliable dashboards. DataStitcher aims to provide the repeatable machinery for discovery, ingestion, profiling, semantic registry, artefact storage, BigQuery marts, generated docs, and public apps.
Current status
Fact: The project dossier marks DataStitcher as in-development and in platform-build.
Fact: The repository contains Cloudflare-hosted business, environment, admin, and docs surfaces; Neon/Postgres registry state; GCS artefact storage; BigQuery analytical marts; and Cloud Run Job-oriented pipeline work.
Known aims and strategic intent
- Keep platform logic separate from data-product configuration and public app UI.
- Use PostgreSQL/Neon as registry truth for metadata, workflow state, approvals, and semantic mappings.
- Use GCS for raw, parsed, profile, manifest, and run configuration artefacts.
- Use BigQuery for raw/staging/canonical/mart analytical tables.
- Keep domain-specific product rules in
data-products/{domain}or public app packages. - Require human review before AI suggestions become registry truth.
Built functionality
Fact: Source docs describe:
- A
data.gov.auCKAN connector. - Generic source/dataset/resource/resource-version model.
- Semantic registry concepts for metrics, dimensions, aliases, caveats, external mappings, and publication rules.
- PostgreSQL migration scaffolds for metadata and registry tables.
- GCS and BigQuery adapter interfaces.
- Python parser package skeleton for XLSX, XML, CSV, and profiling/parsing.
- ABN bulk extract to BigQuery workflow.
- Public audit source ingestion scaffold covering official Australian public records.
- Business and environment data product examples.
- CLI scaffolding for discovery, download, profiling, pipeline runs, and docs generation.
Key apps, jobs, packages, and scripts
| Surface | Role |
|---|---|
apps/websites/business | Australian business dashboard/product app. |
apps/websites/environment | Environment product sample app. |
apps/websites/admin | Private operations console for ingestion and registry workflows. |
packages/pipelines | Cloud Run Job pipeline logic for ABN, ACNC, admin runner, and scheduler jobs. |
packages/connectors | Source adapters and connector logic. |
packages/db | Database and registry contracts. |
packages/data-products | Domain-specific data product configuration. |
packages/docs-generator | Generated documentation workflow. |
python | Parser/profiler job boundary. |
Data sources and integrations
Fact: Source docs mention data.gov.au CKAN, ATO, ABS, ABR, ASIC, ACNC Charity Register, ORIC, GrantConnect, AusTender OCDS, AEC Transparency Register, NDIS Commission, aged-care compliance reports, DFAT Consolidated List, AUSTRAC enforcement actions, Foreign Influence Transparency Scheme, Register of Lobbyists, and state fundraising/incorporated association registers.
Fact: The data platform uses Cloudflare Workers, Neon/Postgres, GCS, BigQuery, Google Cloud Secret Manager, and Cloud Run Jobs.
Deployment and runtime assumptions
- Cloudflare Workers host public and admin Next.js apps through OpenNext.
- Neon/Postgres stores registry and workflow state.
- GCS stores immutable source artefacts and run snapshots.
- BigQuery stores analytical raw/staging/canonical/mart outputs.
- Cloud Run Jobs are the natural boundary for parser, fetch, warehouse, validation, and runner stages.
- ABN name-bearing raw/staging artefacts stay in
australia-southeast1; only aggregate public marts may be copied outward after review.
Reusable capabilities
| Capability | Reuse potential | Notes |
|---|---|---|
| Monitored ingestion pattern | High | Clear candidate for a shared Works ingestion pattern or package. |
| Semantic registry | High | Useful for any data product that needs metric definitions, caveats, external vocabulary mappings, and docs generation. |
| Source fingerprinting and no-change detection | High | Shared by SiteIntel bulk ingest and likely future public-data tools. |
| Resumable pipeline run model | High | Applies to ABN, public audit sources, BigQuery catalogue, and many future ingestion jobs. |
| Data product separation | Medium to high | Strong architectural pattern; not necessarily a package. |
| Region and publication policy model | Medium | Product-specific details vary, but the policy shape is reusable. |
Notable design patterns
- Config truth lives in Git; registry truth lives in Postgres.
- Source-specific behaviour belongs in adapters.
- Jobs are named by responsibility rather than by source where possible.
- Runs store immutable resolved configuration snapshots.
- Retry resumes from failed stage rather than restarting blindly.
- Public apps consume safe marts or generated payloads, not raw/staging tables.
Dependencies on other products or Meridian tooling
Fact: DataStitcher follows Meridian deployment/config conventions and uses shared Meridian deployment tooling.
Inference: DataStitcher is the best first consumer for a future @meridian-stack/works-ingest contract because it already has the clearest generic monitored-ingestion pattern.
Risks and unknowns
- The generic pipeline model should not over-abstract before two real sources prove the boundaries.
- Public audit/source records require careful wording and human review to avoid implying wrongdoing from raw public records.
- ABN/name-bearing raw data has region and publication constraints.
- BigQuery costs and Cloud Run Job complexity need bounded execution and clear operator visibility.
- AI suggestions must stay review-only until approved into registry truth.
Near-term next steps
- Finish one monitored source path end to end with source check, fetch, parse, load, transform, validate, finalise, docs, and app display.
- Keep ABN and public-audit work evidence-first and publication-policy aware.
- Extract only the source/run/artefact contract shape once SiteIntel or BigQuery Atlas becomes a second real consumer.
- Add generated capability metadata in the product repo when the platform shape stabilises.
Longer-term product potential
Opportunity: DataStitcher can become the internal product factory for public-data products: business intelligence, environment dashboards, regional context tools, public audit evidence browsers, and niche APIs.
Opportunity: The semantic registry and docs generator could become a developer-facing package or template for governed data products.
AI notes
- Do not put business-specific logic inside generic platform packages.
- Treat source records as evidence and provenance, not accusations or conclusions.
- Respect region and publication rules for ABN/name-bearing data.
- Prefer configuration plus adapters over one-off workflows.
- Keep public app payloads privacy-safe and mart-derived.
Source coverage
The current source coverage is strong for architecture, ingestion pattern, registry model, data products, and public-audit direction. Implementation depth for every source-specific parser was not inspected in this pass.
Sources consulted
projects/data-stitcher/README.mdprojects/data-stitcher/apps/websites/docs/metadata/project-dossier.jsonprojects/data-stitcher/docs/overview/start-here.mdxprojects/data-stitcher/docs/architecture/overview.mdxprojects/data-stitcher/docs/architecture/storage-and-regions.mdxprojects/data-stitcher/docs/codebase/apps-packages-scripts.mdxprojects/data-stitcher/docs/operations/reusable-monitored-ingestion-pattern.mdprojects/data-stitcher/docs/reference/registry/overview.mdxprojects/data-stitcher/docs/overview/products/business.mdxprojects/data-stitcher/docs/overview/products/environment.mdxprojects/data-stitcher/docs/reference/decisions/0004-platform-data-products-apps.mdxprojects/data-stitcher/docs/public-audit-sources.mdprojects/data-stitcher/docs/operations/deployable-units.mdxprojects/data-stitcher/data-products/README.md