Developer Guide

Quick Start (Recommended)

One command starts the entire local development stack:

make dev

This starts: Docker services (PostgreSQL, MinIO), the Cloudflare Worker, the Inngest dev server, runs Prisma migrations, and starts Next.js.

Prerequisites

• Docker (for PostgreSQL and MinIO)
• Node.js and pnpm (npm install -g pnpm)
• Cloudflare Worker repo cloned alongside this project (default: ../datahub-cloudflare-workers)

Setup

1. Clone the repo:

   git clone https://github.com/datopian/datahub-next
   cd datahub-next
   

2. Install dependencies:

   pnpm i
   

3. Start the dev stack:

   make dev
   

   This auto-copies .env.local.example to .env if no .env exists.

4. Configure credentials in .env:

   • NEXT_PUBLIC_AUTH_GITHUB_ID and AUTH_GITHUB_SECRET (GitHub OAuth - get from team)
   • GH_ACCESS_TOKEN (GitHub personal access token for sync and E2E tests)
   • NEXTAUTH_SECRET (any random string for local dev)

5. Visit:

   • App: http://localhost:3000
   • Dashboard: http://localhost:3000/dashboard
   • Inngest: http://localhost:8288
   • MinIO Console: http://localhost:9001 (minioadmin/minioadmin)

Press Ctrl+C to stop all services. To also stop Docker:

make dev-down

Make targets

Worker directory override

By default, make dev expects the Cloudflare Worker at ../datahub-cloudflare-workers. Override with:

WORKER_DIR=/path/to/workers make dev

Alternative: Cloud Development Environment

If you prefer not to run local Docker, you can connect to shared cloud resources:

1. Copy .env.example (not .env.local.example) to .env
2. Obtain cloud credentials from the team (Neon Postgres + R2 bucket)
3. Configure POSTGRES_PRISMA_URL, POSTGRES_URL_NON_POOLING, and S3_* variables
4. Run pnpm dev directly (no make dev needed)

Environment Variables

See .env.local.example for the full list with comments. Key variables:

GitHub Webhook Forwarding (Optional)

Only needed for automatic content syncing on push. Manual sync is available via the dashboard "Sync" button.

1. Create a webhook channel at smee.io/new
2. Install: npm install --global smee-client
3. Forward: smee --url https://smee.io/your_channel_id --target http://127.0.0.1:3000/api/webhook

Stripe Setup (Optional)

For testing subscription/billing features (requires implementing the webhook route first):

1. Install Stripe CLI: brew install stripe-cli
2. Login: stripe login (use test mode)
3. Forward webhooks: stripe listen --forward-to localhost:3000/api/stripe/webhook
4. Copy the whsec_... secret to STRIPE_WEBHOOK_SECRET in .env
5. Keep Stripe CLI running while testing subscriptions

Note: The Stripe webhook route (app/api/stripe/webhook/route.ts) must be implemented before this setup is functional. Currently only Stripe payment links are used.

Branching Strategy

Workflow

1. Create feature branch from staging
2. Implement changes
3. Submit PR to staging
4. After review, squash-merge to staging
5. Changes are rebased from staging to main for production release

Commit messages follow the Conventional Commits specification.

Testing

Unit Tests (Jest)

# Run all unit tests
pnpm test

# Run in watch mode
pnpm test:watch

Test locations:

• lib/__tests__/ - Utility function tests
• server/api/routers/__tests__/ - tRPC router tests
• lib/*.test.ts - Co-located tests (e.g., extract-title.test.ts)

E2E Tests (Playwright)

E2E tests require the full dev stack running (make dev) and GH_ACCESS_TOKEN set in .env.

Seeding is handled automatically by Playwright's global setup (e2e/global-setup.ts). It creates a test user, publication, and test posts.

# Run all renderer tests on Chromium
pnpm exec playwright test --project=renderer-chromium

# Run specific test file
pnpm exec playwright test e2e/renderer/dataset-datapackage-json.spec.ts

# Run with UI mode
pnpm exec playwright test --project=renderer-chromium --ui

# Debug mode
pnpm exec playwright test --project=renderer-chromium --debug

# Run on both Chromium and Firefox
pnpm exec playwright test --grep renderer

See e2e/README.md for the full E2E testing guide (CI setup, debugging, test repos).

Lint

pnpm lint

Format

pnpm format:write

Infrastructure

Databases

Content Storage

Authentication

GitHub OAuth applications under Datopian account:

• Production: DataHub Cloud
• Staging: DataHub Cloud - Staging
• Development: DataHub Cloud - Dev