49 lines
2.4 KiB
Markdown
49 lines
2.4 KiB
Markdown
# ca-marketplace-scraper
|
|
|
|
## Repo Shape
|
|
|
|
- Bun workspace monorepo with packages under `packages/*`.
|
|
- `packages/core`: scraper behavior, parsing, result types, cookie handling, HTTP helpers.
|
|
- `packages/api-server`: Bun HTTP adapter exposing `/api/*` routes over core.
|
|
- `packages/mcp-server`: MCP/JSON-RPC adapter that proxies to the API server.
|
|
- `cookies/`: local cookie docs/examples only. Treat real cookie files as secrets.
|
|
- `dist/`, `node_modules/`, `.turbo/`, `.direnv/`, `.devenv/`: generated/vendor/cache. Do not edit.
|
|
|
|
## Commands
|
|
|
|
- Install: `bun install`
|
|
- Lint/format/typecheck: `bun run ci`
|
|
- Build all packages: `bun run build`
|
|
- Build bundled runtime output: `bun run build:all`
|
|
- Run tests: `bun test packages/core/test packages/api-server/test packages/mcp-server/test`
|
|
- API dev server: `bun run --cwd packages/api-server dev`
|
|
- MCP dev server: `bun run --cwd packages/mcp-server dev`
|
|
|
|
## Boundaries
|
|
|
|
- Marketplace behavior belongs in `packages/core`, not adapter packages.
|
|
- HTTP route code should parse request input, call core, and map status/errors.
|
|
- MCP code should define tools, validate JSON-RPC flow, and map tool args to API URLs.
|
|
- Keep API query params and MCP tool args in sync.
|
|
- Shared public surface for scraper code is `packages/core/src/index.ts`; update exports deliberately.
|
|
|
|
## Invariants
|
|
|
|
- Cookie precedence in core helpers: explicit/request cookie string before environment variable.
|
|
- Tests must be deterministic and offline. Mock `fetch`; do not hit live marketplace endpoints.
|
|
- Use Bun and Bun-native APIs. Do not add Node-specific tooling unless already required.
|
|
- Biome and strict TypeScript are contract. Fix code; do not relax config.
|
|
|
|
## Verification
|
|
|
|
- Core changes: `bun test packages/core/test && bun run ci`
|
|
- Adapter-only changes: relevant package build plus `bun run ci`
|
|
- Cross-package contract changes: `bun test packages/core/test packages/api-server/test packages/mcp-server/test && bun run ci && bun run build`
|
|
|
|
## Gotchas
|
|
|
|
- `bunfig.toml` points test root at `./do-not-run-tests-from-root`; pass package test paths explicitly.
|
|
- Root `build` cleans `dist`, then Turbo emits bundles for API and MCP.
|
|
- `scripts/start.sh` launches `dist/api/index.js` and `dist/mcp/index.js`.
|
|
- Package `tsconfig.json` files override root `include`; shared ambient declarations under root `types/` must be included from each package that typechecks cross-package source.
|