old4ever/ca-marketplace-scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: format markdown

Browse Source 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
This commit is contained in:
Dmytro Stanchiev
2026-05-01 11:42:54 -04:00
parent d2c3c07e7d
commit 7ab33d0b02
15 changed files with 925 additions and 417 deletions
Download Patch File Download Diff File Expand all files Collapse all files

82
docs/superpowers/plans/2025-07-14-opencode-monorepo-config.md
View File

@@ -1,19 +1,26 @@
# opencode Monorepo Config Adoption Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or superpowers:executing-plans
> to implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Adopt opencode-style monorepo config: Turbo task orchestration, workspace dep catalog, shared root tsconfig, bunfig.toml, and `exports` field in all packages.
**Goal:** Adopt opencode-style monorepo config: Turbo task orchestration, workspace dep
catalog, shared root tsconfig, bunfig.toml, and `exports` field in all packages.
**Architecture:** Pure config changes across 10 files — no source code touched. Root config files are added/updated first, then per-package files updated to reference them. Changes are independent within each task and safe to commit atomically.
**Architecture:** Pure config changes across 10 files — no source code touched.
Root config files are added/updated first, then per-package files updated to reference
them. Changes are independent within each task and safe to commit atomically.
**Tech Stack:** Bun workspaces, Turbo 2.x, @tsconfig/bun, TypeScript (tsgo / @typescript/native-preview)
**Tech Stack:** Bun workspaces, Turbo 2.x, @tsconfig/bun, TypeScript (tsgo /
@typescript/native-preview)
---
* * *
## File Map
| File | Action | Responsible for |
|---|---|---|
| --- | --- | --- |
| `package.json` | Modify | Workspace catalog, turbo devDep, @tsconfig/bun devDep, updated scripts |
| `turbo.json` | Create | Task graph: typecheck, build, test |
| `tsconfig.json` | Create | Shared TS compiler options for all packages |
@@ -25,14 +32,16 @@
| `packages/api-server/tsconfig.json` | Modify | Slim — extends root, paths only |
| `packages/mcp-server/tsconfig.json` | Modify | Slim — extends root, paths only |
---
* * *
### Task 1: Add `bunfig.toml` and `turbo.json`
Two new root config files with no dependencies on other tasks.
**Files:**
- Create: `bunfig.toml`
- Create: `turbo.json`
- [ ] **Step 1: Create `bunfig.toml`**
@@ -83,13 +92,15 @@ git add bunfig.toml turbo.json
git commit -m "chore: add bunfig.toml and turbo.json"
```
---
* * *
### Task 2: Create root `tsconfig.json`
Shared base tsconfig all packages will extend. Extracts the common options currently duplicated in all 3 per-package tsconfigs.
Shared base tsconfig all packages will extend.
Extracts the common options currently duplicated in all 3 per-package tsconfigs.
**Files:**
- Create: `tsconfig.json`
- [ ] **Step 1: Create root `tsconfig.json`**
@@ -130,13 +141,15 @@ git add tsconfig.json
git commit -m "chore: add shared root tsconfig.json"
```
---
* * *
### Task 3: Update root `package.json`
Add workspace catalog, `turbo` + `@tsconfig/bun` devDependencies, and update scripts to use `turbo run`.
Add workspace catalog, `turbo` + `@tsconfig/bun` devDependencies, and update scripts to
use `turbo run`.
**Files:**
- Modify: `package.json`
- [ ] **Step 1: Replace root `package.json`**
@@ -180,7 +193,11 @@ Write this complete file:
}
```
> **Note on catalog versions:** The catalog pins exact versions. The values above are taken from the current package installs. If `@types/bun` was `latest`, check `node_modules/@types/bun/package.json` for the actual installed version and use that. Same for `@typescript/native-preview`.
> **Note on catalog versions:** The catalog pins exact versions.
> The values above are taken from the current package installs.
> If `@types/bun` was `latest`, check `node_modules/@types/bun/package.json` for the
> actual installed version and use that.
> Same for `@typescript/native-preview`.
- [ ] **Step 2: Check actual installed versions**
@@ -208,7 +225,8 @@ Expected: lock file updated, `turbo` and `@tsconfig/bun` appear in `node_modules
bunx turbo run typecheck --dry
```
Expected: output lists the `typecheck` task for each package (even if no `typecheck` script exists yet — turbo will note them as skipped/missing).
Expected: output lists the `typecheck` task for each package (even if no `typecheck`
script exists yet — turbo will note them as skipped/missing).
- [ ] **Step 5: Commit**
@@ -217,15 +235,19 @@ git add package.json bun.lock
git commit -m "chore: add workspace catalog and turbo to root package.json"
```
---
* * *
### Task 4: Update per-package `package.json` files
Rename `type:check` → `typecheck`, replace `main`/`module` with `exports`, swap pinned dep versions for `catalog:` references.
Rename `type:check` → `typecheck`, replace `main`/`module` with `exports`, swap pinned
dep versions for `catalog:` references.
**Files:**
- Modify: `packages/core/package.json`
- Modify: `packages/api-server/package.json`
- Modify: `packages/mcp-server/package.json`
- [ ] **Step 1: Replace `packages/core/package.json`**
@@ -325,7 +347,9 @@ Rename `type:check` → `typecheck`, replace `main`/`module` with `exports`, swa
bun install
```
Expected: no errors. Catalog refs resolved. `bun.lock` updated.
Expected: no errors.
Catalog refs resolved.
`bun.lock` updated.
- [ ] **Step 5: Verify typecheck still works per-package**
@@ -345,15 +369,19 @@ git add packages/core/package.json packages/api-server/package.json packages/mcp
git commit -m "chore: use exports field and catalog refs in all packages"
```
---
* * *
### Task 5: Slim per-package `tsconfig.json` files
Replace the duplicated full tsconfig in each package with a slim `extends`-based one pointing to root.
Replace the duplicated full tsconfig in each package with a slim `extends`-based one
pointing to root.
**Files:**
- Modify: `packages/core/tsconfig.json`
- Modify: `packages/api-server/tsconfig.json`
- Modify: `packages/mcp-server/tsconfig.json`
- [ ] **Step 1: Replace `packages/core/tsconfig.json`**
@@ -400,7 +428,8 @@ Replace the duplicated full tsconfig in each package with a slim `extends`-based
- [ ] **Step 4: Verify `@tsconfig/bun` is resolvable**
The root tsconfig extends `@tsconfig/bun/tsconfig.json`. Confirm the package is installed:
The root tsconfig extends `@tsconfig/bun/tsconfig.json`. Confirm the package is
installed:
```bash
ls node_modules/@tsconfig/bun/tsconfig.json
@@ -414,7 +443,8 @@ Expected: file exists.
bun run typecheck
```
Expected: Turbo runs `typecheck` for all 3 packages in parallel, all pass (or same pre-existing errors — no new ones).
Expected: Turbo runs `typecheck` for all 3 packages in parallel, all pass (or same
pre-existing errors — no new ones).
- [ ] **Step 6: Commit**
@@ -423,7 +453,7 @@ git add packages/core/tsconfig.json packages/api-server/tsconfig.json packages/m
git commit -m "chore: slim per-package tsconfigs to extend root"
```
---
* * *
### Task 6: Smoke test full build pipeline
@@ -437,7 +467,8 @@ Verify everything works end-to-end.
bun run typecheck
```
Expected: Turbo runs `typecheck` across all packages. Exit 0.
Expected: Turbo runs `typecheck` across all packages.
Exit 0.
- [ ] **Step 2: Run full build**
@@ -445,7 +476,8 @@ Expected: Turbo runs `typecheck` across all packages. Exit 0.
bun run build
```
Expected: `dist/` cleaned, Turbo runs `build` (core first, then api-server and mcp-server in parallel), build artifacts appear in `dist/api/` and `dist/mcp/`.
Expected: `dist/` cleaned, Turbo runs `build` (core first, then api-server and
mcp-server in parallel), build artifacts appear in `dist/api/` and `dist/mcp/`.
- [ ] **Step 3: Verify dist artifacts**
@@ -461,7 +493,9 @@ Expected: compiled output files in both directories.
grep -c '\^' bun.lock | head -5
```
With `exact = true` in bunfig.toml, new installs won't add `^` ranges. Existing `^` ranges in `bun.lock` from before are fine — they'll be resolved to exact on next fresh install.
With `exact = true` in bunfig.toml, new installs won’t add `^` ranges.
Existing `^` ranges in `bun.lock` from before are fine — they’ll be resolved to exact on
next fresh install.
- [ ] **Step 5: Final commit if any loose files**

179
docs/superpowers/plans/2026-04-21-cookie-env-only.md
View File

@@ -1,53 +1,64 @@
# Cookie Env-Only Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or superpowers:executing-plans
> to implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Remove cookie files and request-provided cookie overrides so all authenticated marketplace scraping reads raw `Cookie` header strings only from environment variables.
**Goal:** Remove cookie files and request-provided cookie overrides so all authenticated
marketplace scraping reads raw `Cookie` header strings only from environment variables.
**Architecture:** Collapse shared cookie loading to a single env-var reader in `packages/core/src/utils/cookies.ts`, then tighten Facebook and eBay core signatures to stop accepting request/file cookie inputs. Update the API and MCP adapters so they no longer advertise or forward cookie parameters, and rewrite docs/tests to match the env-only contract.
**Architecture:** Collapse shared cookie loading to a single env-var reader in
`packages/core/src/utils/cookies.ts`, then tighten Facebook and eBay core signatures to
stop accepting request/file cookie inputs.
Update the API and MCP adapters so they no longer advertise or forward cookie
parameters, and rewrite docs/tests to match the env-only contract.
**Tech Stack:** Bun, TypeScript, Bun test, Biome, workspace package exports
---
* * *
## File Map
- Modify: `packages/core/src/utils/cookies.ts`
Purpose: remove JSON/file/request-source loading and keep env-only cookie parsing/formatting.
- Modify: `packages/core/src/scrapers/facebook.ts`
Purpose: drop `cookiesSource` / `cookiePath` arguments and env-only error text.
- Modify: `packages/core/src/scrapers/ebay.ts`
Purpose: remove `opts.cookies` request override and use env-only cookie loading.
- Modify: `packages/core/src/index.ts`
Purpose: keep exports aligned with tightened core signatures.
- Modify: `packages/core/test/facebook-core.test.ts`
Purpose: replace missing-file coverage with env-only auth tests.
- Create: `packages/core/test/ebay-core.test.ts`
Purpose: add dedicated eBay auth regression coverage instead of mixing it into Facebook tests.
- Modify: `packages/api-server/src/routes/facebook.ts`
Purpose: stop parsing/forwarding `cookies` query params.
- Modify: `packages/api-server/src/routes/ebay.ts`
Purpose: stop parsing/forwarding `cookies` query params.
- Create: `packages/api-server/test/routes.test.ts`
Purpose: verify Facebook/eBay routes ignore cookie query params and still call core correctly.
- Modify: `packages/mcp-server/src/protocol/tools.ts`
Purpose: remove Facebook/eBay cookie tool inputs and descriptions.
- Modify: `packages/mcp-server/src/protocol/handler.ts`
Purpose: stop mapping removed cookie tool inputs into API URLs.
- Create: `packages/mcp-server/test/protocol.test.ts`
Purpose: verify tool schemas and handler URL building no longer include Facebook/eBay cookie fields.
- Modify: `cookies/AGENTS.md`
Purpose: document env vars as the only supported cookie input.
- Modify: `packages/core/src/utils/cookies.ts` Purpose: remove JSON/file/request-source
loading and keep env-only cookie parsing/formatting.
- Modify: `packages/core/src/scrapers/facebook.ts` Purpose: drop `cookiesSource` /
`cookiePath` arguments and env-only error text.
- Modify: `packages/core/src/scrapers/ebay.ts` Purpose: remove `opts.cookies` request
override and use env-only cookie loading.
- Modify: `packages/core/src/index.ts` Purpose: keep exports aligned with tightened core
signatures.
- Modify: `packages/core/test/facebook-core.test.ts` Purpose: replace missing-file
coverage with env-only auth tests.
- Create: `packages/core/test/ebay-core.test.ts` Purpose: add dedicated eBay auth
regression coverage instead of mixing it into Facebook tests.
- Modify: `packages/api-server/src/routes/facebook.ts` Purpose: stop parsing/forwarding
`cookies` query params.
- Modify: `packages/api-server/src/routes/ebay.ts` Purpose: stop parsing/forwarding
`cookies` query params.
- Create: `packages/api-server/test/routes.test.ts` Purpose: verify Facebook/eBay routes
ignore cookie query params and still call core correctly.
- Modify: `packages/mcp-server/src/protocol/tools.ts` Purpose: remove Facebook/eBay
cookie tool inputs and descriptions.
- Modify: `packages/mcp-server/src/protocol/handler.ts` Purpose: stop mapping removed
cookie tool inputs into API URLs.
- Create: `packages/mcp-server/test/protocol.test.ts` Purpose: verify tool schemas and
handler URL building no longer include Facebook/eBay cookie fields.
- Modify: `cookies/AGENTS.md` Purpose: document env vars as the only supported cookie
input.
### Task 1: Lock core cookie utilities to env-only loading
**Files:**
- Modify: `packages/core/src/utils/cookies.ts:19-227`
- Test: `packages/core/test/facebook-core.test.ts`
- [ ] **Step 1: Write the failing test**
Add or replace the auth-source test block in `packages/core/test/facebook-core.test.ts` with env-only expectations:
Add or replace the auth-source test block in `packages/core/test/facebook-core.test.ts`
with env-only expectations:
```ts
test("should load Facebook cookies from FACEBOOK_COOKIE env var", async () => {
@@ -85,12 +96,14 @@ test("should reject missing Facebook auth env var", async () => {
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-core.test.ts`
Expected: FAIL because the current implementation still allows missing env values to fall through to file/request-based behavior and does not emit the new env-only error.
Run: `bun test packages/core/test/facebook-core.test.ts` Expected: FAIL because the
current implementation still allows missing env values to fall through to
file/request-based behavior and does not emit the new env-only error.
- [ ] **Step 3: Write minimal implementation**
Replace the multi-source loader in `packages/core/src/utils/cookies.ts` with an env-only loader. The target shape is:
Replace the multi-source loader in `packages/core/src/utils/cookies.ts` with an env-only
loader. The target shape is:
```ts
export interface CookieConfig {
@@ -129,8 +142,8 @@ Delete the now-dead helpers and types that exist only for JSON/file/request load
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-core.test.ts`
Expected: PASS for the new env-only tests.
Run: `bun test packages/core/test/facebook-core.test.ts` Expected: PASS for the new
env-only tests.
- [ ] **Step 5: Commit**
@@ -142,10 +155,15 @@ git commit -m "refactor: make cookie loading env-only"
### Task 2: Tighten Facebook core APIs to the new contract
**Files:**
- Modify: `packages/core/src/scrapers/facebook.ts:23-29`
- Modify: `packages/core/src/scrapers/facebook.ts:214-228`
- Modify: `packages/core/src/scrapers/facebook.ts:823-929`
- Modify: `packages/core/src/index.ts:5-15`
- Test: `packages/core/test/facebook-core.test.ts`
- [ ] **Step 1: Write the failing test**
@@ -171,8 +189,9 @@ test("should fail Facebook item fetch when FACEBOOK_COOKIE is unset", async () =
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-core.test.ts`
Expected: FAIL because the current function signatures and error text still mention parameter/file-based auth paths.
Run: `bun test packages/core/test/facebook-core.test.ts` Expected: FAIL because the
current function signatures and error text still mention parameter/file-based auth
paths.
- [ ] **Step 3: Write minimal implementation**
@@ -206,12 +225,14 @@ console.warn(
);
```
Remove the extra cookie arguments from `fetchFacebookItem(...)` and keep `packages/core/src/index.ts` exporting the tightened functions without the old parameter contract.
Remove the extra cookie arguments from `fetchFacebookItem(...)` and keep
`packages/core/src/index.ts` exporting the tightened functions without the old parameter
contract.
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-core.test.ts`
Expected: PASS with the new env-only Facebook API surface.
Run: `bun test packages/core/test/facebook-core.test.ts` Expected: PASS with the new
env-only Facebook API surface.
- [ ] **Step 5: Commit**
@@ -223,8 +244,11 @@ git commit -m "refactor: remove facebook cookie overrides"
### Task 3: Tighten eBay core APIs to env-only auth
**Files:**
- Modify: `packages/core/src/scrapers/ebay.ts:9-15`
- Modify: `packages/core/src/scrapers/ebay.ts:337-389`
- Create: `packages/core/test/ebay-core.test.ts`
- [ ] **Step 1: Write the failing test**
@@ -249,8 +273,8 @@ test("should warn and continue without eBay cookies when EBAY_COOKIE is unset",
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/ebay-core.test.ts`
Expected: FAIL because `loadEbayCookies` still accepts request overrides and mentions file/json sources.
Run: `bun test packages/core/test/ebay-core.test.ts` Expected: FAIL because
`loadEbayCookies` still accepts request overrides and mentions file/json sources.
- [ ] **Step 3: Write minimal implementation**
@@ -276,12 +300,13 @@ async function loadEbayCookies(): Promise<string | undefined> {
}
```
Then remove `cookies` from `fetchEbayItems(..., opts)` and the destructuring that feeds it into `loadEbayCookies()`.
Then remove `cookies` from `fetchEbayItems(..., opts)` and the destructuring that feeds
it into `loadEbayCookies()`.
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/ebay-core.test.ts`
Expected: PASS for the eBay env-only regression coverage.
Run: `bun test packages/core/test/ebay-core.test.ts` Expected: PASS for the eBay
env-only regression coverage.
- [ ] **Step 5: Commit**
@@ -293,13 +318,17 @@ git commit -m "refactor: make ebay auth env-only"
### Task 4: Remove cookie query parameters from the API adapter
**Files:**
- Modify: `packages/api-server/src/routes/facebook.ts:3-33`
- Modify: `packages/api-server/src/routes/ebay.ts:3-52`
- Create: `packages/api-server/test/routes.test.ts`
- [ ] **Step 1: Write the failing test**
Create `packages/api-server/test/routes.test.ts` and mock `@marketplace-scrapers/core` so the route contract is explicit:
Create `packages/api-server/test/routes.test.ts` and mock `@marketplace-scrapers/core`
so the route contract is explicit:
```ts
import { afterEach, describe, expect, mock, test } from "bun:test";
@@ -347,8 +376,9 @@ test("ebayRoute ignores cookies query parameter", async () => {
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/api-server/test/routes.test.ts`
Expected: FAIL because the current routes still parse `reqUrl.searchParams.get("cookies")` and forward it downstream.
Run: `bun test packages/api-server/test/routes.test.ts` Expected: FAIL because the
current routes still parse `reqUrl.searchParams.get("cookies")` and forward it
downstream.
- [ ] **Step 3: Write minimal implementation**
@@ -383,8 +413,8 @@ const items = await fetchEbayItems(SEARCH_QUERY, 1, {
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/api-server/test/routes.test.ts`
Expected: PASS for route coverage and no remaining adapter references to `cookies` for Facebook/eBay.
Run: `bun test packages/api-server/test/routes.test.ts` Expected: PASS for route
coverage and no remaining adapter references to `cookies` for Facebook/eBay.
- [ ] **Step 5: Commit**
@@ -396,13 +426,17 @@ git commit -m "refactor: remove api cookie query overrides"
### Task 5: Remove cookie inputs from MCP tool schemas and request mapping
**Files:**
- Modify: `packages/mcp-server/src/protocol/tools.ts:65-148`
- Modify: `packages/mcp-server/src/protocol/handler.ts:154-211`
- Create: `packages/mcp-server/test/protocol.test.ts`
- [ ] **Step 1: Write the failing test**
Create `packages/mcp-server/test/protocol.test.ts` with schema and URL-building assertions:
Create `packages/mcp-server/test/protocol.test.ts` with schema and URL-building
assertions:
```ts
import { expect, mock, test } from "bun:test";
@@ -445,8 +479,8 @@ expect(calledUrl).not.toContain("cookies=");
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/mcp-server/test/protocol.test.ts`
Expected: FAIL because the current MCP schema and handler still expose and forward those inputs.
Run: `bun test packages/mcp-server/test/protocol.test.ts` Expected: FAIL because the
current MCP schema and handler still expose and forward those inputs.
- [ ] **Step 3: Write minimal implementation**
@@ -465,12 +499,13 @@ Delete the Facebook/eBay cookie tool properties and handler mapping:
// if (args.cookies) params.append("cookies", args.cookies);
```
Leave Kijiji alone; this plan only changes Facebook/eBay env-only auth paths defined by the approved spec.
Leave Kijiji alone; this plan only changes Facebook/eBay env-only auth paths defined by
the approved spec.
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/mcp-server/test/protocol.test.ts`
Expected: PASS with MCP definitions and handler mapping in sync.
Run: `bun test packages/mcp-server/test/protocol.test.ts` Expected: PASS with MCP
definitions and handler mapping in sync.
- [ ] **Step 5: Commit**
@@ -482,12 +517,16 @@ git commit -m "refactor: remove mcp cookie parameters"
### Task 6: Rewrite cookie documentation and run full verification
**Files:**
- Modify: `cookies/AGENTS.md:9-85`
- Modify: `docs/superpowers/specs/2026-04-21-cookie-env-only-design.md` only if implementation reveals a spec mismatch
- Modify: `docs/superpowers/specs/2026-04-21-cookie-env-only-design.md` only if
implementation reveals a spec mismatch
- [ ] **Step 1: Write the failing test**
Treat docs drift as a contract failure. Capture the required state before editing:
Treat docs drift as a contract failure.
Capture the required state before editing:
```md
- Cookie setup docs mention env vars only for Facebook and eBay
@@ -497,14 +536,14 @@ Treat docs drift as a contract failure. Capture the required state before editin
- [ ] **Step 2: Run verification to prove current docs are stale**
Run: `rg -n "facebook\.json|ebay\.json|cookies=" cookies/AGENTS.md`
Expected: matches found
Run: `rg -n "facebook\.json|ebay\.json|cookies=" cookies/AGENTS.md` Expected: matches
found
- [ ] **Step 3: Write minimal implementation**
Rewrite the cookie setup doc so Facebook and eBay each show only env-var setup:
```md
````md
## Cookie Configuration
All supported authenticated scrapers read cookies only from environment variables.
@@ -513,14 +552,14 @@ All supported authenticated scrapers read cookies only from environment variable
```bash
export FACEBOOK_COOKIE='c_user=123; xs=token; fr=request'
```
````
### eBay
```bash
export EBAY_COOKIE='s=VALUE; ds2=VALUE; ebay=VALUE'
```
```
````
Remove the file-based and request-parameter sections entirely.
@@ -534,10 +573,14 @@ Expected: all commands pass
```bash
git add cookies/AGENTS.md docs/superpowers/specs/2026-04-21-cookie-env-only-design.md
git commit -m "docs: align cookie setup with env-only auth"
```
````
## Self-Review
- Spec coverage check: shared cookie utils, Facebook, eBay, API adapter, MCP adapter, tests, and docs each have explicit tasks.
- Placeholder scan: concrete test files are now named for eBay core, API routes, and MCP protocol coverage.
- Type consistency check: `ensureCookies(config)` is the single shared loader name used across Tasks 1-3, and Facebook/eBay route signatures stay aligned with the core changes.
- Spec coverage check: shared cookie utils, Facebook, eBay, API adapter, MCP adapter,
tests, and docs each have explicit tasks.
- Placeholder scan: concrete test files are now named for eBay core, API routes, and MCP
protocol coverage.
- Type consistency check: `ensureCookies(config)` is the single shared loader name used
across Tasks 1-3, and Facebook/eBay route signatures stay aligned with the core
changes.

136
docs/superpowers/plans/2026-04-21-facebook-comet-rewrite.md
View File

@@ -1,34 +1,49 @@
# Facebook Comet Rewrite Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or superpowers:executing-plans
> to implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace the legacy Facebook Marketplace scraper with a route-aware hybrid Comet-bootstrap parser for both search and item routes.
**Goal:** Replace the legacy Facebook Marketplace scraper with a route-aware hybrid
Comet-bootstrap parser for both search and item routes.
**Architecture:** Keep authenticated direct HTTP fetches as the transport. Classify each Facebook response first, then parse route-specific Comet bootstrap/state candidates, and fall back to rendered-HTML extraction only when bootstrap decoding cannot produce the expected search or item shape.
**Architecture:** Keep authenticated direct HTTP fetches as the transport.
Classify each Facebook response first, then parse route-specific Comet bootstrap/state
candidates, and fall back to rendered-HTML extraction only when bootstrap decoding
cannot produce the expected search or item shape.
**Tech Stack:** Bun, TypeScript, `bun:test`, `linkedom`, existing shared cookie/http helpers
**Tech Stack:** Bun, TypeScript, `bun:test`, `linkedom`, existing shared cookie/http
helpers
---
* * *
## File Structure
- Modify: `packages/core/src/scrapers/facebook.ts`
- Owns Facebook fetch flow, response classification, bootstrap candidate extraction, search parsing, item parsing, and HTML fallbacks.
- Owns Facebook fetch flow, response classification, bootstrap candidate extraction,
search parsing, item parsing, and HTML fallbacks.
- Modify: `packages/core/test/facebook-core.test.ts`
- Owns unit coverage for response classification, bootstrap parsing, fallback parsing, and route-aware item/search extraction behavior.
- Owns unit coverage for response classification, bootstrap parsing, fallback parsing,
and route-aware item/search extraction behavior.
- Modify: `packages/core/test/facebook-integration.test.ts`
- Owns higher-level fetch flow tests, auth/degradation behavior, and result shaping for search/item entrypoints.
- Owns higher-level fetch flow tests, auth/degradation behavior, and result shaping
for search/item entrypoints.
### Task 1: Add Route Classification Coverage
**Files:**
- Modify: `packages/core/test/facebook-core.test.ts`
- Modify: `packages/core/src/scrapers/facebook.ts`
- Test: `packages/core/test/facebook-core.test.ts`
- [ ] **Step 1: Write the failing tests**
Add these tests near the Facebook parser tests in `packages/core/test/facebook-core.test.ts`:
Add these tests near the Facebook parser tests in
`packages/core/test/facebook-core.test.ts`:
```ts
test("classifies Comet search responses", () => {
@@ -89,12 +104,14 @@ test("classifies unavailable item responses", () => {
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "classifies"`
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "classifies"`
Expected: FAIL because `classifyFacebookResponse` does not exist yet.
- [ ] **Step 3: Write minimal implementation**
Add this type and function near the parsing section in `packages/core/src/scrapers/facebook.ts`:
Add this type and function near the parsing section in
`packages/core/src/scrapers/facebook.ts`:
```ts
type FacebookResponseKind = "search" | "item" | "auth_gated" | "unavailable" | "unknown";
@@ -128,7 +145,8 @@ export function classifyFacebookResponse(htmlString: HTMLString, responseUrl: st
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "classifies"`
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "classifies"`
Expected: PASS
- [ ] **Step 5: Commit**
@@ -141,8 +159,11 @@ git commit -m "refactor: add facebook response classification"
### Task 2: Add Bootstrap Candidate Extraction
**Files:**
- Modify: `packages/core/test/facebook-core.test.ts`
- Modify: `packages/core/src/scrapers/facebook.ts`
- Test: `packages/core/test/facebook-core.test.ts`
- [ ] **Step 1: Write the failing tests**
@@ -185,7 +206,8 @@ test("keeps candidate order stable for later scoring", () => {
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "bootstrap candidates"`
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "bootstrap candidates"`
Expected: FAIL because `extractFacebookBootstrapCandidates` does not exist.
- [ ] **Step 3: Write minimal implementation**
@@ -218,7 +240,8 @@ export function extractFacebookBootstrapCandidates(htmlString: HTMLString): Reco
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "bootstrap candidates"`
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "bootstrap candidates"`
Expected: PASS
- [ ] **Step 5: Commit**
@@ -231,10 +254,15 @@ git commit -m "refactor: add facebook bootstrap candidate extraction"
### Task 3: Replace Search Parsing With Candidate Scoring
**Files:**
- Modify: `packages/core/test/facebook-core.test.ts`
- Modify: `packages/core/test/facebook-integration.test.ts`
- Modify: `packages/core/src/scrapers/facebook.ts`
- Test: `packages/core/test/facebook-core.test.ts`
- Test: `packages/core/test/facebook-integration.test.ts`
- [ ] **Step 1: Write the failing tests**
@@ -323,12 +351,15 @@ const mockSearchHtml = `
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "Comet bootstrap candidates"`
Expected: FAIL because the current search extractor only understands legacy `marketplace_search` shapes.
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "Comet bootstrap candidates"`
Expected: FAIL because the current search extractor only understands legacy
`marketplace_search` shapes.
- [ ] **Step 3: Write minimal implementation**
Replace the search extraction internals in `extractFacebookMarketplaceData()` with candidate scoring like this:
Replace the search extraction internals in `extractFacebookMarketplaceData()` with
candidate scoring like this:
```ts
function findSearchEdges(candidate: unknown): FacebookEdge[] | null {
@@ -383,7 +414,8 @@ export function extractFacebookMarketplaceData(htmlString: HTMLString): Facebook
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-core.test.ts packages/core/test/facebook-integration.test.ts`
Run:
`bun test packages/core/test/facebook-core.test.ts packages/core/test/facebook-integration.test.ts`
Expected: PASS for the rewritten search fixtures and existing unaffected tests.
- [ ] **Step 5: Commit**
@@ -396,8 +428,11 @@ git commit -m "refactor: rewrite facebook search parser for comet bootstrap"
### Task 4: Replace Item Parsing With Candidate Scoring
**Files:**
- Modify: `packages/core/test/facebook-core.test.ts`
- Modify: `packages/core/src/scrapers/facebook.ts`
- Test: `packages/core/test/facebook-core.test.ts`
- [ ] **Step 1: Write the failing tests**
@@ -438,7 +473,8 @@ test("extracts item details from Comet permalink bootstrap candidates", () => {
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "Comet permalink bootstrap"`
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "Comet permalink bootstrap"`
Expected: FAIL because the current item extractor depends on legacy permalink markers.
- [ ] **Step 3: Write minimal implementation**
@@ -491,8 +527,8 @@ export function extractFacebookItemData(htmlString: HTMLString): FacebookMarketp
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-core.test.ts`
Expected: PASS for current-shape item tests and remaining parser tests.
Run: `bun test packages/core/test/facebook-core.test.ts` Expected: PASS for
current-shape item tests and remaining parser tests.
- [ ] **Step 5: Commit**
@@ -504,8 +540,11 @@ git commit -m "refactor: rewrite facebook item parser for comet bootstrap"
### Task 5: Add HTML Fallback Extraction
**Files:**
- Modify: `packages/core/test/facebook-core.test.ts`
- Modify: `packages/core/src/scrapers/facebook.ts`
- Test: `packages/core/test/facebook-core.test.ts`
- [ ] **Step 1: Write the failing tests**
@@ -549,8 +588,10 @@ test("falls back to rendered item HTML when bootstrap payloads are undecodable",
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "falls back"`
Expected: FAIL because the extractor currently returns `null` without a structured candidate.
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "falls back"`
Expected: FAIL because the extractor currently returns `null` without a structured
candidate.
- [ ] **Step 3: Write minimal implementation**
@@ -607,11 +648,13 @@ function extractItemFallback(htmlString: HTMLString): FacebookMarketplaceItem |
}
```
Then call these helpers as the last fallback inside `extractFacebookMarketplaceData()` and `extractFacebookItemData()`.
Then call these helpers as the last fallback inside `extractFacebookMarketplaceData()`
and `extractFacebookItemData()`.
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-core.test.ts --test-name-pattern "falls back"`
Run:
`bun test packages/core/test/facebook-core.test.ts --test-name-pattern "falls back"`
Expected: PASS
- [ ] **Step 5: Commit**
@@ -624,8 +667,11 @@ git commit -m "refactor: add facebook html fallbacks"
### Task 6: Wire Route-Aware Failures Into Entry Points
**Files:**
- Modify: `packages/core/test/facebook-integration.test.ts`
- Modify: `packages/core/src/scrapers/facebook.ts`
- Test: `packages/core/test/facebook-integration.test.ts`
- [ ] **Step 1: Write the failing tests**
@@ -664,8 +710,10 @@ test("returns null for unavailable item responses", async () => {
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/facebook-integration.test.ts --test-name-pattern "auth-gated|unavailable"`
Expected: FAIL because the entrypoints do not yet classify successful HTML responses by route/auth state.
Run:
`bun test packages/core/test/facebook-integration.test.ts --test-name-pattern "auth-gated|unavailable"`
Expected: FAIL because the entrypoints do not yet classify successful HTML responses by
route/auth state.
- [ ] **Step 3: Write minimal implementation**
@@ -690,12 +738,13 @@ if (itemResponseClass.kind === "unavailable") {
}
```
Use the actual response URL from `fetchHtml` plumbing if that helper is extended to return both HTML and final URL; otherwise start by threading final URL support through the fetch helper in the same task.
Use the actual response URL from `fetchHtml` plumbing if that helper is extended to
return both HTML and final URL; otherwise start by threading final URL support through
the fetch helper in the same task.
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/facebook-integration.test.ts`
Expected: PASS
Run: `bun test packages/core/test/facebook-integration.test.ts` Expected: PASS
- [ ] **Step 5: Commit**
@@ -707,19 +756,22 @@ git commit -m "refactor: handle facebook route-aware failure states"
### Task 7: Run Full Verification And Live Probe
**Files:**
- Modify: `packages/core/src/scrapers/facebook.ts` if small cleanup is required
- Modify: `packages/core/test/facebook-core.test.ts` if small cleanup is required
- Modify: `packages/core/test/facebook-integration.test.ts` if small cleanup is required
- [ ] **Step 1: Run focused Facebook tests**
Run: `bun test packages/core/test/facebook-core.test.ts packages/core/test/facebook-integration.test.ts`
Run:
`bun test packages/core/test/facebook-core.test.ts packages/core/test/facebook-integration.test.ts`
Expected: PASS
- [ ] **Step 2: Run broader core tests**
Run: `bun test packages/core/test`
Expected: PASS
Run: `bun test packages/core/test` Expected: PASS
- [ ] **Step 3: Run live authenticated Facebook probe**
@@ -742,11 +794,14 @@ if (results[0]?.url) {
Expected:
- search returns at least one result
- item fetch returns non-null for the first live result when the route is not stale/unavailable
- item fetch returns non-null for the first live result when the route is not
stale/unavailable
- [ ] **Step 4: Make any minimal cleanup needed to keep tests and live probe green**
If cleanup is needed, keep it limited to naming, dead-code removal caused by the rewrite, or small parser corrections directly exposed by the verification commands.
If cleanup is needed, keep it limited to naming, dead-code removal caused by the
rewrite, or small parser corrections directly exposed by the verification commands.
- [ ] **Step 5: Re-run verification**
@@ -767,6 +822,11 @@ git commit -m "refactor: complete facebook comet scraper rewrite"
## Self-Review
- Spec coverage: the plan covers classification, route-aware search parsing, route-aware item parsing, HTML fallbacks, explicit failure-state handling, test replacement, and live verification.
- Placeholder scan: no `TODO`, `TBD`, or unspecified “handle appropriately” steps remain.
- Type consistency: all planned functions and types use the same names across tasks: `classifyFacebookResponse`, `extractFacebookBootstrapCandidates`, `extractFacebookMarketplaceData`, and `extractFacebookItemData`.
- Spec coverage: the plan covers classification, route-aware search parsing, route-aware
item parsing, HTML fallbacks, explicit failure-state handling, test replacement, and
live verification.
- Placeholder scan: no `TODO`, `TBD`, or unspecified “handle appropriately” steps
remain.
- Type consistency: all planned functions and types use the same names across tasks:
`classifyFacebookResponse`, `extractFacebookBootstrapCandidates`,
`extractFacebookMarketplaceData`, and `extractFacebookItemData`.

190
docs/superpowers/plans/2026-04-22-unstable-listing-mode.md
View File

@@ -1,63 +1,75 @@
# Unstable Listing Mode Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or superpowers:executing-plans
> to implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add an optional shared mode across Facebook, eBay, and Kijiji that moves listings priced below 80% of the median into `unstableResults`, while preserving current default response shapes.
**Goal:** Add an optional shared mode across Facebook, eBay, and Kijiji that moves
listings priced below 80% of the median into `unstableResults`, while preserving current
default response shapes.
**Architecture:** Introduce a shared generic classifier in `packages/core` that splits any listing array into `results` and `unstableResults` using the same median-based rule. Then thread one opt-in flag through the scraper entrypoints, API routes, and MCP tool definitions so all surfaces expose the same behavior without changing existing defaults.
**Architecture:** Introduce a shared generic classifier in `packages/core` that splits
any listing array into `results` and `unstableResults` using the same median-based rule.
Then thread one opt-in flag through the scraper entrypoints, API routes, and MCP tool
definitions so all surfaces expose the same behavior without changing existing defaults.
**Tech Stack:** Bun, TypeScript, Bun test, workspace packages, JSON-RPC MCP server
---
* * *
## File Map
- Create: `packages/core/src/utils/unstable.ts`
Purpose: shared generic median/cutoff classifier for listing arrays.
- Modify: `packages/core/src/types/common.ts`
Purpose: add shared mode types used by scrapers and adapters.
- Modify: `packages/core/src/index.ts`
Purpose: export the new shared classifier/types.
- Modify: `packages/core/src/scrapers/facebook.ts`
Purpose: add the optional mode flag and return bucketed results when enabled.
- Modify: `packages/core/src/scrapers/ebay.ts`
Purpose: add the optional mode flag and return bucketed results when enabled.
- Modify: `packages/core/src/scrapers/kijiji.ts`
Purpose: add the optional mode flag and return bucketed results when enabled.
- Create: `packages/core/test/unstable-listing-mode.test.ts`
Purpose: lock the shared classifier behavior with direct unit tests.
- Modify: `packages/core/test/facebook-core.test.ts`
Purpose: prove Facebook preserves default arrays and returns buckets when enabled.
- Modify: `packages/core/test/ebay-core.test.ts`
Purpose: prove eBay preserves default arrays and returns buckets when enabled.
- Modify: `packages/core/test/kijiji-core.test.ts`
Purpose: prove Kijiji preserves default arrays and returns buckets when enabled.
- Modify: `packages/api-server/src/routes/facebook.ts`
Purpose: expose a shared opt-in query parameter and preserve default response shape.
- Modify: `packages/api-server/src/routes/ebay.ts`
Purpose: expose the same query parameter and preserve default response shape.
- Modify: `packages/api-server/src/routes/kijiji.ts`
Purpose: expose the same query parameter and preserve default response shape.
- Modify: `packages/api-server/test/routes.test.ts`
Purpose: verify route forwarding and route response-shape switching.
- Modify: `packages/mcp-server/src/protocol/tools.ts`
Purpose: document the optional unstable mode in all search tools.
- Modify: `packages/mcp-server/src/protocol/handler.ts`
Purpose: forward the optional mode to API routes for all search tools.
- Modify: `packages/mcp-server/test/protocol.test.ts`
Purpose: verify MCP tool metadata and forwarded URLs include the new option.
- Create: `packages/core/src/utils/unstable.ts` Purpose: shared generic median/cutoff
classifier for listing arrays.
- Modify: `packages/core/src/types/common.ts` Purpose: add shared mode types used by
scrapers and adapters.
- Modify: `packages/core/src/index.ts` Purpose: export the new shared classifier/types.
- Modify: `packages/core/src/scrapers/facebook.ts` Purpose: add the optional mode flag
and return bucketed results when enabled.
- Modify: `packages/core/src/scrapers/ebay.ts` Purpose: add the optional mode flag and
return bucketed results when enabled.
- Modify: `packages/core/src/scrapers/kijiji.ts` Purpose: add the optional mode flag and
return bucketed results when enabled.
- Create: `packages/core/test/unstable-listing-mode.test.ts` Purpose: lock the shared
classifier behavior with direct unit tests.
- Modify: `packages/core/test/facebook-core.test.ts` Purpose: prove Facebook preserves
default arrays and returns buckets when enabled.
- Modify: `packages/core/test/ebay-core.test.ts` Purpose: prove eBay preserves default
arrays and returns buckets when enabled.
- Modify: `packages/core/test/kijiji-core.test.ts` Purpose: prove Kijiji preserves
default arrays and returns buckets when enabled.
- Modify: `packages/api-server/src/routes/facebook.ts` Purpose: expose a shared opt-in
query parameter and preserve default response shape.
- Modify: `packages/api-server/src/routes/ebay.ts` Purpose: expose the same query
parameter and preserve default response shape.
- Modify: `packages/api-server/src/routes/kijiji.ts` Purpose: expose the same query
parameter and preserve default response shape.
- Modify: `packages/api-server/test/routes.test.ts` Purpose: verify route forwarding and
route response-shape switching.
- Modify: `packages/mcp-server/src/protocol/tools.ts` Purpose: document the optional
unstable mode in all search tools.
- Modify: `packages/mcp-server/src/protocol/handler.ts` Purpose: forward the optional
mode to API routes for all search tools.
- Modify: `packages/mcp-server/test/protocol.test.ts` Purpose: verify MCP tool metadata
and forwarded URLs include the new option.
### Task 1: Add the shared unstable-listing classifier
**Files:**
- Create: `packages/core/src/utils/unstable.ts`
- Modify: `packages/core/src/types/common.ts`
- Modify: `packages/core/src/index.ts`
- Test: `packages/core/test/unstable-listing-mode.test.ts`
- [ ] **Step 1: Write the failing test**
Create `packages/core/test/unstable-listing-mode.test.ts` with focused shared-behavior coverage:
Create `packages/core/test/unstable-listing-mode.test.ts` with focused shared-behavior
coverage:
```ts
import { describe, expect, test } from "bun:test";
@@ -127,8 +139,8 @@ describe("classifyUnstableListings", () => {
- [ ] **Step 2: Run test to verify it fails**
Run: `bun test packages/core/test/unstable-listing-mode.test.ts`
Expected: FAIL because `classifyUnstableListings` and the shared mode types do not exist yet.
Run: `bun test packages/core/test/unstable-listing-mode.test.ts` Expected: FAIL because
`classifyUnstableListings` and the shared mode types do not exist yet.
- [ ] **Step 3: Write minimal implementation**
@@ -202,8 +214,8 @@ export { classifyUnstableListings } from "./utils/unstable";
- [ ] **Step 4: Run test to verify it passes**
Run: `bun test packages/core/test/unstable-listing-mode.test.ts`
Expected: PASS with 4 passing tests.
Run: `bun test packages/core/test/unstable-listing-mode.test.ts` Expected: PASS with 4
passing tests.
- [ ] **Step 5: Commit**
@@ -215,16 +227,24 @@ git commit -m "feat: add shared unstable listing classifier"
### Task 2: Thread the optional mode through all core scrapers
**Files:**
- Modify: `packages/core/src/scrapers/facebook.ts`
- Modify: `packages/core/src/scrapers/ebay.ts`
- Modify: `packages/core/src/scrapers/kijiji.ts`
- Modify: `packages/core/test/facebook-core.test.ts`
- Modify: `packages/core/test/ebay-core.test.ts`
- Modify: `packages/core/test/kijiji-core.test.ts`
- [ ] **Step 1: Write the failing tests**
Add one focused opt-in test per scraper. Use the new shared classifier through the public scraper entrypoints instead of testing internal helpers.
Add one focused opt-in test per scraper.
Use the new shared classifier through the public scraper entrypoints instead of testing
internal helpers.
In `packages/core/test/facebook-core.test.ts`, add:
@@ -286,7 +306,8 @@ test("fetchKijijiItems returns stable and unstable buckets when unstable mode is
});
```
Also add one default-mode assertion in one existing scraper test file, for example in `packages/core/test/facebook-core.test.ts`:
Also add one default-mode assertion in one existing scraper test file, for example in
`packages/core/test/facebook-core.test.ts`:
```ts
test("fetchFacebookItems keeps returning an array by default", async () => {
@@ -307,8 +328,10 @@ test("fetchFacebookItems keeps returning an array by default", async () => {
- [ ] **Step 2: Run tests to verify they fail**
Run: `bun test packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts`
Expected: FAIL because the scraper signatures do not yet accept the new option and still always return arrays.
Run:
`bun test packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts`
Expected: FAIL because the scraper signatures do not yet accept the new option and still
always return arrays.
- [ ] **Step 3: Write minimal implementation**
@@ -322,7 +345,8 @@ import {
} from "../index";
```
In `packages/core/src/scrapers/facebook.ts`, extend the default export signature and branch at the end:
In `packages/core/src/scrapers/facebook.ts`, extend the default export signature and
branch at the end:
```ts
export default async function fetchFacebookItems(
@@ -371,7 +395,8 @@ export default async function fetchEbayItems(
}
```
In `packages/core/src/scrapers/kijiji.ts`, add the same final argument after `listingOptions`:
In `packages/core/src/scrapers/kijiji.ts`, add the same final argument after
`listingOptions`:
```ts
export default async function fetchKijijiItems(
@@ -392,12 +417,15 @@ export default async function fetchKijijiItems(
}
```
Keep the default branch untouched in all three files so existing callers still receive arrays.
Keep the default branch untouched in all three files so existing callers still receive
arrays.
- [ ] **Step 4: Run tests to verify they pass**
Run: `bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts`
Expected: PASS, including the new opt-in bucket assertions and the default-array regression assertion.
Run:
`bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts`
Expected: PASS, including the new opt-in bucket assertions and the default-array
regression assertion.
- [ ] **Step 5: Commit**
@@ -409,14 +437,19 @@ git commit -m "feat: add unstable mode to scraper results"
### Task 3: Expose unstable mode in API routes
**Files:**
- Modify: `packages/api-server/src/routes/facebook.ts`
- Modify: `packages/api-server/src/routes/ebay.ts`
- Modify: `packages/api-server/src/routes/kijiji.ts`
- Modify: `packages/api-server/test/routes.test.ts`
- [ ] **Step 1: Write the failing tests**
Extend `packages/api-server/test/routes.test.ts` with route-forwarding coverage for the new query parameter:
Extend `packages/api-server/test/routes.test.ts` with route-forwarding coverage for the
new query parameter:
```ts
test("facebookRoute forwards unstableFilter=true to core", async () => {
@@ -480,8 +513,8 @@ test("kijijiRoute forwards unstableFilter=true to core", async () => {
- [ ] **Step 2: Run tests to verify they fail**
Run: `bun test packages/api-server/test/routes.test.ts`
Expected: FAIL because the routes do not yet parse or forward `unstableFilter`.
Run: `bun test packages/api-server/test/routes.test.ts` Expected: FAIL because the
routes do not yet parse or forward `unstableFilter`.
- [ ] **Step 3: Write minimal implementation**
@@ -533,12 +566,14 @@ const items = await fetchKijijiItems(
);
```
Do not add any response wrapper logic in the routes; simply return whatever the core scraper returns so the default array path remains unchanged.
Do not add any response wrapper logic in the routes; simply return whatever the core
scraper returns so the default array path remains unchanged.
- [ ] **Step 4: Run tests to verify they pass**
Run: `bun test packages/api-server/test/routes.test.ts`
Expected: PASS, including existing cookie-parameter regression tests and the new unstable-mode forwarding assertions.
Run: `bun test packages/api-server/test/routes.test.ts` Expected: PASS, including
existing cookie-parameter regression tests and the new unstable-mode forwarding
assertions.
- [ ] **Step 5: Commit**
@@ -550,13 +585,17 @@ git commit -m "feat: expose unstable mode in api routes"
### Task 4: Document and forward unstable mode in MCP tools
**Files:**
- Modify: `packages/mcp-server/src/protocol/tools.ts`
- Modify: `packages/mcp-server/src/protocol/handler.ts`
- Modify: `packages/mcp-server/test/protocol.test.ts`
- [ ] **Step 1: Write the failing tests**
Extend `packages/mcp-server/test/protocol.test.ts` with metadata and forwarding coverage:
Extend `packages/mcp-server/test/protocol.test.ts` with metadata and forwarding
coverage:
```ts
test("search tools document unstable listing mode", () => {
@@ -601,12 +640,14 @@ Mirror the forwarding assertion for `search_kijiji` and `search_ebay` in the sam
- [ ] **Step 2: Run tests to verify they fail**
Run: `bun test packages/mcp-server/test/protocol.test.ts`
Expected: FAIL because the tools do not yet describe `unstableFilter` and the handler does not append it to API URLs.
Run: `bun test packages/mcp-server/test/protocol.test.ts` Expected: FAIL because the
tools do not yet describe `unstableFilter` and the handler does not append it to API
URLs.
- [ ] **Step 3: Write minimal implementation**
In `packages/mcp-server/src/protocol/tools.ts`, add the same optional property to all three tools:
In `packages/mcp-server/src/protocol/tools.ts`, add the same optional property to all
three tools:
```ts
unstableFilter: {
@@ -617,7 +658,8 @@ unstableFilter: {
},
```
In `packages/mcp-server/src/protocol/handler.ts`, append the shared flag in each search branch:
In `packages/mcp-server/src/protocol/handler.ts`, append the shared flag in each search
branch:
```ts
if (args.unstableFilter !== undefined) {
@@ -629,8 +671,8 @@ Add that snippet to the `search_kijiji`, `search_facebook`, and `search_ebay` br
- [ ] **Step 4: Run tests to verify they pass**
Run: `bun test packages/mcp-server/test/protocol.test.ts`
Expected: PASS, including the new tool-schema assertions and URL-forwarding assertions.
Run: `bun test packages/mcp-server/test/protocol.test.ts` Expected: PASS, including the
new tool-schema assertions and URL-forwarding assertions.
- [ ] **Step 5: Commit**
@@ -642,21 +684,23 @@ git commit -m "docs: expose unstable mode in mcp tools"
### Task 5: Verify the full cross-package feature end to end
**Files:**
- No code changes expected.
- [ ] **Step 1: Run the focused package tests**
Run: `bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts packages/api-server/test/routes.test.ts packages/mcp-server/test/protocol.test.ts`
Run:
`bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts packages/api-server/test/routes.test.ts packages/mcp-server/test/protocol.test.ts`
Expected: PASS with zero failing tests.
- [ ] **Step 2: Run the broader workspace verification**
Run: `bun run ci`
Expected: PASS with clean workspace validation.
Run: `bun run ci` Expected: PASS with clean workspace validation.
- [ ] **Step 3: Commit verification-only follow-ups if needed**
If verification forced any tiny fixes, commit them immediately after the fix with a focused message, for example:
If verification forced any tiny fixes, commit them immediately after the fix with a
focused message, for example:
```bash
git add <exact files changed>
@@ -667,6 +711,8 @@ If no files changed during verification, skip this commit step.
## Self-Review
- Spec coverage: shared classifier, all three scrapers, API exposure, MCP documentation, and tests are each mapped to a task.
- Placeholder scan: no `TODO`, `TBD`, or "write tests later" placeholders remain.
- Type consistency: the plan uses one shared flag name, `unstableFilter`, and one shared core option, `hideUnstableResults`, across all tasks.
- Spec coverage: shared classifier, all three scrapers, API exposure, MCP documentation,
and tests are each mapped to a task.
- Placeholder scan: no `TODO`, `TBD`, or “write tests later” placeholders remain.
- Type consistency: the plan uses one shared flag name, `unstableFilter`, and one shared
core option, `hideUnstableResults`, across all tasks.

104
docs/superpowers/plans/2026-04-28-code-smell-cleanup.md
View File

@@ -1,14 +1,22 @@
# Code Smell Cleanup Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or superpowers:executing-plans
> to implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Fix concrete code smells found in repo review without changing marketplace behavior or relaxing lint/type rules.
**Goal:** Fix concrete code smells found in repo review without changing marketplace
behavior or relaxing lint/type rules.
**Architecture:** Start with correctness bugs at transport boundaries, then remove secret-leaking query/log paths, then reduce duplicate parsing and HTTP code. Keep marketplace behavior inside `packages/core`, API routes thin, and MCP as JSON-RPC transport only.
**Architecture:** Start with correctness bugs at transport boundaries, then remove
secret-leaking query/log paths, then reduce duplicate parsing and HTTP code.
Keep marketplace behavior inside `packages/core`, API routes thin, and MCP as JSON-RPC
transport only.
**Tech Stack:** Bun `1.3.13`, TypeScript strict mode, `bun:test`, Biome, framework-free `Bun.serve` adapters.
**Tech Stack:** Bun `1.3.13`, TypeScript strict mode, `bun:test`, Biome, framework-free
`Bun.serve` adapters.
---
* * *
## File Structure
@@ -18,7 +26,8 @@
- Extract shared API call/query-param helpers.
- Stop logging full URLs with cookie-bearing params.
- Modify: `packages/mcp-server/src/protocol/tools.ts`
- Remove `cookies` from Kijiji MCP schema or mark it as unsupported after API route no longer accepts it.
- Remove `cookies` from Kijiji MCP schema or mark it as unsupported after API route no
longer accepts it.
- Modify: `packages/mcp-server/test/protocol.test.ts`
- Add coverage for `id: 0`.
- Add coverage for zero-valued numeric args.
@@ -53,12 +62,15 @@
- Replace `console.error` with repo logger.
- Modify: `packages/core/test/setup.ts`
- Remove redundant comments and make fetch-mock policy explicit.
- Test: existing package tests under `packages/core/test`, `packages/api-server/test`, `packages/mcp-server/test`.
- Test: existing package tests under `packages/core/test`, `packages/api-server/test`,
`packages/mcp-server/test`.
## Task 1: Fix MCP JSON-RPC `id: 0` Handling
**Files:**
- Modify: `packages/mcp-server/src/protocol/handler.ts:61-74`
- Test: `packages/mcp-server/test/protocol.test.ts`
- [ ] **Step 1: Write failing test for `id: 0`**
@@ -137,7 +149,9 @@ git commit -m "fix: preserve zero json-rpc ids"
## Task 2: Preserve Zero Numeric MCP Arguments
**Files:**
- Modify: `packages/mcp-server/src/protocol/handler.ts:107-216`
- Test: `packages/mcp-server/test/protocol.test.ts`
- [ ] **Step 1: Write failing tests for zero-valued params**
@@ -288,10 +302,15 @@ git commit -m "fix: forward zero-valued mcp params"
## Task 3: Remove Cookie Query Path From MCP and API
**Files:**
- Modify: `packages/mcp-server/src/protocol/tools.ts:55-59`
- Modify: `packages/mcp-server/src/protocol/handler.ts:119`
- Modify: `packages/api-server/src/routes/kijiji.ts:65`
- Test: `packages/mcp-server/test/protocol.test.ts`
- Test: `packages/api-server/test/routes.test.ts`
- [ ] **Step 1: Update MCP tests for no cookie exposure**
@@ -341,7 +360,8 @@ test("search_kijiji should not forward cookies query parameters", async () => {
- [ ] **Step 2: Update API test expectation**
In `packages/api-server/test/routes.test.ts`, replace `kijijiRoute passes cookies query parameter` test with:
In `packages/api-server/test/routes.test.ts`, replace
`kijijiRoute passes cookies query parameter` test with:
```ts
test("kijijiRoute ignores cookies query parameter", async () => {
@@ -374,13 +394,15 @@ test("kijijiRoute ignores cookies query parameter", async () => {
- [ ] **Step 3: Run tests to verify failure**
Run: `bun test packages/mcp-server/test/protocol.test.ts packages/api-server/test/routes.test.ts`
Run:
`bun test packages/mcp-server/test/protocol.test.ts packages/api-server/test/routes.test.ts`
Expected: FAIL because Kijiji cookie query is still exposed/forwarded.
- [ ] **Step 4: Remove Kijiji cookie schema and forwarding**
Delete `cookies` property from `search_kijiji` in `packages/mcp-server/src/protocol/tools.ts`.
Delete `cookies` property from `search_kijiji` in
`packages/mcp-server/src/protocol/tools.ts`.
Delete this line from `packages/mcp-server/src/protocol/handler.ts`:
@@ -396,7 +418,8 @@ cookies: reqUrl.searchParams.get("cookies") || undefined,
- [ ] **Step 5: Run tests**
Run: `bun test packages/mcp-server/test/protocol.test.ts packages/api-server/test/routes.test.ts`
Run:
`bun test packages/mcp-server/test/protocol.test.ts packages/api-server/test/routes.test.ts`
Expected: PASS.
@@ -410,10 +433,15 @@ git commit -m "fix: remove cookie query forwarding"
## Task 4: Add Strict API Integer Parsing
**Files:**
- Create: `packages/api-server/src/routes/helpers.ts`
- Modify: `packages/api-server/src/routes/facebook.ts`
- Modify: `packages/api-server/src/routes/ebay.ts`
- Modify: `packages/api-server/src/routes/kijiji.ts`
- Test: `packages/api-server/test/routes.test.ts`
- [ ] **Step 1: Write failing API validation tests**
@@ -560,7 +588,9 @@ git commit -m "fix: strictly parse route integers"
## Task 5: De-Duplicate MCP API Calls
**Files:**
- Modify: `packages/mcp-server/src/protocol/handler.ts`
- Test: `packages/mcp-server/test/protocol.test.ts`
- [ ] **Step 1: Add regression test for successful tool result after helper extraction**
@@ -645,7 +675,8 @@ Use `"facebook"` and `"ebay"` in their branches.
- [ ] **Step 4: Run MCP tests and build**
Run: `bun test packages/mcp-server/test/protocol.test.ts && bun run --cwd packages/mcp-server build`
Run:
`bun test packages/mcp-server/test/protocol.test.ts && bun run --cwd packages/mcp-server build`
Expected: PASS.
@@ -659,11 +690,17 @@ git commit -m "refactor: share mcp api calls"
## Task 6: Consolidate Core HTTP Fetching
**Files:**
- Modify: `packages/core/src/utils/http.ts`
- Modify: `packages/core/src/scrapers/facebook.ts`
- Modify: `packages/core/src/scrapers/ebay.ts`
- Test: `packages/core/test/http.test.ts`
- Test: `packages/core/test/facebook-core.test.ts`
- Test: `packages/core/test/ebay-core.test.ts`
- [ ] **Step 1: Add shared HTTP test for response URL and deterministic jitter**
@@ -695,7 +732,8 @@ test("fetchHtml can return response URL", async () => {
});
```
If current `Response.url` cannot be set in Bun tests, use a mocked object cast to `Response` instead:
If current `Response.url` cannot be set in Bun tests, use a mocked object cast to
`Response` instead:
```ts
global.fetch = mock(() =>
@@ -827,7 +865,8 @@ Update error property reads from `err.status` to `err.statusCode`.
- [ ] **Step 5: Replace eBay direct fetch with shared helper**
In `packages/core/src/scrapers/ebay.ts`, import `fetchHtml` and `HttpError` from `../utils/http`.
In `packages/core/src/scrapers/ebay.ts`, import `fetchHtml` and `HttpError` from
`../utils/http`.
Replace direct `fetch` block with:
@@ -845,7 +884,8 @@ logger.error(`Failed to fetch eBay search (${err.statusCode}): ${err.message}`);
- [ ] **Step 6: Run core tests**
Run: `bun test packages/core/test/http.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts`
Run:
`bun test packages/core/test/http.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts`
Expected: PASS.
@@ -865,15 +905,20 @@ git commit -m "refactor: share scraper http fetching"
## Task 7: Clean Kijiji Dead Code and Logging
**Files:**
- Modify: `packages/core/src/scrapers/kijiji.ts`
- Test: `packages/core/test/kijiji-core.test.ts`
- Test: `packages/core/test/kijiji-integration.test.ts`
- [ ] **Step 1: Verify `_parseListing` has no callers**
Run: `rg "_parseListing|parseListing" packages/core packages/api-server packages/mcp-server`
Run:
`rg "_parseListing|parseListing" packages/core packages/api-server packages/mcp-server`
Expected: only `_parseListing` definition appears. If any caller appears, stop and update this task to preserve behavior.
Expected: only `_parseListing` definition appears.
If any caller appears, stop and update this task to preserve behavior.
- [ ] **Step 2: Delete dead function**
@@ -911,7 +956,8 @@ Replace `console.error(...)` calls with `logger.error(...)` preserving message t
- [ ] **Step 4: Run Kijiji tests**
Run: `bun test packages/core/test/kijiji-core.test.ts packages/core/test/kijiji-integration.test.ts`
Run:
`bun test packages/core/test/kijiji-core.test.ts packages/core/test/kijiji-integration.test.ts`
Expected: PASS.
@@ -925,7 +971,9 @@ git commit -m "refactor: clean kijiji scraper internals"
## Task 8: Clean Test Setup Comments and Enforce Fetch Mocking
**Files:**
- Modify: `packages/core/test/setup.ts`
- Test: core test suite
- [ ] **Step 1: Update setup file**
@@ -942,7 +990,8 @@ global.fetch = (() => {
Run: `bun test packages/core/test`
Expected: PASS. If failures occur, fix individual tests by mocking `global.fetch` in `beforeEach` and restoring in `afterEach`.
Expected: PASS. If failures occur, fix individual tests by mocking `global.fetch` in
`beforeEach` and restoring in `afterEach`.
- [ ] **Step 3: Commit**
@@ -954,6 +1003,7 @@ git commit -m "test: require explicit fetch mocks"
## Task 9: Final Verification
**Files:**
- Verify all touched packages.
- [ ] **Step 1: Run full deterministic tests**
@@ -991,9 +1041,13 @@ git commit -m "chore: finish code smell cleanup"
## Self-Review
- Spec coverage: all reviewed smells are covered: JSON-RPC id bug, zero args, cookie query leak, strict integer parsing, duplicate route/MCP helper code, duplicate HTTP clients, dead Kijiji function, direct timers/logging, stale setup comments.
- Placeholder scan: no TBD/TODO/fill-in placeholders remain. Each task has target files, code snippets, commands, and expected results.
- Type consistency: route helper names, MCP helper names, and shared HTTP option names are used consistently across tasks.
- Spec coverage: all reviewed smells are covered: JSON-RPC id bug, zero args, cookie
query leak, strict integer parsing, duplicate route/MCP helper code, duplicate HTTP
clients, dead Kijiji function, direct timers/logging, stale setup comments.
- Placeholder scan: no TBD/TODO/fill-in placeholders remain.
Each task has target files, code snippets, commands, and expected results.
- Type consistency: route helper names, MCP helper names, and shared HTTP option names
are used consistently across tasks.
## Execution Handoff
@@ -1001,5 +1055,7 @@ Plan complete and saved to `docs/superpowers/plans/2026-04-28-code-smell-cleanup
Two execution options:
1. Subagent-Driven (recommended) - dispatch fresh subagent per task, review between tasks, fast iteration.
2. Inline Execution - execute tasks in this session using executing-plans, batch execution with checkpoints.
1. Subagent-Driven (recommended) - dispatch fresh subagent per task, review between
tasks, fast iteration.
2. Inline Execution - execute tasks in this session using executing-plans, batch
execution with checkpoints.

52
docs/superpowers/plans/2026-04-30-ebay-dollar-price-inputs.md
View File

@@ -1,38 +1,60 @@
# Marketplace Dollar Price Inputs Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to
> implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make public marketplace price inputs use dollars while preserving core scraper cent-based filtering.
**Goal:** Make public marketplace price inputs use dollars while preserving core scraper
cent-based filtering.
**Architecture:** API server owns HTTP query parsing and converts dollar amounts to cents before calling core. MCP server keeps forwarding numeric dollar values as query params. Core scraper internals remain unchanged because parsed listing prices already use cents. This applies to eBay `minPrice`/`maxPrice` and Kijiji `priceMin`/`priceMax`; Facebook exposes no price filter inputs.
**Architecture:** API server owns HTTP query parsing and converts dollar amounts to
cents before calling core.
MCP server keeps forwarding numeric dollar values as query params.
Core scraper internals remain unchanged because parsed listing prices already use cents.
This applies to eBay `minPrice`/`maxPrice` and Kijiji `priceMin`/`priceMax`; Facebook
exposes no price filter inputs.
**Tech Stack:** Bun, TypeScript, `bun:test`, MCP JSON-RPC adapter, framework-free Bun HTTP routes.
**Tech Stack:** Bun, TypeScript, `bun:test`, MCP JSON-RPC adapter, framework-free Bun
HTTP routes.
---
* * *
### Task 1: API Dollar Parsing
**Files:**
- Modify: `packages/api-server/src/routes/helpers.ts`
- Modify: `packages/api-server/src/routes/ebay.ts`
- Modify: `packages/api-server/src/routes/kijiji.ts`
- Test: `packages/api-server/test/routes.test.ts`
- [ ] **Step 1: Add failing API route tests**
Add tests proving eBay `minPrice=999.99` / `maxPrice=1000` and Kijiji `priceMin=999.99` / `priceMax=1000` are forwarded to core as `99999` and `100000` cents. Add validation tests for empty, whitespace, negative, hex, mixed text, and malformed decimal price values.
Add tests proving eBay `minPrice=999.99` / `maxPrice=1000` and Kijiji `priceMin=999.99`
/ `priceMax=1000` are forwarded to core as `99999` and `100000` cents.
Add validation tests for empty, whitespace, negative, hex, mixed text, and malformed
decimal price values.
Run: `bun test packages/api-server/test/routes.test.ts`
Expected: new forwarding tests fail because route currently rejects decimals and forwards integer dollars unchanged.
Expected: new forwarding tests fail because route currently rejects decimals and
forwards integer dollars unchanged.
- [ ] **Step 2: Implement dollar parser helper**
Add `parseDollarPriceParam(searchParams, name)` in `packages/api-server/src/routes/helpers.ts`. Accept `0`, `1000`, `999.99`, and `0.99`. Reject values that do not match `^\d+(?:\.\d{1,2})?$`. Convert to cents with `Math.round(Number(rawValue) * 100)`.
Add `parseDollarPriceParam(searchParams, name)` in
`packages/api-server/src/routes/helpers.ts`. Accept `0`, `1000`, `999.99`, and `0.99`.
Reject values that do not match `^\d+(?:\.\d{1,2})?$`. Convert to cents with
`Math.round(Number(rawValue) * 100)`.
- [ ] **Step 3: Use dollar parser in eBay route**
Replace `parseNonNegativeIntegerParam` calls for eBay `minPrice`/`maxPrice` and Kijiji `priceMin`/`priceMax` with `parseDollarPriceParam`. Keep pagination/count params on integer parsing.
Replace `parseNonNegativeIntegerParam` calls for eBay `minPrice`/`maxPrice` and Kijiji
`priceMin`/`priceMax` with `parseDollarPriceParam`. Keep pagination/count params on
integer parsing.
- [ ] **Step 4: Verify API tests**
@@ -43,20 +65,25 @@ Expected: all API route tests pass.
### Task 2: MCP Schema Contract
**Files:**
- Modify: `packages/mcp-server/src/protocol/tools.ts`
- Test: `packages/mcp-server/test/protocol.test.ts`
- [ ] **Step 1: Add MCP schema/forwarding tests**
Add tests that `search_ebay` describes `minPrice` and `maxPrice` as dollar filters and forwards numeric dollar values unchanged in API query params.
Add tests that `search_ebay` describes `minPrice` and `maxPrice` as dollar filters and
forwards numeric dollar values unchanged in API query params.
Run: `bun test packages/mcp-server/test/protocol.test.ts`
Expected: description test fails until schema text changes; forwarding behavior should already pass or reveal mapping gaps.
Expected: description test fails until schema text changes; forwarding behavior should
already pass or reveal mapping gaps.
- [ ] **Step 2: Update tool descriptions**
Change eBay `minPrice` and Kijiji `priceMin` descriptions to `Minimum price in dollars`. Change eBay `maxPrice` and Kijiji `priceMax` descriptions to `Maximum price in dollars`.
Change eBay `minPrice` and Kijiji `priceMin` descriptions to `Minimum price in dollars`.
Change eBay `maxPrice` and Kijiji `priceMax` descriptions to `Maximum price in dollars`.
- [ ] **Step 3: Verify MCP tests**
@@ -67,6 +94,7 @@ Expected: all MCP protocol tests pass.
### Task 3: Cross-Package Verification
**Files:**
- No additional edits expected.
- [ ] **Step 1: Run relevant package tests**

67
docs/superpowers/plans/2026-04-30-live-parser-tests.md
View File

@@ -1,25 +1,37 @@
# Live Parser Tests Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
> **For agentic workers:** REQUIRED SUB-SKILL: Use
> superpowers:subagent-driven-development (recommended) or superpowers:executing-plans
> to implement this plan task-by-task.
> Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add explicit live endpoint test suites for each core marketplace scraper, excluded from default tests and runnable through one script.
**Goal:** Add explicit live endpoint test suites for each core marketplace scraper,
excluded from default tests and runnable through one script.
**Architecture:** Live tests live under `packages/core/test/live/` and import public scraper entry points directly. Normal package tests remain offline because the new files are outside current explicit test commands and run only through `bun run test:live`.
**Architecture:** Live tests live under `packages/core/test/live/` and import public
scraper entry points directly.
Normal package tests remain offline because the new files are outside current explicit
test commands and run only through `bun run test:live`.
**Tech Stack:** Bun `1.3.13`, `bun:test`, TypeScript, existing core scraper APIs.
---
* * *
## File Structure
- Create `packages/core/test/live/ebay.live.test.ts`: live eBay search smoke test against `fetchEbayItems`.
- Create `packages/core/test/live/kijiji.live.test.ts`: live Kijiji search smoke test against `fetchKijijiItems`.
- Create `packages/core/test/live/facebook.live.test.ts`: strict live Facebook search smoke test against `fetchFacebookItems` and `FACEBOOK_COOKIE`.
- Modify `package.json`: add root script `test:live` running all files under `packages/core/test/live`.
- Create `packages/core/test/live/ebay.live.test.ts`: live eBay search smoke test
against `fetchEbayItems`.
- Create `packages/core/test/live/kijiji.live.test.ts`: live Kijiji search smoke test
against `fetchKijijiItems`.
- Create `packages/core/test/live/facebook.live.test.ts`: strict live Facebook search
smoke test against `fetchFacebookItems` and `FACEBOOK_COOKIE`.
- Modify `package.json`: add root script `test:live` running all files under
`packages/core/test/live`.
### Task 1: Add eBay Live Suite
**Files:**
- Create: `packages/core/test/live/ebay.live.test.ts`
- [ ] **Step 1: Write the live test file**
@@ -45,12 +57,13 @@ describe("eBay live parser", () => {
- [ ] **Step 2: Run eBay live test**
Run: `bun test packages/core/test/live/ebay.live.test.ts`
Expected: PASS when eBay returns parseable search results; FAIL on endpoint/rate-limit/parser breakage.
Run: `bun test packages/core/test/live/ebay.live.test.ts` Expected: PASS when eBay
returns parseable search results; FAIL on endpoint/rate-limit/parser breakage.
### Task 2: Add Kijiji Live Suite
**Files:**
- Create: `packages/core/test/live/kijiji.live.test.ts`
- [ ] **Step 1: Write the live test file**
@@ -82,12 +95,13 @@ describe("Kijiji live parser", () => {
- [ ] **Step 2: Run Kijiji live test**
Run: `bun test packages/core/test/live/kijiji.live.test.ts`
Expected: PASS when Kijiji returns parseable search and detail pages; FAIL on endpoint/parser breakage.
Run: `bun test packages/core/test/live/kijiji.live.test.ts` Expected: PASS when Kijiji
returns parseable search and detail pages; FAIL on endpoint/parser breakage.
### Task 3: Add Facebook Live Suite
**Files:**
- Create: `packages/core/test/live/facebook.live.test.ts`
- [ ] **Step 1: Write the live test file**
@@ -117,12 +131,14 @@ describe("Facebook live parser", () => {
- [ ] **Step 2: Run Facebook live test**
Run: `bun test packages/core/test/live/facebook.live.test.ts`
Expected: PASS with valid `FACEBOOK_COOKIE`; FAIL when `FACEBOOK_COOKIE` is missing, expired, or parser output is empty.
Run: `bun test packages/core/test/live/facebook.live.test.ts` Expected: PASS with valid
`FACEBOOK_COOKIE`; FAIL when `FACEBOOK_COOKIE` is missing, expired, or parser output is
empty.
### Task 4: Add Root Live Test Script
**Files:**
- Modify: `package.json`
- [ ] **Step 1: Add script**
@@ -137,30 +153,35 @@ Change root `scripts` to include:
- [ ] **Step 2: Run all live tests through script**
Run: `bun run test:live`
Expected: runs eBay, Kijiji, and Facebook live suites. Facebook fails if `FACEBOOK_COOKIE` is unset.
Run: `bun run test:live` Expected: runs eBay, Kijiji, and Facebook live suites.
Facebook fails if `FACEBOOK_COOKIE` is unset.
### Task 5: Verify Default Suite Exclusion
**Files:**
- No code files modified.
- [ ] **Step 1: Run existing core tests**
Run: `bun test packages/core/test`
Expected: existing mocked tests run. If Bun discovers `packages/core/test/live`, change normal verification command to explicit glob `bun test packages/core/test/*.test.ts` and document that in final notes.
Run: `bun test packages/core/test` Expected: existing mocked tests run.
If Bun discovers `packages/core/test/live`, change normal verification command to
explicit glob `bun test packages/core/test/*.test.ts` and document that in final notes.
- [ ] **Step 2: Run static checks**
Run: `bun run ci`
Expected: typecheck and Biome pass. Fix code issues without changing lint or TypeScript rules.
Run: `bun run ci` Expected: typecheck and Biome pass.
Fix code issues without changing lint or TypeScript rules.
## Commit Note
Do not commit during execution unless user explicitly requests a commit. This repo session policy overrides generic plan commit steps.
Do not commit during execution unless user explicitly requests a commit.
This repo session policy overrides generic plan commit steps.
## Self-Review
- Spec coverage: eBay, Kijiji, Facebook live suites; explicit script; strict Facebook auth; excluded from default flow.
- Spec coverage: eBay, Kijiji, Facebook live suites; explicit script; strict Facebook
auth; excluded from default flow.
- Placeholder scan: no `TBD`, `TODO`, or underspecified implementation steps.
- Type consistency: tests use current exported scraper signatures and shared listing fields from `ListingDetails`.
- Type consistency: tests use current exported scraper signatures and shared listing
fields from `ListingDetails`.