167 lines
5.9 KiB
Markdown
167 lines
5.9 KiB
Markdown
# Live Parser Tests Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Add explicit live endpoint test suites for each core marketplace scraper, excluded from default tests and runnable through one script.
|
|
|
|
**Architecture:** Live tests live under `packages/core/test/live/` and import public scraper entry points directly. Normal package tests remain offline because the new files are outside current explicit test commands and run only through `bun run test:live`.
|
|
|
|
**Tech Stack:** Bun `1.3.13`, `bun:test`, TypeScript, existing core scraper APIs.
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
- Create `packages/core/test/live/ebay.live.test.ts`: live eBay search smoke test against `fetchEbayItems`.
|
|
- Create `packages/core/test/live/kijiji.live.test.ts`: live Kijiji search smoke test against `fetchKijijiItems`.
|
|
- Create `packages/core/test/live/facebook.live.test.ts`: strict live Facebook search smoke test against `fetchFacebookItems` and `FACEBOOK_COOKIE`.
|
|
- Modify `package.json`: add root script `test:live` running all files under `packages/core/test/live`.
|
|
|
|
### Task 1: Add eBay Live Suite
|
|
|
|
**Files:**
|
|
- Create: `packages/core/test/live/ebay.live.test.ts`
|
|
|
|
- [ ] **Step 1: Write the live test file**
|
|
|
|
```ts
|
|
import { describe, expect, test } from "bun:test";
|
|
import fetchEbayItems from "../../src/scrapers/ebay";
|
|
|
|
describe("eBay live parser", () => {
|
|
test("scrapes live search results into listing details", async () => {
|
|
const results = await fetchEbayItems("iphone", 1, { maxItems: 3 });
|
|
|
|
expect(results.length).toBeGreaterThan(0);
|
|
for (const listing of results) {
|
|
expect(listing.url).toStartWith("https://");
|
|
expect(listing.title.length).toBeGreaterThan(0);
|
|
expect(listing.listingPrice.cents).toBeGreaterThanOrEqual(0);
|
|
expect(listing.listingPrice.currency.length).toBeGreaterThan(0);
|
|
}
|
|
});
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run eBay live test**
|
|
|
|
Run: `bun test packages/core/test/live/ebay.live.test.ts`
|
|
Expected: PASS when eBay returns parseable search results; FAIL on endpoint/rate-limit/parser breakage.
|
|
|
|
### Task 2: Add Kijiji Live Suite
|
|
|
|
**Files:**
|
|
- Create: `packages/core/test/live/kijiji.live.test.ts`
|
|
|
|
- [ ] **Step 1: Write the live test file**
|
|
|
|
```ts
|
|
import { describe, expect, test } from "bun:test";
|
|
import fetchKijijiItems from "../../src/scrapers/kijiji";
|
|
|
|
describe("Kijiji live parser", () => {
|
|
test("scrapes live search results into detailed listings", async () => {
|
|
const results = await fetchKijijiItems(
|
|
"iphone",
|
|
1,
|
|
"https://www.kijiji.ca",
|
|
{ maxPages: 1 },
|
|
{ includeImages: false, sellerDataDepth: "basic" },
|
|
);
|
|
|
|
expect(results.length).toBeGreaterThan(0);
|
|
for (const listing of results) {
|
|
expect(listing.url).toStartWith("https://www.kijiji.ca/");
|
|
expect(listing.title.length).toBeGreaterThan(0);
|
|
expect(listing.listingPrice.cents).toBeGreaterThanOrEqual(0);
|
|
expect(listing.listingPrice.currency.length).toBeGreaterThan(0);
|
|
}
|
|
});
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run Kijiji live test**
|
|
|
|
Run: `bun test packages/core/test/live/kijiji.live.test.ts`
|
|
Expected: PASS when Kijiji returns parseable search and detail pages; FAIL on endpoint/parser breakage.
|
|
|
|
### Task 3: Add Facebook Live Suite
|
|
|
|
**Files:**
|
|
- Create: `packages/core/test/live/facebook.live.test.ts`
|
|
|
|
- [ ] **Step 1: Write the live test file**
|
|
|
|
```ts
|
|
import { describe, expect, test } from "bun:test";
|
|
import fetchFacebookItems from "../../src/scrapers/facebook";
|
|
|
|
describe("Facebook live parser", () => {
|
|
test("requires FACEBOOK_COOKIE for strict live testing", () => {
|
|
expect(process.env.FACEBOOK_COOKIE?.trim().length ?? 0).toBeGreaterThan(0);
|
|
});
|
|
|
|
test("scrapes live marketplace search results into listing details", async () => {
|
|
const results = await fetchFacebookItems("iphone", 1, "toronto", 3);
|
|
|
|
expect(results.length).toBeGreaterThan(0);
|
|
for (const listing of results) {
|
|
expect(listing.url).toStartWith("https://www.facebook.com/marketplace/item/");
|
|
expect(listing.title.length).toBeGreaterThan(0);
|
|
expect(listing.listingPrice.cents).toBeGreaterThanOrEqual(0);
|
|
expect(listing.listingPrice.currency.length).toBeGreaterThan(0);
|
|
}
|
|
});
|
|
});
|
|
```
|
|
|
|
- [ ] **Step 2: Run Facebook live test**
|
|
|
|
Run: `bun test packages/core/test/live/facebook.live.test.ts`
|
|
Expected: PASS with valid `FACEBOOK_COOKIE`; FAIL when `FACEBOOK_COOKIE` is missing, expired, or parser output is empty.
|
|
|
|
### Task 4: Add Root Live Test Script
|
|
|
|
**Files:**
|
|
- Modify: `package.json`
|
|
|
|
- [ ] **Step 1: Add script**
|
|
|
|
Change root `scripts` to include:
|
|
|
|
```json
|
|
{
|
|
"test:live": "bun test packages/core/test/live"
|
|
}
|
|
```
|
|
|
|
- [ ] **Step 2: Run all live tests through script**
|
|
|
|
Run: `bun run test:live`
|
|
Expected: runs eBay, Kijiji, and Facebook live suites. Facebook fails if `FACEBOOK_COOKIE` is unset.
|
|
|
|
### Task 5: Verify Default Suite Exclusion
|
|
|
|
**Files:**
|
|
- No code files modified.
|
|
|
|
- [ ] **Step 1: Run existing core tests**
|
|
|
|
Run: `bun test packages/core/test`
|
|
Expected: existing mocked tests run. If Bun discovers `packages/core/test/live`, change normal verification command to explicit glob `bun test packages/core/test/*.test.ts` and document that in final notes.
|
|
|
|
- [ ] **Step 2: Run static checks**
|
|
|
|
Run: `bun run ci`
|
|
Expected: typecheck and Biome pass. Fix code issues without changing lint or TypeScript rules.
|
|
|
|
## Commit Note
|
|
|
|
Do not commit during execution unless user explicitly requests a commit. This repo session policy overrides generic plan commit steps.
|
|
|
|
## Self-Review
|
|
|
|
- Spec coverage: eBay, Kijiji, Facebook live suites; explicit script; strict Facebook auth; excluded from default flow.
|
|
- Placeholder scan: no `TBD`, `TODO`, or underspecified implementation steps.
|
|
- Type consistency: tests use current exported scraper signatures and shared listing fields from `ListingDetails`.
|