# Unstable Listing Mode Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Add an optional shared mode across Facebook, eBay, and Kijiji that moves listings priced below 80% of the median into `unstableResults`, while preserving current default response shapes.

**Architecture:** Introduce a shared generic classifier in `packages/core` that splits any listing array into `results` and `unstableResults` using the same median-based rule. Then thread one opt-in flag through the scraper entrypoints, API routes, and MCP tool definitions so all surfaces expose the same behavior without changing existing defaults.

**Tech Stack:** Bun, TypeScript, Bun test, workspace packages, JSON-RPC MCP server

---

## File Map

- Create: `packages/core/src/utils/unstable.ts`
  Purpose: shared generic median/cutoff classifier for listing arrays.
- Modify: `packages/core/src/types/common.ts`
  Purpose: add shared mode types used by scrapers and adapters.
- Modify: `packages/core/src/index.ts`
  Purpose: export the new shared classifier/types.
- Modify: `packages/core/src/scrapers/facebook.ts`
  Purpose: add the optional mode flag and return bucketed results when enabled.
- Modify: `packages/core/src/scrapers/ebay.ts`
  Purpose: add the optional mode flag and return bucketed results when enabled.
- Modify: `packages/core/src/scrapers/kijiji.ts`
  Purpose: add the optional mode flag and return bucketed results when enabled.
- Create: `packages/core/test/unstable-listing-mode.test.ts`
  Purpose: lock the shared classifier behavior with direct unit tests.
- Modify: `packages/core/test/facebook-core.test.ts`
  Purpose: prove Facebook preserves default arrays and returns buckets when enabled.
- Modify: `packages/core/test/ebay-core.test.ts`
  Purpose: prove eBay preserves default arrays and returns buckets when enabled.
- Modify: `packages/core/test/kijiji-core.test.ts`
  Purpose: prove Kijiji preserves default arrays and returns buckets when enabled.
- Modify: `packages/api-server/src/routes/facebook.ts`
  Purpose: expose a shared opt-in query parameter and preserve default response shape.
- Modify: `packages/api-server/src/routes/ebay.ts`
  Purpose: expose the same query parameter and preserve default response shape.
- Modify: `packages/api-server/src/routes/kijiji.ts`
  Purpose: expose the same query parameter and preserve default response shape.
- Modify: `packages/api-server/test/routes.test.ts`
  Purpose: verify route forwarding and route response-shape switching.
- Modify: `packages/mcp-server/src/protocol/tools.ts`
  Purpose: document the optional unstable mode in all search tools.
- Modify: `packages/mcp-server/src/protocol/handler.ts`
  Purpose: forward the optional mode to API routes for all search tools.
- Modify: `packages/mcp-server/test/protocol.test.ts`
  Purpose: verify MCP tool metadata and forwarded URLs include the new option.

### Task 1: Add the shared unstable-listing classifier

**Files:**
- Create: `packages/core/src/utils/unstable.ts`
- Modify: `packages/core/src/types/common.ts`
- Modify: `packages/core/src/index.ts`
- Test: `packages/core/test/unstable-listing-mode.test.ts`

- [ ] **Step 1: Write the failing test**

Create `packages/core/test/unstable-listing-mode.test.ts` with focused shared-behavior coverage:

```ts
import { describe, expect, test } from "bun:test";
import {
  classifyUnstableListings,
  type ListingDetails,
} from "../src/index";

function makeListing(title: string, cents?: number): ListingDetails {
  return {
    url: `https://example.com/${title}`,
    title,
    listingPrice: {
      amountFormatted: cents ? `$${(cents / 100).toFixed(2)}` : "$0.00",
      cents: cents ?? 0,
      currency: "CAD",
    },
    listingType: "item",
    listingStatus: "ACTIVE",
  };
}

describe("classifyUnstableListings", () => {
  test("moves listings below 80% of the median into unstableResults", () => {
    const output = classifyUnstableListings([
      makeListing("cheap", 1000),
      makeListing("mid", 2000),
      makeListing("high", 3000),
    ]);

    expect(output.results.map((item) => item.title)).toEqual(["mid", "high"]);
    expect(output.unstableResults.map((item) => item.title)).toEqual(["cheap"]);
  });

  test("uses the midpoint median for even-sized priced inputs", () => {
    const output = classifyUnstableListings([
      makeListing("a", 1000),
      makeListing("b", 2000),
      makeListing("c", 3000),
      makeListing("d", 4000),
    ]);

    expect(output.results.map((item) => item.title)).toEqual(["b", "c", "d"]);
    expect(output.unstableResults.map((item) => item.title)).toEqual(["a"]);
  });

  test("keeps non-positive prices in results while excluding them from median input", () => {
    const output = classifyUnstableListings([
      makeListing("free", 0),
      makeListing("cheap", 1000),
      makeListing("mid", 2000),
      makeListing("high", 3000),
    ]);

    expect(output.results.map((item) => item.title)).toEqual(["free", "mid", "high"]);
    expect(output.unstableResults.map((item) => item.title)).toEqual(["cheap"]);
  });

  test("returns all listings as results when fewer than two valid prices exist", () => {
    const output = classifyUnstableListings([makeListing("only", 2500)]);

    expect(output.results.map((item) => item.title)).toEqual(["only"]);
    expect(output.unstableResults).toEqual([]);
  });
});
```

- [ ] **Step 2: Run test to verify it fails**

Run: `bun test packages/core/test/unstable-listing-mode.test.ts`
Expected: FAIL because `classifyUnstableListings` and the shared mode types do not exist yet.

- [ ] **Step 3: Write minimal implementation**

Add shared types in `packages/core/src/types/common.ts`:

```ts
export interface UnstableListingBuckets<T> {
  results: T[];
  unstableResults: T[];
}

export interface UnstableListingModeOptions {
  hideUnstableResults?: boolean;
}
```

Create `packages/core/src/utils/unstable.ts` with the shared classifier:

```ts
import type { ListingDetails, UnstableListingBuckets } from "../types/common";

function getMedian(values: number[]): number | null {
  if (values.length < 2) return null;

  const sorted = [...values].sort((a, b) => a - b);
  const middle = Math.floor(sorted.length / 2);

  if (sorted.length % 2 === 0) {
    return (sorted[middle - 1] + sorted[middle]) / 2;
  }

  return sorted[middle];
}

export function classifyUnstableListings<T extends ListingDetails>(
  listings: T[],
): UnstableListingBuckets<T> {
  const pricedValues = listings
    .map((listing) => listing.listingPrice?.cents)
    .filter((cents): cents is number => Number.isFinite(cents) && cents > 0);

  const median = getMedian(pricedValues);
  if (median == null) {
    return { results: listings, unstableResults: [] };
  }

  const threshold = median * 0.8;
  const results: T[] = [];
  const unstableResults: T[] = [];

  for (const listing of listings) {
    const cents = listing.listingPrice?.cents;
    if (Number.isFinite(cents) && cents > 0 && cents < threshold) {
      unstableResults.push(listing);
      continue;
    }

    results.push(listing);
  }

  return { results, unstableResults };
}
```

Export the new symbols from `packages/core/src/index.ts`:

```ts
export * from "./types/common";
export { classifyUnstableListings } from "./utils/unstable";
```

- [ ] **Step 4: Run test to verify it passes**

Run: `bun test packages/core/test/unstable-listing-mode.test.ts`
Expected: PASS with 4 passing tests.

- [ ] **Step 5: Commit**

```bash
git add packages/core/src/utils/unstable.ts packages/core/src/types/common.ts packages/core/src/index.ts packages/core/test/unstable-listing-mode.test.ts
git commit -m "feat: add shared unstable listing classifier"
```

### Task 2: Thread the optional mode through all core scrapers

**Files:**
- Modify: `packages/core/src/scrapers/facebook.ts`
- Modify: `packages/core/src/scrapers/ebay.ts`
- Modify: `packages/core/src/scrapers/kijiji.ts`
- Modify: `packages/core/test/facebook-core.test.ts`
- Modify: `packages/core/test/ebay-core.test.ts`
- Modify: `packages/core/test/kijiji-core.test.ts`

- [ ] **Step 1: Write the failing tests**

Add one focused opt-in test per scraper. Use the new shared classifier through the public scraper entrypoints instead of testing internal helpers.

In `packages/core/test/facebook-core.test.ts`, add:

```ts
test("fetchFacebookItems returns stable and unstable buckets when unstable mode is enabled", async () => {
  process.env.FACEBOOK_COOKIE = "c_user=123; xs=abc";

  global.fetch = mock(() =>
    Promise.resolve({
      ok: true,
      text: () => Promise.resolve(facebookSearchHtmlFixture),
      headers: { get: () => null },
    }),
  );

  const result = await fetchFacebookItems("bike", 1, "toronto", 25, {
    hideUnstableResults: true,
  });

  expect(result).toHaveProperty("results");
  expect(result).toHaveProperty("unstableResults");
});
```

In `packages/core/test/ebay-core.test.ts`, add:

```ts
test("fetchEbayItems returns stable and unstable buckets when unstable mode is enabled", async () => {
  const result = await fetchEbayItems("bike", 1, {
    keywords: ["bike"],
    exclusions: [],
    strictMode: false,
    buyItNowOnly: true,
    canadaOnly: true,
  }, {
    hideUnstableResults: true,
  });

  expect(result).toHaveProperty("results");
  expect(result).toHaveProperty("unstableResults");
});
```

In `packages/core/test/kijiji-core.test.ts`, add:

```ts
test("fetchKijijiItems returns stable and unstable buckets when unstable mode is enabled", async () => {
  const result = await fetchKijijiItems(
    "bike",
    1,
    "https://www.kijiji.ca",
    { maxPages: 1 },
    {},
    { hideUnstableResults: true },
  );

  expect(result).toHaveProperty("results");
  expect(result).toHaveProperty("unstableResults");
});
```

Also add one default-mode assertion in one existing scraper test file, for example in `packages/core/test/facebook-core.test.ts`:

```ts
test("fetchFacebookItems keeps returning an array by default", async () => {
  process.env.FACEBOOK_COOKIE = "c_user=123; xs=abc";

  global.fetch = mock(() =>
    Promise.resolve({
      ok: true,
      text: () => Promise.resolve(facebookSearchHtmlFixture),
      headers: { get: () => null },
    }),
  );

  const result = await fetchFacebookItems("bike");
  expect(Array.isArray(result)).toBe(true);
});
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `bun test packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts`
Expected: FAIL because the scraper signatures do not yet accept the new option and still always return arrays.

- [ ] **Step 3: Write minimal implementation**

Add a small shared helper type import to each scraper:

```ts
import {
  classifyUnstableListings,
  type UnstableListingBuckets,
  type UnstableListingModeOptions,
} from "../index";
```

In `packages/core/src/scrapers/facebook.ts`, extend the default export signature and branch at the end:

```ts
export default async function fetchFacebookItems(
  SEARCH_QUERY: string,
  REQUESTS_PER_SECOND = 1,
  LOCATION = "toronto",
  MAX_ITEMS = 25,
  unstableOptions: UnstableListingModeOptions = {},
): Promise<FacebookListingDetails[] | UnstableListingBuckets<FacebookListingDetails>> {
  // existing fetch/parsing logic

  const limitedItems = pricedItems.slice(0, MAX_ITEMS);
  if (!unstableOptions.hideUnstableResults) {
    return limitedItems;
  }

  const classified = classifyUnstableListings(pricedItems);
  return {
    results: classified.results.slice(0, MAX_ITEMS),
    unstableResults: classified.unstableResults,
  };
}
```

In `packages/core/src/scrapers/ebay.ts`, extend the entrypoint the same way:

```ts
export default async function fetchEbayItems(
  SEARCH_QUERY: string,
  REQUESTS_PER_SECOND = 1,
  options: EbaySearchOptions = {},
  unstableOptions: UnstableListingModeOptions = {},
): Promise<EbayListingDetails[] | UnstableListingBuckets<EbayListingDetails>> {
  // existing fetch/parsing logic

  const limitedResults = maxItems ? listings.slice(0, maxItems) : listings;
  if (!unstableOptions.hideUnstableResults) {
    return limitedResults;
  }

  const classified = classifyUnstableListings(listings);
  return {
    results: maxItems ? classified.results.slice(0, maxItems) : classified.results,
    unstableResults: classified.unstableResults,
  };
}
```

In `packages/core/src/scrapers/kijiji.ts`, add the same final argument after `listingOptions`:

```ts
export default async function fetchKijijiItems(
  SEARCH_QUERY: string,
  REQUESTS_PER_SECOND = 1,
  BASE_URL = "https://www.kijiji.ca",
  searchOptions: SearchOptions = {},
  listingOptions: ListingFetchOptions = {},
  unstableOptions: UnstableListingModeOptions = {},
): Promise<DetailedListing[] | UnstableListingBuckets<DetailedListing>> {
  // existing fetch/parsing logic

  if (!unstableOptions.hideUnstableResults) {
    return allListings;
  }

  return classifyUnstableListings(allListings);
}
```

Keep the default branch untouched in all three files so existing callers still receive arrays.

- [ ] **Step 4: Run tests to verify they pass**

Run: `bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts`
Expected: PASS, including the new opt-in bucket assertions and the default-array regression assertion.

- [ ] **Step 5: Commit**

```bash
git add packages/core/src/scrapers/facebook.ts packages/core/src/scrapers/ebay.ts packages/core/src/scrapers/kijiji.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts
git commit -m "feat: add unstable mode to scraper results"
```

### Task 3: Expose unstable mode in API routes

**Files:**
- Modify: `packages/api-server/src/routes/facebook.ts`
- Modify: `packages/api-server/src/routes/ebay.ts`
- Modify: `packages/api-server/src/routes/kijiji.ts`
- Modify: `packages/api-server/test/routes.test.ts`

- [ ] **Step 1: Write the failing tests**

Extend `packages/api-server/test/routes.test.ts` with route-forwarding coverage for the new query parameter:

```ts
test("facebookRoute forwards unstableFilter=true to core", async () => {
  const { facebookRoute } = await import("../src/routes/facebook");

  await facebookRoute(
    new Request(
      "http://localhost/api/facebook?q=laptop&location=toronto&maxItems=3&unstableFilter=true",
    ),
  );

  expect(fetchFacebookItems).toHaveBeenCalledWith(
    "laptop",
    1,
    "toronto",
    3,
    { hideUnstableResults: true },
  );
});

test("ebayRoute forwards unstableFilter=true to core", async () => {
  const { ebayRoute } = await import("../src/routes/ebay");

  await ebayRoute(
    new Request("http://localhost/api/ebay?q=laptop&unstableFilter=true"),
  );

  expect(fetchEbayItems).toHaveBeenCalledWith(
    "laptop",
    1,
    {
      minPrice: undefined,
      maxPrice: undefined,
      strictMode: false,
      exclusions: [],
      keywords: ["laptop"],
      buyItNowOnly: true,
      canadaOnly: true,
    },
    { hideUnstableResults: true },
  );
});

test("kijijiRoute forwards unstableFilter=true to core", async () => {
  const { kijijiRoute } = await import("../src/routes/kijiji");

  await kijijiRoute(
    new Request("http://localhost/api/kijiji?q=laptop&unstableFilter=true"),
  );

  expect(fetchKijijiItems).toHaveBeenCalledWith(
    "laptop",
    4,
    "https://www.kijiji.ca",
    expect.any(Object),
    {},
    { hideUnstableResults: true },
  );
});
```

- [ ] **Step 2: Run tests to verify they fail**

Run: `bun test packages/api-server/test/routes.test.ts`
Expected: FAIL because the routes do not yet parse or forward `unstableFilter`.

- [ ] **Step 3: Write minimal implementation**

In each route, parse the shared boolean once:

```ts
const hideUnstableResults = reqUrl.searchParams.get("unstableFilter") === "true";
```

Update the core calls to forward the shared option.

In `packages/api-server/src/routes/facebook.ts`:

```ts
const items = await fetchFacebookItems(SEARCH_QUERY, 1, LOCATION, maxItems, {
  hideUnstableResults,
});
```

In `packages/api-server/src/routes/ebay.ts`:

```ts
const items = await fetchEbayItems(
  SEARCH_QUERY,
  1,
  {
    minPrice,
    maxPrice,
    strictMode,
    exclusions,
    keywords,
    buyItNowOnly,
    canadaOnly,
  },
  { hideUnstableResults },
);
```

In `packages/api-server/src/routes/kijiji.ts`:

```ts
const items = await fetchKijijiItems(
  SEARCH_QUERY,
  4,
  "https://www.kijiji.ca",
  searchOptions,
  {},
  { hideUnstableResults },
);
```

Do not add any response wrapper logic in the routes; simply return whatever the core scraper returns so the default array path remains unchanged.

- [ ] **Step 4: Run tests to verify they pass**

Run: `bun test packages/api-server/test/routes.test.ts`
Expected: PASS, including existing cookie-parameter regression tests and the new unstable-mode forwarding assertions.

- [ ] **Step 5: Commit**

```bash
git add packages/api-server/src/routes/facebook.ts packages/api-server/src/routes/ebay.ts packages/api-server/src/routes/kijiji.ts packages/api-server/test/routes.test.ts
git commit -m "feat: expose unstable mode in api routes"
```

### Task 4: Document and forward unstable mode in MCP tools

**Files:**
- Modify: `packages/mcp-server/src/protocol/tools.ts`
- Modify: `packages/mcp-server/src/protocol/handler.ts`
- Modify: `packages/mcp-server/test/protocol.test.ts`

- [ ] **Step 1: Write the failing tests**

Extend `packages/mcp-server/test/protocol.test.ts` with metadata and forwarding coverage:

```ts
test("search tools document unstable listing mode", () => {
  for (const toolName of ["search_kijiji", "search_facebook", "search_ebay"]) {
    const tool = tools.find((entry) => entry.name === toolName);

    expect(tool?.inputSchema.properties).toHaveProperty("unstableFilter");
    expect(tool?.inputSchema.properties.unstableFilter.description).toContain(
      "20% below the median",
    );
    expect(tool?.inputSchema.properties.unstableFilter.description).toContain(
      "unstableResults",
    );
  }
});

test("search_facebook forwards unstableFilter to the API", async () => {
  await handleMcpRequest(
    new Request("http://localhost", {
      method: "POST",
      body: JSON.stringify({
        jsonrpc: "2.0",
        id: 1,
        method: "tools/call",
        params: {
          name: "search_facebook",
          arguments: {
            query: "laptop",
            unstableFilter: true,
          },
        },
      }),
    }),
  );

  const calledUrl = (global.fetch as ReturnType<typeof mock>).mock.calls[0]?.[0];
  expect(String(calledUrl)).toContain("unstableFilter=true");
});
```

Mirror the forwarding assertion for `search_kijiji` and `search_ebay` in the same file.

- [ ] **Step 2: Run tests to verify they fail**

Run: `bun test packages/mcp-server/test/protocol.test.ts`
Expected: FAIL because the tools do not yet describe `unstableFilter` and the handler does not append it to API URLs.

- [ ] **Step 3: Write minimal implementation**

In `packages/mcp-server/src/protocol/tools.ts`, add the same optional property to all three tools:

```ts
unstableFilter: {
  type: "boolean",
  description:
    "Optional: move listings priced more than 20% below the median into unstableResults instead of the main results. When enabled, the response shape changes from a plain list to an object with results and unstableResults.",
  default: false,
},
```

In `packages/mcp-server/src/protocol/handler.ts`, append the shared flag in each search branch:

```ts
if (args.unstableFilter !== undefined) {
  params.append("unstableFilter", args.unstableFilter.toString());
}
```

Add that snippet to the `search_kijiji`, `search_facebook`, and `search_ebay` branches.

- [ ] **Step 4: Run tests to verify they pass**

Run: `bun test packages/mcp-server/test/protocol.test.ts`
Expected: PASS, including the new tool-schema assertions and URL-forwarding assertions.

- [ ] **Step 5: Commit**

```bash
git add packages/mcp-server/src/protocol/tools.ts packages/mcp-server/src/protocol/handler.ts packages/mcp-server/test/protocol.test.ts
git commit -m "docs: expose unstable mode in mcp tools"
```

### Task 5: Verify the full cross-package feature end to end

**Files:**
- No code changes expected.

- [ ] **Step 1: Run the focused package tests**

Run: `bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts packages/api-server/test/routes.test.ts packages/mcp-server/test/protocol.test.ts`
Expected: PASS with zero failing tests.

- [ ] **Step 2: Run the broader workspace verification**

Run: `bun run ci`
Expected: PASS with clean workspace validation.

- [ ] **Step 3: Commit verification-only follow-ups if needed**

If verification forced any tiny fixes, commit them immediately after the fix with a focused message, for example:

```bash
git add <exact files changed>
git commit -m "fix: align unstable mode verification"
```

If no files changed during verification, skip this commit step.

## Self-Review

- Spec coverage: shared classifier, all three scrapers, API exposure, MCP documentation, and tests are each mapped to a task.
- Placeholder scan: no `TODO`, `TBD`, or "write tests later" placeholders remain.
- Type consistency: the plan uses one shared flag name, `unstableFilter`, and one shared core option, `hideUnstableResults`, across all tasks.