ca-marketplace-scraper/docs/superpowers/plans/2026-04-22-unstable-listing-mode.md

Unstable Listing Mode Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add an optional shared mode across Facebook, eBay, and Kijiji that moves listings priced below 80% of the median into unstableResults, while preserving current default response shapes.

Architecture: Introduce a shared generic classifier in packages/core that splits any listing array into results and unstableResults using the same median-based rule. Then thread one opt-in flag through the scraper entrypoints, API routes, and MCP tool definitions so all surfaces expose the same behavior without changing existing defaults.

Tech Stack: Bun, TypeScript, Bun test, workspace packages, JSON-RPC MCP server

---

File Map

• Create: packages/core/src/utils/unstable.ts Purpose: shared generic median/cutoff classifier for listing arrays.
• Modify: packages/core/src/types/common.ts Purpose: add shared mode types used by scrapers and adapters.
• Modify: packages/core/src/index.ts Purpose: export the new shared classifier/types.
• Modify: packages/core/src/scrapers/facebook.ts Purpose: add the optional mode flag and return bucketed results when enabled.
• Modify: packages/core/src/scrapers/ebay.ts Purpose: add the optional mode flag and return bucketed results when enabled.
• Modify: packages/core/src/scrapers/kijiji.ts Purpose: add the optional mode flag and return bucketed results when enabled.
• Create: packages/core/test/unstable-listing-mode.test.ts Purpose: lock the shared classifier behavior with direct unit tests.
• Modify: packages/core/test/facebook-core.test.ts Purpose: prove Facebook preserves default arrays and returns buckets when enabled.
• Modify: packages/core/test/ebay-core.test.ts Purpose: prove eBay preserves default arrays and returns buckets when enabled.
• Modify: packages/core/test/kijiji-core.test.ts Purpose: prove Kijiji preserves default arrays and returns buckets when enabled.
• Modify: packages/api-server/src/routes/facebook.ts Purpose: expose a shared opt-in query parameter and preserve default response shape.
• Modify: packages/api-server/src/routes/ebay.ts Purpose: expose the same query parameter and preserve default response shape.
• Modify: packages/api-server/src/routes/kijiji.ts Purpose: expose the same query parameter and preserve default response shape.
• Modify: packages/api-server/test/routes.test.ts Purpose: verify route forwarding and route response-shape switching.
• Modify: packages/mcp-server/src/protocol/tools.ts Purpose: document the optional unstable mode in all search tools.
• Modify: packages/mcp-server/src/protocol/handler.ts Purpose: forward the optional mode to API routes for all search tools.
• Modify: packages/mcp-server/test/protocol.test.ts Purpose: verify MCP tool metadata and forwarded URLs include the new option.

Task 1: Add the shared unstable-listing classifier

Files:

• Create: packages/core/src/utils/unstable.ts

• Modify: packages/core/src/types/common.ts

• Modify: packages/core/src/index.ts

• Test: packages/core/test/unstable-listing-mode.test.ts

• Step 1: Write the failing test

Create packages/core/test/unstable-listing-mode.test.ts with focused shared-behavior coverage:

import { describe, expect, test } from "bun:test";
import {
  classifyUnstableListings,
  type ListingDetails,
} from "../src/index";

function makeListing(title: string, cents?: number): ListingDetails {
  return {
    url: `https://example.com/${title}`,
    title,
    listingPrice: {
      amountFormatted: cents ? `$${(cents / 100).toFixed(2)}` : "$0.00",
      cents: cents ?? 0,
      currency: "CAD",
    },
    listingType: "item",
    listingStatus: "ACTIVE",
  };
}

describe("classifyUnstableListings", () => {
  test("moves listings below 80% of the median into unstableResults", () => {
    const output = classifyUnstableListings([
      makeListing("cheap", 1000),
      makeListing("mid", 2000),
      makeListing("high", 3000),
    ]);

    expect(output.results.map((item) => item.title)).toEqual(["mid", "high"]);
    expect(output.unstableResults.map((item) => item.title)).toEqual(["cheap"]);
  });

  test("uses the midpoint median for even-sized priced inputs", () => {
    const output = classifyUnstableListings([
      makeListing("a", 1000),
      makeListing("b", 2000),
      makeListing("c", 3000),
      makeListing("d", 4000),
    ]);

    expect(output.results.map((item) => item.title)).toEqual(["b", "c", "d"]);
    expect(output.unstableResults.map((item) => item.title)).toEqual(["a"]);
  });

  test("keeps non-positive prices in results while excluding them from median input", () => {
    const output = classifyUnstableListings([
      makeListing("free", 0),
      makeListing("cheap", 1000),
      makeListing("mid", 2000),
      makeListing("high", 3000),
    ]);

    expect(output.results.map((item) => item.title)).toEqual(["free", "mid", "high"]);
    expect(output.unstableResults.map((item) => item.title)).toEqual(["cheap"]);
  });

  test("returns all listings as results when fewer than two valid prices exist", () => {
    const output = classifyUnstableListings([makeListing("only", 2500)]);

    expect(output.results.map((item) => item.title)).toEqual(["only"]);
    expect(output.unstableResults).toEqual([]);
  });
});

• Step 2: Run test to verify it fails

Run: bun test packages/core/test/unstable-listing-mode.test.ts Expected: FAIL because classifyUnstableListings and the shared mode types do not exist yet.

• Step 3: Write minimal implementation

Add shared types in packages/core/src/types/common.ts:

export interface UnstableListingBuckets<T> {
  results: T[];
  unstableResults: T[];
}

export interface UnstableListingModeOptions {
  hideUnstableResults?: boolean;
}

Create packages/core/src/utils/unstable.ts with the shared classifier:

import type { ListingDetails, UnstableListingBuckets } from "../types/common";

function getMedian(values: number[]): number | null {
  if (values.length < 2) return null;

  const sorted = [...values].sort((a, b) => a - b);
  const middle = Math.floor(sorted.length / 2);

  if (sorted.length % 2 === 0) {
    return (sorted[middle - 1] + sorted[middle]) / 2;
  }

  return sorted[middle];
}

export function classifyUnstableListings<T extends ListingDetails>(
  listings: T[],
): UnstableListingBuckets<T> {
  const pricedValues = listings
    .map((listing) => listing.listingPrice?.cents)
    .filter((cents): cents is number => Number.isFinite(cents) && cents > 0);

  const median = getMedian(pricedValues);
  if (median == null) {
    return { results: listings, unstableResults: [] };
  }

  const threshold = median * 0.8;
  const results: T[] = [];
  const unstableResults: T[] = [];

  for (const listing of listings) {
    const cents = listing.listingPrice?.cents;
    if (Number.isFinite(cents) && cents > 0 && cents < threshold) {
      unstableResults.push(listing);
      continue;
    }

    results.push(listing);
  }

  return { results, unstableResults };
}

Export the new symbols from packages/core/src/index.ts:

export * from "./types/common";
export { classifyUnstableListings } from "./utils/unstable";

• Step 4: Run test to verify it passes

Run: bun test packages/core/test/unstable-listing-mode.test.ts Expected: PASS with 4 passing tests.

• Step 5: Commit

git add packages/core/src/utils/unstable.ts packages/core/src/types/common.ts packages/core/src/index.ts packages/core/test/unstable-listing-mode.test.ts
git commit -m "feat: add shared unstable listing classifier"

Task 2: Thread the optional mode through all core scrapers

Files:

• Modify: packages/core/src/scrapers/facebook.ts

• Modify: packages/core/src/scrapers/ebay.ts

• Modify: packages/core/src/scrapers/kijiji.ts

• Modify: packages/core/test/facebook-core.test.ts

• Modify: packages/core/test/ebay-core.test.ts

• Modify: packages/core/test/kijiji-core.test.ts

• Step 1: Write the failing tests

Add one focused opt-in test per scraper. Use the new shared classifier through the public scraper entrypoints instead of testing internal helpers.

In packages/core/test/facebook-core.test.ts, add:

test("fetchFacebookItems returns stable and unstable buckets when unstable mode is enabled", async () => {
  process.env.FACEBOOK_COOKIE = "c_user=123; xs=abc";

  global.fetch = mock(() =>
    Promise.resolve({
      ok: true,
      text: () => Promise.resolve(facebookSearchHtmlFixture),
      headers: { get: () => null },
    }),
  );

  const result = await fetchFacebookItems("bike", 1, "toronto", 25, {
    hideUnstableResults: true,
  });

  expect(result).toHaveProperty("results");
  expect(result).toHaveProperty("unstableResults");
});

In packages/core/test/ebay-core.test.ts, add:

test("fetchEbayItems returns stable and unstable buckets when unstable mode is enabled", async () => {
  const result = await fetchEbayItems("bike", 1, {
    keywords: ["bike"],
    exclusions: [],
    strictMode: false,
    buyItNowOnly: true,
    canadaOnly: true,
  }, {
    hideUnstableResults: true,
  });

  expect(result).toHaveProperty("results");
  expect(result).toHaveProperty("unstableResults");
});

In packages/core/test/kijiji-core.test.ts, add:

test("fetchKijijiItems returns stable and unstable buckets when unstable mode is enabled", async () => {
  const result = await fetchKijijiItems(
    "bike",
    1,
    "https://www.kijiji.ca",
    { maxPages: 1 },
    {},
    { hideUnstableResults: true },
  );

  expect(result).toHaveProperty("results");
  expect(result).toHaveProperty("unstableResults");
});

Also add one default-mode assertion in one existing scraper test file, for example in packages/core/test/facebook-core.test.ts:

test("fetchFacebookItems keeps returning an array by default", async () => {
  process.env.FACEBOOK_COOKIE = "c_user=123; xs=abc";

  global.fetch = mock(() =>
    Promise.resolve({
      ok: true,
      text: () => Promise.resolve(facebookSearchHtmlFixture),
      headers: { get: () => null },
    }),
  );

  const result = await fetchFacebookItems("bike");
  expect(Array.isArray(result)).toBe(true);
});

• Step 2: Run tests to verify they fail

Run: bun test packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts Expected: FAIL because the scraper signatures do not yet accept the new option and still always return arrays.

• Step 3: Write minimal implementation

Add a small shared helper type import to each scraper:

import {
  classifyUnstableListings,
  type UnstableListingBuckets,
  type UnstableListingModeOptions,
} from "../index";

In packages/core/src/scrapers/facebook.ts, extend the default export signature and branch at the end:

export default async function fetchFacebookItems(
  SEARCH_QUERY: string,
  REQUESTS_PER_SECOND = 1,
  LOCATION = "toronto",
  MAX_ITEMS = 25,
  unstableOptions: UnstableListingModeOptions = {},
): Promise<FacebookListingDetails[] | UnstableListingBuckets<FacebookListingDetails>> {
  // existing fetch/parsing logic

  const limitedItems = pricedItems.slice(0, MAX_ITEMS);
  if (!unstableOptions.hideUnstableResults) {
    return limitedItems;
  }

  const classified = classifyUnstableListings(pricedItems);
  return {
    results: classified.results.slice(0, MAX_ITEMS),
    unstableResults: classified.unstableResults,
  };
}

In packages/core/src/scrapers/ebay.ts, extend the entrypoint the same way:

export default async function fetchEbayItems(
  SEARCH_QUERY: string,
  REQUESTS_PER_SECOND = 1,
  options: EbaySearchOptions = {},
  unstableOptions: UnstableListingModeOptions = {},
): Promise<EbayListingDetails[] | UnstableListingBuckets<EbayListingDetails>> {
  // existing fetch/parsing logic

  const limitedResults = maxItems ? listings.slice(0, maxItems) : listings;
  if (!unstableOptions.hideUnstableResults) {
    return limitedResults;
  }

  const classified = classifyUnstableListings(listings);
  return {
    results: maxItems ? classified.results.slice(0, maxItems) : classified.results,
    unstableResults: classified.unstableResults,
  };
}

In packages/core/src/scrapers/kijiji.ts, add the same final argument after listingOptions:

export default async function fetchKijijiItems(
  SEARCH_QUERY: string,
  REQUESTS_PER_SECOND = 1,
  BASE_URL = "https://www.kijiji.ca",
  searchOptions: SearchOptions = {},
  listingOptions: ListingFetchOptions = {},
  unstableOptions: UnstableListingModeOptions = {},
): Promise<DetailedListing[] | UnstableListingBuckets<DetailedListing>> {
  // existing fetch/parsing logic

  if (!unstableOptions.hideUnstableResults) {
    return allListings;
  }

  return classifyUnstableListings(allListings);
}

Keep the default branch untouched in all three files so existing callers still receive arrays.

• Step 4: Run tests to verify they pass

Run: bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts Expected: PASS, including the new opt-in bucket assertions and the default-array regression assertion.

• Step 5: Commit

git add packages/core/src/scrapers/facebook.ts packages/core/src/scrapers/ebay.ts packages/core/src/scrapers/kijiji.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts
git commit -m "feat: add unstable mode to scraper results"

Task 3: Expose unstable mode in API routes

Files:

• Modify: packages/api-server/src/routes/facebook.ts

• Modify: packages/api-server/src/routes/ebay.ts

• Modify: packages/api-server/src/routes/kijiji.ts

• Modify: packages/api-server/test/routes.test.ts

• Step 1: Write the failing tests

Extend packages/api-server/test/routes.test.ts with route-forwarding coverage for the new query parameter:

test("facebookRoute forwards unstableFilter=true to core", async () => {
  const { facebookRoute } = await import("../src/routes/facebook");

  await facebookRoute(
    new Request(
      "http://localhost/api/facebook?q=laptop&location=toronto&maxItems=3&unstableFilter=true",
    ),
  );

  expect(fetchFacebookItems).toHaveBeenCalledWith(
    "laptop",
    1,
    "toronto",
    3,
    { hideUnstableResults: true },
  );
});

test("ebayRoute forwards unstableFilter=true to core", async () => {
  const { ebayRoute } = await import("../src/routes/ebay");

  await ebayRoute(
    new Request("http://localhost/api/ebay?q=laptop&unstableFilter=true"),
  );

  expect(fetchEbayItems).toHaveBeenCalledWith(
    "laptop",
    1,
    {
      minPrice: undefined,
      maxPrice: undefined,
      strictMode: false,
      exclusions: [],
      keywords: ["laptop"],
      buyItNowOnly: true,
      canadaOnly: true,
    },
    { hideUnstableResults: true },
  );
});

test("kijijiRoute forwards unstableFilter=true to core", async () => {
  const { kijijiRoute } = await import("../src/routes/kijiji");

  await kijijiRoute(
    new Request("http://localhost/api/kijiji?q=laptop&unstableFilter=true"),
  );

  expect(fetchKijijiItems).toHaveBeenCalledWith(
    "laptop",
    4,
    "https://www.kijiji.ca",
    expect.any(Object),
    {},
    { hideUnstableResults: true },
  );
});

• Step 2: Run tests to verify they fail

Run: bun test packages/api-server/test/routes.test.ts Expected: FAIL because the routes do not yet parse or forward unstableFilter.

• Step 3: Write minimal implementation

In each route, parse the shared boolean once:

const hideUnstableResults = reqUrl.searchParams.get("unstableFilter") === "true";

Update the core calls to forward the shared option.

In packages/api-server/src/routes/facebook.ts:

const items = await fetchFacebookItems(SEARCH_QUERY, 1, LOCATION, maxItems, {
  hideUnstableResults,
});

In packages/api-server/src/routes/ebay.ts:

const items = await fetchEbayItems(
  SEARCH_QUERY,
  1,
  {
    minPrice,
    maxPrice,
    strictMode,
    exclusions,
    keywords,
    buyItNowOnly,
    canadaOnly,
  },
  { hideUnstableResults },
);

In packages/api-server/src/routes/kijiji.ts:

const items = await fetchKijijiItems(
  SEARCH_QUERY,
  4,
  "https://www.kijiji.ca",
  searchOptions,
  {},
  { hideUnstableResults },
);

Do not add any response wrapper logic in the routes; simply return whatever the core scraper returns so the default array path remains unchanged.

• Step 4: Run tests to verify they pass

Run: bun test packages/api-server/test/routes.test.ts Expected: PASS, including existing cookie-parameter regression tests and the new unstable-mode forwarding assertions.

• Step 5: Commit

git add packages/api-server/src/routes/facebook.ts packages/api-server/src/routes/ebay.ts packages/api-server/src/routes/kijiji.ts packages/api-server/test/routes.test.ts
git commit -m "feat: expose unstable mode in api routes"

Task 4: Document and forward unstable mode in MCP tools

Files:

• Modify: packages/mcp-server/src/protocol/tools.ts

• Modify: packages/mcp-server/src/protocol/handler.ts

• Modify: packages/mcp-server/test/protocol.test.ts

• Step 1: Write the failing tests

Extend packages/mcp-server/test/protocol.test.ts with metadata and forwarding coverage:

test("search tools document unstable listing mode", () => {
  for (const toolName of ["search_kijiji", "search_facebook", "search_ebay"]) {
    const tool = tools.find((entry) => entry.name === toolName);

    expect(tool?.inputSchema.properties).toHaveProperty("unstableFilter");
    expect(tool?.inputSchema.properties.unstableFilter.description).toContain(
      "20% below the median",
    );
    expect(tool?.inputSchema.properties.unstableFilter.description).toContain(
      "unstableResults",
    );
  }
});

test("search_facebook forwards unstableFilter to the API", async () => {
  await handleMcpRequest(
    new Request("http://localhost", {
      method: "POST",
      body: JSON.stringify({
        jsonrpc: "2.0",
        id: 1,
        method: "tools/call",
        params: {
          name: "search_facebook",
          arguments: {
            query: "laptop",
            unstableFilter: true,
          },
        },
      }),
    }),
  );

  const calledUrl = (global.fetch as ReturnType<typeof mock>).mock.calls[0]?.[0];
  expect(String(calledUrl)).toContain("unstableFilter=true");
});

Mirror the forwarding assertion for search_kijiji and search_ebay in the same file.

• Step 2: Run tests to verify they fail

Run: bun test packages/mcp-server/test/protocol.test.ts Expected: FAIL because the tools do not yet describe unstableFilter and the handler does not append it to API URLs.

• Step 3: Write minimal implementation

In packages/mcp-server/src/protocol/tools.ts, add the same optional property to all three tools:

unstableFilter: {
  type: "boolean",
  description:
    "Optional: move listings priced more than 20% below the median into unstableResults instead of the main results. When enabled, the response shape changes from a plain list to an object with results and unstableResults.",
  default: false,
},

In packages/mcp-server/src/protocol/handler.ts, append the shared flag in each search branch:

if (args.unstableFilter !== undefined) {
  params.append("unstableFilter", args.unstableFilter.toString());
}

Add that snippet to the search_kijiji, search_facebook, and search_ebay branches.

• Step 4: Run tests to verify they pass

Run: bun test packages/mcp-server/test/protocol.test.ts Expected: PASS, including the new tool-schema assertions and URL-forwarding assertions.

• Step 5: Commit

git add packages/mcp-server/src/protocol/tools.ts packages/mcp-server/src/protocol/handler.ts packages/mcp-server/test/protocol.test.ts
git commit -m "docs: expose unstable mode in mcp tools"

Task 5: Verify the full cross-package feature end to end

Files:

• No code changes expected.

• Step 1: Run the focused package tests

Run: bun test packages/core/test/unstable-listing-mode.test.ts packages/core/test/facebook-core.test.ts packages/core/test/ebay-core.test.ts packages/core/test/kijiji-core.test.ts packages/api-server/test/routes.test.ts packages/mcp-server/test/protocol.test.ts Expected: PASS with zero failing tests.

• Step 2: Run the broader workspace verification

Run: bun run ci Expected: PASS with clean workspace validation.

• Step 3: Commit verification-only follow-ups if needed

If verification forced any tiny fixes, commit them immediately after the fix with a focused message, for example:

git add <exact files changed>
git commit -m "fix: align unstable mode verification"

If no files changed during verification, skip this commit step.

Self-Review

• Spec coverage: shared classifier, all three scrapers, API exposure, MCP documentation, and tests are each mapped to a task.
• Placeholder scan: no TODO, TBD, or “write tests later” placeholders remain.
• Type consistency: the plan uses one shared flag name, unstableFilter, and one shared core option, hideUnstableResults, across all tasks.