docs: add unstable listing mode design
This commit is contained in:
@@ -0,0 +1,148 @@
|
||||
# Unstable Listing Mode Design
|
||||
|
||||
## Summary
|
||||
|
||||
Add an optional shared result mode across Facebook, eBay, and Kijiji that moves suspiciously cheap listings out of the main results into a separate `unstableResults` bucket.
|
||||
Listings are considered unstable when their price is more than 20% below the median price of the scraper's priced search results.
|
||||
|
||||
## Goals
|
||||
|
||||
- Support the same optional unstable-listing mode across all scrapers.
|
||||
- Keep current default scraper and route behavior unchanged unless the mode is enabled.
|
||||
- Hide unstable listings from the main results while still returning them separately.
|
||||
- Implement the rule once in shared core code instead of duplicating marketplace-specific logic.
|
||||
- Document the option in MCP tool descriptions so callers can discover it.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Adding marketplace-specific thresholds or heuristics.
|
||||
- Re-ranking results beyond splitting stable and unstable buckets.
|
||||
- Classifying free, missing-price, or invalid-price listings as unstable.
|
||||
- Changing unrelated scraper parsing behavior.
|
||||
|
||||
## Current State
|
||||
|
||||
`packages/core` currently returns plain arrays from scraper search functions.
|
||||
`packages/api-server` forwards those scraper results directly from marketplace routes.
|
||||
`packages/mcp-server` documents search tools per marketplace, but does not expose or describe any result-stability mode.
|
||||
|
||||
There is no shared result-classification utility today.
|
||||
Price filtering exists in some scrapers, but not a cross-marketplace median-based split.
|
||||
|
||||
## Chosen Approach
|
||||
|
||||
Use a shared core utility plus per-route and per-tool opt-in.
|
||||
|
||||
The shared utility will accept parsed listings, compute the median from valid positive prices, and split the data into `results` and `unstableResults`.
|
||||
Each scraper will opt into that utility when the caller enables unstable-listing mode.
|
||||
API routes and MCP tools will expose the same optional mode so the feature is consistently available everywhere scraper search is surfaced.
|
||||
|
||||
This keeps the heuristic centralized, minimizes duplicated logic, and preserves existing consumers by leaving the default path unchanged.
|
||||
|
||||
## Design
|
||||
|
||||
### Shared Core Classification
|
||||
|
||||
Add a shared utility in `packages/core` for listing stability classification.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- accept parsed listing arrays with `listingPrice.cents`
|
||||
- ignore listings whose price is missing, non-numeric, or non-positive when computing the median
|
||||
- compute the median price from valid priced listings
|
||||
- classify listings as unstable when `listingPrice.cents < median * 0.8`
|
||||
- return an object with:
|
||||
- `results`: listings that remain in the main bucket
|
||||
- `unstableResults`: listings moved out of the main bucket
|
||||
|
||||
Listings excluded from median computation because their price is missing or non-positive remain in `results` unchanged.
|
||||
|
||||
### Scraper Integration
|
||||
|
||||
Facebook, eBay, and Kijiji search entrypoints will gain the same optional mode flag.
|
||||
|
||||
Default behavior:
|
||||
|
||||
- return the current plain array result shape
|
||||
|
||||
Opt-in behavior:
|
||||
|
||||
- run the shared classification utility after parsing search results
|
||||
- classify before final result limiting so unstable items do not consume main-result slots
|
||||
- return an object shaped like:
|
||||
|
||||
```ts
|
||||
{
|
||||
results: ListingDetails[];
|
||||
unstableResults: ListingDetails[];
|
||||
}
|
||||
```
|
||||
|
||||
Each scraper will use its existing concrete listing subtype for these arrays.
|
||||
|
||||
### API Surface
|
||||
|
||||
Marketplace API routes will expose an optional query parameter for unstable-listing mode.
|
||||
|
||||
Requirements:
|
||||
|
||||
- keep existing route responses unchanged when the parameter is absent or false
|
||||
- when enabled, return the object payload with `results` and `unstableResults`
|
||||
- use the same semantics across Facebook, eBay, and Kijiji routes
|
||||
|
||||
The exact parameter name should be consistent across routes and intentionally describe the behavior, for example `unstableFilter=true`.
|
||||
|
||||
### MCP Surface
|
||||
|
||||
Marketplace MCP tools will expose the same optional mode as an input field.
|
||||
|
||||
Tool descriptions should explicitly document:
|
||||
|
||||
- that the option is optional
|
||||
- that it moves listings priced more than 20% below the median into `unstableResults`
|
||||
- that enabling it changes the response shape from a plain list to an object with `results` and `unstableResults`
|
||||
- that the behavior is available for Facebook, eBay, and Kijiji search tools
|
||||
|
||||
The wording should be aligned across all three tools so the feature reads as one shared capability.
|
||||
|
||||
### Error Handling
|
||||
|
||||
The unstable-listing mode should be best-effort and non-failing.
|
||||
|
||||
- If there are no valid positive prices, return all listings in `results` and an empty `unstableResults` array.
|
||||
- If there is only one valid priced listing, do not classify it as unstable.
|
||||
- Parsing failures remain governed by existing scraper behavior; the classification layer should not introduce new scraper-specific errors.
|
||||
|
||||
### Testing Strategy
|
||||
|
||||
Follow TDD.
|
||||
Start with shared utility tests, then wire the option through scraper and route tests.
|
||||
|
||||
Coverage targets:
|
||||
|
||||
1. Median calculation for odd-sized valid price sets.
|
||||
2. Median calculation for even-sized valid price sets.
|
||||
3. Strict cutoff behavior where only listings with `price < median * 0.8` move to `unstableResults`.
|
||||
4. Missing, invalid, zero, or negative prices are excluded from median computation and remain in `results`.
|
||||
5. Default scraper behavior still returns plain arrays when the option is disabled.
|
||||
6. Enabled scraper behavior returns `{ results, unstableResults }` for Facebook, eBay, and Kijiji.
|
||||
7. API routes preserve existing response shapes by default and switch to the object payload only when enabled.
|
||||
8. MCP tool metadata documents the new optional mode for all three marketplace search tools.
|
||||
|
||||
Verification target after implementation:
|
||||
|
||||
- `bun test packages/core/test`
|
||||
- `bun test packages/api-server/test`
|
||||
- `bun test packages/mcp-server/test` if MCP metadata tests exist or are added
|
||||
- `bun run ci`
|
||||
|
||||
## Risks
|
||||
|
||||
- The optional mode introduces a union return shape for scraper callers, which can ripple into downstream TypeScript signatures.
|
||||
- Applying classification before final limiting changes which items appear in the main bucket compared with a naive post-limit split.
|
||||
- Kijiji and eBay may have different mixes of priced and unpriced results, so excluding non-positive prices from the median must remain explicit and tested.
|
||||
|
||||
## Rollout Notes
|
||||
|
||||
Land the shared classifier, scraper wiring, route wiring, tests, and MCP description updates together.
|
||||
That avoids a partial rollout where the feature exists in one surface but is undocumented or inconsistent elsewhere.
|
||||
Reference in New Issue
Block a user