old4ever/ca-marketplace-scraper
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
95 Commits 1 Branch 0 Tags
1ee41fb34615e08c61723ccfb44972d6b775828f
Commit Graph
95 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
old4ever 1ee41fb346 feat: add unstable mode to scraper results 2026-04-22 23:23:31 -04:00
old4ever 8141de5b4b feat: add shared unstable listing classifier 2026-04-22 17:56:26 -04:00
old4ever f8975fa91d docs: add unstable listing mode plan 2026-04-22 17:53:45 -04:00
old4ever cb5e1e62d2 docs: add unstable listing mode design 2026-04-22 17:51:07 -04:00
old4ever 9070f76412 refactor: handle facebook route-aware failure states 2026-04-22 11:48:47 -04:00
old4ever 7ddc96dfdf refactor: add facebook html fallbacks 2026-04-22 11:36:47 -04:00
old4ever 63ca006696 refactor: rewrite facebook item parser for comet bootstrap 2026-04-22 02:44:17 -04:00
old4ever c90ee54cc1 refactor: rewrite facebook search parser for comet bootstrap 2026-04-22 02:32:55 -04:00
old4ever cfd7619737 refactor: add facebook bootstrap candidate extraction 2026-04-21 23:46:00 -04:00
old4ever b072599bc6 refactor: add facebook response classification 2026-04-21 23:31:45 -04:00
old4ever 2617afc62f docs: add facebook comet rewrite plan 2026-04-21 23:06:03 -04:00
old4ever ba889a1f9d docs: add facebook comet rewrite design 2026-04-21 23:02:47 -04:00
old4ever 45cff20377 docs: update cookies agents.md 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-04-21 22:05:49 -04:00
old4ever b6e9501448 docs: align cookie setup with env-only auth 2026-04-21 21:53:42 -04:00
old4ever d65d81dbd1 refactor: remove mcp cookie parameters 2026-04-21 21:48:34 -04:00
old4ever 1a2c0cf6b8 refactor: remove api cookie query overrides 2026-04-21 21:47:37 -04:00
old4ever 918ee92441 refactor: make ebay auth env-only 2026-04-21 21:46:40 -04:00
old4ever a7a5eca7ad refactor: remove facebook cookie overrides 2026-04-21 21:45:42 -04:00
old4ever 847ce28590 refactor: make cookie loading env-only 2026-04-21 21:44:12 -04:00
old4ever 7b4b656868 chore: import order 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-04-21 21:07:07 -04:00
old4ever e144dcabeb fix: accept nullable marketplace prices in formatter 2026-04-21 21:01:53 -04:00
old4ever 651d54b837 fix: respect custom facebook cookie path 2026-04-21 21:01:53 -04:00
old4ever 2231603692 chore: fix lint config 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-04-21 20:51:28 -04:00
old4ever 86191e7a45 chore: deep-init 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-04-21 20:47:00 -04:00
old4ever c58d614948 chore: devenv lock update 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-04-21 20:37:55 -04:00
old4ever 7cf21546e2 chore: ai agent config 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-04-21 20:37:55 -04:00
old4ever ffc4a2c5c5 chore: useragent 2026-04-21 19:45:29 -04:00
old4everandClaude Opus 4.5 e4ab145d70 feat: add cookie support to kijiji scraper 
Add optional cookie parameter to bypass bot detection (403 errors).
Cookies can be provided via parameter, KIJIJI_COOKIE env var, or
cookies/kijiji.json file. Supports both JSON array and string formats.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-23 19:29:13 -05:00
old4everandClaude Opus 4.5 1dce0392e3 refactor: use shared cookie utility in ebay scraper 
Replace inline cookie loading with shared utility functions.
Now supports both JSON array and cookie string formats.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-23 19:29:02 -05:00
old4everandClaude Opus 4.5 251fcbb7d9 refactor: use shared cookie utility in facebook scraper 
Replace inline cookie parsing with shared utility functions.
Maintains backward compatibility with existing exports.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-23 19:28:52 -05:00
old4everandClaude Opus 4.5 9bc57d6b54 refactor: add shared cookie utility to core package 
Move cookie parsing logic to a dedicated utility module that can be
shared across all scrapers. Supports both JSON array and cookie string
formats for all input sources (parameter, env var, file).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-23 19:28:44 -05:00
old4ever 4a467c9f02 fix: support both json and string cookies for facebook 2026-01-23 19:00:51 -05:00
old4ever f944d319c2 chore: update dockerignore 2026-01-23 15:43:13 -05:00
old4ever cf9784a565 feat: implement cookie priority hierarchy (URL param > env var > file) for Facebook and eBay scrapers 2026-01-23 15:32:17 -05:00
old4ever df0c528535 fix: correct formatCentsToCurrency usage in facebook scraper 2026-01-23 14:50:41 -05:00
old4ever 2f97d3eafd fix: correct formatCentsToCurrency usage in kijiji scraper 2026-01-23 14:50:41 -05:00
old4ever 65eb8d1724 refactor: increase kijiji scraping request rate to 4 rps 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-01-23 14:50:37 -05:00
old4ever f3839aba54 fix: increase kijiji rate limit to 4 rps 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-01-23 13:59:47 -05:00
old4ever 90b98bfb09 chore: testing mcp server 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-01-23 13:59:28 -05:00
old4ever eb6705df0f feat: add 60-second timeouts to MCP request handlers for reliability 2026-01-23 13:59:28 -05:00
old4ever 72525609ed fix: set idle timeout to 255 seconds in MCP server to prevent premature shutdown 2026-01-23 13:59:28 -05:00
old4ever 8b0a65860c chore: add imports for linkedom and delay utils in ebay scraper 2026-01-23 13:10:44 -05:00
old4ever f9b1c7e096 fix: remove eslint-disable directives 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-01-23 13:08:38 -05:00
old4ever 9edc74cbeb chore: local dev scripts 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-01-23 13:07:01 -05:00
old4ever ee0fca826d style: fix formatting in MCP server 2026-01-23 11:56:54 -05:00
old4ever f7372612fb test: fix formatting in test setup 2026-01-23 11:56:51 -05:00
old4ever bce126664e test: remove unused imports in Kijiji utils tests 2026-01-23 11:56:47 -05:00
old4ever 8cbf11538e test: fix formatting and remove unused HttpError import in Kijiji tests 2026-01-23 11:56:44 -05:00
old4ever 79f47fdaef test: remove unused import in Facebook integration tests 2026-01-23 11:56:41 -05:00
old4ever de5069bf2b test: fix unused variable in Facebook core tests 2026-01-23 11:56:38 -05:00
old4ever 637f1a4e75 fix: resolve biome lint errors and warnings 2026-01-23 10:33:15 -05:00
old4ever 441ff436c4 feat(mcp): extend Kijiji tool with filtering parameters 2026-01-23 09:55:37 -05:00
old4ever 1f53ec912a feat(mcp): add search options to Kijiji and eBay tools 2026-01-23 09:55:21 -05:00
old4ever 053efd815b feat(api/kijiji): add filtering and pagination parameters 2026-01-23 09:54:30 -05:00
old4ever d619fa5d77 feat(api/facebook): add maxItems parameter support 2026-01-23 09:53:51 -05:00
old4ever 050fd0adba feat(api/ebay): add maxItems parameter and error handling 2026-01-23 09:53:00 -05:00
old4ever 7b106c91ce style: format ebay scraper with consistent indentation 2026-01-23 09:52:25 -05:00
old4ever 6e0487f8f3 style: format api-server index with consistent indentation 2026-01-23 09:52:22 -05:00
old4ever da23ca1c3f chore: agents update 2026-01-23 00:52:35 -05:00
old4ever c35aae4c95 chore: biome auto-fixes 2026-01-23 00:52:26 -05:00
old4ever 02162c02f5 chore: biome init 2026-01-23 00:52:10 -05:00
old4everandClaude Opus 4.5 50d56201af feat: port upstream scraper improvements to monorepo 
Kijiji improvements:
- Add error classes: NetworkError, ParseError, RateLimitError, ValidationError
- Add exponential backoff with jitter for retries
- Add request timeout (30s abort)
- Add pagination support (SearchOptions.maxPages)
- Add location/category mappings and resolution functions
- Add enhanced DetailedListing interface with images, seller info, attributes
- Add GraphQL client for seller details

Facebook improvements:
- Add parseFacebookCookieString() for parsing cookie strings
- Add ensureFacebookCookies() with env var fallback
- Add extractFacebookItemData() with multiple extraction paths
- Add fetchFacebookItem() for individual item fetching
- Add extraction metrics and API stability monitoring
- Add vehicle-specific field extraction
- Improve error handling with specific guidance for auth errors

Shared utilities:
- Update http.ts with new error classes and improved fetchHtml

Documentation:
- Port KIJIJI.md, FMARKETPLACE.md, AGENTS.md from upstream

Tests:
- Port kijiji-core, kijiji-integration, kijiji-utils tests
- Port facebook-core, facebook-integration tests
- Add test setup file

Scripts:
- Port parse-facebook-cookies.ts script

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-23 00:34:50 -05:00
old4ever 497c7995a2 feat: ebay 'buy it now' and 'canada only' filters support 2025-12-17 14:38:52 -05:00
old4ever 083b862552 fix healthcheck 2025-12-17 13:58:18 -05:00
old4ever 0a32094e93 feat: adapt Dockerfile for monorepo structure 2025-12-13 20:54:32 -05:00
old4ever a66b5b2362 migrate to monorepo? 2025-12-13 20:31:10 -05:00
old4ever 7da6408d7a fix: add target to a bun build command 2025-10-03 09:06:02 -04:00
old4ever 3863d3139e chore: update docker compose 2025-10-03 09:06:02 -04:00
old4ever 11dbf29f93 chore: update README.md 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-03 08:59:13 -04:00
old4ever 2e2c81b7e5 chore: update package.json 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-03 08:56:03 -04:00
old4ever ea0a69ccd6 feat(docker): enable containerization of marketplace scraper 
Introduce Dockerfile with production build using Bun, .dockerignore for efficient builds, and docker-compose.yml for easy local and prod deployment on port 3000.
 2025-10-03 08:44:51 -04:00
old4ever fa7ac59c45 feat: ebay parser 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-02 13:52:29 -04:00
old4ever 8c52efe5e7 feat(facebook): parse additional listing details like status, images, and seller info 
Enhance Facebook scraping to extract listing status (ACTIVE/SOLD/PENDING/HIDDEN), primary image/video URLs, seller name/ID, category ID, and delivery options, improving response completeness.
 2025-10-02 12:03:59 -04:00
old4ever dcd0da29a2 feat: facebook scraping 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-02 11:50:35 -04:00
old4ever ee09162faa chore: add claude code dev dependency 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-02 10:39:49 -04:00
old4ever ddf7639854 fix: add newlines to debug logging due to overlap with the progress bar 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:12:40 -04:00
old4ever d42c74331e feat: add requests progress bar 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:12:10 -04:00
old4ever 49a32bf6b4 fix: bun's --hot option doesn't like modules 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:11:31 -04:00
old4ever 3166b5a95f chore: remove redundant logging 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:11:09 -04:00
old4ever cb1fb2bae6 feat: add unidecode dependency and slugify function for improved Kijiji search 
This commit introduces the `unidecode` library to handle non-ASCII characters in search queries. A `slugify` function is implemented to improve Kijiji search URL generation by creating more user-friendly and reliable URLs. It converts the user query to a URL friendly string.

It also addresses issues in Kijiji scraping: improves listing filtering logic to ensure only valid listings with prices are added and enhances error handling to provide more informative messages and prevent process termination due to HTTP errors during listing detail retrieval.

Additionally, refactors the Kijiji search query URL, leveraging the new `slugify` function for enhanced URL generation using the search query. Added debugging logs for better traceability.
 2025-09-18 11:35:10 -04:00
old4ever add93dc6bd feat: increase requests per second for Kijiji search 2025-09-18 11:33:47 -04:00
old4ever 99848d1852 build: configure absolute imports using @ alias 
Configures TypeScript path aliases for cleaner and more maintainable imports.
 2025-09-18 11:33:06 -04:00
old4ever d35e37f4c2 feat: implement Kijiji scraping API endpoint 
This commit introduces a new API endpoint that allows users to search for items on Kijiji.
The endpoint accepts a search query as a header or query parameter and returns a JSON response containing the search results. It also handles cases where no query is provided or no results are found, returning appropriate error responses.
 2025-09-17 22:58:11 -04:00
old4ever 2e9547ebaa fix: search GTA only 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-17 22:57:42 -04:00
old4ever 46a8ac92cf feat: extract Kijiji scraping logic into reusable function 
This commit extracts the Kijiji scraping functionality into a reusable function `fetchKijijiItems`. This allows for easier integration into other parts of the application and improves code modularity. The function accepts search query, requests per second, and base URL as parameters, enabling customizable scraping.
 2025-09-17 22:03:24 -04:00
old4ever b7a61423c3 refactor: improve Kijiji scraping architecture 
Introduces types, utilities, and a dedicated HTTP client for improved structure and maintainability.  Includes rate limit handling, retry logic, and error handling.
 2025-09-17 20:45:57 -04:00
old4ever 23dc9cae4c feat: scrape listing details from Kijiji ads 2025-09-17 20:43:07 -04:00
old4ever 257edf394c style: specify source of apollo state interface 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-17 19:00:13 -04:00
old4ever 63683593e4 feat: add kijiji search parser 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-17 18:58:57 -04:00
old4ever c09650b9f4 build: configure build and development scripts 2025-09-17 18:54:19 -04:00
old4ever 030e3c302d feat: integrate linkedom for DOM manipulation 
Adds `linkedom` as a dependency for parsing HTML and manipulating the DOM. This enables scraping capabilities.
 2025-09-17 18:48:23 -04:00
old4ever 3fccbb7a46 minor tweaks 2025-08-14 14:49:41 -04:00
old4ever e0cf995769 init bun 2025-07-29 22:31:17 -04:00
old4ever e013ce5a46 install bun 2025-07-29 22:30:40 -04:00
old4ever cc0bb000f2 init devenv 2025-07-29 22:22:17 -04:00