old4ever/ca-marketplace-scraper
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
102 Commits 1 Branch 0 Tags
0a0723a5600d43362e117c4cfca7306830cccc63
Commit Graph
102 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
old4ever 9edc74cbeb chore: local dev scripts 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2026-01-23 13:07:01 -05:00
old4ever ee0fca826d style: fix formatting in MCP server 2026-01-23 11:56:54 -05:00
old4ever f7372612fb test: fix formatting in test setup 2026-01-23 11:56:51 -05:00
old4ever bce126664e test: remove unused imports in Kijiji utils tests 2026-01-23 11:56:47 -05:00
old4ever 8cbf11538e test: fix formatting and remove unused HttpError import in Kijiji tests 2026-01-23 11:56:44 -05:00
old4ever 79f47fdaef test: remove unused import in Facebook integration tests 2026-01-23 11:56:41 -05:00
old4ever de5069bf2b test: fix unused variable in Facebook core tests 2026-01-23 11:56:38 -05:00
old4ever 637f1a4e75 fix: resolve biome lint errors and warnings 2026-01-23 10:33:15 -05:00
old4ever 441ff436c4 feat(mcp): extend Kijiji tool with filtering parameters 2026-01-23 09:55:37 -05:00
old4ever 1f53ec912a feat(mcp): add search options to Kijiji and eBay tools 2026-01-23 09:55:21 -05:00
old4ever 053efd815b feat(api/kijiji): add filtering and pagination parameters 2026-01-23 09:54:30 -05:00
old4ever d619fa5d77 feat(api/facebook): add maxItems parameter support 2026-01-23 09:53:51 -05:00
old4ever 050fd0adba feat(api/ebay): add maxItems parameter and error handling 2026-01-23 09:53:00 -05:00
old4ever 7b106c91ce style: format ebay scraper with consistent indentation 2026-01-23 09:52:25 -05:00
old4ever 6e0487f8f3 style: format api-server index with consistent indentation 2026-01-23 09:52:22 -05:00
old4ever da23ca1c3f chore: agents update 2026-01-23 00:52:35 -05:00
old4ever c35aae4c95 chore: biome auto-fixes 2026-01-23 00:52:26 -05:00
old4ever 02162c02f5 chore: biome init 2026-01-23 00:52:10 -05:00
old4everandClaude Opus 4.5 50d56201af feat: port upstream scraper improvements to monorepo 
Kijiji improvements:
- Add error classes: NetworkError, ParseError, RateLimitError, ValidationError
- Add exponential backoff with jitter for retries
- Add request timeout (30s abort)
- Add pagination support (SearchOptions.maxPages)
- Add location/category mappings and resolution functions
- Add enhanced DetailedListing interface with images, seller info, attributes
- Add GraphQL client for seller details

Facebook improvements:
- Add parseFacebookCookieString() for parsing cookie strings
- Add ensureFacebookCookies() with env var fallback
- Add extractFacebookItemData() with multiple extraction paths
- Add fetchFacebookItem() for individual item fetching
- Add extraction metrics and API stability monitoring
- Add vehicle-specific field extraction
- Improve error handling with specific guidance for auth errors

Shared utilities:
- Update http.ts with new error classes and improved fetchHtml

Documentation:
- Port KIJIJI.md, FMARKETPLACE.md, AGENTS.md from upstream

Tests:
- Port kijiji-core, kijiji-integration, kijiji-utils tests
- Port facebook-core, facebook-integration tests
- Add test setup file

Scripts:
- Port parse-facebook-cookies.ts script

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-23 00:34:50 -05:00
old4ever 497c7995a2 feat: ebay 'buy it now' and 'canada only' filters support 2025-12-17 14:38:52 -05:00
old4ever 083b862552 fix healthcheck 2025-12-17 13:58:18 -05:00
old4ever 0a32094e93 feat: adapt Dockerfile for monorepo structure 2025-12-13 20:54:32 -05:00
old4ever a66b5b2362 migrate to monorepo? 2025-12-13 20:31:10 -05:00
old4ever 7da6408d7a fix: add target to a bun build command 2025-10-03 09:06:02 -04:00
old4ever 3863d3139e chore: update docker compose 2025-10-03 09:06:02 -04:00
old4ever 11dbf29f93 chore: update README.md 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-03 08:59:13 -04:00
old4ever 2e2c81b7e5 chore: update package.json 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-03 08:56:03 -04:00
old4ever ea0a69ccd6 feat(docker): enable containerization of marketplace scraper 
Introduce Dockerfile with production build using Bun, .dockerignore for efficient builds, and docker-compose.yml for easy local and prod deployment on port 3000.
 2025-10-03 08:44:51 -04:00
old4ever fa7ac59c45 feat: ebay parser 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-02 13:52:29 -04:00
old4ever 8c52efe5e7 feat(facebook): parse additional listing details like status, images, and seller info 
Enhance Facebook scraping to extract listing status (ACTIVE/SOLD/PENDING/HIDDEN), primary image/video URLs, seller name/ID, category ID, and delivery options, improving response completeness.
 2025-10-02 12:03:59 -04:00
old4ever dcd0da29a2 feat: facebook scraping 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-02 11:50:35 -04:00
old4ever ee09162faa chore: add claude code dev dependency 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-10-02 10:39:49 -04:00
old4ever ddf7639854 fix: add newlines to debug logging due to overlap with the progress bar 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:12:40 -04:00
old4ever d42c74331e feat: add requests progress bar 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:12:10 -04:00
old4ever 49a32bf6b4 fix: bun's --hot option doesn't like modules 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:11:31 -04:00
old4ever 3166b5a95f chore: remove redundant logging 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-18 17:11:09 -04:00
old4ever cb1fb2bae6 feat: add unidecode dependency and slugify function for improved Kijiji search 
This commit introduces the `unidecode` library to handle non-ASCII characters in search queries. A `slugify` function is implemented to improve Kijiji search URL generation by creating more user-friendly and reliable URLs. It converts the user query to a URL friendly string.

It also addresses issues in Kijiji scraping: improves listing filtering logic to ensure only valid listings with prices are added and enhances error handling to provide more informative messages and prevent process termination due to HTTP errors during listing detail retrieval.

Additionally, refactors the Kijiji search query URL, leveraging the new `slugify` function for enhanced URL generation using the search query. Added debugging logs for better traceability.
 2025-09-18 11:35:10 -04:00
old4ever add93dc6bd feat: increase requests per second for Kijiji search 2025-09-18 11:33:47 -04:00
old4ever 99848d1852 build: configure absolute imports using @ alias 
Configures TypeScript path aliases for cleaner and more maintainable imports.
 2025-09-18 11:33:06 -04:00
old4ever d35e37f4c2 feat: implement Kijiji scraping API endpoint 
This commit introduces a new API endpoint that allows users to search for items on Kijiji.
The endpoint accepts a search query as a header or query parameter and returns a JSON response containing the search results. It also handles cases where no query is provided or no results are found, returning appropriate error responses.
 2025-09-17 22:58:11 -04:00
old4ever 2e9547ebaa fix: search GTA only 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-17 22:57:42 -04:00
old4ever 46a8ac92cf feat: extract Kijiji scraping logic into reusable function 
This commit extracts the Kijiji scraping functionality into a reusable function `fetchKijijiItems`. This allows for easier integration into other parts of the application and improves code modularity. The function accepts search query, requests per second, and base URL as parameters, enabling customizable scraping.
 2025-09-17 22:03:24 -04:00
old4ever b7a61423c3 refactor: improve Kijiji scraping architecture 
Introduces types, utilities, and a dedicated HTTP client for improved structure and maintainability.  Includes rate limit handling, retry logic, and error handling.
 2025-09-17 20:45:57 -04:00
old4ever 23dc9cae4c feat: scrape listing details from Kijiji ads 2025-09-17 20:43:07 -04:00
old4ever 257edf394c style: specify source of apollo state interface 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-17 19:00:13 -04:00
old4ever 63683593e4 feat: add kijiji search parser 
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
 2025-09-17 18:58:57 -04:00
old4ever c09650b9f4 build: configure build and development scripts 2025-09-17 18:54:19 -04:00
old4ever 030e3c302d feat: integrate linkedom for DOM manipulation 
Adds `linkedom` as a dependency for parsing HTML and manipulating the DOM. This enables scraping capabilities.
 2025-09-17 18:48:23 -04:00
old4ever 3fccbb7a46 minor tweaks 2025-08-14 14:49:41 -04:00
old4ever e0cf995769 init bun 2025-07-29 22:31:17 -04:00