e4ab145d70
feat: add cookie support to kijiji scraper
...
Add optional cookie parameter to bypass bot detection (403 errors).
Cookies can be provided via parameter, KIJIJI_COOKIE env var, or
cookies/kijiji.json file. Supports both JSON array and string formats.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 19:29:13 -05:00
1dce0392e3
refactor: use shared cookie utility in ebay scraper
...
Replace inline cookie loading with shared utility functions.
Now supports both JSON array and cookie string formats.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 19:29:02 -05:00
251fcbb7d9
refactor: use shared cookie utility in facebook scraper
...
Replace inline cookie parsing with shared utility functions.
Maintains backward compatibility with existing exports.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 19:28:52 -05:00
9bc57d6b54
refactor: add shared cookie utility to core package
...
Move cookie parsing logic to a dedicated utility module that can be
shared across all scrapers. Supports both JSON array and cookie string
formats for all input sources (parameter, env var, file).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 19:28:44 -05:00
4a467c9f02
fix: support both json and string cookies for facebook
2026-01-23 19:00:51 -05:00
f944d319c2
chore: update dockerignore
2026-01-23 15:43:13 -05:00
cf9784a565
feat: implement cookie priority hierarchy (URL param > env var > file) for Facebook and eBay scrapers
2026-01-23 15:32:17 -05:00
df0c528535
fix: correct formatCentsToCurrency usage in facebook scraper
2026-01-23 14:50:41 -05:00
2f97d3eafd
fix: correct formatCentsToCurrency usage in kijiji scraper
2026-01-23 14:50:41 -05:00
65eb8d1724
refactor: increase kijiji scraping request rate to 4 rps
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 14:50:37 -05:00
f3839aba54
fix: increase kijiji rate limit to 4 rps
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:59:47 -05:00
90b98bfb09
chore: testing mcp server
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:59:28 -05:00
eb6705df0f
feat: add 60-second timeouts to MCP request handlers for reliability
2026-01-23 13:59:28 -05:00
72525609ed
fix: set idle timeout to 255 seconds in MCP server to prevent premature shutdown
2026-01-23 13:59:28 -05:00
8b0a65860c
chore: add imports for linkedom and delay utils in ebay scraper
2026-01-23 13:10:44 -05:00
f9b1c7e096
fix: remove eslint-disable directives
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:08:38 -05:00
9edc74cbeb
chore: local dev scripts
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:07:01 -05:00
ee0fca826d
style: fix formatting in MCP server
2026-01-23 11:56:54 -05:00
f7372612fb
test: fix formatting in test setup
2026-01-23 11:56:51 -05:00
bce126664e
test: remove unused imports in Kijiji utils tests
2026-01-23 11:56:47 -05:00
8cbf11538e
test: fix formatting and remove unused HttpError import in Kijiji tests
2026-01-23 11:56:44 -05:00
79f47fdaef
test: remove unused import in Facebook integration tests
2026-01-23 11:56:41 -05:00
de5069bf2b
test: fix unused variable in Facebook core tests
2026-01-23 11:56:38 -05:00
637f1a4e75
fix: resolve biome lint errors and warnings
2026-01-23 10:33:15 -05:00
441ff436c4
feat(mcp): extend Kijiji tool with filtering parameters
2026-01-23 09:55:37 -05:00
1f53ec912a
feat(mcp): add search options to Kijiji and eBay tools
2026-01-23 09:55:21 -05:00
053efd815b
feat(api/kijiji): add filtering and pagination parameters
2026-01-23 09:54:30 -05:00
d619fa5d77
feat(api/facebook): add maxItems parameter support
2026-01-23 09:53:51 -05:00
050fd0adba
feat(api/ebay): add maxItems parameter and error handling
2026-01-23 09:53:00 -05:00
7b106c91ce
style: format ebay scraper with consistent indentation
2026-01-23 09:52:25 -05:00
6e0487f8f3
style: format api-server index with consistent indentation
2026-01-23 09:52:22 -05:00
da23ca1c3f
chore: agents update
2026-01-23 00:52:35 -05:00
c35aae4c95
chore: biome auto-fixes
2026-01-23 00:52:26 -05:00
02162c02f5
chore: biome init
2026-01-23 00:52:10 -05:00
50d56201af
feat: port upstream scraper improvements to monorepo
...
Kijiji improvements:
- Add error classes: NetworkError, ParseError, RateLimitError, ValidationError
- Add exponential backoff with jitter for retries
- Add request timeout (30s abort)
- Add pagination support (SearchOptions.maxPages)
- Add location/category mappings and resolution functions
- Add enhanced DetailedListing interface with images, seller info, attributes
- Add GraphQL client for seller details
Facebook improvements:
- Add parseFacebookCookieString() for parsing cookie strings
- Add ensureFacebookCookies() with env var fallback
- Add extractFacebookItemData() with multiple extraction paths
- Add fetchFacebookItem() for individual item fetching
- Add extraction metrics and API stability monitoring
- Add vehicle-specific field extraction
- Improve error handling with specific guidance for auth errors
Shared utilities:
- Update http.ts with new error classes and improved fetchHtml
Documentation:
- Port KIJIJI.md, FMARKETPLACE.md, AGENTS.md from upstream
Tests:
- Port kijiji-core, kijiji-integration, kijiji-utils tests
- Port facebook-core, facebook-integration tests
- Add test setup file
Scripts:
- Port parse-facebook-cookies.ts script
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 00:34:50 -05:00
497c7995a2
feat: ebay 'buy it now' and 'canada only' filters support
2025-12-17 14:38:52 -05:00
083b862552
fix healthcheck
2025-12-17 13:58:18 -05:00
0a32094e93
feat: adapt Dockerfile for monorepo structure
2025-12-13 20:54:32 -05:00
a66b5b2362
migrate to monorepo?
2025-12-13 20:31:10 -05:00
7da6408d7a
fix: add target to a bun build command
2025-10-03 09:06:02 -04:00
3863d3139e
chore: update docker compose
2025-10-03 09:06:02 -04:00
11dbf29f93
chore: update README.md
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-03 08:59:13 -04:00
2e2c81b7e5
chore: update package.json
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-03 08:56:03 -04:00
ea0a69ccd6
feat(docker): enable containerization of marketplace scraper
...
Introduce Dockerfile with production build using Bun, .dockerignore for efficient builds, and docker-compose.yml for easy local and prod deployment on port 3000.
2025-10-03 08:44:51 -04:00
fa7ac59c45
feat: ebay parser
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 13:52:29 -04:00
8c52efe5e7
feat(facebook): parse additional listing details like status, images, and seller info
...
Enhance Facebook scraping to extract listing status (ACTIVE/SOLD/PENDING/HIDDEN), primary image/video URLs, seller name/ID, category ID, and delivery options, improving response completeness.
2025-10-02 12:03:59 -04:00
dcd0da29a2
feat: facebook scraping
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 11:50:35 -04:00
ee09162faa
chore: add claude code dev dependency
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 10:39:49 -04:00
ddf7639854
fix: add newlines to debug logging due to overlap with the progress bar
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:12:40 -04:00
d42c74331e
feat: add requests progress bar
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:12:10 -04:00
