9bc57d6b54
refactor: add shared cookie utility to core package
...
Move cookie parsing logic to a dedicated utility module that can be
shared across all scrapers. Supports both JSON array and cookie string
formats for all input sources (parameter, env var, file).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 19:28:44 -05:00
4a467c9f02
fix: support both json and string cookies for facebook
2026-01-23 19:00:51 -05:00
f944d319c2
chore: update dockerignore
2026-01-23 15:43:13 -05:00
cf9784a565
feat: implement cookie priority hierarchy (URL param > env var > file) for Facebook and eBay scrapers
2026-01-23 15:32:17 -05:00
df0c528535
fix: correct formatCentsToCurrency usage in facebook scraper
2026-01-23 14:50:41 -05:00
2f97d3eafd
fix: correct formatCentsToCurrency usage in kijiji scraper
2026-01-23 14:50:41 -05:00
65eb8d1724
refactor: increase kijiji scraping request rate to 4 rps
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 14:50:37 -05:00
f3839aba54
fix: increase kijiji rate limit to 4 rps
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:59:47 -05:00
90b98bfb09
chore: testing mcp server
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:59:28 -05:00
eb6705df0f
feat: add 60-second timeouts to MCP request handlers for reliability
2026-01-23 13:59:28 -05:00
72525609ed
fix: set idle timeout to 255 seconds in MCP server to prevent premature shutdown
2026-01-23 13:59:28 -05:00
8b0a65860c
chore: add imports for linkedom and delay utils in ebay scraper
2026-01-23 13:10:44 -05:00
f9b1c7e096
fix: remove eslint-disable directives
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:08:38 -05:00
9edc74cbeb
chore: local dev scripts
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2026-01-23 13:07:01 -05:00
ee0fca826d
style: fix formatting in MCP server
2026-01-23 11:56:54 -05:00
f7372612fb
test: fix formatting in test setup
2026-01-23 11:56:51 -05:00
bce126664e
test: remove unused imports in Kijiji utils tests
2026-01-23 11:56:47 -05:00
8cbf11538e
test: fix formatting and remove unused HttpError import in Kijiji tests
2026-01-23 11:56:44 -05:00
79f47fdaef
test: remove unused import in Facebook integration tests
2026-01-23 11:56:41 -05:00
de5069bf2b
test: fix unused variable in Facebook core tests
2026-01-23 11:56:38 -05:00
637f1a4e75
fix: resolve biome lint errors and warnings
2026-01-23 10:33:15 -05:00
441ff436c4
feat(mcp): extend Kijiji tool with filtering parameters
2026-01-23 09:55:37 -05:00
1f53ec912a
feat(mcp): add search options to Kijiji and eBay tools
2026-01-23 09:55:21 -05:00
053efd815b
feat(api/kijiji): add filtering and pagination parameters
2026-01-23 09:54:30 -05:00
d619fa5d77
feat(api/facebook): add maxItems parameter support
2026-01-23 09:53:51 -05:00
050fd0adba
feat(api/ebay): add maxItems parameter and error handling
2026-01-23 09:53:00 -05:00
7b106c91ce
style: format ebay scraper with consistent indentation
2026-01-23 09:52:25 -05:00
6e0487f8f3
style: format api-server index with consistent indentation
2026-01-23 09:52:22 -05:00
da23ca1c3f
chore: agents update
2026-01-23 00:52:35 -05:00
c35aae4c95
chore: biome auto-fixes
2026-01-23 00:52:26 -05:00
02162c02f5
chore: biome init
2026-01-23 00:52:10 -05:00
50d56201af
feat: port upstream scraper improvements to monorepo
...
Kijiji improvements:
- Add error classes: NetworkError, ParseError, RateLimitError, ValidationError
- Add exponential backoff with jitter for retries
- Add request timeout (30s abort)
- Add pagination support (SearchOptions.maxPages)
- Add location/category mappings and resolution functions
- Add enhanced DetailedListing interface with images, seller info, attributes
- Add GraphQL client for seller details
Facebook improvements:
- Add parseFacebookCookieString() for parsing cookie strings
- Add ensureFacebookCookies() with env var fallback
- Add extractFacebookItemData() with multiple extraction paths
- Add fetchFacebookItem() for individual item fetching
- Add extraction metrics and API stability monitoring
- Add vehicle-specific field extraction
- Improve error handling with specific guidance for auth errors
Shared utilities:
- Update http.ts with new error classes and improved fetchHtml
Documentation:
- Port KIJIJI.md, FMARKETPLACE.md, AGENTS.md from upstream
Tests:
- Port kijiji-core, kijiji-integration, kijiji-utils tests
- Port facebook-core, facebook-integration tests
- Add test setup file
Scripts:
- Port parse-facebook-cookies.ts script
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 00:34:50 -05:00
497c7995a2
feat: ebay 'buy it now' and 'canada only' filters support
2025-12-17 14:38:52 -05:00
083b862552
fix healthcheck
2025-12-17 13:58:18 -05:00
0a32094e93
feat: adapt Dockerfile for monorepo structure
2025-12-13 20:54:32 -05:00
a66b5b2362
migrate to monorepo?
2025-12-13 20:31:10 -05:00
7da6408d7a
fix: add target to a bun build command
2025-10-03 09:06:02 -04:00
3863d3139e
chore: update docker compose
2025-10-03 09:06:02 -04:00
11dbf29f93
chore: update README.md
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-03 08:59:13 -04:00
2e2c81b7e5
chore: update package.json
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-03 08:56:03 -04:00
ea0a69ccd6
feat(docker): enable containerization of marketplace scraper
...
Introduce Dockerfile with production build using Bun, .dockerignore for efficient builds, and docker-compose.yml for easy local and prod deployment on port 3000.
2025-10-03 08:44:51 -04:00
fa7ac59c45
feat: ebay parser
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 13:52:29 -04:00
8c52efe5e7
feat(facebook): parse additional listing details like status, images, and seller info
...
Enhance Facebook scraping to extract listing status (ACTIVE/SOLD/PENDING/HIDDEN), primary image/video URLs, seller name/ID, category ID, and delivery options, improving response completeness.
2025-10-02 12:03:59 -04:00
dcd0da29a2
feat: facebook scraping
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 11:50:35 -04:00
ee09162faa
chore: add claude code dev dependency
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 10:39:49 -04:00
ddf7639854
fix: add newlines to debug logging due to overlap with the progress bar
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:12:40 -04:00
d42c74331e
feat: add requests progress bar
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:12:10 -04:00
49a32bf6b4
fix: bun's --hot option doesn't like modules
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:11:31 -04:00
3166b5a95f
chore: remove redundant logging
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:11:09 -04:00
cb1fb2bae6
feat: add unidecode dependency and slugify function for improved Kijiji search
...
This commit introduces the `unidecode` library to handle non-ASCII characters in search queries. A `slugify` function is implemented to improve Kijiji search URL generation by creating more user-friendly and reliable URLs. It converts the user query to a URL friendly string.
It also addresses issues in Kijiji scraping: improves listing filtering logic to ensure only valid listings with prices are added and enhances error handling to provide more informative messages and prevent process termination due to HTTP errors during listing detail retrieval.
Additionally, refactors the Kijiji search query URL, leveraging the new `slugify` function for enhanced URL generation using the search query. Added debugging logs for better traceability.
2025-09-18 11:35:10 -04:00
