d619fa5d77
feat(api/facebook): add maxItems parameter support
2026-01-23 09:53:51 -05:00
050fd0adba
feat(api/ebay): add maxItems parameter and error handling
2026-01-23 09:53:00 -05:00
7b106c91ce
style: format ebay scraper with consistent indentation
2026-01-23 09:52:25 -05:00
6e0487f8f3
style: format api-server index with consistent indentation
2026-01-23 09:52:22 -05:00
da23ca1c3f
chore: agents update
2026-01-23 00:52:35 -05:00
c35aae4c95
chore: biome auto-fixes
2026-01-23 00:52:26 -05:00
02162c02f5
chore: biome init
2026-01-23 00:52:10 -05:00
50d56201af
feat: port upstream scraper improvements to monorepo
...
Kijiji improvements:
- Add error classes: NetworkError, ParseError, RateLimitError, ValidationError
- Add exponential backoff with jitter for retries
- Add request timeout (30s abort)
- Add pagination support (SearchOptions.maxPages)
- Add location/category mappings and resolution functions
- Add enhanced DetailedListing interface with images, seller info, attributes
- Add GraphQL client for seller details
Facebook improvements:
- Add parseFacebookCookieString() for parsing cookie strings
- Add ensureFacebookCookies() with env var fallback
- Add extractFacebookItemData() with multiple extraction paths
- Add fetchFacebookItem() for individual item fetching
- Add extraction metrics and API stability monitoring
- Add vehicle-specific field extraction
- Improve error handling with specific guidance for auth errors
Shared utilities:
- Update http.ts with new error classes and improved fetchHtml
Documentation:
- Port KIJIJI.md, FMARKETPLACE.md, AGENTS.md from upstream
Tests:
- Port kijiji-core, kijiji-integration, kijiji-utils tests
- Port facebook-core, facebook-integration tests
- Add test setup file
Scripts:
- Port parse-facebook-cookies.ts script
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com >
2026-01-23 00:34:50 -05:00
497c7995a2
feat: ebay 'buy it now' and 'canada only' filters support
2025-12-17 14:38:52 -05:00
083b862552
fix healthcheck
2025-12-17 13:58:18 -05:00
0a32094e93
feat: adapt Dockerfile for monorepo structure
2025-12-13 20:54:32 -05:00
a66b5b2362
migrate to monorepo?
2025-12-13 20:31:10 -05:00
7da6408d7a
fix: add target to a bun build command
2025-10-03 09:06:02 -04:00
3863d3139e
chore: update docker compose
2025-10-03 09:06:02 -04:00
11dbf29f93
chore: update README.md
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-03 08:59:13 -04:00
2e2c81b7e5
chore: update package.json
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-03 08:56:03 -04:00
ea0a69ccd6
feat(docker): enable containerization of marketplace scraper
...
Introduce Dockerfile with production build using Bun, .dockerignore for efficient builds, and docker-compose.yml for easy local and prod deployment on port 3000.
2025-10-03 08:44:51 -04:00
fa7ac59c45
feat: ebay parser
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 13:52:29 -04:00
8c52efe5e7
feat(facebook): parse additional listing details like status, images, and seller info
...
Enhance Facebook scraping to extract listing status (ACTIVE/SOLD/PENDING/HIDDEN), primary image/video URLs, seller name/ID, category ID, and delivery options, improving response completeness.
2025-10-02 12:03:59 -04:00
dcd0da29a2
feat: facebook scraping
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 11:50:35 -04:00
ee09162faa
chore: add claude code dev dependency
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-10-02 10:39:49 -04:00
ddf7639854
fix: add newlines to debug logging due to overlap with the progress bar
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:12:40 -04:00
d42c74331e
feat: add requests progress bar
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:12:10 -04:00
49a32bf6b4
fix: bun's --hot option doesn't like modules
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:11:31 -04:00
3166b5a95f
chore: remove redundant logging
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-18 17:11:09 -04:00
cb1fb2bae6
feat: add unidecode dependency and slugify function for improved Kijiji search
...
This commit introduces the `unidecode` library to handle non-ASCII characters in search queries. A `slugify` function is implemented to improve Kijiji search URL generation by creating more user-friendly and reliable URLs. It converts the user query to a URL friendly string.
It also addresses issues in Kijiji scraping: improves listing filtering logic to ensure only valid listings with prices are added and enhances error handling to provide more informative messages and prevent process termination due to HTTP errors during listing detail retrieval.
Additionally, refactors the Kijiji search query URL, leveraging the new `slugify` function for enhanced URL generation using the search query. Added debugging logs for better traceability.
2025-09-18 11:35:10 -04:00
add93dc6bd
feat: increase requests per second for Kijiji search
2025-09-18 11:33:47 -04:00
99848d1852
build: configure absolute imports using @ alias
...
Configures TypeScript path aliases for cleaner and more maintainable imports.
2025-09-18 11:33:06 -04:00
d35e37f4c2
feat: implement Kijiji scraping API endpoint
...
This commit introduces a new API endpoint that allows users to search for items on Kijiji.
The endpoint accepts a search query as a header or query parameter and returns a JSON response containing the search results. It also handles cases where no query is provided or no results are found, returning appropriate error responses.
2025-09-17 22:58:11 -04:00
2e9547ebaa
fix: search GTA only
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-17 22:57:42 -04:00
46a8ac92cf
feat: extract Kijiji scraping logic into reusable function
...
This commit extracts the Kijiji scraping functionality into a reusable function `fetchKijijiItems`. This allows for easier integration into other parts of the application and improves code modularity. The function accepts search query, requests per second, and base URL as parameters, enabling customizable scraping.
2025-09-17 22:03:24 -04:00
b7a61423c3
refactor: improve Kijiji scraping architecture
...
Introduces types, utilities, and a dedicated HTTP client for improved structure and maintainability. Includes rate limit handling, retry logic, and error handling.
2025-09-17 20:45:57 -04:00
23dc9cae4c
feat: scrape listing details from Kijiji ads
2025-09-17 20:43:07 -04:00
257edf394c
style: specify source of apollo state interface
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-17 19:00:13 -04:00
63683593e4
feat: add kijiji search parser
...
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev >
2025-09-17 18:58:57 -04:00
c09650b9f4
build: configure build and development scripts
2025-09-17 18:54:19 -04:00
030e3c302d
feat: integrate linkedom for DOM manipulation
...
Adds `linkedom` as a dependency for parsing HTML and manipulating the DOM. This enables scraping capabilities.
2025-09-17 18:48:23 -04:00
3fccbb7a46
minor tweaks
2025-08-14 14:49:41 -04:00
e0cf995769
init bun
2025-07-29 22:31:17 -04:00
e013ce5a46
install bun
2025-07-29 22:30:40 -04:00
cc0bb000f2
init devenv
2025-07-29 22:22:17 -04:00
