This commit introduces the `unidecode` library to handle non-ASCII characters in search queries. A `slugify` function is implemented to improve Kijiji search URL generation by creating more user-friendly and reliable URLs. It converts the user query to a URL friendly string.
It also addresses issues in Kijiji scraping: improves listing filtering logic to ensure only valid listings with prices are added and enhances error handling to provide more informative messages and prevent process termination due to HTTP errors during listing detail retrieval.
Additionally, refactors the Kijiji search query URL, leveraging the new `slugify` function for enhanced URL generation using the search query. Added debugging logs for better traceability.
This commit introduces a new API endpoint that allows users to search for items on Kijiji.
The endpoint accepts a search query as a header or query parameter and returns a JSON response containing the search results. It also handles cases where no query is provided or no results are found, returning appropriate error responses.
This commit extracts the Kijiji scraping functionality into a reusable function `fetchKijijiItems`. This allows for easier integration into other parts of the application and improves code modularity. The function accepts search query, requests per second, and base URL as parameters, enabling customizable scraping.
Introduces types, utilities, and a dedicated HTTP client for improved structure and maintainability. Includes rate limit handling, retry logic, and error handling.
old4ever