feat: implement cookie priority hierarchy (URL param > env var > file) for Facebook and eBay scrapers
This commit is contained in:
@@ -1,24 +1,33 @@
|
||||
# Facebook Marketplace Cookies Setup
|
||||
# Marketplace Cookies Setup
|
||||
|
||||
To use the Facebook Marketplace scraper, you need to provide valid Facebook session cookies.
|
||||
Both Facebook Marketplace and eBay require valid session cookies to bypass bot detection and access listings.
|
||||
|
||||
## Option 1: Cookies File (`facebook.json`)
|
||||
## Cookie Priority Hierarchy
|
||||
|
||||
1. Log into Facebook in your browser
|
||||
2. Open Developer Tools → Network tab
|
||||
3. Visit facebook.com/marketplace (ensure you're logged in)
|
||||
4. Look for any marketplace-related requests in the Network tab
|
||||
5. Export cookies from the browser's Application/Storage → Cookies section
|
||||
6. Save the cookies as a JSON array to `facebook.json`
|
||||
All scrapers follow this priority order (highest to lowest):
|
||||
1. **URL Parameter** - Passed directly in API/MCP request (overrides all)
|
||||
2. **Environment Variable** - Set as `FACEBOOK_COOKIE` or `EBAY_COOKIE`
|
||||
3. **Cookie File** - Stored in `facebook.json` or `ebay.json` (fallback)
|
||||
|
||||
The `facebook.json` file should contain Facebook session cookies, particularly:
|
||||
---
|
||||
|
||||
## Facebook Marketplace (`facebook.json`)
|
||||
|
||||
### Required Cookies
|
||||
- `c_user`: Your Facebook user ID
|
||||
- `xs`: Facebook session token
|
||||
- `fr`: Facebook request token
|
||||
- `datr`: Data attribution token
|
||||
- `sb`: Session browser token
|
||||
|
||||
Example structure:
|
||||
### Setup Methods
|
||||
|
||||
**Method 1: Cookie File (Lowest Priority)**
|
||||
1. Log into Facebook in your browser
|
||||
2. Open Developer Tools → Application/Storage → Cookies
|
||||
3. Export cookies as JSON array to `facebook.json`
|
||||
|
||||
Example `facebook.json`:
|
||||
```json
|
||||
[
|
||||
{
|
||||
@@ -27,26 +36,59 @@ Example structure:
|
||||
"domain": ".facebook.com",
|
||||
"path": "/",
|
||||
"secure": true
|
||||
},
|
||||
// ... other cookies
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
## Option 2: URL Parameter
|
||||
|
||||
You can pass cookies directly via the `cookies` URL parameter:
|
||||
|
||||
**Method 2: Environment Variable**
|
||||
```bash
|
||||
export FACEBOOK_COOKIE='c_user=123; xs=token; fr=request'
|
||||
```
|
||||
GET /api/facebook?q=laptop&cookies=[{"name":"c_user","value":"123","domain":".facebook.com",...}]
|
||||
|
||||
**Method 3: URL Parameter (Highest Priority)**
|
||||
```
|
||||
GET /api/facebook?q=laptop&cookies=[{"name":"c_user","value":"123",...}]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## eBay (`ebay.json`)
|
||||
|
||||
eBay has aggressive bot detection that blocks requests without valid session cookies.
|
||||
|
||||
### Setup Methods
|
||||
|
||||
**Method 1: Cookie File (Lowest Priority)**
|
||||
1. Log into eBay in your browser
|
||||
2. Open Developer Tools → Network tab
|
||||
3. Visit ebay.ca and inspect any request headers
|
||||
4. Copy the full `Cookie` header value
|
||||
5. Save as plain text to `ebay.json` (see `ebay.json.example`)
|
||||
|
||||
Example `ebay.json`:
|
||||
```
|
||||
s=VALUE; ds2=VALUE; ebay=VALUE; dp1=VALUE; nonsession=VALUE
|
||||
```
|
||||
|
||||
**Method 2: Environment Variable**
|
||||
```bash
|
||||
export EBAY_COOKIE='s=VALUE; ds2=VALUE; ebay=VALUE'
|
||||
```
|
||||
|
||||
**Method 3: URL Parameter (Highest Priority)**
|
||||
```
|
||||
GET /api/ebay?q=laptop&cookies=s=VALUE;ds2=VALUE;ebay=VALUE
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Important Notes
|
||||
|
||||
- Cookies must be from an active Facebook session
|
||||
- Cookies expire, so you may need to refresh them periodically
|
||||
- Never share real cookies or commit them to version control
|
||||
- Facebook may block automated scraping even with valid cookies
|
||||
- Cookies must be from active browser sessions
|
||||
- Cookies expire and need periodic refresh
|
||||
- **NEVER** commit real cookies to version control
|
||||
- Platforms may still block automated scraping despite valid cookies
|
||||
|
||||
## Security
|
||||
|
||||
The cookies file is intentionally left out of version control for security reasons.</content>
|
||||
All `*.json` files in this directory are git-ignored for security.</content>
|
||||
|
||||
Reference in New Issue
Block a user