chore: update ruler docs
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
This commit is contained in:
@@ -1,52 +1,9 @@
|
||||
## Bun Guidelines
|
||||
## Bun Guide
|
||||
|
||||
**CRITICAL**: Do not assume you know full Bun APIs. For **ANY** Bun API you use, confirm them by using `bun-docs` MCP tools.
|
||||
|
||||
Default to using Bun instead of Node.js.
|
||||
|
||||
- Use `bun <file>` instead of `node <file>` or `ts-node <file>`
|
||||
- Use `bun test` instead of `jest` or `vitest`
|
||||
- Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild`
|
||||
- Use `bun install` instead of `npm install` or `yarn install` or `pnpm install`
|
||||
- Use `bun run <script>` instead of `npm run <script>` or `yarn run <script>` or `pnpm run <script>`
|
||||
- Use `bunx <package> <command>` instead of `npx <package> <command>`
|
||||
- Bun automatically loads .env, so don't use dotenv.
|
||||
|
||||
### APIs
|
||||
|
||||
- `Bun.serve()` supports WebSockets, HTTPS, and routes. Don't use `express`.
|
||||
- `bun:sqlite` for SQLite. Don't use `better-sqlite3`.
|
||||
- `Bun.redis` for Redis. Don't use `ioredis`.
|
||||
- `Bun.sql` for Postgres. Don't use `pg` or `postgres.js`.
|
||||
- `WebSocket` is built-in. Don't use `ws`.
|
||||
- Prefer `Bun.file` over `node:fs`'s readFile/writeFile
|
||||
- Bun.$`ls` instead of execa.
|
||||
|
||||
### Testing
|
||||
|
||||
#### Quick Start
|
||||
- Run tests: `bun test`
|
||||
- Write tests in `tests/` folder
|
||||
|
||||
#### Test Structure
|
||||
- Use `describe` blocks to group related tests
|
||||
- Use `test` for individual test cases
|
||||
- Use `beforeEach`/`afterEach` for setup/teardown
|
||||
|
||||
#### Assertions
|
||||
- Import: `import { test, expect, describe, beforeEach, afterEach, mock } from "bun:test";`
|
||||
- Common: `expect(value).toBe(expected)`, `expect(fn).rejects.toThrow()`
|
||||
- Async: `await expect(asyncFn()).resolves.toBe(expected)`
|
||||
|
||||
#### Mocking
|
||||
- Mock functions: `mock(fn)`
|
||||
- Mock globals: `global.fetch = mock(...)`
|
||||
- Restore mocks in `afterEach` or `finally`
|
||||
|
||||
#### Best Practices
|
||||
- Mock external APIs (fetch, file I/O)
|
||||
- Test error cases and edge conditions
|
||||
- Use descriptive test names
|
||||
- Clean up resources in `afterEach`
|
||||
|
||||
For more information, read the Bun API docs in `node_modules/bun-types/docs/**.mdx`.
|
||||
- Package manager/runtime/test runner is Bun `1.3.13`.
|
||||
- Use `bun install`, `bun run <script>`, `bun test`, and `bun build`; do not add npm/yarn/pnpm scripts.
|
||||
- Prefer Bun-native runtime APIs already used in repo: `Bun.serve`, built-in `fetch`, Web APIs, and `bun:test`.
|
||||
- Keep servers framework-free. Do not introduce Express/Koa/Fastify for the adapters.
|
||||
- Bun auto-loads `.env`; do not add `dotenv`.
|
||||
- For tests, import from `bun:test` and restore mocked globals/env in `afterEach` or `finally`.
|
||||
- Root `bun test` is misleading because `bunfig.toml` sets a dummy root. Run package test paths explicitly.
|
||||
|
||||
@@ -2,37 +2,46 @@
|
||||
|
||||
## Repo Shape
|
||||
|
||||
- Bun workspace monorepo.
|
||||
- `packages/core`: scraper logic, parsing, shared cookie/http/format helpers, and the only checked-in tests.
|
||||
- `packages/api-server`: Bun HTTP adapter exposing `/api/*` routes.
|
||||
- `packages/mcp-server`: MCP JSON-RPC adapter that proxies to the API server.
|
||||
- `dist/`: build output. Do not edit generated files here.
|
||||
- `cookies/`: local cookie examples and docs. Never commit real session cookies.
|
||||
- Bun workspace monorepo with packages under `packages/*`.
|
||||
- `packages/core`: scraper behavior, parsing, result types, cookie handling, HTTP helpers.
|
||||
- `packages/api-server`: Bun HTTP adapter exposing `/api/*` routes over core.
|
||||
- `packages/mcp-server`: MCP/JSON-RPC adapter that proxies to the API server.
|
||||
- `cookies/`: local cookie docs/examples only. Treat real cookie files as secrets.
|
||||
- `dist/`, `node_modules/`, `.turbo/`, `.direnv/`, `.devenv/`: generated/vendor/cache. Do not edit.
|
||||
|
||||
## Commands
|
||||
|
||||
- Install: `bun install`
|
||||
- Lint/format check: `bun run ci`
|
||||
- Build everything: `bun run build`
|
||||
- Run tests: `bun test`
|
||||
- Lint/format/typecheck: `bun run ci`
|
||||
- Build all packages: `bun run build`
|
||||
- Build bundled runtime output: `bun run build:all`
|
||||
- Run tests: `bun test packages/core/test packages/api-server/test packages/mcp-server/test`
|
||||
- API dev server: `bun run --cwd packages/api-server dev`
|
||||
- MCP dev server: `bun run --cwd packages/mcp-server dev`
|
||||
|
||||
## Repo Conventions
|
||||
## Boundaries
|
||||
|
||||
- Keep marketplace scraping behavior in `packages/core`. `api-server` and `mcp-server` stay thin adapters.
|
||||
- Preserve cookie precedence everywhere: request parameter > environment variable > cookie file.
|
||||
- Shared public surface for scraper code is `packages/core/src/index.ts`. Update exports deliberately.
|
||||
- Tests should stay deterministic and offline. Mock `fetch`; do not hit live marketplace endpoints.
|
||||
- Use Bun and Bun-native APIs in this repo. Do not introduce Node-specific tooling unless already required.
|
||||
- Biome and strict TypeScript are part of the contract. Fix code to satisfy them; do not relax config.
|
||||
- Marketplace behavior belongs in `packages/core`, not adapter packages.
|
||||
- HTTP route code should parse request input, call core, and map status/errors.
|
||||
- MCP code should define tools, validate JSON-RPC flow, and map tool args to API URLs.
|
||||
- Keep API query params and MCP tool args in sync.
|
||||
- Shared public surface for scraper code is `packages/core/src/index.ts`; update exports deliberately.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Cookie precedence in core helpers: explicit/request cookie string before environment variable.
|
||||
- Tests must be deterministic and offline. Mock `fetch`; do not hit live marketplace endpoints.
|
||||
- Use Bun and Bun-native APIs. Do not add Node-specific tooling unless already required.
|
||||
- Biome and strict TypeScript are contract. Fix code; do not relax config.
|
||||
|
||||
## Verification
|
||||
|
||||
- Core changes: `bun test && bun run ci`
|
||||
- Cross-package contract changes: `bun test && bun run ci && bun run build`
|
||||
- Adapter-only changes: run the relevant package build plus `bun run ci`
|
||||
- Core changes: `bun test packages/core/test && bun run ci`
|
||||
- Adapter-only changes: relevant package build plus `bun run ci`
|
||||
- Cross-package contract changes: `bun test packages/core/test packages/api-server/test packages/mcp-server/test && bun run ci && bun run build`
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The root `build` script emits separate bundles to `dist/api` and `dist/mcp`, then `scripts/start.sh` launches both.
|
||||
- `bunfig.toml` points test root at `./do-not-run-tests-from-root`; pass package test paths explicitly.
|
||||
- Root `build` cleans `dist`, then Turbo emits bundles for API and MCP.
|
||||
- `scripts/start.sh` launches `dist/api/index.js` and `dist/mcp/index.js`.
|
||||
|
||||
@@ -1,55 +1,18 @@
|
||||
# Marketplace Cookies Setup
|
||||
# cookies
|
||||
|
||||
Both Facebook Marketplace and eBay require valid session cookies to bypass bot detection and access listings.
|
||||
## Scope
|
||||
|
||||
## Cookie Configuration
|
||||
- This directory is for cookie setup docs and local examples only.
|
||||
- Treat any real browser cookie export as a secret, even if already present locally.
|
||||
|
||||
Authenticated scrapers now read cookies only from environment variables:
|
||||
1. `FACEBOOK_COOKIE`
|
||||
2. `EBAY_COOKIE`
|
||||
## Runtime Sources
|
||||
|
||||
---
|
||||
- Authenticated scrapers read raw `Cookie` header strings from environment variables such as `FACEBOOK_COOKIE` and `EBAY_COOKIE`.
|
||||
- Some core entrypoints also accept explicit cookie strings from request/options; explicit input takes precedence over environment values.
|
||||
|
||||
## Facebook Marketplace
|
||||
## Safety Rules
|
||||
|
||||
### Required Cookies
|
||||
- `c_user`: Your Facebook user ID
|
||||
- `xs`: Facebook session token
|
||||
- `fr`: Facebook request token
|
||||
- `datr`: Data attribution token
|
||||
- `sb`: Session browser token
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
export FACEBOOK_COOKIE='c_user=123; xs=token; fr=request'
|
||||
```
|
||||
|
||||
Use the raw `Cookie` header string copied from an authenticated browser session.
|
||||
|
||||
---
|
||||
|
||||
## eBay
|
||||
|
||||
eBay has aggressive bot detection that blocks requests without valid session cookies.
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
export EBAY_COOKIE='s=VALUE; ds2=VALUE; ebay=VALUE'
|
||||
```
|
||||
|
||||
Use the raw `Cookie` header string copied from an authenticated browser session.
|
||||
|
||||
---
|
||||
|
||||
## Important Notes
|
||||
|
||||
- Cookies must be from active browser sessions
|
||||
- Cookies expire and need periodic refresh
|
||||
- **NEVER** commit real cookies to version control
|
||||
- Platforms may still block automated scraping despite valid cookies
|
||||
|
||||
## Security
|
||||
|
||||
Do not commit real cookie values or store them in tracked files.
|
||||
- Never commit real cookie values, browser exports, or session files.
|
||||
- Use placeholder values in docs: `c_user=123; xs=token; fr=request`.
|
||||
- Do not paste cookie values into logs, tests, fixtures, or generated agent docs.
|
||||
- If editing this directory, verify diffs do not contain real `c_user`, `xs`, `fr`, `datr`, `sb`, `s`, `ds2`, or `ebay` values.
|
||||
|
||||
@@ -19,5 +19,6 @@
|
||||
|
||||
## Verify
|
||||
|
||||
- `bun test packages/api-server/test`
|
||||
- `bun run --cwd packages/api-server build`
|
||||
- `bun run ci`
|
||||
|
||||
@@ -21,5 +21,6 @@
|
||||
|
||||
## Verify
|
||||
|
||||
- `bun test packages/mcp-server/test`
|
||||
- `bun run --cwd packages/mcp-server build`
|
||||
- `bun run ci`
|
||||
|
||||
Reference in New Issue
Block a user