chore: update ruler docs
Signed-off-by: Dmytro Stanchiev <git@dmytros.dev>
This commit is contained in:
@@ -1,52 +1,9 @@
|
|||||||
## Bun Guidelines
|
## Bun Guide
|
||||||
|
|
||||||
**CRITICAL**: Do not assume you know full Bun APIs. For **ANY** Bun API you use, confirm them by using `bun-docs` MCP tools.
|
- Package manager/runtime/test runner is Bun `1.3.13`.
|
||||||
|
- Use `bun install`, `bun run <script>`, `bun test`, and `bun build`; do not add npm/yarn/pnpm scripts.
|
||||||
Default to using Bun instead of Node.js.
|
- Prefer Bun-native runtime APIs already used in repo: `Bun.serve`, built-in `fetch`, Web APIs, and `bun:test`.
|
||||||
|
- Keep servers framework-free. Do not introduce Express/Koa/Fastify for the adapters.
|
||||||
- Use `bun <file>` instead of `node <file>` or `ts-node <file>`
|
- Bun auto-loads `.env`; do not add `dotenv`.
|
||||||
- Use `bun test` instead of `jest` or `vitest`
|
- For tests, import from `bun:test` and restore mocked globals/env in `afterEach` or `finally`.
|
||||||
- Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild`
|
- Root `bun test` is misleading because `bunfig.toml` sets a dummy root. Run package test paths explicitly.
|
||||||
- Use `bun install` instead of `npm install` or `yarn install` or `pnpm install`
|
|
||||||
- Use `bun run <script>` instead of `npm run <script>` or `yarn run <script>` or `pnpm run <script>`
|
|
||||||
- Use `bunx <package> <command>` instead of `npx <package> <command>`
|
|
||||||
- Bun automatically loads .env, so don't use dotenv.
|
|
||||||
|
|
||||||
### APIs
|
|
||||||
|
|
||||||
- `Bun.serve()` supports WebSockets, HTTPS, and routes. Don't use `express`.
|
|
||||||
- `bun:sqlite` for SQLite. Don't use `better-sqlite3`.
|
|
||||||
- `Bun.redis` for Redis. Don't use `ioredis`.
|
|
||||||
- `Bun.sql` for Postgres. Don't use `pg` or `postgres.js`.
|
|
||||||
- `WebSocket` is built-in. Don't use `ws`.
|
|
||||||
- Prefer `Bun.file` over `node:fs`'s readFile/writeFile
|
|
||||||
- Bun.$`ls` instead of execa.
|
|
||||||
|
|
||||||
### Testing
|
|
||||||
|
|
||||||
#### Quick Start
|
|
||||||
- Run tests: `bun test`
|
|
||||||
- Write tests in `tests/` folder
|
|
||||||
|
|
||||||
#### Test Structure
|
|
||||||
- Use `describe` blocks to group related tests
|
|
||||||
- Use `test` for individual test cases
|
|
||||||
- Use `beforeEach`/`afterEach` for setup/teardown
|
|
||||||
|
|
||||||
#### Assertions
|
|
||||||
- Import: `import { test, expect, describe, beforeEach, afterEach, mock } from "bun:test";`
|
|
||||||
- Common: `expect(value).toBe(expected)`, `expect(fn).rejects.toThrow()`
|
|
||||||
- Async: `await expect(asyncFn()).resolves.toBe(expected)`
|
|
||||||
|
|
||||||
#### Mocking
|
|
||||||
- Mock functions: `mock(fn)`
|
|
||||||
- Mock globals: `global.fetch = mock(...)`
|
|
||||||
- Restore mocks in `afterEach` or `finally`
|
|
||||||
|
|
||||||
#### Best Practices
|
|
||||||
- Mock external APIs (fetch, file I/O)
|
|
||||||
- Test error cases and edge conditions
|
|
||||||
- Use descriptive test names
|
|
||||||
- Clean up resources in `afterEach`
|
|
||||||
|
|
||||||
For more information, read the Bun API docs in `node_modules/bun-types/docs/**.mdx`.
|
|
||||||
|
|||||||
@@ -2,37 +2,46 @@
|
|||||||
|
|
||||||
## Repo Shape
|
## Repo Shape
|
||||||
|
|
||||||
- Bun workspace monorepo.
|
- Bun workspace monorepo with packages under `packages/*`.
|
||||||
- `packages/core`: scraper logic, parsing, shared cookie/http/format helpers, and the only checked-in tests.
|
- `packages/core`: scraper behavior, parsing, result types, cookie handling, HTTP helpers.
|
||||||
- `packages/api-server`: Bun HTTP adapter exposing `/api/*` routes.
|
- `packages/api-server`: Bun HTTP adapter exposing `/api/*` routes over core.
|
||||||
- `packages/mcp-server`: MCP JSON-RPC adapter that proxies to the API server.
|
- `packages/mcp-server`: MCP/JSON-RPC adapter that proxies to the API server.
|
||||||
- `dist/`: build output. Do not edit generated files here.
|
- `cookies/`: local cookie docs/examples only. Treat real cookie files as secrets.
|
||||||
- `cookies/`: local cookie examples and docs. Never commit real session cookies.
|
- `dist/`, `node_modules/`, `.turbo/`, `.direnv/`, `.devenv/`: generated/vendor/cache. Do not edit.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
- Install: `bun install`
|
- Install: `bun install`
|
||||||
- Lint/format check: `bun run ci`
|
- Lint/format/typecheck: `bun run ci`
|
||||||
- Build everything: `bun run build`
|
- Build all packages: `bun run build`
|
||||||
- Run tests: `bun test`
|
- Build bundled runtime output: `bun run build:all`
|
||||||
|
- Run tests: `bun test packages/core/test packages/api-server/test packages/mcp-server/test`
|
||||||
- API dev server: `bun run --cwd packages/api-server dev`
|
- API dev server: `bun run --cwd packages/api-server dev`
|
||||||
- MCP dev server: `bun run --cwd packages/mcp-server dev`
|
- MCP dev server: `bun run --cwd packages/mcp-server dev`
|
||||||
|
|
||||||
## Repo Conventions
|
## Boundaries
|
||||||
|
|
||||||
- Keep marketplace scraping behavior in `packages/core`. `api-server` and `mcp-server` stay thin adapters.
|
- Marketplace behavior belongs in `packages/core`, not adapter packages.
|
||||||
- Preserve cookie precedence everywhere: request parameter > environment variable > cookie file.
|
- HTTP route code should parse request input, call core, and map status/errors.
|
||||||
- Shared public surface for scraper code is `packages/core/src/index.ts`. Update exports deliberately.
|
- MCP code should define tools, validate JSON-RPC flow, and map tool args to API URLs.
|
||||||
- Tests should stay deterministic and offline. Mock `fetch`; do not hit live marketplace endpoints.
|
- Keep API query params and MCP tool args in sync.
|
||||||
- Use Bun and Bun-native APIs in this repo. Do not introduce Node-specific tooling unless already required.
|
- Shared public surface for scraper code is `packages/core/src/index.ts`; update exports deliberately.
|
||||||
- Biome and strict TypeScript are part of the contract. Fix code to satisfy them; do not relax config.
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Cookie precedence in core helpers: explicit/request cookie string before environment variable.
|
||||||
|
- Tests must be deterministic and offline. Mock `fetch`; do not hit live marketplace endpoints.
|
||||||
|
- Use Bun and Bun-native APIs. Do not add Node-specific tooling unless already required.
|
||||||
|
- Biome and strict TypeScript are contract. Fix code; do not relax config.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
- Core changes: `bun test && bun run ci`
|
- Core changes: `bun test packages/core/test && bun run ci`
|
||||||
- Cross-package contract changes: `bun test && bun run ci && bun run build`
|
- Adapter-only changes: relevant package build plus `bun run ci`
|
||||||
- Adapter-only changes: run the relevant package build plus `bun run ci`
|
- Cross-package contract changes: `bun test packages/core/test packages/api-server/test packages/mcp-server/test && bun run ci && bun run build`
|
||||||
|
|
||||||
## Gotchas
|
## Gotchas
|
||||||
|
|
||||||
- The root `build` script emits separate bundles to `dist/api` and `dist/mcp`, then `scripts/start.sh` launches both.
|
- `bunfig.toml` points test root at `./do-not-run-tests-from-root`; pass package test paths explicitly.
|
||||||
|
- Root `build` cleans `dist`, then Turbo emits bundles for API and MCP.
|
||||||
|
- `scripts/start.sh` launches `dist/api/index.js` and `dist/mcp/index.js`.
|
||||||
|
|||||||
@@ -1,55 +1,18 @@
|
|||||||
# Marketplace Cookies Setup
|
# cookies
|
||||||
|
|
||||||
Both Facebook Marketplace and eBay require valid session cookies to bypass bot detection and access listings.
|
## Scope
|
||||||
|
|
||||||
## Cookie Configuration
|
- This directory is for cookie setup docs and local examples only.
|
||||||
|
- Treat any real browser cookie export as a secret, even if already present locally.
|
||||||
|
|
||||||
Authenticated scrapers now read cookies only from environment variables:
|
## Runtime Sources
|
||||||
1. `FACEBOOK_COOKIE`
|
|
||||||
2. `EBAY_COOKIE`
|
|
||||||
|
|
||||||
---
|
- Authenticated scrapers read raw `Cookie` header strings from environment variables such as `FACEBOOK_COOKIE` and `EBAY_COOKIE`.
|
||||||
|
- Some core entrypoints also accept explicit cookie strings from request/options; explicit input takes precedence over environment values.
|
||||||
|
|
||||||
## Facebook Marketplace
|
## Safety Rules
|
||||||
|
|
||||||
### Required Cookies
|
- Never commit real cookie values, browser exports, or session files.
|
||||||
- `c_user`: Your Facebook user ID
|
- Use placeholder values in docs: `c_user=123; xs=token; fr=request`.
|
||||||
- `xs`: Facebook session token
|
- Do not paste cookie values into logs, tests, fixtures, or generated agent docs.
|
||||||
- `fr`: Facebook request token
|
- If editing this directory, verify diffs do not contain real `c_user`, `xs`, `fr`, `datr`, `sb`, `s`, `ds2`, or `ebay` values.
|
||||||
- `datr`: Data attribution token
|
|
||||||
- `sb`: Session browser token
|
|
||||||
|
|
||||||
### Setup
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export FACEBOOK_COOKIE='c_user=123; xs=token; fr=request'
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the raw `Cookie` header string copied from an authenticated browser session.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## eBay
|
|
||||||
|
|
||||||
eBay has aggressive bot detection that blocks requests without valid session cookies.
|
|
||||||
|
|
||||||
### Setup
|
|
||||||
|
|
||||||
```bash
|
|
||||||
export EBAY_COOKIE='s=VALUE; ds2=VALUE; ebay=VALUE'
|
|
||||||
```
|
|
||||||
|
|
||||||
Use the raw `Cookie` header string copied from an authenticated browser session.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Important Notes
|
|
||||||
|
|
||||||
- Cookies must be from active browser sessions
|
|
||||||
- Cookies expire and need periodic refresh
|
|
||||||
- **NEVER** commit real cookies to version control
|
|
||||||
- Platforms may still block automated scraping despite valid cookies
|
|
||||||
|
|
||||||
## Security
|
|
||||||
|
|
||||||
Do not commit real cookie values or store them in tracked files.
|
|
||||||
|
|||||||
@@ -19,5 +19,6 @@
|
|||||||
|
|
||||||
## Verify
|
## Verify
|
||||||
|
|
||||||
|
- `bun test packages/api-server/test`
|
||||||
- `bun run --cwd packages/api-server build`
|
- `bun run --cwd packages/api-server build`
|
||||||
- `bun run ci`
|
- `bun run ci`
|
||||||
|
|||||||
@@ -21,5 +21,6 @@
|
|||||||
|
|
||||||
## Verify
|
## Verify
|
||||||
|
|
||||||
|
- `bun test packages/mcp-server/test`
|
||||||
- `bun run --cwd packages/mcp-server build`
|
- `bun run --cwd packages/mcp-server build`
|
||||||
- `bun run ci`
|
- `bun run ci`
|
||||||
|
|||||||
Reference in New Issue
Block a user