Core serialization library for crowd-pilot's IDE interaction data. Converts IDE events (tab switches, edits, terminal commands, etc.) into conversation format for training language models.

This is a Rust library with:

• Node.js/TypeScript bindings (via napi-rs) - for the VS Code extension (uses character approximation for token counting)
• CLI binary - for batch preprocessing (uses HuggingFace tokenizer via embedded Python)

The serialization logic is the single source of truth, ensuring consistency between runtime inference and training data preprocessing.

• crates/core - Core serialization logic
• crates/napi - Node.js bindings (@crowd-pilot/serializer npm package)
• crates/cli - CLI binary for preprocessing (crowd-pilot-serialize)

• Rust 1.70+
• Node.js 18+ (for napi bindings)
• Python 3.9+ with transformers installed (for CLI tokenizer)

cargo build --release

cd crates/napi npm install npm run build

cargo build --release -p crowd-pilot-serialize

import { ConversationStateManager } from '@crowd-pilot/serializer'; const manager = new ConversationStateManager({ viewportRadius: 10, coalesceRadius: 5, maxTokensPerMessage: 2048, maxTokensPerTerminalOutput: 256, }); manager.handleTabEvent('/path/to/file.ts', 'file contents...'); manager.handleContentEvent('/path/to/file.ts', 10, 0, 'inserted text'); const messages = manager.finalizeForModel();

crowd-pilot-serialize \ --output-format sed \ --csv-root ./data/sessions \ --output-dir ./output \ --tokenizer "Qwen/Qwen2-7B" \ --chat-template qwen3 \ --max-tokens-per-conversation 8192 \ --max-tokens-per-message 2048 \ --val-ratio 0.1

This uses shared CSV coalescing and outputs training.jsonl and validation.jsonl in SED conversation format.

crowd-pilot-serialize \ --output-format zeta \ --csv-root ./data/sessions \ --output-dir ./output-zeta \ --tokenizer "Qwen/Qwen2-7B" \ --chat-template qwen3 \ --zeta-max-editable-tokens 180 \ --zeta-max-context-tokens 350 \ --zeta-diff-context-lines 3

This uses shared CSV coalescing and converts each CSV transition into a Zeta training example.

crowd-pilot-serialize \ --output-format sweep \ --csv-root ./data/sessions \ --output-dir ./output-sweep \ --tokenizer "Qwen/Qwen2-7B" \ --chat-template qwen3 \ --sweep-viewport-lines 21 \ --sweep-opened-file-context full \ --sweep-history-center changed

This uses shared CSV coalescing and converts each CSV transition into a Sweep-style next-edit training example. Sweep applies a hard --max-tokens-per-conversation budget by trimming oldest history first; if even zero-history context does not fit, the sample is dropped.

crowd-pilot-replay \ --session ./recordings \ --terminal-cols 120 \ --terminal-rows 30 \ --delay-ms 100

This replays crowd-code 2.0 recordings and renders editor + terminal viewports. The --session path must be a single source_part_*.tar.gz file or a directory containing those .tar.gz parts. Terminal output is VT-rendered from terminal bytestream events when available; otherwise it falls back to the recorded terminal viewport lines.

Apache 2.0