Harvest Department

Opportunity discovery, scoring, proposal generation, pipeline management

Overview

The Harvest Department is RUSVEL’s freelance opportunity engine. It scans sources (Upwork, LinkedIn, GitHub, RSS feeds), scores discovered opportunities using keyword matching or LLM-based evaluation, generates tailored proposals via an AI agent, and manages a Kanban pipeline (Cold -> Warm -> Hot -> Won/Lost). It also records deal outcomes (won/lost/withdrawn) that feed back into the scoring model, and supports optional RAG via embedding and vector store for similarity-based scoring hints. Browser-captured data from CDP sessions (Upwork job listings, client profiles) is normalized into opportunities and CRM contacts.

Engine (harvest-engine)

• Crate: crates/harvest-engine/src/lib.rs
• Lines: 1013 (lib.rs) + submodules (source, scorer, proposal, pipeline, events, outcomes, cdp_source)
• Status: Wired (real business logic)

Public API

Internal Structure

• HarvestSource trait (source.rs) – Interface for scanning opportunity sources. Returns Vec<RawOpportunity>. Includes MockSource for testing.
• CdpSource (cdp_source.rs) – CDP/browser-based source for Upwork. Includes default JavaScript extraction script.
• OpportunityScorer (scorer.rs) – Scores raw opportunities. Two methods: ScoringMethod::Keyword (fast, pattern-based) and ScoringMethod::Llm (uses AgentPort for AI evaluation). Accepts outcome hints for calibration.
• ProposalGenerator (proposal.rs) – Generates proposals using AgentPort. Produces Proposal with body, estimated value, tone, and metadata.
• Pipeline (pipeline.rs) – Kanban pipeline management via ObjectStore. add(), list(), advance(), stats().
• outcomes (outcomes.rs) – Deal outcome recording and retrieval. record_outcome(), list_outcomes(), recent_outcome_prompt_lines() for LLM hints.
• events (events.rs) – Event kind constants.

Data Types

#![allow(unused)]
fn main() {
pub struct HarvestConfig {
    pub skills: Vec<String>,      // Skills to match (default: ["rust"])
    pub min_budget: Option<f64>,  // Minimum budget filter
}

pub struct OpportunityScoreUpdate {
    pub score: f64,
    pub reasoning: String,
}

pub enum HarvestDealOutcome { Won, Lost, Withdrawn }

pub struct HarvestOutcomeRecord {
    pub id: String,
    pub session_id: SessionId,
    pub opportunity_snapshot: Value,
    pub result: HarvestDealOutcome,
    pub notes: String,
}
}

RAG Integration

When configure_rag() is called with both an EmbeddingPort and VectorStorePort:

1. Scoring hints: Before scoring, the engine embeds the opportunity title+description, searches the vector store for similar past outcomes, and includes the top 5 matches as prompt hints.
2. Outcome indexing: When record_opportunity_outcome() is called, the outcome is embedded and upserted into the vector store with kind: "harvest_outcome" metadata.

Department Wrapper (dept-harvest)

• Crate: crates/dept-harvest/src/lib.rs
• Lines: 99
• Manifest: crates/dept-harvest/src/manifest.rs

The wrapper creates a HarvestEngine with builder pattern (with_events, with_agent), calls configure_rag() to wire optional embedding and vector store from the registration context, and persists the engine reference.

Manifest Declaration

System Prompt

You are the Harvest department of RUSVEL.

Focus: finding opportunities, scoring gigs, drafting proposals. Sources: Upwork, LinkedIn, GitHub.

Capabilities

• opportunity_discovery

Quick Actions

Registered Tools

Personas

Skills

Rules

Jobs

Events

Produced

Consumed

The Harvest department does not consume events from other departments. It is triggered by direct API/CLI calls and browser capture events.

API Routes

CLI Commands

rusvel harvest pipeline    # Show pipeline statistics
rusvel harvest status      # Department status
rusvel harvest list        # List items
rusvel harvest events      # Show recent events

Entity Auto-Discovery

Agents, skills, rules, hooks, and MCP servers scoped to the Harvest department are stored with metadata.engine = "harvest". The shared CRUD API routes filter by this key so each department sees only its own entities.

Chat Flow

sequenceDiagram
    participant User
    participant API as rusvel-api
    participant Chat as Chat Handler
    participant Config as ConfigPort
    participant Skills as Skill Resolver
    participant Rules as Rule Loader
    participant Agent as AgentRuntime
    participant Harvest as HarvestEngine
    participant Events as EventPort
    participant VectorStore as VectorStorePort
    participant Hooks as Hook Dispatcher

    User->>API: POST /api/dept/harvest/chat {message}
    API->>Chat: route to department chat handler
    Chat->>Config: load department config (skills, min_budget)
    Chat->>Skills: resolve_skill() -- check for {{input}} interpolation
    Chat->>Chat: check agent override (custom agent for harvest)
    Chat->>Rules: load_rules_for_engine("harvest") -- append "Human Approval Gate" rule
    Note over Chat: Inject capabilities: opportunity_discovery
    Note over Chat: System prompt: "You are the Harvest department..."
    Chat->>Agent: AgentRuntime::run_streaming(config, tools, prompt)
    Agent->>Harvest: tool calls (harvest.scan, harvest.score, harvest.proposal)
    Harvest->>VectorStore: similarity search for outcome hints (if RAG configured)
    Harvest->>Events: emit harvest.* events
    Agent-->>Chat: SSE stream (AgentEvent chunks)
    Chat-->>User: SSE response
    Chat->>Events: emit chat completion event
    Chat->>Hooks: tokio::spawn hook dispatch

Extending This Department

1. Add a new tool

Register the tool in crates/dept-harvest/src/lib.rs inside the register() method using ctx.tools.add("harvest", "harvest.new_tool", ...). Add a matching ToolContribution entry in crates/dept-harvest/src/manifest.rs.

2. Add a new event kind

Add a new pub const in crates/harvest-engine/src/events.rs. Emit it from the engine method. Add the event kind string to events_produced in crates/dept-harvest/src/manifest.rs.

3. Add a new persona

Add a PersonaContribution entry in the personas vec in crates/dept-harvest/src/manifest.rs.

4. Add a new skill

Add a SkillContribution entry in the skills vec in crates/dept-harvest/src/manifest.rs.

5. Add a new API route

Add a RouteContribution entry in the routes vec in crates/dept-harvest/src/manifest.rs. Implement the handler in crates/rusvel-api/src/engine_routes.rs and wire the route in crates/rusvel-api/src/lib.rs.

6. Add a new harvest source

Implement the HarvestSource trait in a new file under crates/harvest-engine/src/. Return Vec<RawOpportunity> from scan(). The engine’s scan() method will automatically score and persist results.

Port Dependencies

Configuration

{
  "skills": ["rust", "axum", "tokio"],
  "min_budget": 1000.0
}

• skills – Skills to match when scoring opportunities and expanding RSS queries. Default: ["rust"].
• min_budget – Minimum budget filter. Opportunities below this are scored lower.

Object Store Kinds

Scoring System

The OpportunityScorer supports two scoring methods:

Keyword Scoring (ScoringMethod::Keyword)

Fast, pattern-based scoring used when no AgentPort is configured:

• Matches opportunity title and description against configured skills
• Applies budget multiplier based on min_budget threshold
• Returns a normalized score (0.0 to 1.0)

LLM Scoring (ScoringMethod::Llm)

AI-powered scoring used when AgentPort is available:

• Sends opportunity details plus configured skills to the agent
• Includes outcome hints (recent outcomes + vector similarity matches) for calibration
• Agent returns a score and reasoning

Outcome-Based Calibration

When RAG is configured, scoring includes:

1. Recent outcomes: The last 12 HarvestOutcomeRecord entries for the session, formatted as prompt lines
2. Vector similarity: The opportunity title+description is embedded and searched against indexed outcomes; top 5 matches with similarity scores are included

This creates a feedback loop: won/lost outcomes improve future scoring accuracy.

Pipeline Stages

Opportunities flow through a Kanban pipeline:

Cold -> Warm -> Hot -> Won
                   -> Lost
                   -> Withdrawn

• Cold: Newly discovered, not yet evaluated
• Warm: Scored positively, under consideration
• Hot: Actively pursuing, proposal sent
• Won: Deal closed successfully
• Lost: Opportunity not won
• Withdrawn: Withdrawn from consideration

advance_opportunity() moves an opportunity to any valid stage and emits harvest.pipeline.advanced.

Browser Data Capture (CDP)

The on_data_captured() method handles BrowserEvent::DataCaptured payloads from CDP sessions:

Upwork Job Listings (kind: "job_listing")

• Accepts single job or {jobs: [...]} array
• Extracts: title, description, url, budget, skills, posted_at
• Runs through the scorer
• Creates Opportunity in Cold stage
• Emits harvest.opportunity.discovered

Upwork Client Profiles (kind: "client_profile")

• Extracts: name, profile_url, company
• Creates Contact in ObjectStore
• Emits harvest.contact.captured

CDP Extract JavaScript

DEFAULT_CDP_EXTRACT_JS provides a default JavaScript snippet for extracting job listing cards from Upwork pages via Chrome DevTools Protocol.

UI Integration

The manifest declares a dashboard card and 6 tabs:

• Dashboard card: “Opportunity Pipeline” (medium) – Cold, warm, and hot opportunities
• Tabs: actions, engine, agents, skills, rules, events
• has_settings: true (supports skills and budget configuration)

Testing

cargo test -p harvest-engine    # 12 tests

Key test scenarios:

• Scan with mock source returns 3 opportunities
• Pipeline stores and lists correctly
• Pipeline stats match (all Cold after scan)
• Health returns healthy
• Proposals respect session filter
• Outcome recording and retrieval

cargo test -p dept-harvest      # Department wrapper tests

Key test scenarios:

• Department creates with correct manifest ID
• Manifest declares 5 routes, 3 tools, 3 events
• Manifest requires StoragePort, EventPort, AgentPort (non-optional) + ConfigPort (optional)
• Manifest serializes to valid JSON