mirror of https://github.com/ghostfolio/ghostfolio
Browse Source
Implement streaming chat endpoint with Claude via AI SDK. 10 tools: portfolio, holdings, performance, transactions, market-data, symbol-search, account-manage, activity-manage, tag-manage, watchlist-manage. 3-layer verification: hallucination check, output validation, confidence scoring. Lazy-loaded module — zero impact when ANTHROPIC_API_KEY is unset.pull/6458/head
Ryan Waits
1 month ago
29 changed files with 3640 additions and 1 deletions
@ -0,0 +1,279 @@ |
|||||
|
# Agent Module |
||||
|
|
||||
|
AI-powered portfolio assistant built as a NestJS module inside the Ghostfolio fork. Sonnet 4.6 with 6 tools, SSE streaming, structured observability, 2-tier eval suite, and CI-gated golden tests. |
||||
|
|
||||
|
**Live**: https://ghostfolio-4eid.onrender.com/agent |
||||
|
|
||||
|
--- |
||||
|
|
||||
|
## Try It Now (30 seconds) |
||||
|
|
||||
|
A demo user with a seeded portfolio is auto-created on every deploy. |
||||
|
|
||||
|
1. Go to https://ghostfolio-4eid.onrender.com/agent |
||||
|
2. Ask "What do I own?" |
||||
|
|
||||
|
The demo portfolio has AAPL (20 shares), MSFT (10), VOO (20), GOOGL (8), and 0.5 BTC with buys, sells, and dividends spanning Jan 2024 – Jan 2025. |
||||
|
|
||||
|
### New User Flow (bring your own data) |
||||
|
|
||||
|
1. Go to https://ghostfolio-4eid.onrender.com and click **Get Started** |
||||
|
2. Save the Security Token shown -- this is your only login credential |
||||
|
3. Navigate to **Portfolio → Activities → Import** (upload icon) |
||||
|
4. Upload one of the sample CSVs from `seed-data/`: |
||||
|
- `stocks-portfolio.csv` -- US equities (VOO, AAPL, MSFT, GOOGL, AMZN, NVDA, META) |
||||
|
- `crypto-portfolio.csv` -- crypto (BTC, ETH, SOL, LINK, UNI) |
||||
|
- `hybrid-portfolio.csv` -- mixed stocks + crypto |
||||
|
5. Go to `/agent` and chat |
||||
|
|
||||
|
--- |
||||
|
|
||||
|
## Prerequisites |
||||
|
|
||||
|
- Node.js 22+ |
||||
|
- Docker (for local Postgres + Redis) |
||||
|
- npm |
||||
|
|
||||
|
## Architecture |
||||
|
|
||||
|
- **Model**: Sonnet 4.6 via Vercel AI SDK v6 (`ai@6.0.97`, `@ai-sdk/anthropic@3.0.46`) |
||||
|
- **Schemas**: Zod v4 (`zod@4.3.6`) -- required by AI SDK v6 `inputSchema` |
||||
|
- **Max steps**: 6 per chat (multi-tool chaining via `stopWhen: stepCountIs(6)`) |
||||
|
- **Auth**: Ghostfolio JWT + `readAiPrompt` permission guard |
||||
|
- **Stream**: SSE UI message stream (`text-delta`, `tool-input-start` events) |
||||
|
|
||||
|
## Tools |
||||
|
|
||||
|
| Tool | Description | |
||||
|
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | |
||||
|
| `symbol_search` | Disambiguate crypto vs stock symbols, find correct CoinGecko slugs or Yahoo tickers. Use before `market_data` for any non-obvious crypto. | |
||||
|
| `portfolio_analysis` | Holdings, allocations, total value, account breakdown | |
||||
|
| `portfolio_performance` | Returns, net performance, chart data over a date range | |
||||
|
| `holdings_lookup` | Deep dive on a single position (dividends, fees, sectors, countries) | |
||||
|
| `market_data` | Live quotes for 1-10 symbols. Default provider: FMP (Financial Modeling Prep). Also supports CoinGecko, Yahoo, etc. via `dataProviderService.getQuotes()` | |
||||
|
| `transaction_history` | Buy/sell/dividend/fee activity log, filterable + sortable | |
||||
|
|
||||
|
All tools wrapped in try/catch -- errors returned to LLM as `{ error: ... }` so it can recover gracefully. |
||||
|
|
||||
|
## Observability |
||||
|
|
||||
|
| Layer | Detail | |
||||
|
| -------------------- | --------------------------------------------------------------------------------------------------------------- | |
||||
|
| Structured logs | `chat_start`, `step_finish`, `verification`, `chat_complete`, `chat_error` -- JSON with requestId, userId, etc. | |
||||
|
| In-memory metrics | Ring buffer (last 1000 chats) via `AgentMetricsService`, served at `GET /api/v1/agent/metrics?since=1h` | |
||||
|
| Postgres persistence | `AgentChatLog` table with `verificationScore` + `verificationResult` -- survives deploys, queryable via Prisma | |
||||
|
| Verification | 3 systems run on every response: output validation, hallucination detection, confidence scoring (0-1 composite) | |
||||
|
| Feedback | `AgentFeedback` table -- thumbs up/down per response, unique per requestId+userId, summary in metrics endpoint | |
||||
|
| Error handling | `onError` callback in `streamText()` records error metrics; controller wrapped in try/catch returns clean 500 | |
||||
|
| Security | Error messages sanitized -- DB URLs, Redis URLs, API keys redacted before storage/exposure | |
||||
|
|
||||
|
## Verification Systems |
||||
|
|
||||
|
Three deterministic checks run in `onFinish` on every agent response: |
||||
|
|
||||
|
| System | What it checks | Weight | |
||||
|
| ----------------------- | --------------------------------------------------------------------------------------------- | ------ | |
||||
|
| Output Validation | Non-empty, reasonable length, numeric data present when tools called, disclaimer on forecasts | 0.3 | |
||||
|
| Hallucination Detection | Ticker symbols match tool data, dollar amounts approximately match, no phantom holdings | 0.3 | |
||||
|
| Confidence Score | Composite 0-1 from tool success rate (0.3), step efficiency (0.1), validation + hallucination | -- | |
||||
|
|
||||
|
Results are persisted to `AgentChatLog` and queryable via `GET /agent/verification/:requestId`. |
||||
|
|
||||
|
## Feedback |
||||
|
|
||||
|
Chat UI shows thumbs up/down buttons after each assistant message. Feedback is stored in `AgentFeedback` with a unique constraint per (requestId, userId). |
||||
|
|
||||
|
- `POST /agent/feedback` -- submit rating (-1 or 1) + optional comment |
||||
|
- `GET /agent/metrics` -- includes feedback summary (total, positive, negative, satisfaction rate, recent comments) |
||||
|
|
||||
|
## Eval Suite (2-tier, 52 cases) |
||||
|
|
||||
|
| Tier | Cases | Scorers | Threshold | Purpose | |
||||
|
| ---------- | ----- | ---------------------------------------------------------------------------- | --------------- | ---------------------------------- | |
||||
|
| Golden Set | 19 | `GoldenCheck` (deterministic, binary pass/fail, seed-data agnostic) | 100% required | CI gate -- runs every push to main | |
||||
|
| Scenarios | 33 | `ToolCallAccuracy` + `HasResponse` + `ResponseQuality` + `VerificationCheck` | 80%+ acceptable | Manual regression check | |
||||
|
|
||||
|
**Golden Set breakdown**: tool routing (7), structural output (4), no-tool behavioral (2), guardrails (6). |
||||
|
|
||||
|
**Scenarios breakdown**: single-tool (10), multi-tool (10), ambiguous (6), edge (7). |
||||
|
|
||||
|
```bash |
||||
|
# Run golden set (requires ANTHROPIC_API_KEY + TEST_USER_ACCESS_TOKEN in env) |
||||
|
npx evalite run evals/golden/agent-golden.eval.ts |
||||
|
|
||||
|
# Run scenarios |
||||
|
npx evalite run evals/scenarios/agent-scenarios.eval.ts |
||||
|
``` |
||||
|
|
||||
|
## CI Pipeline |
||||
|
|
||||
|
`.github/workflows/golden-evals.yml` -- triggers on push to `main` (agent/eval file changes) or manual dispatch. Hits the deployed Render instance, fails if golden set drops below 100%. |
||||
|
|
||||
|
**Required GitHub secrets**: `RENDER_URL`, `TEST_USER_ACCESS_TOKEN`, `ANTHROPIC_API_KEY` |
||||
|
|
||||
|
## Deployment |
||||
|
|
||||
|
| Resource | Config | |
||||
|
| ------------- | ---------------------------------------------------------- | |
||||
|
| Platform | Render (Docker) via `render.yaml` blueprint | |
||||
|
| URL | https://ghostfolio-4eid.onrender.com | |
||||
|
| Web | Standard plan, 2GB RAM | |
||||
|
| Redis | Starter, volatile-lru eviction | |
||||
|
| Postgres | Basic 1GB | |
||||
|
| Data provider | FMP (paid tier, `batch-quote-short` endpoint) | |
||||
|
| Entrypoint | `prisma migrate deploy` -> `prisma db seed` -> `node main` | |
||||
|
|
||||
|
**Render env vars**: |
||||
|
|
||||
|
| Var | Required | Notes | |
||||
|
| --------------------------------- | -------- | ---------------------------------------------------------------------------------------- | |
||||
|
| `ANTHROPIC_API_KEY` | Yes | Powers the agent LLM | |
||||
|
| `API_KEY_FINANCIAL_MODELING_PREP` | Yes | Primary data provider for stocks/ETFs | |
||||
|
| `API_KEY_COINGECKO_DEMO` | Yes | Free demo key from [CoinGecko](https://www.coingecko.com/en/api/pricing) -- 30 calls/min | |
||||
|
| `DATA_SOURCES` | Yes | `["FINANCIAL_MODELING_PREP","COINGECKO","MANUAL"]` | |
||||
|
| `DATA_SOURCE_EXCHANGE_RATES` | Yes | `FINANCIAL_MODELING_PREP` | |
||||
|
| `DATA_SOURCE_IMPORT` | Yes | `FINANCIAL_MODELING_PREP` | |
||||
|
| `NODE_ENV` | Yes | `production` | |
||||
|
|
||||
|
**Endpoints**: |
||||
|
|
||||
|
- Chat UI: `/agent` (Angular page) |
||||
|
- Chat API: `POST /api/v1/agent/chat` |
||||
|
- Feedback: `POST /api/v1/agent/feedback` |
||||
|
- Verification: `GET /api/v1/agent/verification/:requestId` |
||||
|
- Metrics: `GET /api/v1/agent/metrics?since=1h` (includes feedback summary) |
||||
|
|
||||
|
## Demo User |
||||
|
|
||||
|
A demo user is auto-created by `prisma db seed` on every deploy (and locally via `npx prisma db seed`). |
||||
|
|
||||
|
- **Security token**: `demo-token-2026` |
||||
|
- **Role**: ADMIN |
||||
|
- **Account**: "Main Brokerage" (USD, $5,000 balance) |
||||
|
|
||||
|
### Seeded portfolio |
||||
|
|
||||
|
| Symbol | Type | Qty | Data Source | |
||||
|
| ------- | ---- | --- | ----------- | |
||||
|
| AAPL | BUY | 20 | YAHOO | |
||||
|
| MSFT | BUY | 10 | YAHOO | |
||||
|
| VOO | BUY | 20 | YAHOO | |
||||
|
| GOOGL | BUY | 8 | YAHOO | |
||||
|
| bitcoin | BUY | 0.5 | COINGECKO | |
||||
|
| MSFT | SELL | 3 | YAHOO | |
||||
|
| VOO | DIV | -- | YAHOO | |
||||
|
|
||||
|
### Access token vs auth token |
||||
|
|
||||
|
- **Security token** (`demo-token-2026`): permanent passphrase identifying the user. Enter in the chat UI or Ghostfolio sign-in dialog. |
||||
|
- **Auth token** (JWT): short-lived bearer token for API calls. Obtained by exchanging the security token: |
||||
|
|
||||
|
```bash |
||||
|
curl http://localhost:3333/api/v1/auth/anonymous/demo-token-2026 |
||||
|
# -> { "authToken": "eyJ..." } |
||||
|
``` |
||||
|
|
||||
|
### Sample CSVs for custom portfolios |
||||
|
|
||||
|
Located in `seed-data/` at the repo root: |
||||
|
|
||||
|
| File | Focus | Holdings | |
||||
|
| ---------------------- | ----------- | ---------------------------------------- | |
||||
|
| `stocks-portfolio.csv` | US equities | VOO, AAPL, MSFT, GOOGL, AMZN, NVDA, META | |
||||
|
| `crypto-portfolio.csv` | Crypto | BTC, ETH, SOL, LINK, UNI | |
||||
|
| `hybrid-portfolio.csv` | Mixed | Stocks + crypto | |
||||
|
|
||||
|
Import via Ghostfolio UI: Portfolio → Activities → Import (upload icon). |
||||
|
|
||||
|
## Key Files |
||||
|
|
||||
|
| Path | Purpose | |
||||
|
| ------------------------------------------------------------ | ---------------------------------------------------------------- | |
||||
|
| `apps/api/src/app/endpoints/agent/` | Agent module (controller, service, metrics) | |
||||
|
| `apps/api/src/app/endpoints/agent/tools/` | Tool definitions (6 tools) | |
||||
|
| `apps/api/src/app/endpoints/agent/verification/` | Output validation, hallucination check, confidence | |
||||
|
| `apps/api/src/app/endpoints/agent/agent-feedback.service.ts` | Feedback collection + summary | |
||||
|
| `apps/api/src/app/endpoints/agent/agent-metrics.service.ts` | In-memory metrics + Postgres logging | |
||||
|
| `apps/api/src/app/endpoints/agent/submit-feedback.dto.ts` | Feedback DTO with class-validator decorators | |
||||
|
| `apps/client/src/app/pages/agent/` | Angular chat UI with rich rendering | |
||||
|
| `apps/client/src/app/pages/agent/rendering/` | Marked extensions (allocation, chart, sparkline, metrics, pills) | |
||||
|
| `evals/golden/` | Golden eval set (19 cases) | |
||||
|
| `evals/scenarios/` | Scenario eval set (33 cases) | |
||||
|
| `evals/scorers/` | Scorers (GoldenCheck, ResponseQuality, Verification) | |
||||
|
| `evals/helpers.ts` | Eval utilities (SSE parser with tool results) | |
||||
|
| `seed-data/` | Sample CSVs for demo portfolios | |
||||
|
| `prisma/seed.mts` | Demo user + portfolio seed | |
||||
|
| `prisma/schema.prisma` | AgentChatLog, AgentFeedback models | |
||||
|
| `.github/workflows/golden-evals.yml` | CI workflow | |
||||
|
|
||||
|
## Quickstart |
||||
|
|
||||
|
### Local Dev |
||||
|
|
||||
|
```bash |
||||
|
# 1. Copy env and fill in values |
||||
|
cp .env.example .env |
||||
|
|
||||
|
# 2. Start infra |
||||
|
docker compose -f docker/docker-compose.dev.yml up -d |
||||
|
|
||||
|
# 3. Install deps + run migrations + seed demo user |
||||
|
npm install |
||||
|
npx prisma migrate deploy |
||||
|
npx prisma db seed |
||||
|
|
||||
|
# 4. Start server |
||||
|
npx nx serve api |
||||
|
|
||||
|
# 5. Chat UI (sign in first, then navigate to Agent) |
||||
|
open http://localhost:4200/agent |
||||
|
|
||||
|
# 6. Check metrics |
||||
|
curl http://localhost:3333/api/v1/auth/anonymous/demo-token-2026 |
||||
|
# Use the returned JWT: |
||||
|
curl -H "Authorization: Bearer <jwt>" http://localhost:3333/api/v1/agent/metrics?since=1h |
||||
|
``` |
||||
|
|
||||
|
### Evals |
||||
|
|
||||
|
```bash |
||||
|
# Golden set (requires ANTHROPIC_API_KEY + TEST_USER_ACCESS_TOKEN in env) |
||||
|
TEST_USER_ACCESS_TOKEN=demo-token-2026 \ |
||||
|
npx evalite run evals/golden/agent-golden.eval.ts |
||||
|
|
||||
|
# Scenarios |
||||
|
TEST_USER_ACCESS_TOKEN=demo-token-2026 \ |
||||
|
npx evalite run evals/scenarios/agent-scenarios.eval.ts |
||||
|
|
||||
|
# Against Render |
||||
|
API_BASE=https://ghostfolio-4eid.onrender.com \ |
||||
|
TEST_USER_ACCESS_TOKEN=demo-token-2026 \ |
||||
|
npx evalite run evals/golden/agent-golden.eval.ts |
||||
|
``` |
||||
|
|
||||
|
> **Note**: Without `API_BASE`, evals default to `http://localhost:3333`. |
||||
|
|
||||
|
--- |
||||
|
|
||||
|
## Validation Report |
||||
|
|
||||
|
Metrics snapshot (production, 1h window): |
||||
|
|
||||
|
- 43 chats tracked |
||||
|
- Avg latency: 6.5s, avg 1.7 steps, avg 2.5K tokens |
||||
|
- Tool usage: portfolio_analysis(9), market_data(9), transaction_history(7), portfolio_performance(5), holdings_lookup(3) |
||||
|
- Per-chat detail: requestId, userId, latency, steps, tools, tokens |
||||
|
|
||||
|
### MVP Checklist |
||||
|
|
||||
|
| # | Requirement | Status | Evidence | |
||||
|
| --- | --------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------- | |
||||
|
| 1 | Agent responds to natural language | PASS | All 6 tools return coherent natural language responses | |
||||
|
| 2 | 3+ functional tools | PASS | 6 tools: symbol_search, portfolio_analysis, portfolio_performance, holdings_lookup, market_data, transaction_history | |
||||
|
| 3 | Tool calls execute + structured results | PASS | Tools return tables, dollar amounts, percentages | |
||||
|
| 4 | Agent synthesizes tool results | PASS | Combines tool data into markdown tables, summaries, key takeaways | |
||||
|
| 5 | Conversation history across turns | PASS | "What is its current price?" correctly resolved to VOO from prior turn | |
||||
|
| 6 | Basic error handling | PASS | 401 on bad auth, tool errors caught, clean 500s | |
||||
|
| 7 | Domain-specific verification | PASS | Rejects trades ("read-only"), rejects advice, rejects role-play | |
||||
|
| 8 | 5+ eval test cases | PASS | 19 golden (100%) + 33 scenarios = 52 total | |
||||
|
| 9 | Deployed + publicly accessible | PASS | https://ghostfolio-4eid.onrender.com, health OK | |
||||
@ -0,0 +1,73 @@ |
|||||
|
import { PrismaService } from '@ghostfolio/api/services/prisma/prisma.service'; |
||||
|
|
||||
|
import { Injectable, Logger } from '@nestjs/common'; |
||||
|
|
||||
|
@Injectable() |
||||
|
export class AgentFeedbackService { |
||||
|
private readonly logger = new Logger(AgentFeedbackService.name); |
||||
|
|
||||
|
public constructor(private readonly prismaService: PrismaService) {} |
||||
|
|
||||
|
public async submit({ |
||||
|
requestId, |
||||
|
userId, |
||||
|
rating, |
||||
|
comment |
||||
|
}: { |
||||
|
requestId: string; |
||||
|
userId: string; |
||||
|
rating: number; |
||||
|
comment?: string; |
||||
|
}) { |
||||
|
const feedback = await this.prismaService.agentFeedback.upsert({ |
||||
|
where: { requestId_userId: { requestId, userId } }, |
||||
|
create: { requestId, userId, rating, comment }, |
||||
|
update: { rating, comment } |
||||
|
}); |
||||
|
|
||||
|
this.logger.log( |
||||
|
JSON.stringify({ |
||||
|
event: 'feedback_submitted', |
||||
|
requestId, |
||||
|
userId, |
||||
|
rating |
||||
|
}) |
||||
|
); |
||||
|
|
||||
|
return feedback; |
||||
|
} |
||||
|
|
||||
|
public async getSummary(sinceMs?: number) { |
||||
|
const where = sinceMs |
||||
|
? { createdAt: { gte: new Date(Date.now() - sinceMs) } } |
||||
|
: {}; |
||||
|
|
||||
|
const [total, positive, negative, recent] = await Promise.all([ |
||||
|
this.prismaService.agentFeedback.count({ where }), |
||||
|
this.prismaService.agentFeedback.count({ |
||||
|
where: { ...where, rating: 1 } |
||||
|
}), |
||||
|
this.prismaService.agentFeedback.count({ |
||||
|
where: { ...where, rating: -1 } |
||||
|
}), |
||||
|
this.prismaService.agentFeedback.findMany({ |
||||
|
where: { ...where, comment: { not: null } }, |
||||
|
orderBy: { createdAt: 'desc' }, |
||||
|
take: 10, |
||||
|
select: { |
||||
|
rating: true, |
||||
|
comment: true, |
||||
|
createdAt: true |
||||
|
} |
||||
|
}) |
||||
|
]); |
||||
|
|
||||
|
return { |
||||
|
total, |
||||
|
positive, |
||||
|
negative, |
||||
|
satisfactionRate: total > 0 ? Math.round((positive / total) * 100) : 0, |
||||
|
recentComments: recent |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,168 @@ |
|||||
|
import { PrismaService } from '@ghostfolio/api/services/prisma/prisma.service'; |
||||
|
|
||||
|
import { Injectable, Logger } from '@nestjs/common'; |
||||
|
import type { Prisma } from '@prisma/client'; |
||||
|
|
||||
|
export interface ChatMetric { |
||||
|
requestId: string; |
||||
|
userId: string; |
||||
|
modelId?: string; |
||||
|
latencyMs: number; |
||||
|
totalSteps: number; |
||||
|
toolsUsed: string[]; |
||||
|
promptTokens: number; |
||||
|
completionTokens: number; |
||||
|
totalTokens: number; |
||||
|
errorOccurred?: boolean; |
||||
|
errorMessage?: string; |
||||
|
verificationScore?: number; |
||||
|
verificationResult?: Record<string, unknown>; |
||||
|
timestamp: number; |
||||
|
} |
||||
|
|
||||
|
// Anthropic pricing per million tokens (USD)
|
||||
|
const COST_PER_MTOK: Record<string, { input: number; output: number }> = { |
||||
|
'claude-sonnet-4-6': { input: 3, output: 15 }, |
||||
|
'claude-haiku-4-5-20251001': { input: 1, output: 5 }, |
||||
|
'claude-opus-4-6': { input: 15, output: 75 } |
||||
|
}; |
||||
|
const DEFAULT_COST = COST_PER_MTOK['claude-sonnet-4-6']; |
||||
|
|
||||
|
export function estimateCostUsd( |
||||
|
promptTokens: number, |
||||
|
completionTokens: number, |
||||
|
modelId?: string |
||||
|
): number { |
||||
|
const rates = (modelId && COST_PER_MTOK[modelId]) || DEFAULT_COST; |
||||
|
return ( |
||||
|
(promptTokens * rates.input + completionTokens * rates.output) / 1_000_000 |
||||
|
); |
||||
|
} |
||||
|
|
||||
|
@Injectable() |
||||
|
export class AgentMetricsService { |
||||
|
private readonly logger = new Logger(AgentMetricsService.name); |
||||
|
private readonly metrics: ChatMetric[] = []; |
||||
|
private readonly maxHistory = 1000; |
||||
|
|
||||
|
public constructor(private readonly prismaService: PrismaService) {} |
||||
|
|
||||
|
public record(metric: ChatMetric) { |
||||
|
const sanitized = { |
||||
|
...metric, |
||||
|
errorMessage: metric.errorMessage |
||||
|
? this.sanitizeError(metric.errorMessage) |
||||
|
: undefined |
||||
|
}; |
||||
|
|
||||
|
this.metrics.push(sanitized); |
||||
|
|
||||
|
if (this.metrics.length > this.maxHistory) { |
||||
|
this.metrics.splice(0, this.metrics.length - this.maxHistory); |
||||
|
} |
||||
|
|
||||
|
// Persist to Postgres (fire-and-forget)
|
||||
|
this.persist(sanitized).catch((error) => { |
||||
|
this.logger.warn(`Failed to persist metric: ${error.message}`); |
||||
|
}); |
||||
|
} |
||||
|
|
||||
|
private sanitizeError(message: string): string { |
||||
|
return message |
||||
|
.replace(/postgresql?:\/\/[^\s]*/gi, '[DB_URL_REDACTED]') |
||||
|
.replace(/redis:\/\/[^\s]*/gi, '[REDIS_URL_REDACTED]') |
||||
|
.replace(/sk-ant-[^\s]*/gi, '[API_KEY_REDACTED]') |
||||
|
.replace(/sk-proj-[^\s]*/gi, '[API_KEY_REDACTED]') |
||||
|
.substring(0, 500); |
||||
|
} |
||||
|
|
||||
|
private async persist(metric: ChatMetric) { |
||||
|
await this.prismaService.agentChatLog.create({ |
||||
|
data: { |
||||
|
requestId: metric.requestId, |
||||
|
userId: metric.userId, |
||||
|
modelId: metric.modelId ?? null, |
||||
|
latencyMs: metric.latencyMs, |
||||
|
totalSteps: metric.totalSteps, |
||||
|
toolsUsed: metric.toolsUsed, |
||||
|
promptTokens: metric.promptTokens, |
||||
|
completionTokens: metric.completionTokens, |
||||
|
totalTokens: metric.totalTokens, |
||||
|
errorOccurred: metric.errorOccurred ?? false, |
||||
|
errorMessage: metric.errorMessage ?? null, |
||||
|
verificationScore: metric.verificationScore ?? null, |
||||
|
verificationResult: |
||||
|
(metric.verificationResult as Prisma.InputJsonValue) ?? undefined |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
|
|
||||
|
public getSummary(sinceMs?: number) { |
||||
|
const cutoff = sinceMs ? Date.now() - sinceMs : 0; |
||||
|
const relevant = this.metrics.filter((m) => m.timestamp >= cutoff); |
||||
|
|
||||
|
if (relevant.length === 0) { |
||||
|
return { |
||||
|
totalChats: 0, |
||||
|
avgLatencyMs: 0, |
||||
|
avgSteps: 0, |
||||
|
avgTokens: { prompt: 0, completion: 0, total: 0 }, |
||||
|
cost: { totalUsd: 0, avgPerChatUsd: 0 }, |
||||
|
toolUsage: {} as Record<string, number>, |
||||
|
uniqueUsers: 0, |
||||
|
since: cutoff ? new Date(cutoff).toISOString() : 'all_time' |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
const toolUsage: Record<string, number> = {}; |
||||
|
let totalLatency = 0; |
||||
|
let totalSteps = 0; |
||||
|
let promptTokens = 0; |
||||
|
let completionTokens = 0; |
||||
|
let totalTokens = 0; |
||||
|
let totalCostUsd = 0; |
||||
|
const users = new Set<string>(); |
||||
|
|
||||
|
for (const m of relevant) { |
||||
|
totalLatency += m.latencyMs; |
||||
|
totalSteps += m.totalSteps; |
||||
|
promptTokens += m.promptTokens; |
||||
|
completionTokens += m.completionTokens; |
||||
|
totalTokens += m.totalTokens; |
||||
|
totalCostUsd += estimateCostUsd( |
||||
|
m.promptTokens, |
||||
|
m.completionTokens, |
||||
|
m.modelId |
||||
|
); |
||||
|
users.add(m.userId); |
||||
|
|
||||
|
for (const tool of m.toolsUsed) { |
||||
|
toolUsage[tool] = (toolUsage[tool] || 0) + 1; |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
const n = relevant.length; |
||||
|
|
||||
|
return { |
||||
|
totalChats: n, |
||||
|
avgLatencyMs: Math.round(totalLatency / n), |
||||
|
avgSteps: +(totalSteps / n).toFixed(1), |
||||
|
avgTokens: { |
||||
|
prompt: Math.round(promptTokens / n), |
||||
|
completion: Math.round(completionTokens / n), |
||||
|
total: Math.round(totalTokens / n) |
||||
|
}, |
||||
|
cost: { |
||||
|
totalUsd: +totalCostUsd.toFixed(4), |
||||
|
avgPerChatUsd: +(totalCostUsd / n).toFixed(4) |
||||
|
}, |
||||
|
toolUsage, |
||||
|
uniqueUsers: users.size, |
||||
|
since: cutoff ? new Date(cutoff).toISOString() : 'all_time' |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
public getRecent(count = 20): ChatMetric[] { |
||||
|
return this.metrics.slice(-count); |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,180 @@ |
|||||
|
import { HasPermission } from '@ghostfolio/api/decorators/has-permission.decorator'; |
||||
|
import { HasPermissionGuard } from '@ghostfolio/api/guards/has-permission.guard'; |
||||
|
import { PrismaService } from '@ghostfolio/api/services/prisma/prisma.service'; |
||||
|
import { permissions } from '@ghostfolio/common/permissions'; |
||||
|
import type { RequestWithUser } from '@ghostfolio/common/types'; |
||||
|
|
||||
|
import { |
||||
|
Body, |
||||
|
Controller, |
||||
|
Get, |
||||
|
HttpStatus, |
||||
|
Inject, |
||||
|
Logger, |
||||
|
NotFoundException, |
||||
|
Param, |
||||
|
Post, |
||||
|
Query, |
||||
|
Res, |
||||
|
UseGuards |
||||
|
} from '@nestjs/common'; |
||||
|
import { REQUEST } from '@nestjs/core'; |
||||
|
import { AuthGuard } from '@nestjs/passport'; |
||||
|
import { pipeAgentUIStreamToResponse, type UIMessage } from 'ai'; |
||||
|
import type { Response } from 'express'; |
||||
|
|
||||
|
import { AgentFeedbackService } from './agent-feedback.service'; |
||||
|
import { AgentMetricsService } from './agent-metrics.service'; |
||||
|
import { AgentService } from './agent.service'; |
||||
|
import { SubmitFeedbackDto } from './submit-feedback.dto'; |
||||
|
|
||||
|
@Controller('agent') |
||||
|
export class AgentController { |
||||
|
private readonly logger = new Logger(AgentController.name); |
||||
|
|
||||
|
public constructor( |
||||
|
private readonly agentFeedbackService: AgentFeedbackService, |
||||
|
private readonly agentMetricsService: AgentMetricsService, |
||||
|
private readonly agentService: AgentService, |
||||
|
private readonly prismaService: PrismaService, |
||||
|
@Inject(REQUEST) private readonly request: RequestWithUser |
||||
|
) {} |
||||
|
|
||||
|
@Post('chat') |
||||
|
@HasPermission(permissions.readAiPrompt) |
||||
|
@UseGuards(AuthGuard('jwt'), HasPermissionGuard) |
||||
|
public async chat( |
||||
|
@Body() |
||||
|
body: { |
||||
|
messages: UIMessage[]; |
||||
|
toolHistory?: string[]; |
||||
|
model?: string; |
||||
|
approvedActions?: string[]; |
||||
|
}, |
||||
|
@Res() res: Response |
||||
|
) { |
||||
|
try { |
||||
|
const { agent, requestId } = await this.agentService.chat({ |
||||
|
approvedActions: body.approvedActions, |
||||
|
messages: body.messages, |
||||
|
toolHistory: body.toolHistory, |
||||
|
model: body.model, |
||||
|
userId: this.request.user.id |
||||
|
}); |
||||
|
|
||||
|
await pipeAgentUIStreamToResponse({ |
||||
|
response: res, |
||||
|
agent, |
||||
|
uiMessages: body.messages, |
||||
|
messageMetadata: ({ part }) => { |
||||
|
if (part.type === 'finish') { |
||||
|
return { requestId }; |
||||
|
} |
||||
|
if (part.type === 'finish-step') { |
||||
|
return { stepFinish: true, finishReason: part.finishReason }; |
||||
|
} |
||||
|
return undefined; |
||||
|
} |
||||
|
}); |
||||
|
} catch (error) { |
||||
|
const message = error instanceof Error ? error.message : 'Unknown error'; |
||||
|
|
||||
|
this.logger.error( |
||||
|
`Chat failed: ${message}`, |
||||
|
error instanceof Error ? error.stack : undefined |
||||
|
); |
||||
|
|
||||
|
this.agentMetricsService.record({ |
||||
|
requestId: 'error-' + Date.now(), |
||||
|
userId: this.request.user.id, |
||||
|
latencyMs: 0, |
||||
|
totalSteps: 0, |
||||
|
toolsUsed: [], |
||||
|
promptTokens: 0, |
||||
|
completionTokens: 0, |
||||
|
totalTokens: 0, |
||||
|
errorOccurred: true, |
||||
|
errorMessage: message, |
||||
|
timestamp: Date.now() |
||||
|
}); |
||||
|
|
||||
|
if (!res.headersSent) { |
||||
|
res.status(HttpStatus.INTERNAL_SERVER_ERROR).json({ |
||||
|
error: 'Agent chat failed' |
||||
|
}); |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
@Post('feedback') |
||||
|
@HasPermission(permissions.readAiPrompt) |
||||
|
@UseGuards(AuthGuard('jwt'), HasPermissionGuard) |
||||
|
public async submitFeedback(@Body() body: SubmitFeedbackDto) { |
||||
|
return this.agentFeedbackService.submit({ |
||||
|
requestId: body.requestId, |
||||
|
userId: this.request.user.id, |
||||
|
rating: body.rating, |
||||
|
comment: body.comment |
||||
|
}); |
||||
|
} |
||||
|
|
||||
|
@Get('verification/:requestId') |
||||
|
@HasPermission(permissions.readAiPrompt) |
||||
|
@UseGuards(AuthGuard('jwt'), HasPermissionGuard) |
||||
|
public async getVerification(@Param('requestId') requestId: string) { |
||||
|
const log = await this.prismaService.agentChatLog.findFirst({ |
||||
|
where: { requestId, userId: this.request.user.id }, |
||||
|
select: { |
||||
|
requestId: true, |
||||
|
verificationScore: true, |
||||
|
verificationResult: true, |
||||
|
latencyMs: true, |
||||
|
totalSteps: true, |
||||
|
toolsUsed: true, |
||||
|
promptTokens: true, |
||||
|
completionTokens: true, |
||||
|
totalTokens: true, |
||||
|
createdAt: true |
||||
|
} |
||||
|
}); |
||||
|
|
||||
|
if (!log) { |
||||
|
throw new NotFoundException(`No chat log found for ${requestId}`); |
||||
|
} |
||||
|
|
||||
|
return log; |
||||
|
} |
||||
|
|
||||
|
@Get('metrics') |
||||
|
@HasPermission(permissions.readAiPrompt) |
||||
|
@UseGuards(AuthGuard('jwt'), HasPermissionGuard) |
||||
|
public async getMetrics(@Query('since') since?: string) { |
||||
|
const sinceMs = since ? parseDuration(since) : undefined; |
||||
|
|
||||
|
const [summary, feedback] = await Promise.all([ |
||||
|
this.agentMetricsService.getSummary(sinceMs), |
||||
|
this.agentFeedbackService.getSummary(sinceMs) |
||||
|
]); |
||||
|
|
||||
|
return { |
||||
|
summary, |
||||
|
feedback, |
||||
|
recent: this.agentMetricsService.getRecent(10) |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
function parseDuration(input: string): number | undefined { |
||||
|
const match = /^(\d+)(m|h|d)$/.exec(input); |
||||
|
|
||||
|
if (!match) return undefined; |
||||
|
|
||||
|
const [, value, unit] = match; |
||||
|
const multipliers: Record<string, number> = { |
||||
|
m: 60_000, |
||||
|
h: 3_600_000, |
||||
|
d: 86_400_000 |
||||
|
}; |
||||
|
|
||||
|
return Number(value) * multipliers[unit]; |
||||
|
} |
||||
@ -0,0 +1,69 @@ |
|||||
|
import { AccountBalanceService } from '@ghostfolio/api/app/account-balance/account-balance.service'; |
||||
|
import { AccountService } from '@ghostfolio/api/app/account/account.service'; |
||||
|
import { WatchlistService } from '@ghostfolio/api/app/endpoints/watchlist/watchlist.service'; |
||||
|
import { OrderModule } from '@ghostfolio/api/app/order/order.module'; |
||||
|
import { PortfolioCalculatorFactory } from '@ghostfolio/api/app/portfolio/calculator/portfolio-calculator.factory'; |
||||
|
import { CurrentRateService } from '@ghostfolio/api/app/portfolio/current-rate.service'; |
||||
|
import { PortfolioService } from '@ghostfolio/api/app/portfolio/portfolio.service'; |
||||
|
import { RulesService } from '@ghostfolio/api/app/portfolio/rules.service'; |
||||
|
import { RedisCacheModule } from '@ghostfolio/api/app/redis-cache/redis-cache.module'; |
||||
|
import { UserModule } from '@ghostfolio/api/app/user/user.module'; |
||||
|
import { ApiModule } from '@ghostfolio/api/services/api/api.module'; |
||||
|
import { BenchmarkModule } from '@ghostfolio/api/services/benchmark/benchmark.module'; |
||||
|
import { ConfigurationModule } from '@ghostfolio/api/services/configuration/configuration.module'; |
||||
|
import { DataProviderModule } from '@ghostfolio/api/services/data-provider/data-provider.module'; |
||||
|
import { ExchangeRateDataModule } from '@ghostfolio/api/services/exchange-rate-data/exchange-rate-data.module'; |
||||
|
import { I18nModule } from '@ghostfolio/api/services/i18n/i18n.module'; |
||||
|
import { ImpersonationModule } from '@ghostfolio/api/services/impersonation/impersonation.module'; |
||||
|
import { MarketDataModule } from '@ghostfolio/api/services/market-data/market-data.module'; |
||||
|
import { MarketDataService } from '@ghostfolio/api/services/market-data/market-data.service'; |
||||
|
import { PrismaModule } from '@ghostfolio/api/services/prisma/prisma.module'; |
||||
|
import { PropertyModule } from '@ghostfolio/api/services/property/property.module'; |
||||
|
import { DataGatheringModule } from '@ghostfolio/api/services/queues/data-gathering/data-gathering.module'; |
||||
|
import { PortfolioSnapshotQueueModule } from '@ghostfolio/api/services/queues/portfolio-snapshot/portfolio-snapshot.module'; |
||||
|
import { SymbolProfileModule } from '@ghostfolio/api/services/symbol-profile/symbol-profile.module'; |
||||
|
import { TagModule } from '@ghostfolio/api/services/tag/tag.module'; |
||||
|
|
||||
|
import { Module } from '@nestjs/common'; |
||||
|
|
||||
|
import { AgentFeedbackService } from './agent-feedback.service'; |
||||
|
import { AgentMetricsService } from './agent-metrics.service'; |
||||
|
import { AgentController } from './agent.controller'; |
||||
|
import { AgentService } from './agent.service'; |
||||
|
|
||||
|
@Module({ |
||||
|
controllers: [AgentController], |
||||
|
imports: [ |
||||
|
ApiModule, |
||||
|
BenchmarkModule, |
||||
|
ConfigurationModule, |
||||
|
DataGatheringModule, |
||||
|
DataProviderModule, |
||||
|
ExchangeRateDataModule, |
||||
|
I18nModule, |
||||
|
ImpersonationModule, |
||||
|
MarketDataModule, |
||||
|
OrderModule, |
||||
|
PortfolioSnapshotQueueModule, |
||||
|
PrismaModule, |
||||
|
PropertyModule, |
||||
|
RedisCacheModule, |
||||
|
SymbolProfileModule, |
||||
|
TagModule, |
||||
|
UserModule |
||||
|
], |
||||
|
providers: [ |
||||
|
AccountBalanceService, |
||||
|
AccountService, |
||||
|
AgentFeedbackService, |
||||
|
AgentMetricsService, |
||||
|
AgentService, |
||||
|
CurrentRateService, |
||||
|
MarketDataService, |
||||
|
PortfolioCalculatorFactory, |
||||
|
PortfolioService, |
||||
|
RulesService, |
||||
|
WatchlistService |
||||
|
] |
||||
|
}) |
||||
|
export class AgentModule {} |
||||
@ -0,0 +1,318 @@ |
|||||
|
import { AccountBalanceService } from '@ghostfolio/api/app/account-balance/account-balance.service'; |
||||
|
import { AccountService } from '@ghostfolio/api/app/account/account.service'; |
||||
|
import { WatchlistService } from '@ghostfolio/api/app/endpoints/watchlist/watchlist.service'; |
||||
|
import { OrderService } from '@ghostfolio/api/app/order/order.service'; |
||||
|
import { PortfolioService } from '@ghostfolio/api/app/portfolio/portfolio.service'; |
||||
|
import { RedisCacheService } from '@ghostfolio/api/app/redis-cache/redis-cache.service'; |
||||
|
import { UserService } from '@ghostfolio/api/app/user/user.service'; |
||||
|
import { DataProviderService } from '@ghostfolio/api/services/data-provider/data-provider.service'; |
||||
|
import { PortfolioSnapshotService } from '@ghostfolio/api/services/queues/portfolio-snapshot/portfolio-snapshot.service'; |
||||
|
import { TagService } from '@ghostfolio/api/services/tag/tag.service'; |
||||
|
|
||||
|
import { createAnthropic } from '@ai-sdk/anthropic'; |
||||
|
import { Injectable, Logger } from '@nestjs/common'; |
||||
|
import { |
||||
|
ToolLoopAgent, |
||||
|
stepCountIs, |
||||
|
type ModelMessage, |
||||
|
type PrepareStepFunction, |
||||
|
type UIMessage |
||||
|
} from 'ai'; |
||||
|
import { randomUUID } from 'node:crypto'; |
||||
|
import { join } from 'node:path'; |
||||
|
|
||||
|
import { AgentMetricsService } from './agent-metrics.service'; |
||||
|
import { validateModelId } from './models'; |
||||
|
import { createPrepareStep, loadSkills } from './prepare-step'; |
||||
|
import { createAccountManageTool } from './tools/account-manage.tool'; |
||||
|
import { createActivityManageTool } from './tools/activity-manage.tool'; |
||||
|
import { createHoldingsLookupTool } from './tools/holdings.tool'; |
||||
|
import { createMarketDataTool } from './tools/market-data.tool'; |
||||
|
import { createPortfolioPerformanceTool } from './tools/performance.tool'; |
||||
|
import { createPortfolioAnalysisTool } from './tools/portfolio.tool'; |
||||
|
import { createSymbolSearchTool } from './tools/symbol-search.tool'; |
||||
|
import { createTagManageTool } from './tools/tag-manage.tool'; |
||||
|
import { createTransactionHistoryTool } from './tools/transactions.tool'; |
||||
|
import { createWatchlistManageTool } from './tools/watchlist-manage.tool'; |
||||
|
import { |
||||
|
checkHallucination, |
||||
|
computeConfidence, |
||||
|
validateOutput |
||||
|
} from './verification'; |
||||
|
|
||||
|
const BASE_INSTRUCTIONS = `Be extremely concise. Sacrifice grammar for the sake of concision.
|
||||
|
|
||||
|
You are a financial analysis assistant powered by Ghostfolio. You help users understand their investment portfolio through data-driven insights. |
||||
|
|
||||
|
RULES: |
||||
|
- You can read AND write portfolio data. You can create/update/delete accounts, transactions, watchlist items, and tags. |
||||
|
- Never provide investment advice. Always include "This is not financial advice" when making forward-looking statements. |
||||
|
- Only reference data returned by your tools. Never fabricate numbers or holdings. |
||||
|
- Do not narrate or announce tool calls. Execute tools directly, then respond with results. |
||||
|
- If a tool call fails, tell the user honestly rather than guessing. |
||||
|
- Be concise and data-focused. Use tables and bullet points for clarity. |
||||
|
- When presenting monetary values, use the user's base currency. |
||||
|
- When presenting percentages, round to 2 decimal places. |
||||
|
|
||||
|
FORMATTING: |
||||
|
- Never use emojis. |
||||
|
- Structure data responses with a clear markdown heading (e.g., "## Portfolio Performance (YTD)"). |
||||
|
- Present multi-item or comparative data in markdown tables. Use concise column headers. |
||||
|
- After data tables, include one sentence of insight or context. |
||||
|
- Use **bold** for key metrics mentioned inline. Never use ALL CAPS for emphasis. |
||||
|
- Format currency with commas and two decimals (e.g., $48,210.45). |
||||
|
- Round percentages to two decimal places. Prefix positive returns with +. |
||||
|
- Keep responses focused -- no filler, no disclaimers unless discussing forward-looking statements. |
||||
|
|
||||
|
RICH FORMATTING (use these custom fenced blocks when the data fits): |
||||
|
|
||||
|
1. Allocation breakdowns: use a 2-column markdown table with percentage values. |
||||
|
| Asset Class | Allocation | |
||||
|
|---|---| |
||||
|
| Equities | 65% | |
||||
|
|
||||
|
2. Key metric summaries (2-4 values): use \`\`\`metrics block. One "Label: Value: Delta" per line. Use "--" if no delta.
|
||||
|
\`\`\`metrics
|
||||
|
Net Worth: $85k: +4.2% |
||||
|
Div. Yield: 3.1%: -- |
||||
|
\`\`\` |
||||
|
|
||||
|
3. Follow-up suggestions: ALWAYS end responses with a \`\`\`suggestions block (exactly 2 suggestions, one per line).
|
||||
|
\`\`\`suggestions
|
||||
|
Show my dividend history |
||||
|
Compare YTD vs last year |
||||
|
\`\`\` |
||||
|
|
||||
|
4. Sparklines for trends: \`\`\`sparkline with title and comma-separated values.
|
||||
|
5. Charts when user asks to visualize: \`\`\`chart-area or \`\`\`chart-bar with "Label: Value" per line.`; |
||||
|
|
||||
|
@Injectable() |
||||
|
export class AgentService { |
||||
|
private readonly logger = new Logger(AgentService.name); |
||||
|
private readonly skills: ReturnType<typeof loadSkills>; |
||||
|
|
||||
|
public constructor( |
||||
|
private readonly accountBalanceService: AccountBalanceService, |
||||
|
private readonly accountService: AccountService, |
||||
|
private readonly agentMetricsService: AgentMetricsService, |
||||
|
private readonly dataProviderService: DataProviderService, |
||||
|
private readonly orderService: OrderService, |
||||
|
private readonly portfolioService: PortfolioService, |
||||
|
private readonly portfolioSnapshotService: PortfolioSnapshotService, |
||||
|
private readonly redisCacheService: RedisCacheService, |
||||
|
private readonly tagService: TagService, |
||||
|
private readonly userService: UserService, |
||||
|
private readonly watchlistService: WatchlistService |
||||
|
) { |
||||
|
this.skills = loadSkills(join(__dirname, 'skills')); |
||||
|
} |
||||
|
|
||||
|
public async chat({ |
||||
|
approvedActions, |
||||
|
messages, |
||||
|
model, |
||||
|
toolHistory, |
||||
|
userId |
||||
|
}: { |
||||
|
approvedActions?: string[]; |
||||
|
messages: ModelMessage[] | UIMessage[]; |
||||
|
model?: string; |
||||
|
toolHistory?: string[]; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
const requestId = randomUUID(); |
||||
|
const startTime = Date.now(); |
||||
|
const modelId = validateModelId(model); |
||||
|
|
||||
|
this.logger.log( |
||||
|
JSON.stringify({ |
||||
|
event: 'chat_start', |
||||
|
requestId, |
||||
|
userId, |
||||
|
modelId, |
||||
|
messageCount: messages.length |
||||
|
}) |
||||
|
); |
||||
|
|
||||
|
const anthropic = createAnthropic(); |
||||
|
|
||||
|
const tools = this.buildTools(userId, approvedActions); |
||||
|
const prepareStep = createPrepareStep( |
||||
|
this.skills, |
||||
|
BASE_INSTRUCTIONS, |
||||
|
toolHistory |
||||
|
); |
||||
|
|
||||
|
const agent = new ToolLoopAgent({ |
||||
|
model: anthropic(modelId), |
||||
|
instructions: BASE_INSTRUCTIONS, |
||||
|
tools, |
||||
|
prepareStep: prepareStep as unknown as PrepareStepFunction<typeof tools>, |
||||
|
stopWhen: stepCountIs(10), |
||||
|
onStepFinish: ({ toolCalls, usage, finishReason, stepNumber }) => { |
||||
|
const toolNames = toolCalls.map((tc) => tc.toolName); |
||||
|
this.logger.log( |
||||
|
JSON.stringify({ |
||||
|
event: 'step_finish', |
||||
|
requestId, |
||||
|
userId, |
||||
|
step: stepNumber, |
||||
|
finishReason, |
||||
|
toolsCalled: toolNames.length > 0 ? toolNames : undefined, |
||||
|
tokens: usage |
||||
|
}) |
||||
|
); |
||||
|
}, |
||||
|
onFinish: ({ steps, usage, text }) => { |
||||
|
const latencyMs = Date.now() - startTime; |
||||
|
const allTools = steps.flatMap((s) => |
||||
|
s.toolCalls.map((tc) => tc.toolName) |
||||
|
); |
||||
|
const uniqueTools = [...new Set(allTools)]; |
||||
|
|
||||
|
// Run verification
|
||||
|
const toolResults = steps.flatMap((s) => |
||||
|
s.toolResults.map((tr: any) => ({ |
||||
|
toolName: tr.toolName as string, |
||||
|
result: tr.output |
||||
|
})) |
||||
|
); |
||||
|
const toolErrors = steps.flatMap((s) => |
||||
|
s.toolResults.filter((tr: any) => { |
||||
|
const out = tr.output; |
||||
|
return out && typeof out === 'object' && 'error' in out; |
||||
|
}) |
||||
|
); |
||||
|
|
||||
|
const validation = validateOutput({ |
||||
|
text, |
||||
|
toolCalls: allTools |
||||
|
}); |
||||
|
const hallucination = checkHallucination({ |
||||
|
text, |
||||
|
toolResults |
||||
|
}); |
||||
|
const confidence = computeConfidence({ |
||||
|
toolCallCount: allTools.length, |
||||
|
toolErrorCount: toolErrors.length, |
||||
|
stepCount: steps.length, |
||||
|
maxSteps: 10, |
||||
|
validation, |
||||
|
hallucination |
||||
|
}); |
||||
|
|
||||
|
this.logger.log( |
||||
|
JSON.stringify({ |
||||
|
event: 'verification', |
||||
|
requestId, |
||||
|
userId, |
||||
|
confidence: confidence.score, |
||||
|
validationIssues: validation.issues, |
||||
|
hallucinationIssues: hallucination.issues |
||||
|
}) |
||||
|
); |
||||
|
|
||||
|
this.logger.log( |
||||
|
JSON.stringify({ |
||||
|
event: 'chat_complete', |
||||
|
requestId, |
||||
|
userId, |
||||
|
latencyMs, |
||||
|
totalSteps: steps.length, |
||||
|
totalTokens: usage, |
||||
|
toolsUsed: uniqueTools |
||||
|
}) |
||||
|
); |
||||
|
|
||||
|
this.agentMetricsService.record({ |
||||
|
requestId, |
||||
|
userId, |
||||
|
modelId, |
||||
|
latencyMs, |
||||
|
totalSteps: steps.length, |
||||
|
toolsUsed: uniqueTools, |
||||
|
promptTokens: |
||||
|
(usage as any).promptTokens ?? (usage as any).inputTokens ?? 0, |
||||
|
completionTokens: |
||||
|
(usage as any).completionTokens ?? (usage as any).outputTokens ?? 0, |
||||
|
totalTokens: usage.totalTokens ?? 0, |
||||
|
timestamp: Date.now(), |
||||
|
verificationScore: confidence.score, |
||||
|
verificationResult: { |
||||
|
confidence: confidence.breakdown, |
||||
|
validation: { |
||||
|
valid: validation.valid, |
||||
|
score: validation.score, |
||||
|
issues: validation.issues |
||||
|
}, |
||||
|
hallucination: { |
||||
|
clean: hallucination.clean, |
||||
|
score: hallucination.score, |
||||
|
issues: hallucination.issues |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
|
}); |
||||
|
|
||||
|
return { agent, requestId }; |
||||
|
} |
||||
|
|
||||
|
private buildTools(userId: string, approvedActions?: string[]) { |
||||
|
return { |
||||
|
account_manage: createAccountManageTool({ |
||||
|
accountService: this.accountService, |
||||
|
approvedActions, |
||||
|
portfolioSnapshotService: this.portfolioSnapshotService, |
||||
|
redisCacheService: this.redisCacheService, |
||||
|
userService: this.userService, |
||||
|
userId |
||||
|
}), |
||||
|
activity_manage: createActivityManageTool({ |
||||
|
accountService: this.accountService, |
||||
|
approvedActions, |
||||
|
dataProviderService: this.dataProviderService, |
||||
|
orderService: this.orderService, |
||||
|
portfolioSnapshotService: this.portfolioSnapshotService, |
||||
|
redisCacheService: this.redisCacheService, |
||||
|
userService: this.userService, |
||||
|
userId |
||||
|
}), |
||||
|
portfolio_analysis: createPortfolioAnalysisTool({ |
||||
|
portfolioService: this.portfolioService, |
||||
|
userId |
||||
|
}), |
||||
|
portfolio_performance: createPortfolioPerformanceTool({ |
||||
|
portfolioService: this.portfolioService, |
||||
|
userId |
||||
|
}), |
||||
|
holdings_lookup: createHoldingsLookupTool({ |
||||
|
portfolioService: this.portfolioService, |
||||
|
userId |
||||
|
}), |
||||
|
market_data: createMarketDataTool({ |
||||
|
dataProviderService: this.dataProviderService |
||||
|
}), |
||||
|
symbol_search: createSymbolSearchTool({ |
||||
|
dataProviderService: this.dataProviderService, |
||||
|
userService: this.userService, |
||||
|
userId |
||||
|
}), |
||||
|
tag_manage: createTagManageTool({ |
||||
|
tagService: this.tagService, |
||||
|
userId |
||||
|
}), |
||||
|
transaction_history: createTransactionHistoryTool({ |
||||
|
accountBalanceService: this.accountBalanceService, |
||||
|
accountService: this.accountService, |
||||
|
orderService: this.orderService, |
||||
|
userService: this.userService, |
||||
|
userId |
||||
|
}), |
||||
|
watchlist_manage: createWatchlistManageTool({ |
||||
|
watchlistService: this.watchlistService, |
||||
|
userId |
||||
|
}) |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
@ -0,0 +1,69 @@ |
|||||
|
import { RedisCacheService } from '@ghostfolio/api/app/redis-cache/redis-cache.service'; |
||||
|
import { UserService } from '@ghostfolio/api/app/user/user.service'; |
||||
|
import { PortfolioSnapshotService } from '@ghostfolio/api/services/queues/portfolio-snapshot/portfolio-snapshot.service'; |
||||
|
import { |
||||
|
DEFAULT_CURRENCY, |
||||
|
PORTFOLIO_SNAPSHOT_COMPUTATION_QUEUE_PRIORITY_HIGH, |
||||
|
PORTFOLIO_SNAPSHOT_PROCESS_JOB_NAME, |
||||
|
PORTFOLIO_SNAPSHOT_PROCESS_JOB_OPTIONS |
||||
|
} from '@ghostfolio/common/config'; |
||||
|
import { PerformanceCalculationType } from '@ghostfolio/common/types/performance-calculation-type.type'; |
||||
|
|
||||
|
const SNAPSHOT_TIMEOUT_MS = 30_000; |
||||
|
|
||||
|
export async function warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}: { |
||||
|
portfolioSnapshotService: PortfolioSnapshotService; |
||||
|
redisCacheService: RedisCacheService; |
||||
|
userService: UserService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
// 1. Clear stale snapshots
|
||||
|
await redisCacheService.removePortfolioSnapshotsByUserId({ userId }); |
||||
|
|
||||
|
// 2. Drain any in-flight stale job
|
||||
|
const existingJob = await portfolioSnapshotService.getJob(userId); |
||||
|
|
||||
|
if (existingJob) { |
||||
|
try { |
||||
|
await existingJob.finished(); |
||||
|
} catch {} |
||||
|
|
||||
|
await redisCacheService.removePortfolioSnapshotsByUserId({ userId }); |
||||
|
} |
||||
|
|
||||
|
// 3. Fetch user settings for job params
|
||||
|
const user = await userService.user({ id: userId }); |
||||
|
const userCurrency = |
||||
|
user?.settings?.settings?.baseCurrency ?? DEFAULT_CURRENCY; |
||||
|
const calculationType = |
||||
|
user?.settings?.settings?.performanceCalculationType ?? |
||||
|
PerformanceCalculationType.TWR; |
||||
|
|
||||
|
// 4. Enqueue fresh computation (use return value directly — avoids race
|
||||
|
// where getJob() returns null after removeOnComplete deletes the job)
|
||||
|
const job = await portfolioSnapshotService.addJobToQueue({ |
||||
|
data: { calculationType, filters: [], userCurrency, userId }, |
||||
|
name: PORTFOLIO_SNAPSHOT_PROCESS_JOB_NAME, |
||||
|
opts: { |
||||
|
...PORTFOLIO_SNAPSHOT_PROCESS_JOB_OPTIONS, |
||||
|
jobId: userId, |
||||
|
priority: PORTFOLIO_SNAPSHOT_COMPUTATION_QUEUE_PRIORITY_HIGH |
||||
|
} |
||||
|
}); |
||||
|
|
||||
|
// 5. Await with timeout — don't block forever if computation hangs
|
||||
|
await Promise.race([ |
||||
|
job.finished(), |
||||
|
new Promise((_, reject) => |
||||
|
setTimeout( |
||||
|
() => reject(new Error('Snapshot computation timed out')), |
||||
|
SNAPSHOT_TIMEOUT_MS |
||||
|
) |
||||
|
) |
||||
|
]); |
||||
|
} |
||||
@ -0,0 +1,19 @@ |
|||||
|
export const ANTHROPIC_MODELS = [ |
||||
|
{ id: 'claude-haiku-4-5-20251001', label: 'Haiku 4.5', tier: 'Fast' }, |
||||
|
{ id: 'claude-sonnet-4-6', label: 'Sonnet 4.6', tier: 'Balanced' }, |
||||
|
{ id: 'claude-opus-4-6', label: 'Opus 4.6', tier: 'Best' } |
||||
|
] as const; |
||||
|
|
||||
|
export type AnthropicModelId = (typeof ANTHROPIC_MODELS)[number]['id']; |
||||
|
|
||||
|
export const DEFAULT_MODEL_ID: AnthropicModelId = 'claude-sonnet-4-6'; |
||||
|
|
||||
|
const VALID_IDS = new Set<string>(ANTHROPIC_MODELS.map((m) => m.id)); |
||||
|
|
||||
|
export function validateModelId(id?: string): AnthropicModelId { |
||||
|
if (id && VALID_IDS.has(id)) { |
||||
|
return id as AnthropicModelId; |
||||
|
} |
||||
|
|
||||
|
return DEFAULT_MODEL_ID; |
||||
|
} |
||||
@ -0,0 +1,369 @@ |
|||||
|
import type { ModelMessage, PrepareStepResult, StepResult, ToolSet } from 'ai'; |
||||
|
|
||||
|
import { |
||||
|
createPrepareStep, |
||||
|
getToolCallHistory, |
||||
|
getToolCallsFromMessages, |
||||
|
hasBeenCalled, |
||||
|
loadSkills |
||||
|
} from './prepare-step'; |
||||
|
|
||||
|
const BASE = 'You are a financial assistant.'; |
||||
|
|
||||
|
function callPrepareStep( |
||||
|
prepareStep: ReturnType<typeof createPrepareStep>, |
||||
|
opts: Parameters<ReturnType<typeof createPrepareStep>>[0] |
||||
|
): NonNullable<PrepareStepResult> { |
||||
|
return prepareStep(opts) as NonNullable<PrepareStepResult>; |
||||
|
} |
||||
|
|
||||
|
function makeStep(toolNames: string[]): StepResult<ToolSet> { |
||||
|
return { |
||||
|
toolCalls: toolNames.map((toolName) => ({ |
||||
|
type: 'tool-call' as const, |
||||
|
toolCallId: 'id', |
||||
|
toolName, |
||||
|
args: {} |
||||
|
})), |
||||
|
toolResults: [], |
||||
|
text: '', |
||||
|
reasoning: undefined, |
||||
|
reasoningDetails: [], |
||||
|
files: [], |
||||
|
sources: [], |
||||
|
finishReason: 'tool-calls', |
||||
|
usage: { promptTokens: 0, completionTokens: 0, totalTokens: 0 }, |
||||
|
warnings: [], |
||||
|
request: {}, |
||||
|
response: { |
||||
|
id: 'r1', |
||||
|
timestamp: new Date(), |
||||
|
modelId: 'test', |
||||
|
headers: {} |
||||
|
}, |
||||
|
providerMetadata: undefined, |
||||
|
experimental_providerMetadata: undefined, |
||||
|
stepType: 'initial', |
||||
|
isContinued: false |
||||
|
} as unknown as StepResult<ToolSet>; |
||||
|
} |
||||
|
|
||||
|
describe('getToolCallHistory', () => { |
||||
|
it('extracts tool names from steps', () => { |
||||
|
const steps = [ |
||||
|
makeStep(['portfolio_analysis']), |
||||
|
makeStep(['market_data', 'holdings_lookup']) |
||||
|
]; |
||||
|
expect(getToolCallHistory(steps)).toEqual([ |
||||
|
'portfolio_analysis', |
||||
|
'market_data', |
||||
|
'holdings_lookup' |
||||
|
]); |
||||
|
}); |
||||
|
|
||||
|
it('returns empty array for no steps', () => { |
||||
|
expect(getToolCallHistory([])).toEqual([]); |
||||
|
}); |
||||
|
}); |
||||
|
|
||||
|
describe('hasBeenCalled', () => { |
||||
|
it('returns true when tool is in history', () => { |
||||
|
expect( |
||||
|
hasBeenCalled(['market_data', 'holdings_lookup'], 'market_data') |
||||
|
).toBe(true); |
||||
|
}); |
||||
|
|
||||
|
it('returns false when tool is not in history', () => { |
||||
|
expect(hasBeenCalled(['market_data'], 'account_manage')).toBe(false); |
||||
|
}); |
||||
|
}); |
||||
|
|
||||
|
function makeMessage( |
||||
|
role: 'assistant' | 'user', |
||||
|
toolNames?: string[] |
||||
|
): ModelMessage { |
||||
|
if (role === 'user') { |
||||
|
return { role: 'user', content: [{ type: 'text', text: 'hello' }] }; |
||||
|
} |
||||
|
|
||||
|
const content: any[] = [{ type: 'text', text: 'response' }]; |
||||
|
|
||||
|
for (const toolName of toolNames ?? []) { |
||||
|
content.push({ |
||||
|
type: 'tool-call', |
||||
|
toolCallId: 'tc-1', |
||||
|
toolName, |
||||
|
args: {} |
||||
|
}); |
||||
|
} |
||||
|
|
||||
|
return { role: 'assistant', content }; |
||||
|
} |
||||
|
|
||||
|
describe('getToolCallsFromMessages', () => { |
||||
|
it('extracts tool names from assistant messages', () => { |
||||
|
const messages = [ |
||||
|
makeMessage('user'), |
||||
|
makeMessage('assistant', ['account_manage']), |
||||
|
makeMessage('user') |
||||
|
]; |
||||
|
expect(getToolCallsFromMessages(messages)).toEqual(['account_manage']); |
||||
|
}); |
||||
|
|
||||
|
it('ignores user messages', () => { |
||||
|
expect(getToolCallsFromMessages([makeMessage('user')])).toEqual([]); |
||||
|
}); |
||||
|
|
||||
|
it('returns empty for no messages', () => { |
||||
|
expect(getToolCallsFromMessages([])).toEqual([]); |
||||
|
}); |
||||
|
}); |
||||
|
|
||||
|
describe('loadSkills', () => { |
||||
|
it('loads skills from disk', () => { |
||||
|
const dir = __dirname + '/skills'; |
||||
|
const skills = loadSkills(dir); |
||||
|
|
||||
|
expect(skills.length).toBeGreaterThanOrEqual(2); |
||||
|
|
||||
|
const names = skills.map((s) => s.name); |
||||
|
expect(names).toContain('transaction'); |
||||
|
expect(names).toContain('market-data'); |
||||
|
}); |
||||
|
|
||||
|
it('returns empty array for non-existent dir', () => { |
||||
|
expect(loadSkills('/tmp/nonexistent-skills-dir')).toEqual([]); |
||||
|
}); |
||||
|
}); |
||||
|
|
||||
|
describe('createPrepareStep', () => { |
||||
|
const skills = loadSkills(__dirname + '/skills'); |
||||
|
const prepareStep = createPrepareStep(skills, BASE); |
||||
|
|
||||
|
it('gates write tools on step 0 for read-intent queries', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [], |
||||
|
stepNumber: 0, |
||||
|
model: {} as any, |
||||
|
messages: [ |
||||
|
{ |
||||
|
role: 'user', |
||||
|
content: [{ type: 'text', text: 'How is my portfolio doing?' }] |
||||
|
} |
||||
|
] as ModelMessage[], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
expect(result.activeTools).toContain('portfolio_analysis'); |
||||
|
expect(result.activeTools).toContain('market_data'); |
||||
|
expect(result.activeTools).not.toContain('activity_manage'); |
||||
|
expect(result.activeTools).not.toContain('account_manage'); |
||||
|
expect(result.activeTools).not.toContain('tag_manage'); |
||||
|
expect(result.activeTools).not.toContain('watchlist_manage'); |
||||
|
}); |
||||
|
|
||||
|
it('includes all tools on step 0 for write-intent queries', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [], |
||||
|
stepNumber: 0, |
||||
|
model: {} as any, |
||||
|
messages: [ |
||||
|
{ |
||||
|
role: 'user', |
||||
|
content: [{ type: 'text', text: 'Buy 10 AAPL at $185' }] |
||||
|
} |
||||
|
] as ModelMessage[], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
expect(result.activeTools).toContain('activity_manage'); |
||||
|
expect(result.activeTools).toContain('account_manage'); |
||||
|
expect(result.activeTools).toHaveLength(10); |
||||
|
}); |
||||
|
|
||||
|
it('preserves write intent from earlier messages in multi-turn conversation', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [], |
||||
|
stepNumber: 0, |
||||
|
model: {} as any, |
||||
|
messages: [ |
||||
|
{ |
||||
|
role: 'user', |
||||
|
content: [{ type: 'text', text: 'I want to buy some Bitcoin' }] |
||||
|
}, |
||||
|
{ |
||||
|
role: 'assistant', |
||||
|
content: [{ type: 'text', text: 'Bitcoin is at $66,906. How much?' }] |
||||
|
}, |
||||
|
{ role: 'user', content: [{ type: 'text', text: '0.25' }] } |
||||
|
] as ModelMessage[], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
expect(result.activeTools).toContain('activity_manage'); |
||||
|
expect(result.activeTools).toHaveLength(10); |
||||
|
}); |
||||
|
|
||||
|
it('includes all tools on step 1+ even for read-intent', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [makeStep(['portfolio_analysis'])], |
||||
|
stepNumber: 1, |
||||
|
model: {} as any, |
||||
|
messages: [ |
||||
|
{ |
||||
|
role: 'user', |
||||
|
content: [{ type: 'text', text: 'How is my portfolio doing?' }] |
||||
|
} |
||||
|
] as ModelMessage[], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
expect(result.activeTools).toContain('activity_manage'); |
||||
|
expect(result.activeTools).toHaveLength(10); |
||||
|
}); |
||||
|
|
||||
|
it('includes all tools on read-intent step 0 when priorToolHistory has write tool', () => { |
||||
|
const withHistory = createPrepareStep(skills, BASE, ['account_manage']); |
||||
|
const result = callPrepareStep(withHistory, { |
||||
|
steps: [], |
||||
|
stepNumber: 0, |
||||
|
model: {} as any, |
||||
|
messages: [ |
||||
|
{ |
||||
|
role: 'user', |
||||
|
content: [{ type: 'text', text: 'How is my portfolio doing?' }] |
||||
|
} |
||||
|
] as ModelMessage[], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
expect(result.activeTools).toContain('activity_manage'); |
||||
|
expect(result.activeTools).toHaveLength(10); |
||||
|
}); |
||||
|
|
||||
|
it('includes transaction skill in system prompt on step 0', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [], |
||||
|
stepNumber: 0, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).toContain(BASE); |
||||
|
expect(system).toContain('WRITE SAFETY RULES'); |
||||
|
expect(system).not.toContain('MARKET DATA LOOKUPS'); |
||||
|
}); |
||||
|
|
||||
|
it('includes market-data skill after market_data tool called', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [makeStep(['market_data'])], |
||||
|
stepNumber: 1, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).toContain('MARKET DATA LOOKUPS'); |
||||
|
}); |
||||
|
|
||||
|
it('includes market-data skill after symbol_search called', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [makeStep(['symbol_search'])], |
||||
|
stepNumber: 1, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).toContain('MARKET DATA LOOKUPS'); |
||||
|
}); |
||||
|
|
||||
|
it('returns base-only system when no market tools called', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: [makeStep(['portfolio_analysis'])], |
||||
|
stepNumber: 1, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).toContain(BASE); |
||||
|
expect(system).toContain('WRITE SAFETY RULES'); |
||||
|
expect(system).not.toContain('MARKET DATA LOOKUPS'); |
||||
|
}); |
||||
|
|
||||
|
it('includes all tools when priorToolHistory has write tool (even no messages)', () => { |
||||
|
const withHistory = createPrepareStep(skills, BASE, ['account_manage']); |
||||
|
const result = callPrepareStep(withHistory, { |
||||
|
steps: [], |
||||
|
stepNumber: 0, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
expect(result.activeTools).toContain('activity_manage'); |
||||
|
expect(result.activeTools).toHaveLength(10); |
||||
|
}); |
||||
|
|
||||
|
it('includes market-data skill when priorToolHistory contains market_data', () => { |
||||
|
const withHistory = createPrepareStep(skills, BASE, ['market_data']); |
||||
|
const result = callPrepareStep(withHistory, { |
||||
|
steps: [], |
||||
|
stepNumber: 0, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).toContain('MARKET DATA LOOKUPS'); |
||||
|
}); |
||||
|
|
||||
|
it('adds soft winddown nudge at step 5', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: Array(5).fill(makeStep([])), |
||||
|
stepNumber: 5, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).toContain('wrapping up'); |
||||
|
expect(system).not.toContain('Synthesize'); |
||||
|
}); |
||||
|
|
||||
|
it('adds hard winddown nudge at step 7', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: Array(7).fill(makeStep([])), |
||||
|
stepNumber: 7, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).toContain('Synthesize'); |
||||
|
expect(system).toContain('3 left'); |
||||
|
}); |
||||
|
|
||||
|
it('does not add winddown before step 5', () => { |
||||
|
const result = callPrepareStep(prepareStep, { |
||||
|
steps: Array(4).fill(makeStep([])), |
||||
|
stepNumber: 4, |
||||
|
model: {} as any, |
||||
|
messages: [], |
||||
|
experimental_context: undefined |
||||
|
}); |
||||
|
|
||||
|
const system = result.system as string; |
||||
|
expect(system).not.toContain('wrapping up'); |
||||
|
expect(system).not.toContain('Synthesize'); |
||||
|
}); |
||||
|
}); |
||||
@ -0,0 +1,248 @@ |
|||||
|
import type { |
||||
|
ModelMessage, |
||||
|
PrepareStepFunction, |
||||
|
StepResult, |
||||
|
ToolSet |
||||
|
} from 'ai'; |
||||
|
import { readFileSync, readdirSync } from 'node:fs'; |
||||
|
import { join } from 'node:path'; |
||||
|
|
||||
|
export interface Skill { |
||||
|
name: string; |
||||
|
description: string; |
||||
|
body: string; |
||||
|
} |
||||
|
|
||||
|
const ALL_TOOLS = [ |
||||
|
'portfolio_analysis', |
||||
|
'portfolio_performance', |
||||
|
'holdings_lookup', |
||||
|
'market_data', |
||||
|
'symbol_search', |
||||
|
'transaction_history', |
||||
|
'account_manage', |
||||
|
'tag_manage', |
||||
|
'watchlist_manage', |
||||
|
'activity_manage' |
||||
|
] as const; |
||||
|
|
||||
|
type ToolName = (typeof ALL_TOOLS)[number]; |
||||
|
|
||||
|
const READ_TOOLS: ToolName[] = [ |
||||
|
'portfolio_analysis', |
||||
|
'portfolio_performance', |
||||
|
'holdings_lookup', |
||||
|
'market_data', |
||||
|
'symbol_search', |
||||
|
'transaction_history' |
||||
|
]; |
||||
|
|
||||
|
const WRITE_TOOLS: ToolName[] = [ |
||||
|
'account_manage', |
||||
|
'activity_manage', |
||||
|
'tag_manage', |
||||
|
'watchlist_manage' |
||||
|
]; |
||||
|
|
||||
|
const WRITE_KEYWORDS = [ |
||||
|
// Action verbs
|
||||
|
'create', |
||||
|
'add', |
||||
|
'delete', |
||||
|
'remove', |
||||
|
'buy', |
||||
|
'sell', |
||||
|
'deposit', |
||||
|
'transfer', |
||||
|
'update', |
||||
|
'rename', |
||||
|
'record', |
||||
|
'log', |
||||
|
'withdraw', |
||||
|
'move', |
||||
|
// Domain nouns — queries referencing write-tool entities need those tools
|
||||
|
// even for read-only operations like "list my accounts"
|
||||
|
'account', |
||||
|
'watchlist', |
||||
|
'tag' |
||||
|
]; |
||||
|
|
||||
|
function classifyIntent(messages: ModelMessage[]): 'read' | 'write' { |
||||
|
const userMessages = messages.filter((m) => m.role === 'user'); |
||||
|
|
||||
|
if (userMessages.length === 0) return 'read'; |
||||
|
|
||||
|
for (const msg of userMessages) { |
||||
|
const text = |
||||
|
typeof msg.content === 'string' |
||||
|
? msg.content |
||||
|
: Array.isArray(msg.content) |
||||
|
? msg.content |
||||
|
.filter((p: any) => p.type === 'text') |
||||
|
.map((p: any) => p.text) |
||||
|
.join(' ') |
||||
|
: ''; |
||||
|
|
||||
|
if (WRITE_KEYWORDS.some((kw) => text.toLowerCase().includes(kw))) { |
||||
|
return 'write'; |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
return 'read'; |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Read SKILL.md files from a directory tree. |
||||
|
* Expects: dir/<skill-name>/SKILL.md with YAML frontmatter. |
||||
|
*/ |
||||
|
export function loadSkills(dir: string): Skill[] { |
||||
|
const skills: Skill[] = []; |
||||
|
|
||||
|
let entries: string[]; |
||||
|
|
||||
|
try { |
||||
|
entries = readdirSync(dir, { withFileTypes: true }) |
||||
|
.filter((d) => d.isDirectory()) |
||||
|
.map((d) => d.name); |
||||
|
} catch { |
||||
|
return skills; |
||||
|
} |
||||
|
|
||||
|
for (const name of entries) { |
||||
|
const filePath = join(dir, name, 'SKILL.md'); |
||||
|
|
||||
|
try { |
||||
|
const raw = readFileSync(filePath, 'utf-8'); |
||||
|
const parsed = parseFrontmatter(raw); |
||||
|
|
||||
|
if (parsed) { |
||||
|
skills.push(parsed); |
||||
|
} |
||||
|
} catch { |
||||
|
// skip missing files
|
||||
|
} |
||||
|
} |
||||
|
|
||||
|
return skills; |
||||
|
} |
||||
|
|
||||
|
function parseFrontmatter(raw: string): Skill | null { |
||||
|
const match = /^---\n([\s\S]*?)\n---\n([\s\S]*)$/.exec(raw); |
||||
|
|
||||
|
if (!match) { |
||||
|
return null; |
||||
|
} |
||||
|
|
||||
|
const [, frontmatter, body] = match; |
||||
|
const name = extractYamlValue(frontmatter, 'name'); |
||||
|
const description = extractYamlValue(frontmatter, 'description'); |
||||
|
|
||||
|
if (!name) { |
||||
|
return null; |
||||
|
} |
||||
|
|
||||
|
return { name, description: description ?? '', body: body.trim() }; |
||||
|
} |
||||
|
|
||||
|
function extractYamlValue(yaml: string, key: string): string | null { |
||||
|
const match = new RegExp(`^${key}:\\s*(.+)$`, 'm').exec(yaml); |
||||
|
|
||||
|
return match ? match[1].trim() : null; |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Extract tool names called in previous steps (current session). |
||||
|
*/ |
||||
|
export function getToolCallHistory(steps: StepResult<ToolSet>[]): string[] { |
||||
|
return steps.flatMap((s) => s.toolCalls.map((tc) => tc.toolName)); |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Extract tool names from conversation message history (prior turns). |
||||
|
*/ |
||||
|
export function getToolCallsFromMessages(messages: ModelMessage[]): string[] { |
||||
|
return messages.flatMap((m) => { |
||||
|
if (m.role !== 'assistant' || !Array.isArray(m.content)) { |
||||
|
return []; |
||||
|
} |
||||
|
|
||||
|
return m.content |
||||
|
.filter((part: any) => part.type === 'tool-call') |
||||
|
.map((part: any) => part.toolName); |
||||
|
}); |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Check if a specific tool has been called in history. |
||||
|
*/ |
||||
|
export function hasBeenCalled(history: string[], toolName: string): boolean { |
||||
|
return history.includes(toolName); |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Factory: returns a prepareStep callback that gates tools and composes system prompt. |
||||
|
*/ |
||||
|
export function createPrepareStep( |
||||
|
skills: Skill[], |
||||
|
baseInstructions: string, |
||||
|
priorToolHistory: string[] = [] |
||||
|
): PrepareStepFunction { |
||||
|
const transactionSkill = skills.find((s) => s.name === 'transaction'); |
||||
|
const marketDataSkill = skills.find((s) => s.name === 'market-data'); |
||||
|
|
||||
|
return ({ steps, messages }) => { |
||||
|
const history = [ |
||||
|
...priorToolHistory, |
||||
|
...getToolCallsFromMessages(messages), |
||||
|
...getToolCallHistory(steps) |
||||
|
]; |
||||
|
|
||||
|
// Gate write tools on step 0 for read-intent queries. Once a write tool
|
||||
|
// has been called (current or prior session), all tools stay available.
|
||||
|
const hasWriteHistory = history.some((t) => |
||||
|
WRITE_TOOLS.includes(t as ToolName) |
||||
|
); |
||||
|
const intent = classifyIntent(messages); |
||||
|
const activeTools: ToolName[] = |
||||
|
intent === 'read' && steps.length === 0 && !hasWriteHistory |
||||
|
? [...READ_TOOLS] |
||||
|
: [...ALL_TOOLS]; |
||||
|
|
||||
|
// Skill composition: append skill bodies based on step context
|
||||
|
const today = new Date().toISOString().split('T')[0]; |
||||
|
const systemParts: string[] = [baseInstructions, `Today is ${today}.`]; |
||||
|
|
||||
|
// Transaction skill: always loaded (write tools are always visible)
|
||||
|
if (transactionSkill) { |
||||
|
systemParts.push(transactionSkill.body); |
||||
|
} |
||||
|
|
||||
|
// Market data skill: loaded when symbol_search or market_data has been called
|
||||
|
const marketDataActive = |
||||
|
hasBeenCalled(history, 'symbol_search') || |
||||
|
hasBeenCalled(history, 'market_data'); |
||||
|
|
||||
|
if (marketDataSkill && marketDataActive) { |
||||
|
systemParts.push(marketDataSkill.body); |
||||
|
} |
||||
|
|
||||
|
// Step-aware winddown: nudge synthesis as step budget depletes
|
||||
|
const MAX_STEPS = 10; |
||||
|
const remaining = MAX_STEPS - steps.length; |
||||
|
|
||||
|
if (remaining <= 5 && remaining > 3) { |
||||
|
systemParts.push( |
||||
|
`${steps.length}/${MAX_STEPS} steps used. Start wrapping up unless more data needed.` |
||||
|
); |
||||
|
} else if (remaining <= 3 && remaining > 0) { |
||||
|
systemParts.push( |
||||
|
`${steps.length}/${MAX_STEPS} steps used (${remaining} left). Synthesize findings now. No new tool calls unless essential.` |
||||
|
); |
||||
|
} |
||||
|
|
||||
|
return { |
||||
|
activeTools, |
||||
|
system: systemParts.join('\n\n') |
||||
|
}; |
||||
|
}; |
||||
|
} |
||||
@ -0,0 +1,13 @@ |
|||||
|
--- |
||||
|
name: market-data |
||||
|
description: Data source rules for stock and crypto symbol resolution. |
||||
|
--- |
||||
|
|
||||
|
MARKET DATA LOOKUPS: |
||||
|
|
||||
|
- For stocks and ETFs: use dataSource "YAHOO" with uppercase ticker symbols (e.g. "AAPL", "TSLA", "MSFT"). |
||||
|
- For cryptocurrencies: use dataSource "COINGECKO" with the CoinGecko lowercase slug ID. Do NOT use ticker symbols like "BTC" or "STX" with CoinGecko -- use the full lowercase slug. |
||||
|
- Well-known CoinGecko slugs you can use directly: "bitcoin", "ethereum", "solana". |
||||
|
- For ANY other cryptocurrency, use the symbol_search tool first to find the correct CoinGecko slug. CoinGecko slugs are often non-obvious (e.g. "blockstack" for Stacks, "avalanche-2" for Avalanche, "matic-network" for Polygon). |
||||
|
- If symbol_search returns multiple matches, present the options to the user and let them choose before calling market_data. |
||||
|
- If unsure whether something is a crypto or stock, use symbol_search to find out. |
||||
@ -0,0 +1,12 @@ |
|||||
|
--- |
||||
|
name: transaction |
||||
|
description: Write safety rules for creating, updating, and deleting portfolio transactions and accounts. |
||||
|
--- |
||||
|
|
||||
|
WRITE SAFETY RULES: |
||||
|
|
||||
|
- Write tools have built-in approval gates. The system shows an approval card to the user before executing creates, deletes, transfers, and balance changes. Do NOT ask for text confirmation — just call the write tool directly and let the approval card handle it. |
||||
|
- For multi-step writes (e.g. "deposit then buy"), call each write tool in sequence. Each will show its own approval card. Do not ask the user to confirm the plan first. |
||||
|
- After any write action, briefly confirm what was done (e.g., "Created BUY order: 10 AAPL @ $185.00"). |
||||
|
- Never batch-delete without explicit user consent. |
||||
|
- If a prior write was blocked by a prerequisite (e.g. insufficient funds) and the user asks to resolve the prerequisite, execute both actions. The prior approval still applies. |
||||
@ -0,0 +1,14 @@ |
|||||
|
import { IsIn, IsOptional, IsString, MaxLength } from 'class-validator'; |
||||
|
|
||||
|
export class SubmitFeedbackDto { |
||||
|
@IsString() |
||||
|
requestId: string; |
||||
|
|
||||
|
@IsIn([-1, 1]) |
||||
|
rating: number; |
||||
|
|
||||
|
@IsString() |
||||
|
@IsOptional() |
||||
|
@MaxLength(1000) |
||||
|
comment?: string; |
||||
|
} |
||||
@ -0,0 +1,297 @@ |
|||||
|
import { AccountService } from '@ghostfolio/api/app/account/account.service'; |
||||
|
import { RedisCacheService } from '@ghostfolio/api/app/redis-cache/redis-cache.service'; |
||||
|
import { UserService } from '@ghostfolio/api/app/user/user.service'; |
||||
|
import { PortfolioSnapshotService } from '@ghostfolio/api/services/queues/portfolio-snapshot/portfolio-snapshot.service'; |
||||
|
|
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
import { warmPortfolioCache } from '../helpers/warm-portfolio-cache'; |
||||
|
|
||||
|
export function createAccountManageTool({ |
||||
|
accountService, |
||||
|
approvedActions, |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}: { |
||||
|
accountService: AccountService; |
||||
|
approvedActions?: string[]; |
||||
|
portfolioSnapshotService: PortfolioSnapshotService; |
||||
|
redisCacheService: RedisCacheService; |
||||
|
userService: UserService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Manage investment accounts (brokerages, banks, wallets). Use this to create new accounts, update account details, delete empty accounts, transfer cash between accounts, or list all accounts with balances. IMPORTANT: Deposits and withdrawals are performed by updating the account balance (action: "update"). Do NOT use activity_manage for deposits or withdrawals — just set the new balance here and it will automatically be tracked in transaction_history.', |
||||
|
needsApproval: |
||||
|
process.env.SKIP_APPROVAL === 'true' |
||||
|
? false |
||||
|
: (input) => { |
||||
|
if (input.action === 'list') return false; |
||||
|
const sig = `account_manage:${input.action}:${input.name ?? input.accountId ?? ''}`; |
||||
|
return !approvedActions?.includes(sig); |
||||
|
}, |
||||
|
inputSchema: z.object({ |
||||
|
action: z |
||||
|
.enum(['create', 'update', 'delete', 'transfer', 'list']) |
||||
|
.describe( |
||||
|
"Action to perform. 'create': new account. 'update': modify existing. 'delete': remove empty account. 'transfer': move cash between accounts. 'list': show all accounts." |
||||
|
), |
||||
|
name: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Account display name. Required for 'create'. E.g. 'Fidelity Brokerage', 'Coinbase'." |
||||
|
), |
||||
|
currency: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Account base currency as ISO 4217 code (USD, EUR, GBP). Required for 'create'." |
||||
|
), |
||||
|
balance: z |
||||
|
.number() |
||||
|
.min(0) |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Cash balance. For 'create': initial balance (default 0). For 'update': sets new absolute balance (e.g. to withdraw $5k from $40k, set balance to 35000). For 'transfer': amount to move." |
||||
|
), |
||||
|
accountId: z |
||||
|
.string() |
||||
|
.uuid() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Account ID. Required for 'update' and 'delete'. Get from 'list'." |
||||
|
), |
||||
|
accountIdFrom: z |
||||
|
.string() |
||||
|
.uuid() |
||||
|
.optional() |
||||
|
.describe("Source account ID for 'transfer'. Get from 'list'."), |
||||
|
accountIdTo: z |
||||
|
.string() |
||||
|
.uuid() |
||||
|
.optional() |
||||
|
.describe("Destination account ID for 'transfer'. Get from 'list'."), |
||||
|
comment: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe('Optional note about the account.'), |
||||
|
isExcluded: z |
||||
|
.boolean() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
'If true, exclude from portfolio calculations. Useful for test accounts.' |
||||
|
) |
||||
|
}), |
||||
|
execute: async (input) => { |
||||
|
try { |
||||
|
switch (input.action) { |
||||
|
case 'create': { |
||||
|
const account = await accountService.createAccount( |
||||
|
{ |
||||
|
balance: input.balance ?? 0, |
||||
|
comment: input.comment, |
||||
|
currency: input.currency, |
||||
|
isExcluded: input.isExcluded ?? false, |
||||
|
name: input.name, |
||||
|
user: { connect: { id: userId } } |
||||
|
}, |
||||
|
userId |
||||
|
); |
||||
|
|
||||
|
try { |
||||
|
await warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}); |
||||
|
} catch {} |
||||
|
|
||||
|
return { |
||||
|
id: account.id, |
||||
|
name: account.name, |
||||
|
currency: account.currency, |
||||
|
balance: input.balance ?? 0 |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
case 'update': { |
||||
|
const existing = await accountService.account({ |
||||
|
id_userId: { id: input.accountId, userId } |
||||
|
}); |
||||
|
|
||||
|
if (!existing) { |
||||
|
return { error: 'Account not found or does not belong to you' }; |
||||
|
} |
||||
|
|
||||
|
const data: Record<string, any> = { |
||||
|
id: input.accountId, |
||||
|
user: { connect: { id: userId } } |
||||
|
}; |
||||
|
|
||||
|
if (input.name !== undefined) { |
||||
|
data.name = input.name; |
||||
|
} |
||||
|
|
||||
|
if (input.currency !== undefined) { |
||||
|
data.currency = input.currency; |
||||
|
} |
||||
|
|
||||
|
if (input.balance !== undefined) { |
||||
|
data.balance = input.balance; |
||||
|
} |
||||
|
|
||||
|
if (input.comment !== undefined) { |
||||
|
data.comment = input.comment; |
||||
|
} |
||||
|
|
||||
|
if (input.isExcluded !== undefined) { |
||||
|
data.isExcluded = input.isExcluded; |
||||
|
} |
||||
|
|
||||
|
if (existing.platformId) { |
||||
|
data.platform = { connect: { id: existing.platformId } }; |
||||
|
} |
||||
|
|
||||
|
const account = await accountService.updateAccount( |
||||
|
{ |
||||
|
data, |
||||
|
where: { |
||||
|
id_userId: { id: input.accountId, userId } |
||||
|
} |
||||
|
}, |
||||
|
userId |
||||
|
); |
||||
|
|
||||
|
try { |
||||
|
await warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}); |
||||
|
} catch {} |
||||
|
|
||||
|
return { |
||||
|
id: account.id, |
||||
|
name: account.name, |
||||
|
currency: account.currency, |
||||
|
balance: account.balance |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
case 'delete': { |
||||
|
const existing = await accountService.accountWithActivities( |
||||
|
{ id_userId: { id: input.accountId, userId } }, |
||||
|
{ activities: true } |
||||
|
); |
||||
|
|
||||
|
if (!existing) { |
||||
|
return { error: 'Account not found or does not belong to you' }; |
||||
|
} |
||||
|
|
||||
|
if (existing.activities?.length > 0) { |
||||
|
return { |
||||
|
error: `Account "${existing.name}" has ${existing.activities.length} activities. Delete them first before deleting the account.` |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
await accountService.deleteAccount({ |
||||
|
id_userId: { id: input.accountId, userId } |
||||
|
}); |
||||
|
|
||||
|
try { |
||||
|
await warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}); |
||||
|
} catch {} |
||||
|
|
||||
|
return { deleted: true, name: existing.name }; |
||||
|
} |
||||
|
|
||||
|
case 'transfer': { |
||||
|
if (input.accountIdFrom === input.accountIdTo) { |
||||
|
return { error: 'Cannot transfer to the same account' }; |
||||
|
} |
||||
|
|
||||
|
const accounts = await accountService.getAccounts(userId); |
||||
|
const fromAccount = accounts.find( |
||||
|
(a) => a.id === input.accountIdFrom |
||||
|
); |
||||
|
const toAccount = accounts.find((a) => a.id === input.accountIdTo); |
||||
|
|
||||
|
if (!fromAccount) { |
||||
|
return { error: 'Source account not found' }; |
||||
|
} |
||||
|
|
||||
|
if (!toAccount) { |
||||
|
return { error: 'Destination account not found' }; |
||||
|
} |
||||
|
|
||||
|
if (fromAccount.balance < input.balance) { |
||||
|
return { |
||||
|
error: `Insufficient balance. ${fromAccount.name} has ${fromAccount.currency} ${fromAccount.balance}, but tried to transfer ${input.balance}.` |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
await accountService.updateAccountBalance({ |
||||
|
accountId: input.accountIdFrom, |
||||
|
amount: -input.balance, |
||||
|
currency: fromAccount.currency, |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
await accountService.updateAccountBalance({ |
||||
|
accountId: input.accountIdTo, |
||||
|
amount: input.balance, |
||||
|
currency: fromAccount.currency, |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
try { |
||||
|
await warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}); |
||||
|
} catch {} |
||||
|
|
||||
|
return { |
||||
|
from: fromAccount.name, |
||||
|
to: toAccount.name, |
||||
|
amount: input.balance, |
||||
|
currency: fromAccount.currency |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
case 'list': { |
||||
|
const accounts = await accountService.getAccounts(userId); |
||||
|
|
||||
|
return accounts.map((a: any) => ({ |
||||
|
id: a.id, |
||||
|
name: a.name, |
||||
|
currency: a.currency, |
||||
|
balance: a.balance, |
||||
|
platform: a.platform?.name, |
||||
|
isExcluded: a.isExcluded, |
||||
|
activitiesCount: a.activitiesCount |
||||
|
})); |
||||
|
} |
||||
|
} |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to ${input.action} account: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,409 @@ |
|||||
|
import { AccountService } from '@ghostfolio/api/app/account/account.service'; |
||||
|
import { OrderService } from '@ghostfolio/api/app/order/order.service'; |
||||
|
import { RedisCacheService } from '@ghostfolio/api/app/redis-cache/redis-cache.service'; |
||||
|
import { UserService } from '@ghostfolio/api/app/user/user.service'; |
||||
|
import { DataProviderService } from '@ghostfolio/api/services/data-provider/data-provider.service'; |
||||
|
import { PortfolioSnapshotService } from '@ghostfolio/api/services/queues/portfolio-snapshot/portfolio-snapshot.service'; |
||||
|
|
||||
|
import type { DataSource, Type as ActivityType } from '@prisma/client'; |
||||
|
import { tool } from 'ai'; |
||||
|
import { parseISO } from 'date-fns'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
import { warmPortfolioCache } from '../helpers/warm-portfolio-cache'; |
||||
|
|
||||
|
/** |
||||
|
* Auto-resolve a CoinGecko slug from a raw ticker symbol. |
||||
|
* CoinGecko slugs are always lowercase (e.g. "bitcoin", "stacks"). |
||||
|
* If the symbol is already lowercase, assume it's a valid slug. |
||||
|
*/ |
||||
|
async function resolveCoinGeckoSlug( |
||||
|
dataProviderService: DataProviderService, |
||||
|
user: any, |
||||
|
symbol: string |
||||
|
): Promise<string> { |
||||
|
if (symbol === symbol.toLowerCase()) { |
||||
|
return symbol; |
||||
|
} |
||||
|
|
||||
|
const { items } = await dataProviderService.search({ query: symbol, user }); |
||||
|
const match = items.find((item) => item.dataSource === 'COINGECKO'); |
||||
|
|
||||
|
return match?.symbol ?? symbol.toLowerCase(); |
||||
|
} |
||||
|
|
||||
|
const CRYPTO_KEYWORDS = ['crypto', 'coin', 'wallet', 'defi', 'token']; |
||||
|
const STOCK_KEYWORDS = ['stock', 'brokerage', 'equity', 'equities']; |
||||
|
const ETF_KEYWORDS = ['etf', 'index', 'fund']; |
||||
|
|
||||
|
function resolveAccount( |
||||
|
accounts: { id: string; name: string; activitiesCount?: number }[], |
||||
|
dataSource: string |
||||
|
): string | undefined { |
||||
|
if (accounts.length === 0) return undefined; |
||||
|
if (accounts.length === 1) return accounts[0].id; |
||||
|
|
||||
|
const keywords = |
||||
|
dataSource === 'COINGECKO' |
||||
|
? CRYPTO_KEYWORDS |
||||
|
: dataSource === 'YAHOO' |
||||
|
? [...STOCK_KEYWORDS, ...ETF_KEYWORDS] |
||||
|
: []; |
||||
|
|
||||
|
if (keywords.length > 0) { |
||||
|
const match = accounts.find((a) => |
||||
|
keywords.some((kw) => a.name.toLowerCase().includes(kw)) |
||||
|
); |
||||
|
|
||||
|
if (match) return match.id; |
||||
|
} |
||||
|
|
||||
|
// Fallback: account with most activities
|
||||
|
const sorted = [...accounts].sort( |
||||
|
(a, b) => (b.activitiesCount ?? 0) - (a.activitiesCount ?? 0) |
||||
|
); |
||||
|
|
||||
|
return sorted[0].id; |
||||
|
} |
||||
|
|
||||
|
export function createActivityManageTool({ |
||||
|
accountService, |
||||
|
approvedActions, |
||||
|
dataProviderService, |
||||
|
orderService, |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}: { |
||||
|
accountService: AccountService; |
||||
|
approvedActions?: string[]; |
||||
|
dataProviderService: DataProviderService; |
||||
|
orderService: OrderService; |
||||
|
portfolioSnapshotService: PortfolioSnapshotService; |
||||
|
redisCacheService: RedisCacheService; |
||||
|
userService: UserService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Create, update, or delete portfolio transactions (activities). Supports BUY, SELL, DIVIDEND, FEE, INTEREST, and LIABILITY types.', |
||||
|
needsApproval: |
||||
|
process.env.SKIP_APPROVAL === 'true' |
||||
|
? false |
||||
|
: (input) => { |
||||
|
if (input.action !== 'create' && input.action !== 'delete') |
||||
|
return false; |
||||
|
const sig = `activity_manage:${input.action}:${input.symbol ?? ''}`; |
||||
|
return !approvedActions?.includes(sig); |
||||
|
}, |
||||
|
inputSchema: z.object({ |
||||
|
action: z |
||||
|
.enum(['create', 'update', 'delete']) |
||||
|
.describe( |
||||
|
"Action to perform. 'create': record a new transaction. 'update': modify an existing one. 'delete': permanently remove." |
||||
|
), |
||||
|
type: z |
||||
|
.enum(['BUY', 'SELL', 'DIVIDEND', 'FEE', 'INTEREST', 'LIABILITY']) |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Transaction type. Required for 'create'. BUY/SELL for trades. DIVIDEND for dividends. FEE for fees. INTEREST for interest income." |
||||
|
), |
||||
|
symbol: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Asset ticker symbol. Required for 'create'. Stocks/ETFs: uppercase (AAPL). Crypto: ticker or CoinGecko slug — raw tickers (STX) are auto-resolved to slugs (stacks). FEE/INTEREST: descriptive name." |
||||
|
), |
||||
|
date: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Transaction date as ISO 8601 string (e.g. '2026-02-26'). Required for 'create'." |
||||
|
), |
||||
|
quantity: z |
||||
|
.number() |
||||
|
.min(0) |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Number of shares/units. Required for 'create'. For FEE: use 1." |
||||
|
), |
||||
|
unitPrice: z |
||||
|
.number() |
||||
|
.min(0) |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Price per share/unit. Required for 'create'. For FEE: total fee amount (with quantity=1)." |
||||
|
), |
||||
|
fee: z |
||||
|
.number() |
||||
|
.min(0) |
||||
|
.optional() |
||||
|
.default(0) |
||||
|
.describe('Transaction fee/commission. Default 0.'), |
||||
|
currency: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Transaction currency as ISO 4217 code. Required for 'create'." |
||||
|
), |
||||
|
accountId: z |
||||
|
.string() |
||||
|
.uuid() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
'Account ID. Optional — if omitted, the best-matching account is auto-selected based on asset type.' |
||||
|
), |
||||
|
dataSource: z |
||||
|
.enum(['YAHOO', 'COINGECKO', 'MANUAL']) |
||||
|
.optional() |
||||
|
.default('YAHOO') |
||||
|
.describe( |
||||
|
'Data source. YAHOO for stocks/ETFs. COINGECKO for crypto. MANUAL for custom assets/fees.' |
||||
|
), |
||||
|
comment: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe("Optional note, e.g. 'Quarterly dividend'."), |
||||
|
orderId: z |
||||
|
.string() |
||||
|
.uuid() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Transaction ID. Required for 'update' and 'delete'. Get from transaction_history." |
||||
|
) |
||||
|
}), |
||||
|
execute: async (input) => { |
||||
|
try { |
||||
|
switch (input.action) { |
||||
|
case 'create': { |
||||
|
const user = await userService.user({ id: userId }); |
||||
|
|
||||
|
// Auto-resolve CoinGecko slug from raw ticker (e.g. "STX" → "stacks")
|
||||
|
const symbol = |
||||
|
input.dataSource === 'COINGECKO' && input.symbol |
||||
|
? await resolveCoinGeckoSlug( |
||||
|
dataProviderService, |
||||
|
user, |
||||
|
input.symbol |
||||
|
) |
||||
|
: input.symbol; |
||||
|
|
||||
|
// Auto-resolve account if not provided
|
||||
|
let accountId = input.accountId; |
||||
|
|
||||
|
if (!accountId) { |
||||
|
const accounts = await accountService.getAccounts(userId); |
||||
|
accountId = resolveAccount( |
||||
|
accounts.map((a: any) => ({ |
||||
|
id: a.id, |
||||
|
name: a.name, |
||||
|
activitiesCount: a.activitiesCount |
||||
|
})), |
||||
|
input.dataSource |
||||
|
); |
||||
|
} |
||||
|
|
||||
|
// Validate activity
|
||||
|
await dataProviderService.validateActivities({ |
||||
|
activitiesDto: [ |
||||
|
{ |
||||
|
currency: input.currency, |
||||
|
dataSource: input.dataSource as DataSource, |
||||
|
symbol, |
||||
|
type: input.type as ActivityType |
||||
|
} |
||||
|
], |
||||
|
maxActivitiesToImport: 1, |
||||
|
user |
||||
|
}); |
||||
|
|
||||
|
const order = await orderService.createOrder({ |
||||
|
accountId, |
||||
|
comment: input.comment, |
||||
|
currency: input.currency, |
||||
|
date: parseISO(input.date), |
||||
|
fee: input.fee, |
||||
|
quantity: input.quantity, |
||||
|
SymbolProfile: { |
||||
|
connectOrCreate: { |
||||
|
create: { |
||||
|
currency: input.currency, |
||||
|
dataSource: input.dataSource as DataSource, |
||||
|
symbol |
||||
|
}, |
||||
|
where: { |
||||
|
dataSource_symbol: { |
||||
|
dataSource: input.dataSource as DataSource, |
||||
|
symbol |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
}, |
||||
|
type: input.type as ActivityType, |
||||
|
unitPrice: input.unitPrice, |
||||
|
user: { connect: { id: userId } }, |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
try { |
||||
|
await warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}); |
||||
|
} catch {} |
||||
|
|
||||
|
return { |
||||
|
id: order.id, |
||||
|
type: input.type, |
||||
|
symbol, |
||||
|
date: input.date, |
||||
|
quantity: input.quantity, |
||||
|
unitPrice: input.unitPrice, |
||||
|
fee: input.fee, |
||||
|
currency: input.currency, |
||||
|
accountId |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
case 'update': { |
||||
|
const originalOrder = await orderService.order({ |
||||
|
id: input.orderId |
||||
|
}); |
||||
|
|
||||
|
if (!originalOrder) { |
||||
|
return { error: 'Transaction not found' }; |
||||
|
} |
||||
|
|
||||
|
if (originalOrder.userId !== userId) { |
||||
|
return { |
||||
|
error: 'Transaction does not belong to you' |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
// Fetch current symbol profile for defaults
|
||||
|
const user = await userService.user({ id: userId }); |
||||
|
const userCurrency = |
||||
|
user?.settings?.settings?.baseCurrency ?? 'USD'; |
||||
|
|
||||
|
const { activities } = await orderService.getOrders({ |
||||
|
userCurrency, |
||||
|
userId, |
||||
|
includeDrafts: true, |
||||
|
withExcludedAccountsAndActivities: true |
||||
|
}); |
||||
|
|
||||
|
const currentActivity = activities.find( |
||||
|
(a) => a.id === input.orderId |
||||
|
); |
||||
|
|
||||
|
if (!currentActivity) { |
||||
|
return { error: 'Transaction not found' }; |
||||
|
} |
||||
|
|
||||
|
const newType = (input.type ?? |
||||
|
currentActivity.type) as ActivityType; |
||||
|
const newSymbol = |
||||
|
input.symbol ?? currentActivity.SymbolProfile?.symbol; |
||||
|
const newDataSource = (input.dataSource ?? |
||||
|
currentActivity.SymbolProfile?.dataSource) as DataSource; |
||||
|
const newDate = input.date |
||||
|
? parseISO(input.date) |
||||
|
: currentActivity.date; |
||||
|
const newAccountId = input.accountId ?? currentActivity.accountId; |
||||
|
|
||||
|
await orderService.updateOrder({ |
||||
|
data: { |
||||
|
comment: input.comment ?? currentActivity.comment, |
||||
|
currency: input.currency ?? currentActivity.currency, |
||||
|
date: newDate, |
||||
|
fee: input.fee ?? currentActivity.fee, |
||||
|
quantity: input.quantity ?? currentActivity.quantity, |
||||
|
type: newType, |
||||
|
unitPrice: input.unitPrice ?? currentActivity.unitPrice, |
||||
|
account: newAccountId |
||||
|
? { |
||||
|
connect: { |
||||
|
id_userId: { id: newAccountId, userId } |
||||
|
} |
||||
|
} |
||||
|
: undefined, |
||||
|
SymbolProfile: { |
||||
|
connect: { |
||||
|
dataSource_symbol: { |
||||
|
dataSource: newDataSource, |
||||
|
symbol: newSymbol |
||||
|
} |
||||
|
}, |
||||
|
update: { |
||||
|
assetClass: undefined, |
||||
|
assetSubClass: undefined, |
||||
|
name: newSymbol |
||||
|
} |
||||
|
}, |
||||
|
user: { connect: { id: userId } } |
||||
|
}, |
||||
|
where: { id: input.orderId } |
||||
|
}); |
||||
|
|
||||
|
try { |
||||
|
await warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}); |
||||
|
} catch {} |
||||
|
|
||||
|
return { |
||||
|
updated: true, |
||||
|
id: input.orderId, |
||||
|
type: newType, |
||||
|
symbol: newSymbol, |
||||
|
date: newDate |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
case 'delete': { |
||||
|
const order = await orderService.order({ id: input.orderId }); |
||||
|
|
||||
|
if (!order) { |
||||
|
return { error: 'Transaction not found' }; |
||||
|
} |
||||
|
|
||||
|
if (order.userId !== userId) { |
||||
|
return { |
||||
|
error: 'Transaction does not belong to you' |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
const deletedOrder = await orderService.deleteOrder({ |
||||
|
id: input.orderId |
||||
|
}); |
||||
|
|
||||
|
try { |
||||
|
await warmPortfolioCache({ |
||||
|
portfolioSnapshotService, |
||||
|
redisCacheService, |
||||
|
userService, |
||||
|
userId |
||||
|
}); |
||||
|
} catch {} |
||||
|
|
||||
|
return { |
||||
|
deleted: true, |
||||
|
type: deletedOrder.type, |
||||
|
date: deletedOrder.date |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to ${input.action} activity: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,78 @@ |
|||||
|
import { PortfolioService } from '@ghostfolio/api/app/portfolio/portfolio.service'; |
||||
|
|
||||
|
import { DataSource } from '@prisma/client'; |
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
export function createHoldingsLookupTool({ |
||||
|
portfolioService, |
||||
|
userId |
||||
|
}: { |
||||
|
portfolioService: PortfolioService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Look up detailed information about a specific holding in the portfolio: performance, dividends, fees, historical data, countries, sectors. Use when the user asks about a specific stock, ETF, or crypto they hold.', |
||||
|
inputSchema: z.object({ |
||||
|
symbol: z.string().describe('Ticker symbol (e.g. AAPL, MSFT, BTC-USD)'), |
||||
|
dataSource: z |
||||
|
.enum([ |
||||
|
'ALPHA_VANTAGE', |
||||
|
'COINGECKO', |
||||
|
'EOD_HISTORICAL_DATA', |
||||
|
'FINANCIAL_MODELING_PREP', |
||||
|
'GHOSTFOLIO', |
||||
|
'GOOGLE_SHEETS', |
||||
|
'MANUAL', |
||||
|
'RAPID_API', |
||||
|
'YAHOO' |
||||
|
]) |
||||
|
.optional() |
||||
|
.default('YAHOO') |
||||
|
.describe('Data source. Defaults to YAHOO.') |
||||
|
}), |
||||
|
execute: async ({ symbol, dataSource = 'YAHOO' }) => { |
||||
|
try { |
||||
|
const holding = await portfolioService.getHolding({ |
||||
|
dataSource: dataSource as DataSource, |
||||
|
impersonationId: undefined, |
||||
|
symbol: symbol.toUpperCase(), |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
if (!holding) { |
||||
|
return { error: `No holding found for ${symbol}` }; |
||||
|
} |
||||
|
|
||||
|
return { |
||||
|
symbol: holding.SymbolProfile?.symbol, |
||||
|
name: holding.SymbolProfile?.name, |
||||
|
assetClass: holding.SymbolProfile?.assetClass, |
||||
|
assetSubClass: holding.SymbolProfile?.assetSubClass, |
||||
|
currency: holding.SymbolProfile?.currency, |
||||
|
quantity: holding.quantity, |
||||
|
averagePrice: holding.averagePrice, |
||||
|
marketPrice: holding.marketPrice, |
||||
|
marketPriceMin: holding.marketPriceMin, |
||||
|
marketPriceMax: holding.marketPriceMax, |
||||
|
value: holding.value, |
||||
|
netPerformance: holding.netPerformance, |
||||
|
netPerformancePercent: holding.netPerformancePercent, |
||||
|
dividendInBaseCurrency: holding.dividendInBaseCurrency, |
||||
|
dividendYieldPercent: holding.dividendYieldPercent, |
||||
|
feeInBaseCurrency: holding.feeInBaseCurrency, |
||||
|
activitiesCount: holding.activitiesCount, |
||||
|
dateOfFirstActivity: holding.dateOfFirstActivity, |
||||
|
countries: holding.SymbolProfile?.countries, |
||||
|
sectors: holding.SymbolProfile?.sectors, |
||||
|
tags: holding.tags |
||||
|
}; |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to look up holding ${symbol}: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,73 @@ |
|||||
|
import { DataProviderService } from '@ghostfolio/api/services/data-provider/data-provider.service'; |
||||
|
|
||||
|
import { DataSource } from '@prisma/client'; |
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
export function createMarketDataTool({ |
||||
|
dataProviderService |
||||
|
}: { |
||||
|
dataProviderService: DataProviderService; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Get current market price and quote data for one or more symbols. Requires the exact symbol and dataSource. For stocks/ETFs use dataSource "YAHOO" with uppercase ticker (e.g. "AAPL"). For crypto use dataSource "COINGECKO" with the exact CoinGecko slug — IMPORTANT: only use "bitcoin", "ethereum", or "solana" directly; for all other crypto you MUST call symbol_search first to get the correct slug.', |
||||
|
inputSchema: z.object({ |
||||
|
symbols: z |
||||
|
.array( |
||||
|
z.object({ |
||||
|
symbol: z |
||||
|
.string() |
||||
|
.describe( |
||||
|
'The exact symbol. For YAHOO: uppercase ticker (e.g. "AAPL"). For COINGECKO: exact lowercase slug from symbol_search (e.g. "bitcoin", "blockstack").' |
||||
|
), |
||||
|
dataSource: z |
||||
|
.enum([ |
||||
|
'ALPHA_VANTAGE', |
||||
|
'COINGECKO', |
||||
|
'EOD_HISTORICAL_DATA', |
||||
|
'FINANCIAL_MODELING_PREP', |
||||
|
'GHOSTFOLIO', |
||||
|
'GOOGLE_SHEETS', |
||||
|
'MANUAL', |
||||
|
'RAPID_API', |
||||
|
'YAHOO' |
||||
|
]) |
||||
|
.optional() |
||||
|
.default('YAHOO') |
||||
|
.describe( |
||||
|
'Data source. Use "COINGECKO" for cryptocurrencies, "YAHOO" for stocks/ETFs. Defaults to YAHOO.' |
||||
|
) |
||||
|
}) |
||||
|
) |
||||
|
.min(1) |
||||
|
.max(10) |
||||
|
.describe('Array of symbols to look up (max 10)') |
||||
|
}), |
||||
|
execute: async ({ symbols }) => { |
||||
|
try { |
||||
|
const items = symbols.map(({ symbol, dataSource = 'YAHOO' }) => ({ |
||||
|
dataSource: dataSource as DataSource, |
||||
|
symbol: |
||||
|
dataSource === 'COINGECKO' |
||||
|
? symbol.toLowerCase() |
||||
|
: symbol.toUpperCase() |
||||
|
})); |
||||
|
|
||||
|
const quotes = await dataProviderService.getQuotes({ items }); |
||||
|
|
||||
|
return Object.entries(quotes).map(([symbol, data]) => ({ |
||||
|
symbol, |
||||
|
currency: data.currency, |
||||
|
marketPrice: data.marketPrice, |
||||
|
marketState: data.marketState, |
||||
|
dataSource: data.dataSource |
||||
|
})); |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to fetch market data: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,72 @@ |
|||||
|
import { PortfolioService } from '@ghostfolio/api/app/portfolio/portfolio.service'; |
||||
|
import type { DateRange } from '@ghostfolio/common/types'; |
||||
|
|
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
export function createPortfolioPerformanceTool({ |
||||
|
portfolioService, |
||||
|
userId |
||||
|
}: { |
||||
|
portfolioService: PortfolioService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Get portfolio performance metrics over a time range: returns, net performance, chart data, and annualized performance. Use when the user asks about returns, how their portfolio is doing, or wants performance over a specific period.', |
||||
|
inputSchema: z.object({ |
||||
|
dateRange: z |
||||
|
.enum(['1d', '1y', '5y', 'max', 'mtd', 'wtd', 'ytd']) |
||||
|
.optional() |
||||
|
.describe( |
||||
|
'Time range for performance data. Defaults to max (all time). ' + |
||||
|
'If the user asks for a range that doesn\'t exactly match (e.g. "last 3 months", "since October"), ' + |
||||
|
'pick the closest range that fully covers their request and filter/summarize the relevant portion in your response. ' + |
||||
|
"Never refuse a request just because the exact range isn't available." |
||||
|
) |
||||
|
}), |
||||
|
execute: async ({ dateRange = 'max' }) => { |
||||
|
try { |
||||
|
const result = await portfolioService.getPerformance({ |
||||
|
dateRange: dateRange as DateRange, |
||||
|
filters: undefined, |
||||
|
impersonationId: undefined, |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
// Downsample chart to ~20 points for LLM context efficiency
|
||||
|
const chart = result.chart ?? []; |
||||
|
const sampled: { date: string; netWorth: number }[] = []; |
||||
|
|
||||
|
if (chart.length > 0) { |
||||
|
const step = Math.max(1, Math.floor(chart.length / 20)); |
||||
|
|
||||
|
for (let i = 0; i < chart.length; i += step) { |
||||
|
sampled.push({ |
||||
|
date: chart[i].date, |
||||
|
netWorth: chart[i].netWorth ?? 0 |
||||
|
}); |
||||
|
} |
||||
|
|
||||
|
// Always include the last point
|
||||
|
const last = chart[chart.length - 1]; |
||||
|
|
||||
|
if (sampled[sampled.length - 1]?.date !== last.date) { |
||||
|
sampled.push({ date: last.date, netWorth: last.netWorth ?? 0 }); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
return { |
||||
|
firstOrderDate: result.firstOrderDate, |
||||
|
hasErrors: result.hasErrors, |
||||
|
performance: result.performance, |
||||
|
chart: sampled.length ? sampled : null |
||||
|
}; |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to fetch performance: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,63 @@ |
|||||
|
import { PortfolioService } from '@ghostfolio/api/app/portfolio/portfolio.service'; |
||||
|
import type { DateRange } from '@ghostfolio/common/types'; |
||||
|
|
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
export function createPortfolioAnalysisTool({ |
||||
|
portfolioService, |
||||
|
userId |
||||
|
}: { |
||||
|
portfolioService: PortfolioService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Get portfolio details including holdings, allocations, account breakdown, and summary metrics (total value, cash, currency). Use this when the user asks about their portfolio composition, what they own, or wants an overview.', |
||||
|
inputSchema: z.object({ |
||||
|
dateRange: z |
||||
|
.enum(['1d', '1y', '5y', 'max', 'mtd', 'wtd', 'ytd']) |
||||
|
.optional() |
||||
|
.describe( |
||||
|
'Time range for the analysis. Defaults to max (all time). ' + |
||||
|
'If the user asks for a range that doesn\'t exactly match (e.g. "last 3 months", "since October"), ' + |
||||
|
'pick the closest range that fully covers their request and filter/summarize the relevant portion in your response. ' + |
||||
|
"Never refuse a request just because the exact range isn't available." |
||||
|
) |
||||
|
}), |
||||
|
execute: async ({ dateRange = 'max' }) => { |
||||
|
try { |
||||
|
const details = await portfolioService.getDetails({ |
||||
|
dateRange: dateRange as DateRange, |
||||
|
filters: undefined, |
||||
|
impersonationId: undefined, |
||||
|
userId, |
||||
|
withSummary: true |
||||
|
}); |
||||
|
|
||||
|
const holdings = Object.values(details.holdings).map((h) => ({ |
||||
|
name: h.name, |
||||
|
symbol: h.symbol, |
||||
|
currency: h.currency, |
||||
|
assetClass: h.assetClass, |
||||
|
assetSubClass: h.assetSubClass, |
||||
|
allocationInPercentage: h.allocationInPercentage, |
||||
|
valueInBaseCurrency: h.valueInBaseCurrency, |
||||
|
netPerformancePercent: h.netPerformancePercent, |
||||
|
quantity: h.quantity, |
||||
|
marketPrice: h.marketPrice |
||||
|
})); |
||||
|
|
||||
|
return { |
||||
|
holdings, |
||||
|
summary: details.summary, |
||||
|
accounts: details.accounts |
||||
|
}; |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to fetch portfolio: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,62 @@ |
|||||
|
import { UserService } from '@ghostfolio/api/app/user/user.service'; |
||||
|
import { DataProviderService } from '@ghostfolio/api/services/data-provider/data-provider.service'; |
||||
|
|
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
export function createSymbolSearchTool({ |
||||
|
dataProviderService, |
||||
|
userService, |
||||
|
userId |
||||
|
}: { |
||||
|
dataProviderService: DataProviderService; |
||||
|
userService: UserService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Search for a stock, ETF, or cryptocurrency by name or ticker. Use when unsure of the exact CoinGecko slug or Yahoo symbol. Returns matching assets with their correct symbol and dataSource for use with market_data.', |
||||
|
inputSchema: z.object({ |
||||
|
query: z |
||||
|
.string() |
||||
|
.min(2) |
||||
|
.describe( |
||||
|
'Search query — a coin name, ticker, or company name (e.g. "stacks", "STX", "apple")' |
||||
|
) |
||||
|
}), |
||||
|
execute: async ({ query }) => { |
||||
|
try { |
||||
|
const user = await userService.user({ id: userId }); |
||||
|
|
||||
|
const { items } = await dataProviderService.search({ |
||||
|
query, |
||||
|
user |
||||
|
}); |
||||
|
|
||||
|
return items |
||||
|
.slice(0, 10) |
||||
|
.map( |
||||
|
({ |
||||
|
assetClass, |
||||
|
assetSubClass, |
||||
|
currency, |
||||
|
dataSource, |
||||
|
name, |
||||
|
symbol |
||||
|
}) => ({ |
||||
|
symbol, |
||||
|
name, |
||||
|
dataSource, |
||||
|
assetClass, |
||||
|
assetSubClass, |
||||
|
currency |
||||
|
}) |
||||
|
); |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to search symbols: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,80 @@ |
|||||
|
import { TagService } from '@ghostfolio/api/services/tag/tag.service'; |
||||
|
|
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
export function createTagManageTool({ |
||||
|
tagService, |
||||
|
userId |
||||
|
}: { |
||||
|
tagService: TagService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Manage tags for organizing and categorizing transactions. Tags can be applied to activities to group them (e.g., "Long Term", "Tax Loss Harvest", "Retirement"). Use this to create new tags, rename existing ones, delete unused tags, or list all available tags.', |
||||
|
needsApproval: |
||||
|
process.env.SKIP_APPROVAL === 'true' |
||||
|
? false |
||||
|
: (input) => input.action !== 'list', |
||||
|
inputSchema: z.object({ |
||||
|
action: z |
||||
|
.enum(['create', 'update', 'delete', 'list']) |
||||
|
.describe( |
||||
|
"Action to perform. 'create': make a new tag. 'update': rename an existing tag. 'delete': remove a tag from all transactions. 'list': show all tags." |
||||
|
), |
||||
|
name: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Tag name. Required for 'create' and 'update'. E.g. 'Long Term', 'Speculative', 'Tax Loss Harvest'." |
||||
|
), |
||||
|
tagId: z |
||||
|
.string() |
||||
|
.uuid() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Tag ID. Required for 'update' and 'delete'. Get this from action 'list' results." |
||||
|
) |
||||
|
}), |
||||
|
execute: async (input) => { |
||||
|
try { |
||||
|
switch (input.action) { |
||||
|
case 'create': { |
||||
|
const tag = await tagService.createTag({ |
||||
|
name: input.name, |
||||
|
user: { connect: { id: userId } } |
||||
|
}); |
||||
|
|
||||
|
return { id: tag.id, name: tag.name }; |
||||
|
} |
||||
|
|
||||
|
case 'update': { |
||||
|
const tag = await tagService.updateTag({ |
||||
|
data: { name: input.name }, |
||||
|
where: { id: input.tagId } |
||||
|
}); |
||||
|
|
||||
|
return { id: tag.id, name: tag.name }; |
||||
|
} |
||||
|
|
||||
|
case 'delete': { |
||||
|
const tag = await tagService.deleteTag({ id: input.tagId }); |
||||
|
|
||||
|
return { deleted: true, name: tag.name }; |
||||
|
} |
||||
|
|
||||
|
case 'list': { |
||||
|
const tags = await tagService.getTagsForUser(userId); |
||||
|
|
||||
|
return tags.map(({ id, name, isUsed }) => ({ id, name, isUsed })); |
||||
|
} |
||||
|
} |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to ${input.action} tag: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,211 @@ |
|||||
|
import { AccountBalanceService } from '@ghostfolio/api/app/account-balance/account-balance.service'; |
||||
|
import { AccountService } from '@ghostfolio/api/app/account/account.service'; |
||||
|
import { OrderService } from '@ghostfolio/api/app/order/order.service'; |
||||
|
import { UserService } from '@ghostfolio/api/app/user/user.service'; |
||||
|
|
||||
|
import type { Type as ActivityType } from '@prisma/client'; |
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
const ORDER_TYPES = [ |
||||
|
'BUY', |
||||
|
'SELL', |
||||
|
'DIVIDEND', |
||||
|
'FEE', |
||||
|
'INTEREST', |
||||
|
'LIABILITY' |
||||
|
] as const; |
||||
|
|
||||
|
const CASH_TYPES = ['DEPOSIT', 'WITHDRAWAL'] as const; |
||||
|
|
||||
|
const ALL_TYPES = [...ORDER_TYPES, ...CASH_TYPES] as const; |
||||
|
|
||||
|
export function createTransactionHistoryTool({ |
||||
|
accountBalanceService, |
||||
|
accountService, |
||||
|
orderService, |
||||
|
userService, |
||||
|
userId |
||||
|
}: { |
||||
|
accountBalanceService: AccountBalanceService; |
||||
|
accountService: AccountService; |
||||
|
orderService: OrderService; |
||||
|
userService: UserService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
'Get transaction/activity history: buys, sells, dividends, fees, deposits, withdrawals. Use when the user asks about their trades, transaction history, activity log, deposits, or withdrawals.', |
||||
|
inputSchema: z.object({ |
||||
|
types: z |
||||
|
.array(z.enum(ALL_TYPES)) |
||||
|
.optional() |
||||
|
.describe( |
||||
|
'Filter by activity type. Includes DEPOSIT/WITHDRAWAL for cash movements. Omit to get all types.' |
||||
|
), |
||||
|
take: z |
||||
|
.number() |
||||
|
.int() |
||||
|
.min(1) |
||||
|
.max(100) |
||||
|
.optional() |
||||
|
.default(50) |
||||
|
.describe('Number of results to return (max 100). Defaults to 50.'), |
||||
|
sortDirection: z |
||||
|
.enum(['asc', 'desc']) |
||||
|
.optional() |
||||
|
.default('desc') |
||||
|
.describe('Sort by date. Defaults to desc (newest first).') |
||||
|
}), |
||||
|
execute: async ({ types, take = 50, sortDirection = 'desc' }) => { |
||||
|
try { |
||||
|
const user = await userService.user({ id: userId }); |
||||
|
const userCurrency = user?.settings?.settings?.baseCurrency ?? 'USD'; |
||||
|
|
||||
|
// Split types into order types vs cash types
|
||||
|
const orderTypes = types?.filter( |
||||
|
(t): t is (typeof ORDER_TYPES)[number] => |
||||
|
(ORDER_TYPES as readonly string[]).includes(t) |
||||
|
); |
||||
|
const cashTypes = types?.filter((t): t is (typeof CASH_TYPES)[number] => |
||||
|
(CASH_TYPES as readonly string[]).includes(t) |
||||
|
); |
||||
|
|
||||
|
const wantOrders = !types || orderTypes.length > 0; |
||||
|
const wantCash = !types || cashTypes.length > 0; |
||||
|
|
||||
|
// Fetch orders if needed
|
||||
|
let orderEntries: { |
||||
|
date: Date; |
||||
|
type: string; |
||||
|
symbol?: string; |
||||
|
name?: string; |
||||
|
quantity?: number; |
||||
|
unitPrice?: number; |
||||
|
currency?: string; |
||||
|
fee?: number; |
||||
|
value?: number; |
||||
|
valueInBaseCurrency?: number; |
||||
|
account?: string; |
||||
|
}[] = []; |
||||
|
|
||||
|
if (wantOrders) { |
||||
|
const { activities } = await orderService.getOrders({ |
||||
|
sortDirection: 'asc', |
||||
|
types: (orderTypes?.length > 0 |
||||
|
? orderTypes |
||||
|
: undefined) as ActivityType[], |
||||
|
userCurrency, |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
orderEntries = activities.map((a) => ({ |
||||
|
date: a.date, |
||||
|
type: a.type, |
||||
|
symbol: a.SymbolProfile?.symbol, |
||||
|
name: a.SymbolProfile?.name, |
||||
|
quantity: a.quantity, |
||||
|
unitPrice: a.unitPrice, |
||||
|
currency: a.currency, |
||||
|
fee: a.fee, |
||||
|
value: a.value, |
||||
|
valueInBaseCurrency: a.valueInBaseCurrency, |
||||
|
account: a.account?.name |
||||
|
})); |
||||
|
} |
||||
|
|
||||
|
// Fetch balance deltas if needed
|
||||
|
const cashEntries: typeof orderEntries = []; |
||||
|
|
||||
|
if (wantCash) { |
||||
|
const [{ balances }, accounts] = await Promise.all([ |
||||
|
accountBalanceService.getAccountBalances({ |
||||
|
userCurrency, |
||||
|
userId, |
||||
|
withExcludedAccounts: false |
||||
|
}), |
||||
|
accountService.getAccounts(userId) |
||||
|
]); |
||||
|
|
||||
|
const accountNameById = new Map(accounts.map((a) => [a.id, a.name])); |
||||
|
|
||||
|
// Compute deltas per account
|
||||
|
const lastBalanceByAccount = new Map<string, number>(); |
||||
|
|
||||
|
for (const b of balances) { |
||||
|
const prev = lastBalanceByAccount.get(b.accountId); |
||||
|
lastBalanceByAccount.set(b.accountId, b.valueInBaseCurrency); |
||||
|
|
||||
|
if (prev === undefined) { |
||||
|
// First record — treat as initial deposit if > 0
|
||||
|
if (b.valueInBaseCurrency === 0) continue; |
||||
|
const type = b.valueInBaseCurrency > 0 ? 'DEPOSIT' : 'WITHDRAWAL'; |
||||
|
if ( |
||||
|
types && |
||||
|
!cashTypes.includes(type as (typeof CASH_TYPES)[number]) |
||||
|
) |
||||
|
continue; |
||||
|
|
||||
|
cashEntries.push({ |
||||
|
date: b.date, |
||||
|
type, |
||||
|
value: Math.abs(b.valueInBaseCurrency), |
||||
|
valueInBaseCurrency: Math.abs(b.valueInBaseCurrency), |
||||
|
account: accountNameById.get(b.accountId) |
||||
|
}); |
||||
|
continue; |
||||
|
} |
||||
|
|
||||
|
const delta = b.valueInBaseCurrency - prev; |
||||
|
if (delta === 0) continue; |
||||
|
|
||||
|
const type = delta > 0 ? 'DEPOSIT' : 'WITHDRAWAL'; |
||||
|
if ( |
||||
|
types && |
||||
|
!cashTypes.includes(type as (typeof CASH_TYPES)[number]) |
||||
|
) |
||||
|
continue; |
||||
|
|
||||
|
cashEntries.push({ |
||||
|
date: b.date, |
||||
|
type, |
||||
|
value: Math.abs(delta), |
||||
|
valueInBaseCurrency: Math.abs(delta), |
||||
|
account: accountNameById.get(b.accountId) |
||||
|
}); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Merge and sort
|
||||
|
const merged = [...orderEntries, ...cashEntries]; |
||||
|
merged.sort((a, b) => { |
||||
|
const diff = a.date.getTime() - b.date.getTime(); |
||||
|
return sortDirection === 'asc' ? diff : -diff; |
||||
|
}); |
||||
|
|
||||
|
const sliced = merged.slice(0, take); |
||||
|
|
||||
|
return { |
||||
|
count: merged.length, |
||||
|
activities: sliced.map((e) => ({ |
||||
|
date: e.date, |
||||
|
type: e.type, |
||||
|
symbol: e.symbol, |
||||
|
name: e.name, |
||||
|
quantity: e.quantity, |
||||
|
unitPrice: e.unitPrice, |
||||
|
currency: e.currency, |
||||
|
fee: e.fee, |
||||
|
value: e.value, |
||||
|
valueInBaseCurrency: e.valueInBaseCurrency, |
||||
|
account: e.account |
||||
|
})) |
||||
|
}; |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to fetch transactions: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,90 @@ |
|||||
|
import { WatchlistService } from '@ghostfolio/api/app/endpoints/watchlist/watchlist.service'; |
||||
|
|
||||
|
import { DataSource } from '@prisma/client'; |
||||
|
import { tool } from 'ai'; |
||||
|
import { z } from 'zod'; |
||||
|
|
||||
|
export function createWatchlistManageTool({ |
||||
|
watchlistService, |
||||
|
userId |
||||
|
}: { |
||||
|
watchlistService: WatchlistService; |
||||
|
userId: string; |
||||
|
}) { |
||||
|
return tool({ |
||||
|
description: |
||||
|
"Manage the user's watchlist of tracked securities. Add symbols to watch their price trends and market conditions, remove symbols no longer of interest, or list the current watchlist with performance data.", |
||||
|
needsApproval: |
||||
|
process.env.SKIP_APPROVAL === 'true' |
||||
|
? false |
||||
|
: (input) => input.action !== 'list', |
||||
|
inputSchema: z.object({ |
||||
|
action: z |
||||
|
.enum(['add', 'remove', 'list']) |
||||
|
.describe( |
||||
|
"Action to perform. 'add': add a symbol to watch. 'remove': stop watching a symbol. 'list': show all watched symbols with trends." |
||||
|
), |
||||
|
symbol: z |
||||
|
.string() |
||||
|
.optional() |
||||
|
.describe( |
||||
|
"Ticker symbol. Required for 'add' and 'remove'. Stocks/ETFs: uppercase (AAPL, MSFT). Crypto: CoinGecko slug (bitcoin, ethereum)." |
||||
|
), |
||||
|
dataSource: z |
||||
|
.enum(['YAHOO', 'COINGECKO']) |
||||
|
.optional() |
||||
|
.default('YAHOO') |
||||
|
.describe( |
||||
|
"Data source for the symbol. YAHOO for stocks/ETFs. COINGECKO for crypto. Required for 'add' and 'remove'." |
||||
|
) |
||||
|
}), |
||||
|
execute: async (input) => { |
||||
|
try { |
||||
|
switch (input.action) { |
||||
|
case 'add': { |
||||
|
await watchlistService.createWatchlistItem({ |
||||
|
dataSource: input.dataSource as DataSource, |
||||
|
symbol: input.symbol, |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
return { |
||||
|
added: true, |
||||
|
symbol: input.symbol, |
||||
|
dataSource: input.dataSource |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
case 'remove': { |
||||
|
await watchlistService.deleteWatchlistItem({ |
||||
|
dataSource: input.dataSource as DataSource, |
||||
|
symbol: input.symbol, |
||||
|
userId |
||||
|
}); |
||||
|
|
||||
|
return { removed: true, symbol: input.symbol }; |
||||
|
} |
||||
|
|
||||
|
case 'list': { |
||||
|
const items = await watchlistService.getWatchlistItems(userId); |
||||
|
|
||||
|
return items.map((item) => ({ |
||||
|
symbol: item.symbol, |
||||
|
name: item.name, |
||||
|
dataSource: item.dataSource, |
||||
|
marketCondition: item.marketCondition, |
||||
|
trend50d: item.trend50d, |
||||
|
trend200d: item.trend200d, |
||||
|
allTimeHighPerformance: |
||||
|
item.performances?.allTimeHigh?.performancePercent |
||||
|
})); |
||||
|
} |
||||
|
} |
||||
|
} catch (error) { |
||||
|
return { |
||||
|
error: `Failed to ${input.action} watchlist item: ${error instanceof Error ? error.message : 'unknown error'}` |
||||
|
}; |
||||
|
} |
||||
|
} |
||||
|
}); |
||||
|
} |
||||
@ -0,0 +1,62 @@ |
|||||
|
import type { HallucinationCheckResult } from './hallucination-check'; |
||||
|
import type { OutputValidationResult } from './output-validation'; |
||||
|
|
||||
|
export interface ConfidenceScoreResult { |
||||
|
score: number; |
||||
|
breakdown: { |
||||
|
toolSuccessRate: number; |
||||
|
stepEfficiency: number; |
||||
|
outputValidity: number; |
||||
|
hallucinationScore: number; |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
const WEIGHTS = { |
||||
|
toolSuccessRate: 0.3, |
||||
|
stepEfficiency: 0.1, |
||||
|
outputValidity: 0.3, |
||||
|
hallucinationScore: 0.3 |
||||
|
}; |
||||
|
|
||||
|
/** |
||||
|
* Composite confidence score (0-1) for an agent response. |
||||
|
*/ |
||||
|
export function computeConfidence({ |
||||
|
toolCallCount, |
||||
|
toolErrorCount, |
||||
|
stepCount, |
||||
|
maxSteps, |
||||
|
validation, |
||||
|
hallucination |
||||
|
}: { |
||||
|
toolCallCount: number; |
||||
|
toolErrorCount: number; |
||||
|
stepCount: number; |
||||
|
maxSteps: number; |
||||
|
validation: OutputValidationResult; |
||||
|
hallucination: HallucinationCheckResult; |
||||
|
}): ConfidenceScoreResult { |
||||
|
// Tool success rate: 1.0 if no tools, else (successful / total)
|
||||
|
const toolSuccessRate = |
||||
|
toolCallCount === 0 ? 1 : (toolCallCount - toolErrorCount) / toolCallCount; |
||||
|
|
||||
|
// Step efficiency: penalize using many steps (closer to max = lower)
|
||||
|
const stepEfficiency = maxSteps > 0 ? 1 - stepCount / (maxSteps * 2) : 1; |
||||
|
const clampedEfficiency = Math.max(0, Math.min(1, stepEfficiency)); |
||||
|
|
||||
|
const score = |
||||
|
WEIGHTS.toolSuccessRate * toolSuccessRate + |
||||
|
WEIGHTS.stepEfficiency * clampedEfficiency + |
||||
|
WEIGHTS.outputValidity * validation.score + |
||||
|
WEIGHTS.hallucinationScore * hallucination.score; |
||||
|
|
||||
|
return { |
||||
|
score: Math.round(score * 1000) / 1000, |
||||
|
breakdown: { |
||||
|
toolSuccessRate: Math.round(toolSuccessRate * 1000) / 1000, |
||||
|
stepEfficiency: Math.round(clampedEfficiency * 1000) / 1000, |
||||
|
outputValidity: Math.round(validation.score * 1000) / 1000, |
||||
|
hallucinationScore: Math.round(hallucination.score * 1000) / 1000 |
||||
|
} |
||||
|
}; |
||||
|
} |
||||
@ -0,0 +1,210 @@ |
|||||
|
export interface ToolResult { |
||||
|
toolName: string; |
||||
|
result: unknown; |
||||
|
} |
||||
|
|
||||
|
export interface HallucinationCheckResult { |
||||
|
clean: boolean; |
||||
|
score: number; |
||||
|
issues: string[]; |
||||
|
} |
||||
|
|
||||
|
/** |
||||
|
* Deterministic hallucination detection. |
||||
|
* Checks that the response doesn't reference data not present in tool results. |
||||
|
*/ |
||||
|
export function checkHallucination({ |
||||
|
text, |
||||
|
toolResults |
||||
|
}: { |
||||
|
text: string; |
||||
|
toolResults: ToolResult[]; |
||||
|
}): HallucinationCheckResult { |
||||
|
const issues: string[] = []; |
||||
|
let checks = 0; |
||||
|
let passed = 0; |
||||
|
|
||||
|
if (toolResults.length === 0) { |
||||
|
return { clean: true, score: 1, issues: [] }; |
||||
|
} |
||||
|
|
||||
|
const toolDataStr = JSON.stringify(toolResults.map((tr) => tr.result)); |
||||
|
|
||||
|
// Check: ticker symbols in response should exist in tool data
|
||||
|
const tickerMatches = text.match(/\b[A-Z]{2,5}\b/g) ?? []; |
||||
|
const toolTickers = extractTickers(toolDataStr); |
||||
|
const knownNonTickers = new Set([ |
||||
|
'THE', |
||||
|
'AND', |
||||
|
'FOR', |
||||
|
'NOT', |
||||
|
'YOU', |
||||
|
'ARE', |
||||
|
'HAS', |
||||
|
'WAS', |
||||
|
'ALL', |
||||
|
'CAN', |
||||
|
'HAD', |
||||
|
'HER', |
||||
|
'ONE', |
||||
|
'OUR', |
||||
|
'OUT', |
||||
|
'HIS', |
||||
|
'HOW', |
||||
|
'ITS', |
||||
|
'MAY', |
||||
|
'NEW', |
||||
|
'NOW', |
||||
|
'OLD', |
||||
|
'SEE', |
||||
|
'WAY', |
||||
|
'WHO', |
||||
|
'DID', |
||||
|
'GET', |
||||
|
'LET', |
||||
|
'SAY', |
||||
|
'SHE', |
||||
|
'TOO', |
||||
|
'USE', |
||||
|
'YTD', |
||||
|
'USD', |
||||
|
'ETF', |
||||
|
'IPO', |
||||
|
'CEO', |
||||
|
'CFO', |
||||
|
'ROI', |
||||
|
'EPS', |
||||
|
'ATH', |
||||
|
'APR', |
||||
|
'FEB', |
||||
|
'MAR', |
||||
|
'JAN', |
||||
|
'JUN', |
||||
|
'JUL', |
||||
|
'AUG', |
||||
|
'SEP', |
||||
|
'OCT', |
||||
|
'NOV', |
||||
|
'DEC' |
||||
|
]); |
||||
|
|
||||
|
if (tickerMatches.length > 0) { |
||||
|
checks++; |
||||
|
const unknownTickers = tickerMatches.filter( |
||||
|
(t) => !toolTickers.has(t) && !knownNonTickers.has(t) |
||||
|
); |
||||
|
if (unknownTickers.length === 0) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
// Only flag if many unknown tickers (some might be abbreviations)
|
||||
|
const ratio = unknownTickers.length / tickerMatches.length; |
||||
|
if (ratio > 0.5) { |
||||
|
issues.push( |
||||
|
`Possible hallucinated tickers: ${unknownTickers.slice(0, 5).join(', ')}` |
||||
|
); |
||||
|
} else { |
||||
|
passed++; |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Check: dollar amounts in response should approximately match tool data
|
||||
|
const responseDollars = extractDollarAmounts(text); |
||||
|
const toolDollars = extractDollarAmounts(toolDataStr); |
||||
|
|
||||
|
if (responseDollars.length > 0 && toolDollars.length > 0) { |
||||
|
checks++; |
||||
|
const unmatchedAmounts = responseDollars.filter( |
||||
|
(rd) => !toolDollars.some((td) => isApproximateMatch(rd, td)) |
||||
|
); |
||||
|
if (unmatchedAmounts.length === 0) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
const ratio = unmatchedAmounts.length / responseDollars.length; |
||||
|
if (ratio > 0.5) { |
||||
|
issues.push( |
||||
|
`Dollar amounts not found in tool data: ${unmatchedAmounts |
||||
|
.slice(0, 3) |
||||
|
.map((a) => `$${a}`) |
||||
|
.join(', ')}` |
||||
|
); |
||||
|
} else { |
||||
|
passed++; |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Check: no claims about holdings not in tool results
|
||||
|
const holdingSymbols = extractHoldingSymbols(toolResults); |
||||
|
if (holdingSymbols.size > 0) { |
||||
|
checks++; |
||||
|
// Look for "you hold X" or "your X position" patterns referencing unknown symbols
|
||||
|
const holdingClaims = |
||||
|
text.match( |
||||
|
/(?:you (?:hold|own|have)|your .+ position|holding of)\s+([A-Z]{2,5})/gi |
||||
|
) ?? []; |
||||
|
const claimedSymbols = holdingClaims |
||||
|
.map((m) => { |
||||
|
const match = m.match(/([A-Z]{2,5})$/); |
||||
|
return match?.[1]; |
||||
|
}) |
||||
|
.filter(Boolean) as string[]; |
||||
|
|
||||
|
const unknownHoldings = claimedSymbols.filter( |
||||
|
(s) => !holdingSymbols.has(s) |
||||
|
); |
||||
|
if (unknownHoldings.length === 0) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
issues.push( |
||||
|
`Claims about holdings not in data: ${unknownHoldings.join(', ')}` |
||||
|
); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
const score = checks > 0 ? passed / checks : 1; |
||||
|
|
||||
|
return { |
||||
|
clean: issues.length === 0, |
||||
|
score, |
||||
|
issues |
||||
|
}; |
||||
|
} |
||||
|
|
||||
|
function extractTickers(str: string): Set<string> { |
||||
|
const matches = str.match(/\b[A-Z]{2,5}\b/g) ?? []; |
||||
|
return new Set(matches); |
||||
|
} |
||||
|
|
||||
|
function extractDollarAmounts(str: string): number[] { |
||||
|
const matches = str.match(/\$[\d,]+(?:\.\d{1,2})?/g) ?? []; |
||||
|
return matches.map((m) => parseFloat(m.replace(/[$,]/g, ''))); |
||||
|
} |
||||
|
|
||||
|
function isApproximateMatch(a: number, b: number): boolean { |
||||
|
if (a === 0 && b === 0) return true; |
||||
|
const diff = Math.abs(a - b); |
||||
|
const max = Math.max(Math.abs(a), Math.abs(b)); |
||||
|
// Allow 5% tolerance or $1 absolute
|
||||
|
return diff / max < 0.05 || diff < 1; |
||||
|
} |
||||
|
|
||||
|
function extractHoldingSymbols(toolResults: ToolResult[]): Set<string> { |
||||
|
const symbols = new Set<string>(); |
||||
|
|
||||
|
for (const tr of toolResults) { |
||||
|
if ( |
||||
|
tr.toolName === 'holdings_lookup' || |
||||
|
tr.toolName === 'portfolio_analysis' |
||||
|
) { |
||||
|
const str = JSON.stringify(tr.result); |
||||
|
const matches = str.match(/"symbol"\s*:\s*"([^"]+)"/g) ?? []; |
||||
|
for (const m of matches) { |
||||
|
const sym = /"symbol"\s*:\s*"([^"]+)"/.exec(m)?.[1]; |
||||
|
if (sym) symbols.add(sym); |
||||
|
} |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
return symbols; |
||||
|
} |
||||
@ -0,0 +1,9 @@ |
|||||
|
export { computeConfidence } from './confidence-score'; |
||||
|
export type { ConfidenceScoreResult } from './confidence-score'; |
||||
|
export { checkHallucination } from './hallucination-check'; |
||||
|
export type { |
||||
|
HallucinationCheckResult, |
||||
|
ToolResult |
||||
|
} from './hallucination-check'; |
||||
|
export { validateOutput } from './output-validation'; |
||||
|
export type { OutputValidationResult } from './output-validation'; |
||||
@ -0,0 +1,89 @@ |
|||||
|
export interface OutputValidationResult { |
||||
|
valid: boolean; |
||||
|
score: number; |
||||
|
issues: string[]; |
||||
|
} |
||||
|
|
||||
|
const MIN_LENGTH = 10; |
||||
|
const MAX_LENGTH = 10_000; |
||||
|
const FORWARD_LOOKING_PATTERNS = |
||||
|
/\b(will|should|expect|predict|forecast|going to|likely to)\b.*\b(grow|rise|fall|drop|increase|decrease|return)\b/i; |
||||
|
const DISCLAIMER_PATTERNS = |
||||
|
/not (financial|investment) advice|past performance|no guarantee/i; |
||||
|
const WRITE_CLAIM_PATTERNS = |
||||
|
/\b(created|deleted|removed|updated|transferred|recorded|added)\b/i; |
||||
|
const WRITE_TOOLS = [ |
||||
|
'account_manage', |
||||
|
'activity_manage', |
||||
|
'watchlist_manage', |
||||
|
'tag_manage' |
||||
|
]; |
||||
|
|
||||
|
export function validateOutput({ |
||||
|
text, |
||||
|
toolCalls |
||||
|
}: { |
||||
|
text: string; |
||||
|
toolCalls: string[]; |
||||
|
}): OutputValidationResult { |
||||
|
const issues: string[] = []; |
||||
|
let checks = 0; |
||||
|
let passed = 0; |
||||
|
|
||||
|
// Non-empty
|
||||
|
checks++; |
||||
|
if (text.trim().length >= MIN_LENGTH) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
issues.push(`Response too short (${text.trim().length} chars)`); |
||||
|
} |
||||
|
|
||||
|
// Reasonable length
|
||||
|
checks++; |
||||
|
if (text.length <= MAX_LENGTH) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
issues.push(`Response too long (${text.length} chars)`); |
||||
|
} |
||||
|
|
||||
|
// Tool data referenced: if tools were called, response should contain numbers
|
||||
|
if (toolCalls.length > 0) { |
||||
|
checks++; |
||||
|
if (/\d/.test(text)) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
issues.push('Tools called but no numeric data in response'); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Disclaimer check for forward-looking language
|
||||
|
if (FORWARD_LOOKING_PATTERNS.test(text)) { |
||||
|
checks++; |
||||
|
if (DISCLAIMER_PATTERNS.test(text)) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
issues.push('Forward-looking language without disclaimer'); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
// Write action claim: if response claims a write action, verify a write tool was called
|
||||
|
if (WRITE_CLAIM_PATTERNS.test(text)) { |
||||
|
checks++; |
||||
|
const hasWriteTool = toolCalls.some((tc) => WRITE_TOOLS.includes(tc)); |
||||
|
if (hasWriteTool) { |
||||
|
passed++; |
||||
|
} else { |
||||
|
issues.push( |
||||
|
'Response claims a write action but no write tool was called' |
||||
|
); |
||||
|
} |
||||
|
} |
||||
|
|
||||
|
const score = checks > 0 ? passed / checks : 1; |
||||
|
|
||||
|
return { |
||||
|
valid: issues.length === 0, |
||||
|
score, |
||||
|
issues |
||||
|
}; |
||||
|
} |
||||
Loading…
Reference in new issue