mirror of https://github.com/ghostfolio/ghostfolio
Browse Source
- research.md: 9 architectural decisions (two-tier OCR, Azure+tesseract fallback) - data-model.md: K1ImportSession, CellMapping models, K1ImportStatus enum - contracts/k1-import-api.md: 10 REST endpoints - quickstart.md: file structure, setup guide, workflow - plan.md: summary, technical context, constitution check, project structure - Updated copilot agent context with new tech stack entriespull/6701/head
Robert Patch
2 months ago
6 changed files with 1021 additions and 1 deletions
@ -0,0 +1,381 @@ |
|||
# API Contracts: K-1 Import |
|||
|
|||
**Phase 1 Output** | **Date**: 2026-03-18 |
|||
|
|||
## Base Path |
|||
|
|||
All endpoints under `/api/v1/k1-import/` |
|||
|
|||
## Authentication |
|||
|
|||
All endpoints require JWT authentication (`AuthGuard('jwt')`) and appropriate permissions via `HasPermissionGuard`. |
|||
|
|||
--- |
|||
|
|||
## Endpoints |
|||
|
|||
### POST /api/v1/k1-import/upload |
|||
|
|||
Upload a K-1 PDF and initiate extraction. |
|||
|
|||
**Permission**: `createKDocument` |
|||
|
|||
**Request**: `multipart/form-data` |
|||
|
|||
| Field | Type | Required | Description | |
|||
| --------------- | -------- | -------- | ------------------------------------ | |
|||
| `file` | File | Yes | PDF file (max 25 MB, MIME: application/pdf) | |
|||
| `partnershipId` | `string` | Yes | Target partnership UUID | |
|||
| `taxYear` | `number` | Yes | Tax year for this K-1 | |
|||
|
|||
**Response**: `201 Created` |
|||
|
|||
```json |
|||
{ |
|||
"id": "uuid", |
|||
"partnershipId": "uuid", |
|||
"status": "PROCESSING", |
|||
"taxYear": 2025, |
|||
"fileName": "K1-Smith-Capital-2025.pdf", |
|||
"fileSize": 245760, |
|||
"extractionMethod": "pdf-parse", |
|||
"createdAt": "2026-03-18T00:00:00.000Z" |
|||
} |
|||
``` |
|||
|
|||
**Errors**: |
|||
|
|||
| Status | Condition | |
|||
| ------ | -------------------------------------- | |
|||
| 400 | File is not a valid PDF | |
|||
| 400 | File exceeds 25 MB size limit | |
|||
| 400 | Partnership not found or not owned by user | |
|||
| 400 | Partnership has no active members | |
|||
| 400 | Tax year < partnership inception year | |
|||
|
|||
--- |
|||
|
|||
### GET /api/v1/k1-import/:id |
|||
|
|||
Get the current state of an import session, including extraction results. |
|||
|
|||
**Permission**: `readKDocument` |
|||
|
|||
**Response**: `200 OK` |
|||
|
|||
```json |
|||
{ |
|||
"id": "uuid", |
|||
"partnershipId": "uuid", |
|||
"status": "EXTRACTED", |
|||
"taxYear": 2025, |
|||
"fileName": "K1-Smith-Capital-2025.pdf", |
|||
"fileSize": 245760, |
|||
"extractionMethod": "pdf-parse", |
|||
"rawExtraction": { |
|||
"metadata": { |
|||
"partnershipName": "Smith Capital Partners LP", |
|||
"partnershipEin": "12-3456789", |
|||
"partnerName": "Smith Family Trust", |
|||
"partnerEin": "98-7654321", |
|||
"taxYear": 2025, |
|||
"isAmended": false, |
|||
"isFinal": true |
|||
}, |
|||
"fields": [ |
|||
{ |
|||
"boxNumber": "1", |
|||
"label": "Ordinary business income (loss)", |
|||
"customLabel": null, |
|||
"rawValue": "$52,340", |
|||
"numericValue": 52340, |
|||
"confidence": 0.95, |
|||
"confidenceLevel": "HIGH", |
|||
"isUserEdited": false |
|||
} |
|||
], |
|||
"overallConfidence": 0.92, |
|||
"method": "pdf-parse", |
|||
"pagesProcessed": 2 |
|||
}, |
|||
"verifiedData": null, |
|||
"documentId": "uuid", |
|||
"kDocumentId": null, |
|||
"errorMessage": null, |
|||
"createdAt": "2026-03-18T00:00:00.000Z", |
|||
"updatedAt": "2026-03-18T00:00:05.000Z" |
|||
} |
|||
``` |
|||
|
|||
**Errors**: |
|||
|
|||
| Status | Condition | |
|||
| ------ | ----------------------------------- | |
|||
| 404 | Import session not found | |
|||
| 403 | Import session belongs to different user | |
|||
|
|||
--- |
|||
|
|||
### PUT /api/v1/k1-import/:id/verify |
|||
|
|||
Submit user-verified/edited extraction data. Transitions status from EXTRACTED to VERIFIED. |
|||
|
|||
**Permission**: `updateKDocument` |
|||
|
|||
**Request**: `application/json` |
|||
|
|||
```json |
|||
{ |
|||
"taxYear": 2025, |
|||
"fields": [ |
|||
{ |
|||
"boxNumber": "1", |
|||
"label": "Ordinary business income (loss)", |
|||
"customLabel": null, |
|||
"rawValue": "$52,340", |
|||
"numericValue": 52340, |
|||
"confidence": 0.95, |
|||
"confidenceLevel": "HIGH", |
|||
"isUserEdited": false |
|||
}, |
|||
{ |
|||
"boxNumber": "11", |
|||
"label": "Other income (loss)", |
|||
"customLabel": "Section 1256 contracts", |
|||
"rawValue": "$8,200", |
|||
"numericValue": 8200, |
|||
"confidence": 0.72, |
|||
"confidenceLevel": "MEDIUM", |
|||
"isUserEdited": true |
|||
} |
|||
] |
|||
} |
|||
``` |
|||
|
|||
**Response**: `200 OK` — Updated import session with status `VERIFIED` |
|||
|
|||
**Errors**: |
|||
|
|||
| Status | Condition | |
|||
| ------ | ----------------------------------------------- | |
|||
| 400 | Import session is not in EXTRACTED status | |
|||
| 400 | Fields array is empty | |
|||
| 404 | Import session not found | |
|||
|
|||
--- |
|||
|
|||
### POST /api/v1/k1-import/:id/confirm |
|||
|
|||
Confirm verified data and trigger automatic model object creation (KDocument, Distributions, Document linkage). |
|||
|
|||
**Permission**: `createKDocument` |
|||
|
|||
**Request**: `application/json` |
|||
|
|||
```json |
|||
{ |
|||
"filingStatus": "DRAFT", |
|||
"existingKDocumentAction": null |
|||
} |
|||
``` |
|||
|
|||
| Field | Type | Required | Description | |
|||
| ------------------------- | ----------------------------- | -------- | ---------------------------------------- | |
|||
| `filingStatus` | `"DRAFT" \| "ESTIMATED" \| "FINAL"` | Yes | Status for the created/updated KDocument | |
|||
| `existingKDocumentAction` | `"UPDATE" \| "CREATE_NEW" \| null` | No | Action if KDocument already exists | |
|||
|
|||
**Response**: `201 Created` |
|||
|
|||
```json |
|||
{ |
|||
"importSession": { |
|||
"id": "uuid", |
|||
"status": "CONFIRMED" |
|||
}, |
|||
"kDocument": { |
|||
"id": "uuid", |
|||
"partnershipId": "uuid", |
|||
"type": "K1", |
|||
"taxYear": 2025, |
|||
"filingStatus": "DRAFT", |
|||
"data": { "ordinaryIncome": 52340, "..." : "..." } |
|||
}, |
|||
"distributions": [ |
|||
{ |
|||
"id": "uuid", |
|||
"entityId": "uuid", |
|||
"partnershipId": "uuid", |
|||
"type": "RETURN_OF_CAPITAL", |
|||
"amount": 60000, |
|||
"date": "2025-12-31T00:00:00.000Z" |
|||
} |
|||
], |
|||
"allocations": [ |
|||
{ |
|||
"entityId": "uuid", |
|||
"entityName": "Smith Family Trust", |
|||
"ownershipPercent": 60, |
|||
"allocatedValues": { "ordinaryIncome": 31404, "..." : "..." } |
|||
} |
|||
], |
|||
"document": { |
|||
"id": "uuid", |
|||
"type": "K1", |
|||
"name": "K1-Smith-Capital-2025.pdf" |
|||
} |
|||
} |
|||
``` |
|||
|
|||
**Errors**: |
|||
|
|||
| Status | Condition | |
|||
| ------ | ------------------------------------------------------------- | |
|||
| 400 | Import session is not in VERIFIED status | |
|||
| 400 | Partnership has no active members | |
|||
| 409 | KDocument already exists for this partnership/year and no action specified | |
|||
|
|||
--- |
|||
|
|||
### POST /api/v1/k1-import/:id/cancel |
|||
|
|||
Cancel an import session. No model objects are created. |
|||
|
|||
**Permission**: `updateKDocument` |
|||
|
|||
**Response**: `200 OK` — Updated import session with status `CANCELLED` |
|||
|
|||
**Errors**: |
|||
|
|||
| Status | Condition | |
|||
| ------ | --------------------------------------------- | |
|||
| 400 | Import session is already CONFIRMED or CANCELLED | |
|||
| 404 | Import session not found | |
|||
|
|||
--- |
|||
|
|||
### GET /api/v1/k1-import/history |
|||
|
|||
List import sessions for a partnership, ordered by creation date descending. |
|||
|
|||
**Permission**: `readKDocument` |
|||
|
|||
**Query Parameters**: |
|||
|
|||
| Param | Type | Required | Description | |
|||
| --------------- | -------- | -------- | ------------------------------ | |
|||
| `partnershipId` | `string` | Yes | Partnership UUID | |
|||
| `taxYear` | `number` | No | Filter by tax year | |
|||
|
|||
**Response**: `200 OK` — Array of import session summaries |
|||
|
|||
```json |
|||
[ |
|||
{ |
|||
"id": "uuid", |
|||
"partnershipId": "uuid", |
|||
"status": "CONFIRMED", |
|||
"taxYear": 2025, |
|||
"fileName": "K1-Smith-Capital-2025.pdf", |
|||
"extractionMethod": "pdf-parse", |
|||
"kDocumentId": "uuid", |
|||
"createdAt": "2026-03-18T00:00:00.000Z" |
|||
} |
|||
] |
|||
``` |
|||
|
|||
--- |
|||
|
|||
### POST /api/v1/k1-import/:id/reprocess |
|||
|
|||
Re-run extraction on a previously uploaded PDF using the current cell mapping configuration. |
|||
|
|||
**Permission**: `updateKDocument` |
|||
|
|||
**Response**: `200 OK` — New import session with status `PROCESSING` (original session unchanged) |
|||
|
|||
**Errors**: |
|||
|
|||
| Status | Condition | |
|||
| ------ | ------------------------------------------- | |
|||
| 400 | Original import session has no stored document | |
|||
| 404 | Import session not found | |
|||
|
|||
--- |
|||
|
|||
## Cell Mapping Endpoints |
|||
|
|||
### GET /api/v1/cell-mapping |
|||
|
|||
Get cell mappings for a partnership (with global defaults for unmapped boxes). |
|||
|
|||
**Permission**: `readKDocument` |
|||
|
|||
**Query Parameters**: |
|||
|
|||
| Param | Type | Required | Description | |
|||
| --------------- | -------- | -------- | ---------------------------------------- | |
|||
| `partnershipId` | `string` | No | Partnership UUID (omit for global defaults) | |
|||
|
|||
**Response**: `200 OK` |
|||
|
|||
```json |
|||
[ |
|||
{ |
|||
"id": "uuid", |
|||
"partnershipId": null, |
|||
"boxNumber": "1", |
|||
"label": "Ordinary business income (loss)", |
|||
"description": "IRS Schedule K-1 Box 1", |
|||
"isCustom": false, |
|||
"sortOrder": 1 |
|||
} |
|||
] |
|||
``` |
|||
|
|||
--- |
|||
|
|||
### PUT /api/v1/cell-mapping |
|||
|
|||
Update or create cell mappings for a partnership. |
|||
|
|||
**Permission**: `updateKDocument` |
|||
|
|||
**Request**: `application/json` |
|||
|
|||
```json |
|||
{ |
|||
"partnershipId": "uuid", |
|||
"mappings": [ |
|||
{ |
|||
"boxNumber": "11", |
|||
"label": "Section 1256 contracts", |
|||
"description": "Custom label for Box 11", |
|||
"isCustom": false |
|||
}, |
|||
{ |
|||
"boxNumber": "20-Z", |
|||
"label": "Qualified Business Income (Section 199A)", |
|||
"description": "Custom additional box", |
|||
"isCustom": true |
|||
} |
|||
] |
|||
} |
|||
``` |
|||
|
|||
**Response**: `200 OK` — Updated mappings array |
|||
|
|||
--- |
|||
|
|||
### DELETE /api/v1/cell-mapping/reset |
|||
|
|||
Reset a partnership's cell mappings to IRS defaults (deletes all custom mappings for the partnership). |
|||
|
|||
**Permission**: `updateKDocument` |
|||
|
|||
**Query Parameters**: |
|||
|
|||
| Param | Type | Required | Description | |
|||
| --------------- | -------- | -------- | ------------------ | |
|||
| `partnershipId` | `string` | Yes | Partnership UUID | |
|||
|
|||
**Response**: `200 OK` |
|||
@ -0,0 +1,241 @@ |
|||
# Data Model: K-1 PDF Scan Import |
|||
|
|||
**Phase 1 Output** | **Date**: 2026-03-18 |
|||
|
|||
## Overview |
|||
|
|||
This feature adds 2 new Prisma models and 1 new enum to support K-1 PDF scanning, import session tracking, and cell mapping configuration. It extends the existing models from spec 001-family-office-transform (KDocument, Distribution, Document, PartnershipMembership) with automatic creation from scanned data. |
|||
|
|||
### Entity Relationship Diagram (Conceptual) |
|||
|
|||
``` |
|||
User (existing) |
|||
└── Partnership (existing from 001) |
|||
├── K1ImportSession[] ──┬── Document (uploaded PDF, existing from 001) |
|||
│ (new model) ├── KDocument (auto-created, existing from 001) |
|||
│ └── CellMapping (per-partnership config) |
|||
├── PartnershipMembership[] (existing from 001) |
|||
│ └── [K-1 allocations computed at confirm time] |
|||
├── KDocument[] (existing from 001) |
|||
│ └── Distribution[] (auto-created from Box 19, existing from 001) |
|||
└── CellMapping[] (new model, per-partnership overrides) |
|||
|
|||
Global CellMapping (partnershipId = null) ── IRS default box definitions |
|||
``` |
|||
|
|||
## New Enum |
|||
|
|||
### K1ImportStatus |
|||
|
|||
Tracks the lifecycle of a K-1 import session. |
|||
|
|||
| Value | Description | |
|||
| ------------ | -------------------------------------------------------------- | |
|||
| `PROCESSING` | PDF uploaded, extraction in progress | |
|||
| `EXTRACTED` | Extraction complete, awaiting user review | |
|||
| `VERIFIED` | User has reviewed/edited values, ready for confirmation | |
|||
| `CONFIRMED` | User confirmed, model objects created (KDocument, Distributions) | |
|||
| `CANCELLED` | User cancelled, no model objects created | |
|||
| `FAILED` | Extraction failed (invalid PDF, OCR error, etc.) | |
|||
|
|||
## New Models |
|||
|
|||
### K1ImportSession |
|||
|
|||
A record of a single K-1 PDF import attempt, tracking the full lifecycle from upload through confirmation. |
|||
|
|||
| Field | Type | Constraints | Description | |
|||
| ------------------ | ---------------- | ---------------------------- | --------------------------------------------------------------- | |
|||
| `id` | `String` | PK, UUID, auto-generated | Unique identifier | |
|||
| `partnershipId` | `String` | FK → Partnership.id, indexed | Target partnership for this K-1 import | |
|||
| `userId` | `String` | FK → User.id, indexed | User who initiated the import | |
|||
| `status` | `K1ImportStatus` | Required, Default: PROCESSING | Current lifecycle status | |
|||
| `taxYear` | `Int` | Required | Tax year extracted or specified by user | |
|||
| `fileName` | `String` | Required | Original filename of uploaded PDF | |
|||
| `fileSize` | `Int` | Required | File size in bytes | |
|||
| `extractionMethod` | `String` | Required | Method used: "pdf-parse", "azure", "tesseract" | |
|||
| `rawExtraction` | `Json?` | Optional | Raw extraction results before user edits | |
|||
| `verifiedData` | `Json?` | Optional | User-verified/edited extraction results (K1ExtractionResult) | |
|||
| `documentId` | `String?` | FK → Document.id, optional | Linked uploaded PDF Document record | |
|||
| `kDocumentId` | `String?` | FK → KDocument.id, optional | Resulting KDocument (set on CONFIRMED status) | |
|||
| `errorMessage` | `String?` | Optional | Error details if status is FAILED | |
|||
| `createdAt` | `DateTime` | Default: now() | Upload timestamp | |
|||
| `updatedAt` | `DateTime` | Auto-updated | Last modification timestamp | |
|||
|
|||
**Relations**: |
|||
|
|||
- `partnership` → `Partnership` (many-to-one, cascade delete) |
|||
- `user` → `User` (many-to-one, cascade delete) |
|||
- `document` → `Document?` (many-to-one, optional) |
|||
- `kDocument` → `KDocument?` (many-to-one, optional) |
|||
|
|||
**Indexes**: `@@index([partnershipId, taxYear])` for import history queries per partnership/year. |
|||
|
|||
### CellMapping |
|||
|
|||
A configuration defining how K-1 box numbers map to labels. Supports a global IRS-default set (partnershipId = null) and per-partnership customizations. |
|||
|
|||
| Field | Type | Constraints | Description | |
|||
| --------------- | ---------- | -------------------------------------- | ---------------------------------------------------- | |
|||
| `id` | `String` | PK, UUID, auto-generated | Unique identifier | |
|||
| `partnershipId` | `String?` | FK → Partnership.id, optional, indexed | Partnership this mapping applies to (null = global) | |
|||
| `boxNumber` | `String` | Required | K-1 box identifier (e.g., "1", "6a", "19a", "20-A") | |
|||
| `label` | `String` | Required | Display label (e.g., "Ordinary business income") | |
|||
| `description` | `String?` | Optional | Extended description or IRS instructions | |
|||
| `isCustom` | `Boolean` | Default: false | Whether this is a user-added custom cell | |
|||
| `sortOrder` | `Int` | Required | Display order in the verification screen | |
|||
| `createdAt` | `DateTime` | Default: now() | Creation timestamp | |
|||
| `updatedAt` | `DateTime` | Auto-updated | Last modification timestamp | |
|||
|
|||
**Relations**: |
|||
|
|||
- `partnership` → `Partnership?` (many-to-one, optional, cascade delete) |
|||
|
|||
**Unique constraint**: `@@unique([partnershipId, boxNumber])` — one mapping per box per partnership (or per box globally when partnershipId is null). |
|||
|
|||
## Modifications to Existing Models |
|||
|
|||
### Partnership (from spec 001) |
|||
|
|||
Add back-references — no column changes: |
|||
|
|||
| New Field | Type | Description | |
|||
| ----------------- | -------------------- | ------------------------------------ | |
|||
| `importSessions` | `K1ImportSession[]` | Import attempts for this partnership | |
|||
| `cellMappings` | `CellMapping[]` | Custom cell mapping configurations | |
|||
|
|||
### KDocument (from spec 001) |
|||
|
|||
Add back-reference — no column changes: |
|||
|
|||
| New Field | Type | Description | |
|||
| ---------------- | ------------------- | ---------------------------------------- | |
|||
| `importSession` | `K1ImportSession?` | Import session that created this record | |
|||
|
|||
## Application-Layer Types |
|||
|
|||
### K1ExtractionResult (TypeScript interface) |
|||
|
|||
The structure returned by the extraction service and stored in `K1ImportSession.rawExtraction` and `K1ImportSession.verifiedData`. |
|||
|
|||
```typescript |
|||
interface K1ExtractionResult { |
|||
/** Extracted metadata from the K-1 header */ |
|||
metadata: { |
|||
partnershipName: string | null; |
|||
partnershipEin: string | null; |
|||
partnerName: string | null; |
|||
partnerEin: string | null; |
|||
taxYear: number | null; |
|||
isAmended: boolean; |
|||
isFinal: boolean; |
|||
}; |
|||
|
|||
/** Extracted box values */ |
|||
fields: K1ExtractedField[]; |
|||
|
|||
/** Overall extraction confidence (0.0–1.0) */ |
|||
overallConfidence: number; |
|||
|
|||
/** Extraction method used */ |
|||
method: 'pdf-parse' | 'azure' | 'tesseract'; |
|||
|
|||
/** Number of pages processed */ |
|||
pagesProcessed: number; |
|||
} |
|||
|
|||
interface K1ExtractedField { |
|||
/** Box identifier (e.g., "1", "6a", "19a") */ |
|||
boxNumber: string; |
|||
|
|||
/** Display label from cell mapping */ |
|||
label: string; |
|||
|
|||
/** Custom label override by user (null if not overridden) */ |
|||
customLabel: string | null; |
|||
|
|||
/** Extracted raw text value */ |
|||
rawValue: string; |
|||
|
|||
/** Parsed numeric value (null if unparseable) */ |
|||
numericValue: number | null; |
|||
|
|||
/** Confidence score (0.0–1.0) */ |
|||
confidence: number; |
|||
|
|||
/** Confidence level for display */ |
|||
confidenceLevel: 'HIGH' | 'MEDIUM' | 'LOW'; |
|||
|
|||
/** Whether user has manually edited this value */ |
|||
isUserEdited: boolean; |
|||
} |
|||
``` |
|||
|
|||
### K1ConfirmationRequest (TypeScript interface) |
|||
|
|||
The request body when the user confirms verified K-1 data. |
|||
|
|||
```typescript |
|||
interface K1ConfirmationRequest { |
|||
/** Import session ID */ |
|||
importSessionId: string; |
|||
|
|||
/** Tax year (may have been overridden by user) */ |
|||
taxYear: number; |
|||
|
|||
/** Filing status for the new KDocument */ |
|||
filingStatus: 'DRAFT' | 'ESTIMATED' | 'FINAL'; |
|||
|
|||
/** Verified fields with any user edits applied */ |
|||
fields: K1ExtractedField[]; |
|||
|
|||
/** Whether to update an existing KDocument (null = create new) */ |
|||
existingKDocumentAction: 'UPDATE' | 'CREATE_NEW' | null; |
|||
} |
|||
``` |
|||
|
|||
### Default IRS K-1 Cell Mapping |
|||
|
|||
The standard box definitions seeded as global CellMapping records (partnershipId = null): |
|||
|
|||
| boxNumber | label | sortOrder | |
|||
| --------- | ----------------------------------------- | --------- | |
|||
| 1 | Ordinary business income (loss) | 1 | |
|||
| 2 | Net rental real estate income (loss) | 2 | |
|||
| 3 | Other net rental income (loss) | 3 | |
|||
| 4 | Guaranteed payments for services | 4 | |
|||
| 4a | Guaranteed payments for capital | 5 | |
|||
| 4b | Total guaranteed payments | 6 | |
|||
| 5 | Interest income | 7 | |
|||
| 6a | Ordinary dividends | 8 | |
|||
| 6b | Qualified dividends | 9 | |
|||
| 6c | Dividend equivalents | 10 | |
|||
| 7 | Royalties | 11 | |
|||
| 8 | Net short-term capital gain (loss) | 12 | |
|||
| 9a | Net long-term capital gain (loss) | 13 | |
|||
| 9b | Collectibles (28%) gain (loss) | 14 | |
|||
| 9c | Unrecaptured section 1250 gain | 15 | |
|||
| 10 | Net section 1231 gain (loss) | 16 | |
|||
| 11 | Other income (loss) | 17 | |
|||
| 12 | Section 179 deduction | 18 | |
|||
| 13 | Other deductions | 19 | |
|||
| 14 | Self-employment earnings (loss) | 20 | |
|||
| 15 | Credits | 21 | |
|||
| 16 | Foreign transactions | 22 | |
|||
| 17 | Alternative minimum tax (AMT) items | 23 | |
|||
| 18 | Tax-exempt income and nondeductible expenses | 24 | |
|||
| 19a | Distributions — Cash and marketable securities | 25 | |
|||
| 19b | Distributions — Other property | 26 | |
|||
| 20 | Other information | 27 | |
|||
| 21 | Foreign taxes paid or accrued | 28 | |
|||
|
|||
## Validation Rules |
|||
|
|||
1. **Import session partnership**: Must reference an existing partnership owned by the current user. |
|||
2. **Import session tax year**: Must be ≥ year of the partnership's inception date. |
|||
3. **File upload**: Must be a valid PDF, ≤ 25 MB. System rejects non-PDF MIME types. |
|||
4. **Extraction status transitions**: Only valid transitions: PROCESSING → EXTRACTED → VERIFIED → CONFIRMED/CANCELLED, or PROCESSING → FAILED. No backwards transitions. |
|||
5. **Cell mapping uniqueness**: One mapping per (partnershipId, boxNumber). Custom mappings for a partnership override the global default for that box number. |
|||
6. **Confirmation prerequisites**: Can only confirm when status is VERIFIED, partnership has at least one active member, and verifiedData is not null. |
|||
7. **Duplicate KDocument check**: Before creating a KDocument, check for existing (partnershipId, type=K1, taxYear). If found, require explicit user decision (update existing or reject). |
|||
8. **Distribution allocation**: Box 19a/19b amounts are allocated to members by ownership percentage as of the tax year's fiscal year end. Allocation amounts must sum exactly to the partnership-level total (handle rounding by adjusting the largest member's allocation). |
|||
@ -0,0 +1,121 @@ |
|||
# Implementation Plan: K-1 PDF Scan Import |
|||
|
|||
**Branch**: `004-k1-scan-import` | **Date**: 2026-03-18 | **Spec**: [spec.md](spec.md) |
|||
**Input**: Feature specification from `/specs/004-k1-scan-import/spec.md` |
|||
|
|||
## Summary |
|||
|
|||
Automated K-1 PDF scanning that extracts structured IRS Schedule K-1 (Form 1065) data from uploaded PDFs, presents a verification screen for manual review/correction, and auto-creates downstream model objects (KDocument, Distributions, member allocations, Document). Uses a two-tier extraction approach: `pdf-parse` for digital PDFs (free, instant, local) and Azure AI Document Intelligence / `tesseract.js` fallback for scanned PDFs. Supports per-partnership cell mapping customization and import history with re-processing. |
|||
|
|||
## Technical Context |
|||
|
|||
**Language/Version**: TypeScript 5.9.2, Node.js ≥ 22.18.0 |
|||
**Primary Dependencies**: NestJS 11.x (backend), Angular 21.x (frontend), Prisma 6.x (ORM), pdf-parse (PDF text), @azure/ai-form-recognizer (cloud OCR), tesseract.js (local OCR fallback) |
|||
**Storage**: PostgreSQL via Prisma (structured data), local filesystem `uploads/` (PDF files) |
|||
**Testing**: Jest (unit + integration), test K-1 PDF fixtures in `test/import/` |
|||
**Target Platform**: Docker (node:22-slim), self-hosted or Railway |
|||
**Project Type**: Web application (NestJS API + Angular SPA) — Nx monorepo |
|||
**Performance Goals**: PDF extraction < 30 seconds (SC-001), model creation < 5 seconds (SC-005), 90%+ accuracy for digital PDFs (SC-002) |
|||
**Constraints**: Self-hosted capable (Azure OCR optional), max PDF size 25 MB, K-1 Form 1065 only (V1) |
|||
**Scale/Scope**: Single family office (10–50 partnerships, 10–50 K-1s/year), 2 new API modules, 3 new frontend pages |
|||
|
|||
## Constitution Check |
|||
|
|||
_GATE: Must pass before Phase 0 research. Re-check after Phase 1 design._ |
|||
|
|||
No constitution.md exists for this project. Gates assessed against standard engineering principles: |
|||
|
|||
| Gate | Status | Notes | |
|||
|------|--------|-------| |
|||
| No unnecessary dependencies | PASS | 3 new packages (`pdf-parse`, `@azure/ai-form-recognizer`, `tesseract.js`) — each serves a distinct, justified purpose per research.md | |
|||
| Follows existing patterns | PASS | New NestJS modules follow existing controller/service/DTO pattern (mirrors `k-document`, `upload` modules) | |
|||
| No breaking changes | PASS | 2 new Prisma models + 1 enum, back-references only on existing models — no column changes | |
|||
| Test coverage | PASS | Unit tests for extractors, mapper, allocation; integration tests for full pipeline | |
|||
| Self-hosted compatible | PASS | Core extraction (pdf-parse) is fully local; Azure is optional with tesseract.js fallback | |
|||
|
|||
**Post-Phase 1 re-check**: PASS — data model adds 2 models/1 enum, no existing schema changes beyond back-references. API contracts follow existing REST patterns. No violations identified. |
|||
|
|||
## Project Structure |
|||
|
|||
### Documentation (this feature) |
|||
|
|||
```text |
|||
specs/004-k1-scan-import/ |
|||
├── plan.md # This file |
|||
├── research.md # Phase 0: OCR provider research & decisions |
|||
├── data-model.md # Phase 1: K1ImportSession, CellMapping models |
|||
├── quickstart.md # Phase 1: Setup & dev guide |
|||
├── contracts/ |
|||
│ └── k1-import-api.md # Phase 1: REST API contracts |
|||
├── checklists/ |
|||
│ └── requirements.md # Spec quality checklist |
|||
└── tasks.md # Phase 2 output (created by /speckit.tasks) |
|||
``` |
|||
|
|||
### Source Code (repository root) |
|||
|
|||
```text |
|||
apps/api/src/app/ |
|||
├── k1-import/ |
|||
│ ├── k1-import.module.ts |
|||
│ ├── k1-import.controller.ts |
|||
│ ├── k1-import.service.ts |
|||
│ ├── dto/ |
|||
│ │ ├── upload-k1.dto.ts |
|||
│ │ ├── verify-k1.dto.ts |
|||
│ │ └── confirm-k1.dto.ts |
|||
│ ├── extractors/ |
|||
│ │ ├── k1-extractor.interface.ts |
|||
│ │ ├── pdf-parse-extractor.ts |
|||
│ │ ├── azure-extractor.ts |
|||
│ │ └── tesseract-extractor.ts |
|||
│ ├── k1-field-mapper.service.ts |
|||
│ ├── k1-allocation.service.ts |
|||
│ └── k1-confidence.service.ts |
|||
├── cell-mapping/ |
|||
│ ├── cell-mapping.module.ts |
|||
│ ├── cell-mapping.controller.ts |
|||
│ └── cell-mapping.service.ts |
|||
|
|||
apps/client/src/app/ |
|||
├── pages/ |
|||
│ ├── k1-import/ |
|||
│ │ ├── k1-import-page.component.ts |
|||
│ │ ├── k1-import-page.html |
|||
│ │ ├── k1-import-page.scss |
|||
│ │ ├── k1-import-page.routes.ts |
|||
│ │ ├── k1-verification/ |
|||
│ │ │ ├── k1-verification.component.ts |
|||
│ │ │ ├── k1-verification.html |
|||
│ │ │ └── k1-verification.scss |
|||
│ │ └── k1-confirmation/ |
|||
│ │ ├── k1-confirmation.component.ts |
|||
│ │ ├── k1-confirmation.html |
|||
│ │ └── k1-confirmation.scss |
|||
│ └── cell-mapping/ |
|||
│ ├── cell-mapping-page.component.ts |
|||
│ ├── cell-mapping-page.html |
|||
│ └── cell-mapping-page.routes.ts |
|||
├── services/ |
|||
│ └── k1-import-data.service.ts |
|||
|
|||
libs/common/src/lib/ |
|||
├── interfaces/ |
|||
│ └── k1-import.interface.ts |
|||
├── dtos/ |
|||
│ └── k1-import/ |
|||
│ ├── create-k1-import.dto.ts |
|||
│ ├── verify-k1-import.dto.ts |
|||
│ └── confirm-k1-import.dto.ts |
|||
|
|||
prisma/ |
|||
├── schema.prisma # + K1ImportSession, CellMapping, K1ImportStatus |
|||
├── migrations/ |
|||
│ └── 2026XXXX_added_k1_import/ # New migration |
|||
|
|||
test/import/ |
|||
├── sample-k1-digital.pdf # Test fixture: digital K-1 |
|||
└── sample-k1-scanned.pdf # Test fixture: scanned K-1 |
|||
``` |
|||
|
|||
**Structure Decision**: Follows the existing Nx monorepo convention with new NestJS modules under `apps/api/src/app/` and new Angular pages under `apps/client/src/app/pages/`. Shared interfaces and DTOs in `libs/common/`. This mirrors the existing `k-document`, `upload`, and `family-office` module patterns. |
|||
@ -0,0 +1,120 @@ |
|||
# Quickstart: K-1 PDF Scan Import |
|||
|
|||
**Phase 1 Output** | **Date**: 2026-03-18 |
|||
|
|||
## Prerequisites |
|||
|
|||
1. Spec 001-family-office-transform models are implemented (Entity, Partnership, PartnershipMembership, KDocument, Distribution, Document) |
|||
2. At least one Partnership with one or more member Entities exists in the database |
|||
3. The existing upload infrastructure (`UploadController`, `uploads/` directory) is functional |
|||
4. Node.js ≥ 22.18.0, Docker for PostgreSQL/Redis |
|||
|
|||
## Environment Setup |
|||
|
|||
Add to `.env` (optional — for Azure OCR of scanned PDFs): |
|||
``` |
|||
AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT=https://your-resource.cognitiveservices.azure.com/ |
|||
AZURE_DOCUMENT_INTELLIGENCE_KEY=your-api-key |
|||
``` |
|||
|
|||
If these are empty, scanned PDFs fall back to `tesseract.js` (lower accuracy but fully self-hosted). |
|||
|
|||
## New Dependencies |
|||
|
|||
```bash |
|||
npm install pdf-parse @azure/ai-form-recognizer tesseract.js |
|||
npm install -D @types/pdf-parse |
|||
``` |
|||
|
|||
## Database Migration |
|||
|
|||
After adding the new Prisma models (`K1ImportSession`, `CellMapping`, `K1ImportStatus` enum): |
|||
|
|||
```bash |
|||
npx prisma db push # Development: sync schema |
|||
# OR |
|||
npx prisma migrate dev # Create a migration file |
|||
``` |
|||
|
|||
Seed the default IRS cell mappings (28 rows with partnershipId = null) via the existing seed mechanism or a dedicated seed script. |
|||
|
|||
## Key Files to Create |
|||
|
|||
### Backend (apps/api/src/) |
|||
|
|||
``` |
|||
app/k1-import/ |
|||
├── k1-import.module.ts # NestJS module |
|||
├── k1-import.controller.ts # REST endpoints (see contracts/k1-import-api.md) |
|||
├── k1-import.service.ts # Orchestration: upload → extract → verify → confirm |
|||
├── dto/ |
|||
│ ├── upload-k1.dto.ts # Multipart upload DTO |
|||
│ ├── verify-k1.dto.ts # Verification submission DTO |
|||
│ └── confirm-k1.dto.ts # Confirmation request DTO |
|||
├── extractors/ |
|||
│ ├── k1-extractor.interface.ts # Common extraction interface |
|||
│ ├── pdf-parse-extractor.ts # Tier 1: digital PDF text extraction |
|||
│ ├── azure-extractor.ts # Tier 2: Azure Document Intelligence |
|||
│ └── tesseract-extractor.ts # Tier 2 fallback: tesseract.js OCR |
|||
├── k1-field-mapper.service.ts # Maps raw extraction → K1ExtractedField[] |
|||
├── k1-allocation.service.ts # Allocates K-1 amounts to members by ownership % |
|||
└── k1-confidence.service.ts # Computes confidence scores with validation heuristics |
|||
|
|||
app/cell-mapping/ |
|||
├── cell-mapping.module.ts # NestJS module |
|||
├── cell-mapping.controller.ts # CRUD for cell mappings |
|||
└── cell-mapping.service.ts # Cell mapping business logic + seed data |
|||
``` |
|||
|
|||
### Shared Types (libs/common/src/lib/) |
|||
|
|||
``` |
|||
interfaces/ |
|||
├── k1-import.interface.ts # K1ExtractionResult, K1ExtractedField, K1ConfirmationRequest |
|||
dtos/ |
|||
├── k1-import/ |
|||
│ ├── create-k1-import.dto.ts |
|||
│ ├── verify-k1-import.dto.ts |
|||
│ └── confirm-k1-import.dto.ts |
|||
``` |
|||
|
|||
### Frontend (apps/client/src/app/) |
|||
|
|||
``` |
|||
pages/k1-import/ |
|||
├── k1-import-page.component.ts # Upload + history view |
|||
├── k1-import-page.html |
|||
├── k1-import-page.scss |
|||
├── k1-import-page.routes.ts |
|||
├── k1-verification/ |
|||
│ ├── k1-verification.component.ts # Verification/edit screen |
|||
│ ├── k1-verification.html |
|||
│ └── k1-verification.scss |
|||
└── k1-confirmation/ |
|||
├── k1-confirmation.component.ts # Confirmation result screen |
|||
├── k1-confirmation.html |
|||
└── k1-confirmation.scss |
|||
|
|||
pages/cell-mapping/ |
|||
├── cell-mapping-page.component.ts # Cell mapping configuration UI |
|||
├── cell-mapping-page.html |
|||
└── cell-mapping-page.routes.ts |
|||
|
|||
services/ |
|||
├── k1-import-data.service.ts # HTTP client for k1-import endpoints |
|||
``` |
|||
|
|||
## Verification Workflow |
|||
|
|||
1. **Upload**: User selects PDF → `POST /api/v1/k1-import/upload` → session created with status PROCESSING |
|||
2. **Extract**: Backend detects PDF type (digital vs. scanned) → routes to appropriate extractor → status becomes EXTRACTED |
|||
3. **Review**: Frontend polls/fetches session → displays verification screen with extracted fields, confidence indicators |
|||
4. **Edit**: User corrects values, overrides labels → `PUT /api/v1/k1-import/:id/verify` → status becomes VERIFIED |
|||
5. **Confirm**: User clicks "Confirm & Save" → `POST /api/v1/k1-import/:id/confirm` → KDocument + Distributions + Document created → status becomes CONFIRMED |
|||
|
|||
## Testing Strategy |
|||
|
|||
- **Unit tests**: Extractors (pdf-parse, azure, tesseract), field mapper, confidence scoring, allocation math |
|||
- **Integration tests**: Full upload → extract → verify → confirm flow with test PDF fixtures |
|||
- **Test fixtures**: Include sample K-1 PDFs (digital and scanned) in `test/import/` directory |
|||
- **Allocation accuracy**: Verify rounding behavior — allocated amounts must sum exactly to partnership total |
|||
@ -0,0 +1,154 @@ |
|||
# Research: K-1 PDF Scan Import |
|||
|
|||
**Phase 0 Output** | **Date**: 2026-03-18 |
|||
|
|||
## Decision 1: PDF Text Extraction (Tier 1 — Digital PDFs) |
|||
|
|||
**Decision**: Use `pdf-parse` npm package for digitally-generated K-1 PDFs. |
|||
|
|||
**Rationale**: Digitally-generated PDFs from fund administrators contain embedded text. `pdf-parse` extracts this text losslessly, is free, fully self-hosted, and instant. It has 3M+ weekly npm downloads and a stable API. No external API calls needed. |
|||
|
|||
**Alternatives Considered**: |
|||
- `pdfjs-dist` (Mozilla pdf.js) — lower-level, requires more boilerplate for text extraction; `pdf-parse` wraps this already. |
|||
- Cloud OCR for all PDFs — unnecessary cost and latency for digital PDFs where text extraction is 100% accurate. |
|||
|
|||
--- |
|||
|
|||
## Decision 2: OCR for Scanned PDFs (Tier 2) |
|||
|
|||
**Decision**: Use Azure AI Document Intelligence (Layout model) as primary Tier 2 provider, with `tesseract.js` as self-hosted fallback. |
|||
|
|||
**Rationale**: |
|||
- Azure has the best tax-form pedigree among cloud providers (prebuilt IRS models for W-2, 1098, 1099) |
|||
- Returns per-field confidence scores (0.0–1.0) natively, directly fulfilling FR-006/FR-009 |
|||
- 500 free pages/month covers typical family office volume (10–50 K-1s/year) |
|||
- `@azure/ai-form-recognizer` has full TypeScript types, aligns with NestJS patterns |
|||
- `tesseract.js` runs as WASM in Node.js (no system install), provides ~75% accuracy fallback |
|||
|
|||
**Alternatives Considered**: |
|||
- Google Document AI — good form parsing but no tax-specific models, more expensive for custom processors ($30/1K pages) |
|||
- AWS Textract — strong table extraction but less established for tax forms, requires IAM setup |
|||
- Tesseract.js only — accuracy drops to 70–85% for clean scans, no layout understanding; acceptable as fallback but not primary |
|||
|
|||
--- |
|||
|
|||
## Decision 3: Two-Tier Extraction Architecture |
|||
|
|||
**Decision**: Implement a PDF type detection step that routes digital PDFs to local extraction (free, instant) and scanned PDFs to cloud OCR. |
|||
|
|||
**Rationale**: Most K-1s from fund administrators are digitally generated. The two-tier approach avoids unnecessary API calls and costs for the majority case, while still supporting scanned documents. |
|||
|
|||
**Detection heuristic**: Extract text via `pdf-parse`; if extracted text length < 100 characters or does not contain K-1 keywords ("Schedule K-1", "Form 1065", "Partner's Share"), route to Tier 2 OCR. |
|||
|
|||
**Alternatives Considered**: |
|||
- Cloud OCR for everything — simpler but adds cost ($0.15/page) and latency (3–10s) for digital PDFs that don't need it |
|||
- Local OCR only (Tesseract.js) — insufficient accuracy (75%) for production tax data; too many manual corrections needed |
|||
|
|||
--- |
|||
|
|||
## Decision 4: K-1 Box Extraction Strategy |
|||
|
|||
**Decision**: Use regex-based box extraction for Tier 1 (digital text), and key-value pair extraction from the OCR provider for Tier 2. Both feed into a shared K-1 field mapper that applies the cell mapping configuration. |
|||
|
|||
**Rationale**: The IRS Schedule K-1 (Form 1065) has a consistent, standardized layout: |
|||
- Page 1: Header + Part I (partnership info) + Part II (partner info) + Boxes 1–11 |
|||
- Page 2: Boxes 12–20+ with code/sub-code details |
|||
- Box values sit in a numbered two-column grid: number label → description → value field |
|||
- Layout has been structurally stable for years, making template/regex extraction reliable |
|||
|
|||
**Challenges addressed**: |
|||
- Multi-line sub-codes (Boxes 11, 13, 15, 16, 17, 18, 20) — handle by extracting code-letter/value pairs within each box section |
|||
- Supplemental schedules — out of scope for V1 auto-extraction; captured as additional Document attachments |
|||
- Multi-entity PDFs — detect via repeated "Schedule K-1" headers; split and process each K-1 separately |
|||
|
|||
**Alternatives Considered**: |
|||
- Fixed coordinate-based extraction — too brittle across different PDF generators (varying margins, fonts) |
|||
- Machine learning model — overkill for V1 given the standardized form layout |
|||
|
|||
--- |
|||
|
|||
## Decision 5: Confidence Scoring Approach |
|||
|
|||
**Decision**: Three-level confidence display (High/Medium/Low) derived from extraction method and validation heuristics. |
|||
|
|||
**Rationale**: |
|||
|
|||
For **Tier 1** (digital text): |
|||
- Base confidence: 0.90 (text extraction is inherently reliable) |
|||
- +0.05 if box number regex matched cleanly |
|||
- +0.05 if value format validated (currency, percentage, integer) |
|||
- -0.10 to -0.30 for potential adjacent-box text contamination |
|||
|
|||
For **Tier 2** (cloud OCR): |
|||
- Use Azure's native per-field confidence score directly |
|||
- Layer cross-field validation (e.g., Box 6b ≤ Box 6a, sub-boxes sum to parent) |
|||
|
|||
**Display mapping**: |
|||
- High (≥ 0.85): Green — no user attention needed |
|||
- Medium (0.60–0.84): Yellow — optional review |
|||
- Low (< 0.60): Red — highlighted, requires manual review (FR-009) |
|||
|
|||
**Alternatives Considered**: |
|||
- Binary confidence (confident/not) — too coarse; doesn't guide the user's review attention |
|||
- Numeric score display — too technical for a non-engineer user; three levels with color coding is more actionable |
|||
|
|||
--- |
|||
|
|||
## Decision 6: New Database Models |
|||
|
|||
**Decision**: Add two new Prisma models (`K1ImportSession`, `CellMapping`) to support import tracking and cell mapping configuration, alongside the existing K-document models from spec 001. |
|||
|
|||
**Rationale**: |
|||
- `K1ImportSession` tracks the full import lifecycle (upload → processing → extracted → verified → confirmed/cancelled), enabling import history (FR-022) and re-processing (FR-023) |
|||
- `CellMapping` stores per-partnership cell label customizations (FR-017 through FR-021) separate from the KDocument data itself |
|||
|
|||
**Alternatives Considered**: |
|||
- Store import sessions as JSON metadata on KDocument — would conflate document data with import workflow state; makes import history harder to query |
|||
- Store cell mappings as JSON on Partnership — would work but loses the ability to query/manage mappings independently and doesn't support a global default set |
|||
|
|||
--- |
|||
|
|||
## Decision 7: File Storage |
|||
|
|||
**Decision**: Use the existing `uploads/` directory and `Document` model from spec 001. Uploaded K-1 PDFs are stored on the local filesystem, with metadata in the `Document` table. |
|||
|
|||
**Rationale**: The existing upload infrastructure (UploadController with `FileInterceptor`, Document model, `uploads/` directory) is already in place. No need to add a new storage mechanism. |
|||
|
|||
**Alternatives Considered**: |
|||
- S3/cloud storage — would require new infrastructure; the self-hosted philosophy favors local storage |
|||
- Database blob storage — increases database size and backup time for binary files |
|||
|
|||
--- |
|||
|
|||
## Decision 8: New Environment Variables |
|||
|
|||
**Decision**: Add two optional environment variables for Azure Document Intelligence, following the existing `ConfigurationService` pattern with `str({ default: '' })`. |
|||
|
|||
``` |
|||
AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT — Azure resource endpoint URL |
|||
AZURE_DOCUMENT_INTELLIGENCE_KEY — Azure API key |
|||
``` |
|||
|
|||
**Rationale**: When both are empty (default), the system falls back to `tesseract.js` for scanned PDFs. This makes Azure optional — the feature works fully self-hosted with degraded OCR accuracy. |
|||
|
|||
**Alternatives Considered**: |
|||
- Separate feature flag — unnecessary; empty credentials are sufficient to indicate "not configured" |
|||
- Google/AWS credentials — Azure recommended as primary; could add additional providers later |
|||
|
|||
--- |
|||
|
|||
## Decision 9: New npm Dependencies |
|||
|
|||
**Decision**: Add the following packages: |
|||
|
|||
| Package | Purpose | Tier | |
|||
|---|---|---| |
|||
| `pdf-parse` | Text extraction from digital PDFs | Tier 1 (required) | |
|||
| `@azure/ai-form-recognizer` | Cloud OCR for scanned PDFs | Tier 2 (optional) | |
|||
| `tesseract.js` | Self-hosted OCR fallback | Tier 2 fallback | |
|||
|
|||
**Rationale**: `pdf-parse` is essential for the Tier 1 (free, local) path. Azure SDK is optional (only loaded when credentials are configured). `tesseract.js` provides a zero-config fallback that runs as WASM — no system dependencies needed, works in the existing `node:22-slim` Docker image. |
|||
|
|||
**Alternatives Considered**: |
|||
- `pdfjs-dist` directly instead of `pdf-parse` — more boilerplate, `pdf-parse` wraps it with a simpler API |
|||
- Only cloud OCR — loses the self-hosted story and adds cost for digital PDFs |
|||
Loading…
Reference in new issue