Improve resilience of the portfolio snapshot computation for large portfolios

A portfolio snapshot computation that runs longer than the Bull queue's
lock duration is flagged as stalled and fails, surfacing as HTTP 500 on
all portfolio and account endpoints. Because the snapshot job uses the
user id as its job id and was only configured with removeOnComplete, the
failed job stayed in the queue and every later request re-read it, making
the 500 permanent until Redis was flushed by hand.

- Add removeOnFail to PORTFOLIO_SNAPSHOT_PROCESS_JOB_OPTIONS so a failed
  job is removed and the next request recomputes instead of returning a
  cached failure
- Increase the default portfolio snapshot computation timeout from 30s to
  300s so large portfolios compute successfully out of the box (still
  overridable via PROCESSOR_PORTFOLIO_SNAPSHOT_COMPUTATION_TIMEOUT)

Refs #6914, #3755

CHANGELOG.md

@@ -22,9 +22,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Improved the language localization for Spanish (`es`)
 - Upgraded `bull-board` from version `7.0.0` to `7.1.5`
 - Upgraded `Nx` from version `22.7.1` to `22.7.2`
+- Increased the default timeout of the portfolio snapshot computation from 30 to 300 seconds to support portfolios with a large transaction history
 
 ### Fixed
 
+- Resolved a permanent internal server error on the portfolio and account endpoints by removing failed portfolio snapshot jobs from the queue so that a stalled computation is retried on the next request
 - Resolved an issue with the cash balance calculation of an account for `SELL` activities to ensure fees are correctly subtracted
 - Resolved an exception in the portfolio details endpoint when an asset profile is unmatched
 

libs/common/src/lib/config.ts

@@ -89,7 +89,7 @@ export const DEFAULT_PORT = 3333;
 export const DEFAULT_PROCESSOR_GATHER_ASSET_PROFILE_CONCURRENCY = 1;
 export const DEFAULT_PROCESSOR_GATHER_HISTORICAL_MARKET_DATA_CONCURRENCY = 1;
 export const DEFAULT_PROCESSOR_PORTFOLIO_SNAPSHOT_COMPUTATION_CONCURRENCY = 1;
-export const DEFAULT_PROCESSOR_PORTFOLIO_SNAPSHOT_COMPUTATION_TIMEOUT = 30000;
+export const DEFAULT_PROCESSOR_PORTFOLIO_SNAPSHOT_COMPUTATION_TIMEOUT = 300000;
 
 export const DEFAULT_REDACTED_PATHS = [
   'accounts[*].balance',
@@ -217,7 +217,8 @@ export const INVESTMENT_ACTIVITY_TYPES = [
 
 export const PORTFOLIO_SNAPSHOT_PROCESS_JOB_NAME = 'PORTFOLIO';
 export const PORTFOLIO_SNAPSHOT_PROCESS_JOB_OPTIONS: JobOptions = {
-  removeOnComplete: true
+  removeOnComplete: true,
+  removeOnFail: true
 };
 
 export const HEADER_KEY_IMPERSONATION = 'Impersonation-Id';