Assessments operations and validation plan¶
Status: Proposed operating procedures and acceptance evidence | Last reviewed: 2026-10-20
Use this plan to test, migrate and operate a production Assessments service after implementing the production work packages. The architecture guide describes what exists today. Targets and procedures below are proposed; they are not claims that production controls or performance have already been verified.
Existing verification and what it establishes¶
The October 19 invitation release records 58 automated tests, nine browser scenarios, lint, TypeScript and a container build. The subsequent login-brand change records lint/build checks and two login browser scenarios. Earlier rollout integrity checks retained both completed examples, 152 English evidence objects and three Behavioral recordings. These are historical results in the AI handoff, not tests rerun for this documentation update.
Existing tests provide regression coverage for substantial application behavior. They do not establish sustained capacity, disaster recovery, corporate IAM lifecycle, independent assessment validity, accessibility conformance or production data-policy approval. Mock OIDC browser fixtures do not replace testing the actual corporate sign-in and role-removal path.
For an implementation change, run the app's normal checks in the documented environment:
cd /workspaces/trueserve-framework/apps/proficiency
npm run typecheck
npm run lint
npm test
npm run build
npm run test:e2e
Generate API documentation with npm run openapi after changing contracts. Real-provider smoke commands use synthetic audio and incur provider usage; use them when the affected integration warrants it. The English pipeline smoke and browser suite both use port 3052 and must not run concurrently. Fixtures use disposable schemas/evidence directories; never point destructive fixture cleanup at live data.
Acceptance test matrix¶
| Area | Required scenarios | Evidence to retain |
|---|---|---|
| Invitation access | Correct and incorrect name/email/code; normalized input; expiry; revoke; replace; both/single type; existing attempt recovery | API/browser results; no code/hash leakage; sessions invalidated as designed |
| SSO and roles | Actual corporate flow; state/nonce/PKCE failure; expired tokens; explicit false admin; role removal; restricted production reviewer mapping | Sanitized IAM test record and permission matrix |
| Object authorization | Other user's IDs, campaign scope, direct audio/report/export paths, both auth methods used by one person | Negative tests for every protected object/action |
| CSRF and abuse | Missing/foreign Origin; missing CSRF; stale sessions; replay; shared NAT; forged forwarded headers; burst login/uploads | Security findings and rate-limit/admission measurements |
| Attempt preservation | Saved English and Behavioral analyses; fresh SSO attempts; code replacement; retry-safe enrollment; simultaneous tabs | Before/after row and content hashes; original deadlines retained |
| Audio readiness | Speech, silence, low signal, clipping, stalled input, denied permission, unplug, retry, mobile device switching | Device/browser matrix and local-only capture verification |
| Recording transport | Duplicate/out-of-order/missing chunks, stale writer, reconnect, provider disconnect, end-of-task upload, browser sleep | Acknowledgement/sequence traces without raw content in general logs |
| Replica behavior | Add a second server during live sessions; kill owner; stale takeover; rolling deploy; forced drain timeout | No cross-instance interruption; fencing and recovery proof |
| Worker reliability | 429/5xx, timeout, malformed JSON, invalid references, lease expiry/crash, repeated retry, DB failure during completion | Durable state transitions, bounded cost/retries, stale-owner rejection |
| Evidence integrity | Corrupt ciphertext/hash/manifest, missing object, orphan object, mixed key versions, large audio, rescore after review | Deterministic rejection/reconciliation and historical-result preservation |
| Retention and holds | Both domains, shared invitations/users, explicit deletion, report expiry, appeal hold, retry after partial deletion, backup restore | Data-inventory coverage and purge/hold completion record |
| Human review | Pre-reveal rating, explicit reveal, persisted exposure, self-review across identities, release/disposition, amendment and appeal | Separation of AI/human records and verified candidate delivery |
| Accessibility | Keyboard, screen reader, zoom, focus errors, audio controls, timer/status updates, login and accommodations | Manual WCAG review plus automated findings and resolutions |
| Assessment quality | Locked benchmark, independent ratings, insufficient evidence, irrelevant/repeated answers, injection, subgroup/condition analyses | Versioned evaluation report, uncertainty and approved claims |
| Load and recovery | Agreed peak/burst, long sessions, DB failover, evidence outage, provider degradation, backup restore and rollback | Reproducible load scripts, metrics, SLO report and timed recovery |
Use consented or synthetic fixtures for engineering tests. Do not use synthetic high scores to estimate real-world accuracy or human time savings. For evaluation, preserve person-level partitioning across SSO, invitation identities and retakes.
Capacity and cost model¶
Define the workload first: candidates per day, peak active recordings by type, reviewer playback concurrency, average/upper-percentile response duration, retained days, submission burst size and provider quotas. Record the expected device/network mix and concurrent campaign schedule.
PCM16 mono at 24 kHz contains 24,000 × 2 = 48,000 raw bytes per second per track, or 2.88 MB per minute in decimal units. Five full five-minute Behavioral answers therefore contain about 72 MB of raw audio per attempt, before database/encryption overhead. Base64 upload encoding expands the payload by roughly one third; encryption and transport headers add more. This arithmetic is a sizing input, not measured production traffic.
English stores candidate and assistant tracks. If both contain a full m minutes, raw audio is approximately 2 × 2.88 × m MB; actual track lengths differ. Temporary assembled buffers, WAV/base64 conversion, provider payloads, transcripts and artifacts add memory/storage pressure. Measure peak RSS under simultaneous scoring and recording rather than allocating only the final file size.
Use these formulas as a spreadsheet basis:
Raw recorded bytes/day =
48,000 × sum(recorded track seconds across all attempts that day)
Retained media estimate =
measured stored bytes/day × retention days
+ retained object versions/replicas/backups
Required scoring starts/second =
peak submitted attempts/second × average scoring runs/attempt
Approximate worker concurrency =
required scoring starts/second × measured mean scoring duration
÷ chosen utilization target
Monthly provider cost =
sum(actual usage by model/modality × current contracted unit rate)
+ separately billed features and retried/failed calls
English commonly invokes an audio observer per eligible speaking task followed by two synthesis passes, plus live/TTS usage. A fully answered Behavioral run can require five audio-observation and five competency calls; insufficient evidence can reduce that count, and retries/re-analysis can increase it. Track actual calls and usage rather than billing every attempt as a fixed ten calls.
Infrastructure cost includes runtime CPU/memory, load balancer, database/storage/IO, object operations/storage, backups, logs, network egress/NAT and key/secret operations. Reviewer playback and repeated analysis can materially change the total. Obtain current region-specific pricing and the organization's provider contract when budgeting; this guide does not quote prices or guarantee a per-candidate cost.
The current twelve-live-session default is an admission setting, not a measured capacity promise. SQL connections, large bytea storage, per-IP rate limits, object writes, provider quotas and event-loop pressure can become limiting resources first. Scale workers by queue age and measured service rate; scale voice by active sessions/headroom as well as CPU. Reserve capacity for review and administrative recovery during scoring bursts.
Proposed service objectives¶
These are starting points for owner discussion, not observed results or contractual promises. Approve measurement definitions and targets in P8.
| Objective | Initial proposal | Measurement |
|---|---|---|
| Core API availability | 99.9% monthly within supported service hours | Eligible requests excluding expected authentication/validation rejection; classify provider dependence separately |
| Non-AI command latency | p95 under 500 ms at the agreed load | Server/ingress duration, excluding audio transfer and provider waits |
| Submission durability | Every acknowledged submission recoverable within the documented failure model | Crash tests and reconciliation of accepted commands/manifests |
| Live recording continuity | No unrelated-session interruption when adding/removing a healthy replica | Controlled deployment probes; acknowledged audio-loss bound defined by storage/backup design |
| AI turnaround | Initial p95 target under five minutes at agreed load | Submission-to-analysis time, with provider outage/coverage failures separately visible |
| Candidate support | Owned response time during recruitment campaigns | Support ticket/contact workflow; do not leave a placeholder contact |
| Disaster recovery | Initial planning target RPO ≤5 minutes and RTO ≤1 hour | Timed full-system restore including evidence and key access |
| Deletion | Completion within the approved policy deadline | Oldest overdue item and proof across primary/derived storage; backup expiry documented separately |
Do not promise zero disaster data loss while approving a five-minute RPO. Define which acknowledgements survive process loss, zone failure or full database recovery. Multi-AZ availability and backups/PITR solve different problems; neither guarantees that an independently stored object set and its keys can be restored coherently.
Observability and alerts¶
Instrument app, voice and worker operations with stable request/attempt/run/job correlation IDs. Keep candidate names, emails, codes, tokens, audio, transcripts and full provider prompts/results out of routine metrics/logs. Protected evidence/audit stores remain available to authorized investigations.
Minimum dashboards should show:
- Access and candidate journey: login errors, invitation failures by safe category, authorization denials, starts, submissions, interruptions and stuck states.
- Voice: active/available slots, accepted/acknowledged frames, persistence failures, disconnect reasons, owner heartbeat age and resumed/interrupted runs.
- Scoring: queue age/depth, leases, retries, dead letters, provider latency/errors, parse/validation failures, coverage/abstention and model version.
- Persistence: DB pool saturation/latency, object read/write errors, missing/orphan evidence, integrity failures, storage growth and backup/restore status.
- Lifecycle and governance: deletion backlog, overdue holds/reviews, expired credentials/certificates, audit delivery failures and usage/budget trends.
Page on user-impacting loss of recording persistence, material integrity failures, sustained authentication/service failure and overdue critical lifecycle jobs. Use tickets or lower-urgency alerts for trends where an immediate response cannot help. Test the alert route and escalation; the mere existence of a dashboard is not an operating control.
Migration and cutover procedure¶
- Create the release manifest. Identify the reviewed source, image digest, schema version, public origin/SSO subject mapping, model configurations, object layout and key versions. Record permitted use and passed release gates.
- Build an isolated rehearsal. Restore a protected copy using approved access; include both completed and unfinished attempts and invitation consumption state. Do not send copied personal recordings to a new provider project without the approved policy.
- Apply additive schema changes. Test forward migration and old/new reader compatibility. Preserve attempt IDs, snapshots, accepted answers, review revisions and code-grant relationships.
- Copy evidence with verification. Preserve logical IDs/AAD and plaintext hashes. Reconcile counts and bytes, decrypt sampled/all required records, verify manifests and test both domains. If re-encrypting, record key/version transitions and retain a compatible rollback reader.
- Verify identity and access. Keep OIDC issuer/client/subject continuity or execute a reviewed subject migration. Confirm production groups and invitation candidate isolation. Verify direct media and export authorization.
- Exercise workers and failure handling. Use authorized synthetic attempts for provider smoke tests, interruption recovery and queue retries. Check that replica startup cannot mutate another owner's live session.
- Enter a controlled cutover window. Stop new starts and drain active recording sessions. Allow scoring to complete or safely relinquish leases. Reconcile final writes; choose and document a brief write freeze or a tested replication strategy.
- Switch traffic and verify. Test code login and SSO, completed English/Behavioral analyses, fresh attempts, resume behavior, recording playback, human review, delivery and administrative actions. Observe error/queue/retention metrics.
- Retain a bounded rollback window. Keep the previous compatible image and readers plus protected recovery material. Remove temporary duplicate data/access according to policy after acceptance.
Avoid a casual dual-write scheme across the old local files and new object store. Partial failure can produce two inconsistent truths. If dual writing is necessary, implement a durable ledger/reconciler and prove failure behavior before using it in cutover.
Rollback and disaster recovery¶
An application rollback normally changes the immutable image while retaining the current database and evidence. Confirm schema and encryption compatibility first. Rewinding the database to an old dump loses newer submissions and is a disaster-recovery action, not a routine UI rollback.
The current app has two special compatibility constraints:
- Builds before invitation access do not safely interpret invitation-bound sessions. Follow the existing handoff's session-removal and IAM compatibility procedure before considering that old image; preserve assessment rows and grant consumption.
- Earlier Behavioral code assumed lifetime attempt uniqueness. Do not recreate that constraint after retakes exist. An older schema expectation can fail even when the image starts.
For disaster recovery, restore DB, evidence objects and the matching keys into an isolated destination first. Verify schema/version, users/ownership, invitation scope, sample/full required decryption, manifest/audit consistency, released results and lifecycle state. Check whether provider jobs might run again and disable automatic worker execution until reconciliation prevents unintended duplicate processing.
Record actual recovery point, missing/uncertain records, time to restore and operator decisions. Reapply deletion tombstones/holds according to policy so restored backups do not silently resurrect data into normal access. Reconcile the recovery with the candidate/reviewer support process before reopening.
Incident runbooks to implement¶
| Incident | Immediate response | Recovery proof |
|---|---|---|
| Voice persistence failure | Stop accepting audio/new sessions where durability cannot be met; show accurate interruption status | Last acknowledged chunks/epochs reconcile; original deadlines and saved evidence remain |
| Model provider outage | Keep submissions durable; pause/bound retries and surface pending analysis | Retry authorized jobs after recovery; preserve prior AI/human revisions and usage records |
| Stuck/duplicate scoring | Inspect lease ownership, retries and provider request IDs; fence stale execution | One accepted completion per current generation, with duplicate charges visible |
| Identity/provider failure | Keep authorization fail-closed; do not grant review/admin roles to bypass SSO | Restore verified claims; recheck expired/revoked sessions and candidate isolation |
| Evidence missing/corrupt | Restrict affected result release and preserve incident evidence | Restore a verified copy or mark unavailable/insufficient evidence; never fabricate a transcript |
| Key unavailable or lost | Restrict affected reads/writes; contact key custodian through the approved process | Restore authorized key access/version; prove decryption/signatures before reopening |
| Retention backlog | Alert records owner; prevent unauthorized re-export; repair/retry idempotently | All stores/derived copies reconcile; holds and completion receipts are correct |
| Security/data incident | Contain the affected access path, preserve authorized audit evidence and engage incident/privacy owners | Verified remediation, scoped impact and approved notification/reopening decisions |
Attach runnable, environment-specific commands only after the production platform is selected. Existing local-container commands are not interchangeable with official Dev/QA access or a future production cluster. Follow the repository's official-environment source, access and logging rules during incident investigation.
Documentation artifact verification¶
The diagram uses Archify's architecture type and showcase quality profile. Delivery validates nine artifact checks and records exact specification/HTML SHA-256 hashes and byte counts in the receipt. Browser checks and reviewed screenshots are recorded in the visual-review record.
To revise it, edit the JSON specification, validate after each revision, then deliver to the HTML path. Run from the repository root with the installed Archify skill:
node /home/codespace/.agents/skills/archify/bin/archify.mjs validate architecture \
docs/assets/assessments/assessments-current.architecture.json --quality showcase --json
node /home/codespace/.agents/skills/archify/bin/archify.mjs deliver architecture \
docs/assets/assessments/assessments-current.architecture.json \
docs/assets/assessments/assessments-current.html --quality showcase --json \
> docs/assets/assessments/assessments-current.receipt.json
Inspect the rendered diagram in both themes after delivery, check long labels and default viewport readability, and refresh the static preview and visual-review record. A validation receipt does not replace visual review. Never hand-edit generated HTML and leave the old hashes claiming it is the accepted artifact.
Official references¶
References were checked October 20, 2026. Vendor documentation establishes available behavior and conditions, not the configuration or entitlement of this deployment. Source implementation evidence is separately recorded in the file-hash manifest.
| ID | Primary reference | Use in these guides |
|---|---|---|
| V1 | OpenAI data controls | Training opt-in, abuse monitoring versus application state, store, ZDR eligibility and regional/model conditions |
| V2 | AWS ALB listeners and target-group attributes | Native WebSockets and deregistration/draining behavior |
| V3 | AWS ECS container stop timeout | Fargate's maximum 120-second stop timeout |
| V4 | Amazon RDS Multi-AZ | Availability, backup and restore behavior; application recovery still requires evidence and keys |
| V5 | W3C WCAG 2.2 | Accessibility acceptance criteria and manual review scope |
| V6 | OWASP ASVS | Versioned application-security verification; page identifies stable 5.0.0 |
| V7 | NIST AI Risk Management Framework | Governance, measurement and lifecycle risk management; check ongoing revisions before adopting a version |
| V8 | EEOC employment tests and selection procedures | U.S. validation/job-relatedness and accommodation context; deployment-specific legal review required |
The existing English guide's CEFR references identify Council of Europe materials for specialist review. Those upstream pages/PDFs returned access errors during this review; no newly verified CEFR conformity claim is made.