Assessments production readiness¶

Status: Proposed implementation plan; gates not yet passed | Last reviewed: 2026-10-20

Use this plan to turn the existing Assessments demonstration into a supportable production service. Each work package identifies the current gap, implementation scope, responsible discipline and evidence needed for release. Read the current architecture first; the target below has not been deployed by this documentation task.

Implementation paths below refer to the local working checkout; the ignored app source is not hosted by the documentation site.

Production readiness has two separate outcomes: dependable software operation and justified use of its assessment results. A technically reliable service can still produce unsuitable employment or language-proficiency conclusions. Neither a successful demo nor a human-review button establishes the second outcome.

Preserve the agreed product behavior¶

Keep Name/Email/Code access with internal invitation management and Deloitte SSO. Keep the simple login brand/header and the requested clean main screens. Keep completed Behavioral and English AI analyses available for demonstration alongside fresh attempts, with previous submissions preserved.

Code users stay candidate-only and receive their assigned types. SSO users retain the existing demonstration capabilities until a production access policy is approved. Do not restore removed main-tab marketing/research copy as a substitute for governance work. Required notices, consent, evidence limits and release decisions belong in the appropriate product/legal workflow and technical documentation.

An AI analysis is a stored model result. A human-reviewed result requires an accountable reviewer action. A validated assessment instrument requires evidence beyond either workflow. Demo scripts and production reports must keep these distinctions clear.

Decisions that set the production scope¶

Record these decisions before sizing infrastructure or promising a delivery date. Product and engineering can continue source preservation and tests while policy owners resolve the business choices.

Decision	Accountable owner	Required written outcome
Intended use	Product + recruitment/language-assessment owner	Demonstration, shadow evaluation, coaching or employment selection; role, population, geography and prohibited uses
Identity and access	Security + IAM + recruiting operations	Candidate identity assurance, reviewer/admin groups, campaign scope, conflict rules and access-removal SLA
Assessment content	Subject experts + assessment specialist	Job analysis, question coverage, scoring rubric, language skills covered and evidence required for claims
Data handling	Privacy + security + records owner	Lawful basis/notice review, consent separation, storage/provider regions, retention/holds, export and deletion policy
Service objectives	Product + SRE	Peak concurrency, attempt volume, support hours, latency targets, RPO/RTO and acceptable interruption policy
Candidate operations	Recruiting operations + accessibility owner	Invitation delivery, accommodations, support/contact details, feedback, corrections, appeals and manual fallback
Platform	Platform engineering	Approved runtime/account/network, SSO integration, secrets/KMS ownership, deployment and audit boundaries

No production infrastructure is provisioned, no access policy is changed and no candidate data is moved as part of this plan.

Release gates¶

Gate	Scope	Required evidence
G0 — reproducible engineering baseline	Continued engineering on the existing demo	Reviewed source ownership; clean-clone build; versioned migrations; preserved source/evidence/key recovery set
G1 — controlled operational pilot	Explicitly scoped real-user pilot	P1–P6 controls; candidate support and accommodation path; privacy/provider review; restore and interruption drills; documented limits
G2 — consequential use	Results inform the approved business decision	G1 plus independent P7 validation, approved thresholds/claims, trained reviewers, monitoring/appeals and legal/policy assessment
G3 — scaled service	More replicas, campaigns or sustained traffic	P2 replication proof, P8 load/capacity evidence, tested canary/rollback and service-owner sign-off

Gate approval is a release-management decision supported by artifacts. A checklist entry marked “implemented” without its test or operational evidence does not pass a gate. A limited pilot needs an explicitly bounded use policy; labeling a deployment “pilot” does not itself authorize use in hiring.

Proposed production hosting¶

A suitable AWS reference target is an Application Load Balancer with managed TLS, ECS/Fargate application services, private RDS PostgreSQL and private S3 evidence storage. ALB supports persistent WebSocket upgrades, which fits the existing English transport. This is a design recommendation, not an inventory of current infrastructure. Reuse an approved corporate container platform if required; retain the same application, storage and recovery requirements. Official references V2–V4 are in the operations guide.

Component	Proposed role	Change from today
DNS, TLS and ALB	HTTPS entry point; `/voice` upgrade; certificate lifecycle; bounded ingress	Replace manual certificate installation and host nginx responsibilities where appropriate
Web/API service	SPA, sessions, invitation management, commands and reviewer APIs	Explicit process role; private listener behind trusted ingress; scoped authorization
Voice service	English live connections and durable recording acknowledgements	Owner leases, admission control, session draining and recovery; can share web service initially after correctness work
Scoring workers	English and Behavioral analysis	Independent concurrency, resource limits, queue-age scaling, provider budgets and graceful cancellation
Maintenance runner	Retention, expiry, reconciliation and audit anchoring	Independently scheduled/leased work with alerts; not dependent on a long scoring call finishing
RDS PostgreSQL	Transactional system of record and initial durable job queues	Private access, TLS, role separation, Multi-AZ where the target requires it, automated backups/PITR and tested restores
S3 evidence store	Encrypted media/provider artifacts and explicit lifecycle	Migrate both evidence domains or explicitly retain/budget Behavioral SQL media; reconcile objects and metadata
Secrets/KMS	Runtime credentials, envelope keys and signing-key versions	Service identities, separated key purposes, rotation and old-key recovery
Central telemetry	Logs, metrics, traces, alerts and audit destination	Operational visibility without raw candidate content in ordinary telemetry
CI/CD and registry	Reproducible build, signed/provenanced artifact, controlled promotion	Replace reliance on ignored source and manual image tags

Use separate dev/test/prod data and provider projects. Keep synthetic demo fixtures visibly identified and outside production validation metrics. Browser assets may remain served by the API initially; a CDN is optional and must never cache authenticated evidence or result APIs.

Use private networking and narrowly scoped service identities. If the platform requires TLS beyond the ingress, configure and test backend TLS explicitly. Avoid embedding static cloud keys in task definitions. Approve outbound connectivity to the identity and model providers; capture operational failures without logging authorization headers or candidate recordings.

ALB draining and container stopping are different mechanisms. Fargate's documented stop timeout is at most 120 seconds; an assessment can run much longer. Implement admission shutdown and application-level draining before terminating tasks, with a tested interruption/resume path for forced failures. Sticky routing alone does not fix the current ownerless startup recovery.

Do not copy the Compose resource numbers or host networking directly into a Fargate task definition. Choose a supported CPU/memory pair from measured peak memory, configure awsvpc/private service networking, update the bind address and constrain proxy trust. Model requests assembling long PCM recordings can consume considerably more memory than the raw recording size.

P0 — source ownership and reproducible delivery¶

Owner: application engineering + repository/platform owner. Dependency: none. Gate: G0.

The app's source is mostly ignored and absent from a fresh clone. Its local Docker image and private rollout backups are not an adequate engineering distribution mechanism. The source evidence manifest identifies inspected files, but is not a copy of their contents.

Decide the app's repository/package boundary. Either track reviewed source in this repository or move it into an owned repository with an explicit dependency on shared storage. Preserve the existing local checkout until a clean-clone reproduction succeeds.
Include source, lockfile, public assets, nonsecret configuration examples, schema changes, tests, build/deploy definitions and documentation. Exclude runtime .env.local, credentials, recordings, database dumps, private transcripts and generated test artifacts containing personal data.
Run a secret scan on staged content and dependency/container scanning. Pin build inputs sufficiently to reproduce the image; record source revision, lockfile hash, image digest, SBOM/provenance and migration version.
Replace repeated baseline SQL application as the sole migration mechanism with an ordered migration ledger and compatibility policy. Keep fresh-install tests as well as upgrade tests.
Create app-scoped CI for typecheck, lint, tests, build, migration checks and appropriate browser coverage. Promote the same immutable artifact across environments.

Acceptance: a second clean environment builds and tests the app without recovering files from the original workspace; an upgrade fixture preserves attempts, invitation consumption, recordings and review history; the release manifest identifies both code and schema. No secrets or candidate data appear in the tracked package.

Primary change points: root ignore policy, package/build configuration (apps/proficiency/package.json), apps/proficiency/Dockerfile, apps/proficiency/server/migrate.ts, CI and app documentation.

P1 — production identity and access¶

Owner: IAM + application security + recruiting operations. Dependency: role and scope decisions. Gate: G1.

Broad SSO review is an intentional demo setting. It is too broad to adopt silently for a production candidate population. The legacy administrator fallback and lack of assignment scope also need explicit treatment.

Define candidate, reviewer, assessment administrator and platform operator permissions. Scope evidence access to an organization/campaign/requisition or assigned review queue as the business requires.
Restrict both the application flag and Authentik mappings. Migrate away from the implicit reviewer-to-admin fallback only after explicit administrator claims are present and tested. Preserve explicit false claims.
Retain invitation identity isolation. Add verified person/assignment linkage for conflict-of-interest rules and evaluation partitioning without merging on email. Test a person using both SSO and an invitation.
Specify invitation delivery and identity assurance. Add an approved delivery channel, code expiry/reissue controls, rate limits and administrative auditing. Decide whether possession of a code is adequate or whether inbox verification/stronger identity is required.
Add durable rate/admission limits that distinguish authentication abuse from legitimate concurrent audio uploads behind a shared NAT. Test proxy spoofing and forwarded IPs at the actual ingress.
Test revocation during ordinary requests, uploads and live audio; document the maximum enforcement interval. Review exports and administrative changes as sensitive operations requiring refreshed entitlement.

Acceptance: positive and negative access matrices pass for every API and media path; cross-campaign access, forged IDs and cross-auth self-review fail; revoked access stops within the approved bound; candidate sessions never inherit SSO roles. IAM owners demonstrate joiner/mover/leaver and emergency revocation procedures.

Primary change points: apps/proficiency/server/auth.ts, apps/proficiency/server/invitations.ts, both route modules, schema scope fields and Authentik property/group mappings.

P2 — live session ownership and process separation¶

Owner: backend + platform engineering. Dependency: P0. Gate: before replicas; G1 recovery and G3 scale.

Current voice.recover() interrupts every open voice run on startup. Fix that correctness issue before autoscaling or an overlapping rolling deployment.

Give each voice run an owner instance ID, lease/heartbeat and fencing generation. Recover only an expired owner; an old owner must not append or finalize after takeover.
Make recording acceptance durable and observable. Define the exact point at which the browser is told a chunk is saved. Resume from the last acknowledged position; reject duplicate conflicts, gaps and stale epochs without rewriting original evidence.
Preserve original deadlines and consent through interruption. Distinguish a new authorized upstream voice connection from seamless continuation of the old provider session.
Add process roles/entry points for serving, voice, scoring and maintenance. Separation can be incremental, but a worker-only process must not run voice startup recovery.
Add draining state: stop new sessions, remove readiness, allow active recordings to finish when possible, then stop. Await or safely relinquish scoring jobs before closing the DB. Forced shutdown must leave recoverable leases and unambiguous evidence.
Apply per-instance and fleet admission limits atomically. Queue or reject new starts with actionable candidate feedback before exhausting sockets, memory or provider concurrency.

Acceptance: run two instances against the same test database while one has active English sessions; starting/stopping the other does not change those sessions. Kill the owner during capture and recover only its sessions after lease expiry. Validate bounded acknowledged-audio loss, stale-writer rejection, retained deadlines, no duplicate finalization and safe draining.

Primary change points: apps/proficiency/server/main.ts, apps/proficiency/server/voice.ts, voice-run schema, browser reconnect code and deployment health/drain hooks.

P3 — evidence, keys, retention and recovery¶

Owner: backend + security/KMS + database/records operations. Dependency: data policy and P0. Gate: G1.

The current file encryption, manifests and SQL transactions are a useful base. Production needs a complete data lifecycle and recoverable key management.

Inventory personal content in every table/object/export/log: identity, recordings, transcript events, AI artifacts, reviews, audit payloads, receipts, datasets and backups. Record purpose, region, access, retention trigger and deletion/hold rule for each.
Implement a versioned encryption envelope with separate encryption, manifest-signing and session/CSRF key purposes. Use managed key access and record key identifiers without exposing key material. Retain old-key decryption/verification while data remains.
Exercise rotation on copied evidence, then backfill or rewrap with auditable progress and rollback. Do not replace the current environment key and hope historical data remains readable.
Move English files to a shared durable object store using copy/hash verification and metadata reconciliation. Behavioral bytea needs its own storage migration or an explicit capacity/backup decision. Merely selecting the S3 adapter does not affect it.
Define the SQL/object commit protocol and reconciliation for interrupted uploads/deletions. Preserve immutable logical IDs/content hashes; track object version/location separately if necessary. Detect orphan objects and missing evidence.
Unify deletion and hold behavior. Include the new shared invitations and user identity records, derived reports, model artifacts, dataset membership and exports. English report-expiry and evidence-expiry paths need a common appeal/hold policy.
Specify how backup expiry, object versions and provider retention interact with deletion. If immutable storage is used for selected audit anchors, avoid imposing an incompatible blanket retention lock on candidate audio.
Automate backups and run isolated restores of DB, evidence and keys together. Prove decryptability, manifest verification, authorization and deletion-state correctness after recovery.

Acceptance: a signed data inventory and automated lifecycle tests cover both assessment types; key rotation can read old/new evidence; interrupted migrations reconcile; expiry/holds are consistent; restoring the recovery set meets approved RPO/RTO. No retained copy is silently omitted from the policy.

Primary change points: apps/proficiency/server/crypto.ts, apps/proficiency/server/evidence.ts, apps/proficiency/server/worker.ts, Behavioral maintenance (apps/proficiency/server/bpaas/domain.ts), schemas and backup tooling.

P4 — scoring reliability, provider controls and provenance¶

Owner: AI/backend engineering + provider account owner. Dependency: P0; coordinate P2/P3. Gate: G1.

Keep jobs durable and explicitly at least once. Fence completion by current lease/generation, maintain bounded retry budgets even after crashes, and distinguish retryable provider/network errors from invalid or insufficient evidence.
Add backoff with jitter, timeout/cancellation, dead-letter handling and operator retry reasons. A retry can incur another provider charge even if only one final result is accepted; record that fact.
Give every request a provenance record: assessment snapshot/manifest, prompt/schema version, model ID, provider request ID where available, attempt/run/job IDs, request hash, usage, parse/validation outcome and latency.
Extend usage accounting across live voice, TTS, audio observation, synthesis, coaching, retries and failed/invalid outputs. Add project budgets and per-attempt call limits.
Validate actual provider project settings, data agreement, approved region and every endpoint/model feature. Make HTTP and WebSocket base endpoints configurable. Do not describe store: false as Zero Data Retention.
Maintain an allowlist/change process for model configurations. Evaluate prompt/model changes on a locked benchmark before publication; preserve historical snapshots and results. Avoid silent fallback to a model with different scoring characteristics.
Treat candidate transcripts as untrusted content. Test instruction injection in spoken answers, fabricated quotations, invalid evidence IDs, duplicate references and plausible but unsupported explanations. Withhold or route uncertain/failed analysis to human handling.
Make provider outages a recoverable processing state. Keep saved recordings available and give candidates/reviewers an accurate status; do not invent an analysis or erase prior human results.

Acceptance: fault-injection tests cover 429/5xx/timeouts, malformed output, job-owner death and duplicated work; stale workers cannot publish; operators can identify and retry a failed job without overwriting released reviews; a model change has a reproducible evaluation and rollback record.

Primary change points: both scoring workers, apps/proficiency/server/openai.ts, job schemas, shared configurations (apps/proficiency/shared/bpaas.ts), provider project configuration and usage dashboards.

P5 — candidate completion, accessibility and support¶

Owner: product/frontend + recruiting operations + accessibility. Dependency: candidate policy. Gate: G1.

Complete the journey beyond successful recording. Provide an actual support contact, invitation recovery, understandable processing failures, a tested reviewed-result delivery path for both types and a correction/appeal workflow. Behavioral reviewer dispositions currently do not constitute candidate-facing delivery.

Implement the outstanding automated microphone-readiness request as a separate reviewed change: continuity, sustained signal and clipping checks; clear retries for silence, low level, disconnected input and permissions; optional playback; local-only test audio. Do not report measured energy as proof of intelligibility or submit a fabricated playback acknowledgement.

Target WCAG 2.2 AA with manual assistive-technology tests, not only an automated scanner. Review keyboard navigation, focus/error management, contrast, audio controls, status announcements, mobile zoom and accessible authentication. Timed assessments need an approved accommodation/alternative process and documented treatment of essential timing requirements; silently granting extra time or claiming all timers are exempt are both inadequate. V5 provides the standard.

Acceptance: end-to-end journeys cover invitation/SSO, both completed analyses and fresh attempts, permissions/disconnection recovery, result delivery and appeals. Accessibility specialists test candidate and reviewer flows. Recruiting operations can help a candidate without editing sealed responses or exposing AI results prematurely.

P6 — operational security and observability¶

Owner: SRE/platform + application security. Dependency: P0; coordinate P2–P4. Gate: G1.

Implement separate liveness, readiness and synthetic business checks. Keep provider outages visible without causing every healthy API task to restart. Monitor voice acceptance/acknowledgement failures, queue age and lease expiry, scoring failure/coverage rates, evidence read failures, retention backlog, auth failures, audit delivery and certificate/backup status.

Replace silent worker/maintenance catches with classified counters and sanitized diagnostics. Correlate request, attempt, voice-run and job IDs while keeping raw audio, transcripts, tokens, codes and model prompts out of ordinary logs. Protect operational and audit access separately; hash-chain evidence alone is not an independent audit store.

Use a risk-scoped OWASP ASVS review and penetration test, including object-level authorization, CSRF, replay, injection, exports, upload limits, session fixation, proxy trust and resource exhaustion. Record versioned findings and remediation evidence. Dependency scanning, non-root containers and a restrictive CSP are inputs to this review, not proof of completion. V6 provides ASVS.

Acceptance: alerts fire in controlled failure drills and reach a named on-call route; runbooks restore service/evidence; security review closes release-blocking findings; certificate renewal/install is automated or replaced by a managed listener and verified. Service dashboards distinguish real traffic, synthetic checks and demo attempts.

P7 — assessment validity, fairness and human review¶

Owner: independent language-assessment/industrial-organizational specialists + business/legal owners. Dependency: intended-use specification and candidate/data policy. Gate: G2.

For English, document the construct actually measured: speaking and reading in these tasks. Do not imply that untested listening/writing domains or a whole-person CEFR credential have been established. Establish task specifications, benchmark performances, trained raters, standard setting and empirical evidence for any CEFR linkage/thresholds. Have specialists obtain and review the relevant Council of Europe materials; the existing English guide links them, but the upstream CEFR pages were not retrievable during this documentation review.

For Behavioral, replace generic service principles with a role-specific job analysis and reviewed rubric. Establish whether each question elicits the intended competence, what missing evidence means, which outcomes validate it and how dispositions are used. Do not use speech style, accent, emotion or inferred personality as a substitute for job-relevant content.

Create a locked, consented evaluation set with person-level partitions and representative conditions. Use independent human ratings recorded before AI reveal, adjudication and documented rater training. Measure coverage/abstention, agreement, error at decision thresholds, uncertainty and relevant subgroup/accessibility differences where collection and analysis are authorized. Predefine acceptance rules and sample-size/power needs with a qualified specialist. Avoid selecting thresholds after looking at the held-out results.

Compare against the existing manual process on both quality and time. Measure actual active reviewer time and end-to-end delay; exclude synthetic demos and self-viewing from business-benefit estimates. Two model passes or high human/AI correlation do not replace criterion validity, fairness evaluation or outcome monitoring.

The NIST AI RMF is a useful governance structure, not a certification. U.S. EEOC selection-procedure guidance addresses job-relatedness, validation and accommodation issues; legal owners must determine applicable requirements for each deployment and population. Do not infer global compliance or exemption merely because a human reviews a result. See V7–V8.

Acceptance: an independently reviewed validation report supports the precise permitted claims and business use; unsupported levels/thresholds are disabled; reviewers are trained; adverse outcomes, appeals and model/content drift have owners and monitoring. A technical team cannot sign this package off alone.

P8 — capacity, rollout and release evidence¶

Owner: SRE + QA + backend + release owner. Dependency: P0–P6; P7 for consequential use. Gate: G3.

Use the capacity model and test matrix to test the agreed traffic envelope, including concurrent recording, playback and scoring. Size from measured peak memory/provider quotas and DB pressure. Do not extrapolate capacity from the current MAX_CONCURRENT_VOICE=12 default or one successful demo.

Rehearse migration and rollback on a copy with both completed and unfinished attempts. Stage immutable releases, add schema compatibly, copy/verify evidence, drain sessions and cut over only after checks pass. Keep old and new evidence/key readers compatible during the rollback window. Never rewind the live database to an old snapshot merely to revert a frontend release.

Acceptance: load and failure reports meet approved targets; deployment adds/removes an instance without corrupting sessions; both login methods and both demo scenarios pass after cutover; restore/rollback drills preserve current data; support and incident owners accept the release.

Delivery sequence and ownership¶

Start P0 immediately. Run the product/identity/data decisions and P7 study design alongside it. Next implement P1 and the P2 correctness changes; build P3 recovery/storage and P4 scoring/provider controls in parallel workstreams once interfaces are agreed. Finish P5/P6 for the controlled pilot. Execute P8 against the integrated system and the pilot's actual capacity target. Pass P7 before consequential use.

This order is a dependency plan, not a staffing or calendar estimate. Estimate each package after owners choose scope and inspect integration requirements. External IAM, privacy, procurement and validation-study dependencies can dominate the schedule.

The release owner assembles one evidence pack: source/image/schema manifest, architecture decision records, access matrix, data/provider approvals, security/accessibility results, automated/load/failure tests, recovery/rollback drill, assessment validation where required, operating procedures and named sign-offs. Track unresolved issues with owners, permitted-use limits and expiry dates.