Assessments AI handoff¶
Status: Current | Last reviewed: 2026-10-20
Continue the Language Assessment / unified Assessments app from this handoff when the original Codex thread cannot be opened. The unified Behavioral and English app is deployed and healthy. The next unimplemented user request is to automate microphone readiness checks using incoming audio and volume.
The October 16 continuation deploys repeat demonstrations: home cards and history separate View AI review from Start new assessment / Continue assessment. Behavioral enrollment now permits a separate attempt after submission and enforces uniqueness only for unfinished attempts; its newAttempt flag and operation receipt preserve existing results and retry safety. Follow the current Behavioral guide and demo walkthrough over the original single-attempt descriptions below. Both completed examples already exist under the presenter's account; no generated examples or replacement scores are needed. Verification passed 53 automated tests and eight browser scenarios; original records and evidence were preserved. The microphone follow-up remains separate.
October 20 architecture and production documentation¶
The requested Archify diagram and detailed technical/production write-up are in project documentation and this AI handoff, as the user selected. No in-app documentation page or production migration is part of this change.
- Architecture and technical guide: both login methods, both assessment pipelines, review, data, encryption, AI integrations and current runtime constraints.
- Interactive Archify diagram: current architecture, with an editable specification, delivery receipt and visual-review record.
- Production readiness: proposed hosting, nine work packages, owners, dependencies, acceptance criteria and release gates.
- Operations and validation: tests, capacity/cost formulas, proposed service objectives, migration, rollback, recovery and official references.
The documentation audit rechecked the deployed image identity and recorded source hashes. It did not rerun candidate assessments, change IAM, move data or deploy a new app image. Historical test results remain labeled as historical.
Prioritize source reproducibility and session ownership before production replication: only the app's .env.example is tracked, and current voice.recover() interrupts every open voice run without checking an owning instance. The roadmap also covers broad reviewer access, cross-auth self-review conflicts, key rotation, shared-invitation retention, candidate feedback delivery and independent assessment validation. These are proposed follow-up packages, not completed fixes. The existing automated-microphone request remains separate.
October 19 access and interface update¶
The latest user request restores Name, Email, Code login alongside Deloitte SSO and adds internal management. /invitations is an administrator-only screen for issuing access to Behavioral, English or both, selecting expiry, replacing a code and revoking access. Code users are candidate-only, isolated from SSO users even when emails match, and receive one persistent attempt per assigned type. Replacement resumes those attempts; new invitations authorize retakes. SSO still supports completed AI reviews alongside separate fresh attempts.
The login card has a small Assessments by Deloitte logo/header above the three-field form and SSO button, restored at the user’s follow-up request. Main-tab descriptive subtitles, the footer, header Research/assessment labels, Behavioral metric disclaimer and Validation research box are removed. Mobile navigation, identity metadata and consent/task instructions remain. See current access behavior and invitation instructions. These decisions supersede the earlier SSO-only direction below.
server/invitations.sql is an additive migration applied after both assessment schemas. Do not allow code sessions to inherit ASSESSMENTS_ALL_USERS_REVIEWERS, merge accounts by email, expose invitation hashes, or reset consumed attempt IDs. Code values are shown once and are not stored in audit or idempotency receipts. Before rolling back to a pre-invitation image, stop the app and remove invitation-bound sessions; retain all assessment data and schema. Older builds cannot interpret those sessions safely.
The current login-header build is trueserve-proficiency:login-brand-20261019. It adds the shared brand mark above the login form. Lint, the TypeScript/build checks and both login browser scenarios passed. No schema or assessment logic changed. The previous invitation build is the immediate image rollback; protected rollout details are in .local-data/proficiency/login-brand-rollout.txt.
The October 19 invitation and interface update was deployed as trueserve-proficiency:invitation-access-20261019 (sha256:cc2c5c3077c08e6f7cacba06fa7b09d5a65c856a54c07ed1f492587be388e8be). Verification passed 58 automated tests, nine browser scenarios, lint, TypeScript and the container build; the final field styling also passed the invitation browser scenario. The additive migration preserved every existing assessment, recording and review row, including the unfinished Behavioral attempt. Both completed examples, 152 English evidence objects and three Behavioral recordings passed runtime integrity checks. Protected backups and verification results are identified by .local-data/proficiency/invitation-access-rollout.txt. Full authenticated journeys ran in disposable schemas; the public deployment was checked for health, anonymous protection and the new login assets.
Start here¶
- Work in
/workspaces/trueserve-framework/apps/proficiency. The product is named Assessments; its directory, database and SSO client retain the nameproficiency. - Read app invariants, runtime commands, the shared/English technical guide, and the Behavioral technical guide.
- Read the architecture and production documentation for current system boundaries and the implementation roadmap; do not mistake proposed production controls for deployed features.
- Preserve the current checkout. Most application source is Git-ignored and is absent from a fresh clone.
- Follow the latest user task. The older microphone follow-up remains separate; access and demo behavior are documented above.
- Recovery did not modify thread state. This continuation implements access and presentation changes; it does not implement the separate microphone follow-up.
Recovered thread and stopping point¶
| Item | Recovered value |
|---|---|
| Thread ID | 01a13ae0-5bd8-7493-90e2-652133b90b21 |
| Indexed title | We have an existing Language Assessment app here - https://… |
| Workspace | /workspaces/trueserve-framework |
| Branch | environment/vanguard |
| HEAD during recovery | 8b7ce59f04976814a7971c0590f7359223d5bbf2 |
| Recorded error | conversation_blocked: this conversation cannot be continued |
| First recorded block | October 14, 2026, 11:58:59 a.m. EDT |
| Latest user request | October 14, 2026, 12:09:27 p.m. EDT |
The readable local transcript is:
/home/codespace/.codex/sessions/2026/10/14/rollout-2026-10-14T14-45-44-01a13ae0-5bd8-7493-90e2-652133b90b21.jsonl
The session index is /home/codespace/.codex/session_index.jsonl; metadata is in state_5.sqlite in the same directory. The transcript's final deployment command completed before the first error. The later microphone request failed immediately with the same error and has no implementation actions after it. There is no evidence here establishing why the service blocked the conversation; do not assume deleting a local lock file repairs it.
Recovery used saved user messages, assistant updates, command results, project files and deployment artifacts. No thread database or lock was modified. The raw transcript is private working context and can contain operational details; do not commit or publish it.
Latest product decisions¶
The original brief expands the existing English research app with a working AI-assisted BPaaS candidate screener for the Customer Experience recruitment team and an EDJ demonstration. The supplied HTML prototype informs capabilities, not visual design.
Later user messages override two parts of the original brief:
- Unified access. Use
assessments.dev.trueserve.deloitte.com. Support Name/Email/Code invitation login and Deloitte SSO. Derive SSO identity from Authentik; match code identities to administrator-issued invitations. SSO gets both types; invitations get their assigned types. - Score visibility and roles. The user briefly requested immediate candidate AI scores and restricted reviewers, then explicitly changed that direction. Candidate pages hide AI and promise later feedback. Every admitted Deloitte user can use the reviewer workspace and explicitly reveal AI, including their own result. Human self-review/release stays blocked. Assessment/model administration remains elevated.
The user subsequently authorized completing deployment, the hostname change and Authentik configuration, asking that Authentik use its API. Those steps finished. The final request was:
The mic test should be automated since we can see if data is flowing and seems of right volume.
That microphone request is the next work item. The current task authorizes preparing this handoff; it is not a request to repeat the completed rollout.
Implemented scope¶
| Area | Current behavior |
|---|---|
| Shared access | Name/Email/Code invitations plus corporate SSO, unified home, server-owned identity and role-scoped navigation |
| Behavioral attempt | SSO permits a fresh attempt after submission; an invitation retains one attempt per assigned type; reopening preserves the active attempt |
| Readiness | Consent, Terms/Privacy, optional reuse consent, ten-second local microphone recording with mandatory playback acknowledgement |
| Questions | Five behavioral prompts; 30 seconds to start, up to five minutes to answer, early submission, automatic deadlines and missed-response handling |
| Recovery | Existing audio, submitted answers and original deadlines are preserved; writer rotation prevents old tabs overwriting |
| AI assessment | Audio transcription/observations, transcript-based rubric, verified quotations, relevance/STAR/empathy/problem-solving/domain criteria, coverage flags |
| Reviewer | Queue, recordings, transcript, explicit AI reveal, score/competency evidence, append-only notes and dispositions |
| Administration | Versioned questions/rubrics/models, usage and performance summaries, reuse eligibility manifests |
| English | Existing guided speaking, reading, sealed evidence, research CEFR estimates and separate human review remain available |
Behavioral speech observations are separate from hiring competence. Null means insufficient evidence; a zero criterion requires evidence of counterproductive behavior. Missing answers do not silently become zero competence. AI does not accept, reject or disqualify candidates, and voice must not be used to infer emotion, personality or cultural fit.
Fine-tuning support currently means consent-based eligibility manifests and selecting already deployed model IDs. It does not train a model or automatically upload a dataset.
Live deployment and identity¶
| Item | Value |
|---|---|
| Primary hostname | https://assessments.dev.trueserve.deloitte.com |
| Legacy hostname | https://langtest.dev.trueserve.deloitte.com, redirects to Assessments |
| Dedicated container | tsf-proficiency |
| Loopback listener | 127.0.0.1:3050 |
| Deployed tag | trueserve-proficiency:login-brand-20261019 |
| Image ID | sha256:48258c057bd2a2a1bbb690c697ad924ce5bc9c20b7cd989ea3ee20d186276b09 |
| Immediate rollback tag | trueserve-proficiency:invitation-access-20261019 |
| Authentik provider / client | Provider 125; client and application slug proficiency |
| Access claims | langtest_access, langtest_reviewer, separate assessments_admin |
| Reviewer setting | ASSESSMENTS_ALL_USERS_REVIEWERS=true |
| Administrator membership | Existing Proficiency Reviewers group; includes erigolden@deloitte.com |
The group name is historical: broad reviewer access does not imply administrator membership. An explicit false administrator claim must remain false. Preserve the SSO provider/client ID so existing subjects, users and assessment ownership survive.
The saved deployment record records deployment at 11:49 a.m. EDT and default-resolver verification at 11:58 a.m. EDT. DNS propagation completed; the earlier cached empty DNS answer is resolved. Authentik changes used its API. The browser record shows corporate Microsoft sign-in was reached and protected anonymous requests were rejected.
At the October 16 update, primary HTTPS health, the legacy redirect, anonymous API rejection and the new public frontend assets were verified. Both original completed assessments, 152 English evidence objects and three Behavioral recordings passed integrity checks; saved assessment, recording and review rows matched the pre-deployment hashes. The protected .local-data/proficiency/repeat-demo-rollout.txt points to that rollout's backup and verification record. Corporate interactive login was not repeated; the complete UI journeys ran in disposable test schemas.
Private runtime configuration is apps/proficiency/.env.local; provider credentials come from secret/OpenAIAPIKey.txt. Authentik administration uses secret/AuthentikToken.txt. Read credentials through the existing mechanisms; never copy their contents into a handoff, command argument, log or Git.
The deployment uses dedicated proficiency PostgreSQL tables and .local-data/proficiency/evidence. Protected rollback snapshots are in .local-data/proficiency/backups/; .local-data/proficiency/active-rollout.txt identifies the rollout backup. Preserve database, encrypted evidence and keys together. See the technical guide before rollback: old code requires restoring restricted Authentik claims first.
This app runs in its dedicated Docker container on the Vanguard host, even though most framework applications use Overmind. Its compose mounts refer to the host checkout /home/admin/trueserve-framework. Do not restart core voice services for this app. Follow repository official-environment access/source/logging rules for any later incident investigation.
Next task: automate microphone readiness¶
Confirmed current gap: the Behavioral check displays input activity but still requires full playback plus “I played the check and can hear myself clearly.” The English check records four seconds and uses another manual audibility checkbox. Neither currently computes a readiness verdict from signal quality.
| Change point | Current implementation |
|---|---|
| Behavioral candidate | checkMic() collects PCM and updates a peak meter; heard / heardEnd gate starting |
| Audio capture helper | microphone() returns mono PCM frames and supports an ended-track callback |
| AudioWorklet | Shared mono PCM16 / 24 kHz capture |
| Behavioral routes | microphone/start records server time; microphone/complete requires { heardPlayback: true } and at least ten elapsed seconds |
| English candidate | checkDevice() uses MediaRecorder; deviceOk is set by the manual checkbox |
| English voice helper | Existing RMS meter during live recording |
The following implementation guidance is proposed for the next agent; the user specified automation based on data flow and volume, not exact thresholds:
- Calculate capture continuity, non-empty sample coverage, sustained signal level and excessive clipping during the readiness window. Document thresholds; a single loud peak or nonempty recording must not establish readiness.
- Automatically mark readiness when the check succeeds. Give actionable retry feedback for silence, low level, clipping, denied permission, disconnected input or stalled capture. Keep consent and the explicit assessment-start action.
- Remove mandatory listen-back acknowledgement. Optional playback can remain useful. Describe the outcome as input readiness; audio energy does not prove intelligibility or that the candidate heard playback.
- Update the API contract honestly rather than submitting a fabricated
heardPlayback: true. Keep ownership, consent, CSRF, minimum-duration checks and assessment deadlines. Browser-reported diagnostics are readiness telemetry, not trusted assessment evidence. - Keep microphone-test audio local as currently promised. Release tracks, timers, audio contexts and object URLs on retry/navigation/error.
- Treat Behavioral as the immediate affected flow and align the English readiness flow where practical. The user did not specify different behavior by assessment type.
Update the relevant fixtures in Behavioral API tests, unified tests, demo fixture, and both Behavioral and English browser suites. The current Behavioral browser test explicitly waits for playback and checks the acknowledgement, so it must change with the interaction.
Verify valid speech, silence, low input, clipped input, permission denial, disconnect and retry. Confirm failed readiness cannot begin an assessment and that passing readiness requires no playback checkbox. Preserve existing recording, recovery, score visibility and role tests.
Update both Behavioral and English usage instructions for any changed screens, their technical guides, and app invariants if relevant. Existing user-guide nav entries are in mkdocs.userguide.yml. Regenerate API documentation when changing the microphone endpoint.
Source map¶
| Files | Responsibility |
|---|---|
| src/main.tsx, Home.tsx | Unified navigation and assessment availability |
| server/auth.ts | OIDC identity, provisioning, entitlement refresh, reviewer/admin distinction |
| src/bpaas, server/bpaas, shared/bpaas.ts | Behavioral UI, routes/domain/scoring/schema and shared types |
| server/voice.ts, interview.ts | English voice gateway and guided exchanges |
| server/domain.ts, evidence.ts, worker.ts | English lifecycle, evidence and durable jobs |
| server/migrate.ts | Additive English and Behavioral migrations |
| bootstrap-authentik.py | App-specific Authentik setup; inspect without --apply |
| deploy/compose.yml, nginx.conf | Dedicated runtime and host routing |
| scripts/openapi.ts | Generated app inventory and rendered API reference |
Verification evidence and commands¶
The original thread and verification report record:
- 50 automated tests and seven browser tests passed; lint, TypeScript and production build passed.
- The real-provider synthetic demo completed in 31 seconds with three substantive responses, a repeated-question response and a missed question: 3/5 meaningful, provisional score 97/100.
- result.json records
gpt-audio-1.5andgpt-5.6-terra, with 5,936 input and 2,694 output tokens. - Deployment retained five app users and four English attempts. The saved sealed English submission and 130 encrypted evidence objects passed integrity checks.
- SSO role API checks confirmed administrator and ordinary reviewer separation. Automated account-creation tests used a mock OIDC provider; the production browser check stopped at corporate sign-in.
These are prior implementation results, not newly rerun test results. This handoff only rechecked live health/image identity and documentation references. The synthetic score/timing is demonstration evidence, not validated hiring accuracy or measured reviewer savings.
Run normal verification inside the existing app checkout:
cd /workspaces/trueserve-framework/apps/proficiency
npm run typecheck
npm run lint
npm test
npm run build
npm run test:e2e
# After an API contract change:
npm run openapi
API/browser fixtures create disposable PostgreSQL schemas and temporary evidence directories. Use the configured dedicated database and preserve existing secrets. PLAYWRIGHT_CHROMIUM_EXECUTABLE can select a compatible installed browser.
npm run test:bpaas-demo, npm run test:provider and npm run test:pipeline make real provider calls with synthetic audio. Use them when the affected path warrants it. The English pipeline smoke and browser suite both use port 3052 and must not run together. Do not run live migrations merely to prepare a handoff.
Preservation and remaining limits¶
The root .gitignore explicitly ignores apps/proficiency/* and docs/03-services/frontend/proficiency.md. Only the app's .env.example is currently tracked. Source, app instructions, generated inventory, deployment scripts and artifacts are local. The previous thread created no commit. The root workspace also contains extensive unrelated changes.
Before transferring machines or using a new worktree, preserve the app source and ignored technical guide through a reviewed source-only copy/archive. Exclude .env.local, credentials, evidence, database backups, node_modules, build output and private transcripts. Never use git clean -fdx, discard this checkout, or assume checkout/clone recreates the app. If committing later, resolve the intended source ownership/ignore policy and stage only the relevant files.
Remaining product/operational limits already documented:
- The EDJ rubric uses generic customer-service principles; proprietary role policies and a real recruitment contact are still needed.
- Manual review baseline and reviewer time savings have not been established empirically.
- Behavioral recordings currently use encrypted PostgreSQL storage; scaled deployment needs a storage review.
- English/Behavioral AI assessments remain research/demo estimates requiring human review and validation.
- Certificate installation is manual after renewal. The primary certificate expires April 29, 2027; the legacy redirect certificate expires March 31, 2027. No unattended renewal-install job was added.
Prompt for the next AI thread¶
Continue the unified Assessments app in /workspaces/trueserve-framework/apps/proficiency.
Read docs/_planning/assessments-ai-handoff.md and the app's CLAUDE.md first.
The old thread 01a13ae0-5bd8-7493-90e2-652133b90b21 is conversation_blocked.
The unified app, assessments hostname and Authentik API rollout are already deployed.
Preserve the ignored local app source and unrelated workspace changes.
Implement the last outstanding request: automate the microphone test from audio
data flow and reasonable volume, removing mandatory playback acknowledgement.
Preserve invitation and SSO identity separation, hidden candidate AI scores, SSO reviewer access,
separate administrator permissions, immutable answers and existing evidence.
Update relevant API contracts, fixtures, tests and documentation; report verification.