Codiatech guides
Access Control and Data Freshness for AI Document Assistants
An internal document assistant should use current sources that the person asking is authorized to access at that moment. A plausible answer is only part of the requirement. Teams also need to know which version supplied the evidence, when revoked permissions take effect and where deleted content might remain. This guide turns those questions into a practical pilot plan for retrieval-augmented generation, or RAG. The examples are illustrative. Each organization needs decisions that reflect its own sources, access model and operating responsibilities.
AI Document Assistant Pilot Acceptance Log
Record actual observations for access, document version, deletion and citation tests.
The editable workbook includes Turkish and English sheets. Fill in your own decisions and observations; no proposal prices or successful test results are prefilled.
Download pilot acceptance log (XLSX)Start with One Business Question
Choose a narrow workflow before connecting a whole document estate. An expense-policy pilot might use an approved procedure, a superseded version and a restricted manager note. Confirm that the data owner permits those documents to be processed. Include questions that are ambiguous, unanswered or supported by conflicting sources, alongside straightforward examples. This gives the team a useful picture of everyday failure conditions. Before ingestion starts, agree who owns the source material, where copies may be stored and what should happen when content is withdrawn.
Enforce Access Where the Data Is Retrieved
Authenticate the user on the server and obtain organization, group and document permissions from trusted systems. A role typed into a chat message cannot establish access. Retrieval and the endpoint that opens the original file should follow compatible authorization decisions. Hiding a link in the interface does not protect an exposed API. Check whether restricted titles, snippets, summaries or result counts reveal information. OWASP recommends validating permissions on every request and denying access by default. Record which component makes the decision and how it handles missing identity or permission information.
Recheck Access After a Role Change
Change a manager account to an employee role, keep its session open and repeat a question about a restricted document. The target behavior is that subsequent authorization checks exclude the restricted material and its source links. In a real deployment, tokens, cached group membership and existing connections can delay that change. Measure the propagation time and agree an acceptable limit with the data owner. Preventing a new response cannot retract text the person already viewed or downloaded. Decide separately how historical conversations, exported answers and previously shared links should behave after access changes.
Define What Makes a Source Current
The latest upload is not necessarily the policy currently in force. Track a stable document identity, version, approval state, effective date and any version it replaces. A procedure uploaded today may only become effective next month. Decide whether archived material is searchable and how it is separated from current guidance. The content owner should resolve precedence when two policies disagree. Keep the source's effective date separate from the index's last successful synchronization time. When the system cannot establish freshness, its response should make that limitation clear rather than silently treating an uncertain source as authoritative.
Treat Deletion as a Data Lifecycle Event
Removing an original file does not establish that every derived copy has disappeared. List the extracted passages, embeddings, search index entries, cached answers and conversation records that may contain its content. Use document and version identifiers to manage replacements and trace deletion work. Microsoft's Azure Storage indexing documentation illustrates why change detection and deletion detection require separate attention. The exact mechanism depends on the selected platform. For your project, define how withdrawal events are received, which derived stores must change and what evidence confirms completion.
- Cache Scope Define how cached answers depend on organization, access scope, document version and permission changes. Reusing a result across users requires a server-side decision that their relevant access is equivalent.
- Deletion Check Search for the withdrawn document by name and by a distinctive passage. Try its old source URL. Check that new answers and download endpoints follow the agreed withdrawal policy.
Check What Each Citation Supports
A citation proves little unless the linked passage supports the claim beside it. Review answers against their evidence, including qualifications and exceptions. Where possible, show a document version and a link to the relevant section. Check access again when the person opens that source. Test what happens if the document has moved or been withdrawn since the answer was produced. Missing evidence and conflicting procedures should lead to an explicit limitation or escalation. Instructions embedded inside a retrieved document must not become authority to change the application's permissions or perform an otherwise restricted action.
How to score a RAG pilot and assign maintenance
Ask the supplier to run a pilot on approved sample documents and questions supplied by the people who own those documents. Record each test account, expected passage, expected answer, observed result and who resolves a failure. Agree on acceptance thresholds before the run, and repeat the same checks after changes to the model, search index or access rules.
- Access and data handling Run the same question as two users with different access, then remove the privileged user's access to the restricted document while their session is open. Record how long restricted passages and their source links remain accessible. Ask where documents, embeddings, queries and logs are processed, who can access them, and what the retention, deletion and training terms are.
- Retrieval, citation and answer Use a policy passage that requires an itemized receipt and explicitly exempts one expense category. If the current passage was retrieved and the link opens it, the retrieval and link checks can pass. An answer omitting the applicable exception fails the answer check. Separately check that each cited passage supports the claim it accompanies. Add a question with no permitted source and check that the assistant says it found no supporting evidence.
- Maintenance and release Assign owners for document updates, failed synchronizations and model changes. Agree who can hold or roll back a release when a test fails, who handles reports of wrong answers, and how quickly access revocation and document withdrawal should reach the index and answer cache.
What the Codiatech Exercise Actually Demonstrates
Our downloadable Node.js exercise is a small source-selection example using synthetic expense documents. It contains seven deterministic checks covering current sources and citations, restricted material, manager access, superseded content, missing evidence, human approval and role revocation. After the same account changes from manager to employee, the restricted source is excluded from its next answer while the current employee procedure remains available. The script returns the selected passage directly. It uses no language model, real identity provider, vector database or distributed cache. It therefore does not establish generated-answer quality, production security or deletion propagation.
Download the file and run node document-assistant-lab.mjs to inspect the checks in your environment. All policies and roles are fictional. This is a reproducible exercise, not a customer outcome. The sources below provide implementation guidance; the script shows the full, deliberately limited example.
Write Acceptance Tests Before the Pilot Expands
For each scenario, record the user, source version, initial state, change applied, expected behavior and observed result. Include successful access as well as requests that must be refused. In the actual application, repeat checks with an existing session and a newly authenticated session after changing permissions. Interrupt synchronization during an update or deletion to inspect failure behavior. The following checklist is a proposal to adapt to a project. It is broader than the synthetic exercise and should not be presented as a list of production capabilities already validated by that script.
- Permissions An authorized user gets the intended source. An unauthorized user cannot retrieve its text, summary or file. Measure subsequent request behavior after revoking the same account's role.
- Updates The approved replacement is selected once it becomes effective. If synchronization is delayed, the application follows the agreed fallback instead of presenting outdated guidance as current.
- Deletion Withdrawn passages and affected cached answers stop contributing to new queries within the agreed period. Failed cleanup is visible and retries can be inspected.
- Evidence and Decisions Citations open the correct section and support the answer. Missing evidence is acknowledged. Approval or payment requests stay within the human decision process defined for the project.
Agree Who Operates the System After Launch
Assign ownership for content approval, user access, source connectors, indexing jobs and incident response. An index that stops updating and an identity synchronization failure need different investigations. Define what happens when a model version changes and which regression checks must run again. Keep enough operational evidence to understand a decision without unnecessarily storing document text or personal data in logs. At handover, list the tests completed, the areas outside the pilot and the people responsible for maintenance. Codiatech scopes development, integration and ongoing operations per project, with responsibilities agreed before those activities are undertaken.
Technical Sources
Define a Document Assistant Pilot
Share the business question, source systems and user roles. We can assess a practical scope covering access, freshness and acceptance tests.
Request a Proposal