When a company asks us for an AI assistant, the conversation usually starts with models and channels: which model, WhatsApp or Teams, Cantonese or English. Those matter, but they are rarely what decides whether the assistant is trusted six months later. That is decided by the knowledge base it answers from. A retrieval-augmented assistant (RAG) can only be as accurate, current and consistent as the documents behind it.
These are the five questions we work through with clients before we build anything.
1. Who owns each document — and would they know if it were wrong?
Every document the assistant can cite needs an owner: a person or team who will be asked when the content is challenged. Shared drives accumulate files with no owner, and those files are exactly the ones that produce confident, outdated answers. We ask clients to list the sources for the pilot and to put a name next to each. If nobody claims a document, it does not go in.
2. Is it current, and how would the assistant know?
Price lists, HR policies, product specifications and procedures change. The assistant must be able to tell a current version from a superseded one. In practice that means agreeing a rule — for example, only documents in a specific SharePoint library, with a "valid from" date in the metadata — and connecting the assistant to that location rather than to a one-off export. An assistant that was accurate on launch day and quietly drifts afterwards is worse than no assistant, because people stop checking.
3. Who is allowed to see what?
An internal assistant often has access to more than any single employee does. If a document is restricted — salary bands, board papers, a client matter — the assistant must respect the same permissions the user already has. On Microsoft 365 and Azure this is achievable: the assistant checks the user's identity and filters retrieval by the same access lists that govern SharePoint. But it has to be designed in from the start, and it shapes which sources go into the pilot.
4. Is the content in a form a machine can read?
Scanned PDFs without a text layer, tables in images, bilingual documents with the English and Chinese interleaved, spreadsheets where the meaning lives in colour-coding — these are everyday realities in Hong Kong offices, and each one needs handling. Azure AI Document Intelligence covers a great deal, including scanned documents and tables, but some material is better rewritten than processed. We run a sample of the real documents through the pipeline during discovery so there are no surprises in week three.
5. What are the 50 questions it must get right?
Before building, we ask the client's team to write down the questions they actually get — in the language they get them, including the mixed Cantonese-English ones — and the answer they would consider correct, with the source. Fifty is a good number for a pilot. This list becomes the acceptance test: we measure the assistant against it every week, and it is what a manager signs off on, rather than a demo that went well.
What this means for the project plan
None of these questions needs a long programme. In a four-to-six-week pilot we typically spend the first week on exactly this: choosing sources, naming owners, agreeing the currency rule, checking permissions, and writing the test questions. The build is faster and the result is something the business will keep using. If you are planning an assistant and would like a second opinion on your knowledge base, we are happy to look at it with you.