How to Build a RAG Chatbot Over SharePoint with Azure AI Search and Jinba

Summary

• SharePoint RAG requires four integrated components: Azure AI Search indexing, identity-based security filtering at query time, private Azure OpenAI generation, and complete audit logging.
• Hybrid search with semantic reranking achieves NDCG@3 of 60.1, outperforming vector-only search (43.8) by a wide margin on Microsoft benchmarks.
• The SharePoint indexer remains in preview, so isolate its risk behind a stable orchestration layer rather than coupling directly to it.
• Azure OpenAI states that prompts, completions, and embeddings are not used for training and remain private to the deployment.
• To build and run this pattern with compliance-grade controls, Jinba Flow provides the orchestration, OData filtering, and audit logging needed for regulated enterprises.

When an auditor asks for a specific policy or a staff member needs the exact clause that governs a decision, the answer is buried somewhere in SharePoint. People dig through folders, ask colleagues, and lose hours reopening documents they have already read. The fix is making that content immediately queryable: policies, contracts, compliance procedures, and financial reports, answered in seconds with a citation back to the named source document, and in a way that still respects department boundaries and compliance requirements.

Building a Retrieval-Augmented Generation (RAG) system over SharePoint requires four capabilities delivered in sequence: indexing documents into a retrievable store, enforcing access control at query time, generating grounded answers with a private model, and logging every transaction for audit. Each step has a specific technical requirement that shapes the choice of tooling.

The architecture described here connects SharePoint to Azure AI Search for indexing and retrieval, uses Jinba as the orchestration and compliance layer, and routes generation through a private Azure OpenAI deployment. This is a compliance-first pattern built for production.

The Four-Component Stack

The full data flow consists of four stages:

1. The Azure AI Search indexer connects to SharePoint Online libraries, chunks documents, generates embeddings, and populates a hybrid index containing both keyword and vector fields.
2. A user submits a question. Jinba receives the query, attaches the appropriate security filter derived from the user's identity, and calls Azure AI Search using hybrid search to retrieve the most relevant document chunks.
3. The retrieved chunks are injected as context into a prompt sent to a private Azure OpenAI model, which generates a grounded answer referencing only the provided material.
4. Jinba returns the answer to the user with citations linking directly to the source files in SharePoint, and writes a complete transaction record to the audit log.

Each stage addresses a distinct risk in a deliberate sequence.

Step 1: Index SharePoint with Azure AI Search

Connect the SharePoint indexer

The SharePoint in Microsoft 365 indexer connects Azure AI Search directly to SharePoint Online document libraries. It indexes files, list items, pages, and associated metadata incrementally, picking up new and modified content automatically and detecting deletions.

This indexer requires Azure AI Search on the Basic tier or higher. It does not support OneDrive as a data source.

The SharePoint indexer is currently in preview and requires risk isolation before production use. Microsoft's documentation states the feature is offered "as-is" under Supplemental Terms with best-effort support, is explicitly not recommended for production workloads, and is not guaranteed to reach general availability. For regulated enterprises with production SLAs, this is a material risk. The mitigation is to build against Jinba's stable API layer rather than directly coupling application logic to the preview indexer. The underlying connector can change without breaking the retrieval interface.

Design the hybrid index schema

A hybrid-capable Azure AI Search index stores vector and non-vector fields in the same index. For a SharePoint RAG system, the schema should include:

• A key field (string type) as the unique document chunk identifier
• A vector field such as content_vector, typed as Collection(Edm.Single) with dimensions set to 1536 to match Azure OpenAI's text-embedding-ada-002 embedding model
• A content field containing the human-readable text of each chunk, used for keyword search and for grounding the language model
• Metadata fields for security filtering: department, site_id, library_name, and sensitivity_label are common choices for enterprise SharePoint environments

The metadata fields are where access control is implemented. Vector fields in Azure AI Search cannot be filterable, facetable, or sortable. Security trimming and data isolation must therefore be applied through filter predicates on standard metadata fields. This filter-based approach is the intended pattern.

The SharePoint workflow automation that populates this schema runs continuously via the incremental indexer, keeping the knowledge base current without manual intervention.

Step 2: Orchestrate Retrieval with Jinba

Use hybrid search by default

The Jinba AZURE_AI_SEARCH tool supports text, vector, and hybrid search modes in a single interface, defaulting to hybrid. Hybrid search is the right default for enterprise content.

Microsoft's own benchmarks on customer data quantify the difference. Measuring retrieval quality by NDCG@3:

• Keyword-only search: 40.6
• Vector-only search (Ada-002): 43.8
• Hybrid search: 48.4
• Hybrid with semantic reranker: 60.1

Hybrid search fuses BM25 keyword matching with vector similarity using Reciprocal Rank Fusion (RRF), then applies a semantic reranker to reorder results by meaning. The performance difference between vector-only search and hybrid search with reranking is significant. For SharePoint content that includes policy documents, procedures, and financial tables, the keyword signal is often more reliable than the embedding alone. Hybrid retrieval captures both.

The Azure AI Search integration within Jinba abstracts this complexity. Operations teams configure searchType, answerTotal, and vectorRetrievalCount through a structured tool call rather than managing the retrieval pipeline directly.

Enforce row-level security via OData filters

The most common access control requirement for multi-department SharePoint environments is preventing Finance documents from appearing in HR query results and vice versa. The Jinba AZURE_AI_SEARCH tool includes a filter parameter that accepts an OData filter string, applied server-side on Azure AI Search before any results are returned to the orchestrator.

A workflow that authenticates the user and reads their department attribute constructs the filter dynamically:

{
"tool": "AZURE_AI_SEARCH",
"query": "What were the Q3 revenue projections?",
"externalKbId": "your-sharepoint-kb-id",
"indexName": "sharepoint-hybrid-index",
"searchType": "hybrid",
"filter": "department eq 'Finance'"
}

This condition ensures that only chunks tagged department: Finance are evaluated at retrieval time. The access boundary is enforced at the index query layer rather than in application logic. Documents shared across departments can carry multiple department tags in the metadata field, supporting cross-departmental content without relaxing the filter entirely.

The full parameter reference for the Jinba tool, including skip, select, and vectorRetrievalCount, is available in the Jinba documentation.

Step 3: Generate Answers with a Private Azure OpenAI Model

Ground the answer in retrieved context

The content field from each chunk returned by the Jinba retrieval step is inserted into the prompt before it reaches the language model. The system instruction should be explicit: the model must answer using only the provided context and must not draw on outside knowledge. This keeps the model from hallucinating information absent from the indexed documents.

No special configuration is required beyond structuring the prompt to separate context from the user question. The grounding constraint is a prompt engineering decision rather than a platform feature.

Data privacy guarantees

Regulated enterprises require assurance about where data goes once it enters an AI model. Azure's data privacy documentation states:

• Prompts (inputs), completions (outputs), embeddings, and training data are not available to other customers.
• They are not available to OpenAI or other providers.
• They are not used to improve any model.
• The models are stateless: no prompts or completions are stored within the model itself.

Deployments run within the customer's specified Azure geography. For full network isolation, configure Azure Private Link to keep traffic off the public internet entirely. This is standard practice for regulated workloads and is documented in Microsoft's networking guidance for Azure AI services.

Step 4: Surface Citations and Capture the Audit Trail

Return citations with every answer

An answer without a source is difficult to trust in a regulated environment. Every result object returned by the Jinba AZURE_AI_SEARCH tool includes id, score, content, filename, and downloadUrl. The filename and downloadUrl fields are sufficient to build a citation layer in the chat interface: display the document name alongside the answer and link directly to the file in SharePoint. Users can verify claims without leaving the application.

This makes the retrieval step transparent. Employees can see which documents informed the answer, which supports both trust and accountability when the answer shapes a decision.

Fill the audit logging gap

Native Azure monitoring addresses operations rather than compliance. According to Microsoft's monitoring documentation for Azure OpenAI, Azure resource logs, which are the log stream relevant to an audit trail, are not collected or stored by default. A diagnostic setting must be configured explicitly. Even then, the out-of-the-box dashboards report on HTTP requests and token usage. They do not record which document chunks were retrieved, which query triggered the retrieval, or what the grounded answer contained.

For regulated industries, that gap is significant. Compliance teams typically require a record of:

• Who submitted the query and when
• Which document chunks were retrieved in response
• What answer the model produced
• Which citations were returned to the user

Jinba assembles this full transaction record automatically for every interaction. The orchestration layer sits across the entire flow, from user query to model response, and logs each step without requiring teams to instrument individual Azure services separately. This layer converts the system from an operational deployment into a compliance-grade asset.

What to Verify Before Going Live

Before this system handles production traffic in a regulated environment, confirm the following:

• The Azure AI Search index schema includes all metadata fields needed for security filtering, and those fields are populated correctly during indexing from SharePoint.
• The OData filter in each Jinba tool call is constructed from the authenticated user's identity, not self-reported input.
• The Azure OpenAI deployment is scoped to the correct geography and Private Link is configured if the organization requires network-level isolation.
• Azure resource logs are routed to a diagnostic destination, even if Jinba handles the compliance audit log. Operational logs and compliance logs serve different purposes.
• The SharePoint indexer's preview status is documented in the project's risk register, and the team has a contingency for changes to the connector before it reaches general availability.

From Architecture to Production

This four-component pattern, SharePoint as source, Azure AI Search as the hybrid index, Jinba as the orchestration and compliance layer, and Azure OpenAI as the private generation model, addresses the core requirements that enterprise RAG deployments consistently face.

Access control is enforced at the retrieval layer through metadata filtering rather than through application-level logic. Answer privacy is guaranteed by the Azure OpenAI service's data handling commitments. Audit coverage is provided by the Jinba orchestration layer, filling the gap that native Azure monitoring leaves. The preview risk of the SharePoint indexer is isolated behind Jinba's stable interface, reducing the exposure to connector changes.

To get started, register your Azure AI Search index as an external knowledge base in Jinba, configure the AZURE_AI_SEARCH tool with the appropriate filter logic for your user roles, and connect a private Azure OpenAI deployment as the generation endpoint. The Jinba documentation covers the full tool specification and knowledge base registration process.

Frequently Asked Questions

What is SharePoint RAG?

SharePoint RAG is an architecture that retrieves relevant document chunks from SharePoint libraries and uses a private language model to generate answers grounded in those chunks. Instead of training a model on SharePoint content, the system indexes documents into Azure AI Search, retrieves the most relevant pieces for each question, and passes them as context to Azure OpenAI. The result is a queryable assistant that can answer policy, compliance, and business questions with traceable citations.

How do you build a RAG chatbot over SharePoint?

Connect the SharePoint in Microsoft 365 indexer to Azure AI Search, design a hybrid index with content and metadata fields, orchestrate retrieval through a tool such as Jinba that supports OData filters and hybrid search, and generate answers with a private Azure OpenAI deployment. Each answer should include citations to the source SharePoint files. The full pattern is compliance-first because access control, privacy, and audit logging are part of the pipeline rather than added later.

Is the SharePoint indexer in Azure AI Search production-ready?

No, the SharePoint indexer is currently in preview and Microsoft does not recommend it for production workloads. The documentation states it is offered "as-is" under supplemental terms with best-effort support. For regulated environments, that means the preview status should be treated as a material risk. Organizations can isolate the risk by building against an orchestration layer such as Jinba so the retrieval interface remains stable even if the underlying connector changes.

How does row-level security work in SharePoint RAG?

Row-level security is enforced at retrieval time using metadata filters. The SharePoint indexer stores metadata such as department, site ID, library name, or sensitivity label in Azure AI Search. When a user submits a question, the orchestration layer builds an OData filter from the authenticated user's identity, such as department eq 'Finance'. Azure AI Search applies that filter server-side before returning any documents, so users never see content outside their permitted scope.

Why is hybrid search better than vector-only search for SharePoint?

Hybrid search combines BM25 keyword matching with vector similarity and then applies a semantic reranker. Microsoft benchmarks on customer data show hybrid search with reranking achieves an NDCG@3 of 60.1, compared with 43.8 for vector-only search. SharePoint libraries often include policies, procedures, and tables where exact terms and identifiers matter, so keyword signals improve retrieval quality beyond embeddings alone.

Does Azure OpenAI use my SharePoint data for training?

No. Azure OpenAI documentation states that prompts, completions, embeddings, and training data are not available to other customers, are not available to OpenAI or other providers, and are not used to improve any model. The models are stateless and do not store prompt or completion data within the model itself. Deployments can also run in the selected Azure geography and behind Private Link for network isolation.

How does Jinba provide audit logging for RAG workflows?

Jinba records a complete transaction record for every interaction: who submitted the query, when it was submitted, which document chunks were retrieved, what answer the model produced, and which citations were returned. Native Azure OpenAI monitoring focuses on requests and token usage and does not collect resource logs by default, so Jinba fills the compliance gap by logging the retrieval and generation steps as a unified audit trail.

How do citations work in a SharePoint RAG system?

Each retrieved chunk returned by the Jinba AZURE_AI_SEARCH tool includes fields such as filename and downloadUrl. The application can render the filename as a clickable citation next to the generated answer. The downloadUrl points back to the source file in SharePoint. This allows users to verify the answer against the exact document that informed it, which is important for trust, decision-making, and compliance review.

Can I use OneDrive instead of SharePoint with this architecture?

OneDrive is not supported with the native SharePoint indexer in Azure AI Search. Microsoft's SharePoint Online indexer does not support OneDrive as a data source. If OneDrive content is required, a separate ingestion path or a different connector is needed before the content enters Azure AI Search. The rest of the architecture, including hybrid retrieval, OData filtering, private generation, and audit logging, can remain the same once the documents are indexed.