How to Migrate SharePoint RAG to a Production-Ready Pipeline with Azure AI Search and Jinba

Summary

• Azure OpenAI On Your Data retires October 14, 2026, and Microsoft has stopped onboarding new models; any dependent RAG build needs a migration path before that date.
• The native SharePoint indexer for Azure AI Search is a preview feature that can process or store data outside the Azure compliance boundary, so it is not production-ready for regulated enterprises.
• The stable alternative is to sync SharePoint documents to Azure Blob Storage via Microsoft Graph API, then index with the GA Blob Storage indexer and enable hybrid BM25 + vector search.
• Grounded generation and a complete audit trail require answering only from retrieved context, citing sources, and explicitly configuring Azure OpenAI diagnostic settings, which are not on by default.
• For production SharePoint RAG, Jinba Flow connects Azure AI Search hybrid retrieval to private Azure OpenAI generation and captures full workflow history for audit.

Building a retrieval-augmented generation (RAG) chatbot over SharePoint presents three recurring failure modes: retrieval pulls irrelevant chunks, the LLM generates answers that contradict the source documents, and the compliance team requires a full audit trail that does not exist.

These failure modes are the predictable result of two architectural shortcuts that many teams take by default: relying on the native Azure AI Search SharePoint indexer for ingestion, and relying on the managed Azure OpenAI On Your Data feature for generation. Both paths close soon. Azure OpenAI On Your Data is deprecated and retires on October 14, 2026; Microsoft has stopped onboarding new models. The SharePoint indexer for Azure AI Search is a preview feature offered "as-is" with best-effort support, explicitly not recommended for production, and not guaranteed to reach general availability.

Regulated enterprises cannot build on either. This guide defines the production path: ingest SharePoint documents through a stable pipeline, index them in Azure AI Search with hybrid search enabled, retrieve grounded context through Jinba, generate answers with a private Azure OpenAI instance, and log the full query-to-answer lineage for audit purposes.

Why the Default Paths Fail in Regulated Environments

The native SharePoint indexer uses preview API version 2026-08-01-preview. Microsoft's documentation states that it "can process or store data outside the Azure compliance boundary" and carries a timing lag in recognizing permission and access control changes. For any organization subject to data residency controls, that lag is a compliance exposure.

The managed On Your Data feature removes that concern but introduces a different one: it is being retired. Any team that builds on it today faces a forced migration before October 2026.

The retrieval quality problem is separate but equally important. Without hybrid search combining BM25 keyword matching with vector search, systems routinely surface irrelevant chunks and rank them above the correct ones. The LLM then generates answers from poor context, and the results appear hallucinated even when the model is behaving correctly. The problem is the data fed to it, not the model itself.

A production architecture addresses all three constraints: a stable ingestion pipeline, hybrid retrieval, and verifiable data boundaries.

The Architecture at a Glance

The workflow has four steps:

1. Sync and index: Pull SharePoint documents into Azure Blob Storage via the Microsoft Graph API, then populate an Azure AI Search index using the stable Blob Storage indexer.
2. Retrieve: Query the index using Jinba's native AZURE_AI_SEARCH tool with hybrid search enabled.
3. Generate: Chain the retrieved chunks into Jinba's AZURE_OPENAI_INVOKE tool, backed by a private Azure OpenAI instance.
4. Serve and audit: Deliver cited answers in a chat interface and capture the full execution trail across both the Jinba orchestration layer and Azure OpenAI resource logs.

Each step is covered in detail below.

Step 1: Ingest SharePoint Documents into Azure AI Search

The production-ready ingestion path avoids the preview SharePoint indexer entirely. It uses a two-stage approach that keeps the compliance boundary intact.

Stage 1: Sync SharePoint to Azure Blob Storage

Use the Microsoft Graph API to export documents from the SharePoint site into an Azure Blob Storage container. This SharePoint workflow automation step decouples the compliance-sensitive SharePoint permission model from the indexing process. The app registration requires Sites.Selected Graph API permissions scoped to the specific site, rather than broad tenant-level access. Documents land in Blob Storage under the organization's control, within the Azure geography, before any indexer touches them.

This approach supports the file types most common in enterprise knowledge bases: .txt, .md, .html, .docx, .pptx, and .pdf.

Stage 2: Index into Azure AI Search

Once documents are in Blob Storage, the Azure Blob Storage indexer, a stable, GA feature, handles cracking, chunking, and embedding into the Azure AI Search instance. This is the same canonical ingestion flow that Microsoft describes for grounded generation: source documents are cracked into text, split into chunks, and embedded into the search index.

Configure Hybrid Search

When creating the index, enable both the keyword (BM25) field and a vector field for embeddings. This is the Azure AI Search integration configuration that resolves the irrelevant-chunk problem. Keyword search handles exact terminology; vector search handles semantic similarity. Together they produce ranked results that are both lexically and conceptually relevant to the query. Set the index to use hybrid as the default searchType; this is also the default in Jinba's AZURE_AI_SEARCH tool, which means the two layers align without additional configuration.

Step 2: Retrieve Context with Jinba's Azure AI Search Tool

Jinba provides a native AZURE_AI_SEARCH tool that queries the index registered in the Jinba workspace. Registering the index as a knowledge base in Jinba takes a connection string and an index name; after that, Jinba Flow handles the retrieval logic.

The tool's key parameters:

• query: the submitted question, passed directly from the chat input
• externalKbId: the ID of the registered Azure AI Search knowledge base in Jinba
• indexName: the name of the target search index created in Step 1
• searchType: set to hybrid for combined BM25 and vector retrieval (the default)
• answerTotal: number of document chunks to return (default 5, maximum 1000)
• vectorRetrievalCount: number of vectors used in the retrieval pass (default 5, maximum 1000)

The tool returns, for each retrieved chunk: id, score, content, filename, and downloadUrl. The downloadUrl is the direct link to the source file in SharePoint or Blob Storage. This is what makes citations functional: not just a filename displayed to the user, but a clickable link back to the original document that an auditor or end user can verify.

The score field enables downstream filtering. Organizations that need to suppress low-confidence results can add a score threshold in the Jinba Flow before the retrieved chunks reach the generation step.

Step 3: Generate Grounded Answers with a Private Azure OpenAI Instance

The retrieved chunks from Step 2 pass directly into Jinba's AZURE_OPENAI_INVOKE tool, which calls the private Azure OpenAI deployment. No data transits through OpenAI's public infrastructure.

Microsoft's data privacy documentation for Azure OpenAI states the guarantees explicitly: prompts, completions, embeddings, and training data are not available to other customers, not available to OpenAI, and not used to train any generative AI foundation model without permission. The models are stateless, and data is processed only within the customer-specified geography. For regulated enterprises, the query, the retrieved chunks, and the generated answer all remain within the defined compliance boundary.

Grounding the Prompt

The system prompt is where retrieval quality translates into answer quality. A prompt structured for grounding should instruct the model to:

• Answer only from the provided context chunks
• State explicitly when the context does not contain enough information to answer the question
• Cite the source filename and, where available, the downloadUrl for each claim

This directly counters the failure pattern where the model generates plausible-sounding answers that do not appear in the source documents. If the retrieved chunks do not contain the answer, a grounded prompt surfaces that gap rather than filling it with inference.

The Jinba AZURE_OPENAI_INVOKE step accepts the structured output of AZURE_AI_SEARCH as its context input, so no custom serialisation is required between the two tools. The chain is: user query -> AZURE_AI_SEARCH -> retrieved chunks with metadata -> AZURE_OPENAI_INVOKE with grounding prompt -> cited answer.

Step 4: Audit Logging Across Both Layers

For regulated enterprises, a defensible audit trail requires reconstructing the full lineage of any answer: which query was submitted, which chunks were retrieved, and what the model returned. That lineage spans two systems, and each requires explicit configuration.

Layer 1: Jinba Orchestration Logs

Jinba's History and Versions feature captures the full execution history of each workflow run. Every invocation of the Jinba Flow, including the input query, the tool calls made, the chunks retrieved, and the final answer, is recorded at the orchestration layer. This covers the attribution of each query, which compliance teams review first.

Layer 2: Azure OpenAI Resource Logs

Resource log configuration is frequently omitted. Azure OpenAI's platform metrics and the Azure Monitor activity log are collected automatically, but resource logs, which contain the request and response detail needed for audit purposes, are not collected until a diagnostic setting is created.

The required action: in the Azure portal, navigate to the Azure OpenAI resource, open Diagnostic settings, and create a setting that routes resource logs to a destination. The destination options are Log Analytics workspace, Azure Storage, or Azure Event Hubs. Without this setting, the detailed prompt-and-completion records do not persist, and the audit trail has a gap at the generation layer.

Azure's default dashboards cover usage and telemetry. They do not surface the request-level detail that a compliance review requires. The diagnostic setting is mandatory for any regulated deployment.

With both layers configured, the complete audit trail covers: the original user query (Jinba History), the chunks retrieved with their scores and source URLs (Jinba History), the prompt sent to the model (Azure resource logs), and the completion returned (Azure resource logs).

What to Build Next

The four-step architecture produces a production-grade RAG system: stable ingestion, hybrid retrieval, private generation, and full auditability. The deprecation of Azure OpenAI On Your Data and the preview status of the native SharePoint indexer make this custom pipeline the only viable path for regulated enterprises planning beyond October 2026.

The next decision point is retrieval tuning. Start with the default answerTotal of 5 chunks and monitor the score distribution across real queries. If the score spread is flat, with many chunks returning similar scores, the index configuration or chunk size should be adjusted. The vectorRetrievalCount parameter gives control over the breadth of the vector search pass independently of the final chunk count returned.

Audit log coverage should be verified before the first production query. Confirm that the Azure OpenAI diagnostic setting is active and that resource logs are appearing in the chosen destination. A silent misconfiguration at this layer creates a compliance gap that is only discovered during an audit review.

Frequently Asked Questions

What is the best way to build a production RAG chatbot over SharePoint?

The recommended approach is a custom pipeline: sync SharePoint documents to Azure Blob Storage via Microsoft Graph API, index them with the stable Azure Blob Storage indexer into Azure AI Search with hybrid search enabled, retrieve context using Jinba's AZURE_AI_SEARCH tool, and generate answers with a private Azure OpenAI instance via Jinba's AZURE_OPENAI_INVOKE tool. This avoids deprecated and preview features, maintains data residency, and provides full auditability.

Why is the native SharePoint indexer for Azure AI Search not recommended for production?

The native SharePoint indexer is a preview feature that Microsoft offers "as-is" with best-effort support and explicitly states it can process or store data outside the Azure compliance boundary. It also has a timing lag in recognizing permission changes. For regulated enterprises, these are unacceptable compliance risks. Use the Graph API to sync documents to Blob Storage and then use the stable Blob Storage indexer instead.

How do I enable hybrid search in Azure AI Search for a SharePoint RAG pipeline?

When creating an Azure AI Search index, define both a keyword (BM25) field and a vector field for embeddings. Then set the query searchType to hybrid to combine lexical and semantic retrieval. In Jinba's AZURE_AI_SEARCH tool, hybrid is the default searchType, so no extra configuration is needed when the tool is used with defaults or explicitly set.

What are the steps to ingest SharePoint documents into Azure AI Search?

There are two stages: (1) Sync SharePoint documents to Azure Blob Storage using the Microsoft Graph API with Sites.Selected permissions scoped to the specific site. (2) Configure the Azure Blob Storage indexer (GA) to crack, chunk, and embed the documents into Azure AI Search. This decouples SharePoint permissions from indexing and keeps data within the Azure geography.

How can I ensure my SharePoint RAG chatbot meets data residency and compliance requirements?

Use only GA, stable Azure services and tooling. Avoid the preview SharePoint indexer and deprecated Azure OpenAI On Your Data. Sync data via Graph API into an organization-controlled Blob Storage container, use a private Azure OpenAI instance (which does not share data with OpenAI or other customers), and configure audit logging at both the orchestration layer (Jinba History) and the Azure OpenAI resource level (diagnostic settings). Additionally, scope Graph API permissions to specific sites and verify that all processing occurs in the chosen Azure geography.

How do I set up audit logging for Azure OpenAI in a RAG pipeline?

In the Azure portal, go to the Azure OpenAI resource, open Diagnostic settings, and create a new diagnostic setting that routes resource logs to a Log Analytics workspace, Azure Storage, or Azure Event Hubs. Without this, only platform metrics and activity logs are collected, and the detailed prompt-and-completion records required for audit are not persisted. The Jinba History and Versions feature is also required to capture the orchestration-level lineage.

What happens when Azure OpenAI On Your Data is retired?

Azure OpenAI On Your Data is deprecated and will retire on October 14, 2026. Microsoft has stopped onboarding new models. Any solution built on it must be migrated before that date. The recommended replacement is a custom architecture like the one described here: sync SharePoint to Blob Storage, index with Azure AI Search, retrieve with Jinba, and generate with a private Azure OpenAI instance.

How many chunks should I retrieve in a SharePoint RAG pipeline?

Start with the default answerTotal of 5 chunks in Jinba's AZURE_AI_SEARCH tool. Monitor the score distribution across real queries. If scores are flat (many chunks with similar scores), adjust the index configuration or chunk size. The vectorRetrievalCount parameter can also be tuned independently to control the breadth of the vector search pass without changing the final chunk count.

How do I prevent hallucinations in a RAG chatbot over SharePoint?

Use hybrid search to retrieve relevant chunks, and then ground the generation with a strict system prompt that instructs the model to answer only from the provided context, explicitly state when the context lacks sufficient information, and cite sources with filenames and URLs. Jinba's AZURE_OPENAI_INVOKE tool accepts the structured output of AZURE_AI_SEARCH as context, which enforces grounding through structured context.