How to Migrate SharePoint RAG to a Production-Ready Pipeline with Azure AI Search and Jinba

Summary

• Azure OpenAI "On Your Data" retires October 14, 2026, and the native SharePoint indexer for Azure AI Search remains in public preview, so neither is production-safe for regulated workloads.
• A safer migration path uses Sites.Selected to sync SharePoint documents to Azure Blob Storage, writes each document's ACL into allowed_principals blob metadata, and enforces those permissions at query time with an OData filter.
• Index with the GA Azure AI Search Blob indexer, use a managed identity instead of API keys for embedding calls, and configure a custom SplitSkill chunking strategy for structured SharePoint content.
• Generate answers from retrieved chunks with your private Azure OpenAI deployment and enable diagnostic logging for RequestResponse and Audit before production traffic.
• Jinba Flow orchestrates the hybrid retrieval and private generation steps in one deployable workflow, keeping prompts and completions inside your Azure boundary.

The production migration path for SharePoint RAG is a four-stage pipeline: sync documents from SharePoint to Azure Blob Storage via Microsoft Graph with Sites.Selected permissions, index with the GA Azure AI Search Blob Storage indexer, retrieve with Jinba's AZURE_AI_SEARCH tool using hybrid BM25 and vector search, and generate against your private Azure OpenAI deployment with AZURE_OPENAI_INVOKE. This guide covers each step, including the permission model, indexer configuration, and audit logging.

Two deadlines make this migration non-optional. Azure OpenAI "On Your Data" is deprecated and retires on October 14, 2026. The native SharePoint indexer for Azure AI Search remains in public preview, which means it cannot be treated as a stable production dependency for regulated or compliance-sensitive workloads. Both constraints affect teams operating SharePoint RAG today.

Why the current tools are not production-safe

Azure OpenAI "On Your Data" is already frozen. Microsoft has stopped onboarding new models; the service supports only specific versions of GPT-4o and GPT-4o-mini. Any workload that depends on it will need to migrate before October 2026, and starting that migration now gives teams time to own the replacement properly.

The native SharePoint indexer for Azure AI Search introduces a different risk. Because it remains in public preview, it can process and store data outside your defined Azure compliance boundary. For enterprises with data residency requirements, that is a disqualifying constraint, not a configuration problem. The indexer also produces citations that do not resolve back to the originating SharePoint document, which undermines the utility of retrieval results for end users.

The pipeline described in this guide avoids both constraints. It uses only GA services for data movement and indexing, keeps all compute inside your Azure boundary, and returns a direct download URL for every retrieved document.

Step 1: Sync SharePoint to Azure Blob Storage with Sites.Selected

Most published guides request Files.Read.All or Sites.FullControl.All at the tenant level. These are tenant-wide permissions that violate the principle of least privilege and create unnecessary exposure across every site in the organisation.

The correct approach uses the Sites.Selected application permission. With Sites.Selected, the app registration receives no access by default. Access is granted explicitly, site by site, via a Microsoft Graph API call or PnP PowerShell. No site is readable unless you have approved it.

To grant access to a specific site using PnP PowerShell:

Connect-PnPOnline -Url "https://<tenant>-admin.sharepoint.com" -Interactive
Grant-PnPAzureADAppSitePermission `
-AppId "<your-app-registration-client-id>" `
-DisplayName "<your-app-name>" `
-Site "https://<tenant>.sharepoint.com/sites/<site-name>" `
-Permissions Read

Once the permission is in place, use a Logic App or Azure Function with the Microsoft Graph Files API to export documents from the approved SharePoint sites into an Azure Blob Storage container.

Materialising permissions into blob metadata is the most important step in this stage. The Blob Storage indexer has no awareness of SharePoint access control lists. If you skip this, every indexed document becomes queryable by every user. For each document exported:

1. Query SharePoint via Graph to retrieve the ACL for that item.
2. Extract the Entra ID object IDs of all users and groups with read access.
3. Write those IDs into a metadata field on the blob, for example allowed_principals, as a comma-separated list.

This metadata becomes the source of truth for query-time permission filtering in Step 3.

Step 2: Index with the GA Azure AI Search Blob Indexer

With documents in Blob Storage, configure four Azure AI Search resources: a data source, an index, a skillset, and an indexer.

The data source points to your Blob Storage container. The index schema must include a field that maps to the allowed_principals metadata, a vector field for embeddings, and source URL fields so that citations resolve correctly. The skillset calls your Azure OpenAI embedding model to vectorize each chunk during indexing.

Use a managed identity, not API keys. Enable a system-assigned managed identity on your Azure AI Search resource. Then assign it the Cognitive Services OpenAI User role on your Azure OpenAI resource. This allows the indexer's skillset to call the embedding model for vectorization without any hardcoded credentials in your configuration.

az role assignment create \
--assignee "<ai-search-managed-identity-principal-id>" \
--role "Cognitive Services OpenAI User" \
--scope "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<aoai-resource-name>"

Chunking strategy matters. Azure AI Search's default chunking is a fixed token split, which performs poorly on structured documents such as SharePoint wiki pages, tabular content, or multi-section policy documents. A common outcome is that retrieved chunks lack the surrounding context needed to answer a query accurately. Define a custom SplitSkill configuration in your skillset that matches the structure of your documents: smaller chunks for dense reference material, larger ones for narrative documents.

Once the indexer runs, verify the index contains the allowed_principals field, the vector field, and a resolvable source URL for each document. These three fields are required for the retrieval and citation steps that follow.

Step 3: Orchestrate Retrieval and Generation with Jinba

Jinba connects the index to the application. Register your Azure AI Search instance as an external knowledge base in your Jinba workspace. Connection credentials are stored in the workspace registration and are never written into your flow manifest.

Retrieval with AZURE_AI_SEARCH

The AZURE_AI_SEARCH tool performs hybrid retrieval by default, combining BM25 keyword matching with vector similarity in a single ranked result set. This is the azure ai search integration that closes the gap between keyword recall and semantic relevance without requiring separate query passes.

A sample retrieval step in a Jinba flow:

- tool: AZURE_AI_SEARCH
params:
query: "{{user_query}}"
externalKbId: "{{kb.sharepoint_docs}}"
indexName: "sharepoint-index"
filter: "allowed_principals/any(p: p eq '{{user_entra_object_id}}')"

The filter parameter is an OData expression that evaluates against the allowed_principals field written during the sync step. A document only appears in results if the requesting user's Entra object ID is present. This enforces SharePoint-equivalent permissions at query time, without any round-trip to SharePoint.

The tool returns id, score, content, filename, and a downloadUrl for each result. That downloadUrl resolves directly to the source file in SharePoint or Blob Storage. This directly addresses the citation gap in the preview SharePoint indexer, where out-of-the-box results do not link back to the originating document.

Generation with AZURE_OPENAI_INVOKE

Pass the retrieved content to AZURE_OPENAI_INVOKE to generate a grounded answer against your own Azure OpenAI deployment:

- tool: AZURE_OPENAI_INVOKE
params:
deployment: "{{env.AZURE_OPENAI_MODEL_DEPLOYMENT_NAME}}"
endpoint: "{{env.AZURE_OPENAI_ENDPOINT}}"
api_key: "{{secrets.AZURE_OPENAI_API_KEY}}"
messages:
- role: system
content: "Answer using only the provided context. Cite the source filename."
- role: user
content: "Context: {{search_results.content}}\n\nQuestion: {{user_query}}"

The deployment field references your named Azure OpenAI model deployment. Credentials are passed through environment variables or Jinba's secrets syntax, not hardcoded in the manifest. Because AZURE_OPENAI_INVOKE targets your private deployment, all prompt and completion traffic stays within your Azure compliance boundary.

Where your documents include images or complex layouts, the AZURE_OPENAI_INVOKE_WITH_FILE variant extends the same private deployment to multimodal inputs.

A complete walkthrough of the Jinba RAG flow pattern, including index configuration and semantic ranking, is available in the Jinba RAG chat tutorial.

Step 4: Enable Audit Logging via Diagnostic Settings

Azure OpenAI resource logs are not collected or stored by default. If your organisation requires an audit trail of prompts, completions, or model invocations, you must create a diagnostic setting manually before any production traffic runs.

To configure diagnostic settings on your Azure OpenAI resource:

1. Navigate to your Azure OpenAI resource in the Azure Portal.
2. Under Monitoring, select Diagnostic settings.
3. Click Add diagnostic setting.
4. Select the log categories to capture. For a production audit trail, enable RequestResponse and Audit.
5. Choose a destination:
6. • Log Analytics workspace: supports Kusto queries for investigation and alerting, and incurs ingestion and retention costs.
   • Azure Storage account: lower cost, suitable for long-term archival, but not queryable in place.
   • Event Hubs: streams logs downstream to a SIEM or custom processing pipeline.

Configure this setting before the pipeline handles real user queries. Retroactive log recovery is not possible for requests that occurred before the diagnostic setting existed.

What you control by owning this pipeline

Building this pipeline removes three dependencies that would otherwise constrain future work:

• Chunking control. The default Azure AI Search chunking degrades retrieval quality for structured SharePoint content. Owning the skillset means you can configure chunk size and overlap per document type.
• Permission enforcement. The Sites.Selected app registration, combined with OData filter expressions at query time, replicates SharePoint ACL logic without any preview-stage indexer involvement.
• Citation fidelity. Every result from AZURE_AI_SEARCH carries a downloadUrl. Users and downstream applications receive a direct link to the source document, not a generic reference.
• Compliance boundary. The Blob Storage indexer is GA. AZURE_OPENAI_INVOKE targets your private deployment. No step in the pipeline routes data outside your defined Azure environment.

The October 14, 2026 retirement of Azure OpenAI "On Your Data" creates a fixed deadline, but the preview status of the native SharePoint indexer creates a risk that is active now. A team that builds this pipeline today inherits a GA architecture, a clear permission model, and an audit trail that pre-dates the migration deadline.

Start with a single SharePoint site, grant it Sites.Selected access, validate that allowed_principals is correctly written to blob metadata, run the indexer, and confirm that a test query through AZURE_AI_SEARCH with a known user's Entra object ID returns only documents that user can access in SharePoint. That end-to-end validation is the prerequisite for a safe production rollout.

Frequently asked questions

What is the Azure OpenAI On Your Data retirement date?

Azure OpenAI “On Your Data” retires on October 14, 2026. Workloads that still depend on this service must migrate before that date; after retirement, supported model access may be lost. This guide replaces On Your Data with a GA Azure AI Search Blob indexer and private Azure OpenAI generation.

Why is the native SharePoint indexer for Azure AI Search not production-ready?

Because it remains in public preview, the native SharePoint indexer can process and store data outside your defined Azure compliance boundary. It also produces citations that often do not resolve back to the source SharePoint document. For regulated or compliance-sensitive workloads, this makes the preview indexer unsuitable as a stable production dependency.

How does the Sites.Selected permission model reduce SharePoint migration risk?

The Sites.Selected application permission starts with no tenant-wide access. You grant access explicitly, site by site, using Microsoft Graph or PnP PowerShell. This avoids the broad exposure created by Files.Read.All or Sites.FullControl.All and ensures only approved SharePoint sites are synced to Azure Blob Storage.

How do I enforce SharePoint permissions in Azure AI Search?

You enforce SharePoint permissions by writing each document’s ACL to blob metadata during sync, typically in an allowed_principals field. At query time, apply an OData filter such as allowed_principals/any(p: p eq '{{user_entra_object_id}}'). This prevents users from retrieving documents they cannot access in SharePoint.

How do I configure the Azure AI Search Blob indexer with a managed identity?

Enable a system-assigned managed identity on your Azure AI Search resource. Then assign the Cognitive Services OpenAI User role on your Azure OpenAI resource to that identity. This lets the indexer skillset call your embedding model for vectorization without hardcoded API keys.

What chunking strategy works best for SharePoint documents in Azure AI Search?

A fixed token split is not ideal for structured SharePoint content. Use a custom SplitSkill configuration: smaller chunks for dense reference material, tabular content, or policies, and larger chunks for narrative documents. The goal is to preserve enough surrounding context for accurate retrieval.

How do I audit Azure OpenAI prompts and completions in production?

Configure diagnostic settings on your Azure OpenAI resource before production traffic runs. Enable the RequestResponse and Audit log categories, then choose Log Analytics, Azure Storage, or Event Hubs as the destination. Azure OpenAI resource logs are not collected by default, so this step is required for an audit trail.

How does Jinba’s AZURE_AI_SEARCH tool combine BM25 and vector search?

AZURE_AI_SEARCH performs hybrid retrieval by default. It combines BM25 keyword matching with vector similarity in a single ranked result set, closing the gap between keyword recall and semantic relevance without separate query passes.

Can AZURE_OPENAI_INVOKE keep SharePoint RAG requests inside my Azure compliance boundary?

Yes. AZURE_OPENAI_INVOKE targets your private Azure OpenAI deployment, so prompt and completion traffic remains within your Azure compliance boundary. Use environment variables or Jinba secrets for credentials instead of hardcoding them in the flow manifest.

What should I validate before rolling out this SharePoint RAG pipeline to production?

Start with a single SharePoint site and validate the entire path. Grant Sites.Selected access, sync documents to Blob Storage, confirm allowed_principals is written correctly, run the indexer, and test a query with a known user’s Entra object ID. The test should return only documents that user can access in SharePoint.