Enhancing Enterprise Search with Azure AI Search: Vector Search, AI Enrichment, and Semantic Retrieval

As organizations continue investing in AI-powered digital experiences, search has become one of the most impactful areas for innovation. Enterprise users increasingly expect search experiences that understand intent, context, and meaning—not just keywords. Whether surfacing knowledge across internal systems, improving customer-facing search experiences, or preparing content for AI-driven applications, modern search capabilities are becoming a strategic business requirement.

Azure AI Search provides organizations with the ability to enrich content during indexing, leverage vector search for semantic understanding, and improve retrieval quality across both structured and unstructured data. In this post, we’ll explore how Azure AI Search can be used to enhance search experiences through AI-powered enrichment and vector-based retrieval.

AI Search Overview

Azure AI Search allows for two primary retrieval approaches: Classic Search, which we will focus on in this blog, and Agentic Retrieval, which is a topic for another day.

Classic Search is similar to traditional search implementations. A searchable index is populated through an indexer, and queries are executed against that index. What makes Azure AI Search different is its ability to enrich data during indexing and support vector-based search alongside traditional keyword search.

Scenarios

For this blog post, we’ll consider two search scenarios.

Scenario 1

In this scenario, we have a movie poster repository and want users to search not only by movie title but also by the content of the poster itself.

Scenario 2

For the second scenario, let’s consider a situation many of us have experienced: remembering the plot or details of a movie but not the title. Azure AI Search can help address this problem through vectorization and semantic search.

How to Handle Both Scenarios

Two Azure AI Search capabilities play a critical role in solving these challenges:

Vector Search

Vector search uses numeric representations of content rather than traditional full-text matching. These numeric representations, known as vectors, capture semantic meaning. Queries are converted into vectors and matched against indexed vectors based on similarity, allowing search to identify conceptually related content rather than simply matching keywords.

Skillsets

Skillsets enrich content during the indexing process. They allow custom processing pipelines to execute as content is indexed. In our examples, skillsets generate vectors and extract additional metadata that improves search quality.

Azure AI Search includes numerous built-in skills for enrichment. A complete list can be found in the Microsoft documentation.

Configuration

For this walkthrough, we’ll use Scenario 1.

To build the solution, we’ll need:

• An Azure AI Search instance
• A data source
• An index
• A skillset
• An indexer
• Two models deployed in Azure Foundry

Azure Portal blade options for index, indexer, skillset and datasource configuration

Note: The index configuration below includes vector search profiles and HNSW algorithm settings. The Under the Hood section covers how vectorization and the search algorithm work in depth.

Data Source Configuration

The data source connects Azure AI Search to an Azure Blob Storage container containing movie information.

Index Configuration

The index defines searchable fields, data types, filterability, and vector search profiles.

A key consideration is ensuring that vector dimensions match the dimensions generated during indexing. In this example, the embedding model generates vectors with 1,536 dimensions, so the index field must match that size.

Skillset Configuration

The skillset enriches content during indexing.

For this example, we:

• Analyze poster images using the built-in Image Analysis Skill.
• Flatten generated captions using the Merge Skill.
• Generate embeddings using the Azure OpenAI Embedding Skill.

These embeddings become searchable vectors.

Indexer Configuration

The indexer connects everything together.

It:

• Reads content from the data source.
• Executes the skillset.
• Maps generated fields into the search index.

Because skillsets execute during indexing, any enriched fields they generate can be mapped directly into the index.

Code-First Approach

A code-first approach is also available.

Azure AI Search provides SDKs for:

• .NET
• Java
• JavaScript
• Python

For this project, I used a .NET 10 backend alongside a React front end to configure and interact with Azure AI Search programmatically.

A snippet of code for the index definition

Scenario 1 in Action

After configuration is complete, we can run the indexer.

The Azure Portal provides a useful search testing interface that allows direct querying of the index and makes it easy to experiment with search behavior and tuning.

When reviewing indexed content, however, we quickly discover a challenge.

For example, the built-in Image Analysis Skill produced the following output for the WALL-E movie poster:

• “imageCaption”: “website”
• Along with tags such as:
• “text”,
• “screenshot”,
• “cartoon”,
• “poster”,
• “digital compositing”,
• “pc game”,
• “LEGO”

Clearly, this metadata does not accurately represent the content of the poster.

As a result, semantic search quality suffers. Searching for a phrase like “robot in space” becomes ineffective if the vectorized description simply says “website.”

This is where Custom Skills become valuable.

Azure Portal indexer page

Now that the index is populated we can go ahead and test our search directly in the Azure portal.

Azure Portal index search

This page is very helpful as you can directly search on your index and play around with queries to help tune your search. After taking a look at the returned results, we can see the built in image analysis skill is not giving us exactly what we want. For example, here is a search result for the movie “WALL-E”. Here is the poster.

WALL-E Movie Poster

Here is the JSON:

As you can see, it captioned the image as “website” and the tags aren’t exactly accurate either. With this bad data, it makes the search ineffective. If we search with a poster description, we will not get a good result if the vectorized caption we search on is just “website”. So, what can we do? Well, this is where Custom Skills come into the picture.

Custom Skills

Azure AI Search allows developers to create custom enrichment skills using the WebApiSkill.

These skills execute outside of Azure AI Search but integrate seamlessly into the indexing pipeline.

For this example:

• A GPT-4o model was deployed in Azure Foundry.
• An Azure Function App was created the ingests the poster images.
• The function app passes the images to the GPT-4o model, which generates detailed poster descriptions.
• The detailed poster description is then passed to the Azure OpenAI Embedding Skill which vectorizes the generated descriptions.

The indexed Wall-E entry now looks like this.

The enriched WALL-E poster entry now contains a rich semantic description that accurately captures the image content.

Rather than simply indexing “website,” the model describes:

• The robot WALL-E
• The futuristic setting
• Earth in the background
• Space-themed visual elements
• The movie branding and tagline

This richer content dramatically improves vector search quality.

A query such as:

“Robot in space”

now correctly returns WALL-E as the top result.

Search Results

Scenario 2

The second scenario follows a similar architecture.

Instead of movie posters, we index JSON files containing:

• Movie titles
• Actors
• Genres
• Summaries
• Review summaries

During indexing, Azure OpenAI Embedding Skills generate vectors from summaries and review content.

This enables semantic searches such as:

“Sci-fi movie with existential themes”

or

“A robot goes on an adventure”

Rather than relying on exact keywords, Azure AI Search identifies movies that are conceptually related to the query.

Sci-fi search results

We could also use WALL-E again as an example for a search. By passing something like “A robot goes on an adventure” to our search.

Robot on an adventure search results

As expected, WALL-E ranks highly for the second example.

Under the Hood

Now that we have a good walkthrough of two search scenarios, let’s take a look at how all this works under the hood.

Vectorization

Vectorization converts content into numeric representations that capture semantic meaning.

When a query is submitted, Azure AI Search compares the query vector against indexed vectors and identifies the most similar matches.

The quality of these embeddings is one of the primary factors influencing semantic search accuracy.

Here is the Microsoft Learn diagram for how vector search works.

Search Algorithms

Azure AI Search supports two vector search algorithms:

• Exhaustive K-Nearest Neighbors (KNN)
• Hierarchical Navigable Small World (HNSW)

For most production workloads, HNSW is the preferred option due to its balance of speed and accuracy.

HNSW

HNSW builds a multi-layer graph structure during indexing.

Each indexed item becomes a node connected to neighboring nodes based on similarity.

Higher layers provide broad navigation across the dataset, while lower layers provide increasingly granular relationships.

During query execution:

• Search begins at the highest layer.
• The graph is traversed toward increasingly similar nodes.
• Less relevant paths are pruned.
• The nearest neighbors are returned.

This approach allows Azure AI Search to efficiently retrieve highly relevant results without exhaustively comparing every vector.

HNSW Graph

At query time, the search navigates this constructed graph and enters through the top level. This top level contains the set of vectors that serves as a starting point for the search. The search then traverses the graph level by level from top to bottom. It selects the node to navigate to by choosing the one that is closer to the query vector based on the similarity metric. As it traverses, the algorithm prunes the search space by only considering nodes that are likely to contain the nearest neighbor. As the algorithm moves to lower levels the algorithm considers more neighbors near the query. The search completes when the desired number of nearest neighbors are identified (this is configured in the search). So, if we were to visualize our “A robot goes on an adventure search” we would get the following:

HNSW Graph search visualization

We can see here all the nodes that get hit during the search in green and the location of our vectorized search query as the diamond. The closest nodes to our query are what are returned. These results have the highest similarity score and best search result.

Search Types, Scoring, and Tuning

Azure AI Search supports four primary search approaches:

• Full-text search
• Vector search
• Hybrid search
• Multimodal search

For this blog, we focus primarily on vector and hybrid search.

HNSW Parameters

Several HNSW parameters influence performance and quality:

metric

Defines similarity measurement:

• cosine
• dotProduct
• euclidean
• hamming

For Azure OpenAI embeddings, cosine similarity is typically the correct choice.

m

Controls the number of connections created for each node.

Higher values generally improve recall but increase memory usage and indexing time.

efConstruction

Controls the number of nearest neighbors evaluated during index creation.

Higher values improve index quality but increase indexing time.

efSearch

Controls the number of nearest neighbors evaluated during search.

Higher values improve retrieval quality at the cost of query latency.

Reciprocal Rank Fusion (RRF)

When multiple retrieval methods are used simultaneously, Azure AI Search applies Reciprocal Rank Fusion (RRF).

Examples include:

• Multiple vector searches
• Hybrid search combining vector and full-text retrieval

RRF combines rankings from each retrieval method and promotes results that perform well across multiple searches.

This allows organizations to blend traditional keyword relevance with semantic understanding.

Vector Weighting

Vector weights can be applied to influence search results.

Increasing the weight of a vector field boosts its contribution to the final ranking score during RRF.

This becomes especially useful when balancing multiple vectors or tuning hybrid search experiences.

We can use this to tune our searches. One way we can do this is by assigning a weight to a vector during search. Essentially, this is a multiplier that will be applied to the search score. By doing this, it increases or decreases the score that the vector is given during RRF.

As an example, let’s take a look at the search scores for our “A robot goes on an adventure” search again. Here we can see the score for each vector and the overall score for WALL-E with no weights applied.

Now let’s see it with a weight of 2 applied to the summaryVector. We can see the vector search scores are the same. The weight has been applied and used during RRF which can be seen by the increased overall search score.

For the full-text searches, the score is computed using the BM25 relevance scoring algorithm. We can adjust the results of this via search profiles.

Search Profiles

Search profiles control how full-text search results are scored.

They can:

• Assign weights to fields
• Apply freshness boosts
• Use distance-based scoring
• Incorporate magnitude and tag functions

This provides additional flexibility for fine-tuning relevance beyond vector search.

Similar to vectors we can add a weight to any of the fields which will multiply its search score by the specified weight. You can also specify additional functions. There are four types of functions that can be used in scoring: magnitude, freshness, distance, and tags. The function aggregation specifies how the scores from multiple functions are combined.

You can select the scoring profile to use at query time by specifying it in the query. You can also set a scoring profile as a default that will be applied to all queries (unless a different profile is set in the query).

Considerations During Hybrid Search

One important consideration when combining full-text and vector search is score balancing.

Full-text search scores are effectively unbounded, while vector similarity scores operate within a much narrower range.

Because of this, a seemingly insignificant keyword match can sometimes outweigh a stronger semantic match.

Carefully tuning vector weights and search profiles helps prevent keyword relevance from overpowering semantic relevance when hybrid search is used.

Search with star wars at top

Star wars is now the top result. While two robots are included in that adventure, WALL-E still matches the query more closely. Why did this change happen? Let’s take a look at the search scores.

Star Wars Result:

WALL-E Result:

As you can see, because our search includes the word “A”, which is present in the title of Star Wars but not WALL-E, it ranked higher. While “A” doesn’t result in a huge search score it is enough to put Star Wars over the top of WALL-E. Let’s try again with a weight of 20 applied to the summary vector. We can now see WALL-E back to the top result. You could also use a search profile instead and apply a weight to the title that reduces its search score.

Wall-E At the top of search results

Conclusion

AI is fundamentally changing how organizations approach search. Azure AI Search provides powerful capabilities that allow teams to enrich content, generate semantic representations, and deliver significantly more relevant search experiences across both structured and unstructured data.

Whether you’re building customer-facing search experiences, enterprise knowledge discovery solutions, content retrieval systems, or AI-powered applications, vector search and AI enrichment create opportunities to move beyond keyword matching and toward intent-driven discovery.

At RBA, we help organizations design and implement modern search and AI experiences that align with business goals, improve content discoverability, and create stronger digital experiences. As AI-powered search continues to evolve, establishing the right architecture, enrichment strategy, and governance model will be critical to realizing long-term value from enterprise content investments.