As organizations increasingly adopt generative AI technologies, streamlining and securing AI workflows becomes essential. Snowflake Cortex Search offers a powerful, AI-driven solution for exploring vast amounts of structured and unstructured data directly within the Snowflake environment. To demonstrate how Cortex Search can transform search capabilities with AI, I used a simple library books search system with unstructured data as an example, while highlighting its broader advantages for enterprises seeking to harness AI with strong data governance.

At its core, Cortex Search utilizes vector search, powered by Snowflake’s high-performance and cost-effective Arctic Embed M model. This vector search is enhanced with a hybrid approach that combines lexical search and semantic reranking to optimize retrieval and ranking of results.

This ensemble method leverages:

1. Vector search for semantic similarity : Unlike conventional methods that rely on the exact match of keywords within documents, vector search understands the semantic meaning of queries, retrieving results based on the concepts behind the words.
2. Keyword search for lexical matching : Keyword search focuses on exact word matches, ensuring that specific terms from the query appear in the results. This method is particularly useful for retrieving documents with precise language or terminology.
3. Semantic reranking to surface the most relevant results : Once the initial results are retrieved, semantic reranking reorders them based on their relevance to the query’s intent. This ensures that the most contextually appropriate results appear at the top of the list.

By using this multi-pronged strategy, Cortex Search can handle a wide variety of queries effectively without extensive tuning.

Let's jump into the hands-on section.

Overview:

For this example, we’ll use an unstructured book catalog. We’ll process data from PDFs, Excel files, and CSVs, and create a searchable index within Snowflake. Below is a high-level breakdown of the process:

The table comprises pivotal columns:

1. Data Extraction and Chunking using a Python UDF and LangChain.
2. Raw Data Storage in Snowflake.
3. Cortex Search Service Creation to index and handle search queries.
4. Result Interface with Streamlit in Snowflake, allowing users to interact with the system through a Generative AI-powered interface.

Let’s dive into each of these steps in detail.

Data Extraction and Chunking with Python UDF and LangChain

Create a Python UDF called books_chunk that uses the LangChain library to extract and process text from PDFs, Excel files, and CSVs. The function splits the extracted text into chunks optimized for semantic search.

Here’s how the UDF is structured:

                  
                    CREATE OR REPLACE FUNCTION cortex_search_db.cortex_search_schema.books_chunk(
                      description string, title string, authors string, category string, publisher string
                  )
                  RETURNS TABLE (chunk string, title string, authors string, category string, publisher string)
                  LANGUAGE PYTHON
                  RUNTIME_VERSION = '3.9'
                  HANDLER = 'text_chunker'
                  PACKAGES = ('snowflake-snowpark-python', 'langchain')
                  AS
                  $$
                  from langchain.text_splitter import RecursiveCharacterTextSplitter
                  import copy
                  from typing import Optional
                  
                  class text_chunker:
                      def process(self, description: Optional[str], title: str, authors: str, category: str, publisher: str):
                          if description == None:
                              description = ""  
                              
                          text_splitter = RecursiveCharacterTextSplitter(
                              chunk_size = 2000,
                              chunk_overlap = 300,
                              length_function = len
                          )
                          chunks = text_splitter.split_text(description)
                          for chunk in chunks:
                              yield (title + "\n" + authors + "\n" + chunk, title, authors, category, publisher)
                  $$;
                    
                  

This function ensures that the data is split into smaller, manageable chunks that are compatible with Snowflake Cortex Search’s vector embeddings, which have token limitations.

Raw Data Storage in Snowflake

Store the processed chunks in a raw data table, which will serve as the foundation for our Cortex Search service.

                    
                      CREATE OR REPLACE TABLE cortex_search_db.cortex_search_schema.book_description_chunks AS (
                        SELECT
                            books.*,
                            t.CHUNK as CHUNK
                        FROM cortex_search_db.cortex_search_schema.books_dataset_raw books,
                             TABLE(cortex_search_db.cortex_search_schema.books_chunk(
                                 books.description, books.title, books.authors, books.category, books.publisher)) t
                    );
                    
                      
                    

Creating the Cortex Search Service

Once the data is prepared and stored, we create a Cortex Search Service. This service will allow us to index the book descriptions, making them searchable via Snowflake’s hybrid search mechanism

                  
                    CREATE OR REPLACE CORTEX SEARCH SERVICE CORTEX_SEARCH_DB.CORTEX_SEARCH_SCHEMA.BOOKS_DATASET_SERVICE
                    ON DESCRIPTION
                    ATTRIBUTES CATEGORY, PUBLISHER, AUTHORS, TITLE
                    WAREHOUSE = CORTEX_SEARCH_WH
                    TARGET_LAG = '1 hour'
                     AS (
                          SELECT *
                          FROM cortex_search_db.cortex_search_schema.book_description_chunks
                     );
                    
                  

This SQL statement sets up the Cortex Search service, specifying the search column DESCRIPTION and filter columns CATEGORY, PUBLISHER, AUTHORS and TITLE to search and index. The TARGET_LAG ensures the service refreshes approximately every hour.

Streamlit Query Interface

For user interaction, built a Streamlit application that acts as a Generative AI-powered search interface. Users can input search queries in natural language, and the app returns relevant results by querying the Cortex Search service.

Here's an example of how the search query works within the app:

                  
                    def run_cortex_search(session, database_name, schema_name, service_name, query, columns, filter_condition, limit):
                      search_query = f"""
                      SELECT PARSE_JSON(
                          SNOWFLAKE.CORTEX.SEARCH_PREVIEW(
                              '{database_name}.{schema_name}.{service_name}',
                              '{{ 
                                "query": "{query}",
                                "columns": {columns},
                                "filter": {filter_condition},
                                "limit": {limit}
                              }}'
                          )
                      )['results'] as results;
                      """
                      
                      result_df = session.sql(search_query).to_pandas()
                      return result_df
                    
                  

The function queries the Cortex Search Service using the SEARCH_PREVIEW function and processes the results for display in the Streamlit app. Once Cortex Search becomes generally available (GA), additional filtering options may be introduced, and the “preview” designation may be removed.

The Power of Unified Data and AI in Snowflake

By implementing this solution within Snowflake, we’ve created a powerful AI-driven search system for book catalogs while reaping several key benefits:

1. Simplified AI IntegrationCortex Search abstracts away the complexities of managing vector embeddings, search algorithms, and infrastructure. Developers can focus on building applications instead of managing AI infrastructure.
2. Unified Governance and SecurityHaving both AI and data in the same Snowflake environment enables consistent governance and role-based access control (RBAC). This is crucial for enterprises handling sensitive data, as it ensures security and compliance.
3. Real-time Data AccessCortex Search is designed to reflect changes in the underlying data with minimal lag, ensuring users get up-to-date search results. The TARGET_LAG parameter defines the frequency at which the service refreshes its index.
4. Scalability and Cost EfficiencySnowflake’s architecture allows Cortex Search to scale efficiently across large datasets without the need for a separate vector database or search engine. By leveraging Snowflake’s compute resources, enterprises can manage both data and AI under one roof, reducing overhead and costs.

Possible Use Cases

Customer Support Assistants: Help support agents triage tickets efficiently by searching through prior tickets and documentation to find the best solutions for customer issues.

Financial Analysts: Cortex Search enables financial analysts to retrieve and compare earnings reports across companies, helping to identify trends quickly.

Sales Support: Sales teams can leverage AI-powered search to retrieve relevant case studies, pitch materials, and other resources tailored to their customer’s needs.

R&D Knowledge Discovery: Researchers can use Cortex Search to sift through scientific papers, internal reports, and other literature relevant to their ongoing work, speeding up the research process.

Product Search: Cortex Search can be used for e-commerce platforms to implement lexical and semantic search for products, allowing users to search product catalogs based on descriptions, specifications, and reviews.

Documentation Search: Organizations can build intelligent site search capabilities for navigating technical support and product documentation pages, allowing users to find relevant information quickly.

Legal Document Retrieval: Law firms and legal teams can use Cortex Search to explore large volumes of legal texts, contracts, or case law, retrieving key insights and information from complex legal documents.

Conclusion

Snowflake Cortex Search offers a compelling way to integrate AI-driven search capabilities into enterprise applications while maintaining the security, scalability, and governance benefits of the Snowflake platform. By simplifying the integration of advanced NLP models, Cortex Search allows enterprises to focus on deriving value from their data rather than managing AI infrastructure.