gemini/grounding/intro-grounding-gemini.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Intro to Grounding with Gemini in Vertex AI\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/grounding/intro-grounding-gemini.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fgrounding%2Fintro-grounding-gemini.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/grounding/intro-grounding-gemini.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/grounding/intro-grounding-gemini.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://goo.gle/4jeQyFS\">\n",
        "      <img width=\"32px\" src=\"https://cdn.qwiklabs.com/assets/gcp_cloud-e3a77215f0b8bfa9b3f611c0d2208c7e8708ed31.svg\" alt=\"Google Cloud logo\"><br> Open in  Cloud Skills Boost\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/grounding/intro-grounding-gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/grounding/intro-grounding-gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/grounding/intro-grounding-gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/grounding/intro-grounding-gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/grounding/intro-grounding-gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49e1e41cea0d"
      },
      "source": [
        "| Authors |\n",
        "| --- |\n",
        "| [Holt Skinner](https://github.com/holtskinner) |\n",
        "| [Kristopher Overholt](https://github.com/koverholt) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "**YouTube Video: Introduction to grounding with Gemini on Vertex AI**\n",
        "\n",
        "<a href=\"https://www.youtube.com/watch?v=Ph0g6dnsB4g&list=PLIivdWyY5sqJio2yeg1dlfILOUO2FoFRx\" target=\"_blank\">\n",
        "  <img src=\"https://img.youtube.com/vi/Ph0g6dnsB4g/maxresdefault.jpg\" alt=\"Introduction to grounding with Gemini on Vertex AI\" width=\"500\">\n",
        "</a>\n",
        "\n",
        "[Grounding in Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/ground-gemini) lets you use generative text models to generate content grounded in your own documents and data. This capability lets the model access information at runtime that goes beyond its training data. By grounding model responses in Google Search results or data stores within [Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction), LLMs that are grounded in data can produce more accurate, up-to-date, and relevant responses.\n",
        "\n",
        "Grounding provides the following benefits:\n",
        "\n",
        "- Reduces model hallucinations (instances where the model generates content that isn't factual)\n",
        "- Anchors model responses to specific information, documents, and data sources\n",
        "- Enhances the trustworthiness, accuracy, and applicability of the generated content\n",
        "\n",
        "You can configure two different sources of grounding in Vertex AI:\n",
        "\n",
        "1. Google Search results for data that is publicly available and indexed.\n",
        "   - If you use this service in a production application, you will also need to [use a Google Search entry point](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/grounding-search-entry-points).\n",
        "2. [Data stores in Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/create-datastore-ingest), which can include your own data in the form of website data, unstructured data, or structured data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to:\n",
        "\n",
        "- Generate LLM text and chat model responses grounded in Google Search results\n",
        "- Compare the results of ungrounded LLM responses with grounded LLM responses\n",
        "- Create and use a data store in Vertex AI Search to ground responses in custom documents and data\n",
        "- Generate LLM text and chat model responses grounded in Vertex AI Search results\n",
        "\n",
        "This tutorial uses the following Google Cloud AI services and resources:\n",
        "\n",
        "- Vertex AI\n",
        "- Vertex AI Search\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Configuring the LLM and prompt for various examples\n",
        "- Sending example prompts to generative text and chat models in Vertex AI\n",
        "- Setting up a data store in Vertex AI Search with your own data\n",
        "- Sending example prompts with various levels of grounding (no grounding, web grounding, data store grounding)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "1. Enable the [Vertex AI and Vertex AI Search APIs](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,discoveryengine.googleapis.com).\n",
        "1. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "### Install Google Gen AI SDK for Python\n",
        "\n",
        "Install the following packages required to execute this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "2b4ef9b72d43"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "If you are running this notebook on Google Colab, you will need to authenticate your environment. To do this, run the new cell below. This step is not required if you are using Vertex AI Workbench."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "603adbbf0532"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    # Authenticate user to Google Cloud\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "### Set Google Cloud project information and create client\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment).\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)\n",
        "\n",
        "You can also change the `LOCATION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\"}\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
        "\n",
        "from google import genai\n",
        "\n",
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "from IPython.display import Markdown, display\n",
        "from google.genai.types import (\n",
        "    GenerateContentConfig,\n",
        "    GenerateContentResponse,\n",
        "    GoogleSearch,\n",
        "    Part,\n",
        "    Retrieval,\n",
        "    Tool,\n",
        "    VertexAISearch,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4e569c5d4a49"
      },
      "source": [
        "### Helper functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "307f36dbd36c"
      },
      "outputs": [],
      "source": [
        "def print_grounding_data(response: GenerateContentResponse) -> None:\n",
        "    \"\"\"Prints Gemini response with grounding citations in Markdown format.\"\"\"\n",
        "    if not (response.candidates and response.candidates[0].grounding_metadata):\n",
        "        print(\"Response does not contain grounding metadata.\")\n",
        "        display(Markdown(response.text))\n",
        "        return\n",
        "\n",
        "    grounding_metadata = response.candidates[0].grounding_metadata\n",
        "    markdown_parts = []\n",
        "\n",
        "    # Citation indexes are in bytes\n",
        "    ENCODING = \"utf-8\"\n",
        "    text_bytes = response.text.encode(ENCODING)\n",
        "    last_byte_index = 0\n",
        "\n",
        "    for support in grounding_metadata.grounding_supports:\n",
        "        markdown_parts.append(\n",
        "            text_bytes[last_byte_index : support.segment.end_index].decode(ENCODING)\n",
        "        )\n",
        "\n",
        "        # Generate and append citation footnotes (e.g., \"[1][2]\")\n",
        "        footnotes = \"\".join([f\"[{i + 1}]\" for i in support.grounding_chunk_indices])\n",
        "        markdown_parts.append(f\" {footnotes}\")\n",
        "\n",
        "        # Update index for the next segment\n",
        "        last_byte_index = support.segment.end_index\n",
        "\n",
        "    # Append any remaining text after the last citation\n",
        "    if last_byte_index < len(text_bytes):\n",
        "        markdown_parts.append(text_bytes[last_byte_index:].decode(ENCODING))\n",
        "\n",
        "    markdown_parts.append(\"\\n\\n----\\n## Grounding Sources\\n\")\n",
        "\n",
        "    # Build Grounding Sources Section\n",
        "    markdown_parts.append(\"### Grounding Chunks\\n\")\n",
        "    for i, chunk in enumerate(grounding_metadata.grounding_chunks, start=1):\n",
        "        context = chunk.web or chunk.retrieved_context\n",
        "        if not context:\n",
        "            print(f\"Skipping Grounding Chunk without context: {chunk}\")\n",
        "            continue\n",
        "\n",
        "        uri = context.uri\n",
        "        title = context.title or \"Source\"\n",
        "\n",
        "        # Convert GCS URIs to public HTTPS URLs\n",
        "        if uri and uri.startswith(\"gs://\"):\n",
        "            uri = uri.replace(\"gs://\", \"https://storage.googleapis.com/\", 1).replace(\n",
        "                \" \", \"%20\"\n",
        "            )\n",
        "        markdown_parts.append(f\"{i}. [{title}]({uri})\\n\")\n",
        "\n",
        "    # Add Search/Retrieval Queries\n",
        "    if grounding_metadata.web_search_queries:\n",
        "        markdown_parts.append(\n",
        "            f\"\\n**Web Search Queries:** {grounding_metadata.web_search_queries}\\n\"\n",
        "        )\n",
        "        if grounding_metadata.search_entry_point:\n",
        "            markdown_parts.append(\n",
        "                f\"\\n**Search Entry Point:**\\n{grounding_metadata.search_entry_point.rendered_content}\\n\"\n",
        "            )\n",
        "    elif grounding_metadata.retrieval_queries:\n",
        "        markdown_parts.append(\n",
        "            f\"\\n**Retrieval Queries:** {grounding_metadata.retrieval_queries}\\n\"\n",
        "        )\n",
        "\n",
        "    display(Markdown(\"\".join(markdown_parts)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "55cf2dd17690"
      },
      "source": [
        "Initialize the Gemini model from Vertex AI:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "652a8969dd5a"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.0-flash\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e336da7161af"
      },
      "source": [
        "## Example: Grounding with Google Search results\n",
        "\n",
        "In this example, you'll compare LLM responses with no grounding with responses that are grounded in the results of a Google Search. You'll ask a question about a the most recent solar eclipse."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "6a28ca4abb52"
      },
      "outputs": [],
      "source": [
        "PROMPT = \"When is the next solar eclipse in the US?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "25955ce5d263"
      },
      "source": [
        "### Text generation without grounding\n",
        "\n",
        "Make a prediction request to the LLM with no grounding:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "a2e348ff93e6"
      },
      "outputs": [],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=PROMPT,\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5d7cb7cceb99"
      },
      "source": [
        "### Text generation grounded in Google Search results\n",
        "\n",
        "You can add the `tools` keyword argument with a `Tool` including `GoogleSearch` to instruct Gemini to first perform a Google Search with the prompt, then construct an answer based on the web search results.\n",
        "\n",
        "The search queries and [Search Entry Point](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/grounding-search-entry-points) are available for each `Candidate` in the response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "1d9fb83b0ab9"
      },
      "outputs": [],
      "source": [
        "google_search_tool = Tool(google_search=GoogleSearch())\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=PROMPT,\n",
        "    config=GenerateContentConfig(tools=[google_search_tool]),\n",
        ")\n",
        "\n",
        "print_grounding_data(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6d3920bb2ac0"
      },
      "source": [
        "Note that the response without grounding only has limited information from the LLM about solar eclipses. Whereas the response that was grounded in web search results contains the most up to date information from web search results that are returned as part of the LLM with grounding request."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "59c98ab0f5fb"
      },
      "source": [
        "### Text generation with multimodal input grounded in Google Search results\n",
        "\n",
        "Gemini can also generate grounded responses with multimodal input. Let's try with this image of the Eiffel Tower.\n",
        "\n",
        "![Paris](https://storage.googleapis.com/github-repo/generative-ai/gemini/grounding/paris.jpg)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5ebdda19afad"
      },
      "outputs": [],
      "source": [
        "PROMPT = \"What is the current temperature at this location?\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=[\n",
        "        Part.from_uri(\n",
        "            file_uri=\"gs://github-repo/generative-ai/gemini/grounding/paris.jpg\",\n",
        "            mime_type=\"image/jpeg\",\n",
        "        ),\n",
        "        PROMPT,\n",
        "    ],\n",
        "    config=GenerateContentConfig(\n",
        "        tools=[google_search_tool],\n",
        "    ),\n",
        ")\n",
        "\n",
        "print_grounding_data(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a29c93ef3f34"
      },
      "source": [
        "## Example: Grounding with Enterprise Web Search\n",
        "\n",
        "Grounding with Google Search uses Google Search to perform searches across the web. As part of this offering, Google Search might perform logging of customer queries (see [section 19.k of Google Cloud Service Specific Terms](https://cloud.google.com/terms/service-terms)). This often doesn't meet the compliance requirements of customers in highly regulated industries like Finance or Healthcare.\n",
        "\n",
        "Enterprise Web Search meets these requirements. When a customer uses Enterprise Web Search to ground on the web, this is done without logging of customer data and with full support for VPC SC and ML processing in-region. Enterprise Web Search Grounding is available in an US and EU multi-region.\n",
        "\n",
        "Request and response format for Enterprise Web Search Grounding are very similar to Grounding with Google Search.\n",
        "\n",
        "### Gemini model compatibility\n",
        "\n",
        "Enterprise Web Search is compatible with all Gemini models. Gemini 2.0 Flash supports multimodal input (e.g. images, documents, videos). "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "eb0d3334a18f"
      },
      "outputs": [],
      "source": [
        "%%bash -s \"$PROJECT_ID\" \"$MODEL_ID\"\n",
        "PROJECT_ID=$1\n",
        "MODEL_ID=$2\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  -H \"x-server-timeout: 60\" \\\n",
        "  https://us-central1-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/us-central1/publishers/google/models/$MODEL_ID:generateContent \\\n",
        "  -d '{\n",
        "    \"contents\": [{\n",
        "      \"role\": \"user\",\n",
        "      \"parts\": [{\n",
        "        \"text\": \"Who won the 2024 UEFA European Championship?\"\n",
        "      }]\n",
        "    }],\n",
        "    \"tools\": [{\n",
        "      \"enterpriseWebSearch\": {}\n",
        "    }]\n",
        "  }'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6708e03a1d7b"
      },
      "source": [
        "## Example: Grounding with Google Maps\n",
        "\n",
        "You can also use Google Maps data for grounding with Gemini. See the [announcement blog](https://blog.google/products/earth/grounding-google-maps-generative-ai/) for more information.\n",
        "\n",
        "**NOTE:** This feature is allowlist-only, refer to the [documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/grounding/grounding-with-google-maps) for how to request access."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0a9ce21ca573"
      },
      "source": [
        "First, you will need to create a [Google Maps API Key](https://cloud.google.com/vertex-ai/generative-ai/docs/grounding/grounding-with-google-maps#access-to-google-maps)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "53102c85aa24"
      },
      "outputs": [],
      "source": [
        "GOOGLE_MAPS_API_KEY = \"[your-google-maps-api-key]\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ebfa54b6b14e"
      },
      "outputs": [],
      "source": [
        "%%bash -s \"$PROJECT_ID\" \"$MODEL_ID\" \"$GOOGLE_MAPS_API_KEY\"\n",
        "PROJECT_ID=$1\n",
        "MODEL_ID=$2\n",
        "GOOGLE_MAPS_API_KEY=$3\n",
        "\n",
        "curl -X POST \\\n",
        "  -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
        "  -H \"Content-Type: application/json\" \\\n",
        "  -H \"x-server-timeout: 60\" \\\n",
        "  https://us-central1-aiplatform.googleapis.com/v1beta1/projects/$PROJECT_ID/locations/us-central1/publishers/google/models/$MODEL_ID:generateContent \\\n",
        "  -d '{\n",
        "    \"contents\": [{\n",
        "      \"role\": \"user\",\n",
        "      \"parts\": [{\n",
        "        \"text\": \"Recommend some good vegetarian food in Las Vegas.\"\n",
        "      }]\n",
        "    }],\n",
        "    \"systemInstruction\": {\n",
        "        \"parts\": {\n",
        "            \"text\": \"You are a helpful assistant that provides information about locations. You have access to map data and can answer questions about distances, directions, and points of interest.\"\n",
        "        }\n",
        "    },\n",
        "    \"tools\": [\n",
        "        {\n",
        "            \"googleMaps\": {\n",
        "                \"authConfig\": {\n",
        "                    \"apiKeyConfig\": {\n",
        "                        \"name\": \"api_key\",\n",
        "                        \"apiKeyString\": \"$GOOGLE_MAPS_API_KEY\"\n",
        "                    }\n",
        "                }\n",
        "            }\n",
        "        }\n",
        "    ],\n",
        "  }'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77f0800f8762"
      },
      "source": [
        "## Example: Grounding with custom documents and data\n",
        "\n",
        "In this example, you'll compare LLM responses with no grounding with responses that are grounded in the [results of a search app in Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/create-datastore-ingest).\n",
        "\n",
        "The data store will contain internal documents from a fictional bank, Cymbal Bank. These documents aren't available on the public internet, so the Gemini model won't have any information about them by default."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1b308548c68b"
      },
      "source": [
        "### Creating a data store in Vertex AI Search\n",
        "\n",
        "In this example, you'll use a Google Cloud Storage bucket with a few sample internal documents for our bank. There's some docs about booking business travel, strategic plan for this Fiscal Year and HR docs describing the different jobs available in the company.\n",
        "\n",
        "Follow the tutorial steps in the Vertex AI Search documentation to:\n",
        "\n",
        "1. [Create a data store with unstructured data](https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#unstructured-data) that loads in documents from the GCS folder `gs://cloud-samples-data/gen-app-builder/search/cymbal-bank-employee`.\n",
        "2. [Create a search app](https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#create_a_search_app) that is attached to that data store. You should also enable the **Enterprise edition features** so that you can search indexed records within the data store.\n",
        "\n",
        "**Note:** The data store must be in the same project that you are using for Gemini.\n",
        "\n",
        "You can also follow this notebook to do it with code. [Create a Vertex AI Search Datastore and App](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/search/create_datastore_and_search.ipynb)\n",
        "\n",
        "Once you've created a data store, obtain the App ID and input it below.\n",
        "\n",
        "Note: You will need to wait for data ingestion to finish before using a data store with grounding. For more information, see [create a data store](https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-es)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "fcd767476241"
      },
      "outputs": [],
      "source": [
        "VERTEX_AI_SEARCH_PROJECT_ID = PROJECT_ID  # @param {type: \"string\"}\n",
        "VERTEX_AI_SEARCH_REGION = \"global\"  # @param {type: \"string\"}\n",
        "# Replace this with your App (Engine) ID from Vertex AI Search\n",
        "VERTEX_AI_SEARCH_APP_ID = \"cymbal-bank-onboarding\"  # @param {type: \"string\"}\n",
        "\n",
        "VERTEX_AI_SEARCH_ENGINE_NAME = f\"projects/{VERTEX_AI_SEARCH_PROJECT_ID}/locations/{VERTEX_AI_SEARCH_REGION}/collections/default_collection/engines/{VERTEX_AI_SEARCH_APP_ID}\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ccc156676e0a"
      },
      "source": [
        "Now you can ask a question about the company culture:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9c1e1b1743bd"
      },
      "outputs": [],
      "source": [
        "PROMPT = \"What is the company culture like?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f365681544bb"
      },
      "source": [
        "### Text generation without grounding\n",
        "\n",
        "Make a prediction request to the LLM with no grounding:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "299818ae71e9"
      },
      "outputs": [],
      "source": [
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=PROMPT,\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "073f2ec42ff6"
      },
      "source": [
        "### Text generation grounded in Vertex AI Search results\n",
        "\n",
        "Now we can add the `tools` keyword arg with a grounding tool of `grounding.VertexAISearch()` to instruct the LLM to first perform a search within your search app, then construct an answer based on the relevant documents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d4c5d53a37b4"
      },
      "outputs": [],
      "source": [
        "vertex_ai_search_tool = Tool(\n",
        "    retrieval=Retrieval(\n",
        "        vertex_ai_search=VertexAISearch(engine=VERTEX_AI_SEARCH_ENGINE_NAME)\n",
        "    )\n",
        ")\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=\"What is the company culture like?\",\n",
        "    config=GenerateContentConfig(tools=[vertex_ai_search_tool]),\n",
        ")\n",
        "\n",
        "print_grounding_data(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e3f985c704cd"
      },
      "source": [
        "Note that the response without grounding doesn't have any context about what company we are asking about. Whereas the response that was grounded in Vertex AI Search results contains information from the documents provided, along with citations of the information.\n",
        "\n",
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ Important notes:</b><br>\n",
        "<br>\n",
        "<b>If you get an error when running the previous cell:</b><br>\n",
        "&nbsp;&nbsp;&nbsp;&nbsp;In order for this sample notebook to work with data store in Vertex AI Search,<br>\n",
        "&nbsp;&nbsp;&nbsp;&nbsp;you'll need to create a <a href=\"https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#create_a_data_store\">data store</a> <b>and</b> a <a href=\"https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search#create_a_search_app\">search app</a> associated with it in Vertex AI Search.<br>\n",
        "&nbsp;&nbsp;&nbsp;&nbsp;If you only create a data store, the previous request will return errors when making queries against the data store.\n",
        "<br><br>\n",
        "<b>If you get an empty response when running the previous cell:</b><br>\n",
        "&nbsp;&nbsp;&nbsp;&nbsp;You will need to wait for data ingestion to finish before using a data store with grounding.<br>\n",
        "&nbsp;&nbsp;&nbsp;&nbsp;For more information, see <a href=\"https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-es\">create a data store</a>.\n",
        "</div>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "54562717e2a4"
      },
      "source": [
        "## Example: Grounded chat responses\n",
        "\n",
        "You can also use grounding when using chat conversations in Vertex AI. In this example, you'll compare LLM responses with no grounding with responses that are grounded in the results of a Google Search and a data store in Vertex AI Search."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "490cf1ed3399"
      },
      "outputs": [],
      "source": [
        "PROMPT = \"What are managed datasets in Vertex AI?\"\n",
        "PROMPT_FOLLOWUP = \"What types of data can I use?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b59783e4f1ce"
      },
      "source": [
        "### Chat session grounded in Google Search results\n",
        "\n",
        "Now you can add the `tools` keyword arg with a Tool of `GoogleSearch` to instruct the chat model to first perform a Google Search with the prompt, then construct an answer based on the web search results:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "58edb2bd860f"
      },
      "outputs": [],
      "source": [
        "chat = client.chats.create(\n",
        "    model=MODEL_ID,\n",
        "    config=GenerateContentConfig(tools=[Tool(google_search=GoogleSearch())]),\n",
        ")\n",
        "\n",
        "display(Markdown(\"## Prompt\"))\n",
        "display(Markdown(f\"> {PROMPT}\"))\n",
        "response = chat.send_message(PROMPT)\n",
        "print_grounding_data(response)\n",
        "\n",
        "display(Markdown(\"---\\n\"))\n",
        "\n",
        "display(Markdown(\"## Follow-up Prompt\"))\n",
        "display(Markdown(f\"> {PROMPT_FOLLOWUP}\"))\n",
        "response = chat.send_message(PROMPT_FOLLOWUP)\n",
        "print_grounding_data(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "87be7f661f14"
      },
      "source": [
        "### Chat session grounded in Vertex AI Search results\n",
        "\n",
        "Now we can add the `tools` keyword arg with a grounding tool of `VertexAISearch` to instruct the chat session to first perform a search within your custom search app, then construct an answer based on the relevant documents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8fdad0c3f1f3"
      },
      "outputs": [],
      "source": [
        "PROMPT = \"How do I book business travel?\"\n",
        "PROMPT_FOLLOWUP = \"Give me more details.\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1a824202a8f0"
      },
      "outputs": [],
      "source": [
        "chat = client.chats.create(\n",
        "    model=MODEL_ID,\n",
        "    config=GenerateContentConfig(\n",
        "        tools=[\n",
        "            Tool(\n",
        "                retrieval=Retrieval(\n",
        "                    vertex_ai_search=VertexAISearch(engine=VERTEX_AI_SEARCH_ENGINE_NAME)\n",
        "                )\n",
        "            )\n",
        "        ]\n",
        "    ),\n",
        ")\n",
        "\n",
        "display(Markdown(\"## Prompt\"))\n",
        "display(Markdown(f\"> {PROMPT}\"))\n",
        "response = chat.send_message(PROMPT)\n",
        "print_grounding_data(response)\n",
        "\n",
        "display(Markdown(\"---\\n\"))\n",
        "\n",
        "display(Markdown(\"## Follow-up Prompt\"))\n",
        "display(Markdown(f\"> {PROMPT_FOLLOWUP}\"))\n",
        "response = chat.send_message(PROMPT_FOLLOWUP)\n",
        "print_grounding_data(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To avoid incurring charges to your Google Cloud account for the resources used in this notebook, follow these steps:\n",
        "\n",
        "1. To avoid unnecessary Google Cloud charges, use the [Google Cloud console](https://console.cloud.google.com/) to delete your project if you do not need it. Learn more in the Google Cloud documentation for [managing and deleting your project](https://cloud.google.com/resource-manager/docs/creating-managing-projects).\n",
        "1. If you used an existing Google Cloud project, delete the resources you created to avoid incurring charges to your account. For more information, refer to the documentation to [Delete data from a data store in Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/delete-datastores), then delete your data store.\n",
        "2. Disable the [Vertex AI Search API](https://console.cloud.google.com/apis/api/discoveryengine.googleapis.com) and [Vertex AI API](https://console.cloud.google.com/apis/api/aiplatform.googleapis.com) in the Google Cloud Console."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "intro-grounding-gemini.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}