Build gen AI features powered by your data with Firebase Genkit

1. Before you begin

In this codelab, you will learn to use Firebase Genkit for integrating generative AI into your app. Firebase Genkit is an open source framework that helps you build, deploy, and monitor production-ready AI-powered apps.

Genkit is designed for app developers, to help you easily integrate powerful AI capabilities into your apps with familiar patterns and paradigms. It's built by the Firebase team, leveraging our experience in building tools used by millions of developers worldwide.

Prerequisites

• Familiarity with Firestore, Node.js, and TypeScript.

Key takeaways

• You will learn how to build smarter apps with Firestore's advanced vector similarity search capabilities.
• Learn how to practically implement generative AI in your apps using Genkit.
• Get your solution ready for deployment and integration.

What you'll need

• A browser of your choice, such as Google Chrome
• A development environment with a code editor and terminal
• A Google Account for the creation and management of your Firebase project

2. Review the web app and cloud services used

In this section, you'll review the web app that you'll build with this codelab, as well learn about the cloud services that you'll use.

Web app

In this codelab, you'll work in the codebase of an app called Compass - a vacation planning app. Users can pick a destination, look through activities at the destination, and create an itinerary for their trip.

In this codelab, you will implement two new features that are intended to improve user engagement with the app's homepage. Both ideas are powered by generative AI:

• The app currently displays a static list of destinations. You'll change it to be dynamic!
• You'll implement an auto-populated itinerary to hopefully increase stickiness.

Services used

In this codelab, you'll use many Firebase and Cloud services and features, and most of the starter code for them is provided for you. The following table contains the services that you'll use and the reasons for using them.

3. Set up your development environment

Verify your Node.js version

1. In your terminal, verify that you have Node.js version 20.0.0 or higher installed:

   node -v
   

2. If you don't have Node.js version 20.0.0 or higher, download the latest LTS version and install it.

Get the source code for the codelab

If you have a GitHub account:

1. Create a new repository using our template from github.com/FirebaseExtended/codelab-ai-genkit-rag
2. Create a local clone of the codelab's GitHub repository you just created:

   git clone https://github.com/<your-github-handle>/codelab-ai-genkit-rag
   

If you don't have git installed or prefer to not create a new repo:

Download the GitHub repository as a zip file.

Review the folder structure

On your local machine, find the cloned repository and review the folder structure:

The root folder includes a README.md file that offers a quick start to run the web app, using streamlined instructions. However, if you're a first-time learner, you should complete this codelab (instead of the quickstart) because the codelab contains the most comprehensive set of instructions.

If you're unsure of whether you correctly applied code as instructed throughout this codelab, you can find the solution code in the end git branch.

Install the Firebase CLI

1. Verify that you have the Firebase CLI installed and that it's version 13.6 or higher:

   firebase --version
   

2. If you have the Firebase CLI installed, but it's not version 13.6 or higher, update it:

   npm update -g firebase-tools
   

3. If you don't have the Firebase CLI installed, install it:

   npm install -g firebase-tools
   

If you're unable to update or install the Firebase CLI because of permission errors, see the npm documentation or use another installation option.

Log into Firebase

1. In your terminal, log in to Firebase:

   firebase login
   

   If your terminal says that you're already logged in to Firebase, you can skip to the Set up your Firebase project section of this codelab.
2. In your terminal, depending on whether you want Firebase to collect data, enter Y or N. (either option works for this codelab)
3. In your browser, select your Google Account and click Allow.

Install Google Cloud's gcloud CLI

1. Install the gcloud CLI.
2. In your terminal, log into Google Cloud:

   gcloud auth login
   

4. Set up your Firebase project

In this section, you'll set up a Firebase project and register a Firebase Web App in it. You'll also enable a few services that are used by the sample web app later in this codelab.

All of the steps in this section are performed in the Firebase console.

Create a Firebase project

1. Sign into the Firebase console using the same Google Account you used in the previous step.
2. Click Create a project, and then enter a project name (for example, Compass Codelab).
   Remember the auto-assigned project ID for your Firebase project (or click the Edit icon to set your preferred project ID). You'll need this ID later to identify your Firebase project in the Firebase CLI. If you forget your ID, you can always find it later in the Project Settings.
3. Click Continue.
4. If prompted, review and accept the Firebase terms, and then click Continue.
5. For this codelab, you do not need Google Analytics, so toggle off the Google Analytics option.
6. Click Create project, wait for your project to provision, and then click Continue.

Add a web app to your Firebase project

1. Navigate to the Project Overview screen in your Firebase project, and then click Web.
2. In the App nickname text box, enter a memorable app nickname, such as My Compass Codelab App. You can leave the check box for setting up Firebase Hosting unchecked, because you will be optionally setting up a hosting service in the last step of this codelab.
3. Click Register app > Continue to console.

Great! You have now registered a web app in your new Firebase project.

Upgrade your Firebase billing plan

To use Firebase Genkit and Vertex AI (and their underlying cloud services), you need to upgrade your Firebase project to enable billing.

To upgrade your project's billing plan, follow these steps:

1. Navigate to Firebase Billing Plans within your Firebase project.
2. In the Firebase billing plans dialog, select the Blaze plan and purchase it.

Enable Cloud Firestore

1. Navigate to Build > Firestore Database using the left-navigation pane.
2. Click Create database.
3. Leave the Database ID set to (default).
4. Select your preferred Cloud Firestore location (or just leave it as the default).
5. Click Next.
6. Select Start in test mode.
7. Click Create.

Enable Vertex AI

1. In your terminal, set the default project for the Google Cloud SDK:

   gcloud config set project YOUR_PROJECT_ID
   

You might see a warning message saying "WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues." You will fix this with the next command.

1. Run the following command to set the quota project:

   gcloud auth application-default set-quota-project YOUR_PROJECT_ID
   

2. Enable the Vertex AI service on you project:

   gcloud services enable aiplatform.googleapis.com
   

3. Grant permissions to your service account. Make sure to replace YOUR_PROJECT_ID with your actual project ID

   gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
   --member "serviceAccount:firebase-app-hosting-compute@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
   --role "roles/aiplatform.user" 
   

5. Set up the web app

To run the web app, you'll need to run commands in your terminal and add code within your code editor.

Configure the Firebase CLI to target your project

1. Navigate to the project's root folder: Using your terminal, ensure you're located within the root directory of your codelab project.
2. Link the CLI to your Firebase project: Execute the following command, replacing YOUR_PROJECT_ID with the actual ID of your Firebase project:

   firebase use YOUR_PROJECT_ID
   

Import sample data into Firestore

This codelab provides you with pre-generated sample data to get started quickly. Let's load it into your Firestore instance!

1. To allow the local codebase to run code that would normally use a service account, run the following command in your terminal:

   gcloud auth application-default login
   

   This will open a new tab in your browser. Log in using the same Google Account you used in the previous steps.
2. Run the Data Import Script: in your terminal, run the following commands to import the sample data. Make sure to replace YOUR_PROJECT_ID with the actual ID of your Firebase project.

   cd load-firestore-data
   npm ci
   node index.js YOUR_PROJECT_ID
   cd ..
   

3. Verify Data in the Firebase Console: Go to the Firebase console and open the Firestore section for your project: Firestore. You should now see the imported data schema and its contents displayed in your Firestore instance.

Connect Your Web App to Your Firebase Project

Your web app's codebase needs to be associated with the correct Firebase project to utilize its services, such as the database. To accomplish this, you'll add your Firebase configuration to your app's codebase. This configuration includes essential details like your project's API keys, database URL, and other settings that enable your app to interact securely with Firebase.

1. Obtain your Firebase configuration
   1. Navigate to your Firebase project in the Firebase console.
   2. Click on the gear icon next to "Project Overview" and select Project settings.
   3. In the "Your apps" card, select your web app.
   4. You'll find your Firebase configuration under the "SDK setup and configuration" tab.
   5. Copy const firebaseConfig = {...} fragment of the configuration snippet.
2. Add your Firebase configuration to your web app's codebase:
   1. In your code editor, open the src/lib/genkit/genkit.config.ts file.
   2. Replace the relevant section with the code that you copied.
   3. Save the file.

Preview the web app in your browser

1. In your terminal, install dependencies and then run the web app:

   npm install
   npm run dev
   

2. In your browser, navigate to the locally hosted Hosting URL to view the web app. For example, in most cases, the URL is http://localhost:3000/ or something similar.

Compass is a Next.js app using React Server Components, and this is the homepage.

Click Find my dream trip. You can see it currently displays some hard-coded data for some fixed destinations:

Feel free to explore. When you're ready to continue, click the home button (in the top-right corner).

6. Dive into Generative AI with Genkit

Now you're ready to leverage the power of generative AI in your application! This section of the codelab will guide you through implementing a feature that suggests destinations based on user-provided inspiration. You will use Firebase Genkit and Google Cloud's Vertex AI as the provider for the generative model (you will be using Gemini).

Genkit can store trace and flow state (which allows you to inspect the result of executing Genkit flows). In this codelab, you will be using Firestore to store these traces.

As a final step in this codelab, you will deploy your Genkit app to Firebase App Hosting.

Connect your Genkit app to your Firebase project

Before you can launch Genkit, your codebase needs to be associated with the correct Firebase project to utilize its services, such as the database. To accomplish this, you'll add your Firebase configuration to your Genkit app's codebase. This configuration includes essential details like your project's API keys, database URL, and other settings that enable your app to interact securely with Firebase.

1. Obtain your Firebase configuration
   1. Navigate to your Firebase project in the Firebase console.
   2. Click on the gear icon next to "Project Overview" and select Project settings.
   3. In the "Your apps" card, select your web app.
   4. You'll find your Firebase configuration under the "SDK setup and configuration" tab.
   5. Copy const firebaseConfig = {...} fragment of the configuration snippet.
2. Add your Firebase configuration to your Genkit app's codebase:
   1. In your code editor, open the genkit-functions/src/lib/genkit.config.ts file.
   2. Replace the relevant section with the code that you copied.
   3. Save the file.

Launch the Genkit Developer UI

Genkit comes with a web-based UI that allows you to interact with LLMs, Genkit flows, retrievers, and other AI components.

1. Launch the Genkit Developer UIOpen a new terminal window and navigate to your genkit-functions directory. Then, execute the following command to start the Genkit developer UI:

   cd genkit-functions
   npx genkit start
   

2. In your browser, navigate to the locally hosted Genkit URL. In most cases, it's http://localhost:4000/.

Interact with Gemini

You can now use Genkit's Developer UI to interact with any of the registered models or any of the other AI components, such as the prompts, retrievers, and Genkit flows.

For example, try asking Gemini to suggest a holiday vacation. Note how you can use system instructions to steer the behaviour of the model based on your specific needs. This also works for models that don't natively support system instructions.

Manage prompts

Firebase Genkit introduces Dotprompt, a plugin and text format designed to streamline the creation and management of your generative AI prompts. The core idea behind Dotprompt is treating prompts as code, allowing you to write, maintain, and version control them alongside your application code.

To use Dotprompt, start with a hello-world:

1. In your code editor, open the genkit-functions/prompts/1-hello-world.prompt file.
2. Open dotprompt/1-hello-world in the Firebase Genkit UI.
3. Use any language name or code that you're familiar with, or leave it as an empty string.
4. Click Run.

5. Try a few different values. Large language models are good at understanding abbreviated, misspelled, or incomplete prompts in simple queries like this one.

Enrich your output with structured data

Beyond generating plain text, Genkit empowers you to structure your output for enhanced presentation and integration within your app's UI. By defining a schema, you can instruct the LLM to produce structured data that aligns with your desired format.

Exploring Output Schemas

Let's look at an example for specifying the output schema of an LLM call.

1. Examine the prompt file: Open the dotprompt/2-simple-itinerary file in your code editor. Observe the defined input and output schemas.
2. Interact with the UI: Navigate to the dotprompt/2-simple-itinerary section within the Firebase Genkit UI.
3. Provide input: Populate the place and interests fields with sample data:

   {
       "interests": [
           "Museums"
       ],
       "place": "Paris"
   }
   

4. Click Run.

Understanding schema-driven output

Notice how the generated output conforms to the defined schema. By specifying the desired structure, you've instructed the LLM to produce data that's easily parsed and integrated into your application. Genkit automatically validates the output against the schema, ensuring data integrity.

Moreover, you can configure Genkit to retry or attempt to repair the output if it doesn't match the schema.

Key advantages of output schemas

• Simplified integration: Easily incorporate structured data into your app's UI elements.
• Data validation: Ensure the accuracy and consistency of generated output.
• Error handling: Implement mechanisms to address schema mismatches.

Leveraging output schemas elevates your Genkit experience, enabling you to create tailored, structured data for richer and more dynamic user experiences.

Unlocking multimodal input

Imagine your app suggesting personalized vacation destinations based on images your users find inspiring. Genkit, combined with a multimodal generative model, empowers you to bring this vision to life.

1. Explore the prompt file: Open the genkit-functions/prompts/imgDescription.prompt file in your code editor. Notice the inclusion of {{media url=this}}, a Handlebars syntax element that facilitates incorporating images into your prompt.
2. Open the dotprompt/imgDescription prompt in the Genkit UI.
3. Input an Image URL: In the imageUrls field, paste a URL of an image—for instance, a thumbnail image from Wikipedia showcasing the Eiffel Tower:
4. Input an Image URL: In the imageUrls field, paste a URL of an imag - for instance, a thumbnail image from Wikipedia showing the Eiffel Tower:

   {
       "imageUrls": [
           "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4b/La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg/556px-La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg"
       ]
   }
   

5. Click Run.

7. Implementing retrieval with Vector Similarity Search

While generating creative content with AI models is impressive, practical applications often require grounding the output in a specific context.

In the case of this codelab, you have a database of destinations (places and activities) and aim to ensure that the Gemini model's generated suggestions exclusively reference entries within this database.

To bridge the gap between unstructured queries and relevant content, we'll harness the power of vector similarity search on generated embeddings.

Understanding embeddings and vector similarity

• Vectors: Vectors are numerical representations of data points, often used to model complex information like text or images. Each dimension in a vector corresponds to a specific feature of the data.
• Embedding Models: These specialized AI models transform input data, such as text, into high-dimensional vectors. The fascinating aspect is that similar inputs are mapped to vectors that are close to each other in this high-dimensional space.
• Vector Similarity Search: This technique leverages the proximity of embedding vectors to identify relevant data points. Given an input query, it finds entries in the database with similar embedding vectors, indicating semantic relatedness.

The retrieval process in action

1. Embedding the query: Your user's input (e.g., "romantic dinner in Paris") is passed through an embedding model, generating a query vector.
2. Database Embeddings: Ideally, you've pre-processed your database of destinations, creating embedding vectors for each entry.
3. Similarity Calculation: The query vector is compared to each embedding vector in the database using a similarity metric (e.g., cosine similarity).
4. Retrieval: The most similar entries from the database, based on their proximity to the query vector, are retrieved as relevant suggestions.

By incorporating this retrieval mechanism into your application, you leverage the Gemini model to generate suggestions that are not only creative but also firmly rooted in your specific dataset. This approach ensures that the generated output remains contextually relevant and aligned with the information available in your database.

Enabling Vector Similarity Search in Firestore

In a previous step, you populated your Firestore database with sample places and activities. Each place entry includes a knownFor text field describing its notable attributes, along with a corresponding embedding field containing the vector representation of this description.

To leverage the power of vector similarity search on these embeddings, you'll need to create a Firestore index. This index enables efficient retrieval of places based on the similarity of their embedding vectors to a given query.

Enabling Vector Similarity Search in Firestore

1. In your terminal, run the following command to install the latest alpha component. You need version 2024.05.03 or later:

   gcloud components install alpha
   

2. Create the index, making sure to replace YOUR_PROJECT_ID with your project's ID.

   gcloud alpha firestore indexes composite create --project=YOUR_PROJECT_ID --collection-group=places --query-scope=COLLECTION --field-config=vector-config='{"dimension":"768","flat": "{}"}',field-path=embedding
   

3. Open the placesRetriever in the Genkit Developer UI.
4. Click Run. Observe the scaffolded object with placeholder text, indicating where you'll implement the retriever logic.
5. Open the genkit-functions/src/lib/placesRetriever.ts file in your code editor.
6. Scroll all the way down and replace the placeholder placesRetriever with the following:

   export const placesRetriever = defineFirestoreRetriever({
     name: 'placesRetriever',
     firestore,
     collection: 'places',
     contentField: 'knownFor',
     vectorField: 'embedding',
     embedder: textEmbeddingGecko,
     distanceMeasure: 'COSINE',
   });
   

Testing the retriever

1. Open the placesRetriever retriever in Genkit's Developer UI.
2. Provide the following Query:

   {
       "content": [
           {
               "text": "UNESCO"
           }
       ]
   }
   

3. You can also provide Options, for example to specify how many documents the retriever should return.

   {
       "limit": 4
   }
   

4. Click Run.

You can do additional filtering on the data beyond similarity by adding where clauses to the Options.

8. Retrieval augmented generation (RAG) with Genkit

In previous sections, you've built individual prompts capable of handling text, JSON, and images, generating vacation destinations and other engaging content for your users. You've also implemented a prompt that retrieves relevant destinations from your Firestore database. Now, it's time to orchestrate these components into a cohesive retrieval augmented generation (RAG) flow.

1. Enhance the itinerary prompt: Begin by opening the genkit-functions/prompts/itineraryGen.prompt file in your code editor. Observe how the prompt has been expanded to accept additional inputs, specifically the activities data sourced from the retriever.
2. Discover Genkit Flows: Open the genkit-functions/src/lib/itineraryFlow.ts file. This file introduces you to flows, a powerful Genkit feature.
   • Flows are strongly typed, streamable functions that can be invoked both locally and remotely, with full observability. You can mange and invoke flow both from Genkit's CLI and the Genkit Developer UI.
   • To streamline debugging, consider breaking down lengthy flows into smaller, manageable steps.
3. Integrate the Image Description step: Locate the TODO: 2 comment (around line 70). This marks the spot where you'll enhance your flow. Replace the empty imgDescription placeholder with the output generated by the imgDescription prompt call.
4. Test in the Developer UI: Navigate to the itineraryFlow in Genkit's Developer UI.
5. Use the following input to test successful execution of itineraryFlow with your newly added step:

   {
       "request": "Sightseeing in Paris",
       "imageUrls": [
           "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4b/La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg/556px-La_Tour_Eiffel_vue_de_la_Tour_Saint-Jacques%2C_Paris_ao%C3%BBt_2014_%282%29.jpg"
       ]
   }
   

6. Click Run. Observe the generated output, which should incorporate the image description into the itinerary suggestion.

8. If you encounter any errors or unexpected behaviors, check the Inspect tab for details. You can also use this tab to review the history of executions from the Trace Store.

RAG for your web application

Make sure the web app is still running by visiting http://localhost:3000/ in your browser. If the web app is no longer running, follow these steps:

npm install
npm run dev

Check out the Dream Your Vacation web app page (http://localhost:3000/gemini) and its source code (src/app/gemini/page.tsx) for a Next.js integration example.

9. Deploy your application with Firebase App Hosting

The final step in this journey is deploying your web app. We'll leverage Firebase App Hosting, a framework-aware hosting solution designed to simplify the deployment of Next.js and Angular apps to a serverless backend.

1. Commit your changes to your local git repository and then push to GitHub.
2. In the Firebase console, navigate to App Hosting within your Firebase project.
3. Click Get started > Connect to GitHub.
4. Select your GitHub account and Repository. Click Next.
5. In Deployment setting > Root directory, keep the default value.
6. For the Live branch, select the main branch of your GitHub repo. Click Next.
7. Enter an ID for your backend (for example, compass).
8. When asked whether to Create or associate a Firebase web app, choose Select an existing Firebase web app, and pick the app you created in an earlier step in this codelab.
9. Click Finish and Deploy.

Monitoring Deployment Status

The deployment process will take a few minutes. You can track the progress in the App Hosting section of your Firebase console. Once completed, your web app will be accessible to users.

Automatic Redeployment

Firebase App Hosting streamlines future updates. Whenever you push changes to the main branch of your GitHub repository, Firebase App Hosting will automatically rebuild and redeploy your app, ensuring your users always experience the latest version.

10. Conclusion

Congratulations on completing this comprehensive codelab!

You've successfully harnessed the power of Firebase Genkit, Firestore, and Vertex AI to create a sophisticated "flow" that generates personalized vacation recommendations based on user preferences and inspiration, all while grounding the suggestions in your application's data.

Throughout this journey, you've gained hands-on experience with essential software engineering patterns crucial for building robust generative AI applications. These patterns include:

• Prompt management: Organizing and maintaining prompts as code for better collaboration and version control.
• Multimodal content: Integrating diverse data types, such as images and text, to enhance AI interactions.
• Input/Output Schemas: Structuring data for seamless integration and validation in your application.
• Vector Stores: Leveraging vector embeddings for efficient similarity search and retrieval of relevant information.
• Data Retrieval: Implementing mechanisms to fetch and incorporate data from databases into AI-generated content.
• Retrieval Augmented Generation (RAG): Combining retrieval techniques with generative AI for contextually relevant and accurate outputs.
• Flow Instrumentation: Building and orchestrating complex AI workflows for seamless and observable execution.

By mastering these concepts and applying them within the Firebase ecosystem, you're well-equipped to embark on your own genAI adventures. Explore the vast possibilities, create innovative applications, and continue pushing the boundaries of what's achievable with generative AI.

Exploring alternative deployment options

Genkit offers a variety of deployment options to suit your specific needs, including:

• Cloud Functions for Firebase
• Cloud Run
• Next.js
• Any Node.js environment
• Genkit also supports Go

Just choose the one that is best for you by running the following inside your (package.json) node folder:

npx genkit init

Next steps

• Experiment with prompts and take advantage of the large context windows in Google AI Studio or Vertex AI Studio.
• Learn more about AI retrieval augmented generation (RAG) search.
• Check out the official docs for Firebase Genkit.
• Learn more about similarity search capabilities in Firestore and Cloud SQL for PostgreSQL.
• Go deeper into genAI flows with function calling.