Enhance vertexai integration

[luxzoli]

Why are these changes needed?

• Enhance VertexAI authentication by supporting credentials objects in the llm_config for more flexibility:

import google.auth

credentials, project = google.auth.default()

from google.auth import impersonated_credentials

target_scopes = [
    'https://www.googleapis.com/auth/cloud-platform']

target_credentials = impersonated_credentials.Credentials(
  source_credentials=credentials,
  target_principal='autogen@autogen-with-gemini.iam.gserviceaccount.com',
  target_scopes = target_scopes,
  lifetime=500
)

llm_config = {
        "config_list": [
            { 
                "model": "gemini-1.5-pro",
                "api_type": "google",
                "credentials": target_credentials,
                "project": "autogen-with-gemini"
            }
         ],
}
assistant = AssistantAgent(
    "assistant", llm_config=llm_config, max_consecutive_auto_reply=5
)

• Adding safety settings conversion to VertexAI format from the OAI_CONFIG_LIST

• Support system_message, which are called system_instructions with Gemini

• Consolidate message handling in send_message to use the officially supported PartsType for the content argument

• Consolidate the project_id argument for the GeminiClient passed to the vertexai.init() method

• Fix issue with Gemini message format in chats

Related issue number

Relates to #2387

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[luxzoli]

Code for reproducing the issue with the Gemini message format in chats:

import os
from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union

import autogen
from autogen import Agent, AssistantAgent, ConversableAgent, UserProxyAgent
import google.auth
from google.auth import impersonated_credentials

credentials, project = google.auth.default()


target_scopes = [
    'https://www.googleapis.com/auth/cloud-platform']

target_credentials = impersonated_credentials.Credentials(
  source_credentials=credentials,
  target_principal='autogen@autogen-with-gemini.iam.gserviceaccount.com',
  target_scopes = target_scopes,
  lifetime=500)

llm_config = {
        "config_list": [{"model": "gemini-1.5-pro", "api_type": "google", "credentials": target_credentials, "project": "autogen-with-gemini"}],
}

financial_tasks = [
    """What are the current stock prices of NVDA and TESLA, and how is the performance over the past month in terms of percentage change?""",
    """Investigate possible reasons of the stock performance.""",
    """Plot a graph comparing the stock prices over the past month.""",
]

writing_tasks = ["""Develop an engaging blog post using any information provided."""]

financial_assistant = autogen.AssistantAgent(
    name="Financial_assistant",
    llm_config=llm_config,
)
research_assistant = autogen.AssistantAgent(
    name="Researcher",
    llm_config=llm_config,
)
writer = autogen.AssistantAgent(
    name="writer",
    llm_config=llm_config,
    system_message="""
        You are a professional writer, known for
        your insightful and engaging articles.
        You transform complex concepts into compelling narratives.
        Reply "TERMINATE" in the end when everything is done.
        """,
)

user = autogen.UserProxyAgent(
    name="User",
    human_input_mode="NEVER",
    is_termination_msg=lambda x: x.get("content", "") and x.get("content", "").rstrip().endswith("TERMINATE"),
    code_execution_config={
        "last_n_messages": 1,
        "work_dir": "tasks",
        "use_docker": False,
    },  # Please set use_docker=True if docker is available to run the generated code. Using docker is safer than running the generated code directly.
)

chat_results = user.initiate_chats(
    [
        {
            "recipient": financial_assistant,
            "message": financial_tasks[0],
            "clear_history": True,
            "silent": False,
            "summary_method": "last_msg",
        },
        {
            "recipient": research_assistant,
            "message": financial_tasks[1],
            "summary_method": "reflection_with_llm",
        },
        {
            "recipient": writer,
            "message": writing_tasks[0],
            "carryover": "I want to include a figure or a table of data in the blogpost.",
        },
    ]
)

Details abbout the issue, which the commit fixes:

Traceback (most recent call last):
  File "/home/lux_zoltan_andras/autogen_testing/autogen_google_auth/chat_sequence_gemini.py", line 80, in <module>
    chat_results = user.initiate_chats(
  File "/home/lux_zoltan_andras/.local/lib/python3.10/site-packages/autogen/agentchat/conversable_agent.py", line 1250, in initiate_chats
    self._finished_chats = initiate_chats(_chat_queue)
  File "/home/lux_zoltan_andras/.local/lib/python3.10/site-packages/autogen/agentchat/chat.py", line 199, in initiate_chats
    __post_carryover_processing(chat_info)
  File "/home/lux_zoltan_andras/.local/lib/python3.10/site-packages/autogen/agentchat/chat.py", line 119, in __post_carryover_processing
    ("\n").join([t for t in chat_info["carryover"]])
TypeError: sequence item 2: expected str instance, dict found

[luxzoli]

@sonichi as requested, this is the new PR for the previos Enhance vertexai integration PR #3067

[luxzoli]

Just verify with the notebook and it works with vertax platform.

Here's a few feedback

• in llm_config, use project_id instead of project when setting up project. Otherwise user might put the project name in that field and get a authentication error (like me...)
• use another name other than Google_API_Key when getting gemini key. For example Google_Vertax_Gemini_API_Key, to avoid confliction with other Google API keys.

Thanks @LittleLittleCloud, I have made the suggested changes. I chose the name GOOGLE_GEMINI_API_KEY for the environment variable, as Gemini is not necessarily used via VertexAI. Please let me know if you have a better name in mind 🙂

[LittleLittleCloud]

@qingyun-wu can you approve the OpenAI test

[gitguardian]

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	e43a86c	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	bdb40d7	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	954ca45	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	eff19ac	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	06a0a5d	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	0524c77	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	d7ea410	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	e43a86c	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	841ed31	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	802f099	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	9a484d8	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	e973ac3	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	89650e7	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	e07b06b	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	abe4c41	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	7362fb9	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	e43a86c	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	954ca45	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	bdb40d7	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	abad9ff	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	954ca45	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	c7bb588	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	b97b99d	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	e43a86c	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	49e8053	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	49e8053	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	49e8053	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	d422c63	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	97fa339	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	501610b	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	954ca45	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	bdb40d7	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	e43a86c	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	bdb40d7	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	79dbb7b	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	a86d0fd	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	394561b	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	3eac646	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	f45b553	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	6563248	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	2b3a9ae	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	c03558f	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	c03558f	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	2b3a9ae	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	0a3c6c4	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	76f5f5a	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	954ca45	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	2b3a9ae	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	c03558f	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	76f5f5a	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	0a3c6c4	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	3b79cc6	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	11baa52	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	c03558f	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	2b3a9ae	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	c03558f	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	2b3a9ae	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	3b79cc6	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	11baa52	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	11baa52	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	3b79cc6	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	0a3c6c4	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	76f5f5a	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	c03558f	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	2b3a9ae	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	2b3a9ae	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	c03558f	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}