01.ai Yi Client #3048

uukuguy · 2024-06-30T15:39:50Z

Why are these changes needed?

To enhance the support of non-OpenAI models with AutoGen. The 01.ai Yi family of models includes yi-large etc.
Developers can utilize the AI capabilities of the Yi series LLMs to perform a variety of tasks, such as text generation, language translation, content summarization, logical reasoning, mathematical calculation, and code generation.

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

uukuguy · 2024-06-30T15:42:18Z

@microsoft-github-policy-service agree

Hk669

I couldn't find the pricing as well. please feel free to share some references. Thanks

Hk669 · 2024-07-01T02:53:05Z

autogen/oai/yi.py

+            api_key
+        ), "Please include the api_key in your config list entry for Yi or set the YI_API_KEY env variable."
+
+        self._oai_client = OpenAI(api_key=api_key, base_url=base_url)


I don't understand, why are we initializing an OpenAI client again? can you please explain this.

According to the official documentation, 01.AI provides an HTTP request interface highly compatible with OpenAI API, but no SDKs for Python or other languages have been found so far.
I'm undecided, so I'll first use the method of the OpenAI API client that I usually use. Other possible implementations include:

Using requests

Inheriting the autogen.oai.OpenAIClient
Any suggestions?

i think this looks good. please let us know the status of the tests using the model. a notebook/ tutorial on this is appreciated. thanks @uukuguy

cc @marklysze @qingyun-wu

Is it possible just to use base_url and api_key on a standard config and use the normal OpenAI client rather than use a specific class for Yi.

I also agree with directly using the OpenAI client to access the Yi API. The only major difference is that the Yi API can only use some of the parameters from the OpenAI API. So, still need Yi client?

Yep, that's a good point and I think it's useful to be able to handle client specific parameters. Okay, we move forward with testing then.

@qingyun-wu or @sonichi, let us know if you think this okay to proceed with or if there's another approach.

@uukuguy, @marklysze thanks for the discussion! Yes, it's okay to proceed with the current approach!

Hk669 · 2024-07-01T02:54:16Z

autogen/oai/yi.py

+
+# Cost per thousand tokens - Input / Output (NOTE: Convert $/Million to $/K)
+YI_PRICING_1K = {
+    "yi-large": (0.003, 0.003),


are these models not open sourced? do they have a separate client that could be accessed with the OpenAI python SDK and API key?

According to the official documentation, 01.AI provides an HTTP request interface highly compatible with OpenAI API, but no SDKs for Python or other languages have been found so far.

According to the official documentation, 01.AI provides an HTTP request interface highly compatible with OpenAI API, but no SDKs for Python or other languages have been found so far.

How should we run and test the client? Can you provide a code sample?

# You can get an account with a $5 free credit at https://platform.01.ai/apikeys to create your own YI_API_KEY. # Use openai client import os from openai import OpenAI client = OpenAI(api_key=os.getenv("YI_API_KEY"), base_url=os.getenv("YI_BASE_URL", "https://api.01.ai/v1")) # Use autogen.oai client from autogen.oai.yi import YiClient client = YiClient(api_key=os.getenv("YI_API_KEY"))

Thanks, are you able to test it with these sample autogen scripts?

https://github.com/marklysze/AutoGenCodeTesting

I use them when developing client classes.

marklysze · 2024-07-01T03:14:07Z

Hey @uukuguy, thanks so much for putting this together. In line with @Hk669's questions, can you describe how you use this client?

I can see there's an official Yi API but it's limited to Beta Testers - is this designed to use that?

I'd love to test it but I'm not sure how to get started and whether I need an account somewhere.

uukuguy · 2024-07-01T04:52:02Z

Hey @uukuguy, thanks so much for putting this together. In line with @Hk669's questions, can you describe how you use this client?

I can see there's an official Yi API but it's limited to Beta Testers - is this designed to use that?

I'd love to test it but I'm not sure how to get started and whether I need an account somewhere.

01.AI's LLM API is located at https://platform.01.ai/, currently featuring only the yi-large model, with plans to deliver multiple models such as yi-large-turbo, yi-medium, and yi-vision. The Yi series of open-source models can be found at https://huggingface.co/01-ai, including several popular models like Yi-1.5-34B.

According to the official documentation, 01.AI provides an HTTP request interface highly compatible with OpenAI API, but no SDKs for Python or other languages have been found so far.

uukuguy · 2024-07-01T04:52:42Z

I couldn't find the pricing as well. please feel free to share some references. Thanks

@Hk669 https://platform.01.ai/docs#models-and-pricing

qingyun-wu · 2024-07-03T02:25:16Z

@uukuguy thanks the the amzing PR. It seems there are some format issues. Could you help fix those by running pre-commit? Thanks!

qingyun-wu · 2024-07-07T16:28:25Z

@uukuguy, I am wondering what's the status of this PR. Could you fix conflicts and also the code format issues? Thank you!

yiranwu0 · 2024-07-08T18:44:34Z

If 01.ai is highly compatible with openai, this PR might not be need. For example, deepinfra support openai's format, and we only need to pass in an additional base_url. Can you check if it works this way?

{
        # using DeepInfra
        "model": "meta-llama/Meta-Llama-3-70B-Instruct",
        "api_key": "YOUR_API_KEY",
        "base_url": "https://api.deepinfra.com/v1/openai" # need to specify the base_url
 }

Hk669 · 2024-07-14T09:16:52Z

@uukuguy i believe we can close this PR, as @yiranwu0 mentioned, it should work just with passing an extra argument base_url and is highly compatible with openai. update us with the status. thanks

cc @marklysze @qingyun-wu please share your thoughts on this. thanks

marklysze · 2024-07-23T22:56:30Z

@uukuguy i believe we can close this PR, as @yiranwu0 mentioned, it should work just with passing an extra argument base_url and is highly compatible with openai. update us with the status. thanks

cc @marklysze @qingyun-wu please share your thoughts on this. thanks

Thanks @Hk669, yes I didn't get a response from Yi about function calling either, so this can be revisited perhaps if they update their API to differentiate or provide more capability than just the OpenAI API compatibility.

uukuguy added 2 commits June 30, 2024 23:03

Added oai.yi

1e2298a

Updated stream and tool_call in oai.yi

3be65c9

uukuguy added 4 commits June 30, 2024 23:48

Fixed test_yi.py

7ab72b4

Fixed usage tokens calculation

b0e25fd

Fixed test/oai/test_yi.py

0edbb6c

Merge branch 'main' into main

60e9eec

qingyun-wu requested review from marklysze and Hk669 June 30, 2024 21:39

Hk669 requested changes Jul 1, 2024

View reviewed changes

Updated 01.ai endpoint base_url

c73f609

Merge branch 'microsoft:main' into main

cf5e8ca

Hk669 mentioned this pull request Jul 1, 2024

[Roadmap]: Roadmap about Enhanced non-OpenAI models #2946

Closed

3 tasks

qingyun-wu requested a review from yiranwu0 July 3, 2024 02:19

Merge branch 'main' into main

56ad68b

qingyun-wu closed this Jul 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

01.ai Yi Client #3048

01.ai Yi Client #3048

uukuguy commented Jun 30, 2024

uukuguy commented Jun 30, 2024

Hk669 left a comment

Hk669 Jul 1, 2024

uukuguy Jul 1, 2024

Hk669 Jul 1, 2024 •

edited

Loading

marklysze Jul 2, 2024

uukuguy Jul 2, 2024

marklysze Jul 2, 2024

qingyun-wu Jul 3, 2024

Hk669 Jul 1, 2024

uukuguy Jul 1, 2024

marklysze Jul 1, 2024

uukuguy Jul 1, 2024 •

edited

Loading

marklysze Jul 1, 2024 •

edited

Loading

marklysze commented Jul 1, 2024

uukuguy commented Jul 1, 2024

uukuguy commented Jul 1, 2024 •

edited

Loading

qingyun-wu commented Jul 3, 2024

qingyun-wu commented Jul 7, 2024

yiranwu0 commented Jul 8, 2024 •

edited

Loading

Hk669 commented Jul 14, 2024

marklysze commented Jul 23, 2024

01.ai Yi Client #3048

01.ai Yi Client #3048

Conversation

uukuguy commented Jun 30, 2024

Why are these changes needed?

Checks

uukuguy commented Jun 30, 2024

Hk669 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hk669 Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uukuguy Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

marklysze Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

marklysze commented Jul 1, 2024

uukuguy commented Jul 1, 2024

uukuguy commented Jul 1, 2024 • edited Loading

qingyun-wu commented Jul 3, 2024

qingyun-wu commented Jul 7, 2024

yiranwu0 commented Jul 8, 2024 • edited Loading

Hk669 commented Jul 14, 2024

marklysze commented Jul 23, 2024

Hk669 Jul 1, 2024 •

edited

Loading

uukuguy Jul 1, 2024 •

edited

Loading

marklysze Jul 1, 2024 •

edited

Loading

uukuguy commented Jul 1, 2024 •

edited

Loading

yiranwu0 commented Jul 8, 2024 •

edited

Loading