[AutoBuild] Introducing CaptainAgent

[LeoLjl]

Introducing CaptainAgent, an agent that can break down tasks and dynamically call AutoBuild as a tool to solve the subtasks. In one tool call, the CaptainAgent will provide the building task, execution task and group name. The building task is used to assemble group of experts, the execution task is what the experts need to solve.

We propose a RAG-like pipeline for expert group generation. Given a building task, we perform similarity search in agent library and find topk candidates, then let LLM to select the best fit. If no expert is suitable for the task, new experts will be generated. Tools will be retrieved and binded to the experts.

This is the updated version of PR #2414 #1749.

This PR is still under active construction. Below is the roadmap for merging it.

• Add full content of tool library.
• Add test for captain agent.
• Add agent library. The full content of agent library is too large and requires deduplication.
• Revise the notebook that introduces MetaAgent. The current one only involves tool library.
• Finish blog post.

Additional feature(s) that are on the todo list.

• Make it more customizable. The current design is a static library. Introduce API for users to add their own tools and agents into metaagent library.

Why are these changes needed?

Related issue number

PR #2414 #1749

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.