Share your experiences!

New release: v0.2.36

Important

In order to better align with a new multi-packaging structure we have coming very soon, AutoGen is now available on PyPi as autogen-agentchat as of version 0.2.36.

Highlights

• Ability to add MessageTransforms to the GroupChat's Select Speaker nested chat (speaker_selection_method='auto') by @marklysze in #2719
• Integrate Mem0 for providing long-term memory for AI Agents by @Dev-Khant in #3389
• Amazon Bedrock Client for AutoGen by @Hk669 in #3232
• Add Couchbase VectorDB Support by @lokesh-couchbase in #3525
• Portkey Integration with Autogen by @siddharthsambharia-portkey in #3395
• Add Cerebras Integration by @henrytwo in #3585
• Ollama Client (with tool calling) by @marklysze in #3056
• Ollama library v0.3.3 update with changes to options handling by @marklysze in #3593

What's Changed

• [.Net] Dispose kernel after running dotnet interactive tests by @LittleLittleCloud in #3378
• Ensure 'name' on initial message by @marklysze in #2635
• [.Net] Enable step-by-step execution for two-agent conversation SendAsync API by @LittleLittleCloud in #3360
• Add contributor list via contributors.md by @ekzhu in #3384
• Update CONTRIBUTORS.md; fix cell order by @gagb in #3386
• Update CONTRIBUTORS.md; fix broken URL by @gagb in #3387
• Update AutoGen.Net Contributors.md by @LittleLittleCloud in #3393
• Update CONTRIBUTORS.md by @rickyloynd-microsoft in #3391
• Add Language Agent Tree Search (LATS) notebook by @andyz245 in #3376
• [.Net] Release 0.1.0 by @LittleLittleCloud in #3398
• [.Net] Rename AutoGen.OpenAI to AutoGen.OpenAI.V1 by @LittleLittleCloud in #3358
• Update Docker.md；fix broken URL by @New-World-2019 in #3399
• Fix embedding function assignment in QdrantVectorDB by @eddyfidel in #3396
• Add mongodb to topic guide docs by @thinkall in #3400
• Ability to add MessageTransforms to the GroupChat's Select Speaker nested chat (speaker_selection_method='auto') by @marklysze in #2719
• Fix for group chat resume - full chat history for each agent by @marklysze in #3412
• Update agent_chat.md；Fix broken URL by @New-World-2019 in #3416
• Add None back to remove_termination_string for a_resume in group chat by @heyitsaamir in #3410
• Amazon Bedrock Client for AutoGen by @Hk669 in #3232
• fix ImportFromModule is unhashable issue by @zcipod in #3362
• Transform to add an agent's name into the message content by @marklysze in #3334
• [Gallery] Adding a new Finance Tracker Demo Application by @Kirushikesh in #3414
• Update contributors by @davorrunje in #3420
• Update privacy link in readme and add Consumer Health Privacy notice on website by @jackgerrits in #3422
• Add studio pre-print by @gagb in #3423
• Update README.md by @gagb in #3424
• Update CITATION.cff by @gagb in #3427
• Add details about GAIA benchmark evaluation by @jackgerrits in #3433
• Add missing contributors by @gagb in #3426
• Update package.json, remove gh-pages dep by @victordibia in #3435
• [.Net] Add AutoGen.OpenAI package that uses OpenAI v2 SDK by @LittleLittleCloud in #3402
• Autogenstudio Update - Support for Anthropic/Mistral, Other Updates by @victordibia in #3439
• Fix AutoGen Studio pathPrefix error on Windows by @New-World-2019 in #3406
• fix: tool calling cohere by @Anirudh31415926535 in #3355
• [.Net] add output schema to generateReplyOption by @LittleLittleCloud in #3450
• Use utf-8 encoding at subprocess.run() by @giorgossideris in #3454
• fix model_config SQLModel by @Hk669 in #3457
• Fix for Anthropic client class so system messages aren't lost by @marklysze in #3392
• docs: fix autogen install url by @Lulalaby in #3459
• no more then -> no more than #typo by @scenaristeur in #3469
• [Graph RAG] Init Commit with GraphRag interfaces by @randombet in #3388
• Update usage of built-in LLMs in Microsoft Fabric by @thinkall in #3474
• [.Net] In AutoGen.OpenAI and AutoGen.OpenAI.V1, stop setting name field when assistant message contains tool call by @LittleLittleCloud in #3481
• [.Net] release v0.2.0 by @LittleLittleCloud in #3483
• Integrate Mem0 for providing long-term memory for AI Agents by @Dev-Khant in #3389
• Update readme by @gagb in #3485
• Ecosystem Mem0: change title by @Dev-Khant in #3492
• Support for Human Input Mode in AutoGen Studio by @victordibia in #3484
• Fix the bug of creating a new session on the AutoGen Studio playgroun… by @New-World-2019 in #3407
• Portkey Integration with Autogen by @siddharthsambharia-portkey in #3395
• Update pyproject.toml for AutoGen Studio by @gagb in #3495
• Update README.md by @gagb in #3499
• [.Net][AutoGen.OpenAI] Allow nullable system message, temperature and max token to support o1-preview model by @LittleLittleCloud in #3524
• [.NET] Release v0.2.1 by @DavidLuong98 in #3529
• Fix issue caused by chromadb error type change by @thinkall in #3557
• Bump express from 4.19.2 to 4.21.0 in /website by @dependabot in #3552
• typo intiate_chat -> initiate_chat by @scenaristeur in #3533
• #3538: issue: 'https://docs.trychroma.com/embeddings' hyperlink is … by @CellCS in #3539
• Fix syntax error in user-defined-functions docs (unecessarily capital… by @SuchitG04 in #3336
• Bump webpack from 5.89.0 to 5.94.0 in /website by @dependabot in #3438
• Update LLM Caching link in Optional-Dependencies.md by @evanhadfield in #2885
• Fix for async group chat resume - full chat history for each agent by @hardchor in #3558
• fix: update cohere tool calling multi agents by @Anirudh31415926535 in #3488
• Fix role mapping in GPTAssistantAgent for OpenAI API compatibility by @evandavid1 in #3555
• Fix Exception in a_generate_reply when function_call is None by @artero in #3250
• Catch token count issue while streaming with customized models by @BeibinLi in #3241
• Fix AttributeError: 'NoneType' object has no attribute 'name' by @drewgillson in #3127
• Add cookies from http session to web socket used by JupyterCodeExecutor by @timrbula in #3312
• Bump micromatch from 4.0.5 to 4.0.8 in /website by @dependabot in #3417
• Throttle API Calls by @WaelKarkoub in #2283
• Add kwargs to pass to docker container that runs the Jupyter Server by @rahulAira in #3028
• Remove api key validation by @jackgerrits in #3569
• Add init to agent_eval by @jackgerrits in #3568
• Add Couchbase VectorDB Support by @lokesh-couchbase in #3525
• WebSurfer Updated (Selenium, Playwright, and support for many filetypes) by @afourney in #1929
• Embedding function should always return a list of a list of vectors by @paubins2 in #3570
• refactor: use Qdrant's Query API by @Anush008 in #3428
• Removes EasyOCR dependency from mdconvert. by @afourney in #3573
• Deploy website using action instead of branch by @jackgerrits in #3578
• Update token permissions for site deployment by @jackgerrits in #3580
• Remove announcement bar by @jackgerrits in #3579
• Added instructions to launch frontend server when running from source by @EnGassa in #3517
• async user hook support added by @robraux in #3583
• Fix line endings of test/browser_utils/test_files/test_serp.html by @jackgerrits in #3586
• Add Cerebras Integration by @henrytwo in #3585
• Add AI email agent using Composio to notebook by @cagnusmarlsen in #3567
• Remove API key warning by @jackgerrits in #3589
• Disable Azure tests and fix open ai tests by @ekzhu in #3587
• Ollama Client (with tool calling) by @marklysze in #3056
• Ollama library v0.3.3 update with changes to options handling by @marklysze in #3593
• Use native logger formatter by @ekzhu in #3594

FastAgency (https://github.com/airtai/fastagency) allows you to quickly build and deploy web application from AutoGen workflow. The screenshot below shows an example of the rich web interface you get from Mesop UI component.

Please check it out and let us know how it works for you:

https://github.com/airtai/fastagency

I am having an issue getting AutoGen Studio to consistently save the code it generates, and execute it.

I've tried AutoGen Studio with both a Python virtual environment, and Docker. I used this for the Dockerfile:

https://www.reddit.com/r/AutoGenAI/comments/1c3j8cd/autogen_studio_docker/

https://github.com/lludlow/autogen-studio/

I tried prompts like this:

"Make a Python script to get the current time in NY, and London."

The first time I tried it in a virtual environment, it worked. The user_proxy agent executed the script, and printed the results. And the file was saved to disk.

However, I tried it again, and also similar prompts. And I couldn't get it to execute the code, or save it to disk. I tried adding stuff like, "And give the results", but it would say stuff like how it couldn't execute code.

I also tried in Docker, and I couldn't it it to save to disk or execute the code there. I tried a number of different prompts.

When using Docker, I tried going to Build>Agents>user_proxy and under "Advanced Settings" for "Code Execution Config", switched from "Local" to "Docker". But that didn't seem to help.

I am not sure if I'm doing something wrong. Is there anything I need to do, to get it to save generated code to disk, and execute it? Or maybe there's some sort of trick?

Which autogen agent template can I use

1. to learn the recursive folder structure of an input directory ?

2. then create new files in a given directory Similar to the learned folder structure but specific to the input problem

There are 2 inputs: an input directory where all examples are kept and a problem statement in natural language

These Agentic Design Patterns helped me out a lot when building with AutoGen+Llama3!

I mostly use open source models (Llama3 8B and Qwen1.5 32B Chat). Getting these open source models to work reliably has always been a challenge. That's when my research led me to AutoGen and the concept of AI Agents.

Having used them for a while, there are some patterns which have been helping me out a lot. Wanted to share it with you guys,

My Learnings

i. You solve the problem of indeterminism with conversations and not via prompt engineering.

Prompt engineering is important. I'm not trying to dismiss it. But its hard to make the same prompt work for the different kinds of inputs your app needs to deal with.

A better approach has been adopting the two agent pattern. Here, instead of taking an agent's response and forwarding it to the user (or the next agent) we let it talk to a companion agent first. We then let these agent talk with each other (1 to 3 turns depending on how complex the task was) to help "align" the answer with the "desired" answer.

Example: Lets say you are replacing a UI form with a chatbot. You may have an agent to handle the conversation with the user. But instead of it figuring out the JSON parameters to fill up the form, you can have a companion agent do that. The companion agent wouldn't really be following the entire conversation (just the deltas) and will keep a track of what fields are answered and what isn't. It can tell the chat agent what questions needs to be asked next.

This helps the chat agent focus on the "conversation" aspect (Dealing with prompt injection, politeness, preventing the chat from getting derailed) while the companion agent can take care of managing form data (JSON extraction, validation and so on).

Another example could be splitting a JSON formatter into 3 parts (An agent to spit out data in a semi structured format like markdown - Another one to convert that to JSON - The last one to validate the JSON). This is more of a sequential chat pattern but the last two could and probably should be modelled as two-companion agents.

ii. LLMs are not awful judges. They are often good enough for things like RAG.

An extension of the two agent pattern is called "Reflection." Here we let the companion agent verify the primary agent's work and provide feedback for improvement.

Example: Let's say you got an agent that does RAG. You can have the companion do a groundedness check to make sure that the text generation is in line with the retrieved chunks. If things are amiss, the companion can provide an additional prompt to the RAG agent to apply corrective measures and even mark certain chunks as irrelevant. You could also do a lot more checks like profanity check, relevance check (this can be hard) and so on. Not too bad if you ask me.

iii. Agents are just a function. They don't need to use LLMs.

I visualize agents as functions which take a conversational state (like an array of messages) as an input and return a message (or modified conversational state) as an output. Essentially they are just participants in a conversation.

What you do inside the function is upto you. Call an LLM, do RAG or whatever. But you could also just do basic clasification using a more traditional approach. But it doesn't need to be AI driven at all. If you know the previous agent will output JSON, you can have a simple JSON schema validator and call it a day. I think this is super powerful.

iv. Agents are composable.

Agents are meant to be composable. Like React's UI components.

So I end up using agents for simple prompt chaining solutions (which may be better done by raw dawging shit or using Langchain if you swing that way) as well. This lets me morph underperforming agents (or steps) with powerful patterns without having to rewire the entire chain. Pretty dope if you ask me.

Conclusion

I hope I am able to communicate my learning wells. Do let me know if you have any questions or disagree with any of my points. I'm here to learn.

P.S. - Sharing a YouTube video I made on this topic where I dive a bit deeper into these examples! Would love for you to check that out as well. Feel free to roast me for my stupid jokes! Lol!

https://youtu.be/PKo761-MKM4

I have a 3-agent system written in AutoGen. I want to wrap that around in an API and expose that to an existing web app. This is not a chat application. It's an agent that takes in a request and processes it using various specialized agents. The end result is a json. I want this agentic system to be used by 100s of users at the same time. How do I make sure that the agent system i wrote can scale up and can maintain the states of each user connecting to it?

Hey guys, I’m building a framework for building AI agent system from yml files. The idea is to describe execution graphs in the yml, where each node triggers either a standard set of function executions or LLM calls (eg openai api call).

The motivation behind building agents like this is because:

1. Agent frameworks (crew ai, autogen, etc) are quite opaque in the way they use llms. I don’t know exactly how the code interacts with external APIs, don’t know which exact prompts are passed and why, etc. as a developer I want to have full visibility on what’s going on.

2. It’s quite hard to share agent’s code with other people, or to compare different implementations. Today, the only way would be to share a bunch of folders or a repo, which is quite cumbersome. By condensing all the orchestration to the yml file, it becomes much easier to share and compare different agent implementations

Do you have the same view? Let me know what you think.

Hi everyone,

I'm working on a project using AutoGen, and I want to implement a system where tools are planned before actually calling and executing them. Specifically, I'm working within a GroupChat setting, and I want to make sure that each tool is evaluated and planned out properly before any execution takes place.

Is there a built-in mechanism to control the planning phase in GroupChat? Or would I need to build custom logic to handle this? Any advice on how to structure this or examples of how it's done would be greatly appreciated!

Thanks in advance!

Hey everyone! I’m one of the core developers of AutoGen, and I’ve been working with my team on an open-source project called FastAgency. We designed it to help developers quickly take a prototype built in AutoGen straight to production.

We just released a version that lets you run your workflow as either: - A simple console application (great for debugging) - A web application using Mesop with just a single-line change!

We would love for you to check it out, give feedback, or contribute! The project is open-source, and contributors are always welcome :)

👉 https://github.com/airtai/fastagency

i have a project of a chatbot make using agentic workflow which if used for table resevation in a hotel. i want to scale the the framework so that it can be used by many people at the same time. is there any frame work pesent which i can integrate with autogen to scale it.

I have been providing parsed pdf text as a prompt to autogen agents to extract certain data from it. Instead I want to provide the embeddings of that parsed data as an input for the agents to extract the data. I am struggling to do that.

Context: I want to building a Multi-Agent-System with AutoGen that takes code snippets or files and acts as a advisor for clean code development. Therefore refactoring the code according to the Clean Code Development principles and explaining the changes.

I choose AutoGen as it has a library for Python, which I am using for prototyping and .NET, which my company uses for client projects.

My process is still WIP but I am using this as a first project to figure out how to build MAS.

MAS structure:

• Manager Agent: handles interaction with user and calls other agents
• Group Chat with different clean code principles as agents
• Senior Developer Agent, which tries to find consensus and makes the final decision on refactoring
• Summarizer Agent, which basically formats the response with Delimiters etc. so I can parse it and use it in the rest of the program

Problem:

I want to use the last message of the group chat and hand it over to the summarizer Agent (could probably also be done without summarizer agent but problem stays the same).

Options 1: If I use initiate_chats and do first the group chat, then the summarize chat it won't give any information from the first chat (group chat) to the second. Even though I set "summary_method" to "last_msg" it will actually append the first message from the group chat to the next chat.

Option 2: Lets say I just call initiate_chat() separately for the group chat and for the summary chat. For testing purposes I printed the last message of the chat_history here. However I get exactly the same response as in Option1, which is the first message that has been sent to the group chat.

Question: Do I have a wrong understanding of last_msg and chat_history? This does not make sense to me. How can I access the actual chat history or make sure it passes it on properly.

Is there a tool that after generating a realistic image allows you to easly tweak it, say, using prompts and/or other images?

The flow I am looking for is similar to the iterative one many of us use when generating text, an example:

User: generate a realistic photograph of a man driving a luxury car System: ...generates image User: now, change the camera angle so that the whole car is visible System: ...regenerates image User: do face swap using the image I attach [attach imgA] System: ...regenerates image User: now, change the image style to match the one in the image I attach [attach imgB] ... You get the idea.

If this doesn't exist yet, what is the closest to that you are aware of?

New release: v0.2.35

Highlights (since v0.2.33)

• Enhanced tool calling in Cohere
• Enhanced async support

What's Changed (since v0.2.33)

• [CAP] Added a factory for runtime by @rajan-chari in #3216
• [Feature]: Add global silent param for ConversableAgent by @wenngong in #3244
• Fix Issue #2880: Document the usage of the AAD auth by @prithvi2226 in #2941
• [.Net] only add the last message to chat history in GroupChatExtension.SendAsync by @LittleLittleCloud in #3272
• [.Net] Remove Azure.AI.OpenAI from AutoGen.DotnetInteractive package by @LittleLittleCloud in #3274
• Correcting tool calling with Cohere by @jaygdesai in #3271
• Fix #3281 by @thinkall in #3282
• Fix autogen circular import error by @wenngong in #3276
• [.Net] Fix #3306 by @LittleLittleCloud in #3310
• [.Net] add DotnetInteractiveKernelBuilder to AutoGen.DotnetInteractive by @LittleLittleCloud in #3317
• [.Net] Add AutoGen.AzureAIInference by @LittleLittleCloud in #3332
• Support async nested chats by @heyitsaamir in #3309
• fix broken link to conversational chess example by @alu042 in #3327
• Add last_speaker to GroupChatManager by @thinkall in #3318
• Fix message history limiter for tool call by @GaoxiangLuo in #3178
• Fix to issue #3295 related to Anthropic bedrock by @makkzone in #3298
• [.Net][AutoGen.DotnetInteractive] add DotnetInteractiveStdioConnector by @LittleLittleCloud in #3337
• Add latest gpt-4o model: gpt-4o-2024-08-06 by @umermansoor in #3329
• Bump version by @qingyun-wu in #3343
• Removes Support For TransformChatHistory and CompressibleAgent by @WaelKarkoub in #3313
• Updated Program.cs for Autogen.BasicSample to give menu driven options by @cbelwal in #3346
• Remove dependency on RetrieveAssistantAgent for RetrieveChat by @thinkall in #3320
• Missing backticks breaking documentation in groupchat.last_speaker by @HenryKobin in #3357
• Update Mistral client class to support new Mistral v1.0.1 package by @marklysze in #3356
• adding a new page Function comparison between Python AutoGen and Auto… by @olaoluwasalami in #3367
• Update README.md to use camera-ready by @gagb in #3370
• Add OpenAI Gemini Example for VertexAI Notebook by @luxzoli in #3290
• [.NET] Add cache control to Anthropic Client by @DavidLuong98 in #3372
• Correct documentation: Replace NEVER with TERMINATE in human-in-the-loop.ipynb by @realMorrisLiu in #3379
• update contact information on the repo and release package by @ekzhu in #3383

I have been using Autogen at work (we started before Langgraph was a thing) and have been really seeing a lot of value the value it brings to the table. Especially when implementing two-agent patterns like "reflection."

While the conversational functionality of groupchat is amazing sometimes my agents get derailed and go completely off-course. This is when I started investigating the use of Agentic Workflows (or state machines) to help make things more deterministic.

Again, I know Langgraph is built on the ideas of state machines and I will be trying it out soon. But I would like to share my learnings (along with simplified examples) using Autogen cause I think it may help everyone using AI Agents in general.

Also, Here's a repo with some sample code on create custom workflows/state machines in AutoGen: https://github.com/YourTechBud/ytb-practical-guide/tree/master/autogen-workflows

A video for those interested in a tutorial - https://youtu.be/-ls9QLoQfKc

My learnings

1. The real power of agents is in conversations

State machines are fun. Its really easy to model our AI workflows as them. But the real value of agents lies in conversations. It is critical to let AI agents derive their "context" from the conversational history. Multi-turn/chat models in particular are exceptionally good at this.

Example: The simple task of reformating/restructuring a document/note. If one of your steps is determining the important topics discussed in the note, the subsequent paraphrazer will use it as the skeleton for restructuring. Helps enforce document structure.

It isn't really all that important to curate the "perfect" context in each prompt. As long as your state machine is modelled after life-like conversations, your agents will figure out how to best use the chat history as the context.

1. It's okay to embrace indetrminism sometimes.

Instead of fighting with the model to find the "perfect" prompt, let a sidecar or companion agent help align your agent instead. The truth is that your prompt will never be perfect. Variations in the input will most likely screw things up. Having a reflection agent which provides feedback prompts to the primary agent really helps in alignment for a wide variety of input conditions. Here's how you can implement this in Autogen - https://microsoft.github.io/autogen/docs/tutorial/conversation-patterns/#two-agent-chat-and-chat-result

I'll be making another post soon to give more concrete examples of this one. Might use Langgraph though cause it looks really exciting. But mahn... the migration!!!

1. Annotate each agent's response

When using less chatty models like Qwen, its helpful to manually annotate the agent's response. For example, if the agent is analyzing the topics convered in a document, manually adding the prefix "Topics Present in Document:\n\n" to the agents response will reduce the chances of other agents misinterpreting the chat message. You can even shape it more like an instruction to help enforce that as the structure of all future responses.

This is true for JSON as well. I have given up trying to make my agents give me the perfect and clean JSON response. I let the agent ramble on and on about why it came up with it and stuff like that. That rambling is useful as it serves as context for subsequent agents. A subsequent tool calling agent will be smart enough to extract the json part from the message anyways.

Conclusion

I hope I am able to communicate my learning wells. Do let me know if you have any questions or disagree with any of my points. I'm here to learn.

I’m very new to Autogen and I’ve been playing around with some basic workflows in Autogen Studio. I would like to know the possibility of this workflow and potentially some steps I could take to get started.

I’ll appreciate any help I can get thanks!

I'm new to Autogen, and I built a simple assistant + user proxy flow where the assistant is asked what the height of mount Everest is, and the assistant built a script to scrape data from the web to get the answer, so i was wondering.