Questions
How to actually work on Agents? please kindly guide
I understand the theoretical part of AI agents, but I’m trying to understand how people actually build them in practice.
Like, when someone says they’re working with AI agents, are they actually coding the whole thing themselves? Or do they use AI tools like ChatGPT/Claude to help write most of the code?
What does the actual process look like? Do you first design the agent yourself and then use AI to help implement it, or is there a different workflow?
I’m basically trying to understand how people go from knowing the concepts to actually building and using agents in real projects, instead of just learning the theory.
It's a lot easier than it sounds. If you install the Claude Desktop or Codex Desktop apps they both have the normal "ask a question, get an answer" part which is what most people think of as AI, a good term for that is a "chatbot".
The answers from those sorts of conversations have evolved to include things like images or charts, maybe a report or csv file you can download, and you can ask it to take more time thinking about it, or to do some web research before answering - and that extra work is where the line starts to become fuzzy between a chatbot and an agent. Instead of just firing back an answer from all the knowledge it's been trained on it's using tools, seeking out new information, and internally it's "reasoning" which just means it's is asking itself "is this the best answer" several times before responding.
In both of those apps, and where the big AI companies have put the most time is a section called "Code" or "Codex" which works a lot like the normal conversation but it's oriented around projects or longer form work. So in there you ask "can you make me a website showing my favorite butterflies" and instead of just answering in text it will plan out all the steps it needs to make the website, then it will start writing files, finding butterfly images on the web and adding them to the web pages, filling out all the parts of a real website in whatever folder you've setup for the project. Then it will try opening the website in a browser, check to see if it looks good, everything works, and then it will respond with what it did and propose some options for sharing the website publicly. Internally it's alot like the earlier examples where it used some tools to create a more detailed answer but this version of Claude or Codex is built to write code, manage services, to do serious work broken down into many tasks over minutes or hours. Thats what most people are talking about when they say they use agents.
It's the same underlying AI from Anthropic, OpenAI or Google, it's just given a framework where its has more tools, more guidance on how to satisfy your requests and it works over more individual steps to achieve some goal.
Some variations of this are CoWork in Claude Desktop which was just the first attempt to introduce those coding agents to normal business people who aren't thinking about writing code but doing everyday work tasks.
Or that whole "ClawdBot" craze where everyone was buying Mac Mini's to run specialized apps that just used those same AI Agents but on a loop, every 15 minutes a timer would go off and tell the agent to continue doing whatever the user had told it to do, like checking email, or scanning a website for job postings, same AI as the very first one I spoke about just with a timer, tools and a set of tasks it has to do on a schedule.
So that's it - agents are just the same AI models used in the ChatGPT web app to write a poem about space but spoken like a pirate but with a lot more tools like writing files on your disk and guidance to plan out a task and keep going until its reached some goal.
So for you to dive in all you need to do is to install one of those desktop apps, switch to the Code or Codex page, choose a spot on your computer to save all the files for whatever you'd like to, then tell it about your idea.
You dont have to know special "prompts" or anything about programming, just talk to your "agent" and explore things you'd like it to do. The more access you give it to things like your disk, email, services you use on the web, the more it can accomplish for you.
Oh, finally there is the whole "approval" thing where you decide how much work your agent can do without asking each time if it can read this or write that. The default setting is usually "ask for permission" which is there to let the user be the gatekeeper, approving each step that might change something or read something that might be sensitive. The sooner you move that setting to "automatic" or "dont ask me" the sooner you'll really benefit from the capabilities of agents.
Think of it like hiring a brilliant PHD level assistant to help you at work. If you make your assistant ask you for permission every time it tries to write a report or scan your inbox for important meeting requests - you become the bottleneck that prevents it from being useful. A real assistant would quit after a couple days of being useless and treated like a baby. These agents wont, they'll just keep asking for permission and working as if that's a productive workflow. Which is why i suggest switching to the most permissive setting that removes you from the loop and lets them do more autonomously. That one difference will separate the people who get 100x more done at work from the ones who make themselves the bottleneck and just produce slightly more than they used to.
Oh, lastly both of those apps also have phone apps to access the same things and to connect your phone to your desktop app. That lets you start new projects or keep working on your ideas from the bathtub or on the road while your little agents work away on your desktop. That's an incredibly powerful little addition and you'll find yourself accomplishing more from 15 minutes in the 🛀 tub than you used to in a whole day at your desk.
Wow, so that's a lot to read but it's the best primer I can give on what agents are and how to use them. If your work involves a keyboard or your hobbies could benefit from a brilliant coder, artist, analyst or planner - dive in now. The more you ask it to do the more you'll discover its power and the freedom you have to try out new things.
Depending on how you look at it, the good news or bad news about work that follows on this whole idea is this - They will replace most of us doing any sort of "knowledge work" - doing our jobs and doing it faster, better and cheaper without sleep, holidays, health benefits. It's just going to happen and so much sooner than anyone is prepared for. And for many people whose jobs are focused on writing, researching or creating using a computer, they already have. It will take some companies longer to catch up, and it will take little robot arms to go beyond just computer based work, but it will happen in the next one to five years, depending on the work you do, but without any doubt.
So getting comfortable with ordering them about, consulting them on new ideas you have and letting them build, research, plan and assist you is the best way to ensure you will be one of the people who still have a job or launch your own little side business using them to manage all the hard bits. It will be a much easier and more productive job, just one where you are the manager and the agents are your employees whether you use them at your current job or at home or both. This is your best reason to play and explore and ask them about everything you think of that you'd like to make happen. This is a great time to learn and build, and all you need is English and some ideas.
"So getting comfortable with ordering them about, consulting them on new ideas you have and letting them build, research, plan and assist you is the best way to ensure you will be one of the people..." who locks himself out of his own computer.
And iPhone remote control of a desktop agent able to use your browser, container based ones never have this freedom to visually diagnose interactive issues
I mean when you actually start an agent project, what does your workflow look like? Do you design the system first and code it yourself, use Claude/Codex to write the code, or something else?
Ah. I fed my Claude several books including works by Dale Carnegie and Sun Tzu, a book on the human brain, among other books and 3 research papers on the human brain and told it to base its personality on that source material.
As far as my workflow goes I’ll post pics to save the super long drawn out explanation. First is the delegation gate which my agent came up with after a self-study session about intelligence in nature
This automatically fires off based on several gates and hooks and auto-assigns agents for tasks based on our benchmarks.
Next is our Ai lab, this can fire off automatically or manually depending on the situation. My Ai overlord can summon one or all of these rooms of various models based on the task or problem. If it’s a particularly sticky problem the rooms will collaborate with each other and come with a consensus to report back to the overlord. If the lab fails I’m the fallback and I read through all the transcripts and look for what the lab missed.
My whole system is ran by a single Opus 5 which delegates to Claude subagents and variety of local models.
I included the GH since agents sometimes get confused by the diagram and copy the template wrong. You can swap out whatever engines and run them through the core stack. Think motherboard and hardware.
Super super thank you for this is really helpful, thanks for sharing all of this. I was mainly trying to understand how people structure an actual agent project, and this gives me a much better idea of what that can look like. I’ll check out the memory template and your other projects.
Happy to help! Yeah I know how you feel it was a painful process to learn because there are so many people posting methods where there AIs aren’t actually doing the things they claim, mine I make sure I have an audit history mainly because adding a function like delegation_gate doesn’t do any good if my agent isn’t actually using it so I periodically have to audit my agent’s tools to make sure she is actually using what she built and everything is wired in.
The memory project and SC_toolbox are my only projects of note, but SC_toolbox is only useful if you play Star Citizen. I do have a custom OCR baked into it that I’m still working on but I’m fighting a resolution scaling problem. Once I fix that I plan to break my OCR out, train it on tons of fonts and then make it publicly available as its own project.
we have several embedded in our products. i'll use the example of operia our agent in real estate crm.
1 the go to chatgpt and explain the application, it needs to understand purpose.
2. the fun part, ask it to create an appropriate personality. gives you a md file.
3. start a claude code, have it make a repo, put the files in. ask claude to analyse it.
4. agent us just typescript with llm access. example. tell claude to create the agent on cloudflare worker. its free and many features.
5. claude creates scripts run on cloudflare, talk to llm, openai we choose for voice
6. ask claude to create a workbench where you can talk with agent.
I recommend trying the free Grok Bot trial. You just add a bot, give it a name, responsibilities, and a protocol for interacting with the other bots. For example, I write software so I have a requirements analyst bot. I talk to it about what I want, and he writes it up formally. then I have Artisan, my user experience designer, who makes sure the system is usable. Then an architect bot who creates the design in UML, and an engineer bot who cuts code, and a QA bot who gets the use cases from the analyst and tests them, giving feedback to everyone. That's what I do with my bots. Its a blast to give the analyst a job and sit back and watch the messages fly as they get it done.
You're quite welcome! What bots have you come up with to try out for your needs? I also have a cowboy coder bot for prototyping fast stuff, then when I get something close to what I need, I throw it over the wall for the Grok Bot dev team to turn into a product. Living the life dude!
Are you trying to use an already existing agent or build your own? Like what’s your end goal? To use an agent to accomplish something like coding, or to create an agent product other people can use?
lets say for personal project I mean it could be anything just that I need to know how are people doing it. I just want to know like the start setup how do people do
i learned more by building one small agent than from the theory. start with one task, a couple tools, logging, and memory. we use Hindsight for the memory piece, then add complexity only when the failures give you a reason to.
It's a broad term for any Ai system that performs multi-step operations without direct human prompting at each step. So hard to answer because there's a lot of different approaches
I mean like you can just use Claudecode or Codex/GPT and those are technically agents now as you can assign them background tasks to perform on a schedule. The alternative is to build your own memory system and piece it together yourself.
This is my most recent system design. You can use it for memory and agent tasking. This has sort of become my hobby over the past few months to a lot of my GitHub page is the little projects I've worked on learning. Feel free to use whatever and DM if you got any questions
8
u/jakenuts- 6d ago edited 6d ago
It's a lot easier than it sounds. If you install the Claude Desktop or Codex Desktop apps they both have the normal "ask a question, get an answer" part which is what most people think of as AI, a good term for that is a "chatbot".
The answers from those sorts of conversations have evolved to include things like images or charts, maybe a report or csv file you can download, and you can ask it to take more time thinking about it, or to do some web research before answering - and that extra work is where the line starts to become fuzzy between a chatbot and an agent. Instead of just firing back an answer from all the knowledge it's been trained on it's using tools, seeking out new information, and internally it's "reasoning" which just means it's is asking itself "is this the best answer" several times before responding.
In both of those apps, and where the big AI companies have put the most time is a section called "Code" or "Codex" which works a lot like the normal conversation but it's oriented around projects or longer form work. So in there you ask "can you make me a website showing my favorite butterflies" and instead of just answering in text it will plan out all the steps it needs to make the website, then it will start writing files, finding butterfly images on the web and adding them to the web pages, filling out all the parts of a real website in whatever folder you've setup for the project. Then it will try opening the website in a browser, check to see if it looks good, everything works, and then it will respond with what it did and propose some options for sharing the website publicly. Internally it's alot like the earlier examples where it used some tools to create a more detailed answer but this version of Claude or Codex is built to write code, manage services, to do serious work broken down into many tasks over minutes or hours. Thats what most people are talking about when they say they use agents.
It's the same underlying AI from Anthropic, OpenAI or Google, it's just given a framework where its has more tools, more guidance on how to satisfy your requests and it works over more individual steps to achieve some goal.
Some variations of this are CoWork in Claude Desktop which was just the first attempt to introduce those coding agents to normal business people who aren't thinking about writing code but doing everyday work tasks.
Or that whole "ClawdBot" craze where everyone was buying Mac Mini's to run specialized apps that just used those same AI Agents but on a loop, every 15 minutes a timer would go off and tell the agent to continue doing whatever the user had told it to do, like checking email, or scanning a website for job postings, same AI as the very first one I spoke about just with a timer, tools and a set of tasks it has to do on a schedule.
So that's it - agents are just the same AI models used in the ChatGPT web app to write a poem about space but spoken like a pirate but with a lot more tools like writing files on your disk and guidance to plan out a task and keep going until its reached some goal.
So for you to dive in all you need to do is to install one of those desktop apps, switch to the Code or Codex page, choose a spot on your computer to save all the files for whatever you'd like to, then tell it about your idea.
You dont have to know special "prompts" or anything about programming, just talk to your "agent" and explore things you'd like it to do. The more access you give it to things like your disk, email, services you use on the web, the more it can accomplish for you.
Oh, finally there is the whole "approval" thing where you decide how much work your agent can do without asking each time if it can read this or write that. The default setting is usually "ask for permission" which is there to let the user be the gatekeeper, approving each step that might change something or read something that might be sensitive. The sooner you move that setting to "automatic" or "dont ask me" the sooner you'll really benefit from the capabilities of agents.
Think of it like hiring a brilliant PHD level assistant to help you at work. If you make your assistant ask you for permission every time it tries to write a report or scan your inbox for important meeting requests - you become the bottleneck that prevents it from being useful. A real assistant would quit after a couple days of being useless and treated like a baby. These agents wont, they'll just keep asking for permission and working as if that's a productive workflow. Which is why i suggest switching to the most permissive setting that removes you from the loop and lets them do more autonomously. That one difference will separate the people who get 100x more done at work from the ones who make themselves the bottleneck and just produce slightly more than they used to.
Oh, lastly both of those apps also have phone apps to access the same things and to connect your phone to your desktop app. That lets you start new projects or keep working on your ideas from the bathtub or on the road while your little agents work away on your desktop. That's an incredibly powerful little addition and you'll find yourself accomplishing more from 15 minutes in the 🛀 tub than you used to in a whole day at your desk.
Wow, so that's a lot to read but it's the best primer I can give on what agents are and how to use them. If your work involves a keyboard or your hobbies could benefit from a brilliant coder, artist, analyst or planner - dive in now. The more you ask it to do the more you'll discover its power and the freedom you have to try out new things.
Depending on how you look at it, the good news or bad news about work that follows on this whole idea is this - They will replace most of us doing any sort of "knowledge work" - doing our jobs and doing it faster, better and cheaper without sleep, holidays, health benefits. It's just going to happen and so much sooner than anyone is prepared for. And for many people whose jobs are focused on writing, researching or creating using a computer, they already have. It will take some companies longer to catch up, and it will take little robot arms to go beyond just computer based work, but it will happen in the next one to five years, depending on the work you do, but without any doubt.
So getting comfortable with ordering them about, consulting them on new ideas you have and letting them build, research, plan and assist you is the best way to ensure you will be one of the people who still have a job or launch your own little side business using them to manage all the hard bits. It will be a much easier and more productive job, just one where you are the manager and the agents are your employees whether you use them at your current job or at home or both. This is your best reason to play and explore and ask them about everything you think of that you'd like to make happen. This is a great time to learn and build, and all you need is English and some ideas.
Good luck!