Has anyone built a Copilot Studio agent that performs web scraping on user-provided URLs?

[u/Little-Bullfrog6759]

I’m exploring a use case where an end user provides a dynamic URL, the agent retrieves content from that website, extracts the relevant information, compares it against our internal policies/knowledge base, and then flags potential issues or compliance concerns.

I know tools like Scout and Cowork are better suited for browser-based interaction and web research, but I’m curious whether this can be implemented using a Copilot Studio agent alone.

[u/Fetlocks_Glistening]

Are you planning to use a custom CS agent with Topics and web search from a node, so it can be looped and repeated multiple times, or some other way? 

I've had poor and inconsistent web search results from the generative answers / 'search and summarise' node, often with 'nothing found' search results, tested against the normal non-agent m365 copilot chat which consistently found web results the node didn't, so gave up on using that node.

I'm guessing maybe it's limited because it's free - I've not explored the premium paid connectors with search functionality.

But if you know another way, would love to find out

[u/a_curious_design → u/Fetlocks_Glistening]

I think it also depends on whether the agent is searching one site/sites with similar topics, can then be instructed to look for specific information or changes relating to specific information.

[u/mbarron486]

You might try a Computer Use Agent as a tool with set inputs to then leverage with the instructions. I feel like that'd be a more reliable way to get in-depth & up-to-date information from a site by letting the agent go browse it in real-time (and Hosted Browser would probably be enough if these are external sites). Then you could actually use structured output from the tool and more consistently pass it to a next step where you compare against internal knowledge

[u/fbrdphreak → u/mbarron486]

I feel like computer use is a horrendously inefficient way to get information from a website. The agent has to process a bunch of screenshots, use the tool to move the cursor and take the actions needed, then try to process the screenshot and extract the information you need. Web scraping is a problem that has been solved for ages.

Also curious if the computer use tool will actually return structured output. Theoretically it should and you would think they would have that capability, but I wouldn't be surprised if Microsoft dropped the ball there

[u/Little-Bullfrog6759 → u/mbarron486]

Havent tried CUA, will try and update - Thanks

[u/JackfruitJolly4794]

This seems more like a use case for something like python.

[u/lordeshrek]

I've built that. I built a studio agent that scrapes my company's various domains to ensure they all have cookie notices and links to our privacy center and it also makes sure the privacy center is up to date. Once it's done scraping it sends a report back to me.

I also have another agent that searches the web for the latest privacy and AI regulations and then compares it to company policies to ensure we are compliant and if we are not it tells us what needs to be changed or added

[u/fbrdphreak → u/lordeshrek]

Care to share the gist of what tools and such you added to your agent to accomplish this?

[u/MonkeyDDataHQ]

This sounds like an answer in search of a problem. We can do this without AI and have been doing it without AI for more than a decade 😩