Microsoft has unveiled a new AI-driven feature named “Computer Use” for its Copilot Studio platform, enabling agents to engage directly with websites and desktop applications through simulated mouse clicks, menu selections, and text inputs.
As stated by Charles Lamanna, corporate vice president of business and industry for Microsoft Copilot, this feature represents a significant advancement in Microsoft’s vision for intelligent automation, enabling Copilot Studio agents to interact with graphical user interfaces in a manner akin to that of a human user.
Lamanna stated, “Computer Use adjusts automatically to changes in apps and websites.” It self-corrects in real time through integrated reasoning, allowing work to proceed without disruption. Furthermore, it leverages the strong security protocols and governance structures of Copilot Studio to assist in meeting organizational and industry standards.”
Important use cases for the new feature include:
Automated data input across systems without direct integration;
Automating market research through the gathering and structuring of data from web-based sources; and handling invoices, enhancing financial processes by extracting data and transferring it to accounting systems.
Computer Use operates on Microsoft infrastructure. The company stated that this removes the necessity for organizations to allocate or oversee dedicated RPA environments, thereby decreasing both overhead and maintenance needs.
According to Microsoft, the new feature revitalizes conventional robotic process automation (RPA), enhancing its resilience and accessibility. Users can utilize natural language to describe automation tasks and fine-tune workflows with side-by-side video previews that demonstrate the agent’s interpretation and execution of UI actions, rather than needing extensive coding or scripting.
Further improvements consist of complete transparency, allowing for audits of every action taken by the agent, along with screenshots and reasoning steps.
Next month at its Build conference, Microsoft intends to present Computer Use in more detail.
Microsoft Starts Launching Contentious Recall Feature In other news regarding Microsoft Copilot, the company has recently revealed that it has started to roll out a preview of its Recall feature in Windows, included in the latest Windows 11 preview build.
With Microsoft’s Recall feature for Copilot+ PCs, you can use AI to search for and retrieve past activities across apps, websites, documents, and images by providing a description of what you recall. This feature operates by capturing opt-in snapshots of your screen during the day and necessitates Windows Hello authentication to guarantee that only the device owner can access these snapshots.
The rollout of the feature in preview was originally planned for last June. However, following the emergence of security concerns from the community, the feature was removed. It was planned for an October release, but was removed from the schedule again after it was discovered that the feature was capturing screenshots of users’ credit card details.
Recall is an optional feature that Microsoft has stated can be disabled at any time. It also stated that the new version being released now complies with the company’s dedication to AI and privacy.
Recall does not share snapshots or associated data with Microsoft or third parties, and it is not shared among different Windows users on the same device. Prior to saving snapshots, Windows will request your consent. You maintain control at all times, and you have the ability to delete snapshots, as well as to pause or turn them off whenever you wish. Microsoft stated, “Any future options for the user to share data will necessitate fully informed explicit action by the user.”
