Microsoft reveals Windows 11’s AI roadmap: Smart search, upscaling, more | usagoldmines.com

What’s next for Windows? Microsoft may have just shipped the Windows 11 2024 Update (24H2), but the company is already disclosing its plans for next-gen Windows apps — and there’s some very interesting AI-powered features due by the holidays.

Microsoft disclosed that it is working on several AI additions to Windows and Windows apps: improved Windows search using natural descriptive language, super resolution in Photos, generative fill and erase in Paint, as well as the debut of Recall. All (save Recall) will appear in October as part of the Windows Insider program, with an expected launch in November.

Other breaking Microsoft news you need to know:

Windows 11’s big 2024 Update: How and when you’ll get it

Copilot’s AI will be able to ‘see’ and talk to you, Microsoft says

Windows 11’s 2024 Update: 5 big changes I really like (and more)

All of these features will depend on the NPU inside of Copilot+ PCs, which will now include PCs that feature Qualcomm Snapdragon X Elite processors as well as AMD’s Ryzen AI 300 and Intel’s Lunar Lake. Microsoft also plans several more Copilot features that will run in the cloud, which will include Copilot Voice and Copilot Vision, similar to the innovations used in competing AI services. The timing of these new features will vary by platform, however, as Snapdragon X PCs have shipped for several months now; Microsoft will enable the Copilot+ capable PCs from AMD and Intel with their own updates.

Microsoft has already disclosed more details of its Recall revamp, which the company now says can be bypassed when setting up a new PC or removed later. Windows Recall takes snapshots of your screen every so often, extracting the data and then storing it in case you need it later. The feature came under fire because of claims that it invaded user privacy and was insecure. Now, Microsoft says that it’s storing the screenshots used in Recall and the extracted data inside an encrypted enclave. Security researchers have previously said that the data was stored unencrypted.

Insiders who own Snapdragon X PCs will be able to experience Recall via the Windows Insider program beginning in October. If you own a Copilot+ PC with an AMD or Intel chip inside, the Insider trial will begin in November instead. When will Microsoft release Recall to the public? For now, that’s still a mystery.

New AI features coming for Copilot+ PCs

Microsoft said that it plans to improve search on your PC by using more natural language when searching for files on your PC. You may have already seen this in apps like Microsoft Photos, or Google’s Photos app, too; if you search for “beach,” for example, the apps will use AI to identify beach scenes. Microsoft will apply the same techniques in File Explorer, though it’s not clear what folders or files they’ll apply to.

The improved Windows search does seem to do a better job at identifying scenes than it did before: “BBQ party” was listed as an example search term in a demonstration, below.

“You no longer need to remember file names, settings locations, or even worry about spelling — just type what’s in your mind to find it on a Copilot+ PC,” Microsoft says. It seems unlikely, however, that you’ll be able to find a particular .ini file in the depths of your user folder with the same ease that you’ll find a photo of your aunt’s wedding.

Improved Search will begin with File Explorer, then expand to Windows Search and Settings “in the coming months.”

Super Resolution within Photos may be my favorite potential app, if only because a.) I have many old photos from older, crappy digital cameras, and b.) journalists often are supplied with low-resolution photos that need to be expanded or upscaled for publishing. In either case, the new Super Resolution feature promises to solve those problems.

Microsoft announced Automatic Super Resolution as a way to improve games, but Super Resolution for Photos appears even more practical. There are numerous sites and applications that promise upscaling, and it’s not clear whether this new app will outperform those. Still, Super Resolution for Photos will be free. Tapping the Copilot+ PC’s AI TOPS, you’ll be able to increase the resolution up to eight times, Microsoft says. Super Resolution will be part of Photos, which can already automatically adjust lighting and tint, remove a background, add generative elements, and more.

Speaking of generative AI, you knew that Microsoft had more up its sleeve. In Paint, Microsoft already added Image Creator and Cocreator, which allows for AI-generated images that can be added via layers — also part of Paint. In Photos, generative erase is also a current feature. Photos previously had spot erase, which was wonderful for digitally removing dust that evaded my cleaning cloth when taking a photo of a laptop or docking station. Generative erase, also already in Photos, allows you to define an entire region — such as a power cord, say — and remove that, too.

(You know what’s weird? Generative erase in Photos doesn’t demand an NPU to work. But Microsoft is saying that generative erase in Paint, will.)

Generative Fill basically allows you to define an area within Paint and then create a piece of generative art to fill that region, intelligently working its way into the background. You could sort of already do this in Paint, by creating a second layer, then a piece of AI art, then combining the two. This new feature adds smarts: It instructs the AI to accommodate both your defined region as well as the background when creating the art, making a seamless whole.

The final feature, called Click to Do, is a bit of a puzzle. Normally you can hover your mouse cursor over any part of a web page, right click, and receive any number of options. Now, Microsoft wants you to click the Copilot button, then (left) click somewhere on the screen to receive a list of options that include Copy, Save, Blur background, and more.

“At your request, Click to Do seamlessly connects you to tools and actions to accomplish tasks, like suggesting help with Visual Search with Bing, background blur or erasing objects in Photos, removing backgrounds with Paint, and more,” Microsoft explained. “It also assists with text-related actions such as rewrite, summarize or explain text in line, opening in a text editor, sending an email (with a recognized email address), web searches, and opening websites (with a recognized URL).” Additional actions will be added in the coming months, Microsoft said.

Microsoft demonstrated Click to Do “analyzing” a web page when a user used the appropriate click command. What’s unclear is whether users will remember to actually use the “Copilot click” to do things, or just fall into old habits.