GitHub Copilot brings mockups to life by generating code from images

GitHub has announced a slew of updates for Copilot, while also giving a glimpse into a more agentic future for its AI-powered pair programmer.

Among the notable updates includes a feature called Vision for Copilot, which allows users to attach a screenshot, photo, or diagram to a chat, with Copilot generating the interface, code, and alt text to bring it to life.

So for example, someone on a marketing team could take a screenshot of a web page and illustrate some changes they want made to that page. Rather than requesting such changes via text prompts, it’s now possible to upload an image and just ask Copilot to implement the changes as indicated in the file.

It’s worth noting that this has been available as an extension in VS Code since October, with Microsoft confirming in the app’s description that it would eventually be deprecated in favor of a native feature within GitHub Copilot Chat. Previously, it also required the user to have their own API key.

GitHub is also introducing a new feature dubbed “next edit suggestions” as part of Copilot’s broader code-complete functionality.

Before now, GitHub Copilot in VS Code’s editor worked from the cursor position, but now it will look at other recent edits to predict what you might want to do next. It’s basically using richer contextual cues to automatically identify and suggest what the next edit should be.

If the developer wants to accept the suggestions, they can hit the Tab key, or Esc to reject.

GitHub’s agentic future

GitHub also debuted Copilot Edits last November, serving as a new way to carry out multi-file edits using natural language prompts. This features is now graduating to general availability, while also packing a new “agent mode” that identifies all the files that are relevant to the changes that a developer is trying to make — rather than relying on them to manually select the files that the changes should apply to.

“Copilot 1738862286 does more of the work to figure out the intent that you had with your original request, and then tries to solve that,” GitHub CEO Thomas Dohmke told TechCrunch.

Longer term, the goal is to have such “agent modes” apply to other aspects of Copilot, enabling greater automation across a broader range of interlinked tasks. This is why GitHub has also teased a new initiative it’s calling Project Padawan, which is basically a SWE (software engineering) agent that can independently handle entire tasks under the direction of a developer who assigns issues to Copilot.

Dohmke didn’t give any indication when this might be ready for prime time, merely noting that it will be working with partners and the community to improve it.

“We’re looking for partners that are also building agents, to integrate into that flow, and for customers to work with us and and give us feedback — because we know it’s not going to be perfect at this stage,” Dohmke said.