The Top 10 AI and Copilot Announcements from Microsoft Build 2024

Each year, Microsoft Build brings thousands of developers together so they can learn about new ways to empower them to innovate, create, and build. This year was unique as attendees heard about the power of AI and Copilot coming to life through new features, functionality, and new categories.

What started as a simple AI-powered chat feature has now become a behemoth across all of Microsoft’s business applications and beyond. For me, there were a few standout announcements that showcased the AI and Copilot capabilities that Microsoft is bringing to the developer ecosystem. Additionally, you will notice that the new capabilities span everything from PCs to data to low-code/no-code and more.

1. Defining a New Category with Copilot+ PCs

It’s not often that a new technology category is created. Further, a category that brings two worlds together with a blend of retro tech and cutting-edge tech. But that’s what Microsoft did when they introduced a new category of Windows PCs they have called “Copilot+ PCs”.

Touted as the “fastest, most intelligent Windows PCs ever built”, they will launch with Snapdragon® X Elite and Snapdragon® X Plus processors and features a laundry list of capabilities such as:

• Out-of-the-box robust security powered by the Microsoft Pluton Security processor which is a “chip-to-cloud security technology built with Zero Trust principles at the core. Microsoft Pluton provides hardware-based root of trust, secure identity, secure attestation, and cryptographic services.”
• Forget where you left off? Recall is another feature that allows you to “scroll across time to find the content you need in your timeline across any application, website, document, or more.” Further, Recall “leverages your personal semantic index, built and stored entirely on your device. Your snapshots are yours; they stay locally on your PC.”
• Turn crude drawings and your imagination into a beautifully rendered image with Cocreate. This allows you to bring together your natural language text prompts and digital ink drawings to generate new images in (nearly) real-time.
• Partnerships, apps, and creativity come together with Adobe. Microsoft announced Adobe’s flagship apps Photoshop, Lightroom, and Express are coming to Copilot+ PCs.

2. Extending AI into the OS with Windows Copilot

Continuing the exciting announcements around Copilot+ PCs, new capabilities were launched with Windows Copilot. As an extension of Microsoft Copilot, Windows Copilot Runtime “introduces new ways of interacting with the operating system that utilize AI, such as Phi Silica, the Small Language Model (SLM) created by Microsoft Research that is able to offer many of the same capabilities found in Large Language Models (LLMs), but more compact and efficient so that it can run locally on Windows.”

As part of the Windows Copilot extensions, the Windows Copilot Library provides a set of APIs for developers to create new experiences along with:

• Direct support for Phi 3 mini so that developers can utilize quantized versions of the models. Quantization is a “technique that lets us reap the memory savings from quantization with only a minimal impact on accuracy.”
• The introduction of the developer preview of the Web Neural Network (WebNN) API. WebNN is “an emerging web standard that allows web apps and frameworks to accelerate deep neural networks with GPUs, CPUs, or purpose-built AI accelerators such as NPUs (Neural Processing Units).”
• AI Toolchains and Frameworks that ship with Windows to support BYOM (bring your own models) for developers.

3. Expand the Copilot Platform with Azure AI Studio

The announcement of Azure AI Studio brings the extensibility of AI and Copilot to developers with a full suite of capabilities as a pro-code platform – and it’s a vital part of the Copilot Platform. The services included provide simple setup, API support, and model selection to assist developers with any type of project. For example…

• Azure AI Language is a “service that empowers developers to build intelligent natural language solutions that leverage a set of state-of-the-art language models, including Z-Code++, fine-tuned GPT and more.”
• Azure AI Speech, specifically, Speech analytics, “combines Azure AI services, and PromptFlow, to automatically process and analyze audio data simply by uploading it to cloud storage. With Speech analytics it is easy to gain insights into call center conversations or to extract a conversation summary using AI models from Azure OpenAI as well as Azure AI Language to analyze the accurate transcriptions generated by Azure AI Speech.”
• Azure OpenAI Service with the model catalog provides developers with over 1,600 models to select from and includes “Meta, Mistral, Microsoft, and OpenAI, including GPT 4 Turbo with Vision and Microsoft’s small language model (SLM) Phi3– and new models from Core42 and Nixtla.”

4. Ethical AI Service with Azure AI Content Safety

I wanted to drill into a service that is part of the Azure AI Studio and Azure OpenAI Studio that Microsoft calls “Azure AI Content Safety.” The reason for calling this out is that I have spoken about and covered AI being ethical and safe to ensure that AI has guardrails and people are first in the people + technology equation. As such, the Azure AI Content Safety is a piece of this equation and built on the Microsoft Responsible AI foundation to help with detecting violence, hate, sexual, and self-harm content across text, images, and multimodal content.

• Safety System Messages are part of the generative AI models to provide additional content safety and templates to be quickly deployed to prevent harmful content. The Safety System Messages are available in a playground experience so that developers can test different scenarios before pushing them to production environments.
• Protected Material Detection will soon be upgraded to version 2.0 and “identifies content that highly resembles pre-existing content.” For example, in conjunction with DALL-E, images that are generated from text-based prompts are reviewed in case they “resemble certain types of known creative content, such as studio characters and contemporary artwork.”
• Custom Categories for Azure AI Content Safety “enables you to create your own customized classifier based on your specific needs for content filtering and AI safety whether you want to detect sensitive content, moderate user-generated content, or comply with local regulations.”

5. Invoke Contextual Tools with GitHub Copilot Extensions

Launched in 2008 and acquired by Microsoft in 2018, GitHub has been a mainstay for pro-development. In 2022, GitHub Copilot made its debut to provide AI-assisted coding and redefine software development. Recently, GitHub Copilot Extensions were introduced to help developers “build and deploy to the cloud in their natural language with their preferred tools and services, all without leaving the IDE or GitHub.com.”

GitHub Copilot Extensions, developed by Copilot Partners, are described as “a type of GitHub App that integrates the power of external tools into GitHub Copilot Chat.” As part of the initial launch, the extensions are starting with DataStax, Docker, LambdaTest, LaunchDarkly, McKinsey & Company, Microsoft Azure and Teams, MongoDB, Octopus Deploy, Pangea, Pinecone, Product Science, ReadMe, Sentry, and Stripe.

The concept is simple: “Easily invoke [multiple] tools to get context, perform actions, and generate files and pull requests—accelerating workflows across more tools.”

6. Simplified Copilot Creation with SharePoint Copilot

SharePoint has been a staple of Microsoft Business Applications since 2001 and is now getting an AI boost! Now, admins and business users can create their own custom copilots right from SharePoint in a few simple steps. Further, EVERY SharePoint site will have a copilot!

What I love about this is that AI, through Copilot, is being further democratized in a simple way where the user needs it and in the flow of their work. From within the SharePoint site, you can select “Create Copilot” which will launch a simple wizard to walk you through the process. The process allows you to create a name and description, select the sources (libraries, folders or files) for the AI grounding, and then include a Welcome Message, starting prompts, and instructions. Once created, the custom copilot can now be used and shared with those on your team.

One other aspect I like about this is that the custom copilot respects the SharePoint user permissions. Meaning that the data is only accessible by those provided permission already.

7. Unlocking Data Potential with Copilot in Fabric

Data and AI are inseparable. AI needs data to be grounded and produce intelligent output. On the flip side, AI can be a powerful tool to assist data professionals with data engineering and analysis. That’s why I was excited to see the announcement of the general availability of Copilot in Fabric. With this announcement comes a few new features and functions such as:

• Create a fully functioning report page by providing high-level prompts and identifying the tables, fields, measures, and charts needed.
• Expanded ability to generate custom summaries and answers to questions across full reports which reduces analysis time from hours to seconds.
• A new capability called AI Skills that is designed to “provide a conversational Q&A experience that users can interact with to learn more about their data.”
• Model Explorer that provides “end-to-end visibility of your semantic model in a single tree view, helping you find items in your data fast. You can find items in the search bar, expand and collapse the tree view, see how many of each item are in your semantic mode, and get a single pane access to create or edit semantic model items.”
• Microsoft Fabric Workload Development Kit to help pro and citizen developers to “design, build, and interoperate applications within Microsoft Fabric. Applications built with this kit will appear as a native workload within Fabric, and software developers can publish and monetize their custom workloads through the Microsoft Azure Marketplace.”

8. Amplifying Low-Code/No-Code with Connectors and Flows

Citizen and pro developers are now gaining new capabilities to equip them with their low-code/no-code creation and innovation. This comes in the form of Copilot Connectors and AI Flows. The Copilot Connectors provide over 1,400 connectors for the Power Platform, Microsoft Graph, Power Query, SharePoint, OneDrive, Dataverse, Fabric, and third-party apps.

Further, professional developers can (soon) create and publish custom Copilot extensions that “expand the actions that Microsoft Copilot can take on the user’s behalf, customize grounding knowledge with relevant business data, and enable hand-off to other copilots.”

Another area of excitement is the introduction of AI Flows for Power Automate. This means that “automation shifts from rules-based to AI-based automation that is adaptive and outcome-based.” Think about the last workflow you built and the logic you needed to consider and build for the flow to function properly. With the new AI Flows, it can “interpret process objectives, generate automation plans, autonomously select actions, and validate outcomes—all while allowing for easy adjustments.” On top of this, pro devs can set up “guardrails” that puts safety around the autonomy of the AI Flow.

The capabilities don’t stop there. RPA (robotic process automation) is now AI-powered by allowing developers to create automations through “screen sharing and voice coaching, leveraging multi-modal understanding for comprehensive training.” This functionality reduces the RPA creation time from days to minutes.

The application lifecycle management (ALM) tools in Power Apps are now equipped with native Git integration to synchronize environments with Git repositories. Additionally, developers can “view and use source code in a readable YAML format, create templates for screens, reuse code, and store files in a code repository.”

9. Copilot Studio Enhancements

In November 2023, the new Microsoft Copilot Studio solution was introduced as part of the evolution of Microsoft Copilot. The solution allows people to easily custom Copilot for Microsoft 365; build, test, and publish standalone Copilots; and, manage and secure the Copilots.

Fast forward to Microsoft Build 2024 and Copilot Studio is getting new experiences to enhance the Copilot creation and management process:

• Conversational AI analytics and Insights will help customers understand “what their copilot is doing, how generative AI is responding, when it was unable to give the right answers and why—and recommendations on what to do to improve it.”
• Templates are now available to make the experience even easier for building copilots. Copilot Studio now comes with several templates that align with departments and industries. For example, “Safe Travels for comprehensive travel support, Organization Navigator for organizational clarity, Kudos Copilot for fostering recognition, and Wellness for employee health insights.”
• Security and controls in Copilot Studio have been enhanced to provide administrators with the ability to customize settings. Detailed information about governance, audit logs, inventory, and sensitivity help admins monitor business impact and respond accordingly.

10. Microsoft Copilot Updates

Last, but certainly not least, are the announcements about new Microsoft Copilot capabilities. AI has truly been democratized by putting Copilot into the hands of users to assist them in the flow of work. What I like about these new features is the continuation of the concept of “in the flow of work” by infusing Copilot into the core collaboration solutions of users worldwide – Microsoft Teams, Loop, and Planner – by bringing a new personal assistant to organizations called “Team Copilot”. This new teammate can act as a:

• Meeting facilitator: Copilot enables a more productive discussion in meetings by managing the agenda and taking notes anyone in the meeting can coauthor.
• Group collaborator: Copilot helps everyone get more out of chats, surfacing the most important information, tracking actions items, and addressing unresolved issues.
• Project manager: Copilot ensures every project runs smoothly by creating and assigning tasks, tracking deadlines, and notifying team members when their input is needed.

---