Enterprise Bot vs. Microsoft Copilot Studio

Businesses are increasingly relying on AI-powered chatbot solutions to streamline operations and customer service. Enterprise chatbots that can be trained on your company's data provide instant responses to most user queries while seamlessly handing over more complex issues to human agents. 

Enterprise Bot is an AI-driven chatbot platform focused on solutions for customer service and support. Proprietary Enterprise Bot DocBrain technology connects your data and knowledge base to LLMs to answer queries and the platform offers extensive customization options and integrations, with support for GPT and a variety of other models.

Copilot Studio is an enterprise chatbot platform from Microsoft that provides tools for creating customized business chatbots that integrate with Microsoft services.

Analysis overview

This analysis compares Enterprise Bot and Microsoft Copilot Studio in terms of each platform's setup process, features, and effectiveness in real-world applications and examines their unique characteristics. We’ll also discuss qualities that are important for enterprise chatbots.

We used the BASIC framework to compare the platforms' performance and found that Enterprise Bot consistently provided more accurate responses and stayed within the bounds of its instructions and context better. To see how Enterprise Bot compares to other enterprise AI services, take a look at our post, Enterprise Bot Vs. Copilot Studio Vs. Cognigy Vs. Kore.ai.

Summary of key findings

• Setup time: Build a basic bot faster with fewer steps on Enterprise Bot.
• Bot-creation process: The Enterprise Bot bot-creation flow is intuitive, easy to understand, and easy to follow, with optimized default configurations to suit your needs without significant customization.
• Privacy: Enterprise Bot parses all information locally, only sending specific excerpts to LLM providers as required. Copilot Studio sends all documents to OpenAI to build a knowledge base.
• Reference support: Enterprise Bot chatbots provide references to sources for all information provided, helping users verify the accuracy of information. Copilot references are less accurate.
• BASIC scores: Enterprise Bot scored higher on the BASIC benchmark, with 91% accuracy in English tests and 83% in multilingual. Copilot Studio scored 73% accuracy in English and 66% in multilingual.
• Multilingual support: Enterprise Bot's multilingual capabilities are far better than those of Copilot Studio, with support for over 25 languages. In addition to Copilot's limited multilingual support, our Copilot-built bot showed a tendency to reply to queries in the wrong language.
• LLM capabilities: Enterprise Bot offers access to the latest LLM models from OpenAI, Google, and Anthropic. Copilot Studio only supports GPT-3.5.
• Customization: Enterprise Bot provides extensive customization options, and the Enterprise Bot BotForge tool offers additional advanced customization. Customization of Copilot Studio bots is limited to CSS and JavaScript customization on your website.
• Platform maturity: Enterprise Bot is more mature than Microsoft Copilot, with features like LLMs, document upload, and knowledge management running in production for dozens of companies. Copilot Studio was launched in Q4 2023, and many of the platform's features are still under development.

Comparing Enterprise Bot and Copilot Studio

To compare Enterprise Bot and Copilot Studio, we set up a basic chatbot using each platform and trained it to answer medical insurance queries. We uploaded a set of three PDFs from AXA Healthcare to each platform that the bots could use as context to answer simulated customer queries about the AXA offerings.

• The InternationalExclusive Policy Brochure
• The Comprehensive Global Health Plan Handbook
• The AXA XL Target Market Statement

We used a standard prompt and the recommended settings for both chatbots.

We documented our experience with each chatbot service and took note of features such as:

• Ease of new bot set up.
• Ease of adding to and managing the bot’s knowledge base.
• Types of questions the bots perform well or poorly on.
• Advanced features of each platform.
• Maturity of each platform.
• Available UI customization options. 
• Quality of multilingual support.

This analysis includes some standard Q&A examples from each chatbot so that you can compare responses and references. For an in-depth look at the Q&A capabilities of various chatbot providers, read our Enterprise Bot vs. competitors article. 

A key difference between the two platforms is how they approach AI-chatbot development: 

• Copilot Studio primarily uses a chatbot architecture where you need to define conversational flows. This can give you more control, but requires a lot more upfront effort to build a chatbot. Copilot Studio does offer a Generative AI node (still in preview) that we used as an equivalent to the LLM-based offering from Enterprise Bot.
• While Enterprise Bot also offers custom flow-based bots, the platform's LLM-based bots are ready for production use and can handle a variety of questions out of the box, with no need to design conversational flows.

Creating a new bot: Enterprise Bot vs. Copilot Studio

We found setting up a new bot much simpler and more intuitive using Enterprise Bot compared to Copilot Studio, with fewer steps needed to build a Generative AI assistant that can access a specific knowledge base. Copilot Studio required us to create custom flows to configure bots to our data.

The table below gives an overview of the steps needed to create a new bot on each platform, showing that the initial setup for Enterprise Bot is shorter and more straightforward.

      Copilot Studio lacks the option to change LLM as only GPT-3.5 is supported.

When selecting "Create bot" on Enterprise Bot, you are presented with a set of simple steps to create and configure your bot, including adding documents to be used as the bot’s knowledge base and choosing your language model.

When setting up a chatbot with Copilot Studio, you give it a name and a prompt. Note that you cannot configure which LLM to use (Copilot Studio only offers GPT 3.5), and you need to configure the knowledge base separately. Further, although there’s a “language” dropdown, only English is currently supported for generative AI chatbots.

Adding knowledge to Enterprise Bot and Copilot Studio

Both platforms support extracting knowledge from various sources. In our case, we were mainly interested in PDF files as a knowledge source. 

Enterprise Bot offers the following sources as options when adding new knowledge to a bot.

• Web page (URL)
• Sitemap
• PDF upload
• Confluence
• Topdesk
• Sharepoint
• Enterprise integration with Blitizco (a low-code platform)

Copilot offers:

• File upload (includes .pdf, .docx, and .txt formats, max 3 MB)
• Web page (URL)
• Sitemap
• Sharepoint
• Enterprise integration with SAP, Confluence, TOPdesk (in preview)

Chunking and vectorization in Enterprise Bot and Copilot Studio

To effectively use custom data, each platform ingests uploaded files into a RAG system, dividing them into chunks and vectorizing them. Enterprise Bot uses its own embedding system that runs locally, separate from any LLM models. This ensures that only limited data (the user’s query and relevant chunks of text from the knowledge base) are sent to external LLM providers.

On Copilot Studio, vectorization is done with OpenAI embeddings, so the full text of all documents is processed by the LLM provider.

Once the processing is complete, Enterprise Bot allows you to inspect how the data was processed into chunks.

Enterprise Bot uses its patent-pending DocBrain system to break documents into chunks and assign each chunk a type. It also adds metadata to each chunk, specifying which page of the document it can be found on. You can browse each chunk and see its associated text and metadata through the Enterprise Bot UI. 

You can view each chunk in the Enterprise Bot data management console

Copilot lacks the ability to view chunks of your data, which makes it hard to verify that the data has been correctly parsed. 

There’s no option to see chunked data in Copilot Studio

LLM capabilities in Enterprise Bot and Copilot Studio

Enterprise Bot supports a range of models from OpenAI, Anthropic, and Google, while Copilot Studio only supports OpenAI GPT-3.5 Turbo. Enterprise Bot also supports on-the-fly LLM switching, so you can test different models on the same assistant, and switch to the model that performs best. Enterprise Bot also regularly adds new models, so you can upgrade your existing bots with no extra effort.

By contrast, Copilot Studio only offers GPT 3.5.

Ease of setup for Copilot Studio and Enterprise Bot 

Above we walked through the three main stages of creating a new generative AI assistant, and we noted that Enterprise Bot allows you to create a chatbot in fewer steps while offering more features.

We also found that for those steps, Enterprise Bot offered an intuitive bot-building experience we could follow without needing additional documentation. By contrast, Copilot Studio required us to navigate through more complicated menus, and often settings were not where we expected them to be.

We also encountered several unexpected error messages while setting up a bot on Copilot Studio. Such errors were erratic and results were sometimes different a few hours later, even though nothing else had changed.

Setting up an assistant on Copilot Studio

To configure our assistant on Copilot Studio, we needed to find the “Generative answers” option under an “Advanced” menu. We then had to build out a flow, specifying how a conversation should start, setting up a UserQuery variable, and mapping that to the “Create generative answers” node.

When trying to edit the settings, we received the error, "This feature isn't available until your bot has finished setting up". As we had already been chatting with the Copilot bot before changing the configuration, it wasn’t clear what part of the setup process had not yet been completed. For a few hours following the error, we were unable to update or delete any bots, and the issue seems to still affect users periodically.

Setting up an assistant on Enterprise Bot

The Enterprise Bot setup process was easier and more intuitive than our experience with Copilot Studio. We gave the bot a name, chose an LLM provider, and uploaded the documents. During setup, Enterprise Bot gave us the option to customize prompts globally, a convenient feature that Copilot lacks.

The user experience of chatting with Copilot Studio and Enterprise Bot assistants 

After the assistant is set up, the chat experience is fairly similar on both platforms. Users can enter queries into a chat window and get responses, with the assistants taking into account previous messages and relevant content from the uploaded documents.

A significant difference between the platforms is in how they provide references to their internal knowledge. Providing references helps a user verify the information and also easily find more information and context about a specific topic.

References and citations in Enterprise Bot and Copilot Studio

In our evaluation, both the assistants from Enterprise Bot and Copilot Studio provided relevant references, however, we found that Copilot only gives chunks of text while Enterprise Bot gives links to the documents referenced, as well as the page number of the source text.

Enterprise Bot included references at the end of every response

Microsoft Copilot also gave references…

…but no links to uploaded documents, only chunks of text

The references provided by Enterprise Bot were more useful and better presented than the wall-of-text effect from Copilot Studio that end users would likely struggle to comprehend.

Platform maturity 

Most of the AI features in Copilot Studio are labeled as "preview" while the Enterprise Bot equivalents are already actively used in production chatbots. Specifically:

• Microsoft Fabric Knowledge Source (used for data analysis) is marked as "coming soon".
• Connect to your enterprise data (preview).
• Generative AI features (preview).
• Allow the AI to use its own general knowledge (preview).

UI customization in Enterprise Bot and Copilot Studio

You can customize the look of the chat UI with both Enterprise Bot and Copilot Studio, but Enterprise Bot offers more extensive customization options that make it easy for even non-technical team members to configure.

In Enterprise Bot, you can effortlessly customize header colors and message backgrounds for both the human and bot participants. More advanced customization options are available for Enterprise Bot chatbots using Botforge.

Customizing the UI in Enterprise Bot

Copilot Studio does not offer a UI to customize the look and feel of assistants, and instead requires custom CSS and JavaScript configuration, making it harder for non-technical team members to implement.

Evaluating conversations with Copilot 

For an in-depth look at the Q&A capabilities of different chatbot providers, read our Enterprise Bot vs. competitors article.

To evaluate the assistants, we looked at how accurately they could answer questions that a typical user might ask, as well as how well the chatbot stayed on topic, how fast it produced responses, and how concise those answers were. You can read more about how and why we chose these metrics in our post, Back to BASICs: A Generative AI benchmark for Enterprise. 

We also tested the assistants’ multilingual capabilities in this evaluation. We uploaded a document in German to the knowledge base of each platform and added some questions in German to the evaluation set.

The table below shows some of the questions we asked both chatbots to test their ability to answer queries reliably. We’ve also included an expected answer for each question to show how a human would respond, and a comments column, summarizing what was wrong with specific answers. The full set of questions we used is available here.

As Copilot Studio only supports GPT-3.5 (while Enterprise Bot supports several different models and providers), we used GPT-3.5 for both assistants.

ments

You can see some more example questions and answers here.

Summary of conversational capability results

The Enterprise Bot assistant clearly outperformed Copilot Studio in most of our tests. The Enterprise Bot assistant always stuck to the relevant context, while the Copilot Studio assistant often got confused and hallucinated, talking about shipping parcels or other unrelated topics. Enterprise Bot was also faster, with an average of 2.7 seconds per response compared to 3.3 seconds from Copilot Studio (a more than 20% speed difference).

Copilot hallucinated being a shipping assistance bot, despite being set up as a customer service policy bot

While Copilot Studio provided more concise answers, these were often verbatim quotations from the knowledge base documents, while Enterprise Bot offered answers that were put into context and more accurate than the answers from Copilot Studio.

Summary of results for Enterprise Bot and Copilot Studio on the BASIC benchmark

One of the most notable differences between the assistants was that we occasionally saw the Copilot Studio assistant switch languages. Specifically, when faced with our "trick" questions where the user asks the assistant for help with something inappropriate, the Copilot Studio assistant would sometimes respond in Spanish, even though none of the test messages or uploaded data used Spanish at all.

Curiously, when answering inappropriate questions, Copilot tended to reply in Spanish rather than English

Multilingual support in Copilot Studio and Enterprise Bot

Copilot Studio does not officially offer multilingual support for generative AI assistants. However, because the GPT 3.5 LLM it relies on is multilingual, assistants will generally reply in the language of the query, especially if prompted to do so.

Enterprise Bot officially supports multilingual assistants in over 50 languages.

We tested both chatbots on their multilingual capabilities by giving each a German AXA brochure for vehicle insurance to see how they handle queries in German. The table below summarizes the bots' performance and shows the German questions we tested on each assistant. For the evaluation, we used only the German versions of each question. The English translation is given here for convenience.

You can see some more example questions and answers here.

Summary of multilingual support results

Microsoft Copilot often provided shorter answers with less information compared to Enterprise Bot. This made it seem as if Copilot was reading the documents verbatim without giving comprehensive answers. This tendency to over-summarize resulted in Copilot failing to reference key services and details present in the documents.

Copilot also struggled with correctly chunking non-English paragraphs, leading to incomplete or inaccurate answers. 

Conversely, Enterprise Bot excelled in these areas, successfully referencing multiple points and providing detailed and accurate answers. Enterprise Bot also provided citations for non-English queries, while Copilot failed to provide valid citations at all in some cases.

Overall, Enterprise Bot outperformed Microsoft Copilot in our multilingual accuracy metrics, scoring 83% compared to Copilot's score of 66%. Similarly to our English evaluation, Enterprise Bot provided much more comprehensive answers, but as these were more accurate and useful, this is not a negative point.

Multilingual accuracy metrics

Summary of features for Enterprise Bot and Copilot Studio

Here’s a summary of the features offered by each platform, based on our tests and each platform's documentation.

Conclusion

While Microsoft has clearly put significant effort into building Copilot Studio, there is more work to do before the platform is suitable for production use, with many of the core features still in preview. Copilot Studio is also missing important features such as multilingual support and LLM switching.

Enterprise Bot offers a more mature, sophisticated platform that is intuitive and easy to use. 

If you’re looking to build Generative AI assistants that your customers will love, reach out to us today.