Building a Custom AI Chatbot for Computer Science Education

There are many LLM’s available to students and educators today but getting students to use them effectively poses a number of challenges.

Background

• Merchant Taylors’ School is a leading independent school in the UK, with a strong reputation for academic excellence and innovation.
• The school offers computer science as a subject at both GCSE and A-level and aims to provide students with a solid foundation in computational thinking, programming, and problem-solving skills.
• The school also encourages students to explore the potential and limitations of artificial intelligence, and to develop their own AI projects using various tools and platforms.

To mitigate these challenges and other issues such as reliability we decided to investigate the creation and use of subject specific chatbots with limited but verified domain knowledge.

To investigate this, we used Copilot Studio to create a custom copilot for use by Computer Science students. The current version of the chat chatbot, named Computer Science student copilot, is designed to answer students’ questions about computer science topics, such as definitions, concepts and code structures to support students both in an out of the classroom.

The chatbot uses a combination of document-based and generative AI to produce relevant and accurate answers from multiple sources, such as the GCSE and A-level specifications, core definitions sheets, web pages, and the .Net reference library.

Challenges with LLM use

• Inaccurate or irrelevant answers (hallucinations)
• Lack of source information
• Controlling the level or depth of its answers, which could be too simple or too complex for the students’ needs.
• Tracking and monitoring the student use, which could limit the teachers’ ability to intervene or adjust their teaching strategies.

CoPilot Studio

Copilot Studio is a cloud-based tool designed to allow organisations to create their own custom Copilot powered chatbot solutions for tasks or extend the capabilities of Microsoft Copilot for Microsoft 365. It’s a low-code tool that leverages large language models (LLMs) and other sources of knowledge to build AI-powered conversational interfaces. The LLM is used to analyse the initial question and match it to a topic. Rules for that topic then define the actions that are then taken such as generating the answer from the included sources.

Key features that make it suitable for the school’s needs:

• It allows users to create fixed and generative responses from natural language inputs, which can be tailored to the specific domain and audience of the chatbot.
• It ensures that the generative responses are always referenced, which means that the chatbot can provide the sources or links of its answers, and the users can verify or extend their research.
• It enables users to choose multiple resources to generate answers from, which can include documents, web pages, or other chatbots, and to prioritize or filter them according to the type or level of the questions.
• It supports hierarchal generative responses, which means that the chatbot can generate different levels or depths of answers depending on the user’s input or preference.
• It allows users to publish the chatbot to the web, intranet, or Microsoft Teams channels, which can facilitate the access and distribution of the chatbot to the target users.
• It can trigger other power automate workflows, which can enable the chatbot to perform other tasks or actions, such as sending emails, creating documents, or updating databases.

The Computer Science Copilot

Most of work of the bot is done by the standard conversational boosting feature of CoPilot that generates answers from the set source documents and references them.

Hierarchal generation was implemented for a number of topics such as definitions where it first of all refers to a specific definitions document to generate the answer but if fails to produce the definition from that goes back to the other sources.

The CoPilot has access to the complete .Net framework reference which gives it the ability to explain and write some code

Unlike a full LLM it is unable to solve a complete GCSE problem.

A range of fixed responses such as the times of Computer clubs and help clinics where also implemented.

The CoPilot was made available to students via a channel in their Microsoft teams classes. This meant that the teacher in each class had full visibility of questions asked.

Student Usage

Students where very happy to use the chatbot and felt that it gave them helpful answers. Some students used the references to further gain further understanding. Students used the bot in lessons when the teacher was helping other students as well as outside of the classroom. Some students attempted to lead the bot off topic but were not successful. The bot was heavily used by some students on exam leave in the build up to their GCSE exams.

Advantages of this Solution

• Teacher/school has complete visibility of questions and answers
• Removes off task potential from student AI use as limited to one domain area
• Easy for student to verify answer and extend their research through hyperlinked references
• It can’t answer full questions but can assist with chunks of them.