02 December 2024
Teaching programming with LLMs - CAS AI TC meeting
If you were unable to join us for Teaching programming with LLMs CAS AI thematic community meeting, don’t worry! Here’s a summary of the session Teaching programming and the key points discussed.
Key Takeaways
- Programming error messages are notoriously challenging for students, with limited evidence on how to improve them effectively.
- Large language models (LLMs) show potential for explaining error messages but require thoughtful integration into classrooms.
- Out-of-the-box LLM explanations often help students but may introduce misconceptions or enable shortcuts like copying solutions.
- Teachers’ professional judgment remains crucial in managing how these tools are used to promote learning rather than dependency.
- A new research study is exploring how LLMs can be integrated into classrooms and how both teachers and students respond to these tools.
At the recent session hosted by Veronica Cucuiat from the Raspberry Pi Computing Education Research Centre, we delved into the potential of large language models (LLMs) to explain programming error messages in a way that benefits novice learners. The conversation centred on their usefulness and challenges, underpinned by recent research and classroom trials.
Students often struggle to interpret programming error messages, which can hinder their debugging skills and confidence. Despite ongoing research, there's no consensus on what makes error messages effective, leaving room to explore whether AI tools can bridge the gap.
Initial Research Insights:
A prototype Python editor was tested, featuring an LLM-generated explanation of error messages. Teachers appreciated the detailed, jargon-free feedback but flagged concerns about occasional inaccuracies and students bypassing learning by simply copying solutions.
Feedback Literacy Theory in Focus:
The study applied feedback literacy theory, positioning feedback as a social interaction involving the teacher, student, and the feedback mechanism itself. Teachers highlighted the importance of tailoring feedback to a student’s understanding, which LLMs currently cannot replicate.
Classroom Integration Challenges:
The discussion underscored that while LLMs can save teachers time, their use must align with broader pedagogical goals. Teachers noted the importance of using such tools to guide or develop understanding, rather than providing direct answers.
The Way Forward:
A new study is underway to examine how teachers and students interact with these tools in live classrooms. Early insights suggest that professional development for teachers will be key to effectively integrating AI tools without compromising the broader learning experience.
Next Steps
For teachers curious about using AI in their classrooms, here are some reflective questions and activity ideas:
Reflective Questions:
- How do you currently guide students in understanding programming errors?
- What role could AI tools play in enhancing—or undermining—this process?
- How might you ensure that students develop debugging skills rather than relying on AI-generated solutions?
Further Resources