O'Really?

July 4, 2022

Join us to discuss the implications of the Open AI codex on introductory programming Monday 4th July at 2pm BST

Filed under: education — Duncan Hull @ 8:14 am
Tags: , , , , , , , , , , , , , , , , , , , ,
Robot icon by Freepik – Flaticon

Automatic code generators have been with us a while, but how do modern AI powered bots perform on introductory programming assignments? Join us to discuss the implications of the OpenAI Codex on introductory programming courses on Monday 4th July at 2pm BST. We’ll be discussing a paper by James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly and James Prather [1] for our monthly SIGCSE journal club meetup on zoom. Here is the abstract:

Recent advances in artificial intelligence have been driven by an exponential growth in digitised data. Natural language processing, in particular, has been transformed by machine learning models such as OpenAI’s GPT-3 which generates human-like text so realistic that its developers have warned of the dangers of its misuse. In recent months OpenAI released Codex, a new deep learning model trained on Python code from more than 50 million GitHub repositories. Provided with a natural language description of a programming problem as input, Codex generates solution code as output. It can also explain (in English) input code, translate code between programming languages, and more. In this work, we explore how Codex performs on typical introductory programming problems. We report its performance on real questions taken from introductory programming exams and compare it to results from students who took these same exams under normal conditions, demonstrating that Codex outscores most students. We then explore how Codex handles subtle variations in problem wording using several published variants of the well-known “Rainfall Problem” along with one unpublished variant we have used in our teaching. We find the model passes many test cases for all variants. We also explore how much variation there is in the Codex generated solutions, observing that an identical input prompt frequently leads to very different solutions in terms of algorithmic approach and code length. Finally, we discuss the implications that such technology will have for computing education as it continues to evolve, including both challenges and opportunities. (see accompanying slides)

All welcome, details at sigcse.cs.manchester.ac.uk/join-us. Thanks to Jim Paterson at Glasgow Caledonian University for nominating this months paper.

References

  1. James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, James Prather (2022) The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming ACE ’22: Australasian Computing Education Conference Pages 10–19 DOI:10.1145/3511861.3511863

September 17, 2020

Join us to discuss learning git on Monday 5th October at 2pm

Filed under: engineering,sigcse,technology — Duncan Hull @ 9:18 am
Tags: , , , , , , , , , , , , , , , , , , , , , , , , ,

The use of git is widespread in software engineering, however many novices struggle to get to grips with its complex distributed information model, challenging command line syntax and leaky abstractions. To investigate these pitfalls, we’ll be talking about a paper published by Santiago Perez De Rosso and Daniel Jackson on Purposes, Concepts, Misfits, and a Redesign of Git at OOPSLA. [1] From the abstract:

Git is a widely used version control system that is powerful but complicated. Its complexity may not be an inevitable consequence of its power but rather evidence of flaws in its design. To explore this hypothesis, we analysed the design of Git using a theory that identifies concepts, purposes, and misfits. Some well-known difficulties with Git are described, and explained as misfits in which underlying concepts fail to meet their intended purpose. Based on this analysis, we designed a reworking of Git (called Gitless) that attempts to remedy these flaws.

To correlate misfits with issues reported by users, we conducted a study of Stack Overflow questions. And to determine whether users experienced fewer complications using Gitless in place of Git, we conducted a small user study. Results suggest our approach can be profitable in identifying, analysing, and fixing design problems.

Details of the zoom meeting have been posted on our slack workspace, see sigcse.cs.manchester.ac.uk/join-us for further information. Thanks to Juha Sorva at Aalto University for recommending this paper. The Git logo by Jason Long at git-scm.com/downloads/logos is licensed under a Creative Commons Attribution 3.0 Unported License.

Journal club dates for your diary

We’ll be meeting on the first Monday of every month throughout autumn, so if you’d like to join us next month or a subsequent month, add these journal club dates to your diary:

  • Monday 5th October at 2pm
  • Monday 2nd November at 2pm
  • Monday 7th December at 2pm

References

  1. Santiago Perez De Rosso and Daniel Jackson (2016) Purposes, Concepts, Misfits, and a Redesign of Git in Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, (OOPSLA), pages 292–310 DOI: 10.1145/2983990.2984018

Blog at WordPress.com.