Automatic code generators have been with us a while, but how do modern AI powered bots perform on introductory programming assignments? Join us to discuss the implications of the OpenAI Codex on introductory programming courses on Monday 4th July at 2pm BST. We’ll be discussing a paper by James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly and James Prather [1] for our monthly SIGCSE journal club meetup on zoom. Here is the abstract:
Recent advances in artificial intelligence have been driven by an exponential growth in digitised data. Natural language processing, in particular, has been transformed by machine learning models such as OpenAI’s GPT-3 which generates human-like text so realistic that its developers have warned of the dangers of its misuse. In recent months OpenAI released Codex, a new deep learning model trained on Python code from more than 50 million GitHub repositories. Provided with a natural language description of a programming problem as input, Codex generates solution code as output. It can also explain (in English) input code, translate code between programming languages, and more. In this work, we explore how Codex performs on typical introductory programming problems. We report its performance on real questions taken from introductory programming exams and compare it to results from students who took these same exams under normal conditions, demonstrating that Codex outscores most students. We then explore how Codex handles subtle variations in problem wording using several published variants of the well-known “Rainfall Problem” along with one unpublished variant we have used in our teaching. We find the model passes many test cases for all variants. We also explore how much variation there is in the Codex generated solutions, observing that an identical input prompt frequently leads to very different solutions in terms of algorithmic approach and code length. Finally, we discuss the implications that such technology will have for computing education as it continues to evolve, including both challenges and opportunities. (see accompanying slides)
James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, James Prather (2022) The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming ACE ’22: Australasian Computing Education Conference Pages 10–19 DOI:10.1145/3511861.3511863
The Scream by Edvard Munch 😱, reproduced in LEGO by Nathan Sawaya, the BrickArtist.com
In Canterbury, Glasgow and Manchester, we’re starting a journal club, as part of uki-sigcse.acm.org, the Association for Computing Machinery (ACM) Special Interest Group (SIG) on Computer Science Education (CSE). Journal clubs are like a book clubs, but instead of chatting about books we discuss journal papers instead. Who should come? What’s on the agenda? How can you join and what are our club rules? Read on…
Who should come?
Our journal club will be of interest to:
Educators who teach some flavour of computing or you run a coding boot camp.
Employers who employ and train software engineers, data scientists, developers, coders, programmers, etc
Employees your boss has sent you on a training program or bootcamp to learn or improve your programming
Students what misconceptions about programming have you encountered?
Everyone and anyone who is curious. Our doors are open, this is not an ivory tower. Everyone has something to learn, everyone has something to teach.
Computing educators are often baffled by the misconceptions that their CS1 students hold. We need to understand these misconceptions more clearly in order to help students form correct conceptions. This paper describes one stage in the development of a concept inventory for Computing Fundamentals: investigation of student misconceptions in a series of core CS1 topics previously identified as both important and difficult. Formal interviews with students revealed four distinct themes, each containing many interesting misconceptions. Three of those misconceptions are detailed in this paper: two misconceptions about memory models, and data assignment when primitives are declared. Individual misconceptions are related, but vary widely, thus providing excellent material to use in the development of the CI. In addition, CS1 instructors are provided immediate usable material for helping their students understand some difficult introductory concepts.
In case you’re wondering, CS1 refers to the first course in the introductory sequence of a computer science major (in American parlance), roughly equivalent to first year undergraduate in the UK. CI refers to a Concept Inventory, a test designed to tell teachers exactly what students know and don’t know. According to Reinventing Nerds, the paper has been influential because it was the “first to apply rigorous research methods to investigating misconceptions”. After a brief introduction to the paper and its authors we will discuss the following:
What is good about the paper?
What could be improved?
What is the most surprising or interesting thing you got from the paper?
How convincing is the evidence, arguments and conclusions presented?
How could you use the results and insights in your own teaching or training program?
What are the next steps that follow on from this research? What has already been done to follow on from this work?
Has consensus and opinion moved since the publication of this paper ten years ago? If so, how and why?
Why was this paper voted top 10 of all time by SIGCSE.org members?
Are there any elephants in the room? Does the paper omit anything relevant or gloss over important details?
What do we know that we know (Rumsfeld’s known knowns)
What do we know that we don’t know (Rumsfeld’s known unknowns)
Can’t make it this time? Groups will be running in parallel in Glasgow (23rd March at 1pm with Quintin Cutts) and Canterbury (Friday 27th March, 14.00, Room S132 in the Cornwallis building, School of Computing with Sally Fincher) to discuss the same paper. You can also join us online using the hashtag #SIGCSEJClub. If you’d like to know about future journal clubs in Manchester send an email to listserv@listserv.manchester.ac.uk with the text…
If Manchester, Glasgow or Canterbury aren’t easy for you to get to, start your own journal club by joining SIGCSE at uki-sigcse.acm.org/membership and posting the details to their mailing list. We plan to have regular journal clubs every three months or so where we’ll discuss the same paper nationally during journal club week: this one is Monday 23rd to Friday 27th March.
It will be casual not formal. There will be coffee and refreshments available. We won’t be providing lunch but feel free to bring your own. Some companies call them brown bag meetings, because many of us may will only have an hour so we need to get straight down to business.
It’s about more than just the articles. We are building (and strengthening) communities of practice amongst peers in Computer Science education, not just inside academia but in industry as well. Don’t be shy, all are welcome!
Multidisciplinary is not a dirty word: we aim to foster equality, diversity and inclusion of different people, disciplines, practices and viewpoints. That means we’re open to anyone teaching computer science. That could be in a school, FE college, University, bootcamp, onboarding scheme, company induction or employers staff training program etc. Students are welcome too. The more diverse our journal club is, the stronger it will be.
Topics will reflect the diversity of our membership. We’ve started with student misconceptions, but we invite proposals for which paper we should discuss at our next meeting so we can vote on them.
We’ll pick interesting papers, but they don’t have to be award winning. Papers don’t need to be heavily cited either, but they do have to be thought provoking and provide something meaty to discuss alongside practical tips that can be put into practice straight away.
Any questions? Let me know in the comments section below, via email or twitter.
You might also like…
If you care about the training & education of software engineers and computer scientists, you might also be interested in #CSEdResearchBookClub which will take place on Thursday 5th March at 8pm. They’ll be discussing a paper by Sue Sentanceet al. on using Predict, Run, Investigate, Modify & Make (PRIMM) called Teaching computer programming with PRIMM: a sociocultural perspective. CS education book club is co-ordinated by Jane Waite at Queen Mary University of London (QMUL) see below:
Last month I attended a three day Training of Trainers (ToT) course at the University of Glasgow. Run as an interactive workshop, the course was designed to help leaders of Wikipedia training events to improve their delivery and organisation. Having participated and run several Wikipedia events in the past, such an Ada Lovelace event earlier this year, I was keen to learn how do things better. Here’s a report on the workshop, with some bonus extra curricular Glasgow goodies thrown in for good measure. Thanks again Sara Thomas and Bhavesh Patel for organising and delivering the course.
… work in partnership with organisations from the cultural and education sectors and beyond in order to unlock content, remove barriers to knowledge, develop new ways of engaging with the public and enable learners to benefit fully from the educational potential of the Wikimedia projects.
Most of the workshop participants (pictured top right) were from Gallery, Library, Archive and Museum (GLAM) institutions and a few educational and charitable ones too. Over the three days, here is what we covered:
The second day revisited design skills while touching on delivery skills and group work. This covered elocution, voice projection, body language and an examination the range of experiential activities that can be utilised in workshops. We also discussed aspects of Dave Meier’s accelerated learning (with feedback) and finished the day up with teams preparing for activities for day three.
Day three: The Show Must Go On
The final day of the course finished with the participants divided into four small teams. Each presented a on hour mini-session and had it critiqued by peers. This enabled us to learn from;
Our own mistakes
Other peoples mistakes
Copying / stealing other peoples good ideas, of which there were plenty. Thanks Abd, Daria, Doug, Eoin, Ian, Tara, Ian, Madeleine, Marianne, Saeeda, Tore, Sara and Bhav!
Overall, this was a really useful and memorable training course, one of the best training courses I’ve been on. The content, participants, location were all great and I felt empowered by taking the course as well as making useful contacts from a range of different organisations. It had a clearly defined purpose, well chosen activities and participants, with nothing irrelevant presented. There were tonnes of practical ideas to put into practice straight away which I look forward to doing in 2020. If you’d like to do the course, get in touch with Wikimedia UK.
While in Glasgow, it would be rude not to take advantage of all the bonus extra curricular activities the city has to offer:
Bonus 1: People Make Glasgow Hospitable 🏴
They say that People Make Glasgow, and Glaswegians are very hospitable. In between training sessions our host Sara showed us around the city, including the University cloisters (etc), Inn Deep on the banks of the River Kelvin and Curlers Rest in the West End. Sara’s impressive knowledge of Glasgow and its history is wikipedian in its depth and breadth.
Bonus 2: Glaswegian-Mancunian connections 🇬🇧
To me, Glasgow and Manchester feel like sibling cities separated at birth. If you’re English, Glasgow can feel like a Scottish Manchester. Perhaps Manchester feels like an English Glasgow to the Scots? Here is the case:
Second city syndrome 🥈: As second cities, both Glasgow and Manchester live in the shadow of their more famous capitals, Edinburgh and London. Both cities are the “belly and guts” of their respective nations. Glasgow had its docks, Manchester had its cotton. While both trades are long gone, they leave similar post-industrial legacies on the culture and infrastructure of their respective cities.
Shipping 🚢: Ships, shipping, docks, ports, quays and wharfs run deep in both cities. Glasgow built ships on the River Clyde while Manchester used ships for export and import of goods on its Ship Canal.
Football ⚽: Love it or loathe it, the fitbaw connection between Glasgow and Manchester is strong [1,2]. Scrolling through the list of Manchester United managers I count not just one, two or even three but FOUR Glaswegians. Matt Busby (Belshill is basically Glasgow), Tommy Docherty, Alex Ferguson and David Moyes. Is this a coincidence or catholicism? [1,2] Who knows, but my hypothesis is that being shouted at in a strong Glaswegian accent can make teams perform better (although it didn’t work very well for Moyes). I wonder how many Glasgow kisses Alex Ferguson gave his overpaid prima donna squad to keep them in line? Strangely, the fitbaw manager connection isn’t reciprocated: I can’t find any Mancunians in the list of Celtic managers or the list of Rangers managers
Bonus 3: King Tut’s Wah Wah Hut 🎸
Glasgow is home to the legendary King Tut’s Wah Wah Hut. This humble venue, relatively small with a capacity of only 300, has hosted an impressive range artists including Coldplay, Radiohead, Oasis, Blur, Pulp, Manic Street Preachers, you name it, they’ve played King Tut’s. Curious to find out what all the fuss was about, I arranged to meetup with an old Glaswegian friend for a drink at the venue. Assuming the gig that night would be sold out we asked at the bar who was playing. Turns out they had a handful of tickets left, so we spontaneously bought a pair to see Blanco White. Mixing Andalusian and Latin American influences, Blanco White play melancholic but beautiful tunes using a variety of instruments including the Charango [3]. Part of the reason King Tut’s is legendary is Glaswegian audiences are lively, and it was fun to see the band visibly moved by what Josh Edwards, the lead singer told us was: “easily the best reception we’ve had in months of touring”.
Bonus 4: Like a Brudge over troubled water 🌊
Looking for a walk, run or ride in Glasgow? There are some great routes around the city like the Glasgow River Clyde Bridges, with at least 21 bridges to cross the Clyde on. On an early morning run, I couldn’t find any of the “bridges”, but there were plenty of “brudges” and some fantastic scenery along the Clyde. Och aye!
The Biotechnology and Biological Sciences Research Council (BBSRC) is on the road this autumn in London, Manchester, Bristol, Glasgow and Cambridge. Potential applicants, grant holders and any other interested parties are strongly urged to attend and learn about BBSRC’s plans for the future including new procedures and new Committee structures. The road shows will also provide an opportunity to meet the new members of the BBSRC Senior Management team.
From an original email sent by Alf Game, Deputy Director of Science and Technology Group. See BBSRC Roadshows.
The BBSRC has revised its future strategic priorities and the way in which they will be delivered through responsive mode peer review and is holding a series of road shows “Enabling the Delivery of Excellence with Impact” at various locations across the UK. (more…)
O'Really? 
