In a recent interaction with one of my Discord Mastermind Group members, I learned about an interesting paper from Microsoft Research, GitHub Research, and MIT School of Management titled “The Impact of AI on Developer Prodcutivity: Evidence from GitHub Copilot”.
In this short article, allow me to give you the TLDR. 👇
💡 TL;DR: The paper reveals the productivity gains from employing AI tools in software development. The controlled study, using GitHub Copilot as an AI pair programmer, found that developers with access to this technology were able to complete a task 55.8% faster than those without. Additionally, the results suggested that AI pair programmers could provide significant assistance for individuals transitioning into software development careers.
Wow, what a sales pitch!
(Especially if you consider that GitHub Copilot is a premium product sold by the same company the researchers work for.)
Study Design
From the mid-May to late June 2022, right before Copilot’s grand debut to the general public, 95 code-slinging professionals were rounded up from Upwork, the well-known freelancing platform I’ve already written extensively about:
🔗 Recommended: How to Thrive on Upwork?
This coding rodeo was advertised as a job posting, with eager participants signing contracts and prepping for the coding showdown.
The experiment was conducted under the watchful eye of the Microsoft Research Ethics Review Board. Participants were split into two factions – the treatment group that had GitHub Copilot in their arsenal, and the control group, left to rely on their traditional programming prowess.
All participants were introduced to their task via email and the treatment group received a crash course on GitHub Copilot through a succinct 1-minute video. Surprisingly, five cowboys from the treatment group decided to ride without the Copilot and plunged headlong into the coding arena.
Their task? To construct an HTTP server in JavaScript. The rules were simple: apart from the GitHub Copilot’s assistance to the treatment group, they were free to tap into any resources at their disposal, including the vast knowledge ocean of the internet and wisdom repositories like Stack Overflow and Finxter 👨💻.
Performance was gauged by two metrics – task success and task completion time, giving the researchers an accurate snapshot of the programmers’ prowess. The task was issued and graded using GitHub Classroom, allowing for precise measurement of each participant’s timing and completion.
I have to admit, the experiment was cleverly designed to ensure fairness and accuracy.
Each participant received a personalized copy of the task template, and the time was stamped as soon as they hit ‘start’. The countdown began, with their efforts and progress monitored by the vigilant researchers. The battleground was laden with a dozen tests that the code had to pass – a barometer of submission success. The moment the code passed all twelve tests, the clock stopped, logging the task completion time.
After the task, an exit survey was conducted to gain insights on the participants’ experience. The treatment group was quizzed on their thoughts about GitHub Copilot and how much it expedited their task, while the control group was asked to estimate how much faster they would’ve been if they had the AI assistant at their side after watching a brief demo video.
In essence, this was more than just a coding test; it was a race against the clock, a thrilling tussle between man, machine, and code, all in the pursuit of decoding the enigma of AI-enhanced programming productivity.
Results
In the battle between humans and AI-assisted programmers, the results proved to be compelling.
Of the 166 offers sent, 95 were accepted and these developers were divided into two groups – 45 in the treated and 50 in the control. The majority of the participants were from the 25-34 age group, primarily from India and Pakistan, with an average of 6 years of coding experience and a high level of education.
Fingers flew across keyboards, code was written, and the clock ticked away as the participants embarked on their mission.
The treated group, armed with GitHub Copilot, completed their task in an average of 71.17 minutes.
Meanwhile, the control group, left to their own devices, averaged 160.89 minutes.
🚀 This shows a jaw-dropping 55.8% reduction in completion time for those aided by the AI, a significant boost in productivity.
Even when the four outliers who took more than 300 minutes were dropped from the calculation, the results held firm.
Yet, it wasn’t just about speed.
The study also looked into whether this impact was consistent across different factors such as
- experience,
- employment status,
- income,
- education, and
- software language preference.
(Yeah, some people don’t use Python as their first language.) 🤯
Intriguingly, less experienced developers, those with heavy daily coding loads, and developers aged between 25-44 benefited more from the AI assistant.
In an intriguing twist, participants were asked to estimate their productivity gain using GitHub Copilot. Interestingly, both groups – even those who had not used the tool – estimated a productivity increase of 35%, significantly underestimating the actual boost of 55.8%.
Finally, a glance at the financial implications. The treated group was willing to pay an average of $27.25 per month for GitHub Copilot, while the control group set their cap at $16.91.
No wonder GitHub has launched Copilot as a premium subscription — they are not stupid!
This significant difference, showing the treated group’s higher willingness to pay, indirectly reaffirms the perceived value of Copilot and the benefits the developers derived from using it.
All in all, the results underline the immense potential of AI tools like GitHub Copilot to transform software development, significantly accelerating productivity, particularly for less experienced developers and those with heavier coding loads.
What This Means for the World
If the 55% productivity improvement really proves true, this will completely change the world. To my knowledge, no single tool has ever been able to achieve such an insane leverage potential — think about the ripple effects the productivity boost of software developers may have on society.
Software engineers create software for billions of people, changing their lives in the process. Increasing their output will have a massive impact on the lives of those people.
And don’t worry about losing your job – the world doesn’t run out of software problems anytime soon. If ever.
Imagine a world where software developers, toiling away at tasks that seemingly stretch into eternity, are suddenly gifted an AI-powered tool.
With this tool in hand, their work speed doesn’t just improve; it skyrockets.
Intriguing questions abound: these developers were undoubtedly faster, but did the AI assistance improve the quality of their code, or did reliance on AI result in less attention to the code they were writing?
The plot thickens as the study reveals an unexpected twist – the benefits of AI tools appear to be more pronounced among novices and older developers. It’s as if AI takes on the role of a seasoned mentor guiding an apprentice, hinting at untapped potential for skill initiatives supporting job transitions into software development.
The US hosted over 4.6 million individuals laboring in computer and mathematical occupations, collectively contributing around 2% to the national GDP (2021). Now imagine the socio-economic tsunami if AI tools like GitHub Copilot could supercharge their productivity by 55.8%!
This could herald significant cost savings and drive GDP growth.
But the question remains: how would this productivity windfall be distributed? And how might introducing AI tools rewrite the very definition of these jobs?
The narrative of AI’s influence on software development and beyond is far from over. In fact, it’s only just getting started. It’s a story brimming with suspense, potential, and a future that might be just a keystroke away.
This thrilling exploration is poised to shape the way we understand and engage with the digital world, in software development and far beyond.
With Finxter I try to assist you all the way through this transition so you can remain on the right side of change. Join us free!
🔗 Recommended: Python OpenAI API Cheat Sheet (Free)