🤖 From stagnation to acceleration: Can AI chatbots really boost worker productivity?

💡 5 Quick Questions for … economist Veronique de Rugy on the importance of economic growth

"We can ill afford to rest on our laurels, even to pause in retrospect. Times and conditions change so rapidly that we must keep our aim constantly focused on the future.” - Walt Disney

---

The Essay

🤖 From stagnation to acceleration: Can AI chatbots really boost worker productivity?

We don’t just need dazzling new technologies. We need those technologies to make us more productive. For my part, what we’re seeing now with generative AI models (including Bing AI, ChatGPT, Dall-E, and Midjourney) qualifies as dazzling.

But will they make us more productive? Here are two early-but-interesting data points. In “How Does Artificial Intelligence Affect Productivity? Evidence from an Experiment” Shakked Noy and Whitney Zhang assigned 444 people with college degrees who work in different jobs to write something related to their job and paid them for it. They also randomly chose half of them to use ChatGPT. The results:

Our results show that ChatGPT substantially raises average productivity: time taken decreases by 0.8 SDs and output quality rises by 0.4 SDs. Inequality between workers decreases, as ChatGPT compresses the productivity distribution by benefiting low-ability workers more. ChatGPT mostly substitutes for worker effort rather than complementing worker skills, and restructures tasks towards idea-generation and editing and away from rough-drafting. Exposure to ChatGPT increases job satisfaction and self-efficacy and heightens both concern and excitement about automation technologies.

Or as Bing AI puts it: “Our findings show that ChatGPT helps people work faster and better: they finish their tasks much quicker and their work quality improves a lot.” Specifically, the tasks are completed nearly 40 percent faster. That is amazing.

The second data point, or collection of data points, comes from GitHub, the software development platform, which did a productivity analysis on its Copilot program that turns natural language prompts into coding suggestions. From the analysis:

In the experiment, we measured—on average—how successful each group was in completing the task and how long each group took to finish. The group that used GitHub Copilot had a higher rate of completing the task (78%, compared to 70% in the group without Copilot). The striking difference was that developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot. Specifically, the developers using GitHub Copilot took on average 1 hour and 11 minutes to complete the task, while the developers who didn’t use GitHub Copilot took on average 2 hours and 41 minutes. These results are statistically significant (P=.0017) and the 95% confidence interval for the percentage speed gain is [21%, 89%].

Again, Bing AI breaks it down:

We tested how well and how fast two groups of programmers could do a coding task. One group used GitHub Copilot, an AI tool that helps with writing code. The other group did not use it. We found that the group with GitHub Copilot did better and faster than the group without it. They finished the task in about half the time as the other group. This difference was not just by chance, but very likely because of GitHub Copilot.

Again, super impressive results. How impressive? Wharton’s Ethan Mollick offers perspective:

Ethan Mollick @emollick

It is really hard to look at these early controlled experiments and dismiss AI as entirely hype. For reference, the productivity gains of adding steam power in the mid-1800s was about 25% for a small factory.

Now let me put that context in context: Rich countries experienced a slowdown in productivity growth after 2007, a slowdown that remains ongoing. And while it would be comforting to blame the slowdown on the one-time shock of the Global Financial Crisis or Great Recession, that doesn’t seem to be the case.