Proving GitHub Copilot's Impact at Enterprise Scale
ANZ Bank · 2021–2024 · Published research (arXiv:2402.05636)
Problem: ANZ Bank, with an engineering workforce of 5,000+, wanted rigorous, data-driven evidence — not just vendor claims — on whether an AI pair-programming tool actually improved developer productivity, code quality and security, before rolling it out broadly in a regulated banking environment with real IP and data-security concerns.
Approach: Designed and ran a six-week controlled experiment across 100+ engineers split into Control and Copilot groups, using A/B testing methodology — two weeks of baseline preparation followed by four weeks of active testing with weekly coding challenges. Measured results using GitHub activity metrics, participant surveys, SonarQube static analysis and manual grading, then analysed the data with rigorous non-parametric statistical testing (Wilcoxon Signed Rank Test at 95% confidence). Co-authored the findings into a formal research paper alongside ANZ Bank's Chief Technology Officer and other senior engineering leaders.
Outcome: Found a 42% average improvement in task completion time (rising to 52% for less experienced engineers), alongside statistically significant reductions in bugs and code smells. The findings directly informed ANZ's decision to move to a production rollout across 1,000+ engineers, and the work earned personal appreciation from ANZ Bank's Chief Technology Officer.
Read the published paper (arXiv:2402.05636) ↗