Rapid advances in artificial intelligence (AI) have fueled high expectations for the technology’s potential to fundamentally transform our economy and society through automation. However, given the inscrutability and, sometimes, susceptibility to error of AI systems, we argue that the focus should shift towards fostering effective human-AI collaboration rather than pursuing automation alone. In this context, system decisions must be made available to decision-makers in an explainable and understandable manner, as further required by the EU’s recently passed AI Act. Research shows that there is potential for humans to learn from explainable AI systems and improve their own performance over time. Meanwhile, in addition to enabling humans to benefit from working with AI systems on various everyday tasks, such collaboration ensures the safe and reliable use of AI systems, especially in high-risk areas such as medicine, where human oversight remains paramount.