To Scrum, or to Kanban, that is the question for Data Science Teams
When companies decide to start a data science team, managers are often confused about how to best manage them due to the extensive research work involved and the lack of detailed goals and estimations. As an AI & Data Product manager, I had the opportunity to experiment with both Scrum and Kanban methodologies, compare their results, and select my favorite.
Before I happened upon the team two years ago, they didn’t have anything but a Kanban board that was not managed ideally. I already had experience with Scrum in mixed teams (data science + engineers), so I decided to find a way to make my beloved team happier.
What is the major difference between Scrum and Kanban?
Kanban is more suitable for research projects that are continuous, unpredictable, and don’t have specific deadlines. In contrast, Scrum is designed to deliver an increment and achieve a sprint goal in every 2-week iteration, providing a more structured approach with set deadlines..
At first sight, you can say, yes, please, Kanban is the best fit! I initially thought it would do a great job. However, I ended up with a list of unhappy stakeholders and demotivated engineers.
Kanban seemed nice and reasonable at first. We were creating projects, conducting our research, and feeling pretty chill. But then one day, I realized we had doubled the expected project duration and faced issues syncing with other engineering teams. The data science team wasn’t just doing endless research; they were also improving existing methodologies, providing new ones, and checking implementations. But Kanban left them unsynchronized and isolated from the outside world.
As a result, we lost sight of our overall product timeline and the predictability of our deliveries. I’m not denying the possibility that we were doing something wrong, but that’s not great news for the business, right?
With this in mind, we decided to give Scrum a try, believing it would suit our engineering-oriented approach. :) We learned how to plan projects in detail, split initiatives, and properly estimate even 100% research projects. Of course, we implemented all the necessary Scrum rituals. And, oh my god, it worked! Now, new feature implementations are ✨70%✨ accurate in meeting deadlines, which seems to a good progress!
It worked for us! As a result, we learned to deliver updates and bad news iteratively and on time. 🫡 Stakeholders are happy because they receive results promptly, and the data science team is pleased as well — they finally see their efforts recognized and have a transparent workflow. And I’m happy too because when everyone else is happy, it makes my job easier! 😬
5 steps in how we managed research initiatives within the Scrum framework
- Describe Your Project in Detail
Clearly outline what you expect to achieve from the project. From a product perspective, detail how the project should work and what steps are necessary to reach these results. Identify the hypotheses you need to test and how you plan to do so. - Create an MVP
Consider whether you can create a Minimum Viable Product (MVP) and think about Minimum Viable Data (MVD) to test your hypotheses, which can help minimize research time. - Divide into Steps
Split the project into several key steps. At the end of each step, you should have the opportunity to decide whether to proceed or to halt the project. It’s a good thing to stop on time. 💸 - Estimate Each Part
Provide estimates for each part of the project, and be mindful of potential risks. For science projects, it’s acceptable not to specify exact delivery dates, but you should share expectations in terms of time intervals. - Master the Sprint
When planning sprint content, remember to split tasks into smaller, manageable pieces. Additionally, set a clear sprint aim. While it may not be monumental, it should carry significant meaning for the team.
These are all my observations for now! I hope this gives you something to think about, and I would be happy if someone disagrees with me and proves me wrong. 🤭 Merci!
