Why Can’t I use Agile or CRISP-DM to manage AI and Data Projects?

Millions of people around the globe at organizations and vendors of all sizes certified on leading project management methodologies and approaches such as PMP, PRINCE2, Agile, ScaledAgile (SAFe), and CRISP-DM. Likewise, organizations are rushing to implement and adopt AI. Existing data-centric project management approaches such as CRISP-DM have been around for decades. However, AI project failure rates are very high, with over 70% of AI projects failing according to some statistics. WIth all these existing project management approaches in place, why can’t we use any of the existing project management approaches to achieve better success rates? Why can’t I use Agile or CRISP-DM to manage AI and Data projects?

Why can’t we use existing Project Management Approaches for AI Projects?

Many of these highly adopted, but decades-old (and possibly outdated or too general) project management methodologies, such as the Project Management Institute (PMI) Project Management Professional (PMP) certification are generally-focused approaches for managing projects of a wide range of types. While these general approaches might work for general project management in industries such as construction, legal, IT, healthcare, and other industries, once you try to apply this approach to technology projects you’ll find it’s quickly missing some core and fundamental elements to make technology projects successful. 

Other approaches such as Agile, ScaledAgile, or SCRUM are more focused on the aspects of managing complex application development projects or running technology teams and are not focused on the specific aspects of data-driven projects. While the Cross Industry Standard Process for Data Mining (CRISP-DM) has been around for over two decades, it is no longer maintained, hasn’t been updated to deal with AI projects, and does not take an iterative perspective on data projects. 

The problem of irrelevant or out-dated project management approaches is especially a problem when it comes to AI and advanced data projects. Existing non-AI specific, non-data specific project management approaches lack specific guidance on how to successfully run and manage AI projects, leaving project leaders to figure it out on their own.  This explains in part why you can’t use Agile or CRISP-DM to manage AI and data projects, but let’s dive a bit deeper.

Why can’t we use Agile for AI?

Agile methodologies are extremely popular for a wide range of application development purposes, and for good reason. Prior to the widespread adoption of Agile methodology project Management approaches, organizations found themselves bogged down by traditional waterfall or predictive methodologies for the application development projects which borrowed too much from assembly line methods of production. Rather than wait many months or even years for a software project to wind its way through design, development, testing, and deployment, the Agile approach focuses on tight, short iterations with a goal of rapidly producing a deliverable to meet immediate needs of the business owner, and then continuously iterating as requirements and needs become more refined. 

First popularized by the Agile Manifesto, Agile emphasizes focusing on individuals and interactions over strict processes and tools, delivery of working products over a focus on planning and heavy documentation, continuous customer collaboration versus a drawn out contract negotiation process, and a focus on responding to change rather than strict adherence to a plan. There is no doubt that Agile methodologies have changed the way organizations develop and release software functionality in a world where the pace of change continues to accelerate.

However, with decades of Agile experience under our belts, we’re finding the limitations for trying to implement AI in an agile project management manner. Agile methodologies are challenged by the requirements of AI systems. For one, what exactly is being “delivered” in an AI project? You can say that the machine learning model is a deliverable, but it’s actually just an enabler of a deliverable, not providing any functionality in and of itself. In addition, if you dig deeper into machine learning models, what exactly is in the model? The model consists of algorithmic code plus training model data, the parameters trained by that training data, hyperparameter configuration data, and additional support logic and code that together comprise the model. 

Even that doesn’t really define what the AI system is doing. Yes, you can use the same algorithm with different training data and generate a different model, and you can use a different algorithm with the same training data and that would also generate a different model. So then is the deliverable the algorithm, the training data, the model that aggregates them, the code that uses the model for a particular application, all of the above, none of the above? The answer is yes. Huh? How can it be all of that and none of that at the same time? That’s because it depends on what you want your deliverable to be. Therefore, we need to consider additional approaches to augment Agile in ways that make it more AI-relevant to have an agile methodology for machine learning and AI projects.

Agile forms a good set of practices for an AI methodology but can’t be used solely on its own. We need more guidance to help us figure out how to run projects where the data tells us how the system will perform, and where the resulting system will behave in different ways depending on the environment.

Why can’t we use CRISP-DM for AI Projects?

Before this most recent wave of AI and machine learning interest and hype which started in the 2010s, organizations with data-centric project needs adopted methodologies that focused on their data-centric needs. With roots in data mining and data analytics, some of these earlier methodologies had at their core an iterative cycle focused on data discovery, preparation, modeling, evaluation, and delivery. One of the earliest of these developed is simply known as Knowledge Discovery in Databases (KDD). However, just like with waterfall methodologies, KDD is in some ways too rigid or abstract to deal with continuously evolving models and the needs of AI and advanced data projects.

Responding to the needs for a more iterative approach to data mining and analytics, a consortium of five vendors developed the Cross-industry Standard Process for Data Mining (CRISP-DM) focused on a continuous iteration approach to the various data intensive steps in a data mining project. Specifically, the methodology starts with an iterative loop between business understanding and data understanding, and then a handoff to an iterative loop between data preparation and data modeling, which then gets passed to an evaluation phase, which splits its results to deployment and back to the business understanding. The whole approach is developed in a cyclic iterative loop, which leads to continuous data modeling, preparation, and evaluation as illustrated below:

Source: CRISP-DM 1.0

However, CRISP-DM v1.0 was released almost two decades ago and there was no further development of the CRISP-DM methodology. There were rumors that a second version was underway almost fifteen years ago but further development and versions were never published. IBM and Microsoft have both iterated on the methodology to produce their own variants that add more detail with respect to more iterative loops between data processing and modeling and more specifics around artifacts and deliverables produced during the process. However, both companies are primarily leveraging their modifications in the context of delivering their own premium service engagements or as part of product-centric implementation processes. Clearly vendor-centric, proprietary methodologies can’t be adopted by organizations that have diverse technology needs or desire to utilize vendor-neutral approaches to technology implementation.

A main challenge to making CRISP-DM work is in the context of existing Agile methodologies. From the perspective of Agile, the entire CRISP-DM loop is contained within the development and deployment spheres. It also touches upon the business requirements and testing portions of the Agile loop as well. Indeed, if we bring Agile into the picture, these two independent cycles of application-focused agile AI development and data-focused data methodologies are intertwined in complex ways.

CPMAI: Best Practices approach to Manage AI and Data Projects

So, why can’t I use Agile or CRISP-DM to manage AI and data projects? If using Agile for AI doesn’t work on its own, and CRISP-DM hasn’t evolved in the past 20 years and presents challenges to the ways modern terms work, then what can project managers, owners, and leaders use when managing their AI and advanced data projects? The answer is a methodology that starts from the same root of business requirements as CRISP-DM and splits into two simultaneous iterative loops of Agile project development and Agile-enabled data methodologies. We can think of this as an Agile and iterative data-centric project management approach. This step by step approach is the Cognitive Project Management for AI (CPMAI) methodology.

The CPMAI methodology is a vendor-neutral, data-centric, AI-specific, iterative methodology for running and managing AI, ML, and advanced data projects. Far too often, organizations want to be iterative and agile but deploy a waterfall model for machine learning. CPMAI makes needed enhancements to Agile and CRISP-DM methodologies to meet AI-specific requirements that allow you to develop a model quickly. As we like to say “Think Big. Start Small. Iterate Often.” By extending and enhancing these current methodologies rather than creating a new approach, CPMAI can be implemented immediately at organizations with already-running Agile teams and already-running data projects. 

As we have seen at many organizations, introducing something new and foreign can create instant resistance. So the key is to provide a blended approach that simultaneously delivers the expected results to the organization, using familiar and approachable terminology and concepts, and provides an approach for continued iterative development at the lowest risk possible. Because, at the end of the day, successfully running and managing an AI project with an appropriate AI project methodology  should be everyone’s goal.Interested in learning more about CPMAI? Sign up for CPMAI training and certification.