Millions of people worldwide and the vast majority of organizations implement Agile methodologies and Agile approaches on a daily basis to manage their various development projects. And for good reason: since the early 2000s, Agile has become a best practice for tech teams looking to accelerate development and delivery while reducing the risk that comes from rapidly changing environments, evolving user requirements, and uncertain customer needs. That’s what Agile is all about: focusing on the aspects of managing complex application development projects or running technology teams. Yet, Agile is failing hard when it comes to AI. Up to 80% of AI projects are failing, even in tech-forward firms that have implemented Agile in its most advanced and deepest forms. So, does Agile work for Machine learning? And if not why not and what are the alternatives?

What are the 4 principles of agile?  Are they Relevant for AI?

Since the late 1990s, Agile methodologies have been extremely popular for a wide range of development purposes. Prior to the Agile methodology, organizations found themselves using waterfall or predictive methodologies for the application development projects which would take months or years before the latest update would be released. Waterfall approaches are good  for projects with a defined end goal such as construction projects, but software projects never really have an end. New features can always be added, and new functionality delivered. 

As such, developers and technology teams didn’t want to wait many months or even years for a software project to wind its way through design, development, testing, and deployment. Stemming from this frustration the Agile approach was created to focus on tight, short iterations (think two week sprints)  with a goal of rapidly producing a deliverable to meet immediate needs of the business owner. Teams can then continuously iterate as requirements and needs become more refined. 

The four core values of Agile software development as stated by the Agile Manifesto are:

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

Agile methodologies have forever changed the way organizations develop and release software functionality in a world where the pace of change continues to accelerate. But does Agile work for Machine learning? If the examples of AI project failure are a case in point, the answer is no. 

The reason is that Agile methodologies are focused on functionality and user needs, but AI systems are entirely dependent on data availability and data quality. Even with the most advanced and rapid Agile processes, an approach that isn’t data-forward and data-centric will constantly lose out to the realities of data in the real world. Does that mean we can’t use Agile at all? Or, can we adapt Agile methods to suit our AI and data-centric needs?

Is an Agile Approach in Machine Learning possible?

With decades of Agile experience now under our belts, we’re finding the limitations for this general approach to the agile methodology for machine learning. Agile methodologies are challenged by the requirements of data-centric AI and ML  projects. How exactly do we implement Scrum for machine learning?  And, the nature of a ML project structure is different from traditional software development projects. 

For one, what exactly is being “delivered” in an AI or ML project? You can say that the machine learning model is a deliverable. However it’s actually just an enabler of a deliverable, not providing any functionality in and of itself. In addition, if you dig deeper into machine learning models, what exactly is in the model? The model consists of algorithmic code plus training model data, the parameters trained by that training data, hyperparameter configuration data, and additional support logic and code that together comprise the model. 

Even that doesn’t really define what the AI system is doing. Yes, you can use the same algorithm with different training data and generate a different model, and you can use a different algorithm with the same training data and that would also generate a different model. So then is the deliverable the algorithm, the training data, the model that aggregates them, the code that uses the model for a particular application, all of the above, none of the above? The answer is yes. Huh? How can it be all of that and none of that at the same time? That’s because it depends on what you want your deliverable to be. Therefore, we need to consider additional approaches to augment Agile in ways that make it more AI-relevant to have an agile methodology for machine learning and AI projects.

Rather than reinvent the wheel, we can stand on the shoulders of giants. Agile forms a basis for an AI-specific methodology but can’t be used solely on its own. We need more guidance to help us figure out how to run projects where the data tells us how the system will perform, and where the resulting system will behave in different ways depending on the environment.

How do you develop a ML project?

So if using Agile for AI doesn’t work on its own, then what can ML project managers, owners, and leaders use when managing their AI and advanced data projects? The answer is to leverage a data-focused approach that is also iterative and agile. The data-centric ML project management approach that is fast becoming a best practice approach used by thousands of project managers is the  Cognitive Project Management for AI (CPMAI) methodology. 

The CPMAI methodology is a vendor-neutral, data-centric, AI-specific, iterative methodology for running and managing AI, ML, and advanced data projects. Far too often, organizations want to be iterative and agile but deploy a waterfall model for machine learning. CPMAI makes needed enhancements to existing methodologies to meet AI-specific requirements that allow you to develop a model quickly. As we like to say “Think Big. Start Small. Iterate Often.” By extending and enhancing these current methodologies rather than creating a new approach, CPMAI can be implemented immediately at organizations with already-running Agile teams and already-running data projects. This allows us to take into account machine learning in project management, and therefore allows us to say, with these modifications, Agile does work for Machine Learning.

One challenge with implementing CPMAI is that introducing something new and foreign can create internal resistance. So the key is to provide a trusted approach that simultaneously delivers the expected results to the organization, using familiar and approachable terminology and concepts, and provides an approach for continued iterative development at the lowest risk possible. Because, at the end of the day, successfully running and managing an AI project with an appropriate machine learning project plan should be everyone’s goal.

The best way to get up to speed on CPMAI methodology is to take the machine learning project management course on CPMAI offered by Cognilytica. This training and certification is part of the Project management courses and Project management training that Cognilytica has provided since 2017 to thousands of leading organizations. CPMAI builds up on Agile methodologies and CRISP-DM methods to provide the agile data certification that many organizations desire.Interested in learning more about CPMAI? Interested in adding an additional Project management certification to your resume? Sign up for CPMAI training and certification.