Why Big Data Projects Fail and How to Ensure Success
The failure rates of big data projects are staggering. A NewVantage Partners study found that 77% of businesses report that “business adoption” of big data and AI initiatives is a challenge. Gartner’s 2019 report states, “through 2020, 80% of AI projects will remain alchemy, run by wizards whose talents will not scale in the organization.”
These numbers show that while organizations may want to launch AI or ML projects or have even started to build them, moving into production and getting value from the projects is not happening.
Since so many of these projects don’t get off the ground, it’s time to take a look at why. We’ve compiled our top reasons why big data projects fail, and how companies can have a better chance at success.
Reasons why big data projects fail
There are multiple reasons for failure. For big data projects, there are common occurrences that continually stagnate project success.
Misalignment of technical realities and business assumptions
Often, the technical capabilities do not match the business expectations. Executives may want the technology to have specific functions, but artificial intelligence (AI) and machine learning (ML) can only do so much. While the technology has come very far in recent years, projects may fail if both sides of the company do not understand what the project will actually do -- and what it will not do.
Lack of infrastructure and resources
Leaders make lofty goals with AI and ML in order to maintain competitive advantage and propel the business forward. Big data projects require robust infrastructure and key resources, especially concerning talent. Ernst & Young found and 56% of senior AI professionals say a lack of talent is the greatest barrier to implementation. LinkedIn found that numerous machine learning jobs top the fastest-growing jobs list, growing at exponential rates. AI and ML specialists are in very high demand, and companies that try to push forward without these crucial roles filled usually find their projects halted.
Inflexible project architectures
Some businesses have the resources, skill, and infrastructure to implement a successful big data project, but it still fails. In these cases, it may be because the project scope is too rigid, not allowing for any flexibility once the project starts. Furthermore, many companies wait for the perfect model in the beginning instead of making incremental improvements in the business processes. Using ML even on a subset of the data or on one dimension of the problem can still provide significant business value and reduce risk.
Unrealistic goals
The business side of the equation may make assumptions about the technology that simply cannot be met. Often, the implications of AI technology are more robust than the practicality of the technology, so business leaders set very high expectations of projects.
Models are not moved to production
Probably the most common reason why big data projects fail is that they just cannot move into production. Highly skilled specialists create ML models but then months go by with nothing happening. Often, a company’s IT environments are not equipped to handle the ML models, and the IT team on-hand does not know what to do with them.
How to make a big data project a success
AI technology offers a clear competitive edge, but companies need to take a step back before they create models that never move into production.
Plan strategically
This may seem obvious, but countless companies charge into AI projects just to say that they’re using the technology. Companies need to ensure that the right infrastructure is in place to support the technology and that the skills and resources are there. Furthermore, businesses should ensure that AI projects are aligned to and related to business priorities. The project should either increase the bottom line, decrease operational expenses, or improve customer or employee experiences.
Understand the problem
As part of the planning process, key players must fundamentally understand the business problem they want to solve. What value will this project produce? And, are the goals realistic enough and approved by the technical and business side? What happens often is the business, IT, and data science teams each have a different goal and understanding of the end result. If these three teams can agree, then the project has a better chance of succeeding. Companies should always “size the prize,” and understand whether or not it will deliver business value -- just because it is an interesting data science initiative doesn’t mean it will warrant the investment.
Engage and assess
Organizations should engage partners early on and throughout the entire process. Don’t wait until the end of the project, when a key partner may want massive changes. Assess what the level of business, technical, data, and analytic readiness is necessary to take the project from idea to production. Without this assessment, the project will likely hit snags along the way.
Document any failure
As with any business, failure happens. Sometimes called the “fail forward” strategy, documenting what went wrong, when, and why can help companies avoid making the same mistakes over and over again. After a failed big data project happens, the IT, data science, and business side should understand why it happened and document updated processes to ensure mistakes don’t happen again.
As more organizations turn to AI to help improve business processes, maximize resources, and increase company growth, the more important it becomes to make big data projects successful. A cohesive team of data scientists, IT, and business leaders that all understand the project fully - and the expected results - have a much better chance at a favorable outcome.
For companies that find themselves unable to move their models into production (whether from a technical standpoint or that it takes too much investment and time), find out how we can help at Quickpath. Our platform helps companies take their machine learning models and integrate it with business applications, which means much more business value from data science practice.