How the Right Data Engineer Could Help an Enterprise AI Project Succeed – Ranjit Nair

In Artificial Intelligence, Columns, Technology

The hype around AI is justified by its transformative potential for organizations of all sizes right from SMEs to large enterprises to governments. While its ability to automate and hence reduce costs is well understood, AI’s real potential comes from being able to increase the top line through enabling innovation and improving employee efficiency. According to Gartner, 70 percent of organizations will integrate AI to assist employees’ productivity by 2021. And more and more companies are using AI to shorten the innovation cycle, for instance in drug discovery.

And yet, according to Gartner, 85 percent of AI projects won’t deliver for CIOs. That means, out of 20 artificial intelligence projects, only three will succeed, and 17 of them will fall short. In order to understand what it takes to be among the three that succeed, let’s understand why AI projects often fail and also understand what unifies the AI projects that succeed.

The most common reason for a failure is that AI projects are often poorly conceived. Their adoption may have driven by FOMO or the Fear Of Missing Out and lead to companies simply following the herd. In such cases, the necessary investment of intellectual bandwidth to clearly defining the objectives and desired outcomes of the AI project is not made. What then happens is that there is misalignment between what is being asked of the AI and the actual the business problem.

The other common reasons for failure are not having the right data to train the AI system, poor communication between business teams and the technical teams, setting incorrect expectations, building the wrong training model and using the wrong algorithms.

All of these reasons can be attributed to not having a good data engineering team in place. Enterprises need to heavily focus on hiring and empowering data scientists to create advanced analytics to ensure that they are one of the three that succeed.

A good data engineering team is one that can bridge the gap between business and data. A common refrain from companies whose AI projects have failed is that the data engineers speak geek and not business. They talk using jargon and terminology that is intimidating to the business folks leading to communication gaps and misaligned expectations.

Goals need to be specified in terms of deliverables that will be done in a phase-wise manner so that course corrections that will be inevitably needed can be done early.

Data collection is often a huge pitfall. The required data for the AI project might not be digitized and even if available in a digital format might require a lot of pre-processing before it is fit to be used for training by an AI system. A good data engineer will be able to identify what data is required, where exists, how to collect it and how to prepare it for training.

The choice of AI training models and algorithms can also have a huge impact on the success of AI projects. This is where an experienced data engineer who has seen several projects before comes in handy. It is much better to rely on someone who knows the common pitfalls and what is likely to work for a particular problem.  

The huge potential of AI has meant that AI projects are on the radar of CIOs of organizations of all sizes. But for these AI projects to actually succeed it needs the right data engineering team to be in place to deliver on the promise of AI.

(The author is CEO and Founder of Germin8, a Social Media Intelligence company focused on helping to understand and act in real-time on the gazillions of conversations by the stakeholders.)

You may also read!

Rohan Mujoo, A Tech Graduate Exploring The Engine To Drive Innovation!

The IncubateIND's India Innovation Series is the largest hackathon series being organized in 20+ cities in India aiming to

Read More...

“The City Beautiful” Chandigarh! With Its Remarkable Start-Ups

Chandigarh, the beautiful city, the first planned city of India situated on the foothills of Shivalik, is a union

Read More...

Ved Prakash, An Android Developer Learning To Create And Innovate!

To solve some of the major predicaments continuously growing in our society and to provide a technical solution to

Read More...

Leave a reply:

Your email address will not be published.

Mobile Sliding Menu