If it’s time to implement AI technology in your organization, the task can be overwhelming. Where to start? There are so many options.
Many people turn to reports by various analyst firms to find out who the top AI players are and shorten the list of choices. This makes sense, as long as you are aware of the criteria used for inclusion in the reports. For example, sometimes there requirements for marketing capabilities or global reach that must be met before an AI firm is considered for inclusion and evaluation. These non-technical requirements could eliminate some viable contenders from your consideration (even if they lack a huge marketing budget or global presence).
And here’s another consideration as you peruse the rankings: they only rate the AI platform itself. In my opinion, this is like planning a vacation up through the first day. You pick the destination, plan to get there, and coordinate activities for day one. Then…what are you going to do for the rest of the time? How do you get back home?
While the choice of the AI platform is (of course) important, it’s only one part of the overall process. In many cases, the platform will be sold, implemented, and supported (at least, in part) by another company, whether a specialized AI consultancy, systems integrator, or other type of partner. Since this firm will be the one with whom you have a direct relationship, it’s imperative to evaluate their capabilities.
When guiding clients through the process of choosing a new AI solution, roughly 50% of our effort goes into selection of the platform itself. The other 50% is spent on evaluating the partner, the implementation plan, and the ongoing support.
The “short list” reports do not take this into consideration. They can’t. There are too many options. But the competence of the implementation partner, in the end, will often make or break an AI project.
A complete AI solution requires a platform, implementation, and ongoing support. You must get all three of these right to achieve success.
How do you evaluate implementation and support capabilities for an AI project? Ask your potential partners for these 10 things that will help you evaluate their capabilities:
- A high-level implementation plan. Does it adequately address the complexity of your AI use cases? Is there evidence of a repeatable process in place for model deployment?
- The structure of the AI support team assigned to your organization. Does it contain enough machine learning engineers and data scientists? Is there someone assigned to advocate for you and ensure that your specific AI needs are met?
- An escalation list. When your AI system isn’t performing as expected, how do you raise the issue to a higher level to get results?
- A service level agreement (SLA). While tedious to read, make the time to do so. Does it contain tangible, enforceable performance metrics with financial penalties for failure to meet them? I have seen many agreements that contain aspirational accuracy goals instead of standards. Others limit the circumstances to which the SLA applies. There are many pitfalls lurking in this area, especially with AI systems.
- Clear definition of the acceptance process and your recourse if your AI models do not perform as planned.
- A list of documentation that will accompany the solution. While technical documentation is available online, there should be documentation of your specific model configurations, data pipelines, and instructions for obtaining support.
- A complete description of all training to be delivered, both for end users and for AI system administrators.
- A review of the monitoring tools to be used to track AI performance and identify potential issues.
- A deep dive on troubleshooting, model governance, and reporting capabilities for your AI systems.
- A clear understanding of all players involved and relationships between them. For example, are there third-party data providers that are part of the solution but not called out explicitly? Are there cloud infrastructure providers for certain capabilities? What are the SLAs and expectations for all providers that are part of the overall AI solution?
The time you will spend living with your AI solution far exceeds the time spent choosing it. It’s important to weigh the implementation and support part of the equation as much as the choice of the latest AI technology.
