Original Source Here
Questions To Ask During An AI Model Demo
AI is booming as the technology behind it is advancing at alarming rates. More and more companies are adopting AI and utilizing it to improve the services they provide for both consumers and businesses. I’ve personally been working with a variety of AI models at the company I’m working at with some of the best software developers in South Korea.
The standard way AI models in academic settings are presented is a speaker comes out, introduces the model they have created, wow the listeners with the awesome stuff it can do, and that’s it. The end. But when we consider applying these models to real services, things need to get much more serious. A 5% error might lead to tens of thousands of dissatisfied users depending on how many people uses your service. That’s why it’s vital to know what questions you should ask to properly assess the quality of the model. Here’s a list of questions to help you understand what the model is capable of.
- Can you share some examples where the results didn’t come out as expected and explain why?
Don’t be fooled by the examples shown during the presentation of the model. The presenter will show examples where the model spits out perfect results and you might be fooled to believe this model is simply perfect. But keep in mind, AI models are never perfect. Just like humans beings, AI models have their flaws and it’s vital for you to be aware of just what flaws exist and how critical they are.
For example, let’s say an NLP model is being developed that can be used to detect foul language. Ask for examples where the model detected foul language when there was no trace of it at all. Then ask why this result was created. You can learn the limitations of the model and why such errors were created.
2. How did you accumulate the training data set?
In the current atmosphere of AI, most models are being created by big tech companies such as Google and Facebook, and other smaller companies are simply adopting these models and feeding it different groups of training sets. Thus, these training sets are of vital importance for proper results to be created.
If you ask how exactly the training data set was created, you can learn the quality of the model as a whole and also how much effort was put in to create the model. For example, if you are trying to create a model that can detect the inside and outside of restaurants, but the training data set is comprised of the inside and outside of all buildings, not just restaurants, it is most likely your model will have various flaws. In addition, if it took over a month and $10,000 dollars to accumulate this data set, you should calculate the ROI of the model and decide whether or not to continue this project.
3. How do you plan to improve the model in the future?
As I stated above, no AI model is perfect. So you need to have a plan to continuously improve it in the future. Think of it like this. If you have a 10% error rate, this may not be too bad. It means out of every 10 people who use your service, 1 will be met with an error. But does your service only have 10 users? Probably not. Blow this up to 1 million, and 100 thousand users will be met with an error continuously. Even if it might take some time, you need a plan to improve the model and minimize the error rate in the future. For example, you might have a plan to improve your model by increasing the quality of your training set or implement a newer, more powerful model that has been recently introduced. Either way, a proper plan of initiation should be presented.
Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot