Keep Learning, Keep Growing — How To Use Azure AI for Chest X-ray Diagnosis

Original Source Here

Integrate Azure and Unity

Now we get a trained compute vision model and an API to classify the images based on the tags.

However, I want to integrate the functionality into Unity. Fortunately, Microsoft provides the Azure Custom Vision SDK in C# language. So I think it is possible to integrate them, even if I want to run it on Mac OS.

Since Unity 2018, we get a .net standard 2.0 compatibility level, which should be perfect for NuGet packages. Simply download the package using a separate VS project, then take the netstandard20 version of the DLL and place it in our Unity project. I made a .unitypackage file includes these required dlls, you can download it from here.

You can see the dlls we need in the project in the above image.

Then I will set up the C# code in our Unity project.

void Start()
var ENDPOINT = Environment.GetEnvironmentVariable("CUSTOM_VISION_ENDPOINT");
var predictionKey = Environment.GetEnvironmentVariable("CUSTOM_VISION_PREDICTION_KEY"); _prediction = new CustomVisionPredictionClient(new Microsoft.Azure.CognitiveServices.Vision.CustomVision.Prediction.ApiKeyServiceClientCredentials(predictionKey))
Endpoint = ENDPOINT

Firstly, I set up the environment and create an instance of CustomVisionPredictionClient in the Start method in Unity. As you can see in the code above, we need the endpoint of the Custom Vision resource and the key of the same resource to set up the environment in Unity.

You can find them at the setting window of your Custom Vision project site. You can see it in the following image.

If you don’t set it up correctly, you will not connect to the correct Cognitive services resource.

Then we can call the ClassifyImage method of CustomVisionPredictionClient to classify the input image.

byte[] bytes = sprite.texture.EncodeToPNG();
_testStream = new MemoryStream(bytes);
var publishedModelName = "Iteration2";
var result = _prediction.ClassifyImage(_project.Id, publishedModelName, _testStream);

You need to provide the id of the project, you can find it on your project site too, and the publishedModelName, the name of the model (Iteration) you are using. You can find this name on the performance page of your project website. The last parameter you need to provide is the data of the input image.

In order to convert a texture in Unity context to the required MemoryStream instance, we call the EncodeToPNG method of Texture to convert the texture into a byte array, then create the corresponding MemoryStream instance.

Finally, let’s add some chest x-ray images to test the functionality in Unity.

Then you can see in the video below, our Unity project can classify the chest x-ray images correctly now.

You can find the Unity Project repo here(doesn’t include the Azure part):


Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: