Detecting Model Performance Deterioration in Azure Kubernetes Service | Python SDK Code Examples

How to Collect Data during Inference and Monitor Model Performance on Azure Kubernetes Service using Python SDK.

Question

You have an inference web service deployed on Azure Kubernetes Service.

In order to detect possible deterioration of your model's performance, you want to collect data during the operation.

You need to use the Python SDK and you want to add the necessary statements to your code.

... # script1 global dc_inputs, dc_predictions dc_inputs = ModelDataCollector("best_model", designation="inputs", feature_names=["f1", "f2", "f3"]) dc_predictions = ModelDataCollector("best_model", designation="predictions", feature_names=["p1", "p2"]) ------------ #script2 data = np.array(data) result = model.predict(data) dc_inputs.collect(data) #this call is saving our input data into Azure Blob dc_predictions.collect(result) #this call is saving our input data into Azure Blob ------------------ #script3 aks_config = AksWebservice.deploy_configuration(collect_model_data=True) ...
Which segment goes to which part of your code?

Answers

Explanations

Click on the arrows to vote for the correct answer

A. B. C. D.

Answer: A.

Option A is CORRECT because the ModelDatCollector objects need to be declared in the init() function, so that they can be invoked in the run()

Data collection has to be enabled outside the entry script, as part of the deployment configuration.

Option B is incorrect because data collection is not enabled by default when deploying a model to AKS.

You have to enable it so that you can collect model data, by setting collect_model_data=True.

Option C is incorrect because script1, as a variable declaration, should go to the init() function, while script2 has to go to the run().

Option D is incorrect because data collection must be enabled (script3) outside the entry script, within the deployment configuration.

Reference:

The correct answer is A. script1 to init(); script2 to run(); add script3 to deployment config.

Explanation:

The objective of the question is to identify the correct order of the given code snippets to enable collecting data during the operation of an inference web service deployed on Azure Kubernetes Service.

Script1 defines two ModelDataCollector objects, dc_inputs and dc_predictions. These objects are used to collect data and predictions, respectively. The "designation" parameter in the ModelDataCollector constructor specifies the type of data that the object is collecting, i.e., inputs or predictions. The "feature_names" parameter specifies the names of the features in the collected data.

The ModelDataCollector objects are created in the "init()" method of the inference web service. Therefore, script1 should be added to the "init()" method.

Script2 is the code that collects the data during the operation of the inference web service. The script first converts the input data into a NumPy array and then calls the "collect()" method of the dc_inputs object to save the input data into Azure Blob storage. The script then calls the "predict()" method of the model to get the prediction and calls the "collect()" method of the dc_predictions object to save the prediction into Azure Blob storage.

Therefore, script2 should be added to the "run()" method of the inference web service.

Script3 configures the deployment of the inference web service. The "AksWebservice.deploy_configuration()" method is used to create a deployment configuration object. The "collect_model_data" parameter in this method is set to "True" to enable data collection during the operation of the inference web service.

Therefore, script3 should be added to the deployment configuration code.

In summary, the correct answer is A. script1 should be added to the "init()" method; script2 should be added to the "run()" method; and script3 should be added to the deployment configuration code.