Effective Method Steps for Selecting the Correct Model
Question
You work as a machine learning specialist for a car manufacturer that has developed driverless technology for their new line of cars.
These cars require real-time machine learning models to perform all of the tasks of driving.
You have trained multiple models, using different algorithms and/or different hyperparameters, as candidates to assist in lane line crossover detection using live data from sensors on the undercarriage of the car.
You want to select one of these models as the model to go to production in the line of cars. Using the various options available from SageMaker, which are the most effective method steps you should use to select the correct model? (Select TWO)
Answers
Explanations
Click on the arrows to vote for the correct answer
A. B. C. D. E.Answers: C, E.
Option A is incorrect.
For online testing, you use live data.
For offline testing, you use historical data.
Option B is incorrect.
When performing offline testing of your models, you deploy your trained models to alpha endpoints, not beta endpoints.
Option C is correct.
For online testing, you use live data.
Testing with live data will allow you to perform the steps listed in option.
E.Option D is incorrect.
To use online testing, you deploy your models to a SageMaker endpoint, not a SageMaker training instance.
Option E is correct.
To perform online testing of your models, you deploy the models to a SageMaker endpoint and then send a portion of the data to each model (or production variant), allowing you to evaluate the models.
Reference:
Please see the SageMaker developer guide titled Validate a Machine Learning Model.
To select the correct model for lane line crossover detection using live data from sensors on the undercarriage of the car, you need to evaluate the performance of multiple models. AWS SageMaker provides several options to do that. Two of the most effective methods are:
B. Deploy your trained models to beta endpoints, then using a Jupyter notebook in your SageMaker instance, send inference requests to each model, in turn, using the AWS SDK for Python or the SageMaker high-level Python library and finally evaluate each model.
This method involves deploying the trained models to beta endpoints to create inference endpoints that can process live data. Then you can use a Jupyter notebook in your SageMaker instance to send inference requests to each model, one at a time, using the AWS SDK for Python or the SageMaker high-level Python library. Once you have the inference results, you can evaluate each model's performance using metrics such as accuracy, precision, recall, F1-score, and others. This method allows you to compare the performance of multiple models and select the best one for production.
E. Deploy your models to a SageMaker endpoint, then send a portion of the live data to each model and finally evaluate each model.
This method involves deploying the trained models to a SageMaker endpoint to create inference endpoints that can process live data. Then, you can send a portion of the live data to each model and evaluate each model's performance. This method allows you to simulate the production environment and evaluate each model's performance on a portion of the live data, enabling you to select the best model for production.
Therefore, options B and E are the most effective method steps to use to select the correct model. Options A and C both refer to online testing, which is useful for evaluating a model's performance over time but not for selecting the best model. Option D involves retraining the models on a portion of the live data, which can be time-consuming and may not be necessary if the models are already trained on representative data.
