Pages

Monitor

Once our model is deployed, we want to make sure that it performs as well in production as it did in training. We can opt in to logging by calling logPrediction. Later on, as we get official diagnoses for patients, we can call logTrueValue and use the same identifier as we used in the call to logPrediction.

# Log the prediction. ModelFox.log_prediction(model, %ModelFox.LogPredictionArgs{ identifier: id, options: predict_options, input: input, output: output, }) # Later on, if we get an official diagnosis for the patient, log the true value. Make sure to match the `identifier`. ModelFox.log_true_value(model, %ModelFox.LogTrueValueArgs{ identifier: id, true_value: "Positive", })

// Log the prediction. err = model.LogPrediction(modelfox.LogPredictionArgs{ Identifier: id, Input: input, Options: predictOptions, Output: output, }) if err != nil { log.Fatal(err) } // Later on, if we get an official diagnosis for the patient, log the true value. Make sure to match the `identifier`. err = model.LogTrueValue(modelfox.LogTrueValueArgs{ Identifier: id, TrueValue: "Positive", }) if err != nil { log.Fatal(err) }

// Log the prediction. model.logPrediction({ identifier, input, options, output, }) // Later on, if we get an official diagnosis for the patient, log the true value. Make sure to match the `identifier`. model.logTrueValue({ identifier, trueValue: "Positive", })

// Log the predicton $model->log_prediction( identifier, $input, $output, $options, ); // Later on, if we get an official diagnosis for the patient, log the true value. Make sure to match the `identifier`. $model->log_true_value( identifier, 'Positive', );

# Log the prediction. model.log_prediction( identifier=id, input=input, output=output, options=predict_options, ) # Later on, if we get an official diagnosis for the patient, log the true value. Make sure to match the `identifier`. model.log_true_value( identifier=id, true_value="Positive", )

# Log the prediction. model.log_prediction( identifier: id, input: input, output: output, options: options ) # Later on, if we get an official diagnosis for the patient, log the true value. Make sure to match the `identifier`. model.log_true_value( identifier: id, true_value: 'Positive' )

// Log the prediction. model.log_prediction(modelfox::LogPredictionArgs { identifier: id, input, options: Some(options), output, })?; // Later on, if we get an official diagnosis for the patient, log the true value. Make sure to match the `identifier`. model.log_true_value(modelfox::LogTrueValueArgs { identifier: id, true_value: "Positive".into(), })?;

Back in the app, we can look up a prediction by its identifier, and get an explanation that shows how each feature affects the output.

Production Predictions

Positive

Predicted Class

97.48%

Probability

Now let's see how accurate our model has been in production. Let's open the app and choose Production Metrics in the sidebar.

Production Metrics

Accuracy

-4.47%

83.33%

78.87%

Training

Production

Uh oh! It's a bit lower than we expected. Let's try to find the cause. Under "Production Stats", we see that the "chest_pain" column has an alert and a high invalid values count. Click on the column to view more details.

Production Stats

Column Stats

Status	Column	Type
All good ✓	age	Number
All good ✓	gender	Enum
High Invalid Values Count !	chest_pain	Enum
All good ✓	resting_blood_pressure	Number
All good ✓	cholesterol	Number
All good ✓	fasting_blood_sugar_greater_than_120	Enum
All good ✓	resting_ecg_result	Enum
All good ✓	exercise_max_heart_rate	Number
All good ✓	exercise_induced_angina	Enum
All good ✓	exercise_st_depression	Number
All good ✓	exercise_st_slope	Enum
All good ✓	fluoroscopy_vessels_colored	Enum
All good ✓	thallium_stress_test	Enum

It looks like there is a large discrepancy between the value "asymptomatic" in production versus training. In the table below, we see a high number of invalid values with the string "asx". It looks like we are accidentally using the string "asx" in our code instead of "asymptomatic" for the chest pain column. We can update our code to use the correct value and follow the metrics going forward to confirm they bounce back.

High Invalid Values Count

chest_pain

This Month's Distribution of Unique Values for chest_pain

Training

Production

Unique Values

Value	Training Count	Production Count	Training Fraction	Production Fraction
asymptomatic	133	95	48.72%	23.11%
atypical angina	43	76	15.75%	18.49%
non-angina pain	76	122	27.84%	29.68%
typical angina	21	25	7.69%	6.08%

Invalid Values

Value	Count	Production Fraction
asx	93	22.63%

Hooray! You made it to the end! In this guide, we learned how to train a model, make predictions from our code, tune our model, and monitor it in production. If you want help using ModelFox with your own data, send us an email at [email protected] or ask a question on GitHub Discussions.

< Previous: Inspect your model.