pip install moonwatcherWarning
The demos require wget to be installed on your system.
In the demo the performance of a model on unusual values for brightness, contrast and saturation of the underlying dataset are checked. To see how to create your own specific test scenarios check out Quickstart.
Object detection (the demo will download the val2017 set of COCO and use a subset of it):
python -m moonwatcher.demo_detectionClassification (the demo will download STL-10 as a dataset):
python -m moonwatcher.demo_classification- 🏃♀️ Quickstart
- 1. 🧑🏫 Slices, Checks and Checksuites
- 🍰 Slices
- ✅ Checks
- 📄 Checksuites
- 2. 🤖 Run automated checks
- 3. 👨💻 Write custom checks and checksuites
- 1. 🧑🏫 Slices, Checks and Checksuites
- 🖥️ Web app
Look into the relevant demo (demo_classification.py or demo_detection.py) to see how to create the MoonwatcherModel and MoonwatcherDataset from your data.
from moonwatcher.check import automated_checking from moonwatcher.model.model import MoonwatcherModel from moonwatcher.dataset.dataset import MoonwatcherDataset # Your model (your_model) and dataset (your_dataset) loading somewhere # Look into the relevant demo (demo_classification.py or demo_detection.py) # to see how to create the MoonwatcherModel and MoonwatcherDataset from your data. mw_model = MoonwatcherModel( model=your_model, ... ) mw_dataset = MoonwatcherDataset( dataset=your_dataset, ... ) automated_checking(model=mw_model, dataset=mw_dataset) Writing a custom check works like this.
from moonwatcher.check import Check accuracy_check = Check( name="AccuracyCheck", dataset_or_slice=mw_dataset, metric="Accuracy", operator=">", value=0.8, ) # and run it on your model: check_result = accuracy_check(mw_model)Tip
You can also slice your dataset and use a slice for the check instead of the whole dataset.
Tip
Class/category based checking is not yet supported, but will be part of the next iteration.
Now adding another check and combining both into a checksuite
from moonwatcher.check import Check, CheckSuite precision_check = Check( name="PrecisionCheck", dataset_or_slice=mw_dataset, metric="Precision", operator=">", value=0.8, ) # Combine them into a checksuite first_checksuite = CheckSuite( name="AllChecks", checks=[accuracy_check, precision_check] ) # and run it on your model: checksuite_result = first_checksuite(mw_model)⭐️ Don’t forget to star the project if you want to support open source testing of ML models.
That's it. Have fun! 🌚