model_card_toolkit.model_card_toolkit¶
Model Card Toolkit.
The Model Card Toolkit (MCT) provides a set of utilities to generate Model Cards from trained models, evaluations, and datasets in ML pipelines.
ModelCardToolkit Objects¶
class ModelCardToolkit()
ModelCardToolkit provides utilities to generate a ModelCard.
ModelCardToolkit is a tool for ML practitioners to create Model Cards, documentation for model information such as owners, use cases, training and evaluation data, performance, etc. A Model Card document can be displayed in output formats including HTML, Markdown, etc.
The ModelCardToolkit includes an API designed for a human-in-the-loop process to elaborate the ModelCard.
The ModelCardToolkit organizes the ModelCard assets (e.g., structured data, plots, and UI templates) in a user-specified directory, and updates them incrementally via its API.
Example usage:
import model_card_toolkit
# Initialize the Model Card Toolkit with a path to store generate assets
model_card_output_path = ...
mct = model_card_toolkit.ModelCardToolkit(model_card_output_path)
# Initialize the ModelCard, which can be freely populated
model_card = mct.scaffold_assets()
model_card.model_details.name = 'My Model'
# Write the model card data to a proto file
mct.update_model_card(model_card)
# Return the model card document as an HTML page
html = mct.export_format()
__init__¶
def __init__(output_dir: Optional[str] = None, file_name: Optional[str] = None)
Initializes the ModelCardToolkit.
This function does not generate any assets by itself. Use the other API functions to generate Model Card assets. See class-level documentation for example usage.
Arguments:
output_dir
- The path where MCT assets (such as data files and model cards) are written to. If not provided, a temp directory is used.file_name
- file name of the model card proto file. Defaults to model_card.proto
scaffold_assets¶
def scaffold_assets(path: Optional[str] = None, proto: Optional[message.Message] = None) -> ModelCard
Generates the Model Card Tookit assets.
If a path to an existing model card proto object is provided, it will be copied over as the base card instead of initializing a new one.
Alternatively, if an existing proto object is provided, it will be copied over as the base card.
Assets include the ModelCard proto file, Model Card document, and jinja
template. These are written to the output_dir
declared at
initialization.
Arguments:
path
- The path to model card proto.model_card
- The ModelCard object.
Returns:
A ModelCard representing the given model.
Raises:
FileNotFoundError
- on failure to copy the template files.
update_model_card¶
def update_model_card(model_card: ModelCard) -> None
Updates the Proto file in the MCT assets directory.
Arguments:
model_card
- The updated model card that users want to write back.
Raises:
Error
- when the given model_card is invalid w.r.t. the schema.
export_format¶
def export_format(model_card: Optional[ModelCard] = None, template_path: Optional[str] = None, output_file=_DEFAULT_MODEL_CARD_FILE_NAME) -> str
Generates a model card document based on the MCT assets.
The model card document is both returned by this function, as well as saved to output_file.
Arguments:
model_card
- The ModelCard object, generated fromscaffold_assets()
. If not provided, it will be read from the ModelCard proto file in the assets directory.template_path
- The file path of the Jinja template. If not provided, the default template will be used.output_file
- The file name of the generated model card. If not provided, the default 'model_card.html' will be used. If the file already exists, then it will be overwritten.
Returns:
The model card file content.
Raises:
MCTError
- Ifexport_format
is called beforescaffold_assets
has generated model card assets.
export_test_results_json¶
def export_test_results_json(model_card: ModelCard, output_file=_DEFAULT_MODEL_CARD_RESULTS_FILE_NAME) -> Dict
Generates a document containing model card test results.
The model card result document is both returned by this function, as well as saved to output_file.
Arguments:
model_card
- The ModelCard object, generated fromscaffold_assets()
. If not provided, it will be read from the ModelCard proto file in the assets directory.output_file
- The file name of the generated model card. If not provided, the default 'model_card_results.json' will be used. If the file already exists, then it will be overwritten.
Returns:
The model card file content.
Raises:
MCTError
- Ifexport_test_results_json
is called beforescaffold_assets
has generated model card assets.
group_reports¶
@staticmethod
def group_reports(reports) -> Dict[str, List]
Given a list of model card reports, group them into those with the same type+slice.
Arguments:
reports
- ModelCard reports.
Returns:
Dict of {type+slice: {set of reports with this type+slice}}
find_common_reports¶
@staticmethod
def find_common_reports(reports_a: List, reports_b: List) -> List[Tuple]
Given 2 lists of model card reports, find all reports that have the same type and slice in both lists.
Arguments:
reports_a
- List of ModelCard report.reports_b
- List of ModelCard report.
Returns:
List of [(report A, report B), ...]. The list of (report A, report B) tuples is a cartesian product between reports in A vs reports in B with the same type+slice.
compare_model_cards¶
def compare_model_cards(card_a: ModelCard, card_b: ModelCard, export_path: Optional[str] = None) -> str
Compare reports across given model cards A and B and render them side by side. Only reports that have the same type and slice will be compared.
Arguments:
card_a
- ModelCard for comparison.card_b
- ModelCard for comparison.export_path
- Optional path to export comparison report
Returns:
HTML report of the model card differences.