Skip to content

model_card_toolkit.model_card_toolkit

Model Card Toolkit.

The Model Card Toolkit (MCT) provides a set of utilities to generate Model Cards from trained models, evaluations, and datasets in ML pipelines.

ModelCardToolkit Objects

class ModelCardToolkit()

ModelCardToolkit provides utilities to generate a ModelCard.

ModelCardToolkit is a tool for ML practitioners to create Model Cards, documentation for model information such as owners, use cases, training and evaluation data, performance, etc. A Model Card document can be displayed in output formats including HTML, Markdown, etc.

The ModelCardToolkit includes an API designed for a human-in-the-loop process to elaborate the ModelCard.

The ModelCardToolkit organizes the ModelCard assets (e.g., structured data, plots, and UI templates) in a user-specified directory, and updates them incrementally via its API.

Example usage:

import model_card_toolkit

# Initialize the Model Card Toolkit with a path to store generate assets
model_card_output_path = ...
mct = model_card_toolkit.ModelCardToolkit(model_card_output_path)

# Initialize the ModelCard, which can be freely populated
model_card = mct.scaffold_assets()
model_card.model_details.name = 'My Model'

# Write the model card data to a proto file
mct.update_model_card(model_card)

# Return the model card document as an HTML page
html = mct.export_format()

__init__

def __init__(output_dir: Optional[str] = None, file_name: Optional[str] = None)

Initializes the ModelCardToolkit.

This function does not generate any assets by itself. Use the other API functions to generate Model Card assets. See class-level documentation for example usage.

Arguments:

  • output_dir - The path where MCT assets (such as data files and model cards) are written to. If not provided, a temp directory is used.
  • file_name - file name of the model card proto file. Defaults to model_card.proto

scaffold_assets

def scaffold_assets(path: Optional[str] = None, proto: Optional[message.Message] = None) -> ModelCard

Generates the Model Card Tookit assets.

If a path to an existing model card proto object is provided, it will be copied over as the base card instead of initializing a new one.

Alternatively, if an existing proto object is provided, it will be copied over as the base card.

Assets include the ModelCard proto file, Model Card document, and jinja template. These are written to the output_dir declared at initialization.

Arguments:

  • path - The path to model card proto.
  • model_card - The ModelCard object.

Returns:

A ModelCard representing the given model.

Raises:

  • FileNotFoundError - on failure to copy the template files.

update_model_card

def update_model_card(model_card: ModelCard) -> None

Updates the Proto file in the MCT assets directory.

Arguments:

  • model_card - The updated model card that users want to write back.

Raises:

  • Error - when the given model_card is invalid w.r.t. the schema.

export_format

def export_format(model_card: Optional[ModelCard] = None, template_path: Optional[str] = None, output_file=_DEFAULT_MODEL_CARD_FILE_NAME) -> str

Generates a model card document based on the MCT assets.

The model card document is both returned by this function, as well as saved to output_file.

Arguments:

  • model_card - The ModelCard object, generated from scaffold_assets(). If not provided, it will be read from the ModelCard proto file in the assets directory.
  • template_path - The file path of the Jinja template. If not provided, the default template will be used.
  • output_file - The file name of the generated model card. If not provided, the default 'model_card.html' will be used. If the file already exists, then it will be overwritten.

Returns:

The model card file content.

Raises:

  • MCTError - If export_format is called before scaffold_assets has generated model card assets.

export_test_results_json

def export_test_results_json(model_card: ModelCard, output_file=_DEFAULT_MODEL_CARD_RESULTS_FILE_NAME) -> Dict

Generates a document containing model card test results.

The model card result document is both returned by this function, as well as saved to output_file.

Arguments:

  • model_card - The ModelCard object, generated from scaffold_assets(). If not provided, it will be read from the ModelCard proto file in the assets directory.
  • output_file - The file name of the generated model card. If not provided, the default 'model_card_results.json' will be used. If the file already exists, then it will be overwritten.

Returns:

The model card file content.

Raises:

  • MCTError - If export_test_results_json is called before scaffold_assets has generated model card assets.

group_reports

@staticmethod
def group_reports(reports) -> Dict[str, List]

Given a list of model card reports, group them into those with the same type+slice.

Arguments:

  • reports - ModelCard reports.

Returns:

Dict of {type+slice: {set of reports with this type+slice}}

find_common_reports

@staticmethod
def find_common_reports(reports_a: List, reports_b: List) -> List[Tuple]

Given 2 lists of model card reports, find all reports that have the same type and slice in both lists.

Arguments:

  • reports_a - List of ModelCard report.
  • reports_b - List of ModelCard report.

Returns:

List of [(report A, report B), ...]. The list of (report A, report B) tuples is a cartesian product between reports in A vs reports in B with the same type+slice.

compare_model_cards

def compare_model_cards(card_a: ModelCard, card_b: ModelCard, export_path: Optional[str] = None) -> str

Compare reports across given model cards A and B and render them side by side. Only reports that have the same type and slice will be compared.

Arguments:

  • card_a - ModelCard for comparison.
  • card_b - ModelCard for comparison.
  • export_path - Optional path to export comparison report

Returns:

HTML report of the model card differences.