mbrs.metrics.xcomet module

class mbrs.metrics.xcomet.DeBERTaEncoder(pretrained_model: str, load_pretrained_weights: bool = True, local_files_only: bool = False)

Bases: BERTEncoder

DeBERTa encoder.

Parameters:

• pretrained_model (str) – Pretrained model from hugging face.

• load_pretrained_weights (bool) – If set to True loads the pretrained weights from Hugging Face

• local_files_only (bool) – Whether or not to only look at local files.

forward(input_ids: Tensor, attention_mask: Tensor | None = None, token_type_ids: Tensor | None = None, **kwargs) → dict[str, Tensor]

BERT model forward

Parameters:

• input_ids (torch.Tensor) – tokenized batch.

• attention_mask (torch.Tensor) – batch attention mask.

• token_type_ids (Optional[torch.tensor]) – batch attention mask. Defaults to None

Returns:

dictionary with ‘sentemb’, ‘wordemb’, ‘all_layers’

and ‘attention_mask’.

Return type:

Dict[str, torch.Tensor]

classmethod from_pretrained(pretrained_model: str, load_pretrained_weights: bool = True, local_files_only: bool = False) → Encoder

Function that loads a pretrained encoder from Hugging Face.

Parameters:

• pretrained_model (str) – Name of the pretrain model to be loaded.

• load_pretrained_weights (bool) – If set to True loads the pretrained weights from Hugging Face

• local_files_only (bool) – Whether or not to only look at local files.

Returns:

DeBERTaEncoder object.

Return type:

DeBERTaEncoder

class mbrs.metrics.xcomet.MetricXCOMET(cfg: Config)

Bases: Metric

XCOMET metric class.

Both XCOMET (Guerreiro et al., 2024) and XCOMET-lite (Larionov et al., 2024) are supported.

Supported models:

• Unbabel/XCOMET-XL

• Unbabel/XCOMET-XXL

• myyycroft/XCOMET-lite

class Config(model: str = 'Unbabel/XCOMET-XL', batch_size: int = 8, fp16: bool = False, bf16: bool = False, cpu: bool = False)

Bases: Config

XCOMET metric configuration.

• model (str): Model name or path.

• batch_size (int): Batch size.

• fp16 (bool): Use float16 for the forward computation.

• bf16 (bool): Use bfloat16 for the forward computation.

• cpu (bool): Use CPU for the forward computation.

batch_size: int = 8

bf16: bool = False

cpu: bool = False

fp16: bool = False

model: str = 'Unbabel/XCOMET-XL'

corpus_score(hypotheses: list[str], references_lists: list[list[str]] | None = None, sources: list[str] | None = None) → float

Calculate the corpus-level score.

Parameters:

• hypotheses (list[str]) – Hypotheses.

• references_lists (list[list[str]], optional) – Lists of references.

• sources (list[str], optional) – Sources.

Returns:

The corpus score.

Return type:

float

property device: device

Returns the device of the model.

pairwise_scores(hypotheses: list[str], references: list[str], source: str | None = None) → Tensor

Calculate the pairwise scores.

Parameters:

• hypotheses (list[str]) – Hypotheses.

• references (list[str]) – References.

• source (str, optional) – A source.

Returns:

Score matrix of shape (H, R), where H is the number

of hypotheses and R is the number of references.

Return type:

Tensor

score(hypothesis: str, reference: str | None = None, source: str | None = None) → float

Calculate the score of the given hypothesis.

Parameters:

• hypothesis (str) – A hypothesis.

• reference (str, optional) – A reference.

• source (str, optional) – A source.

Returns:

The score of the given hypothesis.

Return type:

float

scorer: XCOMETMetric

scores(hypotheses: list[str], references: list[str] | None = None, sources: list[str] | None = None) → Tensor

Calculate the scores of the given hypothesis.

Parameters:

• hypotheses (list[str]) – N hypotheses.

• references (list[str], optional) – N references.

• sources (list[str], optional) – N sources.

Returns:

The N scores of the given hypotheses.

Return type:

Tensor

class mbrs.metrics.xcomet.XCOMETLiteMetric(*args, **kwargs)

Bases: XCOMETMetric, PyTorchModelHubMixin

xCOMET-Lite model.