mbrs.metrics.bleurt module

mbrs.metrics.bleurt module#

class mbrs.metrics.bleurt.MetricBLEURT(cfg: Config)[source]#

Bases: Metric

BLEURT metric class.

We employ the PyTorch port version to implement this metric instead of the original version: lucadiliello/bleurt-pytorch (thanks to @lucadiliello)

Available checkpoints:

  • lucadiliello/BLEURT-20

  • lucadiliello/BLEURT-20-D12

  • lucadiliello/BLEURT-20-D3

  • lucadiliello/BLEURT-20-D6

  • lucadiliello/bleurt-base-128

  • lucadiliello/bleurt-base-512

  • lucadiliello/bleurt-large-128

  • lucadiliello/bleurt-large-512

  • lucadiliello/bleurt-tiny-128

  • lucadiliello/bleurt-tiny-512

class Config(model: str = 'lucadiliello/BLEURT-20-D12', batch_size: int = 64, fp16: bool = False, bf16: bool = False, cpu: bool = False)[source]#

Bases: Config

BLEURT metric configuration.

  • model (str): Model name or path.

  • batch_size (int): Batch size.

  • fp16 (bool): Use float16 for the forward computation.

  • bf16 (bool): Use bfloat16 for the forward computation.

  • cpu (bool): Use CPU for the forward computation.

batch_size: int = 64#
bf16: bool = False#
cpu: bool = False#
fp16: bool = False#
model: str = 'lucadiliello/BLEURT-20-D12'#
corpus_score(hypotheses: list[str], references_lists: list[list[str]], sources: list[str] | None = None) → float[source]#

Calculate the corpus-level score.

Parameters:

  • hypotheses (list[str]) – Hypotheses.

  • references_lists (list[list[str]]) – Lists of references.

  • sources (list[str], optional) – Sources.

Returns:

The corpus score.

Return type:

float

property device: device#

Returns the device of the model.

pairwise_scores(hypotheses: list[str], references: list[str], *_, **__) → Tensor[source]#

Calculate the pairwise scores.

Parameters:

  • hypotheses (list[str]) – Hypotheses.

  • references (list[str]) – References.

Returns:

Score matrix of shape (H, R), where H is the number

of hypotheses and R is the number of references.

Return type:

Tensor

score(hypothesis: str, reference: str, *_, **__) → float[source]#

Calculate the score of the given hypothesis.

Parameters:

  • hypothesis (str) – A hypothesis.

  • reference (str) – A reference.

Returns:

The score of the given hypothesis.

Return type:

float

scorer: BleurtForSequenceClassification#
scores(hypotheses: list[str], references: list[str], *_, **__) → Tensor[source]#

Calculate the scores of the given hypothesis.

Parameters:

  • hypotheses (list[str]) – N hypotheses.

  • references (list[str]) – N references.

Returns:

The N scores of the given hypotheses.

Return type:

Tensor

Contents