medcat.stats.kfold

Attributes

MedCATTrainerExportProjectInfo

The project name, project ID, CUIs str, and TUIs str

IntValuedMetric

FloatValuedMetric

Classes

CAT

This is a collection of serialisable model parts.

MedCATTrainerExport

dict() -> new empty dictionary

MedCATTrainerExportProject

dict() -> new empty dictionary

MedCATTrainerExportDocument

dict() -> new empty dictionary

MedCATTrainerExportAnnotation

dict() -> new empty dictionary

SplitType

The split type.

FoldCreator

The FoldCreator based on a MCT export.

SimpleFoldCreator

The FoldCreator based on a MCT export.

PerDocsFoldCreator

The FoldCreator based on a MCT export.

PerAnnsFoldCreator

The FoldCreator based on a MCT export.

WeightedDocumentsCreator

The FoldCreator based on a MCT export.

PerCUIMetrics

Usage docs: https://docs.pydantic.dev/2.9/concepts/models/

Functions

captured_state_cdb(cdb[, save_state_to_disk])

A context manager that captures and re-applies the initial CDB state.

get_stats(cat, data[, epoch, use_project_filters, ...])

TODO: Refactor and make nice

count_all_annotations(export)

Count the number of annotations in a trainer export.

count_all_docs(export)

Count the number of documents in a trainer export.

get_nr_of_annotations(doc)

Get the number of annotations for a tariner export document.

iter_anns(export)

Iterate over all the annotations in a trainer export.

iter_docs(export)

Iterate over all the docs in a trainer export.

get_fold_creator(mct_export, nr_of_folds, split_type)

Get the appropriate fold creator.

get_per_fold_metrics(cat, folds, use_project_filters, ...)

Get per fold metrics for a given set of folds.

_merge_examples(all_examples, cur_examples)

_add_helper(joined, single)

_add_weighted_helper(joined, single, cui2count)

get_metrics_mean(metrics, include_std)

The the mean of the provided metrics.

get_k_fold_stats(cat, mct_export_data[, k, ...])

Get the k-fold stats for the model with the specified data.

Module Contents

class medcat.stats.kfold.CAT(cdb, vocab=None, config=None, model_load_path=None)

Bases: medcat.storage.serialisables.AbstractSerialisable

This is a collection of serialisable model parts.

Parameters:
__init__(cdb, vocab=None, config=None, model_load_path=None)
Parameters:
Return type:

None

cdb
vocab = None
config = None
_trainer: medcat.trainer.Trainer | None = None
_pipeline
usage_monitor
_recreate_pipe(model_load_path=None)
Parameters:

model_load_path (Optional[str])

Return type:

medcat.pipeline.pipeline.Pipeline

classmethod get_init_attrs()
Return type:

list[str]

classmethod ignore_attrs()
Return type:

list[str]

__call__(text)
Parameters:

text (str)

Return type:

Optional[medcat.tokenizing.tokens.MutableDocument]

_ensure_not_training()

Method to ensure config is not set to train.

config.components.linking.train should only be True while training and not during inference. This aalso corrects the setting if necessary.

Return type:

None

get_entities(text: str, only_cui: Literal[False] = False) → medcat.data.entities.Entities
get_entities(text: str, only_cui: Literal[True] = True) → medcat.data.entities.OnlyCUIEntities
get_entities(text: str, only_cui: bool = False) → dict | medcat.data.entities.Entities | medcat.data.entities.OnlyCUIEntities

Get the entities recognised and linked within the provided text.

This will run the text through the pipeline and annotated the recognised and linked entities.

Parameters:

  • text (str) – The text to use.

  • only_cui (bool, optional) – Whether to only output the CUIs rather than the entire context. Defaults to False.

Returns:

Union[dict, Entities, OnlyCUIEntities] – The entities found and linked within the text.

_mp_worker_func(texts_and_indices)
Parameters:

texts_and_indices (list[tuple[str, str, bool]])

Return type:

list[tuple[str, str, Union[dict, medcat.data.entities.Entities, medcat.data.entities.OnlyCUIEntities]]]

_generate_batches_by_char_length(text_iter, batch_size_chars, only_cui)
Parameters:

  • text_iter (Union[Iterator[str], Iterator[tuple[str, str]]])

  • batch_size_chars (int)

  • only_cui (bool)

Return type:

Iterator[list[tuple[str, str, bool]]]

_generate_batches(text_iter, batch_size, batch_size_chars, only_cui)
Parameters:

  • text_iter (Union[Iterator[str], Iterator[tuple[str, str]]])

  • batch_size (int)

  • batch_size_chars (int)

  • only_cui (bool)

Return type:

Iterator[list[tuple[str, str, bool]]]

_generate_simple_batches(text_iter, batch_size, only_cui)
Parameters:

  • text_iter (Union[Iterator[str], Iterator[tuple[str, str]]])

  • batch_size (int)

  • only_cui (bool)

Return type:

Iterator[list[tuple[str, str, bool]]]

_mp_one_batch_per_process(executor, batch_iter, external_processes)
Parameters:

  • executor (concurrent.futures.ProcessPoolExecutor)

  • batch_iter (Iterator[list[tuple[str, str, bool]]])

  • external_processes (int)

Return type:

Iterator[tuple[str, Union[dict, medcat.data.entities.Entities, medcat.data.entities.OnlyCUIEntities]]]

get_entities_multi_texts(texts, only_cui=False, n_process=1, batch_size=-1, batch_size_chars=1000000)

Get entities from multiple texts (potentially in parallel).

If n_process > 1, n_process - 1 new processes will be created and data will be processed on those as well as the main process in parallel.

Parameters:

  • texts (Union[Iterable[str], Iterable[tuple[str, str]]]) – The input text. Either an iterable of raw text or one with in the format of (text_index, text).

  • only_cui (bool) – Whether to only return CUIs rather than other information like start/end and annotated value. Defaults to False.

  • n_process (int) – Number of processes to use. Defaults to 1.

  • batch_size (int) – The number of texts to batch at a time. A batch of the specified size will be given to each worker process. Defaults to -1 and in this case the character count will be used instead.

  • batch_size_chars (int) – The maximum number of characters to process in a batch. Each process will be given batch of texts with a total number of characters not exceeding this value. Defaults to 1,000,000 characters. Set to -1 to disable.

Yields:

Iterator[tuple[str, Union[dict, Entities, OnlyCUIEntities]]] – The results in the format of (text_index, entities).

Return type:

Iterator[tuple[str, Union[dict, medcat.data.entities.Entities, medcat.data.entities.OnlyCUIEntities]]]

_get_entity(ent, doc_tokens, cui)
Parameters:
Return type:

medcat.data.entities.Entity

get_addon_output(ent)

Get the addon output for the entity.

This includes a key-value pair for each addon that provides some. Sometimes same-type addons may combine their output under the same key.

Parameters:

ent (MutableEntity) – The entity in quesiton.

Raises:

ValueError – If unable to merge multiple addon output.

Returns:

dict[str, dict] – All the addon output.

Return type:

dict[str, dict]

_doc_to_out_entity(ent, doc_tokens, only_cui)
Parameters:
Return type:

tuple[int, Union[medcat.data.entities.Entity, str]]

_doc_to_out(doc, only_cui, out_with_text=False)
Parameters:
Return type:

Union[medcat.data.entities.Entities, medcat.data.entities.OnlyCUIEntities]

property trainer

The trainer object.

save_model_pack(target_folder, pack_name=DEFAULT_PACK_NAME, serialiser_type='dill', make_archive=True, only_archive=False, add_hash_to_pack_name=True, change_description=None)

Save model pack.

The resulting model pack name will have the hash of the model pack in its name if (and only if) the default model pack name is used.

Parameters:

  • target_folder (str) – The folder to save the pack in.

  • pack_name (str, optional) – The model pack name. Defaults to DEFAULT_PACK_NAME.

  • serialiser_type (Union[str, AvailableSerialisers], optional) – The serialiser type. Defaults to ‘dill’.

  • make_archive (bool) – Whether to make the arhive /.zip file. Defaults to True.

  • only_archive (bool) – Whether to clear the non-compressed folder. Defaults to False.

  • add_hash_to_pack_name (bool) – Whether to add the hash to the pack name. This is only relevant if pack_name is specified. Defaults to True.

  • change_description (Optional[str]) – If provided, this the description will be added to the model description. Defaults to None.

Returns:

str – The final model pack path.

Return type:

str

_get_hash()
Return type:

str

_versioning(change_description)
Parameters:

change_description (Optional[str])

Return type:

str

classmethod attempt_unpack(zip_path)

Attempt unpack the zip to a folder and get the model pack path.

If the folder already exists, no unpacking is done.

Parameters:

zip_path (str) – The ZIP path

Returns:

str – The model pack path

Return type:

str

classmethod load_model_pack(model_pack_path)

Load the model pack from file.

Parameters:

model_pack_path (str) – The model pack path.

Raises:

ValueError – If the saved data does not represent a model pack.

Returns:

CAT – The loaded model pack.

Return type:

CAT

classmethod load_cdb(model_pack_path)

Loads the concept database from the provided model pack path

Parameters:

model_pack_path (str) – path to model pack, zip or dir.

Returns:

CDB – The loaded concept database

Return type:

medcat.cdb.CDB

get_model_card(as_dict: Literal[True]) → medcat.data.model_card.ModelCard
get_model_card(as_dict: Literal[False]) → str

Get the model card either a (nested) dict or a json string.

Parameters:

as_dict (bool) – Whether to return as dict. Defaults to False.

Returns:

Union[str, ModelCard] – The model card.

__eq__(other)
Parameters:

other (Any)

Return type:

bool

add_addon(addon)
Parameters:

addon (medcat.components.addons.addons.AddonComponent)

Return type:

None

get_strategy()
Return type:

SerialisingStrategy

classmethod include_properties()
Return type:

list[str]

medcat.stats.kfold.captured_state_cdb(cdb, save_state_to_disk=False)

A context manager that captures and re-applies the initial CDB state.

The context manager captures/copies the initial state of the CDB when entering. It then allows the user to modify the state (i.e training). Upon exit re-applies the initial CDB state.

If RAM is an issue, it is recommended to use save_state_to_disk. Otherwise the copy of the original state will be held in memory. If saved on disk, a temporary file is used and removed afterwards.

Parameters:

  • cdb – The CDB to use.

  • save_state_to_disk (bool) – Whether to save state on disk or hold in memory. Defaults to False.

Yields:

None

medcat.stats.kfold.get_stats(cat, data, epoch=0, use_project_filters=False, use_overlaps=False, extra_cui_filter=None, do_print=True)

TODO: Refactor and make nice Print metrics on a dataset (F1, P, R), it will also print the concepts that have the most FP,FN,TP.

Parameters:

  • cat (medcat.cat.CAT) – (CAT): The model pack.

  • data (dict) – The json object that we get from MedCATtrainer on export.

  • epoch (int) – Used during training, so we know what epoch is it.

  • use_project_filters (bool) – Each project in MedCATtrainer can have filters, do we want to respect those filters when calculating metrics.

  • use_overlaps (bool) – Allow overlapping entities, nearly always False as it is very difficult to annotate overlapping entities.

  • use_cui_doc_limit (bool) – If True the metrics for a CUI will be only calculated if that CUI appears in a document, in other words if the document was annotated for that CUI. Useful in very specific situations when during the annotation process the set of CUIs changed.

  • use_groups (bool) – If True concepts that have groups will be combined and stats will be reported on groups.

  • extra_cui_filter (Optional[set]) – This filter will be intersected with all other filters, or if all others are not set then only this one will be used.

  • do_print (bool) – Whether to print stats out. Defaults to True.

Returns:

  • fps (dict) – False positives for each CUI.

  • fns (dict) – False negatives for each CUI.

  • tps (dict) – True positives for each CUI.

  • cui_prec (dict) – Precision for each CUI.

  • cui_rec (dict) – Recall for each CUI.

  • cui_f1 (dict) – F1 for each CUI.

  • cui_counts (dict) – Number of occurrence for each CUI.

  • examples (dict) – Examples for each of the fp, fn, tp. Format will be examples[‘fp’][‘cui’][<list_of_examples>].

Return type:

tuple[dict[str, int], dict[str, int], dict[str, int], dict[str, float], dict[str, float], dict[str, float], dict[str, int], dict]

class medcat.stats.kfold.MedCATTrainerExport

Bases: typing_extensions.TypedDict

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:

d = {} for k, v in iterable:

d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs

in the keyword argument list. For example: dict(one=1, two=2)

projects: list[MedCATTrainerExportProject]
__contains__()

True if the dictionary has the specified key, else False.

__delattr__()

Implement delattr(self, name).

__delitem__()

Delete self[key].

__dir__()

Default dir() implementation.

__eq__()

Return self==value.

__format__()

Default object formatter.

__ge__()

Return self>=value.

__getattribute__()

Return getattr(self, name).

__getitem__()

x.__getitem__(y) <==> x[y]

__gt__()

Return self>value.

__init__()

Initialize self. See help(type(self)) for accurate signature.

__ior__()

Return self|=value.

__iter__()

Implement iter(self).

__le__()

Return self<=value.

__len__()

Return len(self).

__lt__()

Return self<value.

__ne__()

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__or__()

Return self|value.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__()

Return repr(self).

__reversed__()

Return a reverse iterator over the dict keys.

__ror__()

Return value|self.

__setattr__()

Implement setattr(self, name, value).

__setitem__()

Set self[key] to value.

__sizeof__()

D.__sizeof__() -> size of D in memory, in bytes

__str__()

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

clear()

D.clear() -> None. Remove all items from D.

copy()

D.copy() -> a shallow copy of D

get()

Return the value for key if key is in the dictionary, else default.

items()

D.items() -> a set-like object providing a view on D’s items

keys()

D.keys() -> a set-like object providing a view on D’s keys

pop()

D.pop(k[,d]) -> v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault()

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update()

D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

D.values() -> an object providing a view on D’s values

class medcat.stats.kfold.MedCATTrainerExportProject

Bases: typing_extensions.TypedDict

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:

d = {} for k, v in iterable:

d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs

in the keyword argument list. For example: dict(one=1, two=2)

name: str
id: Any
cuis: str
tuis: str | None
documents: list[MedCATTrainerExportDocument]
__contains__()

True if the dictionary has the specified key, else False.

__delattr__()

Implement delattr(self, name).

__delitem__()

Delete self[key].

__dir__()

Default dir() implementation.

__eq__()

Return self==value.

__format__()

Default object formatter.

__ge__()

Return self>=value.

__getattribute__()

Return getattr(self, name).

__getitem__()

x.__getitem__(y) <==> x[y]

__gt__()

Return self>value.

__init__()

Initialize self. See help(type(self)) for accurate signature.

__ior__()

Return self|=value.

__iter__()

Implement iter(self).

__le__()

Return self<=value.

__len__()

Return len(self).

__lt__()

Return self<value.

__ne__()

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__or__()

Return self|value.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__()

Return repr(self).

__reversed__()

Return a reverse iterator over the dict keys.

__ror__()

Return value|self.

__setattr__()

Implement setattr(self, name, value).

__setitem__()

Set self[key] to value.

__sizeof__()

D.__sizeof__() -> size of D in memory, in bytes

__str__()

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

clear()

D.clear() -> None. Remove all items from D.

copy()

D.copy() -> a shallow copy of D

get()

Return the value for key if key is in the dictionary, else default.

items()

D.items() -> a set-like object providing a view on D’s items

keys()

D.keys() -> a set-like object providing a view on D’s keys

pop()

D.pop(k[,d]) -> v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault()

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update()

D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

D.values() -> an object providing a view on D’s values

class medcat.stats.kfold.MedCATTrainerExportDocument

Bases: typing_extensions.TypedDict

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:

d = {} for k, v in iterable:

d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs

in the keyword argument list. For example: dict(one=1, two=2)

name: str
id: Any
last_modified: str
text: str
annotations: list[MedCATTrainerExportAnnotation]
__contains__()

True if the dictionary has the specified key, else False.

__delattr__()

Implement delattr(self, name).

__delitem__()

Delete self[key].

__dir__()

Default dir() implementation.

__eq__()

Return self==value.

__format__()

Default object formatter.

__ge__()

Return self>=value.

__getattribute__()

Return getattr(self, name).

__getitem__()

x.__getitem__(y) <==> x[y]

__gt__()

Return self>value.

__init__()

Initialize self. See help(type(self)) for accurate signature.

__ior__()

Return self|=value.

__iter__()

Implement iter(self).

__le__()

Return self<=value.

__len__()

Return len(self).

__lt__()

Return self<value.

__ne__()

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__or__()

Return self|value.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__()

Return repr(self).

__reversed__()

Return a reverse iterator over the dict keys.

__ror__()

Return value|self.

__setattr__()

Implement setattr(self, name, value).

__setitem__()

Set self[key] to value.

__sizeof__()

D.__sizeof__() -> size of D in memory, in bytes

__str__()

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

clear()

D.clear() -> None. Remove all items from D.

copy()

D.copy() -> a shallow copy of D

get()

Return the value for key if key is in the dictionary, else default.

items()

D.items() -> a set-like object providing a view on D’s items

keys()

D.keys() -> a set-like object providing a view on D’s keys

pop()

D.pop(k[,d]) -> v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault()

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update()

D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

D.values() -> an object providing a view on D’s values

class medcat.stats.kfold.MedCATTrainerExportAnnotation

Bases: MedCATTrainerExportAnnotationRequired

dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object’s

(key, value) pairs

dict(iterable) -> new dictionary initialized as if via:

d = {} for k, v in iterable:

d[k] = v

dict(**kwargs) -> new dictionary initialized with the name=value pairs

in the keyword argument list. For example: dict(one=1, two=2)

id: str | int
validated: bool | None
start: int
end: int
cui: str
value: str
__contains__()

True if the dictionary has the specified key, else False.

__delattr__()

Implement delattr(self, name).

__delitem__()

Delete self[key].

__dir__()

Default dir() implementation.

__eq__()

Return self==value.

__format__()

Default object formatter.

__ge__()

Return self>=value.

__getattribute__()

Return getattr(self, name).

__getitem__()

x.__getitem__(y) <==> x[y]

__gt__()

Return self>value.

__init__()

Initialize self. See help(type(self)) for accurate signature.

__ior__()

Return self|=value.

__iter__()

Implement iter(self).

__le__()

Return self<=value.

__len__()

Return len(self).

__lt__()

Return self<value.

__ne__()

Return self!=value.

__new__()

Create and return a new object. See help(type) for accurate signature.

__or__()

Return self|value.

__reduce__()

Helper for pickle.

__reduce_ex__()

Helper for pickle.

__repr__()

Return repr(self).

__reversed__()

Return a reverse iterator over the dict keys.

__ror__()

Return value|self.

__setattr__()

Implement setattr(self, name, value).

__setitem__()

Set self[key] to value.

__sizeof__()

D.__sizeof__() -> size of D in memory, in bytes

__str__()

Return str(self).

__subclasshook__()

Abstract classes can override this to customize issubclass().

This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached).

clear()

D.clear() -> None. Remove all items from D.

copy()

D.copy() -> a shallow copy of D

get()

Return the value for key if key is in the dictionary, else default.

items()

D.items() -> a set-like object providing a view on D’s items

keys()

D.keys() -> a set-like object providing a view on D’s keys

pop()

D.pop(k[,d]) -> v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem()

Remove and return a (key, value) pair as a 2-tuple.

Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty.

setdefault()

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update()

D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

D.values() -> an object providing a view on D’s values

medcat.stats.kfold.count_all_annotations(export)

Count the number of annotations in a trainer export.

Parameters:

export (MedCATTrainerExport) – The trainer export.

Returns:

int – The total number of annotations.

Return type:

int

medcat.stats.kfold.count_all_docs(export)

Count the number of documents in a trainer export.

Parameters:

export (MedCATTrainerExport) – The trainer export.

Returns:

int – The total number of documents.

Return type:

int

medcat.stats.kfold.get_nr_of_annotations(doc)

Get the number of annotations for a tariner export document.

Parameters:

doc (MedCATTrainerExportDocument) – The trainer export document.

Returns:

int – The number of annotations within the document.

Return type:

int

medcat.stats.kfold.iter_anns(export)

Iterate over all the annotations in a trainer export.

Parameters:

export (MedCATTrainerExport) – The trainer export.

Yields:
Iterator[tuple[MedCATTrainerExportProjectInfo,

MedCATTrainerExportDocument, MedCATTrainerExportAnnotation]]:

The project info, the document, and the annotation.

Return type:

Iterator[tuple[MedCATTrainerExportProjectInfo, MedCATTrainerExportDocument, MedCATTrainerExportAnnotation]]

medcat.stats.kfold.iter_docs(export)

Iterate over all the docs in a trainer export.

Parameters:

export (MedCATTrainerExport) – The trainer export.

Yields:
Iterator[tuple[MedCATTrainerExportProjectInfo,

MedCATTrainerExportDocument]]:

The project info and the document.

Return type:

Iterator[tuple[MedCATTrainerExportProjectInfo, MedCATTrainerExportDocument]]

medcat.stats.kfold.MedCATTrainerExportProjectInfo

The project name, project ID, CUIs str, and TUIs str

class medcat.stats.kfold.SplitType

Bases: enum.Enum

The split type.

DOCUMENTS

Split over number of documents.

ANNOTATIONS

Split over number of annotations.

DOCUMENTS_WEIGHTED

Split over number of documents based on the number of annotations. So essentially this ensures that the same document isn’t in 2 folds while trying to more equally distribute documents with different number of annotations. For example:

If we have 6 documents that we want to split into 3 folds. The number of annotations per document are as follows:

[40, 40, 20, 10, 5, 5]

If we were to split this trivially over documents, we’d end up with the 3 folds with number of annotations that are far from even:

[80, 30, 10]

However, if we use the annotations as weights, we would be able to create folds that have more evenly distributed annotations, e.g:

[[D1,], [D2], [D3, D4, D5, D6]]

where D# denotes the number of the documents, with the number of annotations being equal:

[ 40, 40, 20 + 10 + 5 + 5 = 40]

__new__(value)
_generate_next_value_(start, count, last_values)

Generate the next value when not given.

name: the name of the member start: the initial start value or None count: the number of existing members last_value: the last value assigned or None

classmethod _missing_(value)
__repr__()
__str__()
__dir__()

Returns all members and all public methods

__format__(format_spec)

Returns format using actual value type unless __str__ has been overridden.

__hash__()
__reduce_ex__(proto)
name()

The name of the Enum member.

value()

The value of the Enum member.

class medcat.stats.kfold.FoldCreator(mct_export, nr_of_folds)

Bases: abc.ABC

The FoldCreator based on a MCT export.

Parameters:

  • mct_export (MedCATTrainerExport) – The MCT export dict.

  • nr_of_folds (int) – Number of folds to create.

  • use_annotations (bool) – Whether to fold on number of annotations or documents.

__init__(mct_export, nr_of_folds)
Parameters:
Return type:

None

mct_export
nr_of_folds
_find_or_add_doc(project, orig_doc)
Parameters:
Return type:

medcat.data.mctexport.MedCATTrainerExportDocument

_create_new_project(proj_info)
Parameters:

proj_info (medcat.data.mctexport.MedCATTrainerExportProjectInfo)

Return type:

medcat.data.mctexport.MedCATTrainerExportProject

_create_export_with_documents(relevant_docs)
Parameters:

relevant_docs (Iterable[tuple[medcat.data.mctexport.MedCATTrainerExportProjectInfo, medcat.data.mctexport.MedCATTrainerExportDocument]])

Return type:

medcat.data.mctexport.MedCATTrainerExport

abstract create_folds()

Create folds.

Raises:

ValueError – If something went wrong.

Returns:

list[MedCATTrainerExport] – The created folds.

Return type:

list[medcat.data.mctexport.MedCATTrainerExport]

__slots__ = ()
class medcat.stats.kfold.SimpleFoldCreator(mct_export, nr_of_folds, counter)

Bases: FoldCreator

The FoldCreator based on a MCT export.

Parameters:
__init__(mct_export, nr_of_folds, counter)
Parameters:
Return type:

None

_counter
total
per_fold
_init_per_fold()
Return type:

list[int]

abstract _create_fold(fold_nr)
Parameters:

fold_nr (int)

Return type:

medcat.data.mctexport.MedCATTrainerExport

create_folds()

Create folds.

Raises:

ValueError – If something went wrong.

Returns:

list[MedCATTrainerExport] – The created folds.

Return type:

list[medcat.data.mctexport.MedCATTrainerExport]

mct_export
nr_of_folds
_find_or_add_doc(project, orig_doc)
Parameters:
Return type:

medcat.data.mctexport.MedCATTrainerExportDocument

_create_new_project(proj_info)
Parameters:

proj_info (medcat.data.mctexport.MedCATTrainerExportProjectInfo)

Return type:

medcat.data.mctexport.MedCATTrainerExportProject

_create_export_with_documents(relevant_docs)
Parameters:

relevant_docs (Iterable[tuple[medcat.data.mctexport.MedCATTrainerExportProjectInfo, medcat.data.mctexport.MedCATTrainerExportDocument]])

Return type:

medcat.data.mctexport.MedCATTrainerExport

__slots__ = ()
class medcat.stats.kfold.PerDocsFoldCreator(mct_export, nr_of_folds)

Bases: FoldCreator

The FoldCreator based on a MCT export.

Parameters:

  • mct_export (MedCATTrainerExport) – The MCT export dict.

  • nr_of_folds (int) – Number of folds to create.

  • use_annotations (bool) – Whether to fold on number of annotations or documents.

__init__(mct_export, nr_of_folds)
Parameters:
Return type:

None

nr_of_docs
per_doc_simple
_all_docs
_create_fold(fold_nr)
Parameters:

fold_nr (int)

Return type:

medcat.data.mctexport.MedCATTrainerExport

create_folds()

Create folds.

Raises:

ValueError – If something went wrong.

Returns:

list[MedCATTrainerExport] – The created folds.

Return type:

list[medcat.data.mctexport.MedCATTrainerExport]

mct_export
nr_of_folds
_find_or_add_doc(project, orig_doc)
Parameters:
Return type:

medcat.data.mctexport.MedCATTrainerExportDocument

_create_new_project(proj_info)
Parameters:

proj_info (medcat.data.mctexport.MedCATTrainerExportProjectInfo)

Return type:

medcat.data.mctexport.MedCATTrainerExportProject

_create_export_with_documents(relevant_docs)
Parameters:

relevant_docs (Iterable[tuple[medcat.data.mctexport.MedCATTrainerExportProjectInfo, medcat.data.mctexport.MedCATTrainerExportDocument]])

Return type:

medcat.data.mctexport.MedCATTrainerExport

__slots__ = ()
class medcat.stats.kfold.PerAnnsFoldCreator(mct_export, nr_of_folds)

Bases: SimpleFoldCreator

The FoldCreator based on a MCT export.

Parameters:

  • mct_export (MedCATTrainerExport) – The MCT export dict.

  • nr_of_folds (int) – Number of folds to create.

  • use_annotations (bool) – Whether to fold on number of annotations or documents.

__init__(mct_export, nr_of_folds)
Parameters:
Return type:

None

_add_target_ann(project, orig_doc, ann)
Parameters:
Return type:

None

_targets(start_at)
Parameters:

start_at (int)

Return type:

Iterable[tuple[medcat.data.mctexport.MedCATTrainerExportProjectInfo, medcat.data.mctexport.MedCATTrainerExportDocument, medcat.data.mctexport.MedCATTrainerExportAnnotation]]

_create_fold(fold_nr)
Parameters:

fold_nr (int)

Return type:

medcat.data.mctexport.MedCATTrainerExport

_counter
total
per_fold
_init_per_fold()
Return type:

list[int]

create_folds()

Create folds.

Raises:

ValueError – If something went wrong.

Returns:

list[MedCATTrainerExport] – The created folds.

Return type:

list[medcat.data.mctexport.MedCATTrainerExport]

mct_export
nr_of_folds
_find_or_add_doc(project, orig_doc)
Parameters:
Return type:

medcat.data.mctexport.MedCATTrainerExportDocument

_create_new_project(proj_info)
Parameters:

proj_info (medcat.data.mctexport.MedCATTrainerExportProjectInfo)

Return type:

medcat.data.mctexport.MedCATTrainerExportProject

_create_export_with_documents(relevant_docs)
Parameters:

relevant_docs (Iterable[tuple[medcat.data.mctexport.MedCATTrainerExportProjectInfo, medcat.data.mctexport.MedCATTrainerExportDocument]])

Return type:

medcat.data.mctexport.MedCATTrainerExport

__slots__ = ()
class medcat.stats.kfold.WeightedDocumentsCreator(mct_export, nr_of_folds, weight_calculator)

Bases: FoldCreator

The FoldCreator based on a MCT export.

Parameters:
__init__(mct_export, nr_of_folds, weight_calculator)
Parameters:
Return type:

None

_weight_calculator
_weighted_docs
create_folds()

Create folds.

Raises:

ValueError – If something went wrong.

Returns:

list[MedCATTrainerExport] – The created folds.

Return type:

list[medcat.data.mctexport.MedCATTrainerExport]

mct_export
nr_of_folds
_find_or_add_doc(project, orig_doc)
Parameters:
Return type:

medcat.data.mctexport.MedCATTrainerExportDocument

_create_new_project(proj_info)
Parameters:

proj_info (medcat.data.mctexport.MedCATTrainerExportProjectInfo)

Return type:

medcat.data.mctexport.MedCATTrainerExportProject

_create_export_with_documents(relevant_docs)
Parameters:

relevant_docs (Iterable[tuple[medcat.data.mctexport.MedCATTrainerExportProjectInfo, medcat.data.mctexport.MedCATTrainerExportDocument]])

Return type:

medcat.data.mctexport.MedCATTrainerExport

__slots__ = ()
medcat.stats.kfold.get_fold_creator(mct_export, nr_of_folds, split_type)

Get the appropriate fold creator.

Parameters:

  • mct_export (MedCATTrainerExport) – The MCT export.

  • nr_of_folds (int) – Number of folds to use.

  • split_type (SplitType) – The type of split to use.

Raises:

ValueError – In case of an unknown split type.

Returns:

FoldCreator – The corresponding fold creator.

Return type:

FoldCreator

medcat.stats.kfold.get_per_fold_metrics(cat, folds, use_project_filters, *args, **kwargs)

Get per fold metrics for a given set of folds.

This method captures the state of the before processing each fold. For each fold, it trains on all other folds, and runs metrics on the fold itself.

Parameters:

  • cat (CAT) – The model pack.

  • folds (list[MedCATTrainerExport]) – The folds.

  • use_project_filters (bool) – Whether to use project filters.

Returns:

list[tuple] – The metrics for each fold.

Return type:

list[tuple]

medcat.stats.kfold._merge_examples(all_examples, cur_examples)
Parameters:

  • all_examples (dict)

  • cur_examples (dict)

Return type:

None

medcat.stats.kfold.IntValuedMetric
medcat.stats.kfold.FloatValuedMetric
class medcat.stats.kfold.PerCUIMetrics(/, **data)

Bases: pydantic.BaseModel

Usage docs: https://docs.pydantic.dev/2.9/concepts/models/

A base class for creating Pydantic models.

Parameters:

data (Any)

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

weights: list[int | float] = []
vals: list[int | float] = []
add(val, weight=1)
Parameters:

weight (int)

get_mean()
get_std()
model_config: ClassVar[pydantic.config.ConfigDict]

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_fields: ClassVar[Dict[str, pydantic.fields.FieldInfo]]

Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.

This replaces Model.__fields__ from Pydantic V1.

model_computed_fields: ClassVar[Dict[str, pydantic.fields.ComputedFieldInfo]]

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

__class_vars__: ClassVar[set[str]]

The names of the class variables defined on the model.

__private_attributes__: ClassVar[Dict[str, pydantic.fields.ModelPrivateAttr]]

Metadata about the private attributes of the model.

__signature__: ClassVar[inspect.Signature]

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: ClassVar[bool] = False

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__: ClassVar[pydantic_core.CoreSchema]

The core schema of the model.

__pydantic_custom_init__: ClassVar[bool]

Whether the model has a custom __init__ method.

__pydantic_decorators__: ClassVar[pydantic._internal._decorators.DecoratorInfos]

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: ClassVar[pydantic._internal._generics.PydanticGenericMetadata]

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: ClassVar[Dict[str, Any] | None] = None

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__: ClassVar[None | Literal['model_post_init']]

The name of the post-init method for the model, if defined.

__pydantic_root_model__: ClassVar[bool] = False

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__: ClassVar[pydantic_core.SchemaSerializer]

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__: ClassVar[pydantic_core.SchemaValidator | pydantic.plugin._schema_validator.PluggableSchemaValidator]

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_extra__: dict[str, Any] | None

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__: set[str]

The names of fields explicitly set during instantiation.

__pydantic_private__: dict[str, Any] | None

Values of private attributes set on the model instance.

__slots__ = ('__dict__', '__pydantic_fields_set__', '__pydantic_extra__', '__pydantic_private__')
__init__(/, **data)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (Any)

Return type:

None

property model_extra: dict[str, Any] | None

Get extra fields set during validation.

Returns:

A dictionary of extra fields, or `None` if `config.extra` is not set to `”allow”`.

Return type:

dict[str, Any] | None

property model_fields_set: set[str]

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set, – i.e. that were not filled from defaults.

Return type:

set[str]

classmethod model_construct(_fields_set=None, **values)

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note

model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.

Parameters:

  • _fields_set (set[str] | None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.

  • values (Any) – Trusted or pre-validated data dictionary.

Returns:

A new instance of the `Model` class with validated data.

Return type:

typing_extensions.Self

model_copy(*, update=None, deep=False)

Usage docs: https://docs.pydantic.dev/2.9/concepts/serialization/#model_copy

Returns a copy of the model.

Parameters:

  • update (dict[str, Any] | None) – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.

  • deep (bool) – Set to True to make a deep copy of the model.

Returns:

New model instance.

Return type:

typing_extensions.Self

model_dump(*, mode='python', include=None, exclude=None, context=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False, round_trip=False, warnings=True, serialize_as_any=False)

Usage docs: https://docs.pydantic.dev/2.9/concepts/serialization/#modelmodel_dump

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

  • mode (Literal['json', 'python'] | str) – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.

  • include (IncEx | None) – A set of fields to include in the output.

  • exclude (IncEx | None) – A set of fields to exclude from the output.

  • context (Any | None) – Additional context to pass to the serializer.

  • by_alias (bool) – Whether to use the field’s alias in the dictionary key if defined.

  • exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (bool) – Whether to exclude fields that are set to their default value.

  • exclude_none (bool) – Whether to exclude fields that have a value of None.

  • round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A dictionary representation of the model.

Return type:

dict[str, Any]

model_dump_json(*, indent=None, include=None, exclude=None, context=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False, round_trip=False, warnings=True, serialize_as_any=False)

Usage docs: https://docs.pydantic.dev/2.9/concepts/serialization/#modelmodel_dump_json

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

  • indent (int | None) – Indentation to use in the JSON output. If None is passed, the output will be compact.

  • include (IncEx | None) – Field(s) to include in the JSON output.

  • exclude (IncEx | None) – Field(s) to exclude from the JSON output.

  • context (Any | None) – Additional context to pass to the serializer.

  • by_alias (bool) – Whether to serialize using field aliases.

  • exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (bool) – Whether to exclude fields that are set to their default value.

  • exclude_none (bool) – Whether to exclude fields that have a value of None.

  • round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A JSON string representation of the model.

Return type:

str

classmethod model_json_schema(by_alias=True, ref_template=DEFAULT_REF_TEMPLATE, schema_generator=GenerateJsonSchema, mode='validation')

Generates a JSON schema for a model class.

Parameters:

  • by_alias (bool) – Whether to use attribute aliases or not.

  • ref_template (str) – The reference template.

  • schema_generator (type[pydantic.json_schema.GenerateJsonSchema]) – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications

  • mode (pydantic.json_schema.JsonSchemaMode) – The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

Return type:

dict[str, Any]

classmethod model_parametrized_name(params)

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:

params (tuple[type[Any], Ellipsis]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.

Returns:

String representing the new class where `params` are passed to `cls` as type variables.

Raises:

TypeError – Raised when trying to generate concrete names for non-generic models.

Return type:

str

model_post_init(__context)

Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

Parameters:

__context (Any)

Return type:

None

classmethod model_rebuild(*, force=False, raise_errors=True, _parent_namespace_depth=2, _types_namespace=None)

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

  • force (bool) – Whether to force the rebuilding of the model schema, defaults to False.

  • raise_errors (bool) – Whether to raise errors, defaults to True.

  • _parent_namespace_depth (int) – The depth level of the parent namespace, defaults to 2.

  • _types_namespace (dict[str, Any] | None) – The types namespace, defaults to None.

Returns:

  • Returns `None` if the schema is already “complete” and rebuilding was not required.

  • If rebuilding _was_ required, returns `True` if rebuilding was successful, otherwise `False`.

Return type:

bool | None

classmethod model_validate(obj, *, strict=None, from_attributes=None, context=None)

Validate a pydantic model instance.

Parameters:

  • obj (Any) – The object to validate.

  • strict (bool | None) – Whether to enforce types strictly.

  • from_attributes (bool | None) – Whether to extract data from object attributes.

  • context (Any | None) – Additional context to pass to the validator.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

Return type:

typing_extensions.Self

classmethod model_validate_json(json_data, *, strict=None, context=None)

Usage docs: https://docs.pydantic.dev/2.9/concepts/json/#json-parsing

Validate the given JSON data against the Pydantic model.

Parameters:

  • json_data (str | bytes | bytearray) – The JSON data to validate.

  • strict (bool | None) – Whether to enforce types strictly.

  • context (Any | None) – Extra variables to pass to the validator.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

Return type:

typing_extensions.Self

classmethod model_validate_strings(obj, *, strict=None, context=None)

Validate the given object with string data against the Pydantic model.

Parameters:

  • obj (Any) – The object containing string data to validate.

  • strict (bool | None) – Whether to enforce types strictly.

  • context (Any | None) – Extra variables to pass to the validator.

Returns:

The validated Pydantic model.

Return type:

typing_extensions.Self

classmethod __get_pydantic_core_schema__(source, handler, /)

Hook into generating the model’s CoreSchema.

Parameters:

  • source (type[BaseModel]) – The class we are generating a schema for. This will generally be the same as the cls argument if this is a classmethod.

  • handler (pydantic.annotated_handlers.GetCoreSchemaHandler) – A callable that calls into Pydantic’s internal CoreSchema generation logic.

Returns:

A `pydantic-core` `CoreSchema`.

Return type:

pydantic_core.CoreSchema

classmethod __get_pydantic_json_schema__(core_schema, handler, /)

Hook into generating the model’s JSON schema.

Parameters:

  • core_schema (pydantic_core.CoreSchema) – A pydantic-core CoreSchema. You can ignore this argument and call the handler with a new CoreSchema, wrap this CoreSchema ({‘type’: ‘nullable’, ‘schema’: current_schema}), or just call the handler with the original schema.

  • handler (pydantic.annotated_handlers.GetJsonSchemaHandler) – Call into Pydantic’s internal JSON schema generation. This will raise a pydantic.errors.PydanticInvalidForJsonSchema if JSON schema generation fails. Since this gets called by BaseModel.model_json_schema you can override the schema_generator argument to that function to change JSON schema generation globally for a type.

Returns:

A JSON schema, as a Python object.

Return type:

pydantic.json_schema.JsonSchemaValue

classmethod __pydantic_init_subclass__(**kwargs)

This is intended to behave just like __init_subclass__, but is called by ModelMetaclass only after the class is actually fully initialized. In particular, attributes like model_fields will be present when this is called.

This is necessary because __init_subclass__ will always be called by type.__new__, and it would require a prohibitively large refactor to the ModelMetaclass to ensure that type.__new__ was called in such a manner that the class would already be sufficiently initialized.

This will receive the same kwargs that would be passed to the standard __init_subclass__, namely, any kwargs passed to the class definition that aren’t used internally by pydantic.

Parameters:

**kwargs (Any) – Any keyword arguments passed to the class definition that aren’t used internally by pydantic.

Return type:

None

classmethod __class_getitem__(typevar_values)
Parameters:

typevar_values (type[Any] | tuple[type[Any], Ellipsis])

Return type:

type[BaseModel] | pydantic._internal._forward_ref.PydanticRecursiveRef

__copy__()

Returns a shallow copy of the model.

Return type:

typing_extensions.Self

__deepcopy__(memo=None)

Returns a deep copy of the model.

Parameters:

memo (dict[int, Any] | None)

Return type:

typing_extensions.Self

__getattr__(item)
Parameters:

item (str)

Return type:

Any

_check_frozen(name, value)
Parameters:

  • name (str)

  • value (Any)

Return type:

None

__getstate__()
Return type:

dict[Any, Any]

__setstate__(state)
Parameters:

state (dict[Any, Any])

Return type:

None

__eq__(other)
Parameters:

other (Any)

Return type:

bool

classmethod __init_subclass__(**kwargs)

This signature is included purely to help type-checkers check arguments to class declaration, which provides a way to conveniently set model_config key/value pairs.

```py from pydantic import BaseModel

class MyModel(BaseModel, extra=’allow’): … ```

However, this may be deceiving, since the _actual_ calls to __init_subclass__ will not receive any of the config arguments, and will only receive any keyword arguments passed during class initialization that are _not_ expected keys in ConfigDict. (This is due to the way ModelMetaclass.__new__ works.)

Parameters:

**kwargs (typing_extensions.Unpack[pydantic.config.ConfigDict]) – Keyword arguments passed to the class definition, which set model_config

Note

You may want to override __pydantic_init_subclass__ instead, which behaves similarly but is called after the class is fully initialized.

__iter__()

So dict(model) works.

Return type:

TupleGenerator

__repr__()
Return type:

str

__repr_args__()
Return type:

pydantic._internal._repr.ReprArgs

__repr_name__
__repr_str__
__pretty__
__rich_repr__
__str__()
Return type:

str

property __fields__: dict[str, pydantic.fields.FieldInfo]
Return type:

dict[str, pydantic.fields.FieldInfo]

property __fields_set__: set[str]
Return type:

set[str]

dict(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False)
Parameters:

  • include (IncEx | None)

  • exclude (IncEx | None)

  • by_alias (bool)

  • exclude_unset (bool)

  • exclude_defaults (bool)

  • exclude_none (bool)

Return type:

Dict[str, Any]

json(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False, encoder=PydanticUndefined, models_as_dict=PydanticUndefined, **dumps_kwargs)
Parameters:

  • include (IncEx | None)

  • exclude (IncEx | None)

  • by_alias (bool)

  • exclude_unset (bool)

  • exclude_defaults (bool)

  • exclude_none (bool)

  • encoder (Callable[[Any], Any] | None)

  • models_as_dict (bool)

  • dumps_kwargs (Any)

Return type:

str

classmethod parse_obj(obj)
Parameters:

obj (Any)

Return type:

typing_extensions.Self

classmethod parse_raw(b, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)
Parameters:

  • b (str | bytes)

  • content_type (str | None)

  • encoding (str)

  • proto (pydantic.deprecated.parse.Protocol | None)

  • allow_pickle (bool)

Return type:

typing_extensions.Self

classmethod parse_file(path, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)
Parameters:

  • path (str | pathlib.Path)

  • content_type (str | None)

  • encoding (str)

  • proto (pydantic.deprecated.parse.Protocol | None)

  • allow_pickle (bool)

Return type:

typing_extensions.Self

classmethod from_orm(obj)
Parameters:

obj (Any)

Return type:

typing_extensions.Self

classmethod construct(_fields_set=None, **values)
Parameters:

  • _fields_set (set[str] | None)

  • values (Any)

Return type:

typing_extensions.Self

copy(*, include=None, exclude=None, update=None, deep=False)

Returns a copy of the model.

!!! warning “Deprecated”

This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`py data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

  • include (pydantic._internal._utils.AbstractSetIntStr | pydantic._internal._utils.MappingIntStrAny | None) – Optional set or mapping specifying which fields to include in the copied model.

  • exclude (pydantic._internal._utils.AbstractSetIntStr | pydantic._internal._utils.MappingIntStrAny | None) – Optional set or mapping specifying which fields to exclude in the copied model.

  • update (Dict[str, Any] | None) – Optional dictionary of field-value pairs to override field values in the copied model.

  • deep (bool) – If True, the values of fields that are Pydantic models will be deep-copied.

Returns:

A copy of the model with included, excluded and updated fields as specified.

Return type:

typing_extensions.Self

classmethod schema(by_alias=True, ref_template=DEFAULT_REF_TEMPLATE)
Parameters:

  • by_alias (bool)

  • ref_template (str)

Return type:

Dict[str, Any]

classmethod schema_json(*, by_alias=True, ref_template=DEFAULT_REF_TEMPLATE, **dumps_kwargs)
Parameters:

  • by_alias (bool)

  • ref_template (str)

  • dumps_kwargs (Any)

Return type:

str

classmethod validate(value)
Parameters:

value (Any)

Return type:

typing_extensions.Self

classmethod update_forward_refs(**localns)
Parameters:

localns (Any)

Return type:

None

_iter(*args, **kwargs)
Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

Any

_copy_and_set_values(*args, **kwargs)
Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

Any

classmethod _get_value(*args, **kwargs)
Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

Any

_calculate_keys(*args, **kwargs)
Parameters:

  • args (Any)

  • kwargs (Any)

Return type:

Any

medcat.stats.kfold._add_helper(joined, single)
Parameters:

  • joined (list[dict[str, PerCUIMetrics]])

  • single (list[dict[str, int]])

Return type:

None

medcat.stats.kfold._add_weighted_helper(joined, single, cui2count)
Parameters:

  • joined (list[dict[str, PerCUIMetrics]])

  • single (list[dict[str, float]])

  • cui2count (dict[str, int])

Return type:

None

medcat.stats.kfold.get_metrics_mean(metrics, include_std)

The the mean of the provided metrics.

Parameters:

  • metrics (list[tuple[dict, dict, dict, dict, dict, dict, dict, dict]) – The metrics.

  • include_std (bool) – Whether to include the standard deviation.

Returns:

  • fps (dict) – False positives for each CUI.

  • fns (dict) – False negatives for each CUI.

  • tps (dict) – True positives for each CUI.

  • cui_prec (dict) – Precision for each CUI.

  • cui_rec (dict) – Recall for each CUI.

  • cui_f1 (dict) – F1 for each CUI.

  • cui_counts (dict) – Number of occurrence for each CUI.

  • examples (dict) – Examples for each of the fp, fn, tp. Format will be examples[‘fp’][‘cui’][<list_of_examples>].

Return type:

tuple[dict, dict, dict, dict, dict, dict, dict, dict]

medcat.stats.kfold.get_k_fold_stats(cat, mct_export_data, k=3, use_project_filters=False, split_type=SplitType.DOCUMENTS_WEIGHTED, include_std=False, *args, **kwargs)

Get the k-fold stats for the model with the specified data.

First this will split the MCT export into k folds. You can do this either per document or per-annotation.

For each of the k folds, it will start from the base model, train it with with the other k-1 folds and record the metrics. After that the base model state is restored before doing the next fold. After all the folds have been done, the metrics are averaged.

Parameters:

  • cat (CAT) – The model pack.

  • mct_export_data (MedCATTrainerExport) – The MCT export.

  • k (int) – The number of folds. Defaults to 3.

  • use_project_filters (bool) – Whether to use per project filters. Defaults to False.

  • split_type (SplitType) – Whether to use annodations or docs. Defaults to DOCUMENTS_WEIGHTED.

  • include_std (bool) – Whether to include stanrdard deviation. Defaults to False.

  • *args – Arguments passed to the CAT.train_supervised_raw method.

  • **kwargs – Keyword arguments passed to the CAT.train_supervised_raw method.

Returns:

tuple – The averaged metrics. Potentially with their corresponding standard deviations.

Return type:

tuple