medcat2.components.ner.vocab_based_ner

Attributes

logger

Classes

`MutableDocument`	The mutable parts of the document.
`CoreComponentType`	Generic enumeration.
`AbstractCoreComponent`	Base class for protocol classes.
`BaseTokenizer`	The base tokenizer protocol.
`Vocab`	Vocabulary used to store word embeddings for context similarity
`CDB`	The abstract serialisable base class.
`NER`	Base class for protocol classes.

Functions

maybe_annotate_name(tokenizer, name, tkns, doc, cdb, ...)

Given a name it will check should it be annotated based on config rules.

Module Contents

class medcat2.components.ner.vocab_based_ner.MutableDocument

Bases: Protocol

The mutable parts of the document.

Represents parts of the document that can / should be changed by the various components.

property base: BaseDocument

The base document.

Return type:: BaseDocument

property final_ents: list[MutableEntity]

The linked entities associated with the document.

This should be set by the linker.

Return type:: list[MutableEntity]

property all_ents: list[MutableEntity]

All entities recognised by NER.

This should be set by the NER component.

Return type:: list[MutableEntity]

__iter__()

Return type:: Iterator[MutableToken]

__getitem__(index: int) → MutableToken
__getitem__(index: slice) → MutableEntity

get_tokens(start_index, end_index)

Get the tokens that span the specified character indices.

Parameters:

start_index (int) – The starting character index.
end_index (int) – The ending character index.

Returns:

list[MutableToken] – The list of tokens.

Return type:

list[MutableToken]

set_addon_data(path, val)

Used to add arbitrary data to the entity.

This is generally used by addons to keep track of their data.

NB! The path used needs to be registered using the register_addon_path class method.

Parameters:

path (str) – The data ID / path.
val (Any) – The value to be added.

Return type:

None

get_addon_data(path)

Get data added to the entity.

See add_data for details.

Parameters:: path (str) – The data ID / path.
Returns:: Any – The stored value.
Return type:: Any

classmethod register_addon_path(path, def_val=None, force=True)

Register a custom/arbitrary data path.

This can be used to store arbitrary data along with the entity for use in an addon (e.g MetaCAT).

PS: If using this, it is important to use paths namespaced to the component you’re using in order to avoid conflicts.

Parameters:

path (str) – The path to be used. Should be prefixed by component name (e.g meta_cat_id for an ID tied to the meta_cat addon)
def_val (Any) – Default value. Defaults to None.
force (bool) – Whether to forcefully add the value. Defaults to True.

Return type:

None

__slots__ = ()

_is_protocol = True

_is_runtime_protocol = False

classmethod __init_subclass__(*args, **kwargs)

classmethod __class_getitem__(params)

class medcat2.components.ner.vocab_based_ner.CoreComponentType

Bases: enum.Enum

Generic enumeration.

Derive from this class to define new enumerations.

tagging

token_normalizing

ner

linking

__new__(value)

_generate_next_value_(start, count, last_values)

Generate the next value when not given.

name: the name of the member start: the initial start value or None count: the number of existing members last_value: the last value assigned or None

classmethod _missing_(value)

__repr__()

__str__()

__dir__(): Returns all members and all public methods

__format__(format_spec): Returns format using actual value type unless __str__ has been overridden.

__hash__()

__reduce_ex__(proto)

name(): The name of the Enum member.

value(): The value of the Enum member.

class medcat2.components.ner.vocab_based_ner.AbstractCoreComponent

Bases: CoreComponent

Base class for protocol classes.

Protocol classes are defined as:

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing), for example:

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:

class GenProto(Protocol[T]):
    def meth(self) -> T:
        ...

NAME_PREFIX = 'core_'

property full_name: str

Name with the component type (e.g ner, linking, meta).

Return type:: str

is_core()

Whether the component is a core component or not.

Returns:: bool – Whether this is a core component.
Return type:: bool

get_type()

Return type:: CoreComponentType

property name: str

The name of the component.

Return type:: str

__call__(doc)

Parameters:: doc (medcat2.tokenizing.tokens.MutableDocument)
Return type:: medcat2.tokenizing.tokens.MutableDocument

classmethod get_init_args(tokenizer, cdb, vocab, model_load_path)

Get the init arguments for the component.

Parameters:

tokenizer (BaseTokenizer) – The tokenizer.
cdb (CDB) – The CDB.
vocab (Vocab) – The Vocab.
model_load_path (Optional[str]) – The model load path (or None).

Returns:

list[Any] – The list of init arguments.

Return type:

list[Any]

classmethod get_init_kwargs(tokenizer, cdb, vocab, model_load_path)

Get init keyword arguments for the component.

Parameters:

tokenizer (BaseTokenizer) – The tokenizer.
cdb (CDB) – The CDB.
vocab (Vocab) – The Vocab.
model_load_path (Optional[str]) – The model load path (or None).

Returns:

dict[str, Any] – The keywrod arguments.

Return type:

dict[str, Any]

__slots__ = ()

_is_protocol = True

_is_runtime_protocol = False

classmethod __init_subclass__(*args, **kwargs)

classmethod __class_getitem__(params)

medcat2.components.ner.vocab_based_ner.maybe_annotate_name(tokenizer, name, tkns, doc, cdb, config, label='concept')

Given a name it will check should it be annotated based on config rules. If yes the annotation will be added to the doc.entities array.

Parameters:

tokenizer (BaseTokenizer) – The tokenizer (probably SpaCy).
name (str) – The name found in the text of the document.
tkns (list[MutableToken]) – Tokens that belong to this name in the spacy document.
doc (BaseDocument) – Spacy document to be annotated with named entities.
cdb (CDB) – Concept database.
config (Config) – Global config for medcat.
label (str) – Label for this name (usually concept if we are using a vocab based approach).

Returns:

Optional[BaseEntity] – The entity, if relevant.

Return type:

Optional[medcat2.tokenizing.tokens.MutableEntity]

class medcat2.components.ner.vocab_based_ner.BaseTokenizer

Bases: Protocol

The base tokenizer protocol.

create_entity(doc, token_start_index, token_end_index, label)

Create an entity from a document.

Parameters:

doc (MutableDocument) – The document to use.
token_start_index (int) – The token start index.
token_end_index (int) – The token end index.
label (str) – The label.

Returns:

MutableEntity – The resulting entity.

Return type:

medcat2.tokenizing.tokens.MutableEntity

entity_from_tokens(tokens)

Get an entity from the list of tokens.

Parameters:: tokens (list[MutableToken]) – List of tokens.
Returns:: MutableEntity – The resulting entity.
Return type:: medcat2.tokenizing.tokens.MutableEntity

__call__(text)

Parameters:: text (str)
Return type:: medcat2.tokenizing.tokens.MutableDocument

classmethod get_init_args(config)

Parameters:: config (medcat2.config.Config)
Return type:: list[Any]

classmethod get_init_kwargs(config)

Parameters:: config (medcat2.config.Config)
Return type:: dict[str, Any]

get_doc_class()

Get the document implementation class used by the tokenizer.

This can be used (e.g) to register addon paths.

Returns:: Type[MutableDocument] – The document class.
Return type:: Type[medcat2.tokenizing.tokens.MutableDocument]

get_entity_class()

Get the entity implementation class used by the tokenizer.

Returns:: Type[MutableEntity] – The entity class.
Return type:: Type[medcat2.tokenizing.tokens.MutableEntity]

__slots__ = ()

_is_protocol = True

_is_runtime_protocol = False

classmethod __init_subclass__(*args, **kwargs)

classmethod __class_getitem__(params)

class medcat2.components.ner.vocab_based_ner.Vocab

Bases: medcat2.storage.serialisables.AbstractSerialisable

Vocabulary used to store word embeddings for context similarity calculation. Also used by the spell checker - but not for fixing the spelling only for checking is something correct.

Properties:

vocab (dict[str, WordDescriptor]):

Map from word to attributes, e.g. {‘house’:: {‘vector’: <np.array>, ‘count’: <int>, …}, …}

index2word (dict[int, str]):

From word to an index - used for negative sampling

vec_index2word (dict):

Same as index2word but only words that have vectors

__init__()

Return type:: None

vocab: dict[str, WordDescriptor]

index2word: dict[int, str]

vec_index2word: dict[int, str]

cum_probs: numpy.ndarray

inc_or_add(word, cnt=1, vec=None)

Add a word or increase its count.

Parameters:

word (str) – Word to be added
cnt (int) – By how much should the count be increased, or to what should it be set if a new word. (Default value = 1)
vec (Optional[np.ndarray]) – Word vector (Default value = None)

Return type:

None

remove_all_vectors()

Remove all stored vector representations.

Return type:: None

remove_words_below_cnt(cnt)

Remove all words with frequency below cnt.

Parameters:: cnt (int) – Word count limit.
Return type:: None

_rebuild_index()

inc_wc(word, cnt=1)

Incraese word count by cnt.

Parameters:

word (str) – For which word to increase the count
cnt (int) – By how muhc to increase the count (Default value = 1)

Return type:

None

add_vec(word, vec)

Add vector to a word.

Parameters:

word (str) – To which word to add the vector.
vec (np.ndarray) – The vector to add.

Return type:

None

reset_counts(cnt=1)

Reset the count for all word to cnt.

Parameters:: cnt (int) – New count for all words in the vocab. (Default value = 1)
Return type:: None

update_counts(tokens)

Given a list of tokens update counts for words in the vocab.

Parameters:: tokens (list[str]) – Usually a large block of text split into tokens/words.
Return type:: None

add_word(word, cnt=1, vec=None, replace=True)

Add a word to the vocabulary

Parameters:

word (str) – The word to be added, it should be lemmatized and lowercased
cnt (int) – Count of this word in your dataset (Default value = 1)
vec (Optional[np.ndarray]) – The vector representation of the word (Default value = None)
replace (bool) – Will replace old vector representation (Default value = True)

Return type:

None

add_words(path, replace=True)

Adds words to the vocab from a file, the file is required to have the following format (vec being optional):

<word> <cnt>[ <vec_space_separated>]

e.g. one line: the word house with 3 dimensional vectors: house 34444 0.3232 0.123213 1.231231

Parameters:

path (str) – path to the file with words and vectors
replace (bool) – existing words in the vocabulary will be replaced. Defaults to True.

Return type:

None

init_cumsums()

Initialise cumulative sums.

This is in place of the unigram table. But similarly to it, this approach allows generating a list of indices that match the probabilistic distribution expected as per the word counts of each word.

Return type:: None

get_negative_samples(n=6, ignore_punct_and_num=False)

Get N negative samples.

Parameters:

n (int) – How many words to return (Default value = 6)
ignore_punct_and_num (bool) – Whether to ignore punctuation and numbers. Defaults to False.

Raises:

Exception – If no unigram table is present.

Returns:

list[int] – Indices for words in this vocabulary.

Return type:

list[int]

get_vectors(indices)

Parameters:: indices (list[int])
Return type:: list[numpy.ndarray]

__getitem__(word)

Parameters:: word (str)
Return type:: int

vec(word)

Parameters:: word (str)
Return type:: Optional[numpy.ndarray]

count(word)

Parameters:: word (str)
Return type:: int

item(word)

Parameters:: word (str)
Return type:: WordDescriptor

__contains__(word)

Parameters:: word (str)
Return type:: bool

__eq__(other)

Parameters:: other (Any)
Return type:: bool

get_strategy()

Return type:: SerialisingStrategy

classmethod get_init_attrs()

Return type:: list[str]

classmethod ignore_attrs()

Return type:: list[str]

classmethod include_properties()

Return type:: list[str]

class medcat2.components.ner.vocab_based_ner.CDB(config)

Bases: medcat2.storage.serialisables.AbstractSerialisable

The abstract serialisable base class.

This defines some common defaults.

Parameters:: config (medcat2.config.Config)

__init__(config)

Parameters:: config (medcat2.config.Config)
Return type:: None

config

cui2info: dict[str, medcat2.cdb.concepts.CUIInfo]

name2info: dict[str, medcat2.cdb.concepts.NameInfo]

type_id2info: dict[str, medcat2.cdb.concepts.TypeInfo]

token_counts: dict[str, int]

addl_info: dict[str, Any]

_subnames: set[str]

is_dirty = False

has_changed_names = False

classmethod get_init_attrs()

Return type:: list[str]

_reset_subnames()

has_subname(name)

Whether the CDB has the specified subname.

Parameters:: name (str) – The subname to check.
Returns:: bool – Whether the subname is present in this CDB.
Return type:: bool

get_name(cui)

Returns preferred name if it exists, otherwise it will return the longest name assigned to the concept.

Parameters:: cui (str) – Concept ID or unique identifier in this database.
Returns:: str – The name of the concept.
Return type:: str

weighted_average_function(step)

Get the weighted average for steop.

Parameters:: step (int) – The steop.
Returns:: float – The weighted average.
Return type:: float

add_types(types)

Add type info to CDB.

Parameters:: types (Iterable[tuple[str, str]]) – The raw type info.
Return type:: None

add_names(cui, names, name_status=ST.AUTOMATIC, full_build=False)

Adds a name to an existing concept.

Parameters:

cui (str) – Concept ID or unique identifier in this database, all concepts that have the same CUI will be merged internally.
names (dict[str, NameDescriptor]) –
Names for this concept, or the value that if found in free text can be linked to this concept. Names is an dict like: `{name: {‘tokens’: tokens, ‘snames’: snames,

’raw_name’: raw_name}, …}`

Names should be generated by helper function ‘medcat.preprocessing.cleaners.prepare_name’
name_status (str) – One of P, N, A. Defaults to ‘A’.
full_build (bool) – If True the dictionary self.addl_info will also be populated, contains a lot of extra information about concepts, but can be very memory consuming. This is not necessary for normal functioning of MedCAT (Default value False).

Return type:

None

_add_concept_names(cui, names, name_status)

Parameters:

cui (str)
names (dict[str, medcat2.preprocessors.cleaners.NameDescriptor])
name_status (str)

Return type:

None

_add_full_build(cui, names, ontologies, description, type_ids)

Parameters:

cui (str)
names (dict[str, medcat2.preprocessors.cleaners.NameDescriptor])
ontologies (set[str])
description (str)
type_ids (set[str])

Return type:

None

_add_concept(cui, names, ontologies, name_status, type_ids, description, full_build=False)

Add a concept to internal Concept Database (CDB). Depending on what you are providing this will add a large number of properties for each concept.

Parameters:

cui (str) – Concept ID or unique identifier in this database, all concepts that have the same CUI will be merged internally.
names (dict[str, NameDescriptor]) –
Names for this concept, or the value that if found in free text can be linked to this concept. Names is a dict like: `{name: {‘tokens’: tokens, ‘snames’: snames,

’raw_name’: raw_name}, …}`

Names should be generated by helper function ‘medcat.preprocessing.cleaners.prepare_name’
ontologies (set[str]) – ontologies in which the concept exists (e.g. SNOMEDCT, HPO)
name_status (str) – One of P, N, A
type_ids (set[str]) – Semantic type identifier (have a look at TUIs in UMLS or SNOMED-CT)
description (str) – Description of this concept.
full_build (bool) – If True the dictionary self.addl_info will also be populated, contains a lot of extra information about concepts, but can be very memory consuming. This is not necessary for normal functioning of MedCAT (Default Value False).

Return type:

None

reset_training()

Will remove all training efforts - in other words all embeddings that are learnt for concepts in the current CDB. Please note that this does not remove synonyms (names) that were potentially added during supervised/online learning.

Return type:: None

_remove_names(cui, names)

Remove names from an existing concept - effect is this name will never again be used to link to this concept. This will only remove the name from the linker (namely name2cuis and name2cuis2status), the name will still be present everywhere else. Why? Because it is bothersome to remove it from everywhere, but could also be useful to keep the removed names in e.g. cui2names.

Parameters:

cui (str) – Concept ID or unique identifier in this database.
names (Iterable[str]) – Names to be removed (e.g list, set, or even a dict (in which case keys will be used)).

Return type:

None

__eq__(other)

Parameters:: other (Any)
Return type:: bool

get_strategy()

Return type:: SerialisingStrategy

classmethod ignore_attrs()

Return type:: list[str]

classmethod include_properties()

Return type:: list[str]

medcat2.components.ner.vocab_based_ner.logger

class medcat2.components.ner.vocab_based_ner.NER(tokenizer, cdb)

Bases: medcat2.components.types.AbstractCoreComponent

Base class for protocol classes.

Protocol classes are defined as:

class Proto(Protocol):
    def meth(self) -> int:
        ...

Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing), for example:

class C:
    def meth(self) -> int:
        return 0

def func(x: Proto) -> int:
    return x.meth()

func(C())  # Passes static type check

See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:

class GenProto(Protocol[T]):
    def meth(self) -> T:
        ...

Parameters:

tokenizer (medcat2.tokenizing.tokenizers.BaseTokenizer)
cdb (medcat2.cdb.CDB)

name = 'cat_ner': The name of the component.

__init__(tokenizer, cdb)

Parameters:

tokenizer (medcat2.tokenizing.tokenizers.BaseTokenizer)
cdb (medcat2.cdb.CDB)

Return type:

None

tokenizer

cdb

config

get_type()

Return type:: medcat2.components.types.CoreComponentType

__call__(doc)

Detect candidates for concepts - linker will then be able to do the rest. It adds entities to the doc.entities and each entity can have the entity.link_candidates - that the linker will resolve.

Parameters:: doc (MutableDocument) – Spacy document to be annotated with named entities.
Returns:: doc (MutableDocument) – Spacy document with detected entities.
Return type:: medcat2.tokenizing.tokens.MutableDocument

classmethod get_init_args(tokenizer, cdb, vocab, model_load_path)

Get the init arguments for the component.

Parameters:

tokenizer (BaseTokenizer) – The tokenizer.
cdb (CDB) – The CDB.
vocab (Vocab) – The Vocab.
model_load_path (Optional[str]) – The model load path (or None).

Returns:

list[Any] – The list of init arguments.

Return type:

list[Any]

classmethod get_init_kwargs(tokenizer, cdb, vocab, model_load_path)

Get init keyword arguments for the component.

Parameters:

tokenizer (BaseTokenizer) – The tokenizer.
cdb (CDB) – The CDB.
vocab (Vocab) – The Vocab.
model_load_path (Optional[str]) – The model load path (or None).

Returns:

dict[str, Any] – The keywrod arguments.

Return type:

dict[str, Any]

NAME_PREFIX = 'core_'

property full_name: str

Name with the component type (e.g ner, linking, meta).

Return type:: str

is_core()

Whether the component is a core component or not.

Returns:: bool – Whether this is a core component.
Return type:: bool

__slots__ = ()

_is_protocol = True

_is_runtime_protocol = False

classmethod __init_subclass__(*args, **kwargs)

classmethod __class_getitem__(params)