medcat.components.ner.trf.deid ============================== .. py:module:: medcat.components.ner.trf.deid .. autoapi-nested-parse:: De-identification model. This describes a wrapper on the regular CAT model. The idea is to simplify the use of a DeId-specific model. It tackles two use cases 1) Creation of a deid model 2) Loading and use of a deid model I.e for use case 1: Instead of: cat = CAT(cdb=ner.cdb, addl_ner=ner) You can use: deid = DeIdModel.create(ner) And for use case 2: Instead of: cat = CAT.load_model_pack(model_pack_path) anon_text = deid_text(cat, text) You can use: deid = DeIdModel.load_model_pack(model_pack_path) anon_text = deid.deid_text(text) Or if/when structured output is desired: deid = DeIdModel.load_model_pack(model_pack_path) anon_doc = deid(text) # the spacy document The wrapper also exposes some CAT parts directly: - config - cdb Attributes ---------- .. autoapisummary:: medcat.components.ner.trf.deid.logger Classes ------- .. autoapisummary:: medcat.components.ner.trf.deid.CAT medcat.components.ner.trf.deid.CDB medcat.components.ner.trf.deid.ConfigTransformersNER medcat.components.ner.trf.deid.CoreComponentType medcat.components.ner.trf.deid.NerModel medcat.components.ner.trf.deid.TransformersNER medcat.components.ner.trf.deid.Entity medcat.components.ner.trf.deid.DeIdModel Functions --------- .. autoapisummary:: medcat.components.ner.trf.deid.replace_entities_in_text medcat.components.ner.trf.deid.match_rules medcat.components.ner.trf.deid.merge_all_preds medcat.components.ner.trf.deid.merge_preds Module Contents --------------- .. py:class:: CAT(cdb, vocab = None, config = None, model_load_path = None) Bases: :py:obj:`medcat.storage.serialisables.AbstractSerialisable` This is a collection of serialisable model parts. .. py:method:: __init__(cdb, vocab = None, config = None, model_load_path = None) .. py:attribute:: cdb .. py:attribute:: vocab :value: None .. py:attribute:: config :value: None .. py:attribute:: _trainer :type: Optional[medcat.trainer.Trainer] :value: None .. py:attribute:: _pipeline .. py:attribute:: usage_monitor .. py:method:: _recreate_pipe(model_load_path = None) .. py:method:: get_init_attrs() :classmethod: .. py:method:: ignore_attrs() :classmethod: .. py:method:: __call__(text) .. py:method:: _ensure_not_training() Method to ensure config is not set to train. `config.components.linking.train` should only be True while training and not during inference. This aalso corrects the setting if necessary. .. py:method:: get_entities(text: str, only_cui: Literal[False] = False) -> medcat.data.entities.Entities get_entities(text: str, only_cui: Literal[True] = True) -> medcat.data.entities.OnlyCUIEntities get_entities(text: str, only_cui: bool = False) -> Union[dict, medcat.data.entities.Entities, medcat.data.entities.OnlyCUIEntities] Get the entities recognised and linked within the provided text. This will run the text through the pipeline and annotated the recognised and linked entities. :param text: The text to use. :type text: str :param only_cui: Whether to only output the CUIs rather than the entire context. Defaults to False. :type only_cui: bool, optional :Returns: **Union[dict, Entities, OnlyCUIEntities]** -- The entities found and linked within the text. .. py:method:: _mp_worker_func(texts_and_indices) .. py:method:: _generate_batches_by_char_length(text_iter, batch_size_chars, only_cui) .. py:method:: _generate_batches(text_iter, batch_size, batch_size_chars, only_cui) .. py:method:: _generate_simple_batches(text_iter, batch_size, only_cui) .. py:method:: _mp_one_batch_per_process(executor, batch_iter, external_processes) .. py:method:: get_entities_multi_texts(texts, only_cui = False, n_process = 1, batch_size = -1, batch_size_chars = 1000000) Get entities from multiple texts (potentially in parallel). If `n_process` > 1, `n_process - 1` new processes will be created and data will be processed on those as well as the main process in parallel. :param texts: The input text. Either an iterable of raw text or one with in the format of `(text_index, text)`. :type texts: Union[Iterable[str], Iterable[tuple[str, str]]] :param only_cui: Whether to only return CUIs rather than other information like start/end and annotated value. Defaults to False. :type only_cui: bool :param n_process: Number of processes to use. Defaults to 1. :type n_process: int :param batch_size: The number of texts to batch at a time. A batch of the specified size will be given to each worker process. Defaults to -1 and in this case the character count will be used instead. :type batch_size: int :param batch_size_chars: The maximum number of characters to process in a batch. Each process will be given batch of texts with a total number of characters not exceeding this value. Defaults to 1,000,000 characters. Set to -1 to disable. :type batch_size_chars: int :Yields: *Iterator[tuple[str, Union[dict, Entities, OnlyCUIEntities]]]* -- The results in the format of (text_index, entities). .. py:method:: _get_entity(ent, doc_tokens, cui) .. py:method:: get_addon_output(ent) Get the addon output for the entity. This includes a key-value pair for each addon that provides some. Sometimes same-type addons may combine their output under the same key. :param ent: The entity in quesiton. :type ent: MutableEntity :raises ValueError: If unable to merge multiple addon output. :Returns: **dict[str, dict]** -- All the addon output. .. py:method:: _doc_to_out_entity(ent, doc_tokens, only_cui) .. py:method:: _doc_to_out(doc, only_cui, out_with_text = False) .. py:property:: trainer The trainer object. .. py:method:: save_model_pack(target_folder, pack_name = DEFAULT_PACK_NAME, serialiser_type = 'dill', make_archive = True, only_archive = False, add_hash_to_pack_name = True, change_description = None) Save model pack. The resulting model pack name will have the hash of the model pack in its name if (and only if) the default model pack name is used. :param target_folder: The folder to save the pack in. :type target_folder: str :param pack_name: The model pack name. Defaults to DEFAULT_PACK_NAME. :type pack_name: str, optional :param serialiser_type: The serialiser type. Defaults to 'dill'. :type serialiser_type: Union[str, AvailableSerialisers], optional :param make_archive: Whether to make the arhive /.zip file. Defaults to True. :type make_archive: bool :param only_archive: Whether to clear the non-compressed folder. Defaults to False. :type only_archive: bool :param add_hash_to_pack_name: Whether to add the hash to the pack name. This is only relevant if pack_name is specified. Defaults to True. :type add_hash_to_pack_name: bool :param change_description: If provided, this the description will be added to the model description. Defaults to None. :type change_description: Optional[str] :Returns: **str** -- The final model pack path. .. py:method:: _get_hash() .. py:method:: _versioning(change_description) .. py:method:: attempt_unpack(zip_path) :classmethod: Attempt unpack the zip to a folder and get the model pack path. If the folder already exists, no unpacking is done. :param zip_path: The ZIP path :type zip_path: str :Returns: **str** -- The model pack path .. py:method:: load_model_pack(model_pack_path) :classmethod: Load the model pack from file. :param model_pack_path: The model pack path. :type model_pack_path: str :raises ValueError: If the saved data does not represent a model pack. :Returns: **CAT** -- The loaded model pack. .. py:method:: load_cdb(model_pack_path) :classmethod: Loads the concept database from the provided model pack path :param model_pack_path: path to model pack, zip or dir. :type model_pack_path: str :Returns: **CDB** -- The loaded concept database .. py:method:: get_model_card(as_dict: Literal[True]) -> medcat.data.model_card.ModelCard get_model_card(as_dict: Literal[False]) -> str Get the model card either a (nested) `dict` or a json string. :param as_dict: Whether to return as dict. Defaults to False. :type as_dict: bool :Returns: **Union[str, ModelCard]** -- The model card. .. py:method:: __eq__(other) .. py:method:: add_addon(addon) .. py:method:: get_strategy() .. py:method:: include_properties() :classmethod: .. py:class:: CDB(config) Bases: :py:obj:`medcat.storage.serialisables.AbstractSerialisable` The abstract serialisable base class. This defines some common defaults. .. py:method:: __init__(config) .. py:attribute:: config .. py:attribute:: cui2info :type: dict[str, medcat.cdb.concepts.CUIInfo] .. py:attribute:: name2info :type: dict[str, medcat.cdb.concepts.NameInfo] .. py:attribute:: type_id2info :type: dict[str, medcat.cdb.concepts.TypeInfo] .. py:attribute:: token_counts :type: dict[str, int] .. py:attribute:: addl_info :type: dict[str, Any] .. py:attribute:: _subnames :type: set[str] .. py:attribute:: is_dirty :value: False .. py:attribute:: has_changed_names :value: False .. py:method:: get_init_attrs() :classmethod: .. py:method:: _reset_subnames() .. py:method:: has_subname(name) Whether the CDB has the specified subname. :param name: The subname to check. :type name: str :Returns: **bool** -- Whether the subname is present in this CDB. .. py:method:: get_name(cui) Returns preferred name if it exists, otherwise it will return the longest name assigned to the concept. :param cui: Concept ID or unique identifier in this database. :type cui: str :Returns: **str** -- The name of the concept. .. py:method:: weighted_average_function(step) Get the weighted average for steop. :param step: The steop. :type step: int :Returns: **float** -- The weighted average. .. py:method:: add_types(types) Add type info to CDB. :param types: The raw type info. :type types: Iterable[tuple[str, str]] .. py:method:: add_names(cui, names, name_status = ST.AUTOMATIC, full_build = False) Adds a name to an existing concept. :param cui: Concept ID or unique identifier in this database, all concepts that have the same CUI will be merged internally. :type cui: str :param names: Names for this concept, or the value that if found in free text can be linked to this concept. Names is an dict like: `{name: {'tokens': tokens, 'snames': snames, 'raw_name': raw_name}, ...}` Names should be generated by helper function 'medcat.preprocessing.cleaners.prepare_name' :type names: dict[str, NameDescriptor] :param name_status: One of `P`, `N`, `A`. Defaults to 'A'. :type name_status: str :param full_build: If True the dictionary self.addl_info will also be populated, contains a lot of extra information about concepts, but can be very memory consuming. This is not necessary for normal functioning of MedCAT (Default value `False`). :type full_build: bool .. py:method:: _add_concept_names(cui, names, name_status) .. py:method:: _add_full_build(cui, names, ontologies, description, type_ids) .. py:method:: _add_concept(cui, names, ontologies, name_status, type_ids, description, full_build = False) Add a concept to internal Concept Database (CDB). Depending on what you are providing this will add a large number of properties for each concept. :param cui: Concept ID or unique identifier in this database, all concepts that have the same CUI will be merged internally. :type cui: str :param names: Names for this concept, or the value that if found in free text can be linked to this concept. Names is a dict like: `{name: {'tokens': tokens, 'snames': snames, 'raw_name': raw_name}, ...}` Names should be generated by helper function 'medcat.preprocessing.cleaners.prepare_name' :type names: dict[str, NameDescriptor] :param ontologies: ontologies in which the concept exists (e.g. SNOMEDCT, HPO) :type ontologies: set[str] :param name_status: One of `P`, `N`, `A` :type name_status: str :param type_ids: Semantic type identifier (have a look at TUIs in UMLS or SNOMED-CT) :type type_ids: set[str] :param description: Description of this concept. :type description: str :param full_build: If True the dictionary self.addl_info will also be populated, contains a lot of extra information about concepts, but can be very memory consuming. This is not necessary for normal functioning of MedCAT (Default Value `False`). :type full_build: bool .. py:method:: reset_training() Will remove all training efforts - in other words all embeddings that are learnt for concepts in the current CDB. Please note that this does not remove synonyms (names) that were potentially added during supervised/online learning. .. py:method:: filter_by_cui(cuis_to_keep) Subset the core CDB fields (dictionaries/maps). Note that this will potenitally keep a bit more CUIs then in cuis_to_keep. It will first find all names that link to the cuis_to_keep and then find all CUIs that link to those names and keep all of them. This also will not remove any data from cdb.addl_info - as this field can contain data of unknown structure. :param cuis_to_keep: CUIs that will be kept, the rest will be removed (not completely, look above). :type cuis_to_keep: Collection[str] :raises Exception: If no snames and subsetting is not possible. .. py:method:: remove_cui(cui) This function takes a CUI and removes it the CDB. It also removes the CUI from name specific per_cui_status maps as well as well as removes all the names that do not correspond to any CUIs after the removal of this one. :param cui: The CUI to remove. :type cui: str .. py:method:: _remove_names(cui, names) Remove names from an existing concept - effect is this name will never again be used to link to this concept. This will only remove the name from the linker (namely name2cuis and name2cuis2status), the name will still be present everywhere else. Why? Because it is bothersome to remove it from everywhere, but could also be useful to keep the removed names in e.g. cui2names. :param cui: Concept ID or unique identifier in this database. :type cui: str :param names: Names to be removed (e.g list, set, or even a dict (in which case keys will be used)). :type names: Iterable[str] .. py:method:: __eq__(other) .. py:method:: get_cui2count_train() .. py:method:: get_name2count_train() .. py:method:: get_hash() .. py:method:: get_basic_info() .. py:method:: save(save_path, serialiser = AvailableSerialisers.dill, overwrite = False) Save CDB at path. :param save_path: The path to save at. :type save_path: str :param serialiser: The serialiser. Defaults to AvailableSerialisers.dill. :type serialiser: Union[ str, AvailableSerialisers], optional :param overwrite: Whether to allow overwriting existing files. Defaults to False. :type overwrite: bool, optional .. py:method:: load(path) :classmethod: .. py:method:: get_strategy() .. py:method:: ignore_attrs() :classmethod: .. py:method:: include_properties() :classmethod: .. py:class:: ConfigTransformersNER(/, **data) Bases: :py:obj:`medcat.config.config.SerialisableBaseModel` The transformer NER config .. py:attribute:: general :type: General .. py:class:: Config .. py:attribute:: extra :value: 'allow' .. py:attribute:: validate_assignment :value: True .. py:method:: get_hash(hasher = None) .. py:method:: get_strategy() .. py:method:: get_init_attrs() :classmethod: .. py:method:: ignore_attrs() :classmethod: .. py:method:: include_properties() :classmethod: .. py:method:: merge_config(other) Merge this config with another config's (partial) model dump. The exepctation is that the `other` dict is a partial model dump. Values specified there are overwritten into the current config. Values not specified there are left intact. The `other` config can have keys/values that do not exist in the config or sub-config. And they will be added where possible. :param other: The model dump :type other: dict :raises IncorrectConfigValues: If unable to set the attribute, trying to set incorrect value, or trying to set sub-config values in an incorrect format (non-dict). .. py:method:: load(path) :classmethod: .. py:attribute:: model_config :type: ClassVar[pydantic.config.ConfigDict] Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict]. .. py:attribute:: model_fields :type: ClassVar[Dict[str, pydantic.fields.FieldInfo]] Metadata about the fields defined on the model, mapping of field names to [`FieldInfo`][pydantic.fields.FieldInfo] objects. This replaces `Model.__fields__` from Pydantic V1. .. py:attribute:: model_computed_fields :type: ClassVar[Dict[str, pydantic.fields.ComputedFieldInfo]] A dictionary of computed field names and their corresponding `ComputedFieldInfo` objects. .. py:attribute:: __class_vars__ :type: ClassVar[set[str]] The names of the class variables defined on the model. .. py:attribute:: __private_attributes__ :type: ClassVar[Dict[str, pydantic.fields.ModelPrivateAttr]] Metadata about the private attributes of the model. .. py:attribute:: __signature__ :type: ClassVar[inspect.Signature] The synthesized `__init__` [`Signature`][inspect.Signature] of the model. .. py:attribute:: __pydantic_complete__ :type: ClassVar[bool] :value: False Whether model building is completed, or if there are still undefined fields. .. py:attribute:: __pydantic_core_schema__ :type: ClassVar[pydantic_core.CoreSchema] The core schema of the model. .. py:attribute:: __pydantic_custom_init__ :type: ClassVar[bool] Whether the model has a custom `__init__` method. .. py:attribute:: __pydantic_decorators__ :type: ClassVar[pydantic._internal._decorators.DecoratorInfos] Metadata containing the decorators defined on the model. This replaces `Model.__validators__` and `Model.__root_validators__` from Pydantic V1. .. py:attribute:: __pydantic_generic_metadata__ :type: ClassVar[pydantic._internal._generics.PydanticGenericMetadata] Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these. .. py:attribute:: __pydantic_parent_namespace__ :type: ClassVar[Dict[str, Any] | None] :value: None Parent namespace of the model, used for automatic rebuilding of models. .. py:attribute:: __pydantic_post_init__ :type: ClassVar[None | Literal['model_post_init']] The name of the post-init method for the model, if defined. .. py:attribute:: __pydantic_root_model__ :type: ClassVar[bool] :value: False Whether the model is a [`RootModel`][pydantic.root_model.RootModel]. .. py:attribute:: __pydantic_serializer__ :type: ClassVar[pydantic_core.SchemaSerializer] The `pydantic-core` `SchemaSerializer` used to dump instances of the model. .. py:attribute:: __pydantic_validator__ :type: ClassVar[pydantic_core.SchemaValidator | pydantic.plugin._schema_validator.PluggableSchemaValidator] The `pydantic-core` `SchemaValidator` used to validate instances of the model. .. py:attribute:: __pydantic_extra__ :type: dict[str, Any] | None A dictionary containing extra values, if [`extra`][pydantic.config.ConfigDict.extra] is set to `'allow'`. .. py:attribute:: __pydantic_fields_set__ :type: set[str] The names of fields explicitly set during instantiation. .. py:attribute:: __pydantic_private__ :type: dict[str, Any] | None Values of private attributes set on the model instance. .. py:attribute:: __slots__ :value: ('__dict__', '__pydantic_fields_set__', '__pydantic_extra__', '__pydantic_private__') .. py:method:: __init__(/, **data) Create a new model by parsing and validating input data from keyword arguments. Raises [`ValidationError`][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model. `self` is explicitly positional-only to allow `self` as a field name. .. py:property:: model_extra :type: dict[str, Any] | None Get extra fields set during validation. :Returns: **A dictionary of extra fields, or `None` if `config.extra` is not set to `"allow"`.** .. py:property:: model_fields_set :type: set[str] Returns the set of fields that have been explicitly set on this model instance. :Returns: **A set of strings representing the fields that have been set,** -- i.e. that were not filled from defaults. .. py:method:: model_construct(_fields_set = None, **values) :classmethod: Creates a new instance of the `Model` class with validated data. Creates a new model setting `__dict__` and `__pydantic_fields_set__` from trusted or pre-validated data. Default values are respected, but no other validation is performed. !!! note `model_construct()` generally respects the `model_config.extra` setting on the provided model. That is, if `model_config.extra == 'allow'`, then all extra passed values are added to the model instance's `__dict__` and `__pydantic_extra__` fields. If `model_config.extra == 'ignore'` (the default), then all extra passed values are ignored. Because no validation is performed with a call to `model_construct()`, having `model_config.extra == 'forbid'` does not result in an error if extra values are passed, but they will be ignored. :param _fields_set: A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [`model_fields_set`][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the `values` argument will be used. :param values: Trusted or pre-validated data dictionary. :Returns: **A new instance of the `Model` class with validated data.** .. py:method:: model_copy(*, update = None, deep = False) Usage docs: https://docs.pydantic.dev/2.9/concepts/serialization/#model_copy Returns a copy of the model. :param update: Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data. :param deep: Set to `True` to make a deep copy of the model. :Returns: **New model instance.** .. py:method:: model_dump(*, mode = 'python', include = None, exclude = None, context = None, by_alias = False, exclude_unset = False, exclude_defaults = False, exclude_none = False, round_trip = False, warnings = True, serialize_as_any = False) Usage docs: https://docs.pydantic.dev/2.9/concepts/serialization/#modelmodel_dump Generate a dictionary representation of the model, optionally specifying which fields to include or exclude. :param mode: The mode in which `to_python` should run. If mode is 'json', the output will only contain JSON serializable types. If mode is 'python', the output may contain non-JSON-serializable Python objects. :param include: A set of fields to include in the output. :param exclude: A set of fields to exclude from the output. :param context: Additional context to pass to the serializer. :param by_alias: Whether to use the field's alias in the dictionary key if defined. :param exclude_unset: Whether to exclude fields that have not been explicitly set. :param exclude_defaults: Whether to exclude fields that are set to their default value. :param exclude_none: Whether to exclude fields that have a value of `None`. :param round_trip: If True, dumped values should be valid as input for non-idempotent types such as Json[T]. :param warnings: How to handle serialization errors. False/"none" ignores them, True/"warn" logs errors, "error" raises a [`PydanticSerializationError`][pydantic_core.PydanticSerializationError]. :param serialize_as_any: Whether to serialize fields with duck-typing serialization behavior. :Returns: **A dictionary representation of the model.** .. py:method:: model_dump_json(*, indent = None, include = None, exclude = None, context = None, by_alias = False, exclude_unset = False, exclude_defaults = False, exclude_none = False, round_trip = False, warnings = True, serialize_as_any = False) Usage docs: https://docs.pydantic.dev/2.9/concepts/serialization/#modelmodel_dump_json Generates a JSON representation of the model using Pydantic's `to_json` method. :param indent: Indentation to use in the JSON output. If None is passed, the output will be compact. :param include: Field(s) to include in the JSON output. :param exclude: Field(s) to exclude from the JSON output. :param context: Additional context to pass to the serializer. :param by_alias: Whether to serialize using field aliases. :param exclude_unset: Whether to exclude fields that have not been explicitly set. :param exclude_defaults: Whether to exclude fields that are set to their default value. :param exclude_none: Whether to exclude fields that have a value of `None`. :param round_trip: If True, dumped values should be valid as input for non-idempotent types such as Json[T]. :param warnings: How to handle serialization errors. False/"none" ignores them, True/"warn" logs errors, "error" raises a [`PydanticSerializationError`][pydantic_core.PydanticSerializationError]. :param serialize_as_any: Whether to serialize fields with duck-typing serialization behavior. :Returns: **A JSON string representation of the model.** .. py:method:: model_json_schema(by_alias = True, ref_template = DEFAULT_REF_TEMPLATE, schema_generator = GenerateJsonSchema, mode = 'validation') :classmethod: Generates a JSON schema for a model class. :param by_alias: Whether to use attribute aliases or not. :param ref_template: The reference template. :param schema_generator: To override the logic used to generate the JSON schema, as a subclass of `GenerateJsonSchema` with your desired modifications :param mode: The mode in which to generate the schema. :Returns: **The JSON schema for the given model class.** .. py:method:: model_parametrized_name(params) :classmethod: Compute the class name for parametrizations of generic classes. This method can be overridden to achieve a custom naming scheme for generic BaseModels. :param params: Tuple of types of the class. Given a generic class `Model` with 2 type variables and a concrete model `Model[str, int]`, the value `(str, int)` would be passed to `params`. :Returns: **String representing the new class where `params` are passed to `cls` as type variables.** :raises TypeError: Raised when trying to generate concrete names for non-generic models. .. py:method:: model_post_init(__context) Override this method to perform additional initialization after `__init__` and `model_construct`. This is useful if you want to do some validation that requires the entire model to be initialized. .. py:method:: model_rebuild(*, force = False, raise_errors = True, _parent_namespace_depth = 2, _types_namespace = None) :classmethod: Try to rebuild the pydantic-core schema for the model. This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails. :param force: Whether to force the rebuilding of the model schema, defaults to `False`. :param raise_errors: Whether to raise errors, defaults to `True`. :param _parent_namespace_depth: The depth level of the parent namespace, defaults to 2. :param _types_namespace: The types namespace, defaults to `None`. :Returns: * **Returns `None` if the schema is already "complete" and rebuilding was not required.** * **If rebuilding _was_ required, returns `True` if rebuilding was successful, otherwise `False`.** .. py:method:: model_validate(obj, *, strict = None, from_attributes = None, context = None) :classmethod: Validate a pydantic model instance. :param obj: The object to validate. :param strict: Whether to enforce types strictly. :param from_attributes: Whether to extract data from object attributes. :param context: Additional context to pass to the validator. :raises ValidationError: If the object could not be validated. :Returns: **The validated model instance.** .. py:method:: model_validate_json(json_data, *, strict = None, context = None) :classmethod: Usage docs: https://docs.pydantic.dev/2.9/concepts/json/#json-parsing Validate the given JSON data against the Pydantic model. :param json_data: The JSON data to validate. :param strict: Whether to enforce types strictly. :param context: Extra variables to pass to the validator. :Returns: **The validated Pydantic model.** :raises ValidationError: If `json_data` is not a JSON string or the object could not be validated. .. py:method:: model_validate_strings(obj, *, strict = None, context = None) :classmethod: Validate the given object with string data against the Pydantic model. :param obj: The object containing string data to validate. :param strict: Whether to enforce types strictly. :param context: Extra variables to pass to the validator. :Returns: **The validated Pydantic model.** .. py:method:: __get_pydantic_core_schema__(source, handler, /) :classmethod: Hook into generating the model's CoreSchema. :param source: The class we are generating a schema for. This will generally be the same as the `cls` argument if this is a classmethod. :param handler: A callable that calls into Pydantic's internal CoreSchema generation logic. :Returns: **A `pydantic-core` `CoreSchema`.** .. py:method:: __get_pydantic_json_schema__(core_schema, handler, /) :classmethod: Hook into generating the model's JSON schema. :param core_schema: A `pydantic-core` CoreSchema. You can ignore this argument and call the handler with a new CoreSchema, wrap this CoreSchema (`{'type': 'nullable', 'schema': current_schema}`), or just call the handler with the original schema. :param handler: Call into Pydantic's internal JSON schema generation. This will raise a `pydantic.errors.PydanticInvalidForJsonSchema` if JSON schema generation fails. Since this gets called by `BaseModel.model_json_schema` you can override the `schema_generator` argument to that function to change JSON schema generation globally for a type. :Returns: **A JSON schema, as a Python object.** .. py:method:: __pydantic_init_subclass__(**kwargs) :classmethod: This is intended to behave just like `__init_subclass__`, but is called by `ModelMetaclass` only after the class is actually fully initialized. In particular, attributes like `model_fields` will be present when this is called. This is necessary because `__init_subclass__` will always be called by `type.__new__`, and it would require a prohibitively large refactor to the `ModelMetaclass` to ensure that `type.__new__` was called in such a manner that the class would already be sufficiently initialized. This will receive the same `kwargs` that would be passed to the standard `__init_subclass__`, namely, any kwargs passed to the class definition that aren't used internally by pydantic. :param \*\*kwargs: Any keyword arguments passed to the class definition that aren't used internally by pydantic. .. py:method:: __class_getitem__(typevar_values) :classmethod: .. py:method:: __copy__() Returns a shallow copy of the model. .. py:method:: __deepcopy__(memo = None) Returns a deep copy of the model. .. py:method:: __getattr__(item) .. py:method:: _check_frozen(name, value) .. py:method:: __getstate__() .. py:method:: __setstate__(state) .. py:method:: __eq__(other) .. py:method:: __init_subclass__(**kwargs) :classmethod: This signature is included purely to help type-checkers check arguments to class declaration, which provides a way to conveniently set model_config key/value pairs. ```py from pydantic import BaseModel class MyModel(BaseModel, extra='allow'): ... ``` However, this may be deceiving, since the _actual_ calls to `__init_subclass__` will not receive any of the config arguments, and will only receive any keyword arguments passed during class initialization that are _not_ expected keys in ConfigDict. (This is due to the way `ModelMetaclass.__new__` works.) :param \*\*kwargs: Keyword arguments passed to the class definition, which set model_config .. note:: You may want to override `__pydantic_init_subclass__` instead, which behaves similarly but is called *after* the class is fully initialized. .. py:method:: __iter__() So `dict(model)` works. .. py:method:: __repr__() .. py:method:: __repr_args__() .. py:attribute:: __repr_name__ .. py:attribute:: __repr_str__ .. py:attribute:: __pretty__ .. py:attribute:: __rich_repr__ .. py:method:: __str__() .. py:property:: __fields__ :type: dict[str, pydantic.fields.FieldInfo] .. py:property:: __fields_set__ :type: set[str] .. py:method:: dict(*, include = None, exclude = None, by_alias = False, exclude_unset = False, exclude_defaults = False, exclude_none = False) .. py:method:: json(*, include = None, exclude = None, by_alias = False, exclude_unset = False, exclude_defaults = False, exclude_none = False, encoder = PydanticUndefined, models_as_dict = PydanticUndefined, **dumps_kwargs) .. py:method:: parse_obj(obj) :classmethod: .. py:method:: parse_raw(b, *, content_type = None, encoding = 'utf8', proto = None, allow_pickle = False) :classmethod: .. py:method:: parse_file(path, *, content_type = None, encoding = 'utf8', proto = None, allow_pickle = False) :classmethod: .. py:method:: from_orm(obj) :classmethod: .. py:method:: construct(_fields_set = None, **values) :classmethod: .. py:method:: copy(*, include = None, exclude = None, update = None, deep = False) Returns a copy of the model. !!! warning "Deprecated" This method is now deprecated; use `model_copy` instead. If you need `include` or `exclude`, use: ```py data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) ``` :param include: Optional set or mapping specifying which fields to include in the copied model. :param exclude: Optional set or mapping specifying which fields to exclude in the copied model. :param update: Optional dictionary of field-value pairs to override field values in the copied model. :param deep: If True, the values of fields that are Pydantic models will be deep-copied. :Returns: **A copy of the model with included, excluded and updated fields as specified.** .. py:method:: schema(by_alias = True, ref_template = DEFAULT_REF_TEMPLATE) :classmethod: .. py:method:: schema_json(*, by_alias = True, ref_template = DEFAULT_REF_TEMPLATE, **dumps_kwargs) :classmethod: .. py:method:: validate(value) :classmethod: .. py:method:: update_forward_refs(**localns) :classmethod: .. py:method:: _iter(*args, **kwargs) .. py:method:: _copy_and_set_values(*args, **kwargs) .. py:method:: _get_value(*args, **kwargs) :classmethod: .. py:method:: _calculate_keys(*args, **kwargs) .. py:class:: CoreComponentType Bases: :py:obj:`enum.Enum` Generic enumeration. Derive from this class to define new enumerations. .. py:attribute:: tagging .. py:attribute:: token_normalizing .. py:attribute:: ner .. py:attribute:: linking .. py:method:: __new__(value) .. py:method:: _generate_next_value_(start, count, last_values) Generate the next value when not given. name: the name of the member start: the initial start value or None count: the number of existing members last_value: the last value assigned or None .. py:method:: _missing_(value) :classmethod: .. py:method:: __repr__() .. py:method:: __str__() .. py:method:: __dir__() Returns all members and all public methods .. py:method:: __format__(format_spec) Returns format using actual value type unless __str__ has been overridden. .. py:method:: __hash__() .. py:method:: __reduce_ex__(proto) .. py:method:: name() The name of the Enum member. .. py:method:: value() The value of the Enum member. .. py:class:: NerModel(cat) The NER model. This wraps a CAT instance and simplifies its use as a NER model. It provides methods for creating one from a TransformersNER as well as loading from a model pack (along with some validation). It also exposes some useful parts of the CAT it wraps such as the config and the concept database. .. py:method:: __init__(cat) .. py:attribute:: cat .. py:method:: train(json_path, *args, **kwargs) Train the underlying transformers NER model. All the extra arguments are passed to the TransformersNER train method. :param json_path: The JSON file path to read the training data from. :type json_path: Union[str, list, None] :param \*args: Additional arguments for TransformersNER.train . :param \*\*kwargs: Additional keyword arguments for TransformersNER.train . :Returns: **Tuple[Any, Any, Any]** -- df, examples, dataset .. py:method:: eval(json_path, *args, **kwargs) Evaluate the underlying transformers NER model. All the extra arguments are passed to the TransformersNER eval method. :param json_path: The JSON file path to read the training data from. :type json_path: Union[str, list, None] :param \*args: Additional arguments for TransformersNER.eval . :param \*\*kwargs: Additional keyword arguments for TransformersNER.eval . :Returns: **Tuple[Any, Any, Any]** -- df, examples, dataset .. py:method:: __call__(text, *args, **kwargs) Get the annotated document for text. Undefined arguments and keyword arguments get passed on to the equivalent `CAT` method. :param text: The input text. :type text: Optional[str] :param \*args: Additional arguments for cat.__call__ . :param \*\*kwargs: Additional keyword arguments for cat.__call__ . :Returns: **Optional[Doc]** -- The annotated document. .. py:method:: get_entities(text, *args, **kwargs) Gets the entities recognized within a given text. The output format is identical to `CAT.get_entities`. Undefined arguments and keyword arguments get passed on to CAT.get_entities. :param text: The input text. :type text: str :param \*args: Additional arguments for cat.get_entities . :param \*\*kwargs: Additional keyword arguments for cat.get_entities . :Returns: **dict** -- The output entities. .. py:method:: add_new_concepts(cui2preferred_name, with_random_init = False) Add new concepts to the model and the concept database. Invoking this requires subsequent retraining on the model. :param cui2preferred_name: Dictionary where each key is the literal ID of the concept to be added and each value is its preferred name. :type cui2preferred_name: dict[str, str] :param with_random_init: Whether to use the random init strategy for the new concepts. Defaults to False. :type with_random_init: bool .. py:property:: trf_ner :type: medcat.components.ner.trf.transformers_ner.TransformersNER .. py:property:: config :type: medcat.config.Config .. py:property:: cdb :type: medcat.cdb.CDB .. py:method:: load_model_pack(model_pack_path, config = None) :classmethod: Load NER model from model pack. The method first wraps the loaded CAT instance. :param config: Config for DeId model pack (primarily for stride of overlap window) :param model_pack_path: The model pack path. :type model_pack_path: str :Returns: **NerModel** -- The resulting DeI model. .. py:function:: replace_entities_in_text(text, entities, get_cui_name, redact = False) .. py:class:: TransformersNER(cdb, base_tokenizer, component, config = None, training_arguments=None) Bases: :py:obj:`medcat.components.types.AbstractCoreComponent` Base class for protocol classes. Protocol classes are defined as:: class Proto(Protocol): def meth(self) -> int: ... Such classes are primarily used with static type checkers that recognize structural subtyping (static duck-typing), for example:: class C: def meth(self) -> int: return 0 def func(x: Proto) -> int: return x.meth() func(C()) # Passes static type check See PEP 544 for details. Protocol classes decorated with @typing.runtime_checkable act as simple-minded runtime protocols that check only the presence of given attributes, ignoring their type signatures. Protocol classes can be generic, they are defined as:: class GenProto(Protocol[T]): def meth(self) -> T: ... .. py:attribute:: name :value: 'transformers_ner' The name of the component. .. py:attribute:: _def_serialiser .. py:method:: __init__(cdb, base_tokenizer, component, config = None, training_arguments=None) .. py:attribute:: _component .. py:method:: create_new(cdb, base_tokenizer, config = None, training_arguments=None) :classmethod: .. py:method:: create_new_component(cnf, tokenizer, cdb, vocab, model_load_path) :classmethod: Create a new component or load one off disk if load path presented. This may raise an exception if the wrong type of config is provided. :param cnf: The config relevant to this components. :type cnf: ComponentConfig :param tokenizer: The base tokenizer. :type tokenizer: BaseTokenizer :param cdb: The CDB. :type cdb: CDB :param vocab: The Vocab. :type vocab: Vocab :param model_load_path: Model load path (if present). :type model_load_path: Optional[str] :Returns: **Self** -- The new components. .. py:method:: load_existing(cdb, base_tokenizer, load_path, training_arguments=None, config = None) :classmethod: .. py:method:: get_type() .. py:property:: should_save :type: bool .. py:method:: save(folder, overwrite = False) .. py:method:: __call__(doc) .. py:method:: get_folder_name() .. py:method:: serialise_to(folder_path) .. py:method:: deserialise_from(folder_path, **init_kwargs) :classmethod: .. py:method:: get_strategy() .. py:method:: get_init_attrs() :classmethod: .. py:method:: ignore_attrs() :classmethod: .. py:method:: include_properties() :classmethod: .. py:attribute:: NAME_PREFIX :value: 'core_' .. py:property:: full_name :type: str Name with the component type (e.g ner, linking, meta). .. py:method:: is_core() Whether the component is a core component or not. :Returns: **bool** -- Whether this is a core component. .. py:attribute:: __slots__ :value: () .. py:attribute:: _is_protocol :value: True .. py:attribute:: _is_runtime_protocol :value: False .. py:method:: __init_subclass__(*args, **kwargs) :classmethod: .. py:method:: __class_getitem__(params) :classmethod: .. py:class:: Entity Bases: :py:obj:`TypedDict` dict() -> new empty dictionary dict(mapping) -> new dictionary initialized from a mapping object's (key, value) pairs dict(iterable) -> new dictionary initialized as if via: d = {} for k, v in iterable: d[k] = v dict(**kwargs) -> new dictionary initialized with the name=value pairs in the keyword argument list. For example: dict(one=1, two=2) .. py:attribute:: pretty_name :type: str .. py:attribute:: cui :type: str .. py:attribute:: type_ids :type: list[str] .. py:attribute:: source_value :type: str .. py:attribute:: detected_name :type: str .. py:attribute:: acc :type: float .. py:attribute:: context_similarity :type: float .. py:attribute:: start :type: int .. py:attribute:: end :type: int .. py:attribute:: id :type: int .. py:attribute:: meta_anns :type: dict[str, MetaAnnotation] .. py:attribute:: context_left :type: list[str] .. py:attribute:: context_center :type: list[str] .. py:attribute:: context_right :type: list[str] .. py:method:: __contains__() True if the dictionary has the specified key, else False. .. py:method:: __delattr__() Implement delattr(self, name). .. py:method:: __delitem__() Delete self[key]. .. py:method:: __dir__() Default dir() implementation. .. py:method:: __eq__() Return self==value. .. py:method:: __format__() Default object formatter. .. py:method:: __ge__() Return self>=value. .. py:method:: __getattribute__() Return getattr(self, name). .. py:method:: __getitem__() x.__getitem__(y) <==> x[y] .. py:method:: __gt__() Return self>value. .. py:method:: __init__() Initialize self. See help(type(self)) for accurate signature. .. py:method:: __ior__() Return self|=value. .. py:method:: __iter__() Implement iter(self). .. py:method:: __le__() Return self<=value. .. py:method:: __len__() Return len(self). .. py:method:: __lt__() Return self size of D in memory, in bytes .. py:method:: __str__() Return str(self). .. py:method:: __subclasshook__() Abstract classes can override this to customize issubclass(). This is invoked early on by abc.ABCMeta.__subclasscheck__(). It should return True, False or NotImplemented. If it returns NotImplemented, the normal algorithm is used. Otherwise, it overrides the normal algorithm (and the outcome is cached). .. py:method:: clear() D.clear() -> None. Remove all items from D. .. py:method:: copy() D.copy() -> a shallow copy of D .. py:method:: get() Return the value for key if key is in the dictionary, else default. .. py:method:: items() D.items() -> a set-like object providing a view on D's items .. py:method:: keys() D.keys() -> a set-like object providing a view on D's keys .. py:method:: pop() D.pop(k[,d]) -> v, remove specified key and return the corresponding value. If the key is not found, return the default if given; otherwise, raise a KeyError. .. py:method:: popitem() Remove and return a (key, value) pair as a 2-tuple. Pairs are returned in LIFO (last-in, first-out) order. Raises KeyError if the dict is empty. .. py:method:: setdefault() Insert key with a value of default if key is not in the dictionary. Return the value for key if key is in the dictionary, else default. .. py:method:: update() D.update([E, ]**F) -> None. Update D from dict/iterable E and F. If E is present and has a .keys() method, then does: for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k] .. py:method:: values() D.values() -> an object providing a view on D's values .. py:data:: logger .. py:class:: DeIdModel(cat) Bases: :py:obj:`medcat.components.ner.trf.model.NerModel` The DeID model. This wraps a CAT instance and simplifies its use as a de-identification model. It provides methods for creating one from a TransformersNER as well as loading from a model pack (along with some validation). It also exposes some useful parts of the CAT it wraps such as the config and the concept database. .. py:method:: __init__(cat) .. py:attribute:: cat .. py:method:: train(json_path, *args, **kwargs) Train the underlying transformers NER model. All the extra arguments are passed to the TransformersNER train method. :param json_path: The JSON file path to read the training data from. :type json_path: Union[str, list, None] :param \*args: Additional arguments for TransformersNER.train . :param \*\*kwargs: Additional keyword arguments for TransformersNER.train . :Returns: **Tuple[Any, Any, Any]** -- df, examples, dataset .. py:method:: deid_text(text, redact = False) Deidentify text and potentially redact information. De-identified text. If redaction is enabled, identifiable entities will be replaced with starts (e.g `*****`). Otherwise, the replacement will be the CUI or in other words, the type of information that was hidden (e.g [PATIENT]). :param text: The text to deidentify. :type text: str :param redact: Whether to redact the information. :type redact: bool :Returns: **str** -- The deidentified text. .. py:method:: load_model_pack(model_pack_path, config = None) :classmethod: Load DeId model from model pack. The method first loads the CAT instance. It then makes sure that the model pack corresponds to a valid DeId model. :param config: Config for DeId model pack (primarily for stride of overlap window) :param model_pack_path: The model pack path. :type model_pack_path: str :raises ValueError: If the model pack does not correspond to a DeId model. :Returns: **DeIdModel** -- The resulting DeI model. .. py:method:: _is_deid_model(cat) :classmethod: .. py:method:: _get_reason_not_deid(cat) :classmethod: .. py:method:: create(cdb, cnf) :classmethod: .. py:method:: eval(json_path, *args, **kwargs) Evaluate the underlying transformers NER model. All the extra arguments are passed to the TransformersNER eval method. :param json_path: The JSON file path to read the training data from. :type json_path: Union[str, list, None] :param \*args: Additional arguments for TransformersNER.eval . :param \*\*kwargs: Additional keyword arguments for TransformersNER.eval . :Returns: **Tuple[Any, Any, Any]** -- df, examples, dataset .. py:method:: __call__(text, *args, **kwargs) Get the annotated document for text. Undefined arguments and keyword arguments get passed on to the equivalent `CAT` method. :param text: The input text. :type text: Optional[str] :param \*args: Additional arguments for cat.__call__ . :param \*\*kwargs: Additional keyword arguments for cat.__call__ . :Returns: **Optional[Doc]** -- The annotated document. .. py:method:: get_entities(text, *args, **kwargs) Gets the entities recognized within a given text. The output format is identical to `CAT.get_entities`. Undefined arguments and keyword arguments get passed on to CAT.get_entities. :param text: The input text. :type text: str :param \*args: Additional arguments for cat.get_entities . :param \*\*kwargs: Additional keyword arguments for cat.get_entities . :Returns: **dict** -- The output entities. .. py:method:: add_new_concepts(cui2preferred_name, with_random_init = False) Add new concepts to the model and the concept database. Invoking this requires subsequent retraining on the model. :param cui2preferred_name: Dictionary where each key is the literal ID of the concept to be added and each value is its preferred name. :type cui2preferred_name: dict[str, str] :param with_random_init: Whether to use the random init strategy for the new concepts. Defaults to False. :type with_random_init: bool .. py:property:: trf_ner :type: medcat.components.ner.trf.transformers_ner.TransformersNER .. py:property:: config :type: medcat.config.Config .. py:property:: cdb :type: medcat.cdb.CDB .. py:function:: match_rules(rules, texts, cui2preferred_name) Match a set of rules - pat / cui combos as post processing labels. Uses a cat DeID model for pretty name mapping. :param rules: List of tuples of pattern and cui :type rules: list[tuple[str, str]] :param texts: List of texts to match rules on :type texts: list[str] :param cui2preferred_name: Dictionary of CUI to preferred name, likely to be cat.cdb.cui2preferred_name. :type cui2preferred_name: dict[str, str] .. rubric:: Examples >>> cat = CAT.load_model_pack(model_pack_path) ... >>> rules = [ ('(123) 456-7890', '134'), ('1234567890', '134'), ('123.456.7890', '134'), ('1234567890', '134'), ('1234567890', '134'), ] >>> texts = [ 'My phone number is (123) 456-7890', 'My phone number is 1234567890', 'My phone number is 123.456.7890', 'My phone number is 1234567890', ] >>> matches = match_rules(rules, texts, cat.cdb.cui2preferred_name) :Returns: **List[List[Dict]]** -- List of lists of predictions from `match_rules` .. py:function:: merge_all_preds(model_preds_by_text, rule_matches_per_text, accept_preds = True) Conveniance method to merge predictions from rule based and deID model predictions. :param model_preds_by_text: List of predictions from `cat.get_entities()`, then `[list(m['entities'].values()) for m in model_preds]` :type model_preds_by_text: list[list[Entity]] :param rule_matches_per_text: List of predictions from output of running `match_rules` :type rule_matches_per_text: list[list[Entity]] :param accept_preds: Uses the predicted label from the model, model_preds_by_text, over the rule matches if they overlap. Defaults to using model preds over rules. :type accept_preds: bool :Returns: **list[list[Entity]]** -- List of lists of predictions from `merge_all_preds` .. py:function:: merge_preds(model_preds, rule_matches, accept_preds = True) Merge predictions from rule based and deID model predictions. :param model_preds: predictions from `cat.get_entities()` :type model_preds: list[Entity] :param rule_matches: predictions from output of running `match_rules` on the same text :type rule_matches: list[Entity] :param accept_preds: uses the predicted label from the model, model_preds, over the rule matches if they overlap. Defaults to using model preds over rules. :type accept_preds: bool .. rubric:: Examples >>> # a list of predictions from `cat.get_entities()` >>> model_preds = [ [ {'cui': '134', 'start': 10, 'end': 20, 'acc': 1.0, 'pretty_name': 'Phone Number'}, {'cui': '134', 'start': 25, 'end': 35, 'acc': 1.0, 'pretty_name': 'Phone Number'} ] ] >>> # a list of predictions from `match_rules` >>> rule_matches = [ [ {'cui': '134', 'start': 10, 'end': 20, 'acc': 1.0, 'pretty_name': 'Phone Number'}, {'cui': '134', 'start': 25, 'end': 35, 'acc': 1.0, 'pretty_name': 'Phone Number'} ] ] >>> merged_preds = merge_preds(model_preds, rule_matches) :Returns: **list[Entity]** -- List of predictions from `merge_preds`