medcat.components.addons.meta_cat.mctokenizers.bert_tokenizer
=============================================================

.. py:module:: medcat.components.addons.meta_cat.mctokenizers.bert_tokenizer


Attributes
----------

.. autoapisummary::

   medcat.components.addons.meta_cat.mctokenizers.bert_tokenizer.FAKE_TOKENIZER_PATH


Classes
-------

.. autoapisummary::

   medcat.components.addons.meta_cat.mctokenizers.bert_tokenizer.TokenizerWrapperBase
   medcat.components.addons.meta_cat.mctokenizers.bert_tokenizer.TokenizerWrapperBERT


Module Contents
---------------

.. py:class:: TokenizerWrapperBase(hf_tokenizer = None)

   Bases: :py:obj:`abc.ABC`


   Helper class that provides a standard way to create an ABC using
   inheritance.


   .. py:attribute:: name
      :type:  str


   .. py:method:: __init__(hf_tokenizer = None)


   .. py:attribute:: hf_tokenizers
      :value: None



   .. py:method:: __call__(text: str) -> dict
                  __call__(text: list[str]) -> list[dict]


   .. py:method:: save(dir_path)
      :abstractmethod:



   .. py:method:: load(dir_path, model_variant = '', **kwargs)
      :classmethod:

      :abstractmethod:



   .. py:method:: get_size()
      :abstractmethod:



   .. py:method:: token_to_id(token)
      :abstractmethod:



   .. py:method:: get_pad_id()
      :abstractmethod:



   .. py:method:: ensure_tokenizer()


   .. py:attribute:: __slots__
      :value: ()



.. py:data:: FAKE_TOKENIZER_PATH
   :value: Multiline-String

   .. raw:: html

      <details><summary>Show Value</summary>

   .. code-block:: python

      """#
      /fake-path-not-exist#/"""

   .. raw:: html

      </details>



.. py:class:: TokenizerWrapperBERT(hf_tokenizers = None)

   Bases: :py:obj:`medcat.components.addons.meta_cat.mctokenizers.tokenizers.TokenizerWrapperBase`


   Wrapper around a huggingface BERT tokenizer so that it works with the
   MetaCAT models.

   :param transformers.models.bert.tokenization_bert_fast.BertTokenizerFast: A huggingface Fast BERT.


   .. py:attribute:: name
      :value: 'bert-tokenizer'



   .. py:method:: __init__(hf_tokenizers = None)


   .. py:method:: __call__(text: str) -> dict
                  __call__(text: list[str]) -> list[dict]


   .. py:method:: save(dir_path)


   .. py:method:: load(dir_path, model_variant = '', **kwargs)
      :classmethod:



   .. py:method:: create_new(model_variant)
      :classmethod:



   .. py:method:: get_size()


   .. py:method:: token_to_id(token)


   .. py:method:: get_pad_id()


   .. py:attribute:: hf_tokenizers
      :value: None



   .. py:method:: ensure_tokenizer()


   .. py:attribute:: __slots__
      :value: ()