medcat.storage.serialisers
==========================

.. py:module:: medcat.storage.serialisers


Attributes
----------

.. autoapisummary::

   medcat.storage.serialisers.DEFAULT_SCHEMA_FILE
   medcat.storage.serialisers.logger
   medcat.storage.serialisers.SER_TYPE_FILE
   medcat.storage.serialisers.MANUAL_SERIALISED_TAG
   medcat.storage.serialisers.MANUAL_SERIALISED_RE
   medcat.storage.serialisers._DEF_SER


Exceptions
----------

.. autoapisummary::

   medcat.storage.serialisers.IllegalSchemaException


Classes
-------

.. autoapisummary::

   medcat.storage.serialisers.Serialisable
   medcat.storage.serialisers.ManualSerialisable
   medcat.storage.serialisers.RemappingUnpickler
   medcat.storage.serialisers.Serialiser
   medcat.storage.serialisers.AvailableSerialisers
   medcat.storage.serialisers.DillSerialiser


Functions
---------

.. autoapisummary::

   medcat.storage.serialisers.get_all_serialisable_members
   medcat.storage.serialisers.load_schema
   medcat.storage.serialisers.save_schema
   medcat.storage.serialisers.fix_module_and_cls_name
   medcat.storage.serialisers.get_serialiser
   medcat.storage.serialisers.get_serialiser_type_from_folder
   medcat.storage.serialisers.get_serialiser_from_folder
   medcat.storage.serialisers.serialise
   medcat.storage.serialisers.deserialise


Module Contents
---------------

.. py:class:: Serialisable

   Bases: :py:obj:`Protocol`


   The base serialisable protocol.


   .. py:method:: get_strategy()

      Get the serialisation strategy.

      :Returns: **SerialisingStrategy** -- The strategy.



   .. py:method:: get_init_attrs()
      :classmethod:


      Get the names of the arguments needed for init upon deserialisation.

      :Returns: **list[str]** -- The list of init arguments' names.



   .. py:method:: ignore_attrs()
      :classmethod:


      Get the names of attributes not to serialise.

      :Returns: **list[str]** -- The attribute names that should not be serialised.



   .. py:method:: include_properties()
      :classmethod:



   .. py:attribute:: __slots__
      :value: ()



   .. py:attribute:: _is_protocol
      :value: True



   .. py:attribute:: _is_runtime_protocol
      :value: False



   .. py:method:: __init_subclass__(*args, **kwargs)
      :classmethod:



   .. py:method:: __class_getitem__(params)
      :classmethod:



.. py:class:: ManualSerialisable

   Bases: :py:obj:`Serialisable`, :py:obj:`Protocol`


   The base serialisable protocol.


   .. py:method:: serialise_to(folder_path)

      Serialise to a folder.

      :param folder_path: The folder to serialise to.
      :type folder_path: str



   .. py:method:: deserialise_from(folder_path, **init_kwargs)
      :classmethod:


      Deserialise from a specifc path.

      The init keyword arguments are generally:
      - cnf: The config relevant to the components
      - tokenizer (BaseTokenizer): The base tokenizer for the model
      - cdb (CDB): The CDB for the model
      - vocab (Vocab): The Vocab for the model
      - model_load_path (Optional[str]): The model load path,
          but not the component load path

      :param folder_path: The path to deserialsie form.
      :type folder_path: str

      :Returns: **ManualSerialisable** -- The deserialised object.



   .. py:method:: get_strategy()

      Get the serialisation strategy.

      :Returns: **SerialisingStrategy** -- The strategy.



   .. py:method:: get_init_attrs()
      :classmethod:


      Get the names of the arguments needed for init upon deserialisation.

      :Returns: **list[str]** -- The list of init arguments' names.



   .. py:method:: ignore_attrs()
      :classmethod:


      Get the names of attributes not to serialise.

      :Returns: **list[str]** -- The attribute names that should not be serialised.



   .. py:method:: include_properties()
      :classmethod:



   .. py:attribute:: __slots__
      :value: ()



   .. py:attribute:: _is_protocol
      :value: True



   .. py:attribute:: _is_runtime_protocol
      :value: False



   .. py:method:: __init_subclass__(*args, **kwargs)
      :classmethod:



   .. py:method:: __class_getitem__(params)
      :classmethod:



.. py:function:: get_all_serialisable_members(object)

   Gets all serialisable members of an object.

   This looks for public and protected members, but not private ones.
   It should also be able to return parts of lists and tuples.
   It also provides the name of each serialisable object.

   :param object: The target object.
   :type object: Any

   :Returns: **tuple[list[tuple[Serialisable, str]], dict[str, Any]]** -- list of serialisable objects along with their names


.. py:function:: load_schema(file_name)

   Loads the schema for a folder of deserialisable files from the file.

   :param file_name: The schema file
   :type file_name: str

   :Returns: **tuple[str, list[str]]** -- The class package/name along
             with the parts needed for initialising.


.. py:function:: save_schema(file_name, cls, init_parts)

   Saves the schema of a class to the specified file.

   :param file_name: The file to save to.
   :type file_name: str
   :param cls: The class in question
   :type cls: Type
   :param init_parts list[str]: The parts of the .


.. py:data:: DEFAULT_SCHEMA_FILE
   :value: '.schema.json'


.. py:exception:: IllegalSchemaException(*args)

   Bases: :py:obj:`ValueError`


   Inappropriate argument value (of correct type).


   .. py:method:: __init__(*args)

      Initialize self.  See help(type(self)) for accurate signature.



   .. py:class:: __cause__

      exception cause



   .. py:class:: __context__

      exception context



   .. py:method:: __delattr__()

      Implement delattr(self, name).



   .. py:method:: __dir__()

      Default dir() implementation.



   .. py:method:: __eq__()

      Return self==value.



   .. py:method:: __format__()

      Default object formatter.



   .. py:method:: __ge__()

      Return self>=value.



   .. py:method:: __getattribute__()

      Return getattr(self, name).



   .. py:method:: __gt__()

      Return self>value.



   .. py:method:: __hash__()

      Return hash(self).



   .. py:method:: __le__()

      Return self<=value.



   .. py:method:: __lt__()

      Return self<value.



   .. py:method:: __ne__()

      Return self!=value.



   .. py:method:: __new__()

      Create and return a new object.  See help(type) for accurate signature.



   .. py:method:: __reduce__()


   .. py:method:: __reduce_ex__()

      Helper for pickle.



   .. py:method:: __repr__()

      Return repr(self).



   .. py:method:: __setattr__()

      Implement setattr(self, name, value).



   .. py:method:: __setstate__()


   .. py:method:: __sizeof__()

      Size of object in memory, in bytes.



   .. py:method:: __str__()

      Return str(self).



   .. py:method:: __subclasshook__()

      Abstract classes can override this to customize issubclass().

      This is invoked early on by abc.ABCMeta.__subclasscheck__().
      It should return True, False or NotImplemented.  If it returns
      NotImplemented, the normal algorithm is used.  Otherwise, it
      overrides the normal algorithm (and the outcome is cached).



   .. py:class:: __suppress_context__


   .. py:class:: __traceback__


   .. py:class:: args


   .. py:method:: with_traceback()

      Exception.with_traceback(tb) --
      set self.__traceback__ to tb and return self.



.. py:function:: fix_module_and_cls_name(module_name, cls_name)

.. py:class:: RemappingUnpickler(*args, **kwds)

   Bases: :py:obj:`dill.Unpickler`


   python's Unpickler extended to interpreter sessions and more types


   .. py:method:: find_class(module, name)

      Return an object from a specified module.

      If necessary, the module will be imported. Subclasses may override
      this method (e.g. to restrict unpickling of arbitrary classes and
      functions).

      This method is called whenever a class or a function object is
      needed.  Both arguments passed are str objects.



   .. py:attribute:: _session
      :value: False



   .. py:method:: __init__(*args, **kwds)

      Initialize self.  See help(type(self)) for accurate signature.



   .. py:attribute:: _main


   .. py:attribute:: _ignore
      :value: False



   .. py:method:: load()

      Load a pickle.

      Read a pickled object representation from the open file object given
      in the constructor, and return the reconstituted object hierarchy
      specified therein.



   .. py:method:: __delattr__()

      Implement delattr(self, name).



   .. py:method:: __dir__()

      Default dir() implementation.



   .. py:method:: __eq__()

      Return self==value.



   .. py:method:: __format__()

      Default object formatter.



   .. py:method:: __ge__()

      Return self>=value.



   .. py:method:: __getattribute__()

      Return getattr(self, name).



   .. py:method:: __gt__()

      Return self>value.



   .. py:method:: __hash__()

      Return hash(self).



   .. py:method:: __le__()

      Return self<=value.



   .. py:method:: __lt__()

      Return self<value.



   .. py:method:: __ne__()

      Return self!=value.



   .. py:method:: __reduce__()

      Helper for pickle.



   .. py:method:: __reduce_ex__()

      Helper for pickle.



   .. py:method:: __repr__()

      Return repr(self).



   .. py:method:: __setattr__()

      Implement setattr(self, name, value).



   .. py:method:: __sizeof__()

      Returns size in memory, in bytes.



   .. py:method:: __str__()

      Return str(self).



   .. py:class:: memo


   .. py:class:: persistent_load


   .. py:attribute:: _buffers


   .. py:attribute:: _file_readline


   .. py:attribute:: _file_read


   .. py:attribute:: encoding
      :value: 'ASCII'



   .. py:attribute:: errors
      :value: 'strict'



   .. py:attribute:: proto
      :value: 0



   .. py:attribute:: fix_imports
      :value: True



   .. py:method:: pop_mark()


   .. py:attribute:: dispatch


   .. py:method:: load_proto()


   .. py:method:: load_frame()


   .. py:method:: load_persid()


   .. py:method:: load_binpersid()


   .. py:method:: load_none()


   .. py:method:: load_false()


   .. py:method:: load_true()


   .. py:method:: load_int()


   .. py:method:: load_binint()


   .. py:method:: load_binint1()


   .. py:method:: load_binint2()


   .. py:method:: load_long()


   .. py:method:: load_long1()


   .. py:method:: load_long4()


   .. py:method:: load_float()


   .. py:method:: load_binfloat()


   .. py:method:: _decode_string(value)


   .. py:method:: load_string()


   .. py:method:: load_binstring()


   .. py:method:: load_binbytes()


   .. py:method:: load_unicode()


   .. py:method:: load_binunicode()


   .. py:method:: load_binunicode8()


   .. py:method:: load_binbytes8()


   .. py:method:: load_bytearray8()


   .. py:method:: load_next_buffer()


   .. py:method:: load_readonly_buffer()


   .. py:method:: load_short_binstring()


   .. py:method:: load_short_binbytes()


   .. py:method:: load_short_binunicode()


   .. py:method:: load_tuple()


   .. py:method:: load_empty_tuple()


   .. py:method:: load_tuple1()


   .. py:method:: load_tuple2()


   .. py:method:: load_tuple3()


   .. py:method:: load_empty_list()


   .. py:method:: load_empty_dictionary()


   .. py:method:: load_empty_set()


   .. py:method:: load_frozenset()


   .. py:method:: load_list()


   .. py:method:: load_dict()


   .. py:method:: _instantiate(klass, args)


   .. py:method:: load_inst()


   .. py:method:: load_obj()


   .. py:method:: load_newobj()


   .. py:method:: load_newobj_ex()


   .. py:method:: load_global()


   .. py:method:: load_stack_global()


   .. py:method:: load_ext1()


   .. py:method:: load_ext2()


   .. py:method:: load_ext4()


   .. py:method:: get_extension(code)


   .. py:method:: load_reduce()


   .. py:method:: load_pop()


   .. py:method:: load_pop_mark()


   .. py:method:: load_dup()


   .. py:method:: load_get()


   .. py:method:: load_binget()


   .. py:method:: load_long_binget()


   .. py:method:: load_put()


   .. py:method:: load_binput()


   .. py:method:: load_long_binput()


   .. py:method:: load_memoize()


   .. py:method:: load_append()


   .. py:method:: load_appends()


   .. py:method:: load_setitem()


   .. py:method:: load_setitems()


   .. py:method:: load_additems()


   .. py:method:: load_build()


   .. py:method:: load_mark()


   .. py:method:: load_stop()


.. py:data:: logger

.. py:data:: SER_TYPE_FILE
   :value: '.serialised_by'


.. py:data:: MANUAL_SERIALISED_TAG
   :value: 'MANUALLY_SERIALISED:'


.. py:data:: MANUAL_SERIALISED_RE

.. py:class:: Serialiser

   Bases: :py:obj:`abc.ABC`


   The abstract serialiser base class.

   This class is responsible for both serialising and deserialising.


   .. py:attribute:: RAW_FILE
      :value: 'raw_dict.dat'



   .. py:property:: ser_type
      :type: AvailableSerialisers

      :abstractmethod:


      The serialiser type.


   .. py:method:: serialise(raw_parts, target_file)
      :abstractmethod:


      Serialise the raw attributes / objects.

      :param raw_parts: The raw objects to serialise.
      :type raw_parts: dict[str, Any]
      :param target_file: The file name to write to.
      :type target_file: str



   .. py:method:: deserialise(target_file)
      :abstractmethod:


      Deserialise data written to the specified file.

      :param target_file: The file to read from.
      :type target_file: str

      :Returns: **dict[str, Any]** -- The deserialised raw attributes / objects.



   .. py:method:: get_ser_type_file(folder)
      :classmethod:



   .. py:method:: save_ser_type_file(folder)

      Save the serialiser type into the specified folder.

      :param folder: The folder to use.
      :type folder: str



   .. py:method:: get_manually_serialised_path(folder)
      :classmethod:



   .. py:method:: check_ser_type(folder)

      Check that the folder contains data serialised by this serialiser.

      :param folder: Target folder.
      :type folder: str

      :raises TypeError: If the folder was not serialised by this serialiser.



   .. py:method:: serialise_all(obj, target_folder, overwrite = False)

      Serialise the entire object into the target folder.

      This finds the serialisable parts (attributes) of the object and calls
      the same method on them recursively.
      It also finds the raw attributes (if any) and serialises them.

      :param obj: The object to serialise.
      :type obj: Serialisable
      :param target_folder: The target folder.
      :type target_folder: str
      :param overwrite: Whether to allow overwriting. Defaults to False.
      :type overwrite: bool

      :raises IllegalSchemaException: If there's multiple parts with the same name or
          a file already exists.



   .. py:method:: deserialise_manually(folder_path, man_cls_path, **init_kwargs)
      :classmethod:



   .. py:method:: deserialise_all(folder_path, ignore_folders_prefix = set(), ignore_folders_suffix = set(), **kwargs)

      Deserialise contents of folder.

      Additional initialisation keyword arguments can be provided if needed.

      This loads both the raw attributes for this object as well as the
      serialisable parts / attributes recursively.

      :param folder_path: The folder path.
      :type folder_path: str
      :param ignore_folders_prefix: The prefixes of folders
                                    to ignore.
      :type ignore_folders_prefix: set[str]
      :param ignore_folders_suffix: The suffixes of folders
                                    to ignore.
      :type ignore_folders_suffix: set[str]

      :Returns: **Serialisable** -- The resulting object.



   .. py:attribute:: __slots__
      :value: ()



.. py:class:: AvailableSerialisers

   Bases: :py:obj:`enum.Enum`


   Describes the available serialisers.


   .. py:attribute:: dill


   .. py:attribute:: json


   .. py:method:: write_to(file_path)


   .. py:method:: from_file(file_path)
      :classmethod:



   .. py:method:: __new__(value)


   .. py:method:: _generate_next_value_(start, count, last_values)

      Generate the next value when not given.

      name: the name of the member
      start: the initial start value or None
      count: the number of existing members
      last_value: the last value assigned or None



   .. py:method:: _missing_(value)
      :classmethod:



   .. py:method:: __repr__()


   .. py:method:: __str__()


   .. py:method:: __dir__()

      Returns all members and all public methods



   .. py:method:: __format__(format_spec)

      Returns format using actual value type unless __str__ has been overridden.



   .. py:method:: __hash__()


   .. py:method:: __reduce_ex__(proto)


   .. py:method:: name()

      The name of the Enum member.



   .. py:method:: value()

      The value of the Enum member.



.. py:class:: DillSerialiser

   Bases: :py:obj:`Serialiser`


   The dill based serialiser.


   .. py:attribute:: ser_type

      The serialiser type.


   .. py:method:: serialise(raw_parts, target_file)

      Serialise the raw attributes / objects.

      :param raw_parts: The raw objects to serialise.
      :type raw_parts: dict[str, Any]
      :param target_file: The file name to write to.
      :type target_file: str



   .. py:method:: deserialise(target_file)

      Deserialise data written to the specified file.

      :param target_file: The file to read from.
      :type target_file: str

      :Returns: **dict[str, Any]** -- The deserialised raw attributes / objects.



   .. py:attribute:: RAW_FILE
      :value: 'raw_dict.dat'



   .. py:method:: get_ser_type_file(folder)
      :classmethod:



   .. py:method:: save_ser_type_file(folder)

      Save the serialiser type into the specified folder.

      :param folder: The folder to use.
      :type folder: str



   .. py:method:: get_manually_serialised_path(folder)
      :classmethod:



   .. py:method:: check_ser_type(folder)

      Check that the folder contains data serialised by this serialiser.

      :param folder: Target folder.
      :type folder: str

      :raises TypeError: If the folder was not serialised by this serialiser.



   .. py:method:: serialise_all(obj, target_folder, overwrite = False)

      Serialise the entire object into the target folder.

      This finds the serialisable parts (attributes) of the object and calls
      the same method on them recursively.
      It also finds the raw attributes (if any) and serialises them.

      :param obj: The object to serialise.
      :type obj: Serialisable
      :param target_folder: The target folder.
      :type target_folder: str
      :param overwrite: Whether to allow overwriting. Defaults to False.
      :type overwrite: bool

      :raises IllegalSchemaException: If there's multiple parts with the same name or
          a file already exists.



   .. py:method:: deserialise_manually(folder_path, man_cls_path, **init_kwargs)
      :classmethod:



   .. py:method:: deserialise_all(folder_path, ignore_folders_prefix = set(), ignore_folders_suffix = set(), **kwargs)

      Deserialise contents of folder.

      Additional initialisation keyword arguments can be provided if needed.

      This loads both the raw attributes for this object as well as the
      serialisable parts / attributes recursively.

      :param folder_path: The folder path.
      :type folder_path: str
      :param ignore_folders_prefix: The prefixes of folders
                                    to ignore.
      :type ignore_folders_prefix: set[str]
      :param ignore_folders_suffix: The suffixes of folders
                                    to ignore.
      :type ignore_folders_suffix: set[str]

      :Returns: **Serialisable** -- The resulting object.



   .. py:attribute:: __slots__
      :value: ()



.. py:data:: _DEF_SER

.. py:function:: get_serialiser(serialiser_type = _DEF_SER)

   Get the serialiser based on the type specified.

   :param serialiser_type: The required type. Defaults to 'dill'.
   :type serialiser_type: Union[str, AvailableSerialisers], optional

   :raises ValueError: If no serialiser is found.

   :Returns: **Serialiser** -- The appropriate serialiser.


.. py:function:: get_serialiser_type_from_folder(folder_path)

   Get the serialiser type that was used to serialise data in the folder.

   :param folder_path: The folder in question.
   :type folder_path: str

   :Returns: **AvailableSerialisers** -- The serialiser type.


.. py:function:: get_serialiser_from_folder(folder_path)

   Get the serialiser that was used to serialise the data in the folder.

   :param folder_path: The folder in question.
   :type folder_path: str

   :Returns: **Serialiser** -- The appropriate serialiser.


.. py:function:: serialise(serialiser_type, obj, target_folder, overwrite = False)

   Serialise an object based on the specified serialiser type.

   :param serialiser_type: The serialiser type.
   :type serialiser_type: Union[str, AvailableSerialisers]
   :param obj: The object to serialise.
   :type obj: Serialisable
   :param target_folder: The folder to serialise into.
   :type target_folder: str
   :param overwrite: Whether to allow overwriting. Defaults to False.
   :type overwrite: bool


.. py:function:: deserialise(folder_path, ignore_folders_prefix = set(), ignore_folders_suffix = set(), **init_kwargs)

   Deserialise contents of a folder.

   Extra init keyword arguments can be provided if needed.
   These are generally:
   - cnf: The config relevant to the components
   - tokenizer (BaseTokenizer): The base tokenizer for the model
   - cdb (CDB): The CDB for the model
   - vocab (Vocab): The Vocab for the model
   - model_load_path (Optional[str]): The model load path,
       but not the component load path

   This method finds the serialiser to be used based on the files on disk.

   :param folder_path: The folder to serialise.
   :type folder_path: str
   :param ignore_folders_prefix: The prefixes of folders to ignore.
   :type ignore_folders_prefix: set[str]
   :param ignore_folders_suffix: The suffixes of folders to ignore.
   :type ignore_folders_suffix: set[str]

   :Returns: **Serialisable** -- The deserialised object.