Contains functions that provide general utility and build upon the Python 3 standard library. It has no external dependencies.

• serialization: serialization & deserialization for NamedTuple-deriving & @dataclass decorated classes
• archives: uncompress tar archives
• common: utilities
• schema: obtain a dict-like structure describing the fields & types for any serialzable type (helpful to view as JSON)

This project's most notable functionality are the serialize and deserialize funtions of core_utils.serialization. Take a look at the end of this document for example use.

This project uses poetry for virtualenv and dependency management. We recommend using brew to install poetry system-wide.

To install the project's dependencies, perform:

poetry install 

Every command must be run within the poetry-managed environment. For instance, to open a Python shell, you would execute:

poetry run python 

Alternatively, you may activate the environment by performing poetry shell and directly invoke Python programs.

Install pre-commit git hooks using pre-commit install. Hooks are defined in the .pre-commit-config.yaml file.

CI enforces linting using all pre-commit hooks.

NOTE: Dependencies in hooks MUST be kept in-sync with the dev-dependencies section in pyproject.toml for `poetry.

To run tests, execute:

poetry run pytest -v 

To run tests against all supported environments, use tox:

poetry run tox -p 

NOTE: To run tox, you must have all necessary Python interpreters available. We recommend using pyenv to manage your Python versions.

This project uses ruff for code formatting and linting. Static type checking is enforced using mypy. Use the following commands to ensure code quality:

# code format and lint: applies fixes automatically in-place if possible ruff format . ruff check --fix . # typechecks mypy --ignore-missing-imports --follow-imports=silent --show-column-numbers --warn-unreachable --install-types --non-interactive --check-untyped-defs . 

Lets say you have an address book that you want to write to and from JSON. We'll define our data types for our AddressBook:

from typing import Optional, Union, Sequence from dataclasses import dataclass from enum import Enum, auto @dataclass(frozen=True) class Name: first: str last: str middle: Optional[str] = None class PhoneNumber(NamedTuple): area_code: int number: int extension: Optional[int] = None @dataclass(frozen=True) class EmailAddress: name: str domain: str class ContactType(Enum): personal, professional = auto(), auto() class Emergency(NamedTuple): full_name: str contact: Union[PhoneNumber, EmailAddress] @dataclass(frozen=True) class Entry: name: Name number: PhoneNumber email: EmailAddress contact_type: ContactType emergency_contact: Emergency @dataclass(frozen=True) class AddressBook: entries: Sequence[Entry]

For illustration, let's consider the following instantiated AddressBook:

ab = AddressBook([ Entry(Name('Malcolm', 'Greaves', middle='W'), PhoneNumber(510,3452113), EmailAddress('malcolm','world.com'), contact_type=ContactType.professional, emergency_contact=Emergency("Superman", PhoneNumber(262,1249865,extension=1)) ), ])

We can convert our AddressBook data type into a JSON-formatted string using serialize:

from core_utils.serialization import serialize import json s = serialize(ab) j = json.dumps(s, indent=2) print(j)

And we can easily convert the JSON string back into a new instanitated AddressBook using deserialize:

from core_utils.serialization import deserialize d = json.loads(j) new_ab = deserialize(AddressBook, d) print(ab == new_ab) # NOTE: The @dataclass(frozen=True) is only needed to make this equality work. # Any @dataclass decorated type is serializable. 

Note that the deserialize function needs the type to deserialize the data into. The deserizliation type-matching is structural: it will work so long as the data type's structure (of field names and associated types) is compatible with the supplied data.

In the event that one desires to use serialize and deserialize with data types from third-party libraries (e.g. numpy arrays) or custom-defined classes that are not decorated with @dataclass or derive from NamedTuple, one may supply a CustomFormat.

CustomFormat is a mapping that associates precise types with custom serialization functions. When supplied to serialize, the values in the mapping accept an instance of the exact type and produces a serializable representation. In the deserialize function, they convert such a serialized representation into a bonafide instance of the type.

To illustrate their use, we'll deine CustomFormat dicts that allow us to serialize numpy multi-dimensional arrays:

import numpy as np from core_utils.serialization import * custom_serialization: CustomFormat = { np.ndarray: lambda arr: arr.tolist() } custom_deserialization: CustomFormat = { np.ndarray: lambda lst: np.array(lst) }

Now, we may supply custom_{serialization,deserialization} to our functions. We'll use them to perform a "round-trip" serialization of a four-dimensional array of floating point numbers to and from a JSON-formatted str:

import json v_original = np.random.random((1,2,3,4)) s = serialize(v_original, custom=custom_serialization) j = json.dumps(s) d = json.loads(j) v_deser = deserialize(np.ndarray, d, custom=custom_deserialization) print((v_original == v_deser).all())

It's important to note that, when supplying a CustomFormat the serialization functions take priority over the default behavior (except for Any, as it is always considered a pass-through). Moreover, types must match exactly to the keys in the mapping. Thus, if using a generic type, you must supply separate key-value entires for each distinct type parameterization.