This is a collection of utilities for handling various types of multimedia data. Enhance your experience by seamlessly integrating these utilities with the Clarifai Python SDK. This powerful combination empowers you to address both visual and textual use cases effortlessly through the capabilities of Artificial Intelligence. Unlock new possibilities and elevate your projects with the synergy of versatile data utilities and the robust features offered by the Clarifai Python SDK. Explore the fusion of these tools to amplify the intelligence in your applications! 🌐🚀
Website | Schedule Demo | Signup for a Free Account | API Docs | Clarifai Community | Python SDK Docs | Examples | Colab Notebooks | Discord
Install from PyPi:
pip install clarifai-datautilsInstall from Source:
git clone https://github.com/Clarifai/clarifai-python-datautils cd clarifai-python-datautils python3 -m venv env source env/bin/activate pip3 install -r requirements.txtQuick intro to Image Annotation Conversion feature
from clarifai_datautils.image import ImageAnnotations annotated_dataset = ImageAnnotations.import_from(path= 'folder_path', format= 'annotation_format')-
- Load various annotated image datasets and export to clarifai Platform
- Convert from one annotation format to other supported annotation formats
- Easy to use pipelines to load data from files and ingest into clarifai platfrom.
- Load text files(pdf, doc, etc..) , transform, chunk and upload to the Clarifai Platform
To use Image Annotation Loader, please install the extra libs required for annotations
from clarifai_datautils.image import ImageAnnotations #import from folder coco_dataset = ImageAnnotations.import_from(path='folder_path',format= 'coco_detection') #Using clarifai SDK to upload to Clarifai Platform #export CLARIFAI_PAT={your personal access token} # set PAT as env variable from clarifai.client.dataset import Dataset dataset = Dataset(user_id="user_id", app_id="app_id", dataset_id="dataset_id") dataset.upload_dataset(dataloader=coco_dataset.dataloader) #info about loaded dataset coco_dataset.get_info() #exporting to other formats coco_dataset.export_to('voc_detection')To use Data Ingestion Pipeline, please run
pip install -r requirements-dev.txtfrom clarifai_datautils.text import Pipeline, PDFPartition from clarifai_datautils.text.pipeline.cleaners import Clean_extra_whitespace # Define the pipeline pipeline = Pipeline( name='pipeline-1', transformations=[ PDFPartition(chunking_strategy = "by_title",max_characters = 1024), Clean_extra_whitespace() ] ) # Using SDK to upload from clarifai.client import Dataset dataset = Dataset(dataset_url) dataset.upload_dataset(pipeline.run(files = file_path, loader = True))See many more code examples in this repo.