WhisperKit Android

WhisperKit Android brings Foundation Models On Device for Automatic Speech Recognition. It extends the performance and feature set of WhisperKit from Apple platforms to Android and Linux. The current feature set is a subset of the iOS counterpart, but we are continuing to invest in Android and now welcome contributions from the community.

[Example App] [Blog Post] [Python Tools Repo]

• Getting Started
• Build From Source
• Contributing
• License
• Citation

WhisperKit API is currently experimental and requires explicit opt-in using @OptIn(ExperimentalWhisperKit::class). This indicates that the API may change in future releases. Use with caution in production code.

To use WhisperKit in your Android app, you need to:

1. Add the following dependencies to your app's build.gradle.kts:

dependencies { // 1. WhisperKit SDK implementation("com.argmaxinc:whisperkit:0.3.2") // Check badge above for latest version // 2. QNN dependencies for hardware acceleration implementation("com.qualcomm.qnn:qnn-runtime:2.34.0") implementation("com.qualcomm.qnn:qnn-litert-delegate:2.34.0") }

2. Configure JNI library packaging in your app's build.gradle.kts:

android { // ... packaging { jniLibs { useLegacyPackaging = true } } }

3. Use WhisperKit in your code:

@OptIn(ExperimentalWhisperKit::class) class YourActivity : AppCompatActivity() { private lateinit var whisperKit: WhisperKit override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) // Initialize WhisperKit // Note: Always use applicationContext to avoid memory leaks and ensure proper lifecycle management whisperKit = WhisperKit.Builder() .setModel(WhisperKit.OPENAI_TINY_EN) .setApplicationContext(applicationContext) .setCallback { what, result -> // Handle transcription output when (what) { WhisperKit.TextOutputCallback.MSG_INIT -> { // Model initialized successfully } WhisperKit.TextOutputCallback.MSG_TEXT_OUT -> { // New transcription available val fullText = result.text val segments = result.segments // Process the transcribed text as it becomes available // This callback will be called multiple times as more audio is processed segments.forEach { segment -> // Process each segment val segmentText = segment.text } } WhisperKit.TextOutputCallback.MSG_CLOSE -> { // Cleanup complete } } } .build() // Load the model lifecycleScope.launch { whisperKit.loadModel().collect { progress -> // Handle download progress } // Initialize with audio parameters whisperKit.init(frequency = 16000, channels = 1, duration = 0) // Transcribe audio data in chunks // You can call transcribe() multiple times with different chunks of audio data // Results will be delivered through the callback as they become available val audioChunk1: ByteArray = // First chunk of audio data whisperKit.transcribe(audioChunk1) val audioChunk2: ByteArray = // Second chunk of audio data whisperKit.transcribe(audioChunk2) // Continue processing more chunks as needed... } } override fun onDestroy() { super.onDestroy() whisperKit.deinitialize() } }

Note: The WhisperKit API is currently experimental and may change in future releases. Make sure to handle the @OptIn(ExperimentalWhisperKit::class) annotation appropriately in your code.

make setup

make env

make rebuild-env

make env

make build jni

make env

make build [linux | qnn | gpu]

make adb-push

adb shell cd /sdcard/argmax/tflite export PATH=/data/local/tmp/bin:$PATH export LD_LIBRARY_PATH=/data/local/tmp/lib whisperkit-cli transcribe --model-path /path/to/openai_whisper-base --audio-path /path/to/inputs/jfk_441khz.m4a

./build/linux/whisperkit-cli transcribe --model-path /path/to/my/whisper_model --audio-path /path/to/my/audio_file.m4a --report --report-path /path/to/dump/report.json

make clean [all]

WhisperKit Android is currently in the beta stage. We are actively developing the project and welcome contributions from the community.

The project uses clang-format 14 for C++ code formatting. You'll need to install it based on your operating system:

# Install LLVM 14 brew install llvm@14 # Create symlink for clang-format sudo ln -sf /opt/homebrew/opt/llvm@14/bin/clang-format /opt/homebrew/bin/clang-format

sudo apt update sudo apt install -y clang-format-14 sudo ln -sf /usr/bin/clang-format-14 /usr/local/bin/clang-format

Verify the installation:

clang-format --version

To check C++ code formatting:

./gradlew spotlessCppCheck

This project uses Git hooks to maintain code quality. These hooks help ensure consistent code formatting and quality standards.

To use the Git hooks, run the following command in your repository root:

git config core.hooksPath .githooks

• Runs spotlessApply to automatically fix code formatting issues
• If formatting fixes are applied, they are automatically staged
• The commit will be blocked if spotlessApply fails

If you need to bypass the hooks temporarily (not recommended), you can use:

git commit --no-verify

• We release WhisperKit Android under MIT License.
• FFmpeg open-source (audio decompressing) is released under LGPL
• OpenAI Whisper model open-source checkpoints were released under the MIT License.
• Qualcomm AI Hub .tflite models and QNN libraries for NPU deployment are released under the Qualcomm AI Model & Software License.

If you use WhisperKit for something cool or just find it useful, please drop us a note at info@argmaxinc.com!

If you are looking for managed enterprise deployment with Argmax, please drop us a note at info+sales@argmaxinc.com.

If you use WhisperKit for academic work, here is the BibTeX:

@misc{whisperkit-argmax, title = {WhisperKit}, author = {Argmax, Inc.}, year = {2024}, URL = {https://github.com/argmaxinc/WhisperKitAndroid} }