Skip to content

Yodas2 dataset processing: config and processors#102

Open
ssh-meister wants to merge 92 commits intomainfrom
yodas2
Open

Yodas2 dataset processing: config and processors#102
ssh-meister wants to merge 92 commits intomainfrom
yodas2

Conversation

@ssh-meister
Copy link
Collaborator

@ssh-meister ssh-meister commented Mar 12, 2025

Added:

  • Faster Whisper Inference processor
  • Lambda Expressions processor
  • Extract Tar
@ssh-meister ssh-meister self-assigned this Apr 1, 2025
ssh-meister added 29 commits May 5, 2025 05:49
@ssh-meister
yodas2 config is added
4d287d5 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ListToEntries is added
dc61432 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ListToEntries is added
6921035 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ListRepoFiles and SnapshotDownload are added
ea5f874 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ListRepoFiles and SnapshotDownload docs are added
3fc4895 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Merge branch 'yodas2_pr' of https://github.com/NVIDIA/NeMo-speech-dat…
8fe7d92 
…a-processor into yodas2_pr
@ssh-meister
removed build
0c093d6 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
init updated
29daaa3 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ListYodas2Data
4eb64ef 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ListYodas2Data upd
8938345 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
LambdaExpression
59a2d3f 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
DownloadYodas2Data
c632344 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
init for yodas2 processors
cba6b06 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Fixed docs
613a486 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Fixed docs
8220deb 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ExtractTar
9e71140 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
CreateInitialManifestYodas2
813f670 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Audio converstion processors moved to convert_audio.py
8ea0094 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
RemoveFiles
24f8cc0 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
ASR inference refactoring
e73bd83 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
FasterWhisperInference
5a293c6 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Doc fix
5ab8afc 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Fix typo
312ec51 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
DropSpecifiedFields
c725a7c 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
WhisperHallucinationFeatures
2327a18 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
vLLMInference
29cf370 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
CleanQwenGeneration
71ebb33 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Updated SubRegex
1525c0c 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
CountNumWords updated
994a5f2 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
ssh-meister added 16 commits May 7, 2025 10:17
@ssh-meister
Prevented requirements collection in subfolders
5ae3072 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Prevent no space left during building
dc0391e 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Lightweight granary requirements
8b80117 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Prevent no space left during building
f3210fb 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
E2E tests check
f12f81f 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Branch update
2c41c13 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
requirements mofidifcation
c9e8d43 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Fix device setup for FasterWhisperInference
8f6e1cf 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
added prepare_yodas2_data.py
45fdc8a 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
added text preprocessing in FastTextLangIdClassifier
214a1be 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
prepare_yodas2_data.py updated
98ab125 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
added wget to add tests/prepare_test_data/prepare_yodas2_data.py
6153185 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Removed already imported module
223bd5d 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Added missed param in FasterWhisperInference
54a0af5 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Added HfHubDownloadYodas2Data, HfHubDownload, GetGranarysYodas2
c9e72fd 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Simplified prepare_yodas2_data.py
7a65dbe 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister ssh-meister marked this pull request as ready for review May 19, 2025 11:54
@ssh-meister ssh-meister requested a review from karpnv May 19, 2025 11:54
ssh-meister and others added 10 commits May 20, 2025 02:55
@ssh-meister
New structure of files
33ef48a 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Fix typo in ListYodas2Data
71d84cb 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Added JoinManifests
0bda056 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
mkdir in SnapshotDownload process
674f1fd 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Yodas from Granary
3e12532 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Granary cfg moved
b7ee0f9 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
license
7421bb0 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Removed extra line
ef4515b 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Added constant fields
3b8aff7 
Signed-off-by: Sasha Meister <ameister@nvidia.com>
@ssh-meister
Update README.md
3dc4971
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@ssh-meister @AO696