feat: Add Dimension Mismatch Handling for ChromaDB (#157)#207
Merged
kevin-mindverse merged 3 commits intomindverse:masterfrom Apr 22, 2025
PStarH:master
Merged
feat: Add Dimension Mismatch Handling for ChromaDB (#157)#207kevin-mindverse merged 3 commits intomindverse:masterfrom PStarH:master
kevin-mindverse merged 3 commits intomindverse:masterfrom
PStarH:master
Conversation
- Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier.
|
|
Contributor
| please resolve the conflict. Thanks! |
Contributor
| Hey there, thank you for your contribution, but still there's a few conflicts to solve :) |
Contributor Author
| I resolved the conflict I think |
yingapple approved these changes Apr 18, 2025
kevin-mindverse pushed a commit that referenced this pull request Apr 22, 2025
* Fix Issue #157 Add chroma_utils.py to manage chromaDB and added docs for explanation * Add logging and debugging process - Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier.
lazy-ape pushed a commit to lazy-ape/Second-Me that referenced this pull request Apr 22, 2025
…indverse#207) * Fix Issue mindverse#157 Add chroma_utils.py to manage chromaDB and added docs for explanation * Add logging and debugging process - Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier.
kevin-mindverse added a commit that referenced this pull request Apr 24, 2025
* fix: modify thinking_model loading configuration * feat: realize thinkModel ui * feat:store * feat: add combined_llm_config_dto * add thinking_model_config & database migration * directly add thinking model to user_llm_config * delete thinking model repo dto service * delete thinkingmodel table migration * add is_cot config * feat: allow define is_cot * feat: simplify logs info * feat: add training model * feat: fix is_cot problem * fix: fix chat message * fix: fix progress error * fix: disable no settings thinking * feat: add thinking warning * fix: fix start service error * feat:fix init trainparams problem * feat: change playGround prompt * feat: Add Dimension Mismatch Handling for ChromaDB (#157) (#207) * Fix Issue #157 Add chroma_utils.py to manage chromaDB and added docs for explanation * Add logging and debugging process - Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier. * Change topics_generator timeout to 30 (#263) * quick fix * fix: shade -> shade_merge_info (#265) * fix: shade -> shade_merge_info * add convert array * quick fix import error * add log * add heartbeat * new strategy * sse version * add heartbeat * zh to en * optimize code * quick fix convert function * Feat/new branch management (#267) * feat: new branch management * feat: fix multi-upload * optimize contribute management --------- Co-authored-by: Crabboss Mr <1123357821@qq.com> Co-authored-by: Ye Xiangle <yexiangle@mail.mindverse.ai> Co-authored-by: Xinghan Pan <sampan090611@gmail.com> Co-authored-by: doubleBlack2 <108928143+doubleBlack2@users.noreply.github.com> Co-authored-by: kevin-mindverse <kevin@mindverse.ai> Co-authored-by: KKKKKKKevin <115385420+kevin-mindverse@users.noreply.github.com>
Heterohabilis pushed a commit to Heterohabilis/Second-Me that referenced this pull request May 29, 2025
…indverse#207) * Fix Issue mindverse#157 Add chroma_utils.py to manage chromaDB and added docs for explanation * Add logging and debugging process - Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier.
Heterohabilis pushed a commit to Heterohabilis/Second-Me that referenced this pull request May 29, 2025
* fix: modify thinking_model loading configuration * feat: realize thinkModel ui * feat:store * feat: add combined_llm_config_dto * add thinking_model_config & database migration * directly add thinking model to user_llm_config * delete thinking model repo dto service * delete thinkingmodel table migration * add is_cot config * feat: allow define is_cot * feat: simplify logs info * feat: add training model * feat: fix is_cot problem * fix: fix chat message * fix: fix progress error * fix: disable no settings thinking * feat: add thinking warning * fix: fix start service error * feat:fix init trainparams problem * feat: change playGround prompt * feat: Add Dimension Mismatch Handling for ChromaDB (mindverse#157) (mindverse#207) * Fix Issue mindverse#157 Add chroma_utils.py to manage chromaDB and added docs for explanation * Add logging and debugging process - Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier. * Change topics_generator timeout to 30 (mindverse#263) * quick fix * fix: shade -> shade_merge_info (mindverse#265) * fix: shade -> shade_merge_info * add convert array * quick fix import error * add log * add heartbeat * new strategy * sse version * add heartbeat * zh to en * optimize code * quick fix convert function * Feat/new branch management (mindverse#267) * feat: new branch management * feat: fix multi-upload * optimize contribute management --------- Co-authored-by: Crabboss Mr <1123357821@qq.com> Co-authored-by: Ye Xiangle <yexiangle@mail.mindverse.ai> Co-authored-by: Xinghan Pan <sampan090611@gmail.com> Co-authored-by: doubleBlack2 <108928143+doubleBlack2@users.noreply.github.com> Co-authored-by: kevin-mindverse <kevin@mindverse.ai> Co-authored-by: KKKKKKKevin <115385420+kevin-mindverse@users.noreply.github.com>
EOMZON pushed a commit to EOMZON/Second-Me that referenced this pull request Feb 1, 2026
…indverse#207) * Fix Issue mindverse#157 Add chroma_utils.py to manage chromaDB and added docs for explanation * Add logging and debugging process - Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier.
EOMZON pushed a commit to EOMZON/Second-Me that referenced this pull request Feb 1, 2026
* fix: modify thinking_model loading configuration * feat: realize thinkModel ui * feat:store * feat: add combined_llm_config_dto * add thinking_model_config & database migration * directly add thinking model to user_llm_config * delete thinking model repo dto service * delete thinkingmodel table migration * add is_cot config * feat: allow define is_cot * feat: simplify logs info * feat: add training model * feat: fix is_cot problem * fix: fix chat message * fix: fix progress error * fix: disable no settings thinking * feat: add thinking warning * fix: fix start service error * feat:fix init trainparams problem * feat: change playGround prompt * feat: Add Dimension Mismatch Handling for ChromaDB (mindverse#157) (mindverse#207) * Fix Issue mindverse#157 Add chroma_utils.py to manage chromaDB and added docs for explanation * Add logging and debugging process - Enhanced the`reinitialize_chroma_collections` function in`chroma_utils.py` to properly check if collections exist before attempting to delete them, preventing potential errors when collections don't exist. - Improved error handling in the`_handle_dimension_mismatch` method in`embedding_service.py` by adding more robust exception handling and verification steps after reinitialization. - Enhanced the collection initialization process in`embedding_service.py` to provide more detailed error messages and better handle cases where collections still have incorrect dimensions after reinitialization. - Added additional verification steps to ensure that collection dimensions match the expected dimension after creation or retrieval. - Improved logging throughout the code to provide more context in error messages, making debugging easier. * Change topics_generator timeout to 30 (mindverse#263) * quick fix * fix: shade -> shade_merge_info (mindverse#265) * fix: shade -> shade_merge_info * add convert array * quick fix import error * add log * add heartbeat * new strategy * sse version * add heartbeat * zh to en * optimize code * quick fix convert function * Feat/new branch management (mindverse#267) * feat: new branch management * feat: fix multi-upload * optimize contribute management --------- Co-authored-by: Crabboss Mr <1123357821@qq.com> Co-authored-by: Ye Xiangle <yexiangle@mail.mindverse.ai> Co-authored-by: Xinghan Pan <sampan090611@gmail.com> Co-authored-by: doubleBlack2 <108928143+doubleBlack2@users.noreply.github.com> Co-authored-by: kevin-mindverse <kevin@mindverse.ai> Co-authored-by: KKKKKKKevin <115385420+kevin-mindverse@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the dimension mismatch handling feature for ChromaDB in the Second-Me project, as outlined in Issue #157. This feature enables seamless transitions between embedding models with different output dimensions, enhancing flexibility and reducing manual intervention.
Key Features
1. Automatic Dimension Detection
chroma_utils.py.text-embedding-ada-002(1536),text-embedding-3-large(3072), etc.mistral(768),llama2(1024)2. Dimension Validation
embedding_service.pyandinit_chroma.py.3. Automatic Reinitialization
reinitialize_chroma_collections4. Robust Error Handling and Logging
5. User Documentation
Embedding Model Switching.mdadded:Impact
This implementation significantly improves user experience by:
Related Issue
#157