[WIP] feat: add model architecture configuration#111
[WIP] feat: add model architecture configuration#111aftersnow wants to merge 1 commit intomodelpack:mainfrom
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @aftersnow, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a significant enhancement to the modelpack specification by adding an optional, detailed model architecture configuration. The primary goal is to provide explicit structural information about models, starting with decoder-type transformers, which is currently missing from the modelpack. This new configuration will enable model runtimes to seamlessly execute models by understanding their internal components and parameters, thereby improving interoperability and reducing the need for hardcoded details.
Highlights
- New Model Architecture Specification: I've added a comprehensive new document,
docs/architecture.md, which defines an optional configuration for model architectures. This is a crucial step towards bridging the gap between model images and model runtimes by providing detailed structural information. - Detailed Transformer Architecture Configuration: The new specification currently focuses on decoder-type transformer architectures, outlining detailed properties for components like tokenizers, embeddings (positional and token), transformer layers (attention, MLP, MoE), and normalization. It supports both uniform and mixed layer configurations, providing a granular way to describe model structure.
- Integration with Modelpack Configuration: I've updated
docs/config.mdto include an optionalarchitecture_configfield. This allows modelpack users to associate a detailed architectural description with their models, enhancing the existing specification and enabling better interoperability with model runtimes.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review | Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary | Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help | Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This PR introduces a new architecture_config field to the model configuration and a detailed specification for it in docs/architecture.md. This is a great addition that will improve model interoperability.
I've left a few comments on the documentation for clarity and consistency. The most important feedback is about a broken link in docs/config.md.
Additionally, it seems that the corresponding Go structs and JSON schema have not been updated to reflect these changes. The following files appear to be missing updates:
specs-go/v1/config.goschema/config-schema.json
Keeping the specification documents, Go types, and JSON schema in sync is crucial for consumers of this spec. Please consider adding the necessary changes to these files in this PR.
| - **mlp_layers** _array_, REQUIRED | ||
| | ||
| Layers that use MLP. If empty, moe_frequency determines sparsity. The default is `[]`. |
There was a problem hiding this comment.
The interaction between mlp_layers and moe_frequency is unclear.
- Both are marked as
REQUIRED, which seems contradictory if they are alternative ways of defining layer types. - The description for
mlp_layerssays "If empty, moe_frequency determines sparsity", which suggests they might be mutually exclusive in some way.
Could you please clarify the logic? For example:
- Are both fields always required? If so, how do they work together?
- Should the user provide one or the other? If so, they should probably be
OPTIONALand a constraint likeoneOfshould be defined on the parentmixed_layersobject.
The documentation should be updated to clearly explain how to configure mixed MLP and MoE layers. Also, note that mlp_layers is marked as REQUIRED but has a default value, which is inconsistent as mentioned in another comment.
| Please take a look. More development, testing, and validation are needed. Recently, I have been busy with production environments and haven't had enough time. Everyone is welcome to contribute! |
aftersnow changed the title 
This commit introduces a new document detailing the model architecture configuration, including support for transformer architectures and their components. It outlines terminology, properties, and provides an example configuration for better understanding and implementation. Signed-off-by: Zhao Chen <zhaochen.zju@gmail.com>
aftersnow force-pushed the add-model-arch-config branch from ab3474c to a1258dd Compare 
| This PR is stale because it has been open 40 days with no activity. |
| This PR is stale because it has been open 40 days with no activity. |
| This PR was closed because it has been stalled for 7 days with no activity. |
| This PR is stale because it has been open 40 days with no activity. |
| This PR was closed because it has been stalled for 7 days with no activity. |
Map HuggingFace config.json fields to the ModelPack architecture vocabulary defined in PR modelpack#111 (docs/architecture.md) across four model families: Llama 3.1 8B, Mistral 7B v0.3, Mixtral 8x7B, and DeepSeek-V2-Lite. The document classifies each field mapping as direct, renamed, derived, inferred, or model-specific, and includes the raw config.json values for traceability. Key findings: - Core fields (hidden_size, num_attention_heads, num_key_value_heads) are consistent across all four models - Several spec fields (use_gated_activation, is_causal, norm type) cannot be determined from config.json alone and require model family knowledge - DeepSeek MLA attention uses a fundamentally different field set (q_lora_rank, kv_lora_rank) that does not map cleanly to the current spec vocabulary - sliding_window has no ModelPack equivalent (spec gap) This mapping is intended to guide the auto-generation tooling and JSON Schema work planned for issue modelpack#164. Ref: modelpack#164 Signed-off-by: Luca Secchieri <luca.secchieri@gmail.com>
Add a draft JSON Schema for the architecture configuration blob introduced in PR modelpack#111 (docs/architecture.md). This schema validates the new media type application/vnd.cncf.model.architecture.v1+json. The schema follows the conventions established in config-schema.json: draft-04 format, \$defs for sub-schemas, additionalProperties: false on all objects, and enum constraints for typed fields. Key design decisions: - uniform_layers vs mixed_layers uses oneOf for mutual exclusivity - MLP vs MoE within UniformLayers also uses oneOf - Boolean architectural flags (has_bias, has_norm, etc.) have no schema-level defaults; defaults are documented in architecture.md Known open questions (flagged in PR description): - rope_scaling is left as an open object pending fuller spec definition - MixedLayers does not enforce MLP-or-MoE via oneOf; intentional? - No vendor extension point defined yet (e.g. x- prefixed fields) Ref: modelpack#164 Signed-off-by: Luca Secchieri <luca.secchieri@gmail.com>
Map HuggingFace config.json fields to the ModelPack architecture vocabulary defined in PR modelpack#111 (docs/architecture.md) across four model families: Llama 3.1 8B, Mistral 7B v0.3, Mixtral 8x7B, and DeepSeek-V2-Lite. The document classifies each field mapping as direct, renamed, derived, inferred, or model-specific, and includes the raw config.json values for traceability. Key findings: - Core fields (hidden_size, num_attention_heads, num_key_value_heads) are consistent across all four models - Several spec fields (use_gated_activation, is_causal, norm type) cannot be determined from config.json alone and require model family knowledge - DeepSeek MLA attention uses a fundamentally different field set (q_lora_rank, kv_lora_rank) that does not map cleanly to the current spec vocabulary - sliding_window has no ModelPack equivalent (spec gap) This mapping is intended to guide the auto-generation tooling and JSON Schema work planned for issue modelpack#164. Ref: modelpack#164 Signed-off-by: Luca Secchieri <luca.secchieri@gmail.com>
Map HuggingFace config.json fields to the ModelPack architecture vocabulary defined in PR modelpack#111 (docs/architecture.md) across four model families: Llama 3.1 8B, Mistral 7B v0.3, Mixtral 8x7B, and DeepSeek-V2-Lite. The document classifies each field mapping as direct, renamed, derived, inferred, or model-specific, and includes the raw config.json values for traceability. Key findings: - Core fields (hidden_size, num_attention_heads, num_key_value_heads) are consistent across all four models - Several spec fields (use_gated_activation, is_causal, norm type) cannot be determined from config.json alone and require model family knowledge - DeepSeek MLA attention uses a fundamentally different field set (q_lora_rank, kv_lora_rank) that does not map cleanly to the current spec vocabulary - sliding_window has no ModelPack equivalent (spec gap) This mapping is intended to guide the auto-generation tooling and JSON Schema work planned for issue modelpack#164. Ref: modelpack#164 Signed-off-by: Luca Secchieri <luca.secchieri@gmail.com>
| This PR is stale because it has been open 40 days with no activity. |
…modelpack#164) Adds hf_parser.py that converts HuggingFace config.json into ModelPack transformer spec format (PR modelpack#111 vocabulary). Supports Mistral, Mixtral, Qwen2, GPT-2, DeepSeek-V2 (MLA + mixed layers), and unknown models with NEEDS_REVIEW fallback. Includes 26 unit tests. Improvements over PR modelpack#185's field mapping research: - MLA attention fields (kv_lora_rank, q_lora_rank, qk_nope/rope_head_dim) - DeepSeek MoE routing params (routed_scaling_factor, topk_method) - Mixed layers support (first_k_dense_replace, moe_layer_freq) - Correct learned position embedding for GPT-2/GPT-Neo Signed-off-by: pradhyum6144 <pradhyum314@gmail.com>
2 participants
This commit introduces a new document detailing the model architecture configuration, including support for transformer architectures and their components. It outlines terminology, properties, and provides an example configuration for better understanding and implementation.
Description
Add an optional architecture configuration that describes the detailed structure and components of the model. Currently, only decoder-type transformer architectures are supported.
Motivation and Context
The current modelpack can serve as a standard for building, distributing, and managing AI models. Using this specification, we can:
However, it lacks the architectural and runtime interfaces to run the model. For example, it lacks crucial architectural details such as:
This creates a gap between the model image and the model runtime. Without this additional information, the model runtime cannot execute the model seamlessly (only if the model runtime has pre-built specific support). To address these limitations, we are defining the model architecture configuration as the first step.
This PR adds an optional architecture configuration that describes the detailed structure and components of the model.
Use Cases
Details
The transformer is the most popular architecture for LLMs. It consists of a stack of structured layers, where each layer contains a self-attention block and a feed-forward network, with normalization layers and residual connections. The complete architecture includes a tokenizer, input embedding layer, position embedding layer, transformer layers, and output embedding layer. The transformer architecture has remained relatively stable since Attention is all you need. As shown in the table below, current open-weight model architectures are converging, making it feasible to define a common abstraction.
Note: Each model represents the largest variant within its respective series.
Next Steps
Acknowledgement
This PR is inspired by transformers, vllm, ollama, onnx, tgi, sglang, lmdeploy, curated-transformers, triton-inference-server, transformer-engine, ggml.