Skip to content

fix: improve model_args type coercion in handle_arg_string#3608

Merged
baberabb merged 1 commit intoEleutherAI:mainfrom
ManasVardhan:fix/model-args-type-coercion
Mar 4, 2026
Merged

fix: improve model_args type coercion in handle_arg_string#3608
baberabb merged 1 commit intoEleutherAI:mainfrom
ManasVardhan:fix/model-args-type-coercion

Conversation

@ManasVardhan
Copy link
Contributor

Summary

Improves the type inference/coercion logic in handle_arg_string() (lm_eval/utils.py) to fix several edge cases where model argument values end up as the wrong Python type.

Fixes #2183 (related: #2167)

Changes

handle_arg_string improvements:

  • Negative integers: -1 now correctly becomes int(-1) instead of staying a string (the old str.isnumeric() check only matched non-negative digits)
  • None detection: None/none now returns Python None
  • Explicit quoting: wrapping a value in quotes forces it to stay a string, e.g. revision="123123" keeps the value as str instead of casting to int. This is the recommended fix for the revision issue in [BUG] Huggingface and Neuron runs fail if model revision is an integer #2167.
  • Scientific notation: 1e-5 correctly becomes float(1e-5) via the float fallback

Tests added:

  • Comprehensive tests for handle_arg_string covering bools, None, ints, negative ints, floats, scientific notation, plain strings, and explicit quoting
  • Tests for simple_parse_args_string covering end-to-end parsing with type coercion

Backward compatibility

All existing behavior is preserved. The only additions are:

  1. Negative integers and None are now properly coerced (previously returned as strings)
  2. Users can now quote values to opt out of coercion (new feature, no existing behavior changed)
- Support negative integers (e.g. -1) which isnumeric() missed - Add None/none detection - Add explicit quoting to force string type (e.g. revision="123123") - Add scientific notation support via float() fallback - Add comprehensive tests for handle_arg_string and simple_parse_args_string Fixes EleutherAI#2183, related to EleutherAI#2167
@baberabb baberabb merged commit bcdf34b into EleutherAI:main Mar 4, 2026
2 of 6 checks passed
@baberabb
Copy link
Contributor

baberabb commented Mar 4, 2026

Thanks for the PR! LGTM

Tracin pushed a commit to Tracin/lm-evaluation-harness that referenced this pull request Mar 11, 2026
…I#3608) - Support negative integers (e.g. -1) which isnumeric() missed - Add None/none detection - Add explicit quoting to force string type (e.g. revision="123123") - Add scientific notation support via float() fallback - Add comprehensive tests for handle_arg_string and simple_parse_args_string Fixes EleutherAI#2183, related to EleutherAI#2167
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants