dynamically convert python datatype

Question

I have a use case where I am reading some data from an API call, but need to transform the data before inserting it into a database. The data comes in a integer format, and I need to save it as a string. The database does not offer a datatype conversion, so the conversion needs to happen in Python before inserting.

Within a config file I have like:

config = {"convert_fields": ["payment", "cash_flow"], "type": "str"}

Then within python I am using the eval() function to check what type to convert the fields to.

So the code ends up being like data['field'] = eval(config['type'])(data['field'])

Does anyone have a better suggestion how I can dynamically change these values, maybe without storing the python class type within a config file.

To add, like sure I could just do str(), but there may be a need to have other fields to convert at some point, which are not string. So I want it to be dynamic, from whatever is defined in the config file for the required conversion fields.

To me it sounds like you need a router class that can accept a source type, an arbitrary value, and a destination type, and then has internal logic to route the source type to the destination type, or raise exceptions if such a conversion is not possible. A library may exist for this, but as far as I know there isn't a magic method that can convert anything to anything in the standard library. This library may be of some value. — Zulfe
– Zulfe, Commented Feb 2, 2022 at 15:26
Let me add the standard warning against using eval on unverified strings. — Daniel Walker
– Daniel Walker, Commented Feb 2, 2022 at 15:56
Are there any restrictions on this? For example, are the types well-defined or there be arbitrary files? Is the config JSON or Python? Are the config/API results trusted? — MisterMiyagi
– MisterMiyagi, Commented Feb 2, 2022 at 16:06
In general, I find there is very little use left for eval, once you accept that is a security risk - almost everything can be done differently and you even have ast.literal_eval to help if really, really, needed. It's just a stinky, stinky, code smell. — JL Peyret
– JL Peyret, Commented Feb 2, 2022 at 16:37
If the config file is just Python source code, then you can write "type": str inside the dict, and directly use the value rather than having to eval() it. — jasonharper
– jasonharper, Commented Feb 2, 2022 at 16:38

JonSG · Accepted Answer · 2022-02-02 16:19:25Z

How about using getattr() and __builtins__ that I feel is a little better than exec()/eval() in this instance.

def cast_by_name(type_name, value): return getattr(__builtins__, type_name)(value) print(cast_by_name("bool", 1))

Should spit back:

True

You will likely want to include some support for exceptions and perhaps defaults but this should get you started.

@mistermiyagi Points out a critical flaw that of course eval is a bulitin as well. We might want to limit this to safe types:

def cast_by_name(type_name, value): trusted_types = ["int", "float", "complex", "bool", "str"] ## others as needed if type_name in trusted_types: return getattr(__builtins__, type_name)(value) return value print(cast_by_name("bool", 1))

JL Peyret · Accepted Answer · 2022-02-02 17:56:00Z

Build up a conversion lookup dictionary in advance.

Faster
Easier to debug

 config = {"convert_fields": {"payment" : "str", "cash_flow" : "str", "customer_id" : "int", "name" : "name_it"} } def name_it(s : str): return s.capitalize() data_in = dict( payment = 101.00, customer_id = 3, cash_flow = 1, name = "bill", city = "london" ) convert_functions = { #support builtins and custom functions fieldname : globals().get(funcname) or getattr(__builtins__, funcname) for fieldname, funcname in config["convert_fields"].items() if not funcname in {"eval"} } print(f"{convert_functions=}") data_db = { fieldname : #if no conversion is specified, use `str` convert_functions.get(fieldname, str)(value) for fieldname, value in data_in.items() } print(f"{data_db=}")

Output:

convert_functions={'payment': <class 'str'>, 'cash_flow': <class 'str'>, 'customer_id': <class 'int'>, 'name': <function name_it at 0x10f0fbe20>} data_db={'payment': '101.0', 'customer_id': 3, 'cash_flow': '1', 'name': 'Bill', 'city': 'london'}

if the config could be stored in code, rather than a json-type approach, I'd look into Pydantic though that is not exactly your problem space here:

from pydantic import BaseModel class Data_DB(BaseModel): payment : str customer_id : int cash_flow : str #you'd need a custom validator to handle capitalization name : str city : str pydata = Data_DB(**data_in) print(f"{pydata=}") print(pydata.dict())

output:

pydata=Data_DB(payment='101.0', customer_id=3, cash_flow='1', name='bill', city='london') {'payment': '101.0', 'customer_id': 3, 'cash_flow': '1', 'name': 'bill', 'city': 'london'}

Collectives™ on Stack Overflow

dynamically convert python datatype

2 Answers 2

2 Comments

Output:

output:

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Output:

output:

Comments

Related