According to ConfigMap's docs, a ConfigMap object can have 'data' and 'binaryData' fields.The data field is designed to contain UTF-8 strings while the binaryData field is designed to contain binary data as base64-encoded strings. Therefore, ConfigMaps also can have base64 data in it. But this option is positioned as secondary I would say, 'data' field is used much frequently.
According to Secret's docs, a Secret object can specify the 'data' and/or the 'stringData'. The values for all keys in the data field have to be base64-encoded strings. If the conversion to base64 string is not desirable, you can choose to specify the stringData field instead, which accepts arbitrary strings as values. So, here, in contrary to ConfigMaps, binary approach is considered as primary, as used more frequently.
But we should understand that base64 encoding is not a an encryption, it does not protect your secrets from being used. All who have access to Pods/Deployments in namespace X, can access Secret objects in this namespace. So, these people can simply decode these base64 string to plain ones.
I haven't found any official explanation why the primary way for storing Secrets data is base64, but I have a hypothesis that it is made at least for visual defense, in order to make to see and remember them buy eyes complex maybe.