The file extension means nothing. At all. It's just a part of a file name.
If you want a custom-format dump for use with pg_restore use -Fc as an argument to pg_dump.
pg_dump defaults to generating SQL-format dumps for use with psql.
See the manual for more details.