No, this is not a good idea, because there are more things that need to be removed, not only the database tables. Deleting these tables without checking their usage first (see troubleshooting below) results in this error Cron error message. table or view not found: 1146
The deleted fields are purged by cron, but only 50 rows per run. If you want to accelerate the process, run cron more often or increase the batch size in field.settings.yml:
purge_batch_size: 50
Instead of cron you can run field_purge_batch() from the command line, then with a higher batch limit, for example 1000 or how many rows you have and are able to delete without hitting a memory/time limit:
drush ev "field_purge_batch(1000)"
Troubleshooting
If cron or the drush command has no effect, even not deleting some of the rows (if there are any more rows left in the tables), then check whether there are still field storages registered for these tables:
drush ev "var_dump(\Drupal::state()->get('field.storage.deleted'))"
If this is empty, then you can remove the tables in the database as well, because Drupal has no chance to guess the hash keys of the table names without this information.
If this is not empty, then you can use this output of the field storage definition(s) to debug this further (see comment of @MarioSteinitz).
