Can't resist mentioning, that the "Open" button on the ribbon and the link on the folder title are still there, and they really do the same what you want, with only one click cost.
Anyway, certainly there are several approaches to achieve what you want.
Approach 1: iframe
I believe the most simple way would be what I call "dirty js hack"... It is NOT recommended way, but it will take you only few minutes, and if you need to do the job quickly, it will do.
The idea is basically to display the corresponding page in iframe, inside the modal dialog.
Here is the precise sequence:
- Switch the folder view form into edit mode. For IE, you can achieve this following way:
- Inside the modal dialog, right-click and select Properties in the context menu
- Copy-paste the page address into new browser tab, strip "IsDlg=1" from the query string, and hit Enter
- Select Site Actions -> Edit page
- Add "Content Editor" webpart to the page

, and drag it so it is placed to the bottom of the default list form webpart:

- Follow the "Click here to add new content" advice :) Then, on the ribbon, select HTML-> Edit HTML:

Copy and paste the following code into edit box:
<script type="text/javascript"> document.write('<iframe src="' + $get('SPFieldFile').childNodes[1].href + '?IsDlg=1" height="320" width="650"><' + '/iframe>'); </script>
Save and exit page edit mode
You will get the following final look (you may need to clear the browser cache, F12 -> Cache -> Always update from server; F5):

I admit, it doesn't look very cool. Also, it is very fragile: if you have customized list display form or something like this, this solution could suddenlty stop working.
So, about the alternative solution.
Approach 2: custom web part
This approach involves Visual Studio. You should develop a very simple custom web part, which adds ListViewByQuery inside itself, and configures it so the ListViewByQuery.Query.Folder points to the corresponding folder of the list. You can determine the current item and current list from the webpart code, using SPContext.Current.ListItem and SPContext.Current.List accordingly.
This is of course the preferable solution.
Once the webpart is created, you can put in to the folder display form either manually, as I've described above, or programmatically, during the feature activation.
Please, do not hesitate to ask if you have any additional questions.