You should use GIO:
GError *error = NULL; if (!g_app_info_launch_default_for_uri ("file:///", NULL, &error)) { g_warning ("Failed to open uri: %s", error->message); }
Note that this almost certainly requires a running GLib main loop (but if you're using GTK+, you already have that).
Also, there's no 100% guarantee that a file manager will be opened, but on any sanely set up system that is going to be the default handler for uris that are directories.
On a normal GNOME system @keltars xdg-open method forks, runs a shell script that starts a binary that then runs the same g_app_info_launch_default_for_uri() function.