do_robots()

Displays the default robots.txt file content.

Source

function do_robots() {	header( 'Content-Type: text/plain; charset=utf-8' );	/** * Fires when displaying the robots.txt file. * * @since 2.1.0 */	do_action( 'do_robotstxt' );	$output = "User-agent: *\n";	$public = (bool) get_option( 'blog_public' );	$site_url = parse_url( site_url() );	$path = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';	$output .= "Disallow: $path/wp-admin/\n";	$output .= "Allow: $path/wp-admin/admin-ajax.php\n";	/** * Filters the robots.txt output. * * @since 3.0.0 * * @param string $output The robots.txt output. * @param bool $public Whether the site is considered "public". */	echo apply_filters( 'robots_txt', $output, $public ); }

View all references View on Trac View on GitHub

Hooks

do_action( ‘do_robotstxt’ ): Fires when displaying the robots.txt file.
apply_filters( ‘robots_txt’, string $output, bool $public ): Filters the robots.txt output.

Uses	Description
site_url()`wp-includes/link-template.php`	Retrieves the URL for the current site where WordPress application files (e.g. wp-blog-header.php or the wp-admin/ folder) are accessible.
do_action()`wp-includes/plugin.php`	Calls the callback functions that have been added to an action hook.
apply_filters()`wp-includes/plugin.php`	Calls the callback functions that have been added to a filter hook.
get_option()`wp-includes/option.php`	Retrieves an option value based on an option name.

Show 2 more Show less

Changelog

Version	Description
5.3.0	Remove the "Disallow: /" output if search engine visibility is discouraged in favor of robots meta HTML tag via wp_robots_no_robots() filter callback.
2.1.0	Introduced.

User Contributed Notes

wp_kc 5 years ago

Some examples of how robots.txt file content can be altered.

/** * Add Disallow for some file types. * Add "Disallow: /wp-login.php/\n". * Remove "Allow: /wp-admin/admin-ajax.php\n". * Calculate and add a "Sitemap:" link. */ add_filter( 'robots_txt', function( $output, $public ) {	/** * If "Search engine visibility" is disabled, * strongly tell all robots to go away. */	if ( '0' == $public ) {	$output = "User-agent: *\nDisallow: /\nDisallow: /*\nDisallow: /*?\n";	} else {	/** * Disallow some file types */	foreach( array( 'jpeg','jpg','gif','png','mp4','webm','woff','woff2','ttf','eot' ) as $ext ) {	$output .= "Disallow: /*.{$ext}$\n";	}	/** * Get site path. */	$site_url = parse_url( site_url() );	$path = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';	/** * Add new disallow. */	$output .= "Disallow: $path/wp-login.php\n";	/** * Remove line that allows robots to access AJAX interface. */	$robots = preg_replace( '/Allow: [^\0\s]*\/wp-admin\/admin-ajax\.php\n/', '', $output );	/** * If no error occurred, replace $output with modified value. */	if ( ! is_null( robots ) ) {	$output = $robots;	}	/** * Calculate and add a "Sitemap:" link. * Modify as needed. */	$output .= "Sitemap: {$site_url['scheme']}://{$site_url[ 'host' ]}/sitemap_index.xml\n";	}	return $output; }, 99, 2 ); // Priority 99, Number of Arguments 2.

In this article

Source

Hooks

Related

Changelog

User Contributed Notes