2

I'm working with Mediawiki 1.25.1 on CentOS 7.2 running Apache 2.4.3. I'm working with Mediawiki 1.26.4 on CentOS 7.2 running Apache 2.4.6. I am experiencing an Internal Server Error after editing a page and submitting it:

enter image description here

There is no additional information in /var/log/httpd/error_log. According to Manual:How to debug, I added the following to LocalSettings, and then restarted Apache:

$ sudo tail -7 LocalSettings.php # Added by JW for debugging (JUN 2016). Keep commented unless needed. $wgShowExceptionDetails=true; $wgShowSQLErrors=true; error_reporting( -1 ); ini_set( 'display_errors', 1 ); ?>

Adding $wgShowExceptionDetails, error_reporting and friends does not provide any additional information that we can find. error_log is clean, access_log is clean. We enabled error_log = php_errors.log, but the server does not produce a php_errors.log. We have no idea where else to look.

We ran mysqlcheck my_wiki --auto-repair --user=... --password=... and it reports no errors. apachectl configtest immediately returns Syntax OK. However, the problem persists.

We also performed an upgrade to MediaWiki 1.26.4 (from 1.25.1) in hopes of fixing this internal server error with no joy. It still persists, and I still cannot get additional information about it.

QUESTIONS: How does one really enable debugging information? How can we gather more information about the problem plaguing this server?

(Please provide actionable items in response to the question; and please don't provide off-site links telling us to try some of the stuff on some other page that may or may not work. And please provide answers for the specific question that was asked, and not other questions that were not asked.)

Here's the most recent "useless error message" from error_log after upgrading to MediaWiki 1.26. Its from the tail of /var/log/httpd/error_log when a request was submitted to update a wiki page.

... [Fri Oct 28 21:05:04.456126 2016] [suexec:notice] [pid 1053] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Fri Oct 28 21:05:04.457260 2016] [:notice] [pid 1053] ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/) configured. [Fri Oct 28 21:05:04.457283 2016] [:notice] [pid 1053] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8" [Fri Oct 28 21:05:04.457290 2016] [:notice] [pid 1053] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30" [Fri Oct 28 21:05:04.457295 2016] [:notice] [pid 1053] ModSecurity: LUA compiled version="Lua 5.1" [Fri Oct 28 21:05:04.457299 2016] [:notice] [pid 1053] ModSecurity: LIBXML compiled version="2.9.1" [Fri Oct 28 21:05:04.495066 2016] [auth_digest:notice] [pid 1053] AH01757: generating secret for digest authentication ... [Fri Oct 28 21:05:04.632965 2016] [mpm_prefork:notice] [pid 1053] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips configured -- resuming normal operations [Fri Oct 28 21:05:04.632994 2016] [core:notice] [pid 1053] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

A couple of folks have commented about the startup messages. I did not realize it was abnormal since Apache always did it. We recently made the following change to address it (but it did not help out "no information" problem):

# diff /etc/httpd/conf.d/mpm_prefork.conf.bu /etc/httpd/conf.d/mpm_prefork.conf 9,13c9,13 < StartServers 1 < MinSpareServers 1 < MaxSpareServers 5 < ServerLimit 10 < MaxClients 10 --- > StartServers 4 > MinSpareServers 4 > MaxSpareServers 8 > ServerLimit 32 > MaxClients 32

None of us are professional admins, so we don't know if its enough to resolve the issue. We are a bunch of free and open software developers who run a web server and wiki to help users.

Here is the "maximum brevity" /etc/php.ini:

# cat /etc/php.ini | egrep -v '(^;|^\[)' | sed '/^$/d' engine = On short_open_tag = Off asp_tags = Off precision = 14 output_buffering = 4096 zlib.output_compression = Off implicit_flush = Off unserialize_callback_func = serialize_precision = 17 disable_functions = disable_classes = zend.enable_gc = On expose_php = On max_execution_time = 30 max_input_time = 60 memory_limit = 128M error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT display_errors = Off display_startup_errors = Off log_errors = On log_errors_max_len = 1024 ignore_repeated_errors = Off ignore_repeated_source = Off report_memleaks = On track_errors = Off html_errors = On variables_order = "GPCS" request_order = "GP" register_argc_argv = Off auto_globals_jit = On post_max_size = 8M auto_prepend_file = auto_append_file = default_mimetype = "text/html" doc_root = user_dir = enable_dl = Off file_uploads = On upload_max_filesize = 2M max_file_uploads = 20 allow_url_fopen = On allow_url_include = Off default_socket_timeout = 60 cli_server.color = On date.timezone = "UTC" pdo_mysql.cache_size = 2000 pdo_mysql.default_socket= SMTP = localhost smtp_port = 25 sendmail_path = /usr/sbin/sendmail -t -i mail.add_x_header = On sql.safe_mode = Off odbc.allow_persistent = On odbc.check_persistent = On odbc.max_persistent = -1 odbc.max_links = -1 odbc.defaultlrl = 4096 odbc.defaultbinmode = 1 ibase.allow_persistent = 1 ibase.max_persistent = -1 ibase.max_links = -1 ibase.timestampformat = "%Y-%m-%d %H:%M:%S" ibase.dateformat = "%Y-%m-%d" ibase.timeformat = "%H:%M:%S" mysql.allow_local_infile = On mysql.allow_persistent = On mysql.cache_size = 2000 mysql.max_persistent = -1 mysql.max_links = -1 mysql.default_port = mysql.default_socket = mysql.default_host = mysql.default_user = mysql.default_password = mysql.connect_timeout = 60 mysql.trace_mode = Off mysqli.max_persistent = -1 mysqli.allow_persistent = On mysqli.max_links = -1 mysqli.cache_size = 2000 mysqli.default_port = 3306 mysqli.default_socket = mysqli.default_host = mysqli.default_user = mysqli.default_pw = mysqli.reconnect = Off mysqlnd.collect_statistics = On mysqlnd.collect_memory_statistics = Off pgsql.allow_persistent = On pgsql.auto_reset_persistent = Off pgsql.max_persistent = -1 pgsql.max_links = -1 pgsql.ignore_notice = 0 pgsql.log_notice = 0 sybct.allow_persistent = On sybct.max_persistent = -1 sybct.max_links = -1 sybct.min_server_severity = 10 sybct.min_client_severity = 10 bcmath.scale = 0 session.save_handler = files session.use_cookies = 1 session.use_only_cookies = 1 session.name = PHPSESSID session.auto_start = 0 session.cookie_lifetime = 0 session.cookie_path = / session.cookie_domain = session.cookie_httponly = session.serialize_handler = php session.gc_probability = 1 session.gc_divisor = 1000 session.gc_maxlifetime = 1440 session.bug_compat_42 = Off session.bug_compat_warn = Off session.referer_check = session.cache_limiter = nocache session.cache_expire = 180 session.use_trans_sid = 0 session.hash_function = 0 session.hash_bits_per_character = 5 url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry" mssql.allow_persistent = On mssql.max_persistent = -1 mssql.max_links = -1 mssql.min_error_severity = 10 mssql.min_message_severity = 10 mssql.compatability_mode = Off mssql.secure_connection = Off tidy.clean_output = Off soap.wsdl_cache_enabled=1 soap.wsdl_cache_dir="/tmp" soap.wsdl_cache_ttl=86400 soap.wsdl_cache_limit = 5 ldap.max_links = -1
Improve this question
7
  • Even without these debugging statements, you should get useful information in your server's error_log. Commented Jun 20, 2016 at 7:59
  • Thanks @Stephen. I added error_log information after causing an internal server error. Commented Jul 2, 2016 at 18:31
  • Have you addressed the MaxRequestWorkers problem? Commented Oct 29, 2016 at 1:37
  • "Here's the most recent 'useless error message'" - those aren't error messages. Those are notices, consistent with having additional debug information logged. What version of Apache are you using? Presumably you have access to the server config? Do you know what "log level" Apache is currently set at? Commented Oct 29, 2016 at 9:03
  • @w3dk - "those aren't error messages. Those are notices..." - I can't explain why they are provided error_log. "What version of Apache are you using..." - Apache 2.4.6. "Do you know what log level..." - Cat'ing httpd.conf shows LogLevel warn. Commented Oct 29, 2016 at 9:20

4 Answers 4

Reset to default
1

PHP Parse error: syntax error, unexpected 'error_reporting'...

You are missing a semicolon (;) from the end of the preceding line, ie:

$wgShowExceptionDetails=true;

An "Internal Server Error" (code: 500) is a very generic (catch all) error. Which is why you need the server's error_log for more information. If using the Apache webserver it is often caused by syntax errors in per-directory Apache config files (ie. .htaccess files). Syntax errors can also occur if required server modules are not installed (since the directives cannot be interpreted).

PHP will also trigger a 500 error if a fatal error occurs and error_reporting / display_errors are suppressed.

Improve this answer
3
  • Oh, thanks. Let me give it a go and accept if its progress. Commented Jun 20, 2016 at 7:56
  • OK, I was missing the semi colon. However, adding all the suggested statements with the semi-colons did not produce any more information. Maybe Apache is broke instead of MediaWiki. It sure would be nice if something was logged somewhere rather than making you and me guess. Commented Jul 2, 2016 at 18:37
  • I just upgraded to MediaWiki 1.26.4 in an effort to fix the wiki... Still broken. Nothing displayed on the web page. Nothing written to error_log. I guess it is pick my poison: keep suffering or keep guessing. Commented Oct 29, 2016 at 1:03
1
+25

This type of error is due to mis-confiuration of .htaccess file. php has nothing to do with it. You need to debug apache with some useful diagnostic commands

apachectl configtest

useful help is here https://serverfault.com/questions/232145/command-to-check-validity-of-apache-server-config-files

Improve this answer
2
  • Thanks Vikas. You might as well take a look at How to test '.htaccess' from command line at Unix & Linux Stack Exchange. Commented Oct 29, 2016 at 8:32
  • Thanks again Vikas. I'm trying to figure out what exactly I should do next since following MediaWiki's Manual:How to debug does not work. apachectl configtest immediately returns Syntax OK. Can you please prescribe something I should do? Commented Oct 30, 2016 at 10:21
1

You say you got an internal server error at 14:20, but this does not appear in your log extract (presumably you didn't see the 500 response in your access log either). This makes me think the child process crashed - on a default centos build, this should have resulted in the core being dumped.

If this is the case, then there's a problem with the apache or PHP build. Code written in PHP cannot cause such a crash (it can only trigger the the binary code which then crashes). Upgrading MediaWiki will not help in such a scenario. You could have a look at the core file in gdb, but really the first step is to ensure that both PHP and Apache are patched up to date (and include the versions of both when asking questions here).

It is somewhat alarming to see you are reaching maxrequestworker twice in your log file. And is there a reason for running -DFOREGROUND?

Improve this answer
3
  • Regarding the -DFOREGROUND, I'm guessing its from either (1) nearly 20 years of running a web server so its cruft carried forward, or (2) Confused about DFOREGROUND with Apache and Michael Hampton's answer. Its probably the latter. Commented Oct 30, 2016 at 3:51
  • Regarding MaxRequestWorkers, I'm guessing that is CentOS 7 defaults. I changed the defaults as edit'ed into the original question. I guess I assumed that was normal, and Apache was managing its own resources. Commented Oct 30, 2016 at 4:00
  • Thanks again symcbean. Please forgive my ignorance... what exactly I should do next? I can't quite parse the actionable item in the answer you provided. I am disgusted I cannot get any useful information out of this server. Commented Oct 30, 2016 at 10:25
0

I think we figured out how to diagnose the issue. All this time we were focusing on error logs because we consider a 500 server crash an error, and we specified errors should be logged to error_log in the configuration files.

Out of desperation, we grep'ed the entire file system for *"Internal Server Error". Lo and behold, mod_security is not honoring the configuration settings:

# grep -IR 'Internal Server Error' /var/log ... /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error /var/log/httpd/modsec_audit.log:HTTP/1.1 500 Internal Server Error ... /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161024:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161016:HTTP/1.0 500 Internal Server Error /var/log/httpd/modsec_audit.log-20161016:HTTP/1.0 500 Internal Server Error ... <no error_log entries>

I'm OK with mod_security logging to their own file. I take exception that they did not log to where we told them to through php.ini, and it has wasted months of man hours and trouble shooting. All they had to do was write an occasional message into error_log and this would have been fixed months ago.

Improve this answer