I need to setup replication via SSL. 2 nodes master/master for H/A. Then 2 slave nodes for backup. Node 3 will slave off of node 1, and node 4 will slave off of node 2 for redundant backups. Nodes 3 & 4 will store backups to a directory mapped to S3 via S3FS. All nodes are physical.
We're using mariadb-5.5.41 server on CentOS 7 hosts.
It's a neat plan!
However I am unable to get authentication to work with ssl required of the user.
We already have a CA certificate and key established in the environment. And we used those to generate the cert and key to be used with mariadb.
This is the process we used to generate the cert/key:
openssl genrsa -des3 -out db1.example.com.key 4096 openssl req -new -key db1.example.com.key -out db1.example.com.csr openssl x509 -req -days 3650 -in db1.example.com.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out db1.example.com.crt Using those keys I put together this my.cnf file. Haven't gotten around to configuring replication yet, as I have yet to get ssl logins to work.
[root@db1:~] #cat /etc/my.cnf [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 # Settings user and group are ignored when systemd is used. # If you need to run mysqld under a different user or group, # customize your systemd unit file for mariadb according to the # instructions in http://fedoraproject.org/wiki/Systemd ssl ssl-ca=/opt/mysql/ca.crt ssl-cert=/opt/mysql/db1.example.com.crt ssl-key=/opt/mysql/db1.example.com.key [mysqld_safe] general_log_file=/var/log/mariadb/mariadb.log general_log=1 log-error=/var/log/mariadb/mariadb_error.log pid-file=/var/run/mariadb/mariadb.pid log_slow_queries=/var/log/mysql/mysql-slow.log long_query_time=2 log-queries-not-using-indexes # # include all files from the config directory # !includedir /etc/my.cnf.d Made sure mariadb could read the cert files:
[root@db1:~] #ls -ld /opt/mysql/ /opt/mysql/* drwx------. 2 mysql mysql 86 Jul 20 06:20 /opt/mysql/ -r--------. 1 mysql mysql 2212 Jul 20 05:14 /opt/mysql/ca.crt -r--------. 1 mysql mysql 1956 Jul 20 05:17 /opt/mysql/db1.example.com.crt -r--------. 1 mysql mysql 3247 Jul 20 05:15 /opt/mysql/db1.example.com.key Verified that the ssl libraries were linked against the mysqld binary
[root@db1:~] #ldd $(which mysqld) | grep ssl libssl.so.10 => /lib64/libssl.so.10 (0x00007f79669ba000) And restarted mariadb. And if I take a look at my SSL variables in my mysql command line, everything is looking good.
MariaDB [mysql]> show variables like '%ssl%'; +---------------+---------------------------------+ | Variable_name | Value | +---------------+---------------------------------+ | have_openssl | YES | | have_ssl | YES | | ssl_ca | /opt/mysql/ca.crt | | ssl_capath | | | ssl_cert | /opt/mysql/db1.example.com.crt | | ssl_cipher | | | ssl_key | /opt/mysql/db1.example.com.key | +---------------+---------------------------------+ 7 rows in set (0.00 sec) If I create one replication user without SSL required, he is able to log in from node 2 to node 1 no problem:
MariaDB [mysql]> grant replication slave on *.* to 'slave1'@'db2.example.com' identified by 'secret'; Query OK, 0 rows affected (0.00 sec) [root@db2:~] #mysql -uslave1 -p -h db1.example.com Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 15 Server version: 5.5.41-MariaDB MariaDB Server Copyright (c) 2000, 2014, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> If I show grants on this user I can confirm that SSL is not required
MariaDB [mysql]> show grants for 'slave1'@'db2.example.com'; +--------------------------------------------------------------------------------------------------------------------------------------------------------+ | Grants for [email protected] | +--------------------------------------------------------------------------------------------------------------------------------------------------------+ | GRANT REPLICATION SLAVE ON *.* TO 'slave1'@'db2.example.com' IDENTIFIED BY PASSWORD '*somelongpasswordhash' | +--------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) If I create the second slave user with the SSL requirement, and even flush privileges, I can not log into node 1 from node 2
MariaDB [mysql]> grant replication slave on *.* to 'slave2'@'db2.example.com' identified by 'test' require ssl; Query OK, 0 rows affected (0.00 sec) MariaDB [mysql]> flush privileges; Query OK, 0 rows affected (0.00 sec) [root@db2:~] #mysql -uslave2 -p -h db1.example.com Enter password: ERROR 1045 (28000): Access denied for user 'slave2'@'db2.example.com' (using password: YES) And if I look at the grants for this user I able to tell that ssl is required for him to be able to login:
MariaDB [mysql]> show grants for 'slave2'@'db2.example.com; +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Grants for [email protected] | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | GRANT REPLICATION SLAVE ON *.* TO 'slave2'@'db2.example.com' IDENTIFIED BY PASSWORD '*somelongpasswordhash' REQUIRE SSL | +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) If I take a look at the error log:
[root@db1:~] #grep error /etc/my.cnf log-error=/var/log/mariadb/mariadb_error.log [root@db1:~] #tail /var/log/mariadb/mariadb_error.log 150720 5:18:24 InnoDB: Initializing buffer pool, size = 128.0M 150720 5:18:24 InnoDB: Completed initialization of buffer pool 150720 5:18:24 InnoDB: highest supported file format is Barracuda. 150720 5:18:24 InnoDB: Waiting for the background threads to start 150720 5:18:25 Percona XtraDB (http://www.percona.com) 5.5.40-MariaDB-36.1 started; log sequence number 120637807 150720 5:18:25 [Note] Plugin 'FEEDBACK' is disabled. 150720 5:18:25 [Note] Server socket created on IP: '0.0.0.0'. 150720 5:18:25 [Note] Event Scheduler: Loaded 0 events 150720 5:18:25 [Note] /usr/libexec/mysqld: ready for connections. Version: '5.5.41-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server I see absolutely NO indication as to why the login for the 'slave2' user (that requires SSL) is failing.
And I notice that if I do a \s it tells me that ssl is not in use:
MariaDB [(none)]> \s -------------- mysql Ver 15.1 Distrib 5.5.41-MariaDB, for Linux (x86_64) using readline 5.1 Connection id: 36 Current database: Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server: MariaDB Server version: 5.5.41-MariaDB MariaDB Server Protocol version: 10 Connection: Localhost via UNIX socket Server characterset: latin1 Db characterset: latin1 Client characterset: utf8 Conn. characterset: utf8 UNIX socket: /var/lib/mysql/mysql.sock Uptime: 22 hours 9 min 2 sec Threads: 1 Questions: 311 Slow queries: 0 Opens: 39 Flush tables: 3 Open tables: 24 Queries per second avg: 0.003 -------------- From the above output, there's this line:
SSL: Not in use
Why would that be the case, given the way I've configured this? How can I correct this?
Also 1) how to I bump up the verbosity on the logs so I can get an indication as to why this is failing? and 2) what is the best way to troubleshoot this?
OK so question 2 may seem a little redundant to question 1. But I am truly stumped.
Any help would be appreciated.
