Download as PDF, PPTX
![Copyright 2018 Severalnines AB •Swapping - a process of moving data from memory to disk •MySQL doesn’t like active swapping - we want data in memory, not on disk but we even more don’t want to be killed when system runs out of memory •vm.swappiness = 1 - starting from kernel 3.5 this is the setting you want to use - allow some swapping but only when there’s no other option •You can also deprioritize MySQL when it comes to OOM Killer: echo -1000 > /proc/[pid]/oom_score_adj should help in environments shared with other processes Tuning the operating system for MySQL - memory](https://image.slidesharecdn.com/mysqlmariadbperformancetuning-1-180627110658/75/Webinar-slides-Our-Guide-to-MySQL-MariaDB-Performance-Tuning-18-2048.jpg)
![Copyright 2018 Severalnines AB •When running MySQL on bare iron, you need to make sure you have NUMA interleave policy set •Starting from 5.6.27 a variable is available: innodb_numa_interleave •Before that you could run MySQL as: numactl --interleave=all $command •Memory allocation library may impact performance of MySQL Many choices: glibc malloc, tcmalloc and jemalloc Use LD_PRELOAD or malloc-lib variable in [mysqld_safe] section of my.cnf Tuning the operating system for MySQL - memory](https://image.slidesharecdn.com/mysqlmariadbperformancetuning-1-180627110658/75/Webinar-slides-Our-Guide-to-MySQL-MariaDB-Performance-Tuning-19-2048.jpg)
Uploaded bySeveralnines
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
The document is a comprehensive guide on tuning MySQL and MariaDB performance, detailing the tuning process, configuration settings, and operating system optimizations necessary for optimal database performance. It emphasizes the importance of a test environment, the iterative nature of tuning, and provides practical tips and tools for effective performance management. Additionally, it covers updates in MySQL 8.0 and offers resources for further learning.
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
- 1. krzysztof@severalnines.com Copyright 2018 SeveralninesAB Presenter Krzysztof Książek, Senior Support Engineer @Severalnines Guide to MySQL & MariaDB Performance Tuning 26th June 2018
- 2. Copyright 2018 SeveralninesAB I'm JJ from the Severalnines Team and I'm your host for today's webinar! Feel free to ask any questions in the Questions section of this application or via the Chat box. You can also contact me directly via the chat box or via email: jj@severalnines.com during or after the webinar. Your host & some logistics
- 4.
- 5. Copyright 2017 SeveralninesAB Automation & Management Deployment # Deploy a Cluster in Minutes # On-Prem or Cloud (AWS/Azure/Google) Monitoring # Systems View with 1 sec Resolution # DB / OS stats & Performance Advisors # Configurable Dashboards # Query Analyzer # Real-time / historical Management # Backup Management # Upgrades & Patching # Security & Compliance # Operational Reports # Automatic Recovery & Repair # Performance Management # Automatic Performance Advisors
- 6. Copyright 2017 SeveralninesAB Supported Databases
- 7. Copyright 2017 SeveralninesAB Our Customers
- 8. krzysztof@severalnines.com Copyright 2018 SeveralninesAB Presenter Krzysztof Książek, Senior Support Engineer @Severalnines Guide to MySQL & MariaDB Performance Tuning 26th June 2018
- 9. Copyright 2018 SeveralninesAB •What to tune and why? •Tuning process •Operating system tuning Memory I/O performance Agenda •MySQL configuration tuning Memory I/O performance •Useful tools •Do’s and do not’s of MySQL tuning •What’s new in MySQL 8.0?
- 10. Copyright 2018 SeveralninesAB •Operating system Swapping I/O scheduler CPU scheduler TCP/IP stack Configuration tuning - what to tune and why? MySQL configuration •Per-session buffers •InnoDB Buffer Pool •InnoDB I/O settings
- 11. Copyright 2017 SeveralninesAB Copyright 2018 Severalnines AB Tuning Process
- 12. Copyright 2018 SeveralninesAB •Never-ending story which starts once you install MySQL on the host •You have to tune for a specific workload •Workload may change in time More data can make it I/O-bound Different query mix may increase CPU load and put stress on different parts of the InnoDB •Keep in mind that configuration tuning is not likely to give you a huge increase in performance (except if the server is really badly configured) Tuning process
- 13. Copyright 2018 SeveralninesAB Tuning process
- 14. Copyright 2018 SeveralninesAB •You need a deterministic, test environment to make sure you can measure the impact of the changes •Environment should mirror production as close as possible, to make it more relevant •Changes should be introduced one at a time to ensure you understand the impact of each of them •Benchmark the system using queries as close to production as possible •Restore it to the original state for another round of tweaking •Rinse and repeat until you are happy with results Tuning process
- 15. Copyright 2018 SeveralninesAB •Grab a backup of your production systems •Restore it on a host, restart MySQL or reboot the host itself to clear caches •Capture real-world queries using slow log or tcpdump •Do a baseline run, replay queries using Percona Playback or pt-upgrade •Restore backup again, restart MySQL or reboot the host itself to clear caches •Make _one_ change in my.cnf or OS settings •Replay queries using Percona Playback or pt-upgrade •Measure the difference, repeat the process by restoring the backup if you want to make one more change Tuning process
- 16. Copyright 2017 SeveralninesAB Copyright 2018 Severalnines AB Tuning the operating system for MySQL
- 17. Copyright 2018 SeveralninesAB •Minimize the flushing operations in OS to ensure no stalls are happening when dirty data is flushed to disk vm.dirty_ratio - a hard limit of the memory that can be used to cache dirty disk pages vm.dirty_background_ratio - percentage of system memory that can be used to cache modified (“dirty”) pages before the background flush process kicks in •We want them to be set to low numbers - 5 - 10% to trigger flushing often and make it quick •Default settings may result in a situation where 40% of your memory has to be flushed to disk at once For large instances (128GB) it can be even >50GB of data to flush Tuning the operating system for MySQL - memory
- 18. Copyright 2018 SeveralninesAB •Swapping - a process of moving data from memory to disk •MySQL doesn’t like active swapping - we want data in memory, not on disk but we even more don’t want to be killed when system runs out of memory •vm.swappiness = 1 - starting from kernel 3.5 this is the setting you want to use - allow some swapping but only when there’s no other option •You can also deprioritize MySQL when it comes to OOM Killer: echo -1000 > /proc/[pid]/oom_score_adj should help in environments shared with other processes Tuning the operating system for MySQL - memory
- 19. Copyright 2018 SeveralninesAB •When running MySQL on bare iron, you need to make sure you have NUMA interleave policy set •Starting from 5.6.27 a variable is available: innodb_numa_interleave •Before that you could run MySQL as: numactl --interleave=all $command •Memory allocation library may impact performance of MySQL Many choices: glibc malloc, tcmalloc and jemalloc Use LD_PRELOAD or malloc-lib variable in [mysqld_safe] section of my.cnf Tuning the operating system for MySQL - memory
- 20. Copyright 2018 SeveralninesAB •Increase buffers: net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 # Interface buffering net.core.netdev_max_backlog = 2500 Tuning the operating system for MySQL - network •Increase connection backlog: net.ipv4.tcp_max_syn_backlog = 4096 net.core.somaxconn = 1024 •Reduce timeouts: net.ipv4.tcp_syn_retries = 0 net.ipv4.tcp_synack_retries = 0 net.ipv4.tcp_keepalive_time = 30 net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 2
- 21. Copyright 2018 SeveralninesAB •Disk schedulers may impact performance of I/O-bound workloads •CFQ is tuned for desktop workloads •Noop or deadline are usually better for MySQL Benchmark to see which one of them will work better for you •Filesystem - most often you’ll use EXT4 or XFS Performance depends on workload, hardware and kernel version Benchmark to see which one will work better in your environment Tuning the operating system for MySQL - I/O
- 22. Copyright 2017 SeveralninesAB Copyright 2018 Severalnines AB Tuning MySQL configuration
- 23. Copyright 2018 SeveralninesAB •InnoDB buffer pool - used to cache data and store dirty pages •More is better but you need to leave some memory for other buffers Per join buffers Per session buffers Temporary tables •You may have heard about 80% rule It’s more like 90% for large (i.e. 128GB) hosts Tuning MySQL configuration - memory •Make sure you err on the side of ‘too small’ •Unless you run MySQL 5.7 or 8.0 where you can resize InnoDB buffer pool dynamically, without restart •For fairly loaded (~20-30 running threads) host with 128GB of memory it should be ok to leave ~15GB of memory free •All depends on the workload so your mileage may vary
- 24. Copyright 2018 SeveralninesAB •Per-session buffers in InnoDB: sort_buffer_size, read_buffer_size, read_rnd_buffer_size •Per-join buffer: join_buffer_size •By default - small values •More _not_ always better At 256KB the way how memory allocates change smaller chunks use malloc() which is faster than mmap() •Make sure to benchmark your system after any change to those settings Tuning MySQL configuration - memory
- 25. Copyright 2018 SeveralninesAB •innodb_flush_log_at_trx_commit - governs the durability in InnoDB 1 - full ACID compliance 2 - you may lose up to 1s of transactions when hardware crashes 0 - you may lose up to 1s of transactions when MySQL crashes •Significant change in the I/O performance - less flushes means less I/O and less overhead •Pick whatever you like and whatever you need Slaves may not require full durability if you have many of them Galera Cluster nodes may also not require full durability Tuning MySQL configuration - I/O performance
- 26. Copyright 2018 SeveralninesAB •innodb_io_capacity, innodb_io_capacity_max and innodb_lru_scan_depth - define number of disk operations InnoDB can execute •Set it too low and you may not fully utilize your hardware •More not always better - aggressive flushing is not always the best option Redo logs are there for a reason - to minimize number of writes to tablespaces •innodb_flush_method: O_DIRECT for BBU-backed hardware O_DSYNC may work better with SAN Benchmark your setup before you go live Tuning MySQL configuration - I/O performance
- 27. Copyright 2018 SeveralninesAB •InnoDB Redo Logs are used to store write transactions and they are written sequentially •MySQL must not run out of space in them •Larger logs help with better write merging •Larger logs help with more stable flushing •Larger logs may seriously impact recovery time in case of a crash •The rule of thumb is to make them large enough to store at least 1h of writes Tuning MySQL configuration - I/O performance
- 28. Copyright 2018 SeveralninesAB Tuning MySQL configuration
- 29. Copyright 2018 SeveralninesAB •innodb_buffer_pool_instances, table_open_cache_instances •metadata_locks_hash_instances, innodb_adaptive_hash_index_partitions •Those options can help you to reduce contention on some of those structures •Increase number of buffer pools or adaptive hash index partitions if you notice a congestion on them •Or, preemptively, if you have to handle highly concurrent traffic •Don’t use buffer pool instances smaller than 1GB (use 2GB+, more instances can slow down the system) Tuning MySQL configuration - contentions
- 30. Copyright 2018 SeveralninesAB •max_connections - keep it large enough to handle incoming connections •If you need to handle thousands of connections, check the connection pooling options or a proxy, ideally with connection multiplexing (ProxySQL) •log_bin - you want to have binlogs enabled Consider sync_binlog=0 for MySQL 5.7 - it used to be a default on MySQL 5.6, it has been changed in 5.7.7. Better performance, less durability •skip_name_resolve - just to make sure your database won’t suffer when DNS will not be reachable •Query cache - optimize it away by disabling Use external caching layer (ProxySQL, Redis, Memcached) Tuning MySQL configuration
- 31. Copyright 2017 SeveralninesAB Copyright 2018 Severalnines AB Tools useful in tuning the configuration
- 32. Copyright 2018 SeveralninesAB •Percona developed two great tools, part of the Percona Toolkit suite, which helps you to collect useful data about MySQL and underlying operating system •pt-summary collects data about hardware and OS settings CPU, memory, disk, load on the system •pt-mysql-summary connects to MySQL instance and gives you data about workload and configuration Type of queries that are running, handlers, configuration snapshot of current traffic and many others Tools useful in tuning the configuration
- 33. Copyright 2018 SeveralninesAB Tools useful in tuning the configuration - pt-summary
- 34. Copyright 2018 SeveralninesAB Tools useful in tuning the configuration - pt-mysql-summary
- 35. Copyright 2018 SeveralninesAB Tools useful in tuning the configuration - pt-mysql-summary
- 36. Copyright 2017 SeveralninesAB Copyright 2018 Severalnines AB Do’s and do not’s of MySQL tuning
- 37. Copyright 2018 SeveralninesAB •Tuning MySQL is not a simple task - it’s very time-consuming to go through the process of change - test - change - test… •People like to simplify things and look for shortcuts •Auto-tuning tools have been created mysqltuner.pl •Configurators and wizards have been created (i.e. by Percona) •Monitoring and management tools offer advisors (i.e. ClusterControl) What to avoid when tuning MySQL?
- 38. Copyright 2018 SeveralninesAB •Some of those tools base on counters from MySQL internal status and this is tricky The fact you have joins performed without indices doesn’t justify increasing join_buffer_size •Sometimes making a good change is a bad decision If you run MySQL without binlog enabled, without double_buffer or without XA enabled, you may see a suggestion to enable those features Binlog is great for point-in-time recovery double_buffer and XA helps keep your data safe Enabling them will add serious I/O load on the system - it may not handle it What to avoid when tuning MySQL?
- 39. Copyright 2018 SeveralninesAB •Make sure you follow the tuning process as we described - one change at a time •If you use external tools that provide suggestions, take them with a grain of salt How competent is the vendor in databases? Even if the vendor has the competence - do I understand what this change is and how it may impact my system? •If you setup a new host, you may use external tools more freely This would be initial tuning, not changing the way how application already works •Make sure you use common sense and each of your changes has a reason What to keep in mind when tuning MySQL?
- 40. Copyright 2017 SeveralninesAB Copyright 2018 Severalnines AB What’s new in MySQL 8.0?
- 41. Copyright 2018 SeveralninesAB •Undo tablespaces are now configurable at runtime •innodb_dedicated_server=1, autotunes: innodb_buffer_pool_size innodb_log_file_size innodb_flush_method •Configuration persistence with SET PERSIST What’s new in MySQL 8.0? •innodb_log_buffer_size is now dynamic •Spin delay is now configurable for threads waiting for redo log flushing innodb_log_wait_for_flush_spin_hwm innodb_log_spin_cpu_abs_lwm innodb_log_spin_cpu_pct_hwm
- 42. Copyright 2018 SeveralninesAB •Blogs in “Become a MySQL DBA” series: http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance •How to Benchmark Performance of MySQL & MariaDB using SysBench https://severalnines.com/blog/how-benchmark-performance-mysql-mariadb-using-sysbench •How to Design Highly Available Open Source Database Environments https://severalnines.com/resources/whitepapers/how-design-highly-available-open-source-database- environments •MySQL Replication Blueprint http://severalnines.com/blog/new-whitepaper-mysql-replication-blueprint • Contact: jj@severalnines.com Thank you!