# Debugging Slab Cache Pressure in Kontric Construction Portfolios The deployment of the [Kontric – Construction Company WordPress Theme](https://gplpal.com/product/kontric-construction-company-wordpress-theme/) on a cluster of Rocky Linux 9 servers revealed a specific performance profile during the rendering of project portfolios. The theme is designed to showcase construction site progress through high-resolution image galleries and complex project meta-data. Our stack consists of Nginx 1.24, PHP 8.2-FPM, and MariaDB 10.11, running on an XFS-formatted NVMe partition. After several weeks of content entry, we observed a subtle increase in the 99th percentile of response times for the `/projects/` archive page. This was not a resource exhaustion issue in the traditional sense; CPU and memory usage remained well within nominal ranges. Instead, we noted an intermittent I/O wait jitter that coincided with the directory traversal required to load the construction project assets. ### The Hook: Metadata Jitter in Construction Asset Directories The project portfolio within the Kontric theme creates a significant volume of metadata. Each construction project entry utilizes multiple custom post types and a dedicated subdirectory structure within the `wp-content/uploads/` path. As the number of completed projects grew, the directory containing these high-resolution assets reached a density where the Linux Virtual File System (VFS) began to exhibit overhead during lookup operations. The jitter was most prominent when the PHP-FPM workers attempted to verify the existence of specific thumbnails—an operation common in construction themes where different image ratios are required for desktop and mobile project grids. We are not looking at a volume issue, but a metadata lookup latency issue. In environments where users often [Free Download WooCommerce Theme](https://gplpal.com/product-category/wordpress-themes/) and plugin assets for testing, the filesystem is often overlooked in favor of database tuning. However, the Kontric theme's reliance on specific project folders means that the kernel must constantly resolve paths like `uploads/kontric-projects/2023/bridge-reconstruction/stage-1/`. On XFS, this involves searching through allocation groups for the specific inode associated with each file. ### Diagnostic Path: Analysis via iostat and slabtop To isolate the cause, we moved away from application-level profiling and focused on the kernel's memory allocation for filesystem objects. Using `iostat -xz 1`, we observed that while the `avgqu-sz` (average queue size) remained low, the `await` time for the NVMe device spiked to 12ms during portfolio reloads, which is excessive for solid-state storage. This suggested that the bottleneck was not the data transfer itself, but the time spent waiting for the metadata response. We then utilized `slabtop -o` to monitor the kernel's internal object caches. The output showed that the `dentry` and `xfs_inode` caches were consuming an increasing share of the slab. Specifically, the `xfs_inode` object count was high, but the "active" count was fluctuating. This indicated that the kernel was frequently reclaiming these objects to satisfy other memory requests, only to have to reload them from the disk immediately after. The `vfs_cache_pressure` setting was at the default of 100, which, in our environment, was causing the kernel to reclaim dentry and inode caches too aggressively. ### VFS and Dentry Cache Deep Dive The Linux VFS layer maintains the dentry cache (directory entry cache) to bridge the gap between file names and inodes. For a theme like Kontric, which might have 50 project folders each containing 100 images (each with 10 thumbnails), the number of directory entries is considerable. When a PHP-FPM process calls `file_exists()` or `is_readable()` on a project asset, the kernel first looks in the dentry cache. If the dentry is not found, it must read the directory block from the disk. Inside `slabtop`, we tracked the `dentry` slab. The size of the dentry cache is proportional to the number of unique paths accessed. Because construction projects often have long, descriptive file names for SEO and internal tracking, the memory footprint per dentry is higher than average. When the dentry cache is evicted, the subsequent `stat()` call in PHP-FPM blocks, leading to the observed jitter. ### XFS Allocation Groups and Inode Locality XFS divides a partition into Allocation Groups (AGs) to manage I/O in parallel. For the Kontric theme, the way assets are written to the disk affects the AG layout. When project images are uploaded in batches, they are likely to be contiguous within an AG. However, as projects are updated over time—adding new construction stages or floor plans—the inodes for a single project portfolio may become scattered across different AGs. When the kernel needs to load an inode, it calculates the position based on the inode number. If the inodes are fragmented across AGs, the disk head (or in the case of NVMe, the controller logic) must navigate multiple metadata structures. By using `xfs_db` to inspect the inode layout, we found that the project directory inodes were not optimally clustered. This was exacerbated by the theme's "Project Log" feature, which writes small text files for site updates into the same directories as the large images. ### Memory Pressure and Slab Reclaim The slab allocator is responsible for managing memory for small objects. When the system faces memory pressure, the `kswapd` daemon or direct reclaim processes will look at the slab caches to free up pages. The `dentry` and `inode` caches are often the first targets because they are marked as shrinkable. In our Rocky Linux 9 configuration, the presence of other background tasks—such as backup synchronization for the project galleries—created transient memory spikes. These spikes triggered the slab reclaimer. Because the Kontric theme assets are accessed frequently but sporadically (as different users browse different projects), the kernel's "Least Recently Used" (LRU) algorithm for dentries was not effectively keeping the active project metadata in memory. ### PHP-FPM and the stat() Cache PHP itself has an internal `stat` cache. However, this cache is cleared at the end of every request. For a long construction portfolio page, a single request might perform 200 `stat()` calls to determine which thumbnails to display. If the kernel's dentry cache is cold, every one of these calls results in a context switch to the kernel and a potential disk I/O. We checked the `open_basedir` restriction in `php.ini`. While a necessary security feature, `open_basedir` prevents the PHP-FPM process from using the system's global realpath cache efficiently. For every file access, PHP must verify every parent directory in the path against the `open_basedir` string. This effectively multiplies the number of dentry lookups. For a construction project path five levels deep, this is five additional lookups per file. ### Tuning the vfs_cache_pressure The core of the fix involved adjusting how the kernel prioritizes its metadata caches. By default, `vm.vfs_cache_pressure = 100` tells the kernel to reclaim dentries and inodes at the same rate as page cache (file data). For a metadata-heavy construction theme like Kontric, the metadata is more valuable than the file data. If the file data for a 5MB construction site photo is evicted from the page cache, the subsequent sequential read is fast on NVMe. If the metadata for the directory is evicted, the random lookup to find the file is slow. We reduced `vm.vfs_cache_pressure` to 50. This instructs the kernel to be 50% less aggressive in reclaiming dentries and inodes. Following this change, `slabtop` showed a more stable `xfs_inode` count, and the `await` time in `iostat` dropped back to sub-millisecond levels during portfolio renders. ### XFS Log Buffer and Delaylog Another aspect we tuned was the XFS log. The Kontric theme's construction log feature produces frequent metadata updates. XFS uses a journaling log to track these changes. If the log buffers are too small, the system must flush the log to the disk more frequently, which can block read operations. We increased the number of log buffers and the buffer size using the `logbsize` mount option. We also ensured the `delaylog` option was enabled (which is the default in newer kernels). This allows XFS to aggregate metadata updates in memory before writing them to the journal, reducing the overall I/O overhead of the project logging system. ### Directory Hash Indexes in XFS XFS uses a version 2 directory format which utilizes a B+tree for indexing entries within a directory. For very large project galleries, this index is efficient. However, it can become fragmented if files are frequently added and deleted. Since the construction projects in the Kontric theme are mostly "write-once" (except for the occasional thumbnail regeneration), the B+tree remains relatively clean. However, we used `xfs_fsr` (filesystem reorganizer) to ensure the directory extents were contiguous. While XFS is generally resistant to fragmentation, metadata fragmentation within an AG can still occur. Reorganizing the specific partition ensured that the directory blocks were as close to the inode blocks as possible, further reducing lookup latency. ### PHP-FPM Realpath Cache Tuning To assist the kernel, we also looked at the PHP layer. The `realpath_cache_size` in `php.ini` was increased from the default 4096k to 16M. This allows PHP to store the resolved paths of the construction project images across the duration of a single request more effectively. While this doesn't help across different requests (as the cache is per-process), it significantly reduces the number of duplicate system calls when the Kontric theme renders multiple galleries on a single page. We also adjusted `realpath_cache_ttl` to 3600 seconds. Since construction project directory structures are unlikely to change within an hour, this provides a stable cache for the PHP workers, reducing the total load on the kernel's dentry cache. ### Slab Memory Fragmentation In `slabtop`, we noticed that the `xfs_inode` slab had a high "wasted" memory percentage. This is a symptom of slab fragmentation, where a slab page cannot be freed because one or two objects are still in use. This often happens on long-running servers where memory has been allocated and freed in a disorganized manner. We performed a manual drop of the caches using `echo 2 > /proc/sys/vm/drop_caches` during a low-traffic window. This forced the kernel to clear the dentry and inode caches and re-allocate them in a more contiguous manner. Following the drop, the slab efficiency for `xfs_inode` improved from 72% to 94%. This is a temporary measure, but it provided a clean state for the new `vfs_cache_pressure` settings to take effect. ### Monitoring Directory Extent Counts We used `xfs_bmap -v` on the `uploads/kontric-projects/` directory to see the extent count. A directory is essentially a special file that contains names and inode numbers. If a construction project has thousands of files across different versions, this directory file can become fragmented into multiple extents. In our case, the main project directory had 12 extents. This meant the kernel had to perform multiple lookups just to read the directory map. We used the `xfs_db` tool to verify the block distribution. While we did not move the directory to a new partition, knowing the extent count allowed us to anticipate which projects might exhibit higher latency. ### Transparent Huge Pages and Slab Cache We also investigated the role of Transparent Huge Pages (THP). While THP is generally beneficial for large memory allocations, it can sometimes interfere with the slab allocator's ability to reclaim memory efficiently, as the granularity of reclamation is much larger. We set `transparent_hugepage` to `madvise` in the kernel boot parameters. This ensures that only applications specifically requesting huge pages (like the MariaDB buffer pool) use them, while the kernel's internal slab management for the construction theme metadata remains on standard 4KB pages. This change stabilized the memory footprint of the PHP-FPM workers. Previously, we saw workers occasionally ballooning in size as they interacted with the filesystem metadata under THP. Moving to 4KB pages for the VFS metadata reduced the frequency of direct reclaim events. ### Inode 64-bit Addressing On larger XFS partitions, inodes can be placed anywhere on the disk, resulting in 64-bit inode numbers. Some older applications or 32-bit libraries can have issues with this, but our stack is fully 64-bit. We ensured the `inode64` mount option was active (it is the default in Rocky Linux 9). This allows XFS to place inodes close to the data in all AGs, rather than forcing all inodes into the first 1TB of the disk. For the high-res images in the Kontric theme, this ensures better locality between the metadata and the actual construction project photos. ### Block Device Scheduler Impact Finally, we looked at the block device scheduler. For NVMe, the `none` or `mq-deadline` scheduler is typically used. We were using `none`. We tested `mq-deadline` to see if its ability to prioritize read operations over write operations would help the portfolio loading time. The `mq-deadline` scheduler helped slightly by ensuring that the metadata reads for the project galleries were not delayed by the background log writes of the construction log feature. We tuned the `read_expire` parameter in `/sys/block/nvme0n1/queue/iosched/` to 50ms, ensuring that metadata reads are prioritized. This further reduced the tail latency for users browsing the construction projects. ### System Configuration Snippet For sites running the Kontric theme with extensive project portfolios, the following kernel and filesystem adjustments are recommended to manage inode and dentry pressure. ```bash # Reduce pressure on VFS cache to retain construction project metadata sysctl -w vm.vfs_cache_pressure=50 # Optimize PHP-FPM for large project directory structures # Add to php.ini realpath_cache_size = 16M realpath_cache_ttl = 3600 # Mount XFS with optimized log buffers for construction logging # Add to /etc/fstab UUID=xxxx-xxxx /var/www xfs defaults,noatime,logbsize=256k,inode64 0 0 ``` Ensure that the directory structure under `wp-content/uploads/kontric-projects/` is not overly flat. While XFS handles large directories well, keeping project assets organized by year or region helps maintain better locality within the B+tree index and reduces the memory footprint of individual dentry lookups. Monitoring the `xfs_inode` slab with `slabtop` should be a routine part of the maintenance cycle for construction themes with high asset counts. Avoid using `atime` updates on construction project galleries; the metadata write overhead for every image view is unnecessary and exacerbates slab pressure. The `noatime` mount option is a mandatory baseline for this theme's performance profile. Final check on the NVMe device using `smartctl` confirms that the jitter was purely a software/kernel configuration issue and not a symptom of NAND wear or controller failure. The system now handles the construction project galleries with consistent sub-millisecond metadata lookups. Maintain a strict inode count monitor. If the `xfs_inode` count in `slabtop` exceeds 500,000 objects, consider increasing the physical RAM allocated to the kernel slab to prevent reclaim thrashing. The Kontric theme's project grid is only as fast as the filesystem that serves its metadata. Stop checking the database for project portfolio slowness; check the slab.
有疑问加站长微信联系（非本文作者））
