Linux Performance Tuning — CPU, Memory, Disk & Network Optimisation

Performance Analysis Methodology — USE Method & Bottleneck Identification

Effective Linux performance tuning begins with a structured analysis to identify bottlenecks and prioritize optimization efforts. The USE Method—comprising Utilization, Saturation, and Errors—is a systematic approach to evaluate system components such as CPU, memory, disk, and network. By measuring these metrics, administrators can determine which resource is limiting system performance and address it accordingly.

Utilization indicates how much of a resource is actively used; Saturation reveals whether the resource is reaching its maximum capacity; Errors highlight issues like dropped packets or failed I/O operations. For example, high CPU utilization coupled with low saturation suggests the CPU is busy but not fully saturated, perhaps due to waiting on I/O or network delays. Conversely, high saturation with errors indicates a critical bottleneck requiring immediate attention.

Identifying bottlenecks involves leveraging tools like top, htop, iostat, vmstat, and sar. For CPU analysis, tools like mpstat and perf provide granular insights into core-specific usage and kernel performance counters. Memory bottlenecks can be diagnosed using vmstat and free, highlighting swap activity and free memory levels that may cause thrashing.

Disk I/O bottlenecks are identified through iostat and iotop, which reveal I/O wait times and processes involved in heavy disk activity. Network issues can be diagnosed with iftop or iperf3, measuring throughput and packet loss. The critical step in Linux performance tuning is correlating these metrics to understand the root cause of slowdowns and formulate targeted solutions.

CPU Performance — mpstat, perf, load average & CPU Scheduling

CPU performance tuning is fundamental in optimizing Linux server operations. Tools like mpstat and perf enable detailed analysis of CPU utilization, process scheduling, and kernel performance counters. The load average metrics—displayed via uptime or top—provide a snapshot of CPU demand over 1, 5, and 15-minute intervals, reflecting how busy the system is.

mpstat offers per-core utilization statistics, helping identify uneven load distribution or hyper-threading inefficiencies. Example command:

mpstat -P ALL 1

This outputs CPU usage per core every second, revealing cores that are under or over-utilized. perf allows in-depth profiling of system calls, CPU cycles, cache misses, and kernel events, enabling pinpointing of performance bottlenecks at the instruction level. For example:

perf stat -e cycles,instructions,cache-misses -a sleep 10

Understanding CPU scheduling is crucial for performance tuning. The Linux kernel scheduler manages process execution order, affecting responsiveness and throughput. Tuning parameters like sched_latency_ns, sched_wakeup_granularity_ns, and sched_min_granularity_ns via /proc/sys/kernel or sysctl can optimize context switches and process fairness in high-load environments.

For example, decreasing sched_wakeup_granularity_ns reduces latency for interactive processes, improving user experience. Conversely, increasing sched_min_granularity_ns can prevent CPU thrashing under heavy loads. Properly configuring these kernel parameters, along with monitoring with top and htop, ensures optimal CPU resource utilization in production environments. To learn more about advanced Linux server optimisation, visit Networkers Home.

Memory Management — vmstat, free, swap, OOM Killer & Hugepages

Memory management is a critical aspect of Linux performance tuning. Tools like vmstat and free provide real-time insights into memory usage, swap activity, and system cache efficiencies. High swap usage indicates insufficient RAM or memory leaks, leading to increased disk I/O and degraded performance. For example:

vmstat 1 5

This command samples memory stats every second for five iterations, helping spot abnormal memory consumption patterns. The swap partition acts as an overflow buffer, but excessive swapping causes I/O bottlenecks. Monitoring swapon -s and free -m helps assess whether swap needs tuning or if physical memory upgrades are necessary.

The Out-Of-Memory (OOM) Killer is invoked when the system cannot allocate memory, terminating processes to recover stability. Tuning OOM behavior involves adjusting vm.overcommit_memory and vm.overcommit_ratio. For example, setting vm.overcommit_memory=2 enforces strict overcommit limits, preventing memory over-allocation.

Hugepages provide a performance boost for applications like databases or virtual machines that benefit from large, contiguous memory pages. Configuring hugepages involves editing /etc/sysctl.conf and mounting hugepages manually. Example configuration:

vm.nr_hugepages=1024

Optimizing memory can significantly impact latency and throughput. Ensuring proper kernel parameters and monitoring memory metrics helps maintain system stability and performance. For comprehensive memory tuning strategies, visit Networkers Home Blog.

Disk I/O — iostat, iotop, Filesystem Tuning & I/O Schedulers

Disk I/O performance often becomes a bottleneck in high-throughput Linux servers. Tools like iostat and iotop provide visibility into I/O wait times, throughput, and process-level disk activity. For instance, running iostat -xz 1 offers detailed device utilization metrics, highlighting devices with high utilization or latency issues.

Filesystem tuning involves selecting appropriate options and parameters. For example, choosing a journal mode like ext4 with data=writeback or data=ordered impacts write performance and data integrity. Tuning parameters such as commit interval or enabling/disabling barriers can optimize performance based on workload.

I/O schedulers determine how block I/O requests are ordered and dispatched. Linux offers multiple schedulers: CFQ, Deadline, NOOP, and newer options like bfq. To list available schedulers:

cat /sys/block/sdX/queue/scheduler

To change the scheduler:

echo deadline > /sys/block/sdX/queue/scheduler

Benchmarking different schedulers under workload can identify the best fit for specific applications, such as databases or web servers. Comparing their impact using tools like fio or bonnie++ provides data-driven choices. For example, Networkers Home offers advanced training to master disk optimization techniques.

Network Performance — iperf3, ethtool, TCP Tuning & Buffer Sizes

Network performance tuning is essential for maximizing throughput and minimizing latency, especially in data centers and cloud environments. Tools like iperf3 enable bandwidth testing between hosts, providing metrics on throughput, jitter, and packet loss. Example command:

iperf3 -c server_ip -t 30

Optimizing network interfaces involves configuring ethtool settings, such as enabling features like offloading, checksum offload, and interrupt moderation. For instance:

ethtool -K eth0 gro on gso on tso on

TCP tuning parameters include adjusting window sizes, congestion control algorithms, and offloading features. Modifying buffer sizes can significantly improve throughput, especially over high-latency links. Example:

sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=26214400
sysctl -w net.ipv4.tcp_rmem='4096 87380 26214400'
sysctl -w net.ipv4.tcp_wmem='4096 65536 26214400'

Choosing the right congestion control algorithm, such as BBR or Reno, can improve performance based on network conditions. Set the algorithm with:

sysctl -w net.ipv4.tcp_congestion_control=bbr

Monitoring network buffers and interface statistics with ifconfig and ethtool helps identify bottlenecks. Properly tuned network stack parameters ensure high throughput and low latency, crucial for enterprise workloads. For comprehensive network tuning techniques, refer to Networkers Home Blog.

sysctl Parameters — Kernel Tuning for Production Workloads

The sysctl interface allows real-time tuning of kernel parameters to optimize Linux performance for specific workloads. Proper sysctl tuning impacts CPU scheduling, memory management, networking, and I/O. For example, increasing the maximum number of open files:

fs.file-max = 2097152

Network parameters such as net.ipv4.tcp_tw_reuse and net.ipv4.tcp_fin_timeout influence connection reuse and timeout durations, reducing latency in high-connection environments. Example settings:

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

For disk I/O, adjusting parameters like vm.dirty_ratio and vm.dirty_background_ratio controls how aggressively data is flushed to disk, balancing throughput and latency. Example:

vm.dirty_ratio=20
vm.dirty_background_ratio=10

Kernel parameter tuning should be based on workload profiling and benchmarking data. A comparison table of common sysctl parameters for different workloads can guide tuning efforts:

Applying these configurations requires thorough testing and validation. Regular review of Networkers Home Blog provides insights into latest kernel tuning best practices for production systems.

Profiling & Tracing — strace, ltrace, perf & eBPF

Advanced Linux performance tuning involves profiling and tracing tools to diagnose application and kernel-level bottlenecks. strace and ltrace trace system calls and library calls made by processes, revealing delays caused by I/O, locking, or network operations. For example:

strace -p 

Enables real-time monitoring of a running process. perf provides low-overhead profiling of CPU cycles, cache misses, and kernel events, useful for identifying hotspots in code execution. Example command:

perf record -g -p 

Post-processing with perf report visualizes call graphs and hotspots. For even more detailed kernel tracing, eBPF (Extended Berkeley Packet Filter) allows custom tracing of kernel functions and network packets with high performance and minimal overhead. Tools like bpftrace and BCC scripts enable deep diagnostics in real time.

Combining these tools provides a comprehensive view of system behavior, enabling precise tuning. For instance, identifying slow system calls with strace and correlating them with CPU hotspots from perf data guides targeted kernel parameter adjustments. For advanced training on Linux profiling, visit Networkers Home.

sysbench cpu --cpu-max-prime=20000 run

[fio_job]
name=SequentialRead
ioengine=libaio
rw=read
bs=4k
numjobs=4
size=10G
runtime=60
group_reporting

fio fio_job.fio

stress-ng --cpu 4 --timeout 120s --metrics-brief