If you have been a Linux user for some time now, then you must have come across more than one article telling you how to improve the speed and responsiveness and whatnot of your Linux distribution at home. In fact, the Web is awash with guides and tips on tuning your system to the max, offering the promise of greatness to those bold enough to redirect values into /proc.

The reason why we are here, today, we few, we happy few, we band of geeks, is to discuss this delicate topic, or rather, to debunk it. Because Linux desktop tweaking is nothing short of a big, juicy placebo, right there alongside Windows tweaking, which gives you more or less the same outcome, only with more GUI.

Reality check

Let’s start with the overview of what Linux is. It’s a product of thousands of people. Specifically, the kernel is developed by highly skilled programmers, employed by serious companies, with tons of contributions from many more thousands of volunteers and enthusiasts across the world. The Linux kernel is used in all the top supercomputers, it powers the backbone of the Internet, it’s used for media rendering, and everything else besides. It’s a pretty robust beast, well designed to meet the rigorous demands of the world’s colorful digital markets.

Yet, for some reason, now and then, people online disagree. It ain’t tuned enough.

Does this sound plausible to you? Do you really think that occasional users know better than the collective mind of a whole army of professionals, coding veterans, kernel gurus, filesystem and network experts, and the assorted lot of knights and paladins? Do you really think that Linux is just not good enough by design that its defaults need to be altered?

With this in mind, let us take a look at some of the typical examples for where would-be tuning is supposed to render miracles with your boxes, give them that extra juice you don’t normally get. In other words, all the cases where the collective intelligence of the Linux development world fails the arbitrary forum thread post test.

I/O scheduler

Oh my, if you change noop to deadline, you will get a whatever percentage boost. Not. I have seen this written in at least nine thousand tutorials, almost copypasta, with little regard to the actual problem at hand. The thing is, applying any tunable without knowing what the target software is supposed to do is pointless. More so if you have a generic system, which serves a generic purpose, with different programs, running almost at random times, each with their own unique work profile.

But let’s put that aside. Why would changing the scheduler help? Do you mean to say the folks who ship Linux the way it is have never actually seen and tested their software, and simply decided to go with an arbitrary option? What kind of I/O are you going to be doing? What is so special about your desktop that it merits tweaking? In the worst case, you will be working with a lot of small files on the disk, in which case, you want a faster disk. In the best case, you will be working with large files, in which case, you want a faster disk. In both scenarios, you won’t go beyond the physical capabilities of the device called hard disk.

Most people do not use their hard disks that much to notice any significant improvement. In fact, let’s illustrate. Assume you need to copy ten 4GB files and one thousand 40KB files every single day, from one hard disk to another. This could be your typical backup, which presumably runs at night. I/O scheduler tweaking will probably yield minimal results when working with large files, so your total copy time will be roughly 40GB/disk speed, or approx. 100MB/sec. In other words, you will copy your big data in about 400 seconds, or seven minutes.

With the small files, I/O tweaking could help shorten the time needed to process requests. This is where we might want to take a look at the iostat command, and its output. Interesting fields are the following r/s, w/s, await, svctm. The first two tell us how many requests are being processed each second for the selected device. The third (await) one tells us the average time, in milliseconds, it took for a request to be processed. Finally, svctm (service time) is the time it takes for the relevant block device to complete the request.

There are no golden numbers, but if you look around, you will see that a typical system can process something like 500-600 requests per second. Moreover, the wait time will be at most 15-30 ms, while the service time will often not exceed 20 ms under normal load conditions. If we look at our task, which is to copy 1,000 files, even without any optimization whatsoever, the system could process all the requests in about one and a half second. And we would not notice anything, because the human threshold for interactive latency is about 400 ms. Indeed, copying 40MB of data is well below the maximum throughput of even average hard disks. And to make things worse, the copying is actually done into the disk cache first.

Yes, you may have wondered about that. But most hard disk also come with a cache, often 32-64MB in size, which is used as the target for the write requests. Data is placed there first, and then asynchronously written to the actual permanent storage. This should not concern you, whatsoever, because the disk controller is in charge of this whole scheme, and you should let it be.

But we’re talking about optimization. Let’s assume a magical 50% improvement. This means you would save virtually nothing for large files, and about half a second for the small ones. And if you could double the speed somehow, the grand improvement would still be about three and a half minutes every day.

And now, some real results. My daily home backup runs over a 1Gbps line from one standard mechanical hard disk to another. A grand total of 262,000 files (and growing) are being processed every time, with 35,500 files/min, an average of 6,700 new files are copied every day, at an average rate of 900 files/min. The total data is worth 133GB, with some 650MB copied at an average rate of about 1.5MB/sec. The backup job takes approximately seven minutes.

In this case, the disk capabilities and the I/O scheduling are less critical than the CPU power needed to read all the file attributes and compare them to the destination, in order to decide whether to perform a copy, or leave the target as it. Even your best tuning will not make even the slightest difference. On average, the disks are hardly loaded at some 2% of its total capacity.

SSD TRIM

Here’s another hot potato. It sounds fancy, but it actually means the following: Allow the operating system to inform the SSD disk controller that certain blocks of data are no longer in use, so they can be wiped. Why would we want something like this? Well, to improve write speed, supposedly. When the SSD is empty, then writing data is just one operation. When there’s old, unused junk there, then it needs to be deleted first, and then the new stuff written, so basically, two operations.

Sounds good. And therefore, the Internet is full of articles telling people to activate this function for their devices in Linux distributions that do not support the functionality out of the box, or in those cases where TRIM is available and has not been turned on by default.

First, as to the actual benefit of doing something like this, please see the previous section. As to whether you should do this, think about it. Someone, probably more knowledgeable than you has thought it prudent not to include TRIM in the system defaults, possibly because of the risk of data consistency and corruption. Not because they wanted to give you a sub-par experience. Not because they wanted their product to suck, so you can be disappointed. Most likely because the drawbacks outweigh the benefits. And now, you’re about to play with your hard disk, which contains valuable information, and you’re about to use an unsupported command. Good luck. Speed! Sure, you’ll get your three and a half minutes daily. After you spend a few hours trying to recover data from a slaughtered filesystem.

Swappiness & vm tunables

What a cool word, is it not. I have actually written about this in my third hacking article, with another long, numerical exercise on the actual benefits of making these kind of changes. Sure, tuning the swappiness values can help with old laptops, which have a shortage of physical memory and slow hard disks, but this only proves my point. Get better hardware, and Bob’s your uncle. The same applies to all memory-related tunables.

Network tuning

Upgrade your network card, your ISP, your router, something. There is no problem with your Linux. And if it’s not doing torrents just as well as you expected, the problem is most likely NOT with the internal functionality of the network stack in your kernel. Why do you think that your distro can’t handle the few MB of movies or whatnot that you’re trying to download? Do you honestly believe that all Linux developers are just 54Kbps users and that they do not care about fast Internet?

Here’s an article from a lad who knows his stuff. Most people will look at this and say, I wanna. Wait. Did you read the little paragraph that says high-performance computing? Your laptop is not a high-performance computing machine. It says, multi-user desktop, good, systems serving thousands of concurrent network clients, bad. That’s exactly the point. Linux distributions are configured for the widest range of use cases, and they offer the most balanced and optimal experience overall. You will never need anything else.

Speaking of that earlier resource, results and an actual usage profile would be cool, because otherwise, we don’t have any good visibility into how these tunables affect the server, and what the affected server might actually be. Which brings me to a personal example.

My personal experience

Now, let’s brag a little. In all my years working with Linux and tweaking the hell out of it for clear business purposes, and this is what I’m paid to do, among other cool things, I have only encountered a single situation where kernel tuning really made any real difference to the overall performance of the running software. The sad sobering reality is, most of the time, easy and quick gains are achieved by upgrading the hardware, like the CPU or disk, and fixing and optimizing the actual software. Never the kernel.

Well, except in this one case, a 32-core, 64-thread MySQL server with 128GB RAM, processing some 1,500 requests every second, and it wasn’t handling the load. I spent about a week figuring out the actual customer flow. I figured out how long it took for a single request to complete, and the rate the server was gobbling then. Them, I tweaked the TCP parameters to more quickly rinse old connections, and I carefully matched the MySQL settings to the server capabilities, in order not to create a bottleneck during the initial torrent of requests, so that the CPU queue was filled with about 128 processes, twice the number of worker threads. And so, all considered, I merely changed just three tunables under /proc, and more importantly, tweaked the MySQL memory and I/O cache values, in the software’s own configuration file. That was it.

On the other hand, there was this other situation where I spent a couple of weeks fiddling with every conceivable CPU scheduler tunable, playing with critical nanosecond values, only to get a total improvement of absolutely nothing, and simply by optimizing the multi-threading model in the customer software, I got much better and consistent results.

So yes, in the vast majority of cases, upgrade the CPU to one with a higher frequency, dump in an SSD or such, and you will have improved your system tenfold over any, most careful tuning and tweaking that you can possibly think of.

Conclusion

Server tuning is one thing. But wait. All those with a real server, raise your hand. Good. Not you. Now, desktop users, raise your hands. Excellent. You don’t need to do anything. Kernel tuning at home is black magic, voodoo, hocus-pocus, now you see me now you don’t. It’s a pointless and even dangerous exercise that will buy you a few seconds or minutes at most, and cost you all you hold dear, at worst.

This is actually true for all desktop operating systems. Let them be. Don’t poke them. The benefits are not worth the hassle, not worth the possible complications down the road, which you will not even be able to correlate to the few supposedly innocent tweaks you introduced back then. Let your system run as it is, it is already optimized. For a headache-free experience.

Cheers.