We are running a python application on an RHEL 7 VM machine hosted in Azure. Machine has 8GB of memory & 2GB of swap space configured as swap file. Below the output of free command from the server.
#-> free -h
total used free shared buff/cache available
Mem: 7.8G 3.4G 582M 381M 3.8G 3.6G
Swap: 2.0G 2.0G 0B
Here is the output of top command with list of top processing
All of them are python processes eating up whole of swap space. Whats puzzling for me is that memory is under utilized and swap is 100% utilized. We are planning to increase the swap space to 8 GB to see if system resources(memory) are optimally utilized.
I would like to understand this behavior in depth. Why is swap space getting consumed first than memory. What needs to be done for making the system performance better?
Not answering your question directly, but speaking honestly......
Swap space, especially on Linux systems in the year 2020 is a "relic of the past" (kind of like vacuum tubes before the transistor was invented) .
Why?
Because years ago, computer RAM was very expensive, so the best alternative was disk swap.
However, that equation changed many years ago; RAM became so cheap, most people (including me) never use swap because we have enough RAM; and if the machine started or "needed" to really "swap" that would be an "indicator" to upgrade RAM, because RAM is so cheap. We do not want our systems to swap, ever.
Not knowing the full details of your application, except that it is a python application running in a VM, I suggest you turn off swap and see what happens.
This is easy enough on Linux:
swapoff -a
But before you do this, it is a good idea to review the man page... here is one for Suse, but should be the same (or very similar) for RedHat:
Let me predict what will happen without swap:
the machine will have spikes of high load, and during the spikes will pause its productive work.
Perhaps it will even invoke the OOM process killer.
My observation: a Linux kernel needs (little) swap to smoothly handle certain workloads.
Because it never frees memory without "pressure", and because of the "unused RAM is wasted RAM" philosophy.
So, for safety have a few GB of swap! Even if the kernel.org developers plus a million followers swear these situations do not exist.
In comparison, the Solaris kernel tries to free memory as a background task, in order to avoid a sudden memory pressure. Solaris can run without swap.
However, this was obscured by ZFS, that was developed with a "only free cache under pressure" philosophy. So in Solaris with ZFS either limit the ZFS cache or have some swap!
Every system is different depending on the application
MadeInGermany does not have enough information to make a prediction.... accurate predictions require more knowledge than just a "guess" without system knowledge.
No one can make a prediction based on the very little information provided in the original post ( a python app running in a vm.... is very little information..... really)
In fact, I will illustrate this by turning off swap for a day on two different Linux Ubuntu server, one in a VM and one dedicated hardware (see below)
It should be clear to anyone, that if you have enough memory (and memory is very cheap) that you do not need swap on a modern Linux system. This is fairly well documented; although some swap is recommended.
There is no harm in the original poster turning off swap and observing themselves, what happens.
To further illustrate this point, I have a Linux (Ubuntu) server running (in VM mode) only MQTT services and Node-RED (for the most part), and I just did this on a real, live server in production:
root@localhost:~# swapoff -a
I am also graphing the performance, which currently looks like this:
Then, after that, I will do the same for an even busier Linux server (Ubuntu, dedicated server, running many web services and sites, MQTT, backups, and more)..... to illustrate.
24 hours from now I will post the graphs and update on this VM Ubuntu instance.
Nothing but the facts .... no guesses, no opinions, and no personal preferences in this test. Just "turn off swap" and show the results at the same time, a day later.
And tomorrow, I will take this dedicated Ubuntu server, currently looks like this (runs MQTT, mysql, apache2 and a number of web sites, does backups, etc):
top - 06:40:25 up 35 days, 6 min, 1 user, load average: 0.05, 0.12, 0.16
Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.5 us, 0.9 sy, 0.0 ni, 97.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32652952 total, 16986476 free, 694036 used, 14972440 buff/cache
KiB Swap: 7996412 total, 7996412 free, 0 used. 31071156 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
950 mysql 20 0 2749912 333616 19552 S 0.7 1.0 542:11.32 mysqld
8346 www-data 20 0 479644 32704 21104 S 0.7 0.1 0:00.02 apache2
8353 www-data 20 0 477684 24696 13064 S 0.7 0.1 0:00.02 apache2
901 www-data 20 0 343920 25120 2356 S 0.3 0.1 73:38.94 memcached
7461 www-data 20 0 477100 15708 4556 S 0.3 0.0 0:00.05 apache2
7607 www-data 20 0 477100 15708 4556 S 0.3 0.0 0:00.03 apache2
8205 www-data 20 0 477100 15708 4556 S 0.3 0.0 0:00.01 apache2
1 root 20 0 185144 5792 4040 S 0.0 0.0 2:55.18 systemd
..
...
...
...
Will do the same... will turn off swap complete for a day and compare before and after.
In this manner, everyone can see two very different real-world, live Linux server examples of running with swap on and off; in both cases, there is plenty of memory. In fact, the server in this post (the dedicated server) shows zero swap being used even though the uptime is:
Thanks Neo for your comments and taking out time to perform those tests. I will also try checking my system's performance by turning off swap.
But if you or anyone else can help me understanding my current system behaviour, it will help me understand Linux Memory management better.
As part of my python application, we are making some rest api calls to Azure and using various azure services. Please let me know what other information i can provide for commenting on this behaviour?
You can try tweaking this back a lower percentage (eg add/change vm.swappyness=10 in /etc/sysctl.conf and reboot) will cause more pages to be dropped from cache in preference to using swap.
In the link above, the price of server RAM is about $20 USD for 8 GB.
Now, I am not sure how much time you have spent working on this issue and how much you think your time is worth to you; but I can tell you that already I have spend time and effort on answering your question for you with a value of at least $40 USD, including replying to you and others, doing some quick Internet searches, and logging into a remote system and turning off swap (to illustrate a point for the "naysayers" who remain in love with swap from the yesterdays of expensive RAM nostalgia).
Let's say that the time you and others have spent on the problem (on your end) is also work about $40 USD. This means that $80 USD total human capital has already been spent on this problem.
$80 USD divided by $20 USD turns out to be roughly the same in terms of resources, as purchasing 32 GB of Samsung server RAM for your application. Of course, this is different in a VM service, but that is a different topic; but the same idea applies (just differently, and I tend NOT to use VMs and cloud services for security, and command and control reasons)
Let's say you try tweaking swappiness (which I have done many times before on Linux systems) and playing around with more Linux system parameters (like cache clearing, refresh rates, etc) to try to further optimize. You are spending more resources, all of which could be used to add RAM to you system, if you value your time (in the least). Only in an "academic" context, doing a study for the sake of learning (not operations) is all this optimization an efficient use of time and energy. You have indicated you are working on a real-world application and are not in academics.
This is my point.
RAM is very cheap (and it gets cheaper every year). If your system is swapping, that is an indicator, and a very strong indicator, that you need more RAM. Tweaking and tweaking your system parameters and encouraging others to engage in this kind of tweaking with you, is an inefficient use of human resources and capital. It's more "academic" in nature, versus practical, in operational terms.
In answering you in this reply, this reply is at least work 8 GB of server RAM in 2020; but let's be very conservative, and say all this thinking and typing and error checking (my in-browser spell checker seems to be in a plot against me, LOL) is only worth 4GB of RAM, either way; you are talking about your system which you say has 8GB of RAM installed. That RAM is worth about $20 USD, today (in hardware terms, not inflated, artificial VM terms).
If you want better performance, go out and spend $20 more USD and buy 8GB more and you will have 16GB of RAM.
This is a better use of your time and your energy. Your application will run faster and better. The world will be a better place.
The reason I have turned off swap on one server (yesterday), and am posting the results tonight in this thread, is to prove a simple point to others who are "swap nostalgic". This is not "a test" for me. I have been working with Linux since before Slackware 0.8, which was released before 1993. This translates to using Linux for around 27 years, continuously. I know the results before I turn off swap. There will be no change in system performance because the server and the applications I am running on that server are (1) not constrained by RAM and (2) there are no memory leaks. Linux runs well and I always configure all my Linux systems to have plenty of RAM and upgrade if RAM becomes an issue.
There are a lot of people who like to talk about the virtue of swap. This is like saying that when the glass of milk is full, I will go to the shelf and get another glass to store my milk until the main glass is less full, as I pour into the main glass. While this might be good if you are pouring milk very fast into your glass; most people would get a bigger glass.
The same in true for computer RAM.
RAM is so cheap these days (and has been for many years), you are better off getting more RAM versus spending a lot of time and energy trying to tweak a system to squeeze more performance from your system, which is asking you for more RAM because it is starting to swap.
You asked for ideas; so here is mine, put very simply. I would never run a Linux server application with 8GB of RAM and 2GB of swap because I do not want my systems to swap, ever, for any reason. I run 64 GB RAM on my desktop. I only use 8 GB VMs for "playgrounds", not production apps. If I built an app in a 8GB sandbox and it then became memory constrained, I would move it to a more robust server or upgrade it (if in VM, but I tend to move real-world apps to dedicated hardware, but that is just me because I like having more control over the OS and the upgrade cycles, etc). This is because time is worth more than money. We cannot gain back time. The arrow of time runs only in one direction (in this universe we are in now).
What I do is to always make sure my computer systems, all of them, remote servers, desktops all of them; have more than enough RAM to do the job because RAM is so cheap and I have better things to do with my time, which I consider valuable and a limited resource which I have little control.
Tonight I will post the graph of my MQTT / Node-RED server with swap off for 24 hours. You will see that there is no issue at all because there is enough RAM. If your system is memory constrained and you turn off swap, you will more-than-likely have some performance issues; but my point is not "turn off swap".. my point is "upgrade and add more RAM". For me "swap" is a "big red, flashing LED" telling me "please upgrade".
Now, in the VM case, RAM is a bit more expensive because when running in a VM and we are paying more for the RAM; but that is an entirely different discussion (that is a business issue, not a technology issue); but the point is still the same.
If you system is swapping, install more RAM.
You don't need to turn swap on and off. I was trying to make a point. If your system is memory bound (constrained) it will perform poorly even if you use swap; although it will perform better with swap than without swap (generally speaking) when you are really constrained by RAM. Like in my milk metaphor, it is OK to swap glasses when you are pouring milk to fast into a small glass; but it's even better (and smarter) to get a bigger glass.
The best choice, as I said in my first post, is to upgrade your RAM and move on to other tasks, unless (1) you are doing this as an academic exercise or (2) your time has very little value.
This is why I have replied this way. Time is an arrow which only moves in one directly. Use your time and energy wisely.
I can continue this for a few of my "busy servers", some very busy, but all have plenty of RAM, so all will perform just fine with swap turned off (since they do not really use any swap with it turned on).
Anyway, I don't want to beat this dead horse any longer...
Computer RAM is cheap.... if your system is swapping, buy more RAM.
You are not wrong that swap is used for hibernation in systems that are configured to use swap-based hibernation. But, I am confused by your reply, to be honest. This is not a question of "right and wrong". Not everyone hibernates, in fact, hibernation is a something some people choose to use and others choose not to hibernate. This is the same as any computer configuration. For example in Ubuntu, hibernation is not enabled by default; and as I recall, Ubuntu stopped supporting hibernation (in general) and only makes it available for users who want it.
Personally, I have zero systems (desktop or server) configured to hibernate. That is my choice (right?); others may like hibernation. I do not. The original poster did not ask a question about hibernation, to be honest; and it is very likely the OP does not use hibernation, since on Linux systems (at least the ones I am familiar with, like Ubuntu), hibernation is not enabled by default.
The title of this thread by the OP is "Underutilization of Linux memory" not "How to Enable Hibernation in Linux", isn't that correct?
As for me, I have a number of very busy Ubuntu servers in production and I do not enable hibernation on a single one of them.
My reply early discusses how RAM in 2020 is so cheap, that when you have system swapping for performance reasons (not for those people who configuration their systems to hibernate), that is a sign that you should invest in more RAM.
That is my choice and this is how I always advice others to admin their systems after over 25 years of hands on Linux work since Slackware 0.8. I like lots of deep, cheap RAM and never hibernate.
As a side, I just checked and Slackware 1.0 was released in 1993. I started deploying Linux for the United States Air Force at 27 Air Combat Command bases worldwide as Slackware 0.8, so that was at least 27 years ago. Perhaps over 27 years of using Linux in production has informed me "just a little" on how to configure systems.
https://en.wikipedia.org/wiki/Slackware
Everyone has a choice. You can build your system with plenty of "deep, cheap" RAM or not. Maybe you like having you systems move bits to disk and like to be "frugal" with RAM, LOL. However, I prefer RAM, it is CHEAP; and I never enable hibernation on any system. That is my choice and my advise to most people. If you system is swapping for performance reasons (this is not a hibernation discussion) it is a blinking light to deploy more RAM.
So, If anyone wants to use swap and you like having less RAM and you like your system thrashing pushing bits from RAM and disk and back, and that is your idea of "good computing", then go for it. I am not telling or advising anyone to use or not to use swap or to enable nor disable it. Everyone has a choice, right? I have posted what I do, and I have been working with Linux on a daily basic in production since Slackware 0.8, for over two decades (nearly three decades).
This means, quite simply, have told you that for me, I find RAM cheap and when a system is swapping, that is generally a very clear sign for me to increase RAM (this is not a discussion about hibernation, and never was, to be frank). However, on all Linux systems I have deployed for the past 10 years, I deploy them with a lot of RAM, so they never need to swap. None of my systems and servers swap, and none of my systems and server hibernate.
If anyone wants to swap and hibernate, then you should do so. Everyone has a choice when they configure their system. Many people advise based on "theory". I am telling you what I do based on 27 years of hands on production experience with Linux servers deployed. Most of my systems tend to have a lot of RAM. If I have a Linux system with a small amount of RAM deployed (like a VM slice), and it gets busy because the app grows and it starts to swap, I will deploy more RAM. I never configure hibernation.
As a final thought (hopefully), I can recall many people here at unix.com who were not happy when I gave Linux and Unix "equal status" at unix.com many years ago. There were a lot of "hard core" UNIX users (back then) who were really negative about Linux (and they did not use Linux back then either). I find it interesting that, now, so many of those same Unix-diehards of "yester-years" are now Linux experts today.
There is a big difference between "computing in theory" and "operational system deployment, administration and maintenance" (real-life operations).
A lot of people "talk to walk" but very few people actually "walk the talk". This is especially true today, in my view. We have a lot of "arm chair" experts in computer science and IT security (bloggers, writers, YT video makers), who have never really ran an operational servers in a busy production environment, fought hackers and DDOS attacks "in the real world" or been forced to make data center change in a day or two when a hosting business goes belly up.
There is a huge different between "theory" and "practice" and in server operations, the same is true, for sure! The world today is "pregnant" with arm-chair IT experts, bloggers, commenters, influencers, YT video makers, forum posters, who have never actually ran a production server application.