Varnish 6 bans_persisted_bytes ballooning, but memory usage is fine

Toon Spin

2/6/24, 9:27 AM

In a Varnish 6 server I'm responsible for, there are short bursts of very many bans. These are the result of an application I am not able to modify. This has caused RAM usage to very quickly balloon out of control, but I have been able to fix that by setting ban_cutoff to 100. That seems to be doing the trick very well. RAM usage is now very much under control.

However, bans_persisted_bytes still grows rapidly each burst and doesn't seem to be going down at all. As I'm writing this it's at about 29GiB but the total memory usage of the host, a Debian Bullseye machine, is at like 1.6 GiB.

Should I be worried about the size of bans_persisted_bytes? After all, it doesn't seem to be using RAM. But I have to wonder what the value means if not RAM usage.

I hope someone can shed some light on this. TIA!

2 + 0

varnish

memory-usage

Score:0

Server

Nils Goroll

3/8/24, 11:34 AM

While I am happy to hear that the ban cutoff (which I authored) works for you as intended, I do not think that the answer you gave yourself is correct.

bans_persisted_bytes is a gauge showing the ban list in bytes, see varnish-counters(7). bans_persisted_fragmentation is the number of bytes "wasted" for complete and duplicate bans until they can be removed.

So unless we have a bug, bans_persisted_bytes is the actual size of your ban list, and if you basically need the cutoff to keep it under control, it is to be expected to be huge.

+ 4

Toon Spin

3/9/24, 1:24 PM

Thank you for replying! Great to hear from the horse's mouth. However I think I must be confused as to what the ban list actually is. Because you're saying that the `ban_cleantail` function removes bans, and I don't understand the VTAILQ function calls but can see that counters are being decreased where applicable based on the bit flags, but nowhere in `ban_cleantail` is `bans_persisted_bytes` decreased. If bans are being removed, shouldn't they be struck from the ban list?

Nils Goroll

4/5/24, 8:41 PM

Bans are only ever removed from the tail of the ban list. Bans which are for whatever reason not required any more (e.g. duplicated or fully processed by the ban lurker), are marked as completed (`C` marker in `varnishadm ban.list`).

Nils Goroll

4/5/24, 8:51 PM

When bans are marked completed, their length is added to `bans_persisted_fragmentation` in `ban_mark_completed()`. In `ban_export()`, that counter is deducted from `bans_persisted_bytes`. This is how the `bans_persisted_bytes` shrinks.

Toon Spin

4/12/24, 8:48 AM

Looking at the `ban_export` function I see how the `bans_persisted_bytes` can shrink now ([GitHub link to the actual spot](https://github.com/varnishcache/varnish-cache/blob/6.6/bin/varnishd/cache/cache_ban.c#L340)). I will mark your answer as correct because I see that my answer is wrong now. Thank you for indulging me!

Score:0

Server

Toon Spin

2/13/24, 1:02 PM

My own below answer turns out to be incorrect. Please refer to the accepted answer instead.

To answer my own question I have perused the source code for Varnish 6.1. It seems to be the same in Varnish 7.2. There are two places where the value bans_persisted_bytes is changed, and in both cases a value is added that is guaranteed to be zero or greater.

This means that no matter how big the ban list is, the value bans_persisted_bytes will never decrease and should therefore be considered to be a cumulative value.

Given the fact that setting the cutoff value succeeds in curbing the memory footprint, I feel that I can safely conclude that the growth of the value bans_persisted_bytes is in fact nothing to worry about.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Varnish 6 bans_persisted_bytes ballooning, but memory usage is fine

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.