efivarfs partition saturation on a dedicated Supermicro X10SRi-F

Bare Metal and VPS
1

Hi,

I'm encountering a problem on a baremetal equipped with a Supermicro X10SRi-F motherboard.

At every reboot (and there are a lot right now :frowning: ) the efivarfs partition fills up inexorably.
I hadn't noticed it until I received a monitoring alert because it was becoming a bit critical:

efivarfs 512K 473K 35K 94% /sys/firmware/efi/efivars

From what I understand, we need to act at the BIOS level, but I must admit I'm not very comfortable configuring a BIOS on a remote machine.
I found posts that say you need to shut down the machine and reboot, but I don't think that's possible in this case?
Anyone have an opinion? @le_sbraz?

Supermicro X10SRi-F rev 1.01B SN: NM15CS006083
American Megatrends 3.4.V1

Thanks!

2

Hello @TTY

The solution almost always involves readjusting the variable management system from the Supermicro BIOS itself. It can be done without manually touching the hardware, although it does require rebooting the server and accessing the firmware settings.

https://www.manualshelf.com/manual/supermicro/x10sri-f/user-s-manual-1-0b.html#3

The Supermicro X10SRi-F board manual indicates that the options you need are located within the BIOS configuration. Specifically, the board uses an AMI BIOS, and the relevant option for managing the saturation of the system event log is usually found in the Event Logs section.

Some motherboards allow you to perform this management system reset from within the operating system using IPMI tools, which would avoid a full reboot, but I’m not sure if that’s possible in your case...

Let us know how it goes, cheers.
Sergio Turpin

3

Quick update,

I tried a halt from the SOL console and then a reboot via the manager.
Problem not solved and the partition grew by 12 KB more :frowning:

I opened a ticket with support CS16630315 to see what they tell me.

I'm continuing my research, I found fwupd which can do some things https://www.linuxtricks.fr/wiki/fwupd-mettre-a-jour-les-firmwares-et-uefi-depuis-linux but it's not suitable.
Now I'm on https://www.supermicro.com/en/solutions/management-software/supermicro-update-manager to see if I can troubleshoot...

I have another server affected since a reboot this morning with a Supermicro X10SDV-4C-TLN2F card, AMI 2.3.V1

4

Hi @TTY

The fwupd tool is a good option for newer systems, but on Supermicro X10 boards the support via LVFS is limited, so it's not the best route :smirking_face:

My advice is to wait for support's response. They can run SUM remotely via the BMC and update the firmware without you having to do anything else :crossed_fingers:

Keep me informed!

5

Hello @TTY ,

Check if in /sys/firmware/efi/efivars/ you see anything duplicating or getting larger.

6

Well, the thing is that what I see when listing doesn't come close at all to the size reported by df -h:

efivarfs 512K 485K 23K 96% /sys/firmware/efi/efivars

root@kvm0:/sys/firmware/efi/efivars# ll /sys/firmware/efi/efivars
total 0
drwxr-xr-x 2 root root 0 Sep 2 10:25 .
drwxr-xr-x 5 root root 0 Sep 2 10:48 ..
-rw-r--r-- 1 root root 88 Sep 2 10:25 Boot0005-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 310 Sep 2 10:25 Boot0006-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 322 Sep 2 10:25 Boot0007-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 310 Sep 2 10:25 Boot0008-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 322 Sep 2 10:25 Boot0009-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 6 Sep 2 10:25 BootCurrent-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 8 Sep 2 10:25 BootOptionSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 14 Sep 2 10:25 BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 14 Sep 2 10:25 BootOrderBeforeIpmiBoot-842680f2-1a9c-48e6-a433-be9acb0dd438
-rw-r--r-- 1 root root 25 Sep 2 10:25 BstLogVar-01239999-fc0e-4b6e-9e79-d54d5db6cd20
-rw-r--r-- 1 root root 141 Sep 2 10:25 ConIn-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 141 Sep 2 10:25 ConInDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 143 Sep 2 10:25 ConOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 143 Sep 2 10:25 ConOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 14 Sep 2 10:25 DefaultBootOrder-45cf35f6-0d6e-4d04-856a-0370a5b16f53
-rw-r--r-- 1 root root 36 Sep 2 10:25 DefaultUefiDevOrder-0c923ca9-df73-4ac8-b6d2-98ddc30d99fc
-rw-r--r-- 1 root root 1.3K Sep 2 10:25 DmiArray-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 15 Sep 2 10:25 DmiVar0200020400-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 13 Sep 2 10:25 DmiVar0200020500-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 10 Sep 2 10:25 DmiVar0200020600-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 17 Sep 2 10:25 DmiVar0200020700-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 143 Sep 2 10:25 ErrOut-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 143 Sep 2 10:25 ErrOutDev-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 8 Sep 2 10:25 FPDT_Volatile-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root 516 Sep 2 10:25 FixedBoot-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9
-rw-r--r-- 1 root root 19 Sep 2 10:25 FixedBootGroup-ec87d643-eba4-4bb5-a1e5-3f3e36b20da9
-rw-r--r-- 1 root root 12 Sep 2 10:25 HiiDB-1b838190-4625-4ead-abc9-cd5e6af18fe0
-rw-r--r-- 1 root root 1.6K Sep 2 10:25 KEKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 78 Sep 2 10:25 LoaderDevicePartUUID-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f
-rw-r--r-- 1 root root 16 Sep 2 10:25 LoaderEntryRebootReason-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f
-rw-r--r-- 1 root root 44 Sep 2 10:25 LoaderInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f
-rw-r--r-- 1 root root 6 Sep 2 10:25 MaximumTableSize-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 8 Sep 2 10:25 MonotonicCounter-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root 28 Sep 2 10:25 OA3MSDMvariable-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root 14 Sep 2 10:25 OldBootOrder-01368881-c4ad-4b1d-b631-d57a8ec8db6b
-rw-r--r-- 1 root root 36 Sep 2 10:25 OriUefiDevOrder-0c923ca9-df73-4ac8-b6d2-98ddc30d99fc
-rw-r--r-- 1 root root 12 Sep 2 10:25 OsIndicationsSupported-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 979 Sep 2 10:25 PKDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 10 Sep 2 10:25 PlatformLang-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 10 Sep 2 10:25 PlatformLangCodes-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 88 Sep 2 10:25 PreviousBoot-36d08fa7-cf0b-42f5-8f14-68df73ed3740
-rw-r--r-- 1 root root 8 Sep 2 10:25 RTC-378d7b65-8da9-4773-b6e4-a47826a833e1
-rw-r--r-- 1 root root 8 Sep 2 10:25 RTC-378d7b65-8da9-4773-b6e4-a47826a833e2
-rw-r--r-- 1 root root 12 Sep 2 10:25 S3SS-4bafc2b4-02dc-4104-b236-d6f1b98d9e84
-rw-r--r-- 1 root root 5 Sep 2 10:25 SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 5 Sep 2 10:25 SetupMode-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 132 Sep 2 10:25 SignatureSupport-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 12 Sep 2 10:25 SmbiosEntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 12 Sep 2 10:25 SmbiosScratchBuffer-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 12 Sep 2 10:25 SmbiosV3EntryPointTable-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 6 Sep 2 10:25 Timeout-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 36 Sep 2 10:25 UefiDevOrder-0c923ca9-df73-4ac8-b6d2-98ddc30d99fc
-rw-r--r-- 1 root root 5 Sep 2 10:25 VendorKeys-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 68 Sep 2 10:25 WriteOnceStatus-4b3082a3-80c6-4d7e-9cd0-583917265df1
-rw-r--r-- 1 root root 5.1K Sep 2 10:25 dbDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c
-rw-r--r-- 1 root root 11K Sep 2 10:25 dbxDefault-8be4df61-93ca-11d2-aa0d-00e098032b8c

about twenty kilobytes.

Here's what Claude says about this:

*df does not sum the files. On efivarfs, the kernel has no idea of the store size: efivarfs_statfs() calls the UEFI runtime service QueryVariableInfo() and copies what the firmware returns. The 485 KB are therefore a declaration from the BIOS about the state of its SPI chip, not a calculation on the inodes. No relation to what ls shows.

What we will never see from Linux. Two categories:*

  • NV+BS variables without RT: the UEFI spec forbids their access after ExitBootServices, and GetNextVariableName does not enumerate them at runtime. This is where the AMI BIOS settings live (Setup, AMITSESetup, PchSetup, SaSetup…), each several KB. Their total absence from the list while every AMI board has them is indirect proof. They are very likely the ones consuming the 12 KB per boot—a rewrite on boot, an old record marked dead;
  • uncompacted dead records, already documented.

On the Linux side, nothing writes: efi_pstore is loaded but disabled at boot (pstore: backend 'erst' already in use: ignoring 'efi_pstore'). The consumption is 100 % firmware.

I'm not sure the server doesn't boot at all because it's ultimately a PXE boot.
I'll wait for any possible response on the ticket before doing reboots until saturation (I've already migrated all VMs from this hypervisor) to test this.

7

It may be that the space will eventually be cleared by the BIOS and that the occupied space is essentially "trash" that it will clean up when it actually becomes critical (probably not long from now).

My idea is that after a few reboots or a "save" in the BIOS it will eventually resolve the issue, which isn’t really a problem, perhaps.

8

Hello,
That doesn't ring a bell. For your X10SDV-4C-TLN2F on BIOS 2.3.V1, I think you can ask support for an update. We have a BIOS 2.6.
For the X10SRi-F, you’re already on the latest version with 3.4.V1.

Does the partition eventually fill up to 100 %? Does it generate errors in the logs?

9

I rebooted twice to saturate the partition and it went back to 6%.

Apparently the Munin monitoring worried me for nothing...
Sorry for the inconvenience :frowning: and thanks to those who took their time (precious) to help me.

Edit: I marked it resolved on @le_sbraz's post but your intuition was also correct @LibreMaster .

10

My colleagues ran tests and came to the same conclusion. After several reboots, usage goes up and then eventually drops again.
I advise you to simply ignore this mountpoint in your monitor :smiley:

11

Sorry again for making you waste your time.

12

No problem. Now people who are looking for the error will end up on your thread :slight_smile:

I'm noting here for reference the test ticket we did internally: PUBM-55381.