Safe way to clear all packages from yum cache?

I know that yum clean all doesn't necessarily clear all the package files. I know from "Maybe you want: rm -rf /var/cache/yum" that I might have to, good grief, bluntly clear all of /var/cache/yum. But I've just discovered that I very definitely do not want to clear all of /var/cache/yum, because evidently the record of which packages are installed is also kept down in there somewhere. Without certain files in /var/cache/yum, rpm -qa seems to print nothing.

So is there a simple rule for which other files underneath /var/cache/yum can be manually deleted, to save space by deleting package files that aren't needed (or can be easily redownloaded), but preserving the installation state of the system?

It's possible I'm completely mistaken on one or more of my assumptions here. Corrections welcome if so.

Update: I'm skipping some details here, but copying all of /var/cache/yum from the system that had it to the system that didn't did not fix rpm -qa (it's still printing nothing), so it's looking like one of my assumptions is flawed, and the system's installed state is kept somewhere under /var other than /var/cache/yum.

Update 2: I was wrong in one of my assumptions. As I eventually discovered, and as @shodanshok points out in an answer, the database of packages currently installed is down under /var/lib/rpm, not /var/cache/yum. So clearing all of /var/cache/yum should be safe.

A Red Hat box with no rpm packages installed at all is broken.

First, confirm this is a host that should have a real rpm database. Not some weird container or chroot or other stripped down thing. Determine if the system is supposed to get updates by yum update. And check yum history for if the most recent yum commands are recorded. yum would not get into this state by itself, it would refuse to remove protected packages.

Having a functional system, but nothing in rpm queries, implies someone did something very unsafe to rpm. Such as removed the rpm database without erasing packages. Will be a real mess to clean up. As discussed, removing yum caches will not do this.

Consider reinstalling a new host from the installer. Restore data. Advantages of this clean start include starting with a known good rpm system. And possibly you can take the opportunity to do major software upgrades.

Or, it may be possible to recover from a corrupted rpm database. Assuming any database still exists. Only do this if you cannot restore from backup, and understand which database backend your rpm is using. Back up the host, including rpm db, prior to attempting. The host would need to be of some value to you, if doing surgery on its package database is easier than a reinstall.