Finding all accounts without a domain in proxyAddresses (or Where-Object FilterScript on array that doesn't contain entry by wildcard)

I'm trying to get a list of Active Directory accounts that don't have a an address with a given domain name in their proxyAddresses. I know that to find ones that do, I can do:

Get-AdUser -Filter 'proxyAddresses -like ''smtp:*@domain.com'''

However, the reverse doesn't work, because if they have even one other entry in proxyAddresses (and all our accounts do, like X500 addresses and onmicrosoft addresses), it'll match that and still return the object. I also know I could do something like:

Get-AdUser -Filter * -Properties proxyAddresses |
  ForEach-Object -Begin { $filteredList = @() } -Process {
    $notfound = $true
    $_.proxyAddresses | ForEach-Object -Process {
      if ($_ -like 'smtp:*@domain.com') {
        $notfound = $false
      }
    }
    if ($notfound) { $filteredList += $_ }
  }

Is there a way I can do this in a Where-Object FilterScript instead? i.e., filter server-side rather than client-side? I've tried this to see if I could reverse the true / false from "-like", but it didn't seem to work, I still got all accounts:

Get-AdUser -Filter * -Properties proxyAddresses | Where-Object -FilterScript { if ($_.proxyAddresses -like 'smtp:*@domain.com') { $false } else { $true } }

Any ideas?