Join Log in
Score:1
avatar Server

How should i debug a Microsoft CIM that fails because MSFT_NetRoute be "not found"?

co flag
jayunit100

In general, how should one debug a failure related to CmdletizationQuery_NotFound messages, and why would the MSFT_NetRoute based queries register such an error?

Details

Specifically, my Kubernetes CNI provider (antrea) is posting an error message related to MSFT_NetRoute when it starts up, because it can't succesfully run a powershell query necessary for bootstrapping itself. The error message has a suspicious string, which:

  • Might be related to powershell version (I disconfirmed later in this post, but its a reasonable hypothesis)
  • Im also considering that maybe this error came from external things i havent installed...
  • Finally, without fully understanding the nature of cmdletizationQuery error messages, its reasonable to assume this is related to a lower level networking setup issue.
   + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetRoute

The above snippet of this message, highlights the Get-NetRoute Related error which occurs below, which is puzzling because it seems to be indicating that Get-NetRoute is not a valid query to run. But https://docs.microsoft.com/en-us/powershell/module/nettcpip/get-netroute?view=windowsserver2019-ps seems to imply thet Get-NetRoute is available by default in powershell.

The error message

ROOT/StandardCimv2/MSFT_NetRoute class on the  CIM server: SELECT * FROM MSFT_NetRoute  WHERE ((DestinationPrefix LIKE
'0.0.0.0/0')) AND ((InterfaceIndex = 26)). Verify query parameters and retry.
At line:1 char:3
+ $(Get-NetRoute -InterfaceIndex 26 -DestinationPrefix 0.0.0.0/0 ).Next ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (MSFT_NetRoute:String) [Get-NetRoute], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetRoute

Powershell version

The powershell version on this machine is seemlingy pretty up to date... (5.1 i think is normal on windows server 2019)...

Name                           Value
----                           -----
PSVersion                      5.1.17763.1852
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17763.1852
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Note: I've tested this in similar versions of windows server 2019 with a similar 5.1.x powershell version (5.1.17763.1490) and not seen this failure. So,

Questions

  • is there something i need to do to make Get-NetRoute work properly on my machines?
  • is this related to something else (i.e. the way network interfaces are configured on the individual VM) ? Note that this is a VirtualBox VM, so if there was a issue with bridged or NAT networking, I woudln't be totally shocked.
252
1 + 6
JosefZ avatar
th flag
JosefZ
Does `Get-NetRoute` run at all? And does `Get-NetRoute -DestinationPrefix 0.0.0.0/0` show `26` in the `ifIndex` column?
0
Reply
cn flag
Greg Askew
It's in the error message. MSFT_NetRoute is not present. You can validate this by performing a WMI query against the namespace/path.
0
Reply
co flag
jayunit100
yeah , i played more with this and confirmed: Get-NetRoute works...
0
Reply
co flag
jayunit100
i suspect its simply as mentioned that MSFT_NETROUTE is missing , but not sure, that specific MSFT_NETROUTE object which has -InterfaceIndex 26 -DestinationPrefix 0.0.0.0/0. is not existing... i suspect this is because the interface address 26 doesnt have a 0.0.0.0/0 destination. My CNI provider expects this .. so maybe the interface 26 that is being queried isnt the right one
0
Reply
cn flag
Greg Askew
This is easy to confirm, no need to suspect anything. The WMI class is either present or not.
0
Reply
co flag
jayunit100
Yes, i think i can self answer this question tomorrow once i clear my thoughts and collect some data on other systems...
0
Reply
Score:0
avatar Server
co flag
jayunit100

So, in general, if you get a CmdletizationQuery_NotFound error, may not always be accompanied by info about the failing field if there are many fields in the query...

➜  sig-windows-dev-tools git:(antrea-node-ip-hardcoding) ✗ vagrant winrm winw1 --shell=powershell --command="Get-NetRoute -InterfaceIndex 123 "
No MSFT_NetRoute objects found with property 'InterfaceIndex' equal to '123'.  Verify the value of the property and retry.
At line:1 char:1
+ Get-NetRoute -InterfaceIndex 123
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (123:UInt32) [Get-NetRoute], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound_InterfaceIndex,Get-NetRoute

It is because the specific CIMs query failed, and thus the object you were looking for wasnt found.

However, it appears that sometimes a CIMS query will fail to return a result without giving you the specific part of the query which failed (i.e. CmdletizationQuery_NotFound_InterfaceIndex above was nice and easy to read, but the same query below when we add the DestinationPrefix search field, gives a more cryptic error message)...

➜  sig-windows-dev-tools git:(antrea-node-ip-hardcoding) ✗ vagrant winrm winw1 --shell=powershell --command="Get-NetRoute -InterfaceIndex 7 -DestinationPrefix 0.0.0.0/1"
No matching MSFT_NetRoute objects found by CIM query for instances of the ROOT/StandardCimv2/MSFT_NetRoute class on the  CIM server: SELECT * FROM MSFT_NetRoute  WHERE ((DestinationPrefix LIKE '0.0.0.0/1')) AND ((InterfaceIndex = 7)). Verify query parameters and retry.
At line:1 char:1
+ Get-NetRoute -InterfaceIndex 7 -DestinationPrefix 0.0.0.0/1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (MSFT_NetRoute:String) [Get-NetRoute], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetRoute

In this specific case:

  • there was no existing IP interface with id = 26 which ALSO had the 0.0.0.0/0 destination prefix

In more general cases, if you get a similar error, it could be caused by the fact that your just crafting a CIMS query that is not able to return any data.

Now, since the original question was related to a Kubernetes CNI provider, I'll address that part:

In Kubernetes on Windows

CNI providers such as antrea need this information when coming online, and the general thing to make sure of is the fact that your windows kubelet is setting its IP address correctly (i.e. via the node-ip field on startup).

In my case, I found that after setting this, this query was generated correctly, and it started looking at the right interface for this value (i.e. the one that corresponded to my Node's internal IP address).

There is the broader question of how, in general, DestinationPrefixes should be setup in Kubernetes windows VMs, but that is outside the scope of my original question, but in general, if you have setup your network correctly such that:

  • your node-ip as shown by kubectl get nodes -o wide is the correct one that you want for your windows kubelet and
  • that node-ip is associated w/ an interface which has a destination prefix IP address = 0.0.0.0/0

Then specifically the antrea CNI provider will be able to accurately determine the correct nextHop for its gateway, which ultimately is used to configure the OVS routing rules on a node for your pod networks.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.