inconsistent CUDA toolkit path /usr vs /usr/lib, tensorflow build from source

When I use the default tensorflow build I get Illegal instruction core dumped error.

Based on my research https://stackoverflow.com/questions/60858317/how-to-fix-illegal-instruction-core-dumped & https://github.com/tensorflow/tensorflow/issues/17411 I need to build tensorflow from source.

I started to build tensorflow from source based on: https://www.tensorflow.org/install/source

When I would like to run ./configure I was getting an error:

Could not find any cuda.h matching version '10' in any subdirectory:
    ''
    'include'
    'include/cuda'
    'include/*-linux-gnu'
    'extras/CUPTI/include'
    'include/cuda/CUPTI'
    'local/cuda/extras/CUPTI/include'
of:
    '/lib'
    '/lib/i386-linux-gnu'
    '/lib/x86_64-linux-gnu'
    '/usr'
    '/usr/lib/x86_64-linux-gnu/libfakeroot'

Therefore I installed cuda toolkit and cudnn based on this post: https://towardsdatascience.com/installing-tensorflow-gpu-in-ubuntu-20-04-4ee3ca4cb75d

now I have:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

and Cudddn:

cudnn-10.1-linux-x64-v7.6.5.32 

my cuda is in:

whereis cuda
cuda: /usr/lib/cuda /usr/include/cuda.h

and nvidia-smi returns:

Now I would expect that I'd be able to run ./configure I get the following messages:

WARNING: current bazel installation is not a release version.
Make sure you are running at least bazel 3.7.2
Please specify the location of python. [Default is /usr/bin/python3]: 


 Found possible Python library paths:
   /usr/lib/python3/dist-packages
    /usr/local/lib/python3.8/dist-packages
   Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

  Do you wish to build TensorFlow with ROCm support? [y/N]: 
  No ROCm support will be enabled for TensorFlow.

  Do you wish to build TensorFlow with CUDA support? [y/N]: y
  CUDA support will be enabled for TensorFlow.

  Do you wish to build TensorFlow with TensorRT support? [y/N]: 
  No TensorRT support will be enabled for TensorFlow.

  Inconsistent CUDA toolkit path: /usr vs /usr/lib
  Asking for detailed CUDA configuration...


 Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 10]: 


 Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7]: 


 Please specify the locally installed NCCL version you want to use. [Leave empty to use 
 http://github.com/nvidia/nccl]: 


 Please specify the comma-separated list of base paths to look for CUDA libraries and headers. 
 [Leave empty to use the default]: 

but can not pass this error:

Inconsistent CUDA toolkit path: /usr vs /usr/lib Asking for detailed CUDA configuration...

What is wrong? how do I resolve this problem?

$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
$ sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
$ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub
$ sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"
$ sudo apt-get update
$ sudo apt-get -y install cuda

Source: https://developer.nvidia.com/cuda-11.3.0-download-archive.

In my case these instructions installed CUDA under /usr/local/ and not under /usr/ directly. This way the configuration process was able to find the CUDA installation.

I hope this helps someone.