How can I remove some measure units from the Physical Fields widget?

What the widget returns for the Measurement field is the rendering array built with the following code. (See MeasurementDefaultWidget::formElement().)

$element = [
  '#type' => 'physical_measurement',
  '#measurement_type' => $this->fieldDefinition->getSetting('measurement_type'),
  '#allow_unit_change' => $this->getSetting('allow_unit_change'),
  '#default_value' => $items[$delta]->getValue(),
] + $element;

if (!$this->getSetting('allow_unit_change')) {
   $element['#available_units'] = [$default_unit];
}

There isn't any option list, because that is built from the physical_measurement form element class. (See Measurement::processElement().)

$element['unit'] = [
  '#type' => 'select',
  '#options' => $units,
  '#default_value' => $default_value ? $default_value['unit'] : $unit_class::getBaseUnit(),
  '#title_display' => 'invisible',
  '#field_suffix' => '',
];

In hook_field_widget_form_alter(), the list of the allowed units are contained in $element['#available_units'], which could not be set when all the units for that measure are allowed.
The following code can be used to get all the units a physical_measurement field uses (which are restricted from its measurement type).

use Drupal\physical\MeasurementType;

$unit_class = MeasurementType::getUnitClass($element['#measurement_type']);
$units = array_keys($unit_class::getLabels());

When $element['#allow_unit_change'] is FALSE, which is used to mean it's not possible to change the unit associated with the field, $element['#available_units'] contains the only allowed unit. In that case, hook_field_widget_form_alter() should not alter $element['#available_units'].

As side note, since the question is asking about removing measure units, the module code doesn't allow to add more units because the classes to handle them have code similar to the following one, used by the VolumeUnit class.

/**
 * {@inheritdoc}
 */
public static function getBaseFactor($unit) {
  self::assertExists($unit);
  $factors = [
    self::MILLILITER => '0.000001',
    self::CENTILITER => '0.00001',
    self::DECILITER => '0.0001',
    self::LITER => '0.001',
    self::CUBIC_MILLIMETER => '0.000000001',
    self::CUBIC_CENTIMETER => '0.000001',
    self::CUBIC_METER => '1',
    self::CUBIC_INCH => '0.00001638706',
    self::CUBIC_FOOT => '0.02831685',
    self::FLUID_OUNCE => '0.00002957353',
    self::US_GALLON => '0.0037854118',
  ];

  return $factors[$unit];
}

/**
 * {@inheritdoc}
 */
public static function assertExists($unit) {
  $allowed_units = [
    self::MILLILITER, self::CENTILITER, self::DECILITER, self::LITER,
    self::CUBIC_MILLIMETER, self::CUBIC_CENTIMETER, self::CUBIC_METER,
    self::CUBIC_INCH, self::CUBIC_FOOT, self::FLUID_OUNCE, self::US_GALLON,
  ];
  if (!in_array($unit, $allowed_units)) {
    throw new \InvalidArgumentException(sprintf('Invalid volume unit "%s" provided.', $unit));
  }
}