Add third party settings to a basefield definition

Is there a way to add third party settings when defining a basefield for a custom Content Entity?

I am creating a new Entity type and want to keep the fields defined in code vs config as much as I can. I have an address field and a geofield field defined. I need to add third party settings to the geofield to allow it to geolocate from the Address field.

I was able to get the settings from the yaml config which I translated to a php array and have tried applying the settings to the field via the ->setSettings() method, however that doesn't seem to work.

Anyone got any ideas?

$fields['location'] = BaseFieldDefinition::create('geofield')
      ->setRevisionable(TRUE)
      ->setTranslatable(TRUE)
      ->setLabel(t('Location Coordinates'))
      ->setRequired(FALSE)
      ->setCardinality(1)
      ->setDisplayConfigurable('form', TRUE)
      ->setDisplayOptions('form', [
        'type' => 'geofield_latlon',
        'weight' => 4,
        'settings' => [
          'html5_geolocation' => FALSE,
        ],
      ])
      ->setSettings(
        [
          'third_party_settings' => [
            'geocoder_field' => [
              'method' => 'geocode',
              'weight' => 0,
              'field' => 'address',
              'skip_not_empty_value' => false,
              'disabled' => false,
              'hidden' => false,
              'providers' => [
                0 => 'googlemaps',
              ],
              'dumper' => 'wkt',
              'delta_handling' => 'default',
              'failure' => [
                'handling' => 'preserve',
                'status_message' => true,
                'log' => true,
              ],
              'reverse_geocode' => [
                'field' => '',
              ],
            ],
          ],
        ]
      )
      ->setDisplayConfigurable('view', TRUE)
      ->setDisplayOptions('view', [
        'type' => 'geofield_latlon',
        'label' => 'hidden',
        'weight' => 10,
      ]);