How to create configurable plugins under a plugin manager?

I have a custom plugin type and annotation. The purpose of it is so people can develop their own clients for an API integration, and tell the system which plugin to use to perform these actions. I am able to load all defined plugins and add their sub forms on the main settings configuration form page I have implemented.

When the form submits, I can see the values go through to the appropriate plugin, but the values are not persisted nor in the database. I'm having trouble figuring out how to get this to save.

    $clients = $form_state->getValue('connections');

    if (!empty($clients)) {
      foreach ($clients as $key => $client) {
        $instance = $this->clientPluginManager->createInstance($key, []);
        $client_form_state = SubformState::createForSubform($form['connections'][$key], $form, $form_state);
        $instance->submitConfigurationForm($form['connections'][$key], $client_form_state);
      }
    }

    parent::submitForm($form, $form_state);

In the plugin:

  /**
   * {@inheritdoc}
   */
  public function submitConfigurationForm(array &$form, FormStateInterface $form_state) {
    $this->setConfiguration($form_state->getValues());
  }

  /**
   * {@inheritdoc}
   */
  public function setConfiguration(array $configuration) {
    $this->configuration = NestedArray::mergeDeep($this->defaultConfiguration(), $configuration);
  }

In my module (from looking around at other modules):

mymodule.default_client_configuration:
  type: config_entity
  label: 'Default client configuration'
  mapping:
    api_key:
      type: string
      label: 'The client API key.'
    hostname:
      type: string
      label: 'The hostname or base URI of the API.'

plugin.plugin_configuration.mymodule_client.*:
  type: mymodule.default_client_configuration

I am not entirely certain how the schema should be set. Do I also have to create a custom config entity class definition too? How does configuration make it from the form into config state?

To clarify the comment, it's true that instantiated plugins don't know from where they get the configuration, but you can add a method to the plugin manager to determine plugin id and configuration to instantiate with $this->createInstance().

To answer the question

How to create configurable plugins under a plugin manager?

with a minimal code example:

PluginManager::getInstance

  public function getInstance(array $options) {
    $settings = $this->configFactory->get('mymodule.settings');
    $plugin_id = $settings->get('default_client.plugin_id');
    $configuration = $settings->get('default_client.configuration');
    if ($plugin_id) {
      return $this->createInstance($plugin_id, $configuration);
    }
  }

A plugin by the way is not a form class. Even if it looks like this with build and submit methods, those are not fully working form functions. They are called by the corresponding methods of the parent form. The form instance is not the same as the plugin instance and the configuration is not saved to the database. You can do this in the parent form and by setting #tree you can usually submit the form structure as it is.