Custom reference field saving empty values

I've created a custom field based on entity reference field but the target_id it saves comes from an external system.

I have the field type, field widget and autocomplete element defined and working and the correct values are being saved to the associated database table for valid values.

The bit I'm stuck on is that if the field is left blank it still saves a record to the database but with an empty target_id. How should I stop it doing that?

Here's my implementation of isEmpty(), which I cribbed from the EntityReference field definition. I'm guessing this is the culprit but I'm not sure how to fix it:

  /**
   * {@inheritdoc}
   */
  public function isEmpty() {
    $target_id = isset($this->target_id) ? $this->target_id : NULL;
    if ($target_id !== NULL) {
      return FALSE;
    }
    return TRUE;
  }