stub: ensure BlockingClientCall tasks run after cancellation

[benjaminp]

After canceling a BlockingClientCall, the caller may not interact with it further. That means the call executor, a ThreadSafeThreadlessExecutor, will not execute any more tasks. There may still be tasks submitted to the call executor, though, until the underlying call completes. Some of these tasks (e.g., server messages available) may leak native resources unless executed. So, we convert the executor to a "direct" executor during cancellation in order to ensure all call tasks run.

Fixes #12355.

[ejona86]

I expect we really want to drain the executor until the Listener.onClose() is delivered.

[benjaminp]

The trouble is I don't think .cancel should be blocking.

[benjaminp]

Another option might be to dump post-cancellation tasks on whatever executor would have been used for the call if we didn't force ThreadsafeThreadlessExecutor in.

[benjaminp]

I expect we really want to drain the executor until the Listener.onClose() is delivered.

So you're proposal would be to make BlockingClientCall.cancel look something like this?

  public void cancel(String message, Throwable cause) {
    writeClosed = true;
    call.cancel(message, cause);
    boolean interrupted = Thread.interrupted();
    while (true) {
      try {
        executor.waitAndDrain(x -> x.closeState.get() != null, this);
        break;
      } catch (InterruptedException e) {
        interrupted = true;
      }
    }
    if (interrupted) {
      Thread.currentThread().interrupt();
    }
  }

[apm1467]

We just hit this issue in production.

With a BlockingV2Stub bidirectional stream, cancelling the stream does not release its native memory until the caller manually reads all remaining messages from the stream. This is unexpected and hard to debug. The phenomenon in production is a Netty direct memory leak.

Please consider fixing this issue, or documenting that callers must drain the stream after cancellation. The same drain-after-cancel requirement is documented for v1 blocking stubs, but not for v2.