Package org.apache.kafka.clients.producer.internals

Class BufferPool

java.lang.Object
org.apache.kafka.clients.producer.internals.BufferPool
public class BufferPool extends Object
A pool of ByteBuffers kept under a given memory limit. This class is fairly specific to the needs of the producer. In particular it has the following properties:
  1. There is a special "poolable size" and buffers of this size are kept in a free list and recycled
  2. It is fair. That is all memory is given to the longest waiting thread until it has sufficient memory. This prevents starvation or deadlock when a thread asks for a large chunk of memory and needs to block until multiple buffers are deallocated.

  • Constructor Details

    • BufferPool

      public BufferPool(long memory, int poolableSize, org.apache.kafka.common.metrics.Metrics metrics, org.apache.kafka.common.utils.Time time, String metricGrpName)
      Create a new buffer pool
      Parameters:
      memory - The maximum amount of memory that this buffer pool can allocate
      poolableSize - The buffer size to cache in the free list rather than deallocating
      metrics - instance of Metrics
      time - time instance
      metricGrpName - logical group name for metrics

  • Method Details

    • allocate

      public ByteBuffer allocate(int size, long maxTimeToBlockMs) throws InterruptedException
      Allocate a buffer of the given size. This method blocks if there is not enough memory and the buffer pool is configured with blocking mode.
      Parameters:
      size - The buffer size to allocate in bytes
      maxTimeToBlockMs - The maximum time in milliseconds to block for buffer memory to be available
      Returns:
      The buffer
      Throws:
      InterruptedException - If the thread is interrupted while blocked
      IllegalArgumentException - if size is larger than the total memory controlled by the pool (and hence we would block forever)

    • recordWaitTime

      protected void recordWaitTime(long timeNs)

    • allocateByteBuffer

      protected ByteBuffer allocateByteBuffer(int size)

    • deallocate

      public void deallocate(ByteBuffer buffer, int size)
      Return buffers to the pool. If they are of the poolable size add them to the free list, otherwise just mark the memory as free.
      Parameters:
      buffer - The buffer to return
      size - The size of the buffer to mark as deallocated, note that this may be smaller than buffer.capacity since the buffer may re-allocate itself during in-place compression

    • deallocate

      public void deallocate(ByteBuffer buffer)

    • availableMemory

      public long availableMemory()
      the total free memory both unallocated and in the free list

    • freeSize

      protected int freeSize()

    • unallocatedMemory

      public long unallocatedMemory()
      Get the unallocated memory (not in the free list or in use)

    • queued

      public int queued()
      The number of threads blocked waiting on memory

    • poolableSize

      public int poolableSize()
      The buffer size that will be retained in the free list after use

    • totalMemory

      public long totalMemory()
      The total memory managed by this pool

    • close

      public void close()
      Closes the buffer pool. Memory will be prevented from being allocated, but may be deallocated. All allocations awaiting available memory will be notified to abort.