Package org.apache.kafka.clients.consumer.internals

Class Fetcher<K,V>

java.lang.Object

org.apache.kafka.clients.consumer.internals.AbstractFetch

org.apache.kafka.clients.consumer.internals.Fetcher<K,V>

All Implemented Interfaces:

Closeable, AutoCloseable

public class Fetcher<K,V>
extends AbstractFetch

This class manages the fetching process with the brokers.

Thread-safety: Requests and responses of Fetcher may be processed by different threads since heartbeat thread may process responses. Other operations are single-threaded and invoked only from the thread polling the consumer.

• If a response handler accesses any shared state of the Fetcher (e.g. FetchSessionHandler), all access to that state must be synchronized on the Fetcher instance.
• If a response handler accesses any shared state of the coordinator (e.g. SubscriptionState), it is assumed that all access to that state is synchronized on the coordinator instance by the caller.
• At most one request is pending for each node at any time. Nodes with pending requests are tracked and updated after processing the response. This ensures that any state (e.g. epoch) updated while processing responses on one thread are visible while creating the subsequent request on a different thread.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.kafka.clients.consumer.internals.AbstractFetch

AbstractFetch.ResponseHandler<T>

Field Summary

Fields inherited from class org.apache.kafka.clients.consumer.internals.AbstractFetch

decompressionBufferSupplier, fetchBuffer, fetchConfig, logContext, metadata, metricsManager, nodesWithPendingFetchRequests, subscriptions, time

Constructor Summary

Constructors

Constructor	Description
Fetcher(org.apache.kafka.common.utils.LogContext logContext, ConsumerNetworkClient client, ConsumerMetadata metadata, SubscriptionState subscriptions, FetchConfig fetchConfig, Deserializers<K,V> deserializers, FetchMetricsManager metricsManager, org.apache.kafka.common.utils.Time time, org.apache.kafka.clients.ApiVersions apiVersions)

Method Summary

Modifier and Type	Method	Description
void	clearBufferedDataForUnassignedPartitions(Collection<org.apache.kafka.common.TopicPartition> assignedPartitions)
protected void	closeInternal(org.apache.kafka.common.utils.Timer timer)	This method is called by AbstractFetch.close(Timer) which is guarded by the IdempotentCloser) such as to only be executed once the first time that any of the AbstractFetch.close() methods are called.
Fetch<K,V>	collectFetch()
protected boolean	isUnavailable(org.apache.kafka.common.Node node)	Check if the node is disconnected and unavailable for immediate reconnection (i.e.
protected void	maybeCloseFetchSessions(org.apache.kafka.common.utils.Timer timer)
protected void	maybeThrowAuthFailure(org.apache.kafka.common.Node node)	Checks for an authentication error on a given node and throws the exception if it exists.
int	sendFetches()	Set up a fetch request for any node that we have assigned partitions for which doesn't already have an in-flight fetch or pending fetch data.

Methods inherited from class org.apache.kafka.clients.consumer.internals.AbstractFetch

close, close, createFetchRequest, handleCloseFetchSessionFailure, handleCloseFetchSessionSuccess, handleFetchFailure, handleFetchSuccess, hasAvailableFetches, prepareCloseFetchSessionRequests, prepareFetchRequests, sessionHandler

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Fetcher

public Fetcher(org.apache.kafka.common.utils.LogContext logContext,
 ConsumerNetworkClient client,
 ConsumerMetadata metadata,
 SubscriptionState subscriptions,
 FetchConfig fetchConfig,
 Deserializers<K,V> deserializers,
 FetchMetricsManager metricsManager,
 org.apache.kafka.common.utils.Time time,
 org.apache.kafka.clients.ApiVersions apiVersions)

Method Details

isUnavailable

protected boolean isUnavailable(org.apache.kafka.common.Node node)

Description copied from class: AbstractFetch

Check if the node is disconnected and unavailable for immediate reconnection (i.e. if it is in reconnect backoff window following the disconnect).

Specified by:

isUnavailable in class AbstractFetch

Parameters:

node - Node to check for availability

See Also:

• NetworkClientUtils.isUnavailable(KafkaClient, Node, Time)

maybeThrowAuthFailure

protected void maybeThrowAuthFailure(org.apache.kafka.common.Node node)

Description copied from class: AbstractFetch

Checks for an authentication error on a given node and throws the exception if it exists.

Specified by:

maybeThrowAuthFailure in class AbstractFetch

Parameters:

node - Node to check for a previous AuthenticationException; if found it is thrown

See Also:

• NetworkClientUtils.maybeThrowAuthFailure(KafkaClient, Node)

clearBufferedDataForUnassignedPartitions

public void clearBufferedDataForUnassignedPartitions(Collection<org.apache.kafka.common.TopicPartition> assignedPartitions)

sendFetches

public int sendFetches()

Set up a fetch request for any node that we have assigned partitions for which doesn't already have an in-flight fetch or pending fetch data.

Returns:

number of fetches sent

maybeCloseFetchSessions

protected void maybeCloseFetchSessions(org.apache.kafka.common.utils.Timer timer)

collectFetch

public Fetch<K,V> collectFetch()

closeInternal

protected void closeInternal(org.apache.kafka.common.utils.Timer timer)

This method is called by AbstractFetch.close(Timer) which is guarded by the IdempotentCloser) such as to only be executed once the first time that any of the AbstractFetch.close() methods are called. Subclasses can override this method without the need for extra synchronization at the instance level.

Note: this method is synchronized to reinstitute the 3.5 behavior:

Shared states (e.g. sessionHandlers) could be accessed by multiple threads (such as heartbeat thread), hence, it is necessary to acquire a lock on the fetcher instance before modifying the states.

Overrides:

closeInternal in class AbstractFetch

Parameters:

timer - Timer to enforce time limit