Enable follower fetching in Aiven for Apache Kafka®

Enabling follower fetching in Aiven for Apache Kafka® allows your consumers to fetch data from the nearest replica instead of the leader, optimizing data fetching and enhancing performance.

Prerequisites

• Aiven for Apache Kafka service version 3.6 or later.
• Availability zone (AZ) information for your Aiven for Apache Kafka service.
• Aiven CLI client.
• Aiven Provider for Terraform.

note

Follower fetching is supported on AWS (Amazon Web Services) and Google Cloud.

Identify availability zone

Before configuring client-side rack awareness, identify the AZs where your Kafka brokers run.

• AWS: Availability zone (AZ) names can vary across different accounts. The same physical location might have different AZ names in different accounts. To ensure consistency when configuring client.rack, use the AZ ID, which remains the same across accounts.

  To map AZ names to AZ IDs, see the AWS Knowledge Center article and the AWS documentation on AZ IDs.

• Google Cloud: Use the AZ name directly as the client.rack value. For more information, see Google Cloud's regions and zones documentation.

Enable follower fetching

Use one of the following methods to enable follower fetching on your Aiven for Apache Kafka service.

Console

1. Access the Aiven Console, and select your Aiven for Apache Kafka service.
2. Click Service settings.
3. Scroll to Advanced configuration and click Configure.
4. Click Add configuration options.
5. Select follower_fetching.enabled from the list and set the value to Enabled.
6. Click Save configurations.

CLI

Enable follower fetching on an existing service with the Aiven CLI:

avn service update <service-name> -c follower_fetching.enabled=true

Parameters:

• <service-name>: Name of your Aiven for Apache Kafka service.
• follower_fetching={"enabled": true}: Enables the follower fetching feature.

API

Use the ServiceUpdate API to enable follower fetching on an existing service:

curl --request PUT \
    --url https://api.aiven.io/v1/project/YOUR_PROJECT_NAME/service/YOUR_SERVICE_NAME \
    --header 'Authorization: Bearer YOUR_BEARER_TOKEN' \
    --header 'content-type: application/json' \
    --data '{
        "user_config": {
            "follower_fetching": {
                "enabled": true
            }
        }
    }'

Parameters:

• YOUR_PROJECT_NAME: Name of your project.
• YOUR_SERVICE_NAME: Name of your service.
• YOUR_BEARER_TOKEN: API token for authentication.
• follower_fetching={"enabled": true}: Enables the follower fetching feature.

Terraform

Use the follower_fetching attribute in your aiven_kafka resource.

Client-side configuration

To enable follower fetching at the client level, configure the client.rack setting in the Apache Kafka client. Set the client.rack value to the corresponding AZ ID for AWS or AZ name for Google Cloud for each client. This ensures the client fetches data from the nearest replica.

Example configuration for your consumer properties file:

client.rack=use1-az1 # AWS example
client.rack=europe-west1-b # Google Cloud example

Example scenario: follower fetching in different AZs

Assume you have an Aiven for Apache Kafka cluster running in two AZs in the us-east-1 region for AWS and in the europe-west1 region for Google Cloud:

Cluster setup and consumer distribution

Cloud	Region	AZs for brokers	AZs for consumers
AWS	us-east-1	use1-az1, use1-az2	use1-az1, use1-az2, use1-az3
Google Cloud	europe-west1	europe-west1-b, europe-west1-c	europe-west1-b, europe-west1-c, europe-west1-d

Consumer configuration

Set the client.rack value to the respective AZ ID for AWS or AZ name for Google Cloud for each consumer:

# AWS consumers in use1-az1
client.rack=use1-az1

# AWS consumers in use1-az2
client.rack=use1-az2

# AWS consumers in use1-az3
client.rack=use1-az3

# Google Cloud consumers in europe-west1-b
client.rack=europe-west1-b

# Google Cloud consumers in europe-west1-c
client.rack=europe-west1-c

# Google Cloud consumers in europe-west1-d
client.rack=europe-west1-d

Fetching behavior

Cloud	Consumer location	Fetching behavior	Notes
AWS	use1-az1	Fetch from the nearest replica in their AZ	Reduced latency and network costs
AWS	use1-az2	Fetch from the nearest replica in their AZ	Reduced latency and network costs
AWS	use1-az3	Fetch from the leader (no matching broker.rack)	No follower fetching possible
Google Cloud	europe-west1-b	Fetch from the nearest replica in their AZ	Reduced latency and network costs
Google Cloud	europe-west1-c	Fetch from the nearest replica in their AZ	Reduced latency and network costs
Google Cloud	europe-west1-d	Fetch from the leader (no matching broker.rack)	No follower fetching possible

Use follower fetching with Kafka Connect and MirrorMaker 2

Aiven for Apache Kafka® Connect and Aiven for Apache Kafka® MirrorMaker 2 use follower fetching when it is enabled on your Aiven for Kafka service.

Kafka Connect

Kafka Connect sets consumer.client.rack based on each node’s availability zone. To disable rack awareness for a specific connector, set:

{
  "consumer.override.client.rack": "noop"
}

MirrorMaker 2

MirrorMaker 2 uses a rack ID based on the node’s availability zone when follower_fetching_enabled=true (default). You can override this by setting a rack_id in the integration configuration.

Rack awareness does not apply to external Kafka clusters.

See Configure rack awareness in MirrorMaker 2.

Verify follower fetching

After configuring follower fetching, monitor for a decrease in cross-availability zone network costs to verify its effectiveness.

Related pages

• Follower fetching in Aiven for Apache Kafka®