[Kafka] - 18. KAFKA_LISTENERS vs KAFKA_ADVERTISED_LISTENERS

개발/Kafka 2022. 8. 23. 23:38

Docker-compose로 Kafka를 구성하던 도중에 내부 설정 값에 따라 계속하여 Container가 exited되는 현상이 발생되었습니다.

내부 Network Connection이 문제였기 때문에 해결하기 위해서는 Listeners라는 개념을 알아야 합니다. ( 삽질... )

Error connecting to node kafka1:19092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka1: Name or service not known

What is difference Listeners and Advertised_Listeners

Listeners

brokers should be able to define protocol for binding

list of listeners and the host/IP and port to which Kafka binds to for listening.
internal traffic hits the brokers directly (performance, cost).
The default is 0.0.0.0, which means listening on all interfaces.

Advertised_listeners

brokers should be able to define protocol for sharing so that internal, external and replication traffic can be separated if required.

list of listeners with their host/IP and port. This is the metadata that’s passed back to clients.
External traffic goes through a proxy/load-balancer (security, flexibility)

간단한 예를 한가지 들어보겠습니다.

3개의 A,B,C Node(server)가 존재하며 해당 Node들 안에는 Kafka(broker)가 존재합니다.

여기서 A Node에는 Service A_1이라는 Application이 존재합니다.

ServiceA_1은 Broker와 같은 Local에서 동작 중임으로 localhost로 접근이 가능합니다.( default 0.0.0.0 )

외부 Service에서 Broker에게 접근하려면 advertised.listeners가 없다면 접근은 절대 불가능합니다.

즉, Internal External 오픈할 특정 IP를 별도로 두기 위해서 listeners, advertised.listeners 가 존재

해결 방법 1) /etc/hosts 수동 Mapping

처음에는 /etc/hosts 값을 추가하여 Mapping 시켜주어 이상없이 수행하였습니다.
Environment에서 값이 설정되는 것이 아닌 수기로 해주어야 함으로 추천하지 않습니다.

$ cat /etc/hosts
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1       localhost
127.0.1.1       DESKTOP-G8J2EVJ.localdomain     DESKTOP-G8J2EVJ
127.0.0.1       kafka1
127.0.0.1       kafka2
127.0.0.1       kafka3

121.135.126.5   host.docker.internal
121.135.126.5   gateway.docker.internal
127.0.0.1       kubernetes.docker.internal

version: '3'
services:
  zookeeper-1:
    hostname: zookeeper1
    image: confluentinc/cp-zookeeper:6.2.0
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 12181
      ZOOKEEPER_DATA_DIR: /zookeeper/data
      ZOOKEEPER_SERVERS: zookeeper1:22888:23888;zookeeper2:32888:33888;zookeeper3:42888:43888
    ports:
      - 12181:12181
      - 22888:22888
      - 23888:23888
    volumes:
      - ./zookeeper/data/1:/zookeeper/data

  zookeeper-2:
    hostname: zookeeper2
    image: confluentinc/cp-zookeeper:6.2.0
    environment:
      ZOOKEEPER_SERVER_ID: 2
      ZOOKEEPER_CLIENT_PORT: 22181
      ZOOKEEPER_DATA_DIR: /zookeeper/data
      ZOOKEEPER_SERVERS: zookeeper1:22888:23888;zookeeper2:32888:33888;zookeeper3:42888:43888
    ports:
      - 22181:22181
      - 32888:32888
      - 33888:33888
    volumes:
      - ./zookeeper/data/2:/zookeeper/data

  zookeeper-3:
    hostname: zookeeper3
    image: confluentinc/cp-zookeeper:6.2.0
    environment:
      ZOOKEEPER_SERVER_ID: 3
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_DATA_DIR: /zookeeper/data
      ZOOKEEPER_SERVERS: zookeeper1:22888:23888;zookeeper2:32888:33888;zookeeper3:42888:43888
    ports:
      - 32181:32181
      - 42888:42888
      - 43888:43888
    volumes:
      - ./zookeeper/data/3:/zookeeper/data
  
  kafka-1:
    image: confluentinc/cp-kafka:6.2.0
    hostname: kafka1
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:12181,zookeeper2:22181,zookeeper3:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:19092
      KAFKA_LOG_DIRS: /kafka
    ports:
      - 19092:19092
    volumes:
      - ./kafka/logs/1:/kafka

  kafka-2:
    image: confluentinc/cp-kafka:6.2.0
    hostname: kafka2
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:12181,zookeeper2:22181,zookeeper3:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:29092
      KAFKA_LOG_DIRS: /kafka
    ports:
      - 29092:29092
    volumes:
      - ./kafka/logs/2:/kafka

  kafka-3:
    image: confluentinc/cp-kafka:6.2.0
    hostname: kafka3
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:12181,zookeeper2:22181,zookeeper3:32181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:39092
      KAFKA_LOG_DIRS: /kafka
    ports:
      - 39092:39092
    volumes:
      - ./kafka/logs/3:/kafka

해결 방안 2) Docker-compose Mapping

advertised_listeners에 값을 설정하여 접근이 가능하도록 만들어줍니다.
공식문서를 참조하여 구현하였습니다.
DESKTOP-G8J2EVJ 값은 Hostname 입니다.

version: '3'
services:
  zookeeper-1:
    hostname: zookeeper1
    image: confluentinc/cp-zookeeper:6.2.0
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 12181
      ZOOKEEPER_DATA_DIR: /zookeeper/data
      ZOOKEEPER_SERVERS: zookeeper1:22888:23888;zookeeper2:32888:33888;zookeeper3:42888:43888
    ports:
      - 12181:12181
      - 22888:22888
      - 23888:23888
    volumes:
      - ./zookeeper/data/1:/zookeeper/data

  zookeeper-2:
    hostname: zookeeper2
    image: confluentinc/cp-zookeeper:6.2.0
    environment:
      ZOOKEEPER_SERVER_ID: 2
      ZOOKEEPER_CLIENT_PORT: 22181
      ZOOKEEPER_DATA_DIR: /zookeeper/data
      ZOOKEEPER_SERVERS: zookeeper1:22888:23888;zookeeper2:32888:33888;zookeeper3:42888:43888
    ports:
      - 22181:22181
      - 32888:32888
      - 33888:33888
    volumes:
      - ./zookeeper/data/2:/zookeeper/data

  zookeeper-3:
    hostname: zookeeper3
    image: confluentinc/cp-zookeeper:6.2.0
    environment:
      ZOOKEEPER_SERVER_ID: 3
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_DATA_DIR: /zookeeper/data
      ZOOKEEPER_SERVERS: zookeeper1:22888:23888;zookeeper2:32888:33888;zookeeper3:42888:43888
    ports:
      - 32181:32181
      - 42888:42888
      - 43888:43888
    volumes:
      - ./zookeeper/data/3:/zookeeper/data
  
  kafka-1:
    image: confluentinc/cp-kafka:6.2.0
    hostname: kafka1
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:12181,zookeeper2:22181,zookeeper3:32181
      KAFKA_LISTENERS: LISTENER_BOB://kafka1:19093,LISTENER_FRED://kafka1:19092
      KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka1:19093,LISTENER_FRED://DESKTOP-G8J2EVJ:19092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB
      KAFKA_LOG_DIRS: /kafka
    ports:
      - 19092:19092

    volumes:
      - ./kafka/logs/1:/kafka

  kafka-2:
    image: confluentinc/cp-kafka:6.2.0
    hostname: kafka2
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:12181,zookeeper2:22181,zookeeper3:32181
      KAFKA_LISTENERS: LISTENER_BOB://kafka2:29093,LISTENER_FRED://kafka2:29092
      KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka2:29093,LISTENER_FRED://DESKTOP-G8J2EVJ:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB
      KAFKA_LOG_DIRS: /kafka
    ports:
      - 29092:29092

    volumes:
      - ./kafka/logs/2:/kafka

  kafka-3:
    image: confluentinc/cp-kafka:6.2.0
    hostname: kafka3
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:12181,zookeeper2:22181,zookeeper3:32181
      KAFKA_LISTENERS: LISTENER_BOB://kafka3:39093,LISTENER_FRED://kafka3:39092
      KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka3:39093,LISTENER_FRED://DESKTOP-G8J2EVJ:39092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB
      KAFKA_LOG_DIRS: /kafka
    ports:
      - 39092:39092

    volumes:
      - ./kafka/logs/3:/kafka

Reference

https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic

https://www.confluent.io/ko-kr/blog/kafka-listeners-explained/

https://leeyh0216.github.io/posts/kafka_docker/

'개발 > Kafka' 카테고리의 다른 글

[Kafka] - 20. Filebeats to Kafka(Log produce) (0)	2022.08.25
[Kafka] - 19. Kafka GUI(AKHQ) (0)	2022.08.24
[Kafka] - 16. Kafka 설치 (0)	2022.08.18
[Kafka] - 17. Docker-compose Kafka 개발환경 구축 및 테스트 (0)	2022.08.11
[Kafka] - 15. EOS Transaction (0)	2022.08.10

ABOUT ME

코딩개발 코딩개발

What is difference Listeners and Advertised_Listeners

Listeners

Advertised_listeners

해결 방법 1) /etc/hosts 수동 Mapping

해결 방안 2) Docker-compose Mapping

Reference

'개발 > Kafka' 카테고리의 다른 글

티스토리툴바

ABOUT ME

What is difference Listeners and Advertised_Listeners

Listeners

Advertised_listeners

해결 방법 1) /etc/hosts 수동 Mapping

해결 방안 2) Docker-compose Mapping

Reference

'개발 > Kafka' 카테고리의 다른 글

관련글 관련글 더보기

티스토리툴바