Kubernetes Service(4): envoy overview

  • kkumtree

2024-10-15T10:16:38+09:00

kans
envoy
proxy
kubernetes

λ”°λ‘œ μŠ₯ μ°Ύμ•„λ³΄λ‹ˆ, envoyλŠ” Micro Service Architecture λ“± κ΅¬ν˜„λœ λ‹¨μœ„ κΈ°λŠ₯κ°„μ˜ 톡신을 μœ„ν•œ L7 Proxy 라고 ν•©λ‹ˆλ‹€.
Docker Compose μ •λ„λ‚˜ 일반 μ„œλΉ„μŠ€μ—μ„œλŠ” ꡳ이 ν•„μš”ν•˜μ§€λŠ” μ•Šμ„ 것 κ°™μ§€λ§Œ, Service Mesh ν™˜κ²½μ—μ„œλŠ” μ•Œμ•„λ‘λ©΄ 쒋을 것 κ°™μ•„ ν›μ–΄λ΄…λ‹ˆλ‹€.

CloudNet@μ—μ„œ μ§„ν–‰ν•˜κ³  μžˆλŠ” K8s Advanced Network Study(μ΄ν•˜, KANS)λ₯Ό 톡해 ν•™μŠ΅ν•œ λ‚΄μš©μ„ μ •λ¦¬ν•©λ‹ˆλ‹€.

1. Envoy Installation

wget -O- https://apt.envoyproxy.io/signing.key | sudo gpg --dearmor -o /etc/apt/keyrings/envoy-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/envoy-keyring.gpg] https://apt.envoyproxy.io jammy main" | sudo tee /etc/apt/sources.list.d/envoy.list
sudo apt-get update
sudo apt-get install envoy
envoy --version

ν•™μŠ΅ν™˜κ²½μ€ root둜 μ ‘μ†λ˜μ–΄ μžˆκΈ°μ— sudoλŠ” 쓰지 μ•Šμ•˜μŠ΅λ‹ˆλ‹€.

wget -O- https://apt.envoyproxy.io/signing.key | sudo gpg --dearmor -o /etc/apt/keyrings/envoy-keyring.gpg
--2024-10-15 09:46:22--  https://apt.envoyproxy.io/signing.key
Resolving apt.envoyproxy.io (apt.envoyproxy.io)... 13.215.144.61, 13.251.96.10, 2406:da18:880:3802::c8, ...
Connecting to apt.envoyproxy.io (apt.envoyproxy.io)|13.215.144.61|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3158 (3.1K) [application/vnd.apple.keynote]
Saving to: β€˜STDOUT’

-                   100%[===================>]   3.08K  --.-KB/s    in 0s      

2024-10-15 09:46:23 (86.8 MB/s) - written to stdout [3158/3158]

echo "deb [signed-by=/etc/apt/keyrings/envoy-keyring.gpg] https://apt.envoyproxy.io jammy main" | sudo tee /etc/apt/sources.list.d/envoy.list
deb [signed-by=/etc/apt/keyrings/envoy-keyring.gpg] https://apt.envoyproxy.io jammy main
apt-get update && apt-get install envoy -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  envoy
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 73.2 MB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://apt.envoyproxy.io jammy/main amd64 envoy amd64 1.31.2 [73.2 MB]
Fetched 73.2 MB in 6s (12.2 MB/s)
Selecting previously unselected package envoy.
(Reading database ... 66661 files and directories currently installed.)
Preparing to unpack .../envoy_1.31.2_amd64.deb ...
Unpacking envoy (1.31.2) ...
Setting up envoy (1.31.2) ...

You have installed the Envoy proxy server.

You can check your Envoy version by running the following in a terminal:

  $ envoy --version

Documentation for your version is available at:

  https://www.envoyproxy.io/docs

The Envoy project can be found at:

  https://github.com/envoyproxy/envoy

Scanning processes...                                                           
Scanning linux images...                                                        

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
envoy --version

envoy  version: cc4a75482810de4b84c301d13deb551bd3147339/1.31.2/Clean/RELEASE/BoringSSL
  • μ˜΅μ…˜ 확인

envoy μ˜΅μ…˜μ€ envoy -h 둜 확인가λŠ₯ν•©λ‹ˆλ‹€.
man pageλŠ” λ”°λ‘œ μ„€μΉ˜λ˜μ§€ μ•ŠλŠ” 것 κ°™μŠ΅λ‹ˆλ‹€.

man envoy
# No manual entry for envoy

2. Envoy Quick start

  • 잘 λͺ¨λ₯΄κ² μœΌλ‹ˆ κ·Έλƒ₯ λ”°λΌν•©λ‹ˆλ‹€.
  • Envoy Docs

(a) Config 데λͺ¨ 적용

ν•œ μͺ½μ—λŠ” Envoyλ₯Ό 켜고, ν•œ μͺ½μ—μ„œλŠ” 접속 ν…ŒμŠ€νŠΈλ₯Ό ν•΄λ³Ό κ²λ‹ˆλ‹€.
μŠ€ν„°λ””μ—μ„œ 같은 μ„œλΈŒλ„· ꡬ성이 된 ν™˜κ²½μ„ μ œκ³΅ν•΄μ£Όμ…¨κΈ°μ—, 이 점은 μ–‘ν•΄λ°”λžλ‹ˆλ‹€.

  • Terminal 0) Turn On Envoy
    • foreground μƒνƒœλΌ, μΌœλ†“μ€ μƒνƒœμ—μ„œ λ‹€λ₯Έ 터미널을 μ—½λ‹ˆλ‹€.
curl -O https://www.envoyproxy.io/docs/envoy/latest/_downloads/92dcb9714fb6bc288d042029b34c0de4/envoy-demo.yaml
envoy -c envoy-demo.yaml
  • Terminal 1) ν…ŒμŠ€νŠΈ
ss -tnlp
# State    Recv-Q   Send-Q     Local Address:Port        Peer Address:Port   Process                                                                    
# LISTEN   0        4096       127.0.0.53%lo:53               0.0.0.0:*       users:(("systemd-resolve",pid=347,fd=14))                                 
# LISTEN   0        128              0.0.0.0:22               0.0.0.0:*       users:(("sshd",pid=703,fd=3))                                             
# LISTEN   0        4096             0.0.0.0:10000            0.0.0.0:*       users:(("envoy",pid=10390,fd=25))                                         
# LISTEN   0        4096             0.0.0.0:10000            0.0.0.0:*       users:(("envoy",pid=10390,fd=24))                                         
# LISTEN   0        511                    *:80                     *:*       users:(("apache2",pid=2376,fd=4),("apache2",pid=2375,fd=4),("apache2",pid=2373,fd=4))
# LISTEN   0        128                 [::]:22                  [::]:*       users:(("sshd",pid=703,fd=4))  
curl -s http://127.0.0.1:10000 | grep -o "<title>.*</title>"
# <title>Envoy proxy - home</title>
echo -e "http://$(curl -s ipinfo.io/ip):10000"
# http://54.180.163.59:10000

envoy-hello-world

  • Terminal2) Test in k3s master node
    • 192.168.10.200: Where envoy is running
curl -s http://192.168.10.200:10000 | grep -o "<title>.*</title>" 
# <title>Envoy proxy - home</title>

(b) Config μ„€μ • λ³€κ²½

μ•žμ„œ κ΅¬λ™ν•œ envoyλ₯Ό μ’…λ£Œν•˜κ³ , λ‹€μ‹œ μ‹€ν–‰ν•©λ‹ˆλ‹€.
-c λ‚˜ --config-path μ˜΅μ…˜μ€ λ™μΌν•©λ‹ˆλ‹€.
λ‹€λ§Œ, μ˜΅μ…˜ overrideλ₯Ό ν•  λ•Œ, μΆ”κ°€λ‘œ merging λ˜λŠ” ν™˜κ²½λ³€μˆ˜λŠ”
--config-path μ˜΅μ…˜μ„ μ‚¬μš©ν•˜λ„λ‘ κΆŒν•˜λŠ” 것 κ°™μŠ΅λ‹ˆλ‹€.

cat <<EOT> envoy-override.yaml
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9902
EOT

envoy -c envoy-demo.yaml --config-path "$(cat envoy-override.yaml)"

이 경우, 10000 Port 외에 μΆ”κ°€λ‘œ 9902 포트λ₯Ό 톡해 Admin νŽ˜μ΄μ§€μ— μ ‘κ·Ό ν•  수 μžˆμ—ˆμŠ΅λ‹ˆλ‹€.

envoy-admin

(c) Config μœ νš¨μ„± 검사

--mode validate μ˜΅μ…˜μ„ 톡해, μ„€μ • 파일의 μœ νš¨μ„±μ„ 검사할 수 μžˆμŠ΅λ‹ˆλ‹€.

envoy --mode validate -c envoy-demo.yaml  
# [2024-10-19 15:45:46.382][10661][info][main] [source/server/server.cc:879] runtime: {}
# [2024-10-19 15:45:46.383][10661][info][config] [source/server/configuration_impl.cc:168] loading tracing configuration
# [2024-10-19 15:45:46.383][10661][info][config] [source/server/configuration_impl.cc:124] loading 0 static secret(s)
# [2024-10-19 15:45:46.383][10661][info][config] [source/server/configuration_impl.cc:130] loading 1 cluster(s)
# [2024-10-19 15:45:46.384][10661][info][config] [source/server/configuration_impl.cc:138] loading 1 listener(s)
# [2024-10-19 15:45:46.386][10661][warning][misc] [source/extensions/filters/network/http_connection_manager/config.cc:88] internal_address_config is not configured. The existing default behaviour will trust RFC1918 IP addresses, but this will be changed in next release. Please explictily config internal address config as the migration step or config the envoy.reloadable_features.explicit_internal_address_config to true to untrust all ips by default
# [2024-10-19 15:45:46.389][10661][info][config] [source/server/configuration_impl.cc:154] loading stats configuration
# configuration 'envoy-demo.yaml' OK

(d) Envoy logging μ„€μ •

기본적으둜 /dev/stderr에 λ‘œκΉ…μ„ ν•œλ‹€κ³  ν•©λ‹ˆλ‹€.
character special file(문자 특수 파일)μ΄λ„€μš”.

λž„κΉŒ, container ν™˜κ²½μ—μ„œλŠ” stderr/stdout을 톡해 일반적으둜 λ‘œκΉ…ν•˜λŠ” 것 κ°™κΈ΄ ν•©λ‹ˆλ‹€.

readlink -e /dev/stderr
# /dev/pts/3
readlink /dev/stderr
# /proc/self/fd/2
readlink /proc/self/fd/2
# /dev/pts/3
ls -l /dev/pts/3
# crw------- 1 root tty 136, 3 Oct 19 16:00 /dev/pts/3

[택1] μ‹€ν–‰μ‹œ νŒŒλΌλ―Έν„° μ„€μ •

--log-level μ˜΅μ…˜μ„ 톡해, λ‘œκΉ…ν•  경둜λ₯Ό 지정할 수 μžˆμŠ΅λ‹ˆλ‹€.

# ls /var/log/envoy
# ls: cannot access '/var/log/envoy': No such file or directory
# mkdir -p /var/log/envoy
mkdir -p /tmp/envoy-logs
envoy -c envoy-demo.yaml --log-path /tmp/envoy-logs/custom.log

[택2] Admin μΈν„°νŽ˜μ΄μŠ€μ—μ„œ μ„€μ •

envoy-admin-logging

cat envoy-demo.yaml | grep -A 3 -B 3 access_log:
        # typed_config:
        #   "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        #   stat_prefix: ingress_http
        #   access_log:
        #   - name: envoy.access_loggers.stdout
        #     typed_config:
        #       "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog

[이외] Log extension

  • Log extension을 톡해, λ‹€μ–‘ν•œ λ‘œκΉ… 섀정을 ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

(e) Envoy networking

기본값은 IPv6와 IPv4λ₯Ό λͺ¨λ‘ ν™œμ„±ν™”ν•˜λ‚˜ IPv6λ₯Ό λΉ„ν™œμ„±ν™”ν•˜μ—¬μ•Όν•˜λŠ” 상황이 μžˆλ‹€λ©΄,
데λͺ¨ μ„€μ •νŒŒμΌκ°™μ΄ dns_lookup_familyλ₯Ό V4_ONLY둜 μ„€μ •ν•˜λ©΄ λ˜κ² μŠ΅λ‹ˆλ‹€.
linux ν˜ΈμŠ€νŠΈκ°€ μ•„λ‹Œ ν™˜κ²½μ—μ„œλ„ ν•΄λ‹Ή μΌ€μ΄μŠ€κ°€ μžˆμ„ 수 μžˆλ‹€κ³  ν•©λ‹ˆλ‹€. (Docker Docs)

cat envoy-demo.yaml | grep -A 7 -B 4 dns_lookup_family
  clusters:
  - name: service_envoyproxy_io
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: service_envoyproxy_io
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:

(f) Envoy debugging

[택1] basic

-l ν˜Ήμ€ --log-level μ˜΅μ…˜μ„ 톡해, λ‘œκΉ… λ ˆλ²¨μ„ μ„€μ •ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

  • Default: info
  • List: trace, debug, info, warning/warn, error, critical, off

[택2] component

--component-log-level μ˜΅μ…˜μ„ 톡해, μ»΄ν¬λ„ŒνŠΈλ³„λ‘œ λ‘œκΉ…μ„ 지정할 수 μžˆμŠ΅λ‹ˆλ‹€.
μ „μ—­ λ‘œκΉ… λ ˆλ²¨μ„ off둜 μ„€μ •ν•˜κ³ , νŠΉμ • μ»΄ν¬λ„ŒνŠΈλ§Œ λ‘œκΉ…ν•˜κ³  싢을 λ•Œ μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€.

envoy -c envoy-demo.yaml -l off --component-log-level upstream:debug,connection:trace

kkumtree

plumber for infra

kkumtree

Source code on GitHub

Β© 2025 kkumtree and contributors All rights reserved.
Licensed under
CC BY-NC-ND 4.0