架構(gòu)下的日志集中管理實(shí)踐與EFK方案)
1. 為什么需要多服務(wù)日志集中管理在微服務(wù)架構(gòu)中一個(gè)典型的生產(chǎn)環(huán)境往往包含多個(gè)后端服務(wù)和前端應(yīng)用。以我最近部署的一個(gè)電商平臺(tái)為例整套系統(tǒng)包含3個(gè)Node.js后端服務(wù)用戶服務(wù)/訂單服務(wù)/商品服務(wù)2個(gè)Vue前端應(yīng)用主站/管理后臺(tái)1個(gè)Nginx網(wǎng)關(guān)Redis和MySQL各1個(gè)實(shí)例當(dāng)所有服務(wù)都通過(guò)Docker Compose部署后最頭疼的問(wèn)題就是日志分散在各個(gè)容器中。上周排查一個(gè)支付失敗問(wèn)題時(shí)我不得不同時(shí)docker logs order-service查看訂單服務(wù)日志docker logs payment-service查看支付服務(wù)日志docker logs nginx查看網(wǎng)關(guān)日志再到Kibana中篩選前端錯(cuò)誤日志這種碎片化的排查方式效率極低特別是當(dāng)多個(gè)服務(wù)間存在調(diào)用鏈時(shí)。更糟的是docker logs默認(rèn)只保留最新日志歷史數(shù)據(jù)隨時(shí)可能丟失。2. 基礎(chǔ)版方案直接掛載日志卷2.1 目錄結(jié)構(gòu)設(shè)計(jì)我采用的日志目錄結(jié)構(gòu)如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 關(guān)鍵Compose配置以訂單服務(wù)為例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 實(shí)際踩坑記錄權(quán)限問(wèn)題首次啟動(dòng)時(shí)出現(xiàn)Permission denied錯(cuò)誤解決方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生產(chǎn)環(huán)境應(yīng)配置精確權(quán)限日志輪轉(zhuǎn)失效發(fā)現(xiàn)日志文件超過(guò)10MB后未切割原因是必須同時(shí)設(shè)置max-size和max-file某些應(yīng)用如Nginx需要在其配置中也開啟日志輪轉(zhuǎn)磁盤空間預(yù)警建議添加監(jiān)控腳本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 進(jìn)階版方案EFK全棧日志系統(tǒng)3.1 架構(gòu)設(shè)計(jì)graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 關(guān)鍵組件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch調(diào)優(yōu)docker-compose.efk.yml內(nèi)存設(shè)置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根據(jù)服務(wù)器內(nèi)存調(diào)整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能優(yōu)化技巧日志過(guò)濾在Fluentd中丟棄DEBUG級(jí)別日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自動(dòng)創(chuàng)建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查詢加速為常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署實(shí)戰(zhàn)案例4.1 場(chǎng)景需求開發(fā)環(huán)境使用基礎(chǔ)版日志卷生產(chǎn)環(huán)境使用EFK全棧方案兩種環(huán)境配置無(wú)縫切換4.2 條件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}啟動(dòng)命令示例# 開發(fā)環(huán)境 LOG_DRIVERjson-file docker-compose up # 生產(chǎn)環(huán)境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 監(jiān)控指標(biāo)設(shè)計(jì)Grafana儀表盤建議包含日志量趨勢(shì)按服務(wù)/級(jí)別錯(cuò)誤日志TOP 10日志存儲(chǔ)增長(zhǎng)預(yù)測(cè)關(guān)鍵服務(wù)響應(yīng)時(shí)間關(guān)聯(lián)分析5. 異常處理手冊(cè)5.1 Fluentd常見(jiàn)問(wèn)題癥狀日志未出現(xiàn)在Kibana中 排查步驟檢查Fluentd容器狀態(tài)docker-compose logs fluentd | grep -i error驗(yàn)證Elasticsearch連接curl -XGET http://elasticsearch:9200/_cat/indices?v查看緩沖隊(duì)列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丟失預(yù)防啟用本地備份雙寫策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match設(shè)置磁盤水位警戒線docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 訪問(wèn)控制Kibana基礎(chǔ)認(rèn)證services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成證書 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 掛載到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脫敏Fluentd過(guò)濾器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本優(yōu)化實(shí)踐7.1 冷熱數(shù)據(jù)分離PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采樣配置對(duì)DEBUG日志進(jìn)行50%采樣filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter 微服務(wù)架構(gòu)下的日志集中管理實(shí)踐與EFK方案
1. 為什么需要多服務(wù)日志集中管理在微服務(wù)架構(gòu)中一個(gè)典型的生產(chǎn)環(huán)境往往包含多個(gè)后端服務(wù)和前端應(yīng)用。以我最近部署的一個(gè)電商平臺(tái)為例整套系統(tǒng)包含3個(gè)Node.js后端服務(wù)用戶服務(wù)/訂單服務(wù)/商品服務(wù)2個(gè)Vue前端應(yīng)用主站/管理后臺(tái)1個(gè)Nginx網(wǎng)關(guān)Redis和MySQL各1個(gè)實(shí)例當(dāng)所有服務(wù)都通過(guò)Docker Compose部署后最頭疼的問(wèn)題就是日志分散在各個(gè)容器中。上周排查一個(gè)支付失敗問(wèn)題時(shí)我不得不同時(shí)docker logs order-service查看訂單服務(wù)日志docker logs payment-service查看支付服務(wù)日志docker logs nginx查看網(wǎng)關(guān)日志再到Kibana中篩選前端錯(cuò)誤日志這種碎片化的排查方式效率極低特別是當(dāng)多個(gè)服務(wù)間存在調(diào)用鏈時(shí)。更糟的是docker logs默認(rèn)只保留最新日志歷史數(shù)據(jù)隨時(shí)可能丟失。2. 基礎(chǔ)版方案直接掛載日志卷2.1 目錄結(jié)構(gòu)設(shè)計(jì)我采用的日志目錄結(jié)構(gòu)如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 關(guān)鍵Compose配置以訂單服務(wù)為例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 實(shí)際踩坑記錄權(quán)限問(wèn)題首次啟動(dòng)時(shí)出現(xiàn)Permission denied錯(cuò)誤解決方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生產(chǎn)環(huán)境應(yīng)配置精確權(quán)限日志輪轉(zhuǎn)失效發(fā)現(xiàn)日志文件超過(guò)10MB后未切割原因是必須同時(shí)設(shè)置max-size和max-file某些應(yīng)用如Nginx需要在其配置中也開啟日志輪轉(zhuǎn)磁盤空間預(yù)警建議添加監(jiān)控腳本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 進(jìn)階版方案EFK全棧日志系統(tǒng)3.1 架構(gòu)設(shè)計(jì)graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 關(guān)鍵組件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch調(diào)優(yōu)docker-compose.efk.yml內(nèi)存設(shè)置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根據(jù)服務(wù)器內(nèi)存調(diào)整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能優(yōu)化技巧日志過(guò)濾在Fluentd中丟棄DEBUG級(jí)別日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自動(dòng)創(chuàng)建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查詢加速為常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署實(shí)戰(zhàn)案例4.1 場(chǎng)景需求開發(fā)環(huán)境使用基礎(chǔ)版日志卷生產(chǎn)環(huán)境使用EFK全棧方案兩種環(huán)境配置無(wú)縫切換4.2 條件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}啟動(dòng)命令示例# 開發(fā)環(huán)境 LOG_DRIVERjson-file docker-compose up # 生產(chǎn)環(huán)境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 監(jiān)控指標(biāo)設(shè)計(jì)Grafana儀表盤建議包含日志量趨勢(shì)按服務(wù)/級(jí)別錯(cuò)誤日志TOP 10日志存儲(chǔ)增長(zhǎng)預(yù)測(cè)關(guān)鍵服務(wù)響應(yīng)時(shí)間關(guān)聯(lián)分析5. 異常處理手冊(cè)5.1 Fluentd常見(jiàn)問(wèn)題癥狀日志未出現(xiàn)在Kibana中 排查步驟檢查Fluentd容器狀態(tài)docker-compose logs fluentd | grep -i error驗(yàn)證Elasticsearch連接curl -XGET http://elasticsearch:9200/_cat/indices?v查看緩沖隊(duì)列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丟失預(yù)防啟用本地備份雙寫策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match設(shè)置磁盤水位警戒線docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 訪問(wèn)控制Kibana基礎(chǔ)認(rèn)證services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成證書 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 掛載到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脫敏Fluentd過(guò)濾器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本優(yōu)化實(shí)踐7.1 冷熱數(shù)據(jù)分離PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采樣配置對(duì)DEBUG日志進(jìn)行50%采樣filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter