lumberjack改造前
This commit is contained in:
robin
@@ -347,3 +347,67 @@ Fluent Bit 写入时使用 `json_date_key timestamp` 和 `json_date_format epoch
|
||||
若在管理端设置了公用访问日志策略的文件 `path`，节点会优先使用该目录；否则才使用 `EDGE_LOG_DIR`。Fluent Bit 的 `Path` 需与实际目录一致。
|
||||
|
||||
以上完成即完成 Fluent Bit 的部署与验证。
|
||||
|
||||
---
|
||||
|
||||
## 九、HTTPS 模式（ClickHouse）
|
||||
|
||||
当 ClickHouse 只开放 HTTPS（如 8443）或链路必须加密时，使用本目录新增模板：
|
||||
|
||||
- `fluent-bit-https.conf`：Node+DNS 同机采集（HTTP+DNS 双输入）
|
||||
- `fluent-bit-dns-https.conf`：仅 DNS 节点采集
|
||||
- `fluent-bit-windows-https.conf`：Windows 节点 HTTPS 采集
|
||||
|
||||
### 9.1 什么时候用 HTTPS 模板
|
||||
|
||||
- ClickHouse 仅开放 HTTPS 端口；
|
||||
- 节点到 ClickHouse 跨公网或需要传输加密；
|
||||
- 你希望启用证书校验和 SNI。
|
||||
|
||||
### 9.2 最小切换步骤（Linux）
|
||||
|
||||
1. 备份当前配置：
|
||||
```bash
|
||||
sudo cp /etc/fluent-bit/fluent-bit.conf /etc/fluent-bit/fluent-bit.conf.bak
|
||||
```
|
||||
|
||||
2. 切换为 HTTPS 模板（Node+DNS 同机示例）：
|
||||
```bash
|
||||
sudo cp /path/to/fluent-bit-https.conf /etc/fluent-bit/fluent-bit.conf
|
||||
```
|
||||
|
||||
3. 设置账号密码（按你的服务文件方式设置）：
|
||||
```bash
|
||||
export CH_USER=default
|
||||
export CH_PASSWORD='your_password'
|
||||
```
|
||||
|
||||
4. 修改模板中的关键项：
|
||||
- `Host` / `Port`（HTTPS 常见端口 `8443`）
|
||||
- `tls.verify`：`On`/`Off`
|
||||
- `tls.ca_file`：自签名证书建议配置 CA 文件
|
||||
- `tls.vhost`：证书 CN/SAN 对应主机名（SNI）
|
||||
|
||||
5. 重启并检查：
|
||||
```bash
|
||||
sudo systemctl restart fluent-bit
|
||||
sudo systemctl status fluent-bit
|
||||
journalctl -u fluent-bit -f
|
||||
```
|
||||
|
||||
### 9.3 验证点
|
||||
|
||||
- `default.logs_ingest` 有新增数据（HTTP）
|
||||
- `default.dns_logs_ingest` 有新增数据（DNS）
|
||||
- Fluent Bit 日志中无 TLS 握手失败（`certificate`, `x509`, `tls`）
|
||||
|
||||
### 9.4 回滚
|
||||
|
||||
TLS 配置错误导致中断时，快速回滚：
|
||||
|
||||
```bash
|
||||
sudo cp /etc/fluent-bit/fluent-bit.conf.bak /etc/fluent-bit/fluent-bit.conf
|
||||
sudo systemctl restart fluent-bit
|
||||
```
|
||||
|
||||
回滚后恢复原 HTTP 模式，不影响平台 API/管理端配置。
|
||||
|
||||
39
deploy/fluent-bit/fluent-bit-dns-https.conf
Normal file
39
deploy/fluent-bit/fluent-bit-dns-https.conf
Normal file
@@ -0,0 +1,39 @@
|
||||
# DNS 节点专用 HTTPS：使用 HTTP 输出写入 ClickHouse（无需 out_clickhouse 插件）
|
||||
# 启动前设置：CH_USER、CH_PASSWORD；按需修改 Host、Port（默认 127.0.0.1:8443）
|
||||
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
storage.path /var/lib/fluent-bit/storage
|
||||
storage.sync normal
|
||||
storage.checksum off
|
||||
storage.backlog.mem_limit 128MB
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-dns/*.log
|
||||
Tag app.dns.logs
|
||||
Parser json
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/dns-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.dns.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT%20INTO%20default.dns_logs_ingest%20FORMAT%20JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file /etc/ssl/certs/ca-certificates.crt
|
||||
# tls.vhost clickhouse.example.com
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
74
deploy/fluent-bit/fluent-bit-https.conf
Normal file
74
deploy/fluent-bit/fluent-bit-https.conf
Normal file
@@ -0,0 +1,74 @@
|
||||
# Fluent Bit HTTPS 配置（边缘节点日志采集 -> ClickHouse HTTPS）
|
||||
# HTTP: /var/log/edge/edge-node/*.log
|
||||
# DNS: /var/log/edge/edge-dns/*.log
|
||||
#
|
||||
# 启动前请设置环境变量：
|
||||
# CH_USER=default
|
||||
# CH_PASSWORD=your_password
|
||||
# 如需改地址/端口，请修改 OUTPUT 中 Host/Port（默认 127.0.0.1:8443）
|
||||
# 如证书为公网CA可省略 tls.ca_file；自签名证书请配置 tls.ca_file
|
||||
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
storage.path /var/lib/fluent-bit/storage
|
||||
storage.sync normal
|
||||
storage.checksum off
|
||||
storage.backlog.mem_limit 128MB
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-node/*.log
|
||||
Tag app.http.logs
|
||||
Parser json
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/http-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-dns/*.log
|
||||
Tag app.dns.logs
|
||||
Parser json
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/dns-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.http.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT%20INTO%20default.logs_ingest%20FORMAT%20JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file /etc/ssl/certs/ca-certificates.crt
|
||||
# tls.vhost clickhouse.example.com
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.dns.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT%20INTO%20default.dns_logs_ingest%20FORMAT%20JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file /etc/ssl/certs/ca-certificates.crt
|
||||
# tls.vhost clickhouse.example.com
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
62
deploy/fluent-bit/fluent-bit-windows-https.conf
Normal file
62
deploy/fluent-bit/fluent-bit-windows-https.conf
Normal file
@@ -0,0 +1,62 @@
|
||||
[SERVICE]
|
||||
Flush 1
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
storage.path ./storage
|
||||
storage.sync normal
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path E:\var\log\edge\edge-node\*.log
|
||||
Tag app.http.logs
|
||||
Parser json
|
||||
Refresh_Interval 1
|
||||
Read_from_Head true
|
||||
DB ./http-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path E:\var\log\edge\edge-dns\*.log
|
||||
Tag app.dns.logs
|
||||
Parser json
|
||||
Refresh_Interval 1
|
||||
Read_from_Head true
|
||||
DB ./dns-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.http.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT+INTO+logs_ingest+FORMAT+JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file C:\\path\\to\\ca.pem
|
||||
# tls.vhost clickhouse.example.com
|
||||
Json_Date_Key timestamp
|
||||
Json_Date_Format epoch
|
||||
Retry_Limit 10
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.dns.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT+INTO+dns_logs_ingest+FORMAT+JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file C:\\path\\to\\ca.pem
|
||||
# tls.vhost clickhouse.example.com
|
||||
Json_Date_Key timestamp
|
||||
Json_Date_Format epoch
|
||||
Retry_Limit 10
|
||||
Reference in New Issue
Block a user