Initial commit (code only without large binaries)
This commit is contained in:
197
deploy/clickhouse/README.md
Normal file
197
deploy/clickhouse/README.md
Normal file
@@ -0,0 +1,197 @@
|
||||
# ClickHouse + Fluent Bit 快速部署(Ubuntu 22.04 / Amazon Linux 2023)
|
||||
|
||||
## 1. 脚本说明
|
||||
|
||||
- `setup_clickhouse.sh`:一键入口(推荐),默认顺序执行 安装 ClickHouse -> 配置 HTTPS -> 应用运行参数 -> 初始化日志表。
|
||||
- `install_clickhouse_linux.sh`:安装 `clickhouse-server`、`clickhouse-client`,并启动服务。
|
||||
- `configure_clickhouse_https.sh`:生成自签名 `server.crt + server.key`,写入 HTTPS 配置并重启服务。
|
||||
- `configure_clickhouse_runtime.sh`:默认将日志级别设为 `warning`,并禁用高开销系统日志表(`text_log`、`part_log`、`metric_log`、`asynchronous_metric_log`、`trace_log`)。
|
||||
- `init_waf_logs_tables.sh`:执行建表脚本。
|
||||
- `init_waf_logs_tables.sql`:`logs_ingest`、`dns_logs_ingest` 表结构定义。
|
||||
|
||||
进入脚本所在目录
|
||||
```bash
|
||||
cd /opt/waf-platform/deploy/clickhouse
|
||||
chmod +x setup_clickhouse.sh
|
||||
```
|
||||
|
||||
## 2. 一键部署
|
||||
|
||||
### 2.1 方式A:不设置 ClickHouse 密码(用户名固定 `default`)
|
||||
|
||||
```bash
|
||||
sudo ./setup_clickhouse.sh
|
||||
```
|
||||
|
||||
说明:
|
||||
- ClickHouse 连接用户是 `default`
|
||||
- 未设置密码时,后续平台连接密码留空
|
||||
|
||||
### 2.2 方式B:设置用户名/密码(示例使用 `default`)
|
||||
|
||||
```bash
|
||||
sudo CH_USER='default' \
|
||||
CH_PASSWORD='YourStrongPassword' \
|
||||
CH_DATABASE='default' \
|
||||
./setup_clickhouse.sh
|
||||
```
|
||||
|
||||
说明:
|
||||
- `CH_USER`/`CH_PASSWORD`:初始化日志表时用于连接 ClickHouse
|
||||
- 如果你使用自定义用户,把 `CH_USER` 改为你的用户名,并保证该用户已有对应数据库权限
|
||||
|
||||
可选:单独应用运行参数(日志级别/系统日志表开关):
|
||||
|
||||
```bash
|
||||
sudo CH_LOG_LEVEL=warning ./setup_clickhouse.sh runtime
|
||||
```
|
||||
|
||||
## 3. ClickHouse 安装后关键目录
|
||||
|
||||
- 配置目录:`/etc/clickhouse-server/`
|
||||
- 客户端配置目录:`/etc/clickhouse-client/`
|
||||
- 数据目录:`/var/lib/clickhouse/`
|
||||
- 日志目录:`/var/log/clickhouse-server/`
|
||||
- HTTPS 覆盖配置:`/etc/clickhouse-server/config.d/waf-https.xml`
|
||||
- 运行参数覆盖配置:`/etc/clickhouse-server/config.d/waf-runtime.xml`
|
||||
- HTTPS 证书和私钥:`/etc/clickhouse-server/server.crt`、`/etc/clickhouse-server/server.key`
|
||||
- 证书生成中间文件目录:`/etc/clickhouse-server/pki/`
|
||||
|
||||
## 4. 管理平台配置(EdgeAdmin)
|
||||
|
||||
页面路径:
|
||||
- 左侧菜单:`系统设置` -> `高级设置`
|
||||
- 顶部标签:`日志数据库(ClickHouse)`
|
||||
|
||||
表单填写:
|
||||
- `连接地址(Host)`:ClickHouse 地址(IP 或域名),如 `10.0.0.8` 或 `clickhouse.example.com`
|
||||
- `协议(Scheme)`:`https`
|
||||
- `端口(Port)`:`8443`
|
||||
- `用户名(User)`:`default`(或你自定义的用户名)
|
||||
- `密码(Password)`:对应用户密码
|
||||
- `数据库(Database)`:`default`(或你初始化日志表时使用的库名)
|
||||
|
||||
提交顺序:
|
||||
1. 点“测试连接”
|
||||
2. 连接成功后点“保存”
|
||||
|
||||
## 5. Fluent Bit(两种方式)
|
||||
|
||||
### 5.1 跟随节点在线自动安装(推荐)
|
||||
|
||||
说明:
|
||||
- Node / DNS 在线安装或升级时,平台会自动安装/升级 Fluent Bit 并下发配置。
|
||||
- 默认由平台托管,不需要逐台手改配置文件。
|
||||
|
||||
安装后所在节点关键文件:
|
||||
- `/etc/fluent-bit/fluent-bit.conf`:Fluent Bit 主配置(输入日志路径、输出 ClickHouse、性能参数)。
|
||||
- `/etc/fluent-bit/parsers.conf`:日志解析器定义(当前主要使用 JSON parser)。
|
||||
- `/etc/fluent-bit/.edge-managed.env`:平台下发的 ClickHouse 认证环境变量(`CH_USER`/`CH_PASSWORD`)。
|
||||
- `/etc/fluent-bit/.edge-managed.json`:平台下发的元数据(角色、配置哈希、版本、更新时间)。
|
||||
|
||||
|
||||
说明:
|
||||
- 在线安装时,节点上的 `/etc/fluent-bit/fluent-bit.conf` 会被平台下发覆盖。
|
||||
|
||||
fluent-bit中ClickHouse 账号密码下发与更新逻辑:
|
||||
- 下发来源:管理平台 -日志数据库(ClickHouse)中保存的账号密码。
|
||||
- 落地文件:平台在线安装或升级时写入节点 `/etc/fluent-bit/.edge-managed.env`,内容为 `CH_USER`、`CH_PASSWORD`。
|
||||
- 更新触发:当平台里的 ClickHouse 账号或密码变更后,需触发一次节点安装/升级任务以下发新凭证。
|
||||
|
||||
- 常见问题:只在 ClickHouse 侧改密码、未同步更新平台配置时,Fluent Bit 会出现认证失败(401/unauthorized)。
|
||||
|
||||
高配机器调优(当前默认按 4C8G 参数):
|
||||
- 当前默认参数:`Flush=1`、`storage.backlog.mem_limit=512MB`、`Mem_Buf_Limit=256MB`、`workers=2`。
|
||||
- 机器升配后优先调这 4 个参数:
|
||||
- `storage.backlog.mem_limit`:总缓冲上限(先增大,降低突发堆积丢日志风险)。
|
||||
- `Mem_Buf_Limit`:每个 tail input 的内存缓冲(HTTP 与 DNS 两段都要改)。
|
||||
- `workers`:输出并发写入线程数(HTTP 与 DNS 两段都要改)。
|
||||
- `Flush`:刷盘/发送间隔(值越小越实时,CPU/网络开销更高)。
|
||||
- 8C16G 参考值可按 `deploy/fluent-bit/fluent-bit-sample-8c16g.conf`:
|
||||
- `storage.backlog.mem_limit=1024MB`
|
||||
- `Mem_Buf_Limit=512MB`
|
||||
- `workers=4`
|
||||
- `Refresh_Interval=1`
|
||||
- 修改方法:
|
||||
1. 编辑 `EdgeAPI/internal/installers/fluent_bit.go` 的 `renderManagedConfig()`。
|
||||
2. 按上面参数同步修改 Node/DNS 两段 `[INPUT]` 和 `[OUTPUT]`。
|
||||
3. 重新发布 API 并触发节点安装/升级任务,下发新配置。
|
||||
|
||||
检查:
|
||||
|
||||
```bash
|
||||
sudo systemctl status fluent-bit --no-pager
|
||||
sudo cat /etc/fluent-bit/.edge-managed.json
|
||||
sudo journalctl -u fluent-bit -n 100 --no-pager
|
||||
```
|
||||
|
||||
### 5.2 手动安装(自动安装失败时)
|
||||
|
||||
说明:
|
||||
- 适合节点在线自动安装 Fluent Bit 失败的场景。
|
||||
- 采用在线安装方式,由你手动安装并维护配置。
|
||||
|
||||
步骤:
|
||||
|
||||
1. 在线安装 Fluent Bit。
|
||||
|
||||
Ubuntu 22.04:
|
||||
|
||||
```bash
|
||||
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
|
||||
sudo apt-get update -y
|
||||
sudo apt-get install -y fluent-bit
|
||||
```
|
||||
|
||||
AWS 2023:
|
||||
|
||||
```bash
|
||||
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
|
||||
sudo dnf makecache -y
|
||||
sudo dnf install -y fluent-bit
|
||||
```
|
||||
|
||||
2. 放置配置文件:
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /etc/fluent-bit
|
||||
sudo cp /opt/waf-platform/deploy/fluent-bit/fluent-bit.conf /etc/fluent-bit/
|
||||
sudo cp /opt/waf-platform/deploy/fluent-bit/clickhouse-upstream.conf /etc/fluent-bit/
|
||||
sudo cp /opt/waf-platform/deploy/fluent-bit/parsers.conf /etc/fluent-bit/
|
||||
```
|
||||
|
||||
3. 修改 `/etc/fluent-bit/clickhouse-upstream.conf` 的 ClickHouse `Host`、`Port`(如 `8443`)。
|
||||
4. 配置认证环境变量(按需):
|
||||
|
||||
```bash
|
||||
sudo tee /etc/fluent-bit/fluent-bit.env >/dev/null <<'EOF'
|
||||
CH_USER=default
|
||||
CH_PASSWORD=YourStrongPassword
|
||||
EOF
|
||||
```
|
||||
|
||||
5. 让 systemd 读取环境变量:
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /etc/systemd/system/fluent-bit.service.d
|
||||
sudo tee /etc/systemd/system/fluent-bit.service.d/override.conf >/dev/null <<'EOF'
|
||||
[Service]
|
||||
EnvironmentFile=/etc/fluent-bit/fluent-bit.env
|
||||
EOF
|
||||
```
|
||||
|
||||
6. 启动并检查:
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable fluent-bit
|
||||
sudo systemctl restart fluent-bit
|
||||
sudo systemctl status fluent-bit --no-pager
|
||||
sudo journalctl -u fluent-bit -n 100 --no-pager
|
||||
```
|
||||
|
||||
## 6. 验证
|
||||
|
||||
```bash
|
||||
sudo journalctl -u fluent-bit -f
|
||||
```
|
||||
158
deploy/clickhouse/configure_clickhouse_https.sh
Normal file
158
deploy/clickhouse/configure_clickhouse_https.sh
Normal file
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "[ERROR] please run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f /etc/os-release ]]; then
|
||||
echo "[ERROR] /etc/os-release not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
source /etc/os-release
|
||||
os_id="$(echo "${ID:-}" | tr '[:upper:]' '[:lower:]')"
|
||||
os_ver="${VERSION_ID:-}"
|
||||
is_ubuntu22=false
|
||||
is_amzn2023=false
|
||||
|
||||
if [[ "${os_id}" == "ubuntu" && "${os_ver}" == 22.04* ]]; then
|
||||
is_ubuntu22=true
|
||||
fi
|
||||
if [[ "${os_id}" == "amzn" && "${os_ver}" == 2023* ]]; then
|
||||
is_amzn2023=true
|
||||
fi
|
||||
|
||||
if [[ "${is_ubuntu22}" != "true" && "${is_amzn2023}" != "true" ]]; then
|
||||
echo "[ERROR] only Ubuntu 22.04 or Amazon Linux 2023 is supported. current: ID=${ID:-unknown}, VERSION_ID=${VERSION_ID:-unknown}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v openssl >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then
|
||||
if [[ "${is_ubuntu22}" == "true" ]]; then
|
||||
apt-get update -y
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y openssl curl ca-certificates
|
||||
else
|
||||
dnf makecache -y
|
||||
dnf install -y openssl curl ca-certificates
|
||||
fi
|
||||
fi
|
||||
|
||||
CH_HTTPS_PORT="${CH_HTTPS_PORT:-8443}"
|
||||
CH_LISTEN_HOST="${CH_LISTEN_HOST:-::}"
|
||||
CH_CERT_CN="${CH_CERT_CN:-$(hostname -f 2>/dev/null || hostname)}"
|
||||
CH_CERT_DNS="${CH_CERT_DNS:-}"
|
||||
CH_CERT_IP="${CH_CERT_IP:-}"
|
||||
CH_CERT_DAYS="${CH_CERT_DAYS:-825}"
|
||||
|
||||
CH_DIR="/etc/clickhouse-server"
|
||||
CH_CONFIG_D_DIR="${CH_DIR}/config.d"
|
||||
PKI_DIR="${CH_DIR}/pki"
|
||||
SERVER_CERT="${CH_DIR}/server.crt"
|
||||
SERVER_KEY="${CH_DIR}/server.key"
|
||||
OVERRIDE_FILE="${CH_CONFIG_D_DIR}/waf-https.xml"
|
||||
|
||||
mkdir -p "${CH_CONFIG_D_DIR}" "${PKI_DIR}"
|
||||
|
||||
split_csv() {
|
||||
local raw="$1"
|
||||
if [[ -z "${raw}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
IFS=',' read -r -a arr <<<"${raw}"
|
||||
for item in "${arr[@]}"; do
|
||||
item="$(echo "${item}" | xargs)"
|
||||
if [[ -n "${item}" ]]; then
|
||||
echo "${item}"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
build_san_line() {
|
||||
local san_entries=()
|
||||
while IFS= read -r dns_item; do
|
||||
san_entries+=("DNS:${dns_item}")
|
||||
done < <(split_csv "${CH_CERT_DNS}")
|
||||
while IFS= read -r ip_item; do
|
||||
san_entries+=("IP:${ip_item}")
|
||||
done < <(split_csv "${CH_CERT_IP}")
|
||||
|
||||
if [[ ${#san_entries[@]} -eq 0 ]]; then
|
||||
san_entries+=("DNS:${CH_CERT_CN}")
|
||||
fi
|
||||
|
||||
local san_line
|
||||
san_line="$(IFS=,; echo "${san_entries[*]}")"
|
||||
echo "${san_line}"
|
||||
}
|
||||
|
||||
generate_self_signed_cert() {
|
||||
echo "[INFO] generating self-signed server certificate (crt+key only) ..."
|
||||
local server_key="${PKI_DIR}/server.key"
|
||||
local server_csr="${PKI_DIR}/server.csr"
|
||||
local server_crt="${PKI_DIR}/server.crt"
|
||||
local ext_file="${PKI_DIR}/server.ext"
|
||||
local san_line
|
||||
san_line="$(build_san_line)"
|
||||
|
||||
openssl genrsa -out "${server_key}" 2048
|
||||
openssl req -new -key "${server_key}" -out "${server_csr}" -subj "/CN=${CH_CERT_CN}"
|
||||
|
||||
cat >"${ext_file}" <<EOF
|
||||
subjectAltName=${san_line}
|
||||
keyUsage=digitalSignature,keyEncipherment
|
||||
extendedKeyUsage=serverAuth
|
||||
EOF
|
||||
|
||||
openssl x509 -req -in "${server_csr}" -signkey "${server_key}" \
|
||||
-out "${server_crt}" -days "${CH_CERT_DAYS}" -sha256 -extfile "${ext_file}"
|
||||
|
||||
cp -f "${server_crt}" "${SERVER_CERT}"
|
||||
cp -f "${server_key}" "${SERVER_KEY}"
|
||||
}
|
||||
|
||||
generate_self_signed_cert
|
||||
|
||||
chown clickhouse:clickhouse "${SERVER_CERT}" "${SERVER_KEY}" || true
|
||||
chmod 0644 "${SERVER_CERT}"
|
||||
chmod 0640 "${SERVER_KEY}"
|
||||
|
||||
echo "[INFO] writing ClickHouse HTTPS override config ..."
|
||||
cat >"${OVERRIDE_FILE}" <<EOF
|
||||
<clickhouse>
|
||||
<https_port>${CH_HTTPS_PORT}</https_port>
|
||||
<listen_host>${CH_LISTEN_HOST}</listen_host>
|
||||
<openSSL>
|
||||
<server>
|
||||
<certificateFile>${SERVER_CERT}</certificateFile>
|
||||
<privateKeyFile>${SERVER_KEY}</privateKeyFile>
|
||||
<verificationMode>none</verificationMode>
|
||||
<loadDefaultCAFile>true</loadDefaultCAFile>
|
||||
<cacheSessions>true</cacheSessions>
|
||||
<disableProtocols>sslv2,sslv3</disableProtocols>
|
||||
<preferServerCiphers>true</preferServerCiphers>
|
||||
<invalidCertificateHandler>
|
||||
<name>RejectCertificateHandler</name>
|
||||
</invalidCertificateHandler>
|
||||
</server>
|
||||
</openSSL>
|
||||
</clickhouse>
|
||||
EOF
|
||||
|
||||
echo "[INFO] restarting clickhouse-server ..."
|
||||
systemctl restart clickhouse-server
|
||||
sleep 2
|
||||
|
||||
echo "[INFO] service status ..."
|
||||
systemctl --no-pager -l status clickhouse-server | sed -n '1,15p'
|
||||
|
||||
echo "[INFO] verifying HTTPS endpoint ..."
|
||||
curl -sk "https://127.0.0.1:${CH_HTTPS_PORT}/?query=SELECT%201" || true
|
||||
echo
|
||||
|
||||
echo "[OK] ClickHouse HTTPS setup finished"
|
||||
echo " HTTPS port : ${CH_HTTPS_PORT}"
|
||||
echo " cert file : ${SERVER_CERT}"
|
||||
echo " key file : ${SERVER_KEY}"
|
||||
50
deploy/clickhouse/configure_clickhouse_runtime.sh
Normal file
50
deploy/clickhouse/configure_clickhouse_runtime.sh
Normal file
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "[ERROR] please run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
CH_LOG_LEVEL="${CH_LOG_LEVEL:-warning}"
|
||||
CH_DIR="/etc/clickhouse-server"
|
||||
CH_CONFIG_D_DIR="${CH_DIR}/config.d"
|
||||
OVERRIDE_FILE="${CH_CONFIG_D_DIR}/waf-runtime.xml"
|
||||
|
||||
case "${CH_LOG_LEVEL}" in
|
||||
none|fatal|critical|error|warning|notice|information|debug|trace|test)
|
||||
;;
|
||||
*)
|
||||
echo "[ERROR] invalid CH_LOG_LEVEL: ${CH_LOG_LEVEL}"
|
||||
echo " allowed: none,fatal,critical,error,warning,notice,information,debug,trace,test"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
mkdir -p "${CH_CONFIG_D_DIR}"
|
||||
|
||||
echo "[INFO] writing ClickHouse runtime override config ..."
|
||||
cat >"${OVERRIDE_FILE}" <<EOF
|
||||
<clickhouse>
|
||||
<logger>
|
||||
<level>${CH_LOG_LEVEL}</level>
|
||||
</logger>
|
||||
|
||||
<text_log remove="1"/>
|
||||
<part_log remove="1"/>
|
||||
<metric_log remove="1"/>
|
||||
<asynchronous_metric_log remove="1"/>
|
||||
<trace_log remove="1"/>
|
||||
</clickhouse>
|
||||
EOF
|
||||
|
||||
echo "[INFO] restarting clickhouse-server ..."
|
||||
systemctl restart clickhouse-server
|
||||
sleep 2
|
||||
|
||||
echo "[INFO] service status ..."
|
||||
systemctl --no-pager -l status clickhouse-server | sed -n '1,15p'
|
||||
|
||||
echo "[OK] ClickHouse runtime config applied"
|
||||
echo " file : ${OVERRIDE_FILE}"
|
||||
echo " logger level: ${CH_LOG_LEVEL}"
|
||||
38
deploy/clickhouse/init_waf_logs_tables.sh
Normal file
38
deploy/clickhouse/init_waf_logs_tables.sh
Normal file
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SQL_FILE="${SCRIPT_DIR}/init_waf_logs_tables.sql"
|
||||
|
||||
if [[ ! -f "${SQL_FILE}" ]]; then
|
||||
echo "[ERROR] SQL file not found: ${SQL_FILE}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v clickhouse-client >/dev/null 2>&1; then
|
||||
echo "[ERROR] clickhouse-client not found. Please install ClickHouse client first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
CH_HOST="${CH_HOST:-127.0.0.1}"
|
||||
CH_PORT="${CH_PORT:-9000}"
|
||||
CH_USER="${CH_USER:-default}"
|
||||
CH_PASSWORD="${CH_PASSWORD:-}"
|
||||
CH_DATABASE="${CH_DATABASE:-default}"
|
||||
|
||||
args=(--host "${CH_HOST}" --port "${CH_PORT}" --user "${CH_USER}")
|
||||
if [[ -n "${CH_PASSWORD}" ]]; then
|
||||
args+=(--password "${CH_PASSWORD}")
|
||||
fi
|
||||
|
||||
echo "[INFO] creating database if not exists: ${CH_DATABASE}"
|
||||
clickhouse-client "${args[@]}" --query "CREATE DATABASE IF NOT EXISTS ${CH_DATABASE}"
|
||||
|
||||
echo "[INFO] initializing tables in database: ${CH_DATABASE}"
|
||||
clickhouse-client "${args[@]}" --database "${CH_DATABASE}" < "${SQL_FILE}"
|
||||
|
||||
echo "[INFO] checking table status ..."
|
||||
clickhouse-client "${args[@]}" --database "${CH_DATABASE}" --query \
|
||||
"SELECT name, engine FROM system.tables WHERE database='${CH_DATABASE}' AND name IN ('logs_ingest','dns_logs_ingest') ORDER BY name"
|
||||
|
||||
echo "[OK] ClickHouse ingest tables are ready in database '${CH_DATABASE}'"
|
||||
69
deploy/clickhouse/init_waf_logs_tables.sql
Normal file
69
deploy/clickhouse/init_waf_logs_tables.sql
Normal file
@@ -0,0 +1,69 @@
|
||||
-- Initialize HTTP and DNS ingest tables for GoEdge access logs.
|
||||
-- Run with:
|
||||
-- clickhouse-client --database <db_name> < init_waf_logs_tables.sql
|
||||
|
||||
CREATE TABLE IF NOT EXISTS logs_ingest
|
||||
(
|
||||
timestamp DateTime CODEC(DoubleDelta, ZSTD(1)),
|
||||
node_id UInt64,
|
||||
cluster_id UInt64,
|
||||
server_id UInt64,
|
||||
host LowCardinality(String),
|
||||
ip String,
|
||||
method LowCardinality(String),
|
||||
path String CODEC(ZSTD(1)),
|
||||
status UInt16,
|
||||
bytes_in UInt64 CODEC(Delta, ZSTD(1)),
|
||||
bytes_out UInt64 CODEC(Delta, ZSTD(1)),
|
||||
cost_ms UInt32 CODEC(Delta, ZSTD(1)),
|
||||
ua String CODEC(ZSTD(1)),
|
||||
referer String CODEC(ZSTD(1)),
|
||||
log_type LowCardinality(String),
|
||||
trace_id String,
|
||||
firewall_policy_id UInt64 DEFAULT 0,
|
||||
firewall_rule_group_id UInt64 DEFAULT 0,
|
||||
firewall_rule_set_id UInt64 DEFAULT 0,
|
||||
firewall_rule_id UInt64 DEFAULT 0,
|
||||
request_headers String CODEC(ZSTD(3)) DEFAULT '',
|
||||
request_body String CODEC(ZSTD(3)) DEFAULT '',
|
||||
response_headers String CODEC(ZSTD(3)) DEFAULT '',
|
||||
response_body String CODEC(ZSTD(3)) DEFAULT '',
|
||||
INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 4,
|
||||
INDEX idx_ip ip TYPE bloom_filter(0.01) GRANULARITY 4,
|
||||
INDEX idx_host host TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4,
|
||||
INDEX idx_fw_policy firewall_policy_id TYPE minmax GRANULARITY 4,
|
||||
INDEX idx_status status TYPE minmax GRANULARITY 4
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
PARTITION BY toYYYYMMDD(timestamp)
|
||||
ORDER BY (timestamp, node_id, server_id, trace_id)
|
||||
SETTINGS index_granularity = 8192;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS dns_logs_ingest
|
||||
(
|
||||
timestamp DateTime CODEC(DoubleDelta, ZSTD(1)),
|
||||
request_id String,
|
||||
node_id UInt64,
|
||||
cluster_id UInt64,
|
||||
domain_id UInt64,
|
||||
record_id UInt64,
|
||||
remote_addr String,
|
||||
question_name String,
|
||||
question_type LowCardinality(String),
|
||||
record_name String,
|
||||
record_type LowCardinality(String),
|
||||
record_value String,
|
||||
networking LowCardinality(String),
|
||||
is_recursive UInt8,
|
||||
error String CODEC(ZSTD(1)),
|
||||
ns_route_codes Array(String),
|
||||
content_json String CODEC(ZSTD(3)) DEFAULT '',
|
||||
INDEX idx_request_id request_id TYPE bloom_filter(0.01) GRANULARITY 4,
|
||||
INDEX idx_remote_addr remote_addr TYPE bloom_filter(0.01) GRANULARITY 4,
|
||||
INDEX idx_question_name question_name TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4,
|
||||
INDEX idx_domain_id domain_id TYPE minmax GRANULARITY 4
|
||||
)
|
||||
ENGINE = MergeTree
|
||||
PARTITION BY toYYYYMMDD(timestamp)
|
||||
ORDER BY (timestamp, request_id, node_id)
|
||||
SETTINGS index_granularity = 8192;
|
||||
95
deploy/clickhouse/install_clickhouse_linux.sh
Normal file
95
deploy/clickhouse/install_clickhouse_linux.sh
Normal file
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "[ERROR] please run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f /etc/os-release ]]; then
|
||||
echo "[ERROR] /etc/os-release not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
source /etc/os-release
|
||||
os_id="$(echo "${ID:-}" | tr '[:upper:]' '[:lower:]')"
|
||||
os_ver="${VERSION_ID:-}"
|
||||
is_ubuntu22=false
|
||||
is_amzn2023=false
|
||||
|
||||
if [[ "${os_id}" == "ubuntu" && "${os_ver}" == 22.04* ]]; then
|
||||
is_ubuntu22=true
|
||||
fi
|
||||
if [[ "${os_id}" == "amzn" && "${os_ver}" == 2023* ]]; then
|
||||
is_amzn2023=true
|
||||
fi
|
||||
|
||||
if [[ "${is_ubuntu22}" != "true" && "${is_amzn2023}" != "true" ]]; then
|
||||
echo "[ERROR] only Ubuntu 22.04 or Amazon Linux 2023 is supported. current: ID=${ID:-unknown}, VERSION_ID=${VERSION_ID:-unknown}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${is_ubuntu22}" == "true" ]]; then
|
||||
echo "[INFO] detected Ubuntu 22.04"
|
||||
echo "[INFO] installing prerequisites ..."
|
||||
apt-get update -y
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y curl ca-certificates gnupg apt-transport-https lsb-release
|
||||
|
||||
echo "[INFO] configuring ClickHouse apt repository ..."
|
||||
install -d -m 0755 /etc/apt/keyrings
|
||||
if [[ ! -f /etc/apt/keyrings/clickhouse.gpg ]]; then
|
||||
curl -fsSL https://packages.clickhouse.com/CLICKHOUSE-KEY.GPG | gpg --dearmor -o /etc/apt/keyrings/clickhouse.gpg
|
||||
fi
|
||||
|
||||
cat >/etc/apt/sources.list.d/clickhouse.list <<'EOF'
|
||||
deb [signed-by=/etc/apt/keyrings/clickhouse.gpg arch=amd64,arm64] https://packages.clickhouse.com/deb stable main
|
||||
EOF
|
||||
|
||||
echo "[INFO] installing clickhouse-server and clickhouse-client ..."
|
||||
apt-get update -y
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -y clickhouse-server clickhouse-client clickhouse-common-static
|
||||
fi
|
||||
|
||||
if [[ "${is_amzn2023}" == "true" ]]; then
|
||||
echo "[INFO] detected Amazon Linux 2023"
|
||||
echo "[INFO] installing prerequisites ..."
|
||||
dnf makecache -y
|
||||
dnf install -y curl ca-certificates gnupg2 dnf-plugins-core
|
||||
|
||||
echo "[INFO] configuring ClickHouse yum repository ..."
|
||||
cat >/etc/yum.repos.d/clickhouse.repo <<'EOF'
|
||||
[clickhouse-stable]
|
||||
name=ClickHouse Stable Repository
|
||||
baseurl=https://packages.clickhouse.com/rpm/stable/$basearch
|
||||
enabled=1
|
||||
gpgcheck=1
|
||||
gpgkey=https://packages.clickhouse.com/rpm/stable/repodata/repomd.xml.key
|
||||
https://packages.clickhouse.com/rpm/clickhouse-static.key
|
||||
EOF
|
||||
|
||||
echo "[INFO] installing clickhouse-server and clickhouse-client ..."
|
||||
dnf clean all
|
||||
dnf makecache -y
|
||||
if ! dnf install -y clickhouse-server clickhouse-client clickhouse-common-static; then
|
||||
dnf install -y clickhouse-server clickhouse-client
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "[INFO] enabling clickhouse-server ..."
|
||||
systemctl enable clickhouse-server >/dev/null 2>&1 || true
|
||||
systemctl restart clickhouse-server
|
||||
sleep 2
|
||||
|
||||
if [[ -n "${CLICKHOUSE_DEFAULT_PASSWORD:-}" ]]; then
|
||||
echo "[INFO] setting default user password ..."
|
||||
if [[ "${CLICKHOUSE_DEFAULT_PASSWORD}" == *"'"* ]]; then
|
||||
echo "[ERROR] CLICKHOUSE_DEFAULT_PASSWORD contains single quote, please set password manually with clickhouse-client"
|
||||
exit 1
|
||||
fi
|
||||
clickhouse-client --query "ALTER USER default IDENTIFIED WITH plaintext_password BY '${CLICKHOUSE_DEFAULT_PASSWORD}'"
|
||||
fi
|
||||
|
||||
echo "[INFO] health check ..."
|
||||
clickhouse-client --query "SELECT version()"
|
||||
echo "[OK] ClickHouse install completed: ID=${ID:-unknown}, VERSION_ID=${VERSION_ID:-unknown}"
|
||||
108
deploy/clickhouse/setup_clickhouse.sh
Normal file
108
deploy/clickhouse/setup_clickhouse.sh
Normal file
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
INSTALL_SCRIPT="${SCRIPT_DIR}/install_clickhouse_linux.sh"
|
||||
HTTPS_SCRIPT="${SCRIPT_DIR}/configure_clickhouse_https.sh"
|
||||
RUNTIME_SCRIPT="${SCRIPT_DIR}/configure_clickhouse_runtime.sh"
|
||||
TABLES_SCRIPT="${SCRIPT_DIR}/init_waf_logs_tables.sh"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage:
|
||||
sudo ./setup_clickhouse.sh [all|install|https|runtime|tables]
|
||||
|
||||
Modes:
|
||||
all Install ClickHouse, configure HTTPS, apply runtime config, init ingest tables (default)
|
||||
install Install ClickHouse only
|
||||
https Configure HTTPS only
|
||||
runtime Apply ClickHouse runtime config only
|
||||
tables Initialize ingest tables only
|
||||
|
||||
Common env vars:
|
||||
CLICKHOUSE_DEFAULT_PASSWORD Default user password set during install
|
||||
CH_HTTPS_PORT HTTPS port (default: 8443)
|
||||
CH_CERT_CN Certificate CN
|
||||
CH_CERT_DNS Certificate SAN DNS list (comma-separated)
|
||||
CH_CERT_IP Certificate SAN IP list (comma-separated)
|
||||
CH_CERT_DAYS Certificate validity days (default: 825)
|
||||
CH_LOG_LEVEL ClickHouse logger level (default: warning)
|
||||
CH_HOST ClickHouse host for table init (default: 127.0.0.1)
|
||||
CH_PORT ClickHouse port for table init (default: 9000)
|
||||
CH_USER ClickHouse user for table init (default: default)
|
||||
CH_PASSWORD ClickHouse password for table init
|
||||
CH_DATABASE Database for table init (default: default)
|
||||
EOF
|
||||
}
|
||||
|
||||
require_script() {
|
||||
local script="$1"
|
||||
if [[ ! -f "${script}" ]]; then
|
||||
echo "[ERROR] required file not found: ${script}"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_install() {
|
||||
echo "[INFO] step 1/3: install ClickHouse ..."
|
||||
bash "${INSTALL_SCRIPT}"
|
||||
}
|
||||
|
||||
run_https() {
|
||||
echo "[INFO] step 2/3: configure ClickHouse HTTPS ..."
|
||||
bash "${HTTPS_SCRIPT}"
|
||||
}
|
||||
|
||||
run_runtime() {
|
||||
echo "[INFO] step 3/4: apply ClickHouse runtime config ..."
|
||||
bash "${RUNTIME_SCRIPT}"
|
||||
}
|
||||
|
||||
run_tables() {
|
||||
echo "[INFO] step 4/4: initialize ingest tables ..."
|
||||
bash "${TABLES_SCRIPT}"
|
||||
}
|
||||
|
||||
MODE="${1:-all}"
|
||||
|
||||
case "${MODE}" in
|
||||
-h|--help|help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
all|install|https|runtime|tables)
|
||||
;;
|
||||
*)
|
||||
echo "[ERROR] invalid mode: ${MODE}"
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
require_script "${INSTALL_SCRIPT}"
|
||||
require_script "${HTTPS_SCRIPT}"
|
||||
require_script "${RUNTIME_SCRIPT}"
|
||||
require_script "${TABLES_SCRIPT}"
|
||||
|
||||
case "${MODE}" in
|
||||
all)
|
||||
run_install
|
||||
run_https
|
||||
run_runtime
|
||||
run_tables
|
||||
;;
|
||||
install)
|
||||
run_install
|
||||
;;
|
||||
https)
|
||||
run_https
|
||||
;;
|
||||
runtime)
|
||||
run_runtime
|
||||
;;
|
||||
tables)
|
||||
run_tables
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "[OK] setup completed: mode=${MODE}"
|
||||
348
deploy/deploy.sh
Normal file
348
deploy/deploy.sh
Normal file
@@ -0,0 +1,348 @@
|
||||
#!/bin/bash
|
||||
# CloudWAF 部署脚本 - Gname DNS 提供商更新
|
||||
# 使用方法: ./deploy.sh [zip文件路径]
|
||||
|
||||
set -e # 遇到错误立即退出
|
||||
|
||||
# ========== 配置区域 ==========
|
||||
# 支持环境变量和配置文件
|
||||
# 从环境变量读取
|
||||
INSTALL_BASE_DIR="${BRAND_INSTALL_PATH:-/usr/local/goedge}"
|
||||
|
||||
# 从配置文件读取(如果存在)
|
||||
if [ -f "/etc/goedge/brand.conf" ]; then
|
||||
source /etc/goedge/brand.conf
|
||||
if [ -n "$BRAND_INSTALL_PATH" ]; then
|
||||
INSTALL_BASE_DIR="$BRAND_INSTALL_PATH"
|
||||
fi
|
||||
fi
|
||||
|
||||
# 如果通过参数传入 zip 文件路径,则使用参数;否则使用默认路径
|
||||
ZIP_FILE_RAW="${1:-/tmp/edge-admin-linux-amd64-plus-v1.3.8.zip}"
|
||||
# 获取绝对路径,防止 cd 到临时目录后找不到文件
|
||||
if [[ "$ZIP_FILE_RAW" = /* ]]; then
|
||||
ZIP_FILE="$ZIP_FILE_RAW"
|
||||
else
|
||||
ZIP_FILE="$(pwd)/$ZIP_FILE_RAW"
|
||||
fi
|
||||
|
||||
TARGET_DIR="${INSTALL_BASE_DIR}/edge-admin"
|
||||
BACKUP_DIR="${INSTALL_BASE_DIR}/backup_$(date +%Y%m%d_%H%M%S)"
|
||||
TEMP_DIR="/tmp/edge-admin-update-$(date +%s)"
|
||||
|
||||
# 颜色输出
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# 日志函数
|
||||
log_info() {
|
||||
echo -e "${GREEN}[INFO]${NC} $1"
|
||||
}
|
||||
|
||||
log_warn() {
|
||||
echo -e "${YELLOW}[WARN]${NC} $1"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo -e "${RED}[ERROR]${NC} $1"
|
||||
}
|
||||
|
||||
# ========== 检查前置条件 ==========
|
||||
check_prerequisites() {
|
||||
log_info "检查前置条件..."
|
||||
|
||||
# 检查 zip 文件是否存在
|
||||
if [ ! -f "$ZIP_FILE" ]; then
|
||||
log_error "未找到更新包: $ZIP_FILE"
|
||||
log_info "使用方法: $0 [zip文件路径]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查目标目录是否存在
|
||||
if [ ! -d "$TARGET_DIR" ]; then
|
||||
log_error "目标目录不存在: $TARGET_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查是否有足够的磁盘空间(至少 500MB)
|
||||
available_space=$(df "$TARGET_DIR" | tail -1 | awk '{print $4}')
|
||||
if [ "$available_space" -lt 512000 ]; then
|
||||
log_warn "可用磁盘空间不足 500MB,可能影响部署"
|
||||
fi
|
||||
|
||||
log_info "前置条件检查通过"
|
||||
}
|
||||
|
||||
# ========== 1. 备份关键目录 ==========
|
||||
backup_files() {
|
||||
log_info "=========================================="
|
||||
log_info "步骤 1: 备份关键目录"
|
||||
log_info "=========================================="
|
||||
|
||||
mkdir -p "$BACKUP_DIR"
|
||||
|
||||
# 备份 EdgeAdmin 配置
|
||||
if [ -d "$TARGET_DIR/configs" ]; then
|
||||
log_info "备份 EdgeAdmin 配置文件..."
|
||||
cp -r "$TARGET_DIR/configs" "$BACKUP_DIR/edge-admin-configs"
|
||||
log_info "✅ EdgeAdmin 配置已备份"
|
||||
else
|
||||
log_warn "EdgeAdmin configs 目录不存在,跳过备份"
|
||||
fi
|
||||
|
||||
# 备份 EdgeAdmin 日志(可选)
|
||||
if [ -d "$TARGET_DIR/logs" ]; then
|
||||
log_info "备份 EdgeAdmin 日志文件..."
|
||||
cp -r "$TARGET_DIR/logs" "$BACKUP_DIR/edge-admin-logs" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# 备份 EdgeAPI 配置和数据
|
||||
if [ -d "$TARGET_DIR/edge-api" ]; then
|
||||
if [ -d "$TARGET_DIR/edge-api/configs" ]; then
|
||||
log_info "备份 EdgeAPI 配置文件..."
|
||||
cp -r "$TARGET_DIR/edge-api/configs" "$BACKUP_DIR/edge-api-configs"
|
||||
log_info "✅ EdgeAPI 配置已备份"
|
||||
fi
|
||||
|
||||
if [ -d "$TARGET_DIR/edge-api/logs" ]; then
|
||||
log_info "备份 EdgeAPI 日志文件..."
|
||||
cp -r "$TARGET_DIR/edge-api/logs" "$BACKUP_DIR/edge-api-logs" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
if [ -d "$TARGET_DIR/edge-api/data" ]; then
|
||||
log_info "备份 EdgeAPI 数据文件..."
|
||||
cp -r "$TARGET_DIR/edge-api/data" "$BACKUP_DIR/edge-api-data" 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
|
||||
log_info "✅ 备份完成,备份位置: $BACKUP_DIR"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ========== 2. 停止服务 ==========
|
||||
stop_services() {
|
||||
log_info "=========================================="
|
||||
log_info "步骤 2: 停止服务"
|
||||
log_info "=========================================="
|
||||
|
||||
if [ -f "$TARGET_DIR/bin/edge-admin" ]; then
|
||||
log_info "停止 EdgeAdmin 服务..."
|
||||
"$TARGET_DIR/bin/edge-admin" stop || true
|
||||
sleep 2
|
||||
|
||||
# 检查是否还有进程在运行 (精确匹配二进制路径,避免杀死解析 zip 路径的脚本自身)
|
||||
if pgrep -f "$TARGET_DIR/bin/edge-admin" > /dev/null; then
|
||||
log_warn "检测到 edge-admin 进程仍在运行,尝试强制停止..."
|
||||
pkill -9 -f "$TARGET_DIR/bin/edge-admin" || true
|
||||
sleep 1
|
||||
fi
|
||||
|
||||
log_info "✅ 服务已停止"
|
||||
else
|
||||
log_warn "未找到 edge-admin 可执行文件,跳过停止步骤"
|
||||
fi
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ========== 3. 解压新版本 ==========
|
||||
extract_package() {
|
||||
log_info "=========================================="
|
||||
log_info "步骤 3: 解压新版本"
|
||||
log_info "=========================================="
|
||||
|
||||
log_info "解压更新包到临时目录: $TEMP_DIR"
|
||||
mkdir -p "$TEMP_DIR"
|
||||
cd "$TEMP_DIR"
|
||||
|
||||
if ! unzip -q "$ZIP_FILE"; then
|
||||
log_error "解压失败,请检查 zip 文件是否损坏"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log_info "✅ 解压完成"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ========== 4. 替换文件 ==========
|
||||
replace_files() {
|
||||
log_info "=========================================="
|
||||
log_info "步骤 4: 替换文件"
|
||||
log_info "=========================================="
|
||||
|
||||
# 替换 EdgeAdmin bin
|
||||
if [ -d "$TEMP_DIR/edge-admin/bin" ]; then
|
||||
log_info "替换 EdgeAdmin 可执行文件..."
|
||||
cp -r "$TEMP_DIR/edge-admin/bin"/* "$TARGET_DIR/bin/"
|
||||
log_info "✅ EdgeAdmin bin 已更新"
|
||||
fi
|
||||
|
||||
# 替换 EdgeAdmin web(排除 tmp)
|
||||
if [ -d "$TEMP_DIR/edge-admin/web" ]; then
|
||||
log_info "替换 EdgeAdmin 前端文件..."
|
||||
if command -v rsync > /dev/null; then
|
||||
rsync -av --exclude='tmp' \
|
||||
"$TEMP_DIR/edge-admin/web/" "$TARGET_DIR/web/"
|
||||
else
|
||||
# 如果没有 rsync,使用 cp
|
||||
cp -r "$TEMP_DIR/edge-admin/web"/* "$TARGET_DIR/web/" 2>/dev/null || true
|
||||
rm -rf "$TARGET_DIR/web/tmp"/* 2>/dev/null || true
|
||||
fi
|
||||
# 清空 tmp 目录
|
||||
rm -rf "$TARGET_DIR/web/tmp"/* 2>/dev/null || true
|
||||
log_info "✅ EdgeAdmin web 已更新"
|
||||
fi
|
||||
|
||||
# 替换 EdgeAPI 文件
|
||||
if [ -d "$TEMP_DIR/edge-admin/edge-api" ]; then
|
||||
log_info "替换 EdgeAPI 文件..."
|
||||
|
||||
# 确保 edge-api 目录存在
|
||||
mkdir -p "$TARGET_DIR/edge-api"
|
||||
|
||||
# 替换 bin
|
||||
if [ -d "$TEMP_DIR/edge-admin/edge-api/bin" ]; then
|
||||
mkdir -p "$TARGET_DIR/edge-api/bin"
|
||||
cp -r "$TEMP_DIR/edge-admin/edge-api/bin"/* \
|
||||
"$TARGET_DIR/edge-api/bin/" 2>/dev/null || true
|
||||
log_info "✅ EdgeAPI bin 已更新"
|
||||
fi
|
||||
|
||||
# 替换 deploy(节点安装包)
|
||||
if [ -d "$TEMP_DIR/edge-admin/edge-api/deploy" ]; then
|
||||
mkdir -p "$TARGET_DIR/edge-api/deploy"
|
||||
cp -r "$TEMP_DIR/edge-admin/edge-api/deploy"/* \
|
||||
"$TARGET_DIR/edge-api/deploy/" 2>/dev/null || true
|
||||
log_info "✅ EdgeAPI deploy 已更新"
|
||||
fi
|
||||
|
||||
# 替换 installers(安装工具)
|
||||
if [ -d "$TEMP_DIR/edge-admin/edge-api/installers" ]; then
|
||||
mkdir -p "$TARGET_DIR/edge-api/installers"
|
||||
cp -r "$TEMP_DIR/edge-admin/edge-api/installers"/* \
|
||||
"$TARGET_DIR/edge-api/installers/" 2>/dev/null || true
|
||||
log_info "✅ EdgeAPI installers 已更新"
|
||||
fi
|
||||
fi
|
||||
|
||||
log_info "✅ 文件替换完成"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ========== 5. 恢复配置文件 ==========
|
||||
restore_configs() {
|
||||
log_info "=========================================="
|
||||
log_info "步骤 5: 恢复配置文件"
|
||||
log_info "=========================================="
|
||||
|
||||
# 恢复 EdgeAdmin 配置
|
||||
if [ -d "$BACKUP_DIR/edge-admin-configs" ]; then
|
||||
log_info "恢复 EdgeAdmin 配置文件..."
|
||||
cp -r "$BACKUP_DIR/edge-admin-configs"/* "$TARGET_DIR/configs/"
|
||||
log_info "✅ EdgeAdmin 配置已恢复"
|
||||
else
|
||||
log_warn "未找到 EdgeAdmin 配置备份,请手动检查配置文件"
|
||||
fi
|
||||
|
||||
# 恢复 EdgeAPI 配置
|
||||
if [ -d "$BACKUP_DIR/edge-api-configs" ]; then
|
||||
log_info "恢复 EdgeAPI 配置文件..."
|
||||
mkdir -p "$TARGET_DIR/edge-api/configs"
|
||||
cp -r "$BACKUP_DIR/edge-api-configs"/* "$TARGET_DIR/edge-api/configs/" 2>/dev/null || true
|
||||
log_info "✅ EdgeAPI 配置已恢复"
|
||||
else
|
||||
log_warn "未找到 EdgeAPI 配置备份,如果存在 edge-api 目录,请手动检查配置文件"
|
||||
fi
|
||||
|
||||
log_info "✅ 配置文件恢复完成"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ========== 6. 清理临时文件 ==========
|
||||
cleanup() {
|
||||
log_info "=========================================="
|
||||
log_info "步骤 6: 清理临时文件"
|
||||
log_info "=========================================="
|
||||
|
||||
rm -rf "$TEMP_DIR"
|
||||
log_info "✅ 临时文件已清理"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ========== 7. 启动服务 ==========
|
||||
start_services() {
|
||||
log_info "=========================================="
|
||||
log_info "步骤 7: 启动服务"
|
||||
log_info "=========================================="
|
||||
|
||||
if [ -f "$TARGET_DIR/bin/edge-admin" ]; then
|
||||
log_info "启动 EdgeAdmin 服务..."
|
||||
"$TARGET_DIR/bin/edge-admin" start
|
||||
sleep 3
|
||||
|
||||
# 检查服务状态
|
||||
if pgrep -f "$TARGET_DIR/bin/edge-admin" > /dev/null; then
|
||||
log_info "✅ 服务启动成功"
|
||||
else
|
||||
log_warn "服务可能未正常启动,请检查日志"
|
||||
log_info "查看日志: tail -f $TARGET_DIR/logs/run.log"
|
||||
fi
|
||||
else
|
||||
log_error "未找到 edge-admin 可执行文件"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
}
|
||||
|
||||
# ========== 主函数 ==========
|
||||
main() {
|
||||
echo ""
|
||||
log_info "=========================================="
|
||||
log_info "CloudWAF 部署脚本 - Gname DNS 提供商更新"
|
||||
log_info "=========================================="
|
||||
echo ""
|
||||
|
||||
log_info "部署配置:"
|
||||
log_info " ZIP 文件: $ZIP_FILE"
|
||||
log_info " 目标目录: $TARGET_DIR"
|
||||
log_info " 备份目录: $BACKUP_DIR"
|
||||
echo ""
|
||||
|
||||
# 确认操作
|
||||
read -p "确认开始部署? (y/N): " -n 1 -r
|
||||
echo
|
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
|
||||
log_info "部署已取消"
|
||||
exit 0
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 执行部署步骤
|
||||
check_prerequisites
|
||||
backup_files
|
||||
stop_services
|
||||
extract_package
|
||||
replace_files
|
||||
restore_configs
|
||||
cleanup
|
||||
start_services
|
||||
|
||||
# 完成
|
||||
log_info "=========================================="
|
||||
log_info "✅ 部署完成!"
|
||||
log_info "=========================================="
|
||||
echo ""
|
||||
log_info "备份位置: $BACKUP_DIR"
|
||||
log_info "请检查服务运行状态和日志"
|
||||
echo ""
|
||||
log_info "验证步骤:"
|
||||
log_info "1. 检查服务进程: ps aux | grep \"$TARGET_DIR/bin/edge-admin\""
|
||||
log_info "2. 检查服务日志: tail -f $TARGET_DIR/logs/run.log"
|
||||
log_info "3. 访问管理后台,测试 Gname DNS 提供商功能"
|
||||
echo ""
|
||||
}
|
||||
|
||||
# 执行主函数
|
||||
main
|
||||
|
||||
11
deploy/fluent-bit/clickhouse-upstream.conf
Normal file
11
deploy/fluent-bit/clickhouse-upstream.conf
Normal file
@@ -0,0 +1,11 @@
|
||||
# ClickHouse 上游配置(单机或集群只改此文件)
|
||||
# 单机:保留一个 NODE;集群:按需增加 NODE
|
||||
# 有密码时:不在此文件配置,通过环境变量 CH_USER、CH_PASSWORD 在 fluent-bit.conf 的 OUTPUT 中生效。
|
||||
|
||||
[UPSTREAM]
|
||||
Name ch_backends
|
||||
|
||||
[NODE]
|
||||
Name node-01
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
39
deploy/fluent-bit/fluent-bit-dns-https.conf
Normal file
39
deploy/fluent-bit/fluent-bit-dns-https.conf
Normal file
@@ -0,0 +1,39 @@
|
||||
# DNS 节点专用 HTTPS:使用 HTTP 输出写入 ClickHouse(无需 out_clickhouse 插件)
|
||||
# 启动前设置:CH_USER、CH_PASSWORD;按需修改 Host、Port(默认 127.0.0.1:8443)
|
||||
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
storage.path /var/lib/fluent-bit/storage
|
||||
storage.sync normal
|
||||
storage.checksum off
|
||||
storage.backlog.mem_limit 128MB
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-dns/*.log
|
||||
Tag app.dns.logs
|
||||
Parser json
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/dns-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.dns.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT%20INTO%20default.dns_logs_ingest%20FORMAT%20JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file /etc/ssl/certs/ca-certificates.crt
|
||||
# tls.vhost clickhouse.example.com
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
40
deploy/fluent-bit/fluent-bit-dns.conf
Normal file
40
deploy/fluent-bit/fluent-bit-dns.conf
Normal file
@@ -0,0 +1,40 @@
|
||||
# DNS 节点专用:使用 HTTPS 输出写入 ClickHouse(无需 out_clickhouse 插件)
|
||||
# 启动前设置:CH_USER、CH_PASSWORD;若 ClickHouse 不在本机,请修改 Host、Port
|
||||
# Read_from_Head=true:首次启动会发送已有日志;若只采新日志建议改为 false
|
||||
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
storage.path /var/lib/fluent-bit/storage
|
||||
storage.sync normal
|
||||
storage.checksum off
|
||||
storage.backlog.mem_limit 128MB
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-dns/*.log
|
||||
Tag app.dns.logs
|
||||
Parser json
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/dns-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.dns.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT%20INTO%20default.dns_logs_ingest%20FORMAT%20JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file /etc/ssl/certs/ca-certificates.crt
|
||||
# tls.vhost clickhouse.example.com
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
74
deploy/fluent-bit/fluent-bit-https.conf
Normal file
74
deploy/fluent-bit/fluent-bit-https.conf
Normal file
@@ -0,0 +1,74 @@
|
||||
# Fluent Bit HTTPS 配置(边缘节点日志采集 -> ClickHouse HTTPS)
|
||||
# HTTP: /var/log/edge/edge-node/*.log
|
||||
# DNS: /var/log/edge/edge-dns/*.log
|
||||
#
|
||||
# 启动前请设置环境变量:
|
||||
# CH_USER=default
|
||||
# CH_PASSWORD=your_password
|
||||
# 如需改地址/端口,请修改 OUTPUT 中 Host/Port(默认 127.0.0.1:8443)
|
||||
# 如证书为公网CA可省略 tls.ca_file;自签名证书请配置 tls.ca_file
|
||||
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
storage.path /var/lib/fluent-bit/storage
|
||||
storage.sync normal
|
||||
storage.checksum off
|
||||
storage.backlog.mem_limit 128MB
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-node/*.log
|
||||
Tag app.http.logs
|
||||
Parser json
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/http-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-dns/*.log
|
||||
Tag app.dns.logs
|
||||
Parser json
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/dns-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.http.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT%20INTO%20default.logs_ingest%20FORMAT%20JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file /etc/ssl/certs/ca-certificates.crt
|
||||
# tls.vhost clickhouse.example.com
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
|
||||
[OUTPUT]
|
||||
Name http
|
||||
Match app.dns.logs
|
||||
Host 127.0.0.1
|
||||
Port 8443
|
||||
URI /?query=INSERT%20INTO%20default.dns_logs_ingest%20FORMAT%20JSONEachRow
|
||||
Format json_lines
|
||||
http_user ${CH_USER}
|
||||
http_passwd ${CH_PASSWORD}
|
||||
tls On
|
||||
tls.verify On
|
||||
# tls.ca_file /etc/ssl/certs/ca-certificates.crt
|
||||
# tls.vhost clickhouse.example.com
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
56
deploy/fluent-bit/fluent-bit.conf
Normal file
56
deploy/fluent-bit/fluent-bit.conf
Normal file
@@ -0,0 +1,56 @@
|
||||
# Fluent Bit 主配置(边缘节点日志采集 → ClickHouse)
|
||||
# HTTP: /var/log/edge/edge-node/*.log
|
||||
# DNS: /var/log/edge/edge-dns/*.log
|
||||
|
||||
[SERVICE]
|
||||
Flush 5
|
||||
Log_Level info
|
||||
Parsers_File parsers.conf
|
||||
storage.path /var/lib/fluent-bit/storage
|
||||
storage.sync normal
|
||||
storage.checksum off
|
||||
storage.backlog.mem_limit 128MB
|
||||
|
||||
@INCLUDE clickhouse-upstream.conf
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-node/*.log
|
||||
Tag app.http.logs
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/http-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[INPUT]
|
||||
Name tail
|
||||
Path /var/log/edge/edge-dns/*.log
|
||||
Tag app.dns.logs
|
||||
Refresh_Interval 5
|
||||
Read_from_Head false
|
||||
DB /var/lib/fluent-bit/dns-logs.db
|
||||
Mem_Buf_Limit 128MB
|
||||
Skip_Long_Lines On
|
||||
|
||||
[OUTPUT]
|
||||
Name clickhouse
|
||||
Match app.http.logs
|
||||
Upstream ch_backends
|
||||
Table logs_ingest
|
||||
Http_User ${CH_USER}
|
||||
Http_Passwd ${CH_PASSWORD}
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
|
||||
[OUTPUT]
|
||||
Name clickhouse
|
||||
Match app.dns.logs
|
||||
Upstream ch_backends
|
||||
Table dns_logs_ingest
|
||||
Http_User ${CH_USER}
|
||||
Http_Passwd ${CH_PASSWORD}
|
||||
json_date_key timestamp
|
||||
json_date_format epoch
|
||||
Retry_Limit 10
|
||||
9
deploy/fluent-bit/parsers.conf
Normal file
9
deploy/fluent-bit/parsers.conf
Normal file
@@ -0,0 +1,9 @@
|
||||
# Fluent Bit 解析器(与主配置 Parsers_File 对应)
|
||||
# 边缘节点日志为 JSON Lines,使用 json 解析器即可
|
||||
|
||||
[PARSER]
|
||||
Name json
|
||||
Format json
|
||||
Time_Key timestamp
|
||||
Time_Format %s
|
||||
Time_Keep On
|
||||
141
deploy/升级部署手册.md
Normal file
141
deploy/升级部署手册.md
Normal file
@@ -0,0 +1,141 @@
|
||||
# CloudWAF 升级部署手册
|
||||
|
||||
## I. 管理端 (EdgeAdmin) 升级
|
||||
|
||||
本文档介绍如何使用 `deploy.sh` 自动化脚本对 EdgeAdmin 服务进行升级部署。
|
||||
|
||||
### 1. 准备工作
|
||||
|
||||
在开始升级之前,请确保已满足以下条件:
|
||||
|
||||
1. **文件准备**
|
||||
- `deploy.sh`: 部署脚本
|
||||
- `edge-admin-linux-amd64-plus-vX.X.X.zip`: 对应版本的 EdgeAdmin 更新包
|
||||
|
||||
2. **上传文件**
|
||||
将上述两个文件上传到服务器的同一目录下(例如 `/root` 或 `/tmp`)。
|
||||
|
||||
3. **赋予脚本执行权限**
|
||||
在终端中执行命令:
|
||||
```bash
|
||||
chmod +x deploy.sh
|
||||
```
|
||||
|
||||
### 2. 确认安装路径
|
||||
|
||||
脚本默认的安装路径为 `/usr/local/goedge`。如果您的服务安装在其他位置,请按以下方法修改:
|
||||
|
||||
**方法 A:修改脚本(推荐)**
|
||||
使用文本编辑器(如 vi/nano)打开 `deploy.sh`,修改第 10 行:
|
||||
```bash
|
||||
# 将默认路径修改为您的实际安装路径
|
||||
INSTALL_BASE_DIR="/您实际的/安装路径"
|
||||
```
|
||||
|
||||
**方法 B:使用环境变量(临时)**
|
||||
在执行脚本时指定环境变量:
|
||||
```bash
|
||||
export BRAND_INSTALL_PATH="/您实际的/安装路径"
|
||||
```
|
||||
|
||||
### 3. 执行升级
|
||||
|
||||
使用 root 用户或 sudo 执行脚本,参数为更新包的路径。
|
||||
|
||||
#### 3.1 运行命令
|
||||
```bash
|
||||
# 示例:更新当前目录下的 zip 包
|
||||
sudo ./deploy.sh ./edge-admin-linux-amd64-plus-v1.3.8.zip
|
||||
```
|
||||
|
||||
#### 3.2 确认升级
|
||||
脚本运行后会显示升级信息预览,请检查路径是否正确:
|
||||
```text
|
||||
CloudWAF 部署脚本 - Gname DNS 提供商更新
|
||||
==========================================
|
||||
|
||||
部署配置:
|
||||
ZIP 文件: ...
|
||||
目标目录: /usr/local/goedge/edge-admin
|
||||
备份目录: /usr/local/goedge/backup_20240215_120000
|
||||
|
||||
确认开始部署? (y/N):
|
||||
```
|
||||
输入 `y` 并回车确认。
|
||||
|
||||
### 4. 升级过程说明
|
||||
|
||||
脚本会自动执行以下步骤:
|
||||
1. **备份**:将当前的配置 (`configs/`) 和数据备份到 `backup_YYYYMMDD_HHMMSS` 目录。
|
||||
2. **停止服务**:停止正在运行的 `edge-admin` 进程。
|
||||
3. **解压**:解压新的安装包到临时目录。
|
||||
4. **替换**:
|
||||
- 覆盖 `bin/edge-admin` 可执行文件。
|
||||
- 覆盖 `web/` 前端文件(保留 `web/tmp`)。
|
||||
- 覆盖 `edge-api/` 相关文件(如果存在)。
|
||||
5. **恢复配置**:将备份的 `configs/api_admin.yaml` 等配置文件恢复到新目录,**确保原有配置不丢**。
|
||||
6. **启动服务**:重新启动 `edge-admin` 服务。
|
||||
|
||||
### 5. 验证与故障排查
|
||||
|
||||
升级完成后,请进行以下检查:
|
||||
- **检查进程**:`ps aux | grep edge-admin`
|
||||
- **查看日志**:`tail -f /安装路径/edge-admin/logs/run.log`
|
||||
- **登录验证**:使用浏览器访问管理后台,确认升级成功。
|
||||
|
||||
---
|
||||
|
||||
## II. 用户端 (EdgeUser) 升级
|
||||
|
||||
EdgeUser (用户控制台) 尚未提供自动化脚本,请按照以下步骤手动升级。
|
||||
|
||||
### ⚠️ 重要提示:关于自定义样式
|
||||
如果您修改过 `web/` 目录下的 HTML/CSS (例如登录页),请务必在升级前**手动备份这些文件**。
|
||||
升级过程中的 `web` 目录覆盖会导致您的自定义修改丢失。
|
||||
|
||||
### 1. 备份 (Backup)
|
||||
```bash
|
||||
# 假设安装目录在 /opt/edge-user
|
||||
cp -r /opt/edge-user /opt/edge-user-backup-$(date +%Y%m%d)
|
||||
```
|
||||
*关键文件是 `configs/api_user.yaml`,这是连接 API 的凭证,**千万不要丢失**。*
|
||||
|
||||
### 2. 上传与准备 (Prepare)
|
||||
将 `edge-user-linux-amd64-plus-vX.X.X.zip` 上传到服务器并解压到临时目录。
|
||||
```bash
|
||||
unzip edge-user-linux-amd64-plus-v1.3.8.zip -d /tmp/edge-user-update
|
||||
```
|
||||
|
||||
### 3. 停止服务 (Stop)
|
||||
```bash
|
||||
/opt/edge-user/bin/edge-user stop
|
||||
```
|
||||
|
||||
### 4. 替换文件 (Replace)
|
||||
|
||||
执行以下命令替换程序文件:
|
||||
|
||||
```bash
|
||||
# 1. 替换二进制文件
|
||||
cp -f /tmp/edge-user-update/edge-user/bin/edge-user /opt/edge-user/bin/
|
||||
|
||||
# 2. 替换前端资源 (Web)
|
||||
# [警告] 此操作会重置所有页面样式
|
||||
cp -rf /tmp/edge-user-update/edge-user/web /opt/edge-user/
|
||||
|
||||
# 3. 恢复自定义样式 (如果有备份)
|
||||
# cp /备份路径/login/index.html /opt/edge-user/web/views/@default/login/
|
||||
```
|
||||
|
||||
**注意:不要覆盖 `configs` 目录,以保留您的 `api_user.yaml` 配置。**
|
||||
|
||||
### 5. 重启服务 (Start)
|
||||
```bash
|
||||
/opt/edge-user/bin/edge-user start
|
||||
```
|
||||
|
||||
### 6. 验证 (Verify)
|
||||
查看日志确定启动成功,并访问用户平台页面。
|
||||
```bash
|
||||
tail -f /opt/edge-user/logs/run.log
|
||||
```
|
||||
Reference in New Issue
Block a user