robin/waf-platform
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Initial commit (code only without large binaries)

Browse Source
This commit is contained in:
robin
2026-02-15 18:58:44 +08:00
commit 35df75498f
9442 changed files with 1495866 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

197
deploy/clickhouse/README.md Normal file
View File

@@ -0,0 +1,197 @@
# ClickHouse + Fluent Bit 快速部署（Ubuntu 22.04 / Amazon Linux 2023）
## 1. 脚本说明
- `setup_clickhouse.sh`：一键入口（推荐），默认顺序执行 安装 ClickHouse -> 配置 HTTPS -> 应用运行参数 -> 初始化日志表。
- `install_clickhouse_linux.sh`：安装 `clickhouse-server`、`clickhouse-client`，并启动服务。
- `configure_clickhouse_https.sh`：生成自签名 `server.crt + server.key`，写入 HTTPS 配置并重启服务。
- `configure_clickhouse_runtime.sh`：默认将日志级别设为 `warning`，并禁用高开销系统日志表（`text_log`、`part_log`、`metric_log`、`asynchronous_metric_log`、`trace_log`）。
- `init_waf_logs_tables.sh`：执行建表脚本。
- `init_waf_logs_tables.sql`：`logs_ingest`、`dns_logs_ingest` 表结构定义。
进入脚本所在目录
```bash
cd /opt/waf-platform/deploy/clickhouse
chmod +x setup_clickhouse.sh
```
## 2. 一键部署
### 2.1 方式A：不设置 ClickHouse 密码（用户名固定 `default`）
```bash
sudo ./setup_clickhouse.sh
```
说明：
- ClickHouse 连接用户是 `default`
- 未设置密码时，后续平台连接密码留空
### 2.2 方式B：设置用户名/密码（示例使用 `default`）
```bash
sudo CH_USER='default' \
CH_PASSWORD='YourStrongPassword' \
CH_DATABASE='default' \
./setup_clickhouse.sh
```
说明：
- `CH_USER`/`CH_PASSWORD`：初始化日志表时用于连接 ClickHouse
- 如果你使用自定义用户，把 `CH_USER` 改为你的用户名，并保证该用户已有对应数据库权限
可选：单独应用运行参数（日志级别/系统日志表开关）：
```bash
sudo CH_LOG_LEVEL=warning ./setup_clickhouse.sh runtime
```
## 3. ClickHouse 安装后关键目录
- 配置目录：`/etc/clickhouse-server/`
- 客户端配置目录：`/etc/clickhouse-client/`
- 数据目录：`/var/lib/clickhouse/`
- 日志目录：`/var/log/clickhouse-server/`
- HTTPS 覆盖配置：`/etc/clickhouse-server/config.d/waf-https.xml`
- 运行参数覆盖配置：`/etc/clickhouse-server/config.d/waf-runtime.xml`
- HTTPS 证书和私钥：`/etc/clickhouse-server/server.crt`、`/etc/clickhouse-server/server.key`
- 证书生成中间文件目录：`/etc/clickhouse-server/pki/`
## 4. 管理平台配置（EdgeAdmin）
页面路径：
- 左侧菜单：`系统设置` -> `高级设置`
- 顶部标签：`日志数据库（ClickHouse）`
表单填写：
- `连接地址（Host）`：ClickHouse 地址（IP 或域名），如 `10.0.0.8` 或 `clickhouse.example.com`
- `协议（Scheme）`：`https`
- `端口（Port）`：`8443`
- `用户名（User）`：`default`（或你自定义的用户名）
- `密码（Password）`：对应用户密码
- `数据库（Database）`：`default`（或你初始化日志表时使用的库名）
提交顺序：
1. 点“测试连接”
2. 连接成功后点“保存”
## 5. Fluent Bit（两种方式）
### 5.1 跟随节点在线自动安装（推荐）
说明：
- Node / DNS 在线安装或升级时，平台会自动安装/升级 Fluent Bit 并下发配置。
- 默认由平台托管，不需要逐台手改配置文件。
安装后所在节点关键文件：
- `/etc/fluent-bit/fluent-bit.conf`：Fluent Bit 主配置（输入日志路径、输出 ClickHouse、性能参数）。
- `/etc/fluent-bit/parsers.conf`：日志解析器定义（当前主要使用 JSON parser）。
- `/etc/fluent-bit/.edge-managed.env`：平台下发的 ClickHouse 认证环境变量（`CH_USER`/`CH_PASSWORD`）。
- `/etc/fluent-bit/.edge-managed.json`：平台下发的元数据（角色、配置哈希、版本、更新时间）。
说明：
- 在线安装时，节点上的 `/etc/fluent-bit/fluent-bit.conf` 会被平台下发覆盖。
fluent-bit中ClickHouse 账号密码下发与更新逻辑：
- 下发来源：管理平台 -日志数据库（ClickHouse）中保存的账号密码。
- 落地文件：平台在线安装或升级时写入节点 `/etc/fluent-bit/.edge-managed.env`，内容为 `CH_USER`、`CH_PASSWORD`。
- 更新触发：当平台里的 ClickHouse 账号或密码变更后，需触发一次节点安装/升级任务以下发新凭证。
- 常见问题：只在 ClickHouse 侧改密码、未同步更新平台配置时，Fluent Bit 会出现认证失败（401/unauthorized）。
高配机器调优（当前默认按 4C8G 参数）：
- 当前默认参数：`Flush=1`、`storage.backlog.mem_limit=512MB`、`Mem_Buf_Limit=256MB`、`workers=2`。
- 机器升配后优先调这 4 个参数：
- `storage.backlog.mem_limit`：总缓冲上限（先增大，降低突发堆积丢日志风险）。
- `Mem_Buf_Limit`：每个 tail input 的内存缓冲（HTTP 与 DNS 两段都要改）。
- `workers`：输出并发写入线程数（HTTP 与 DNS 两段都要改）。
- `Flush`：刷盘/发送间隔（值越小越实时，CPU/网络开销更高）。
- 8C16G 参考值可按 `deploy/fluent-bit/fluent-bit-sample-8c16g.conf`：
- `storage.backlog.mem_limit=1024MB`
- `Mem_Buf_Limit=512MB`
- `workers=4`
- `Refresh_Interval=1`
- 修改方法：
1. 编辑 `EdgeAPI/internal/installers/fluent_bit.go` 的 `renderManagedConfig()`。
2. 按上面参数同步修改 Node/DNS 两段 `[INPUT]` 和 `[OUTPUT]`。
3. 重新发布 API 并触发节点安装/升级任务，下发新配置。
检查：
```bash
sudo systemctl status fluent-bit --no-pager
sudo cat /etc/fluent-bit/.edge-managed.json
sudo journalctl -u fluent-bit -n 100 --no-pager
```
### 5.2 手动安装（自动安装失败时）
说明：
- 适合节点在线自动安装 Fluent Bit 失败的场景。
- 采用在线安装方式，由你手动安装并维护配置。
步骤：
1. 在线安装 Fluent Bit。
Ubuntu 22.04：
```bash
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
sudo apt-get update -y
sudo apt-get install -y fluent-bit
```
AWS 2023：
```bash
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
sudo dnf makecache -y
sudo dnf install -y fluent-bit
```
2. 放置配置文件：
```bash
sudo mkdir -p /etc/fluent-bit
sudo cp /opt/waf-platform/deploy/fluent-bit/fluent-bit.conf /etc/fluent-bit/
sudo cp /opt/waf-platform/deploy/fluent-bit/clickhouse-upstream.conf /etc/fluent-bit/
sudo cp /opt/waf-platform/deploy/fluent-bit/parsers.conf /etc/fluent-bit/
```
3. 修改 `/etc/fluent-bit/clickhouse-upstream.conf` 的 ClickHouse `Host`、`Port`（如 `8443`）。
4. 配置认证环境变量（按需）：
```bash
sudo tee /etc/fluent-bit/fluent-bit.env >/dev/null <<'EOF'
CH_USER=default
CH_PASSWORD=YourStrongPassword
EOF
```
5. 让 systemd 读取环境变量：
```bash
sudo mkdir -p /etc/systemd/system/fluent-bit.service.d
sudo tee /etc/systemd/system/fluent-bit.service.d/override.conf >/dev/null <<'EOF'
[Service]
EnvironmentFile=/etc/fluent-bit/fluent-bit.env
EOF
```
6. 启动并检查：
```bash
sudo systemctl daemon-reload
sudo systemctl enable fluent-bit
sudo systemctl restart fluent-bit
sudo systemctl status fluent-bit --no-pager
sudo journalctl -u fluent-bit -n 100 --no-pager
```
## 6. 验证
```bash
sudo journalctl -u fluent-bit -f
```

158
deploy/clickhouse/configure_clickhouse_https.sh Normal file
View File

@@ -0,0 +1,158 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ "${EUID}" -ne 0 ]]; then
echo "[ERROR] please run as root"
exit 1
fi
if [[ ! -f /etc/os-release ]]; then
echo "[ERROR] /etc/os-release not found"
exit 1
fi
# shellcheck disable=SC1091
source /etc/os-release
os_id="$(echo "${ID:-}" | tr '[:upper:]' '[:lower:]')"
os_ver="${VERSION_ID:-}"
is_ubuntu22=false
is_amzn2023=false
if [[ "${os_id}" == "ubuntu" && "${os_ver}" == 22.04* ]]; then
is_ubuntu22=true
fi
if [[ "${os_id}" == "amzn" && "${os_ver}" == 2023* ]]; then
is_amzn2023=true
fi
if [[ "${is_ubuntu22}" != "true" && "${is_amzn2023}" != "true" ]]; then
echo "[ERROR] only Ubuntu 22.04 or Amazon Linux 2023 is supported. current: ID=${ID:-unknown}, VERSION_ID=${VERSION_ID:-unknown}"
exit 1
fi
if ! command -v openssl >/dev/null 2>&1 || ! command -v curl >/dev/null 2>&1; then
if [[ "${is_ubuntu22}" == "true" ]]; then
apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get install -y openssl curl ca-certificates
else
dnf makecache -y
dnf install -y openssl curl ca-certificates
fi
fi
CH_HTTPS_PORT="${CH_HTTPS_PORT:-8443}"
CH_LISTEN_HOST="${CH_LISTEN_HOST:-::}"
CH_CERT_CN="${CH_CERT_CN:-$(hostname -f 2>/dev/null || hostname)}"
CH_CERT_DNS="${CH_CERT_DNS:-}"
CH_CERT_IP="${CH_CERT_IP:-}"
CH_CERT_DAYS="${CH_CERT_DAYS:-825}"
CH_DIR="/etc/clickhouse-server"
CH_CONFIG_D_DIR="${CH_DIR}/config.d"
PKI_DIR="${CH_DIR}/pki"
SERVER_CERT="${CH_DIR}/server.crt"
SERVER_KEY="${CH_DIR}/server.key"
OVERRIDE_FILE="${CH_CONFIG_D_DIR}/waf-https.xml"
mkdir -p "${CH_CONFIG_D_DIR}" "${PKI_DIR}"
split_csv() {
local raw="$1"
if [[ -z "${raw}" ]]; then
return 0
fi
IFS=',' read -r -a arr <<<"${raw}"
for item in "${arr[@]}"; do
item="$(echo "${item}" | xargs)"
if [[ -n "${item}" ]]; then
echo "${item}"
fi
done
}
build_san_line() {
local san_entries=()
while IFS= read -r dns_item; do
san_entries+=("DNS:${dns_item}")
done < <(split_csv "${CH_CERT_DNS}")
while IFS= read -r ip_item; do
san_entries+=("IP:${ip_item}")
done < <(split_csv "${CH_CERT_IP}")
if [[ ${#san_entries[@]} -eq 0 ]]; then
san_entries+=("DNS:${CH_CERT_CN}")
fi
local san_line
san_line="$(IFS=,; echo "${san_entries[*]}")"
echo "${san_line}"
}
generate_self_signed_cert() {
echo "[INFO] generating self-signed server certificate (crt+key only) ..."
local server_key="${PKI_DIR}/server.key"
local server_csr="${PKI_DIR}/server.csr"
local server_crt="${PKI_DIR}/server.crt"
local ext_file="${PKI_DIR}/server.ext"
local san_line
san_line="$(build_san_line)"
openssl genrsa -out "${server_key}" 2048
openssl req -new -key "${server_key}" -out "${server_csr}" -subj "/CN=${CH_CERT_CN}"
cat >"${ext_file}" <<EOF
subjectAltName=${san_line}
keyUsage=digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
EOF
openssl x509 -req -in "${server_csr}" -signkey "${server_key}" \
-out "${server_crt}" -days "${CH_CERT_DAYS}" -sha256 -extfile "${ext_file}"
cp -f "${server_crt}" "${SERVER_CERT}"
cp -f "${server_key}" "${SERVER_KEY}"
}
generate_self_signed_cert
chown clickhouse:clickhouse "${SERVER_CERT}" "${SERVER_KEY}" || true
chmod 0644 "${SERVER_CERT}"
chmod 0640 "${SERVER_KEY}"
echo "[INFO] writing ClickHouse HTTPS override config ..."
cat >"${OVERRIDE_FILE}" <<EOF
<clickhouse>
<https_port>${CH_HTTPS_PORT}</https_port>
<listen_host>${CH_LISTEN_HOST}</listen_host>
<openSSL>
<server>
<certificateFile>${SERVER_CERT}</certificateFile>
<privateKeyFile>${SERVER_KEY}</privateKeyFile>
<verificationMode>none</verificationMode>
<loadDefaultCAFile>true</loadDefaultCAFile>
<cacheSessions>true</cacheSessions>
<disableProtocols>sslv2,sslv3</disableProtocols>
<preferServerCiphers>true</preferServerCiphers>
<invalidCertificateHandler>
<name>RejectCertificateHandler</name>
</invalidCertificateHandler>
</server>
</openSSL>
</clickhouse>
EOF
echo "[INFO] restarting clickhouse-server ..."
systemctl restart clickhouse-server
sleep 2
echo "[INFO] service status ..."
systemctl --no-pager -l status clickhouse-server | sed -n '1,15p'
echo "[INFO] verifying HTTPS endpoint ..."
curl -sk "https://127.0.0.1:${CH_HTTPS_PORT}/?query=SELECT%201" || true
echo
echo "[OK] ClickHouse HTTPS setup finished"
echo " HTTPS port : ${CH_HTTPS_PORT}"
echo " cert file : ${SERVER_CERT}"
echo " key file : ${SERVER_KEY}"

50
deploy/clickhouse/configure_clickhouse_runtime.sh Normal file
View File

@@ -0,0 +1,50 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ "${EUID}" -ne 0 ]]; then
echo "[ERROR] please run as root"
exit 1
fi
CH_LOG_LEVEL="${CH_LOG_LEVEL:-warning}"
CH_DIR="/etc/clickhouse-server"
CH_CONFIG_D_DIR="${CH_DIR}/config.d"
OVERRIDE_FILE="${CH_CONFIG_D_DIR}/waf-runtime.xml"
case "${CH_LOG_LEVEL}" in
none|fatal|critical|error|warning|notice|information|debug|trace|test)
;;
*)
echo "[ERROR] invalid CH_LOG_LEVEL: ${CH_LOG_LEVEL}"
echo " allowed: none,fatal,critical,error,warning,notice,information,debug,trace,test"
exit 1
;;
esac
mkdir -p "${CH_CONFIG_D_DIR}"
echo "[INFO] writing ClickHouse runtime override config ..."
cat >"${OVERRIDE_FILE}" <<EOF
<clickhouse>
<logger>
<level>${CH_LOG_LEVEL}</level>
</logger>
<text_log remove="1"/>
<part_log remove="1"/>
<metric_log remove="1"/>
<asynchronous_metric_log remove="1"/>
<trace_log remove="1"/>
</clickhouse>
EOF
echo "[INFO] restarting clickhouse-server ..."
systemctl restart clickhouse-server
sleep 2
echo "[INFO] service status ..."
systemctl --no-pager -l status clickhouse-server | sed -n '1,15p'
echo "[OK] ClickHouse runtime config applied"
echo " file : ${OVERRIDE_FILE}"
echo " logger level: ${CH_LOG_LEVEL}"

38
deploy/clickhouse/init_waf_logs_tables.sh Normal file
View File

@@ -0,0 +1,38 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SQL_FILE="${SCRIPT_DIR}/init_waf_logs_tables.sql"
if [[ ! -f "${SQL_FILE}" ]]; then
echo "[ERROR] SQL file not found: ${SQL_FILE}"
exit 1
fi
if ! command -v clickhouse-client >/dev/null 2>&1; then
echo "[ERROR] clickhouse-client not found. Please install ClickHouse client first."
exit 1
fi
CH_HOST="${CH_HOST:-127.0.0.1}"
CH_PORT="${CH_PORT:-9000}"
CH_USER="${CH_USER:-default}"
CH_PASSWORD="${CH_PASSWORD:-}"
CH_DATABASE="${CH_DATABASE:-default}"
args=(--host "${CH_HOST}" --port "${CH_PORT}" --user "${CH_USER}")
if [[ -n "${CH_PASSWORD}" ]]; then
args+=(--password "${CH_PASSWORD}")
fi
echo "[INFO] creating database if not exists: ${CH_DATABASE}"
clickhouse-client "${args[@]}" --query "CREATE DATABASE IF NOT EXISTS ${CH_DATABASE}"
echo "[INFO] initializing tables in database: ${CH_DATABASE}"
clickhouse-client "${args[@]}" --database "${CH_DATABASE}" < "${SQL_FILE}"
echo "[INFO] checking table status ..."
clickhouse-client "${args[@]}" --database "${CH_DATABASE}" --query \
"SELECT name, engine FROM system.tables WHERE database='${CH_DATABASE}' AND name IN ('logs_ingest','dns_logs_ingest') ORDER BY name"
echo "[OK] ClickHouse ingest tables are ready in database '${CH_DATABASE}'"

69
deploy/clickhouse/init_waf_logs_tables.sql Normal file
View File

@@ -0,0 +1,69 @@
-- Initialize HTTP and DNS ingest tables for GoEdge access logs.
-- Run with:
-- clickhouse-client --database <db_name> < init_waf_logs_tables.sql
CREATE TABLE IF NOT EXISTS logs_ingest
(
timestamp DateTime CODEC(DoubleDelta, ZSTD(1)),
node_id UInt64,
cluster_id UInt64,
server_id UInt64,
host LowCardinality(String),
ip String,
method LowCardinality(String),
path String CODEC(ZSTD(1)),
status UInt16,
bytes_in UInt64 CODEC(Delta, ZSTD(1)),
bytes_out UInt64 CODEC(Delta, ZSTD(1)),
cost_ms UInt32 CODEC(Delta, ZSTD(1)),
ua String CODEC(ZSTD(1)),
referer String CODEC(ZSTD(1)),
log_type LowCardinality(String),
trace_id String,
firewall_policy_id UInt64 DEFAULT 0,
firewall_rule_group_id UInt64 DEFAULT 0,
firewall_rule_set_id UInt64 DEFAULT 0,
firewall_rule_id UInt64 DEFAULT 0,
request_headers String CODEC(ZSTD(3)) DEFAULT '',
request_body String CODEC(ZSTD(3)) DEFAULT '',
response_headers String CODEC(ZSTD(3)) DEFAULT '',
response_body String CODEC(ZSTD(3)) DEFAULT '',
INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_ip ip TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_host host TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4,
INDEX idx_fw_policy firewall_policy_id TYPE minmax GRANULARITY 4,
INDEX idx_status status TYPE minmax GRANULARITY 4
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (timestamp, node_id, server_id, trace_id)
SETTINGS index_granularity = 8192;
CREATE TABLE IF NOT EXISTS dns_logs_ingest
(
timestamp DateTime CODEC(DoubleDelta, ZSTD(1)),
request_id String,
node_id UInt64,
cluster_id UInt64,
domain_id UInt64,
record_id UInt64,
remote_addr String,
question_name String,
question_type LowCardinality(String),
record_name String,
record_type LowCardinality(String),
record_value String,
networking LowCardinality(String),
is_recursive UInt8,
error String CODEC(ZSTD(1)),
ns_route_codes Array(String),
content_json String CODEC(ZSTD(3)) DEFAULT '',
INDEX idx_request_id request_id TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_remote_addr remote_addr TYPE bloom_filter(0.01) GRANULARITY 4,
INDEX idx_question_name question_name TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4,
INDEX idx_domain_id domain_id TYPE minmax GRANULARITY 4
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (timestamp, request_id, node_id)
SETTINGS index_granularity = 8192;

95
deploy/clickhouse/install_clickhouse_linux.sh Normal file
View File

@@ -0,0 +1,95 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ "${EUID}" -ne 0 ]]; then
echo "[ERROR] please run as root"
exit 1
fi
if [[ ! -f /etc/os-release ]]; then
echo "[ERROR] /etc/os-release not found"
exit 1
fi
# shellcheck disable=SC1091
source /etc/os-release
os_id="$(echo "${ID:-}" | tr '[:upper:]' '[:lower:]')"
os_ver="${VERSION_ID:-}"
is_ubuntu22=false
is_amzn2023=false
if [[ "${os_id}" == "ubuntu" && "${os_ver}" == 22.04* ]]; then
is_ubuntu22=true
fi
if [[ "${os_id}" == "amzn" && "${os_ver}" == 2023* ]]; then
is_amzn2023=true
fi
if [[ "${is_ubuntu22}" != "true" && "${is_amzn2023}" != "true" ]]; then
echo "[ERROR] only Ubuntu 22.04 or Amazon Linux 2023 is supported. current: ID=${ID:-unknown}, VERSION_ID=${VERSION_ID:-unknown}"
exit 1
fi
if [[ "${is_ubuntu22}" == "true" ]]; then
echo "[INFO] detected Ubuntu 22.04"
echo "[INFO] installing prerequisites ..."
apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get install -y curl ca-certificates gnupg apt-transport-https lsb-release
echo "[INFO] configuring ClickHouse apt repository ..."
install -d -m 0755 /etc/apt/keyrings
if [[ ! -f /etc/apt/keyrings/clickhouse.gpg ]]; then
curl -fsSL https://packages.clickhouse.com/CLICKHOUSE-KEY.GPG | gpg --dearmor -o /etc/apt/keyrings/clickhouse.gpg
fi
cat >/etc/apt/sources.list.d/clickhouse.list <<'EOF'
deb [signed-by=/etc/apt/keyrings/clickhouse.gpg arch=amd64,arm64] https://packages.clickhouse.com/deb stable main
EOF
echo "[INFO] installing clickhouse-server and clickhouse-client ..."
apt-get update -y
DEBIAN_FRONTEND=noninteractive apt-get install -y clickhouse-server clickhouse-client clickhouse-common-static
fi
if [[ "${is_amzn2023}" == "true" ]]; then
echo "[INFO] detected Amazon Linux 2023"
echo "[INFO] installing prerequisites ..."
dnf makecache -y
dnf install -y curl ca-certificates gnupg2 dnf-plugins-core
echo "[INFO] configuring ClickHouse yum repository ..."
cat >/etc/yum.repos.d/clickhouse.repo <<'EOF'
[clickhouse-stable]
name=ClickHouse Stable Repository
baseurl=https://packages.clickhouse.com/rpm/stable/$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.clickhouse.com/rpm/stable/repodata/repomd.xml.key
https://packages.clickhouse.com/rpm/clickhouse-static.key
EOF
echo "[INFO] installing clickhouse-server and clickhouse-client ..."
dnf clean all
dnf makecache -y
if ! dnf install -y clickhouse-server clickhouse-client clickhouse-common-static; then
dnf install -y clickhouse-server clickhouse-client
fi
fi
echo "[INFO] enabling clickhouse-server ..."
systemctl enable clickhouse-server >/dev/null 2>&1 || true
systemctl restart clickhouse-server
sleep 2
if [[ -n "${CLICKHOUSE_DEFAULT_PASSWORD:-}" ]]; then
echo "[INFO] setting default user password ..."
if [[ "${CLICKHOUSE_DEFAULT_PASSWORD}" == *"'"* ]]; then
echo "[ERROR] CLICKHOUSE_DEFAULT_PASSWORD contains single quote, please set password manually with clickhouse-client"
exit 1
fi
clickhouse-client --query "ALTER USER default IDENTIFIED WITH plaintext_password BY '${CLICKHOUSE_DEFAULT_PASSWORD}'"
fi
echo "[INFO] health check ..."
clickhouse-client --query "SELECT version()"
echo "[OK] ClickHouse install completed: ID=${ID:-unknown}, VERSION_ID=${VERSION_ID:-unknown}"

108
deploy/clickhouse/setup_clickhouse.sh Normal file
View File

@@ -0,0 +1,108 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INSTALL_SCRIPT="${SCRIPT_DIR}/install_clickhouse_linux.sh"
HTTPS_SCRIPT="${SCRIPT_DIR}/configure_clickhouse_https.sh"
RUNTIME_SCRIPT="${SCRIPT_DIR}/configure_clickhouse_runtime.sh"
TABLES_SCRIPT="${SCRIPT_DIR}/init_waf_logs_tables.sh"
usage() {
cat <<'EOF'
Usage:
sudo ./setup_clickhouse.sh [all|install|https|runtime|tables]
Modes:
all Install ClickHouse, configure HTTPS, apply runtime config, init ingest tables (default)
install Install ClickHouse only
https Configure HTTPS only
runtime Apply ClickHouse runtime config only
tables Initialize ingest tables only
Common env vars:
CLICKHOUSE_DEFAULT_PASSWORD Default user password set during install
CH_HTTPS_PORT HTTPS port (default: 8443)
CH_CERT_CN Certificate CN
CH_CERT_DNS Certificate SAN DNS list (comma-separated)
CH_CERT_IP Certificate SAN IP list (comma-separated)
CH_CERT_DAYS Certificate validity days (default: 825)
CH_LOG_LEVEL ClickHouse logger level (default: warning)
CH_HOST ClickHouse host for table init (default: 127.0.0.1)
CH_PORT ClickHouse port for table init (default: 9000)
CH_USER ClickHouse user for table init (default: default)
CH_PASSWORD ClickHouse password for table init
CH_DATABASE Database for table init (default: default)
EOF
}
require_script() {
local script="$1"
if [[ ! -f "${script}" ]]; then
echo "[ERROR] required file not found: ${script}"
exit 1
fi
}
run_install() {
echo "[INFO] step 1/3: install ClickHouse ..."
bash "${INSTALL_SCRIPT}"
}
run_https() {
echo "[INFO] step 2/3: configure ClickHouse HTTPS ..."
bash "${HTTPS_SCRIPT}"
}
run_runtime() {
echo "[INFO] step 3/4: apply ClickHouse runtime config ..."
bash "${RUNTIME_SCRIPT}"
}
run_tables() {
echo "[INFO] step 4/4: initialize ingest tables ..."
bash "${TABLES_SCRIPT}"
}
MODE="${1:-all}"
case "${MODE}" in
-h|--help|help)
usage
exit 0
;;
all|install|https|runtime|tables)
;;
*)
echo "[ERROR] invalid mode: ${MODE}"
usage
exit 1
;;
esac
require_script "${INSTALL_SCRIPT}"
require_script "${HTTPS_SCRIPT}"
require_script "${RUNTIME_SCRIPT}"
require_script "${TABLES_SCRIPT}"
case "${MODE}" in
all)
run_install
run_https
run_runtime
run_tables
;;
install)
run_install
;;
https)
run_https
;;
runtime)
run_runtime
;;
tables)
run_tables
;;
esac
echo "[OK] setup completed: mode=${MODE}"