Skip to main content

使用 Docker 启动 Databend | 推荐

· 4 min read
wubx
wubx
Data AI

在早期的 CentOS 8 或是 OpenEubler 20 版本中启动 Databend 可能会遇到官方发布的 gnu 包无法使用的问题。 从实践上讲推荐使用 docker + host 网络直接启动，基本相当于直接启动一个 Databend 。

例如下面：

在有网的环境中部署​

利用 docker 启动 databend-meta​

dmeta.sh

docker run --name databend-meta    \
      --privileged \
      --network=host \
      -v /home/vagrant/meta:/var/lib/databend/meta \
      -v /data/databend/log:/var/log/databend \
      -v /etc/databend:/etc/databend \
      -e METASRV_CONFIG_FILE='/etc/databend/databend-meta.toml' \
      -d registry.databend.cn/public/databend-meta:v1.2.731-nightly

利用 docker 启动 databend-query​

dquery.sh

docker stop databend3307
docker rm databend3307
docker run --name databend3307    \
      --privileged \
      --network=host \
      -v /etc/databend:/etc/databend \
      -v /etc/localtime:/etc/localtime \
      -v /data/databend/3307:/var/log/databend \
      -v /data/databend/3307disk:/var/lib/databend \
      -e CONFIG_FILE="/etc/databend/databend-query-3307.toml" \
      -e _DATABEND_INTERNAL_MAX_CONURRENT_IO_REQUEST=300 \
      -e _DATABEND_INTERNAL_RETRY_IO_TIMEOUT=60 \
      -d registry.databend.cn/public/databend-query:v1.2.731-nightly

#      -e RUST_BACKTRACE=1 \
#      -e  STORAGE_S3_ENABLE_VIRTUAL_HOST_STYLE=true \ # 用 oss 的 s3 方式

配置文件参考​

databend-meta.toml​

log_dir                 = "/var/log/databend/"
limit                   = 12
admin_api_address       = "0.0.0.0:28101"
grpc_api_address        = "0.0.0.0:9191"
# databend-query fetch this address to update its databend-meta endpoints list,
# in case databend-meta cluster changes.
grpc_api_advertise_host = "192.168.1.201"

max_applied_log_to_keep = 1000
snapshot_logs_since_last=10000


[raft_config]
id            = 1
raft_dir      = "/var/lib/databend/meta"
raft_api_port = 28103

# Assign raft_{listen|advertise}_host in test config.
# This allows you to catch a bug in unit tests when something goes wrong in raft meta nodes communication.
raft_listen_host = "192.168.1.201"
raft_advertise_host = "192.168.1.201"

# per
snapshot_logs_since_last=10240
# Start up mode: single node cluster
single        = true

databend-query-3307.toml​

[query]
max_active_sessions = 300
shutdown_wait_timeout_ms=15000

flight_api_address = "192.168.1.201:9091"
admin_api_address = "0.0.0.0:8080"
metric_api_address = "0.0.0.0:7070"

mysql_handler_host = "0.0.0.0"
mysql_handler_port = 3307

http_handler_host = "0.0.0.0"
http_handler_port = 8000

flight_sql_handler_host = "0.0.0.0"
flight_sql_handler_port = 8900

tenant_id = "wubx"
cluster_id = "default"

table_engine_memory_enabled = true
default_storage_format = 'parquet'

#default_compression = 'lz4'
max_memory_limit_enabled=true
max_server_memory_usage=5368709120

enable_udf_server = true
udf_server_allow_insecure = true
udf_server_allow_list = ['http://192.168.1.201:8815']

[[setttings]]
max_storage_io_requests=6

[[query.users]]
name = "root"
auth_type = "no_password"

[log]

[log.file]
level = "DEBUG"
limit = 24
format = "text"
include_node_id=true
dir = "/var/log/databend"

[log.query]
on = true

[log.profile]
on = true
dir = "/var/log/databend/profile/"

[meta]
endpoints = ["0.0.0.0:9191"]
username = "root"
password = "root"
client_timeout_in_second = 60
auto_sync_interval = 60

[storage]
type = "s3"
allow_insecure = true
[storage.s3]
bucket = "v-wubx"
root="t0807"
endpoint_url = "http://192.168.1.100:9900"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
enable_virtual_host_style = false

[spill]
spill_local_disk_path = '/data/databend/3307/spill/'
spill_local_disk_reserved_space_percentage = 5

[spill.storage]
type = "s3"
[spill.storage.s3]
bucket = "myspill"
root = "admin"
endpoint_url = "http://192.168.1.100:9900"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
allow_insecure = true

[cache]
table_data_cache_population_queue_size = 64
data_cache_storage = "none"
enable_table_meta_cache = true
table_data_deserialized_data_bytes =1073741824
enable_table_bloom_index_cache=true
table_bloom_index_meta_count=3000
table_bloom_index_filter_size=1073741824

[cache.disk]
path = "/var/lib/databend/cache"
max_bytes = 21474836480

https://github.com/databendlabs/databend/tree/main/scripts/distribution/configs

在无网络的环境部署​

在有一些生产环境中没有网络，也无法连外部的 Docker 源，这时需要下载对应的 Docker image 然后 load 到部署机器或是 push 到公司内网的 docker repo 中

image 下载​

# 本地可以联网环境有去 pull 对应的 image 然后打包上传到上产环境
docker pull registry.databend.cn/public/databend-query:v1.2.735-nightly
docker pull registry.databend.cn/public/databend-meta:v1.2.735-nightly
# 导出对应的 image 
docker save registry.databend.cn/public/databend-query:v1.2.735-nightly |gzip >databend-query-v1.2.735-nightly.tar.gz
docker save registry.databend.cn/public/databend-meta:v1.2.735-nightly |gzip >databend-meta-v1.2.735-nightly.tar.gz

image 在对应机器上​

#upload 生产环境后

gzip -d databend-query-v1.2.735-nightly.tar.gz
gzip -d databend-meta-v1.2.735-nightly.tar.gz 

docker load -i ./databend-query-v1.2.735-nightly.tar
docker load -i ./databend-meta-v1.2.735-nightly.tar

然后启动方式就可以按上面的方式进行。

进阶推入私有化 docker repo​

假设私有化的 docker repo 地址为： registry.databend.local

docker tag registry.databend.cn/public/databend-meta:v1.2.735-nightly registry.databend.local/datafuselabs/databend-query:v1.2.735-nightly 
docker push registry.databend.local/databendlabs/databend-query:v1.2.735-nightly

后续的启动就可以使用：

docker run --name databend3307    \
      --privileged \
      --network=host \
      -v /etc/databend:/etc/databend \
      -v /etc/localtime:/etc/localtime \
      -v /data/databend/3307:/var/log/databend \
      -v /data/databend/3307disk:/var/lib/databend \
      -e CONFIG_FILE="/etc/databend/databend-query-3307.toml" \
      -d registry.databend.local/databendlabs/databend-query:v1.2.735-nightly

💬 社区支持
有问题与我们的团队联系：Slack

微信：82565387