CentOS7上部署ollama+qwen3.8:27b-q8_0

最近需要在CentOS7.9A40服务器上部署ollama记录操作过程如下版本配配好CentOS7.9.2009 kernel3.10.0-1160.119.1.el7.x86_64 NVIDIA Driver:550.144.03(Data Center Driver, RHEL7 通道)CUDA Toolkit:12.4.1(cuda-toolkit-12-4不装驱动组件)nvidia-container-toolkit:1.20.0(离线rpm四件套)Docker CE24.x nvidia-ctk configure--runtimedocker Ollama: 容器内跑OLLAMA_HOST0.0.0.0:11434操作系统yum源可选由于CentOS官方不维护因此在线安装存在问题需要切换为aliyun的yum源不然建议还是换系统1. 备份原有源mv/etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup2. 下载阿里云 CentOS7归档源配置wget-O/etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo3. 如果你希望严格锁定在7.6版本可选但某些环境需要可手动修改 repo 文件将 baseurl 路径中的$releasever统一改为7.6.1810例如baseurlhttps://mirrors.aliyun.com/centos-vault/7.6.1810/os/$basearch/4. 清理并重建缓存 yum clean allyum makecache安装安装驱动# 下载驱动文件wgethttps://download.nvidia.com/XFree86/Linux-x86_64/550.144.03/NVIDIA-Linux-x86_64-550.144.03.run# 切命令行SSH 可跳过本地有桌面必须做systemctl isolate multi-user.target# 清旧驱动首次装可跳过/usr/bin/nvidia-uninstall-s2/dev/nullchmodx NVIDIA-Linux-x86_64-550.144.03.run ./NVIDIA-Linux-x86_64-550.144.03.run\--silent--dkms--disable-nouveau --no-opengl-files\--kernel-source-path/usr/src/kernels/$(uname-r)modprobe nvidia nvidia-smi# 显示以下信息表示安装成功了[roottest ~]# modprobe nvidia[roottest~]# nvidia-smiSun Aug2300:37:182026-----------------------------------------------------------------------------------------|NVIDIA-SMI550.144.03 Driver Version:550.144.03 CUDA Version:12.4||---------------------------------------------------------------------------------------|GPU Name Persistence-M|Bus-Id Disp.A|Volatile Uncorr. ECC||Fan Temp Perf Pwr:Usage/Cap|Memory-Usage|GPU-Util Compute M.||||MIG M.||||0NVIDIA A40 On|00000000:00:0C.0 Off|0||0% 49C P0 64W / 300W|1MiB / 46068MiB|0% Default||||N/A|--------------------------------------------------------------------------------------- -----------------------------------------------------------------------------------------|Processes:||GPU GI CI PID Type Process name GPU Memory||ID ID Usage||||No running processes found|-----------------------------------------------------------------------------------------如果需要一直监听显卡状态使用如下命令nvidia-smi -l 1安装nvidia-toolkit离线版# 找有外网的服务器执行yuminstall-ycurlcurl-s-Lhttps://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo|\sudotee/etc/yum.repos.d/nvidia-container-toolkit.reposudoyuminstall--downloadonly--downloaddir./nvidia-toolkit-offline nvidia-container-toolkit# 下载指定版本的nvidia-container-toolkityuminstall--downloadonly--downloaddir./nvidia-toolkit-offline nvidia-container-toolkit-1.16.2-1# 压缩文件供后面使用tar-zcvfnvidia-toolkit-offline.tar.gz nvidia-toolkit-offline# 解压后有以下几个文件-rw-r--r--.1root root1318229Aug2209:49 libnvidia-container1-1.20.0-1.x86_64.rpm -rw-r--r--.1root root44697Aug2209:49 libnvidia-container-tools-1.20.0-1.x86_64.rpm -rw-r--r--.1root root1470145Aug2209:49 nvidia-container-toolkit-1.20.0-1.x86_64.rpm -rw-r--r--.1root root7937485Aug2209:49 nvidia-container-toolkit-base-1.20.0-1.x86_64.rpm# 直接装忽略签名最快离线机常用cd/root/nvidia-toolkit-offline yum localinstall-y--disablerepo*--nogpgcheck*.rpm配置nvidia-ctk注意这里docker一定要重启否则可能无法生效nvidia-ctk runtime configure--runtimedocker systemctl restartdockerDocker部署推荐dockerstop ollamadockerrmollamadockerrun-d--nameollama--restartunless-stopped--networkhost\--gpusall\-vollama:/root/.ollama\-eOLLAMA_HOST0.0.0.0:11434\-eOLLAMA_GPU_LAYERS64\-eCUDA_VISIBLE_DEVICES0\-eOLLAMA_KEEP_ALIVE10m\ollama/ollama:latest备选dockerstop ollamadockerrm-follamadockerrun-d--restartalways--networkhost\--gpusall\--privileged\-vollama:/root/.ollama\--nameollama\--security-optlabeldisable\--security-optseccompunconfined\--security-optapparmorunconfined\ollama/ollama拉取模型# 进入正在运行的 ollama 容器并执行拉取dockerexec-itollama ollama list# 拉取模型dockerexec-itollama ollama pull qwen3-coder:30b# 运行dockerexec-itollama ollama run qwen3-coder:30bdockerexec-itollama ollama run qwen3-coder:480bdockerexec-itollama ollama run qwen3.5:27bdockerexec-itollama ollama run qwen3.8:27b-q8_0ollama中setting.json配置{env:{CLAUDE_CODE_MAX_OUTPUT_TOKENS:64000,ANTHROPIC_AUTH_TOKEN:ollama,ANTHROPIC_BASE_URL:http://localhost:11434,ANTHROPIC_DEFAULT_HAIKU_MODEL:qwen3.8:27b-q8_0,ANTHROPIC_DEFAULT_OPUS_MODEL:qwen3.8:27b-q8_0,ANTHROPIC_DEFAULT_SONNET_MODEL:qwen3.8:27b-q8_0,ANTHROPIC_MODEL:qwen3.8:27b-q8_0,API_TIMEOUT_MS:3000000,CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC:0,CLAUDE_CODE_ENABLE_TELEMETRY:0},includeCoAuthoredBy:false,model:opus,enabledPlugins:{code-reviewclaude-plugins-official:true,document-skillsanthropic-agent-skills:true,frontend-designclaude-plugins-official:true,githubclaude-plugins-official:true,planning-with-filesplanning-with-files:true,skill-creatorclaude-plugins-official:false,superpowersclaude-plugins-official:true,superpowerssuperpowers-marketplace:true},extraKnownMarketplaces:{anthropic-agent-skills:{source:{source:github,repo:anthropics/skills}},claude-code-plugins:{source:{source:github,repo:anthropics/claude-code}},planning-with-files:{source:{source:github,repo:OthmanAdi/planning-with-files}},superpowers-marketplace:{source:{source:github,repo:obra/superpowers-marketplace}}},skipWebFetchPreflight:true,promptSuggestionEnabled:false,terminalProgressBarEnabled:false}如果需要代理上网可在env中增加配置HTTPS_PROXY:http://127.0.0.1:7897,HTTP_PROXY:http://127.0.0.1:7897,ALL_PROXY:http://127.0.0.1:7897,