python_market_data_service/DEPLOY.md

# 统一行情数据服务 - 部署文档

本文档详细说明行情数据服务的部署方式，支持 **Go** 和 **Python** 两种实现。

## 实现方式选择

| 特性 | Go实现 | Python实现 |
|------|--------|------------|
| 性能 | ⭐⭐⭐ 高性能 | ⭐⭐ 良好 |
| 开发效率 | ⭐⭐ 中等 | ⭐⭐⭐ 高 |
| 数据源对接 | ⭐⭐ 需自行封装 | ⭐⭐⭐ Tushare原生支持 |
| 生态丰富度 | ⭐⭐ 中等 | ⭐⭐⭐ 丰富 |
| 部署复杂度 | ⭐⭐⭐ 单二进制 | ⭐⭐ 依赖较多 |

**推荐**: 
- 生产环境高并发场景选择 **Go实现**
- 快速原型、数据源对接优先选择 **Python实现**

---

# 一、Go 实现部署

## 1.1 部署架构

```
┌─────────────────────────────────────────────────────────────┐
│                        负载均衡器 (可选)                      │
│                         Nginx/HAProxy                        │
└─────────────────────┬───────────────────────────────────────┘
                      │
        ┌─────────────┼─────────────┐
        │             │             │
┌───────▼──────┐ ┌────▼─────┐ ┌────▼─────┐
│  服务实例1    │ │ 服务实例2 │ │ 服务实例3 │
│  :8080       │ │  :8080   │ │  :8080   │
└───────┬──────┘ └────┬─────┘ └────┬─────┘
        │             │            │
        └─────────────┼────────────┘
                      │
        ┌─────────────▼──────────────┐
        │      PostgreSQL 主从       │
        │    (分区表 + 读写分离)      │
        └────────────────────────────┘
```

## 1.2 环境要求

### 服务器配置

| 组件 | 最低配置 | 推荐配置 |
|------|---------|---------|
| 应用服务 | 2核4G | 4核8G |
| PostgreSQL | 4核8G | 8核16G |
| 存储 | 200GB SSD | 500GB SSD |
| 网络 | 10Mbps | 100Mbps |

### 软件依赖

- **操作系统**: Ubuntu 20.04+ / CentOS 8+ / Debian 11+
- **Go**: 1.21+
- **PostgreSQL**: 15+ (支持分区表)
- **Nginx**: 1.18+ (可选，用于负载均衡)

## 1.3 部署步骤

### 安装依赖

```bash
# Ubuntu/Debian
sudo apt update
sudo apt install -y postgresql-15 postgresql-contrib-15 nginx

# 安装Go 1.21
wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz
echo 'export PATH=$PATH:/usr/local/go/bin' >> ~/.bashrc
source ~/.bashrc
```

### 配置PostgreSQL

```bash
# 启动PostgreSQL
sudo systemctl start postgresql
sudo systemctl enable postgresql

# 创建数据库和用户
sudo -u postgres psql -c "CREATE USER marketdata WITH PASSWORD 'your_password';"
sudo -u postgres psql -c "CREATE DATABASE marketdata OWNER marketdata;"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE marketdata TO marketdata;"

# 启用分区表支持
sudo -u postgres psql -d marketdata -c "CREATE EXTENSION IF NOT EXISTS pg_partman;"

# 配置允许远程连接 (可选)
sudo sed -i "s/#listen_addresses = 'localhost'/listen_addresses = '*'/g" /etc/postgresql/15/main/postgresql.conf
echo "host all all 0.0.0.0/0 scram-sha-256" | sudo tee -a /etc/postgresql/15/main/pg_hba.conf
sudo systemctl restart postgresql
```

### 部署应用

```bash
# 创建应用目录
sudo mkdir -p /opt/market-data-service
sudo chown $USER:$USER /opt/market-data-service

# 解压项目
tar -xzf market-data-service.tar.gz -C /opt/
cd /opt/market-data-service

# 下载依赖
go mod download

# 构建
make build-all

# 创建配置文件目录
mkdir -p /opt/market-data-service/config
mkdir -p /opt/market-data-service/logs
```

### 配置环境变量

创建 `/opt/market-data-service/.env`:

```bash
# 应用配置
export PORT=8080
export GIN_MODE=release
export API_KEY=your_secure_api_key_here

# 数据库配置
export DATABASE_URL="postgres://marketdata:your_password@localhost:5432/marketdata?sslmode=disable"

# Tushare配置
export TUSHARE_TOKEN=your_tushare_token_here

# 日志配置
export LOG_LEVEL=info
export LOG_FILE=/opt/market-data-service/logs/app.log
```

### 初始化数据库

```bash
cd /opt/market-data-service
source .env

# 执行数据库初始化脚本
psql $DATABASE_URL -f memory/2026-03-07-database-schema.sql

# 同步基础数据
cd /opt/market-data-service
./bin/market-data-sync -type stocks
./bin/market-data-sync -type futures
./bin/market-data-sync -type calendar -start 20240101 -end 20241231
```

### 创建Systemd服务

创建 `/etc/systemd/system/market-data-service.service`:

```ini
[Unit]
Description=Market Data Service
After=network.target postgresql.service

[Service]
Type=simple
User=marketdata
Group=marketdata
WorkingDirectory=/opt/market-data-service
EnvironmentFile=/opt/market-data-service/.env
ExecStart=/opt/market-data-service/bin/market-data-service
Restart=always
RestartSec=5
StandardOutput=append:/opt/market-data-service/logs/service.log
StandardError=append:/opt/market-data-service/logs/error.log

[Install]
WantedBy=multi-user.target
```

```bash
# 创建用户
sudo useradd -r -s /bin/false marketdata
sudo chown -R marketdata:marketdata /opt/market-data-service

# 启动服务
sudo systemctl daemon-reload
sudo systemctl enable market-data-service
sudo systemctl start market-data-service

# 查看状态
sudo systemctl status market-data-service
sudo journalctl -u market-data-service -f
```

### Docker部署 (Go)

```dockerfile
# Dockerfile
FROM golang:1.21-alpine AS builder

WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o bin/market-data-service ./cmd/server

FROM alpine:latest
RUN apk --no-cache add ca-certificates tzdata
WORKDIR /root/

COPY --from=builder /app/bin/market-data-service .
COPY --from=builder /app/config ./config

EXPOSE 8080

CMD ["./market-data-service"]
```

---

# 二、Python 实现部署

## 2.1 部署架构

Python实现的部署架构与Go相同，但运行环境需要Python解释器和依赖库。

```
┌─────────────────────────────────────────────────────────────┐
│                    Nginx (反向代理)                          │
└─────────────────────┬───────────────────────────────────────┘
                      │
        ┌─────────────┼─────────────┐
        │             │             │
┌───────▼──────┐ ┌────▼─────┐ ┌────▼─────┐
│  Python服务1  │ │ Python   │ │ Python   │
│  Uvicorn     │ │ 服务2    │ │ 服务3    │
│  :8080       │ │  :8081   │ │  :8082   │
└───────┬──────┘ └────┬─────┘ └────┬─────┘
        │             │            │
        └─────────────┼────────────┘
                      │
        ┌─────────────▼──────────────┐
        │      PostgreSQL            │
        └────────────────────────────┘
```

## 2.2 环境要求

- **操作系统**: Ubuntu 20.04+ / CentOS 8+ / Windows 10+
- **Python**: 3.10+
- **PostgreSQL**: 15+
- **Nginx**: 1.18+ (可选)

## 2.3 部署步骤

### 安装Python环境

```bash
# Ubuntu/Debian
sudo apt update
sudo apt install -y python3.10 python3.10-venv python3-pip postgresql-15 nginx

# CentOS/RHEL
sudo yum install -y python310 python310-pip postgresql15-server nginx
```

### 创建虚拟环境

```bash
# 创建应用目录
sudo mkdir -p /opt/python-market-data-service
sudo chown $USER:$USER /opt/python-market-data-service

cd /opt/python-market-data-service

# 创建虚拟环境
python3.10 -m venv venv

# 激活虚拟环境
source venv/bin/activate

# 升级pip
pip install --upgrade pip
```

### 安装依赖

```bash
# 复制项目文件
cp -r /path/to/python_market_data_service/* /opt/python-market-data-service/

# 安装依赖
pip install -r requirements.txt

# 安装Tushare（需单独安装）
pip install tushare
```

### 配置环境变量

创建 `/opt/python-market-data-service/.env`:

```bash
# 应用配置
export PORT=8080
export DATABASE_URL="postgresql://marketdata:your_password@localhost:5432/marketdata"
export TUSHARE_TOKEN="your_tushare_token"
export API_KEY="your_api_key"

# Python路径
export PYTHONPATH=/opt/python-market-data-service
```

### 初始化数据库

```bash
source venv/bin/activate
source .env

# 使用Python初始化数据库（SQLAlchemy会自动创建表）
python -c "from app.repositories.database import init_db; init_db()"

# 或使用SQL脚本
psql $DATABASE_URL -f memory/2026-03-07-database-schema.sql
```

### 同步基础数据

```bash
source venv/bin/activate
source .env

# 同步股票列表
python scripts/sync_data.py --type stocks

# 同步期货列表
python scripts/sync_data.py --type futures

# 同步交易日历
python scripts/sync_data.py --type calendar --start 20240101 --end 20241231
```

### 启动服务

**方式1: 直接启动（开发模式）**

```bash
source venv/bin/activate
python -m app.main
```

**方式2: 使用Uvicorn（生产模式）**

```bash
source venv/bin/activate

# 单进程
uvicorn app.main:app --host 0.0.0.0 --port 8080

# 多进程（推荐）
uvicorn app.main:app --host 0.0.0.0 --port 8080 --workers 4
```

**方式3: 使用Gunicorn（更高性能）**

```bash
source venv/bin/activate

# 4个worker
 gunicorn app.main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8080
```

### 创建Systemd服务 (Python)

创建 `/etc/systemd/system/python-market-data-service.service`:

```ini
[Unit]
Description=Python Market Data Service
After=network.target postgresql.service

[Service]
Type=simple
User=marketdata
Group=marketdata
WorkingDirectory=/opt/python-market-data-service
Environment=PATH=/opt/python-market-data-service/venv/bin
EnvironmentFile=/opt/python-market-data-service/.env
ExecStart=/opt/python-market-data-service/venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8080
Restart=always
RestartSec=5
StandardOutput=append:/opt/python-market-data-service/logs/service.log
StandardError=append:/opt/python-market-data-service/logs/error.log

[Install]
WantedBy=multi-user.target
```

```bash
# 创建用户和日志目录
sudo useradd -r -s /bin/false marketdata
sudo mkdir -p /opt/python-market-data-service/logs
sudo chown -R marketdata:marketdata /opt/python-market-data-service

# 启动服务
sudo systemctl daemon-reload
sudo systemctl enable python-market-data-service
sudo systemctl start python-market-data-service

# 查看状态
sudo systemctl status python-market-data-service
sudo journalctl -u python-market-data-service -f
```

### Docker部署 (Python)

创建 `Dockerfile`:

```dockerfile
# Dockerfile for Python
FROM python:3.10-slim

WORKDIR /app

# 安装系统依赖
RUN apt-get update && apt-get install -y \
    gcc \
    postgresql-client \
    && rm -rf /var/lib/apt/lists/*

# 复制依赖文件
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install tushare

# 复制应用代码
COPY . .

# 暴露端口
EXPOSE 8080

# 启动命令
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080"]
```

创建 `docker-compose.yml`:

```yaml
version: '3.8'

services:
  app:
    build: .
    ports:
      - "8080:8080"
    environment:
      - PORT=8080
      - DATABASE_URL=postgresql://marketdata:password@postgres:5432/marketdata
      - TUSHARE_TOKEN=${TUSHARE_TOKEN}
      - API_KEY=${API_KEY}
    depends_on:
      - postgres
    restart: always
    volumes:
      - ./logs:/app/logs

  postgres:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=marketdata
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=marketdata
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    restart: always

volumes:
  postgres_data:
```

```bash
# 构建并启动
docker-compose up -d

# 查看日志
docker-compose logs -f app

# 同步数据
docker-compose exec app python scripts/sync_data.py --type stocks
```

---

# 三、Nginx配置（通用）

## 3.1 配置Nginx

创建 `/etc/nginx/sites-available/market-data-service`:

```nginx
upstream market_data_backend {
    server 127.0.0.1:8080;
    # 多实例时添加
    # server 127.0.0.1:8081;
    # server 127.0.0.1:8082;
}

server {
    listen 80;
    server_name api.marketdata.example.com;

    # WebSocket支持
    location /v1/stream {
        proxy_pass http://market_data_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-API-Key $http_x_api_key;
        proxy_read_timeout 86400;
    }

    # REST API
    location /v1/ {
        proxy_pass http://market_data_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-API-Key $http_x_api_key;
        
        # 限流
        limit_req zone=api_limit burst=20 nodelay;
    }

    # 管理后台
    location /admin {
        proxy_pass http://market_data_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # 健康检查
    location /health {
        proxy_pass http://market_data_backend/v1/admin/health;
        access_log off;
    }
}

# 限流配置
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;
```

```bash
sudo ln -s /etc/nginx/sites-available/market-data-service /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx
```

---

# 四、监控与维护（通用）

## 4.1 查看服务状态

```bash
# Go服务
sudo systemctl status market-data-service

# Python服务
sudo systemctl status python-market-data-service

# 实时日志
sudo tail -f /opt/market-data-service/logs/service.log
# 或
sudo tail -f /opt/python-market-data-service/logs/service.log

# 数据库连接数
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"
```

## 4.2 备份策略

```bash
#!/bin/bash
# 数据库备份脚本
BACKUP_DIR=/opt/backups
DATE=$(date +%Y%m%d_%H%M%S)

pg_dump $DATABASE_URL | gzip > $BACKUP_DIR/marketdata_$DATE.sql.gz

# 保留最近7天备份
find $BACKUP_DIR -name "marketdata_*.sql.gz" -mtime +7 -delete
```

## 4.3 水平扩展

**Go:**
```bash
PORT=8081 ./bin/market-data-service &
PORT=8082 ./bin/market-data-service &
PORT=8083 ./bin/market-data-service &
```

**Python:**
```bash
# 使用不同端口启动多个实例
uvicorn app.main:app --host 0.0.0.0 --port 8081 &
uvicorn app.main:app --host 0.0.0.0 --port 8082 &
uvicorn app.main:app --host 0.0.0.0 --port 8083 &
```

---

# 五、故障排查

## 5.1 常见问题

**问题1**: 服务无法启动，提示数据库连接失败
```bash
# 检查PostgreSQL状态
sudo systemctl status postgresql

# 测试连接
psql $DATABASE_URL -c "SELECT 1;"
```

**问题2**: Python服务依赖安装失败
```bash
# 检查Python版本
python3 --version  # 需 >= 3.10

# 安装系统依赖（Ubuntu）
sudo apt install -y python3-dev libpq-dev gcc

# 重新安装
pip install -r requirements.txt --force-reinstall
```

**问题3**: WebSocket连接断开
```bash
# 检查连接数
netstat -an | grep :8080 | wc -l

# 查看错误日志
sudo tail -f /opt/market-data-service/logs/error.log
# 或
sudo tail -f /opt/python-market-data-service/logs/service.log
```

**问题4**: 数据同步失败
```bash
# Go
./bin/market-data-sync -type stocks 2>&1 | head -20

# Python
python scripts/sync_data.py --type stocks 2>&1 | head -20
```

---

# 六、性能优化

## 6.1 数据库优化

```sql
-- 添加额外索引
CREATE INDEX CONCURRENTLY idx_klines_symbol_ts ON stock.klines_1m(symbol_id, ts DESC);
```

## 6.2 连接池配置

**Go:**
```bash
export DB_MAX_OPEN_CONNS=100
export DB_MAX_IDLE_CONNS=10
```

**Python:**
```python
# 在 app/repositories/database.py 中修改
engine = create_engine(
    config.database.database_url,
    pool_size=10,
    max_overflow=20,
    pool_pre_ping=True,
)
```

## 6.3 查询缓存

高频查询结果可缓存到Redis，Python实现已预留Redis配置。

---

# 七、安全建议

1. **API Key管理**: 定期更换API Key，使用环境变量存储
2. **数据库安全**: 使用强密码，限制远程访问，启用SSL
3. **网络安全**: 配置防火墙，仅开放必要端口
4. **日志脱敏**: 日志中避免输出敏感信息

---

**文档结束**