流程管理¶
使用 我们定制的 MET-Airflow 进行作业控制
因为客户机器未必可以与Internet相连，所以，我们事先将需要的 Docker image 打包成tar文件。首先装载需要的Images：
$ cd docker-airflow $ docker load -i postgres9.6.tar # Load postgres docker image 67ecfc9591c8: Loading layer [==================================================>] 58.51MB/58.51MB 92d5b04a5b39: Loading layer [==================================================>] 10.44MB/10.44MB 3910ebbe4a2d: Loading layer [==================================================>] 339.5kB/339.5kB 0d061d4cfd5a: Loading layer [==================================================>] 4.068MB/4.068MB 8d3d7cf7bbbb: Loading layer [==================================================>] 17.1MB/17.1MB b4266bc4925d: Loading layer [==================================================>] 1.103MB/1.103MB 1b5a8fd1e08a: Loading layer [==================================================>] 1.536kB/1.536kB e0efe0273e67: Loading layer [==================================================>] 8.704kB/8.704kB fca7b0f5ab0e: Loading layer [==================================================>] 125.9MB/125.9MB f9539ee11f46: Loading layer [==================================================>] 53.25kB/53.25kB 908419ba3b5e: Loading layer [==================================================>] 2.048kB/2.048kB 95e7fed1d125: Loading layer [==================================================>] 3.072kB/3.072kB ffb432d2bbe6: Loading layer [==================================================>] 8.704kB/8.704kB 82d9e01ff013: Loading layer [==================================================>] 1.536kB/1.536kB Loaded image: postgres:9.6 $ docker image ls -a REPOSITORY TAG IMAGE ID CREATED SIZE postgres 9.6 afd8110f1813 4 months ago 211MB $ docker load -i redis5.0.5.tar # Load redis docker image 2db44bce66cd: Loading layer [==================================================>] 72.48MB/72.48MB 96550adba4f9: Loading layer [==================================================>] 338.4kB/338.4kB a0e3cc85530d: Loading layer [==================================================>] 4.081MB/4.081MB 00352274272b: Loading layer [==================================================>] 24.65MB/24.65MB ddf2402ca3ea: Loading layer [==================================================>] 1.536kB/1.536kB cd1b418d5a74: Loading layer [==================================================>] 3.584kB/3.584kB Loaded image: redis:5.0.5 $ docker image ls -a REPOSITORY TAG IMAGE ID CREATED SIZE postgres 9.6 afd8110f1813 4 months ago 211MB redis 5.0.5 63130206b0fa 6 months ago 98.2MB $ docker load -i docker-airflow-met81.tar # Load airflow-met docker 896510bee743: Loading layer [==================================================>] 6.895MB/6.895MB 859394076549: Loading layer [==================================================>] 87.07MB/87.07MB 86339b326932: Loading layer [==================================================>] 4.608kB/4.608kB 5833990cb8e5: Loading layer [==================================================>] 7.909MB/7.909MB 4479c2d219e2: Loading layer [==================================================>] 607.5MB/607.5MB 822c15da002c: Loading layer [==================================================>] 4.608kB/4.608kB fab09785bbaf: Loading layer [==================================================>] 32.77kB/32.77kB 84e949db958d: Loading layer [==================================================>] 39.42kB/39.42kB Loaded image: docker-airflow_met81:latest $ docker image ls -a REPOSITORY TAG IMAGE ID CREATED SIZE docker-airflow_met81 latest 1e4488277470 20 hours ago 3.15GB postgres 9.6 afd8110f1813 4 months ago 211MB redis 5.0.5 63130206b0fa 6 months ago 98.2MB
启动 docker-composer
Note
注意修改 docker-compose-CeleryExecutor.yml 中的各image的版本号，与上一步所使用的images的版本号一致
Note
将本地所需要的目录，挂载到 docker container 中：
@@ -74,12 +74,14 @@ services:
command: scheduler
worker:
image: docker-airflow_met81
restart: always
depends_on:
- scheduler
volumes:
- ./dags:/usr/local/airflow/dags
- /home/wrf/MET:/MET
- /home/wrf/prepbufr:/prepbufr
- /home/wrf/fcst:/fcst
- /home/wrf/netcdf:/netcdf
- /home/wrf/ptstat:/ptstat
# Uncomment to include custom plugins
# - ./plugins:/usr/local/airflow/plugins
environment:
$ docker-compose -f docker-compose-CeleryExecutor.yml up -d
Creating docker-airflow_redis_1 ... done
Creating docker-airflow_postgres_1 ... done
Creating docker-airflow_flower_1 ... done
Creating docker-airflow_webserver_1 ... done
Creating docker-airflow_scheduler_1 ... done
Creating docker-airflow_worker_1 ... done
open http://10.86.41.176:8080/ in a web browser
在 airflow_worker_1 中产生 ssh public key，拷入 IBM 机器，这样在 docker worker 中可以无密码访问 IBM
$ docker container exec -it docker-airflow_worker_1 bash $ airflow@d850a4897e8b:~/.ssh$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/usr/local/airflow/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /usr/local/airflow/.ssh/id_rsa. Your public key has been saved in /usr/local/airflow/.ssh/id_rsa.pub. The key fingerprint is: SHA256:Q148sag+xEFGGmeo3K0bOljoRLfVhVvyzoeATmHOmLU airflow@d850a4897e8b The key's randomart image is: +---[RSA 2048]----+ | .o* . . | | .@ o = o | | . oO.* O = | | .o+.E.O o . | |... =.+ S . | |....o+ = . | |oo . oo . | |..o . . | | . | +----[SHA256]-----+ $ cd ~/.ssh $ ssh-copy-id -i id_rsa.pub hailang@10.86.29.22 $ ssh hailang@10.86.29.22 # test the keyless login