[BigDL 2.0] Traceable CI/CD Jenkins

[glorysdj]

• Jenkins Slave from Dockerfile
• Jenkins Master from Dockerfile
• Jenkins Job to build images
• Jenkins Auto deployment with Docker images
• Jenkins Conf auto save and backup
• Jenkins Job auto save and backup
• github repo created
• image dependencies/libs determination #34
• migrate jenkins job
• migrate jenkins slave node (Gondolin server)
• migrate jenkins slave node (ppml/k8s/deploy/mac/clx servers/ray/nano)
• release nodes backup
• jenkins master backup
• Automatically create jenkins jobs

[pinggao187]

feature request: Jenkin config should also be control by git and go though code review process intel-analytics/analytics-zoo#411
feature request: Testing environment (dockerfile) should also be controled by git intel-analytics/analytics-zoo#412

[pinggao187]

feature request: Jenkin config should also be control by git and go though code review process intel-analytics/analytics-zoo#411 feature request: Testing environment (dockerfile) should also be controled by git intel-analytics/analytics-zoo#412

request: Update jenkins docker image os to support orca openvino estimator testcase intel-analytics/analytics-zoo#416

[pinggao187]

Already planned and expected to be completed in Q4.
current progress: image dependencies/libs determination intel-analytics/analytics-zoo-internal#34

[pinggao187]

• Jenkins Slave fromDockerfile(11/12(https://github.com/analytics-zoo/cicd/pull/5) add jenkins slave dockerfile intel-analytics/analytics-zoo#2
• Create new Jenkins and migrate jenkins jobs(12/03) Jenkins CI/CD: create new Jenkins and migrate jenkins jobs intel-analytics/analytics-zoo#13
• Jenkins Master from Dockerfile intel-analytics/analytics-zoo-internal#28
• Jenkins Job to build images(12/10)
• Jenkins Auto deployment with Docker images(12/10)
• Jenkins Conf auto save and backup(12/17)
• Jenkins Job auto save and backup(12/17)
• github repo created
• image dependencies/libs determination #5035
• jenkins NB/PR job synchronization(12/03) Jenkins CI/CD: create new Jenkins and migrate jenkins jobs intel-analytics/analytics-zoo#13
• chronos test docker image
• migrate jenkins job
• migrate jenkins slave node (Gondolin server)
• migrate jenkins slave node (ppml/k8s/deploy/mac/clx servers/ray/nano)
• release nodes backup
• jenkins master backup
• Automatically create jenkins jobs

[pinggao187]

There are 7 conda envs in the jenkins test, "py36,py37,py37-horovod-pytorch,py37-horovod-tf,colabtf15,colabtf2,colab-pytorch".
We plan to use requirements-XXX.txt as the dev-ops interface. each conda env corresponds to a docker image.

1.python-requirements(datails(https://github.com/analytics-zoo/cicd/pull/6))
(1).requirements-py36py37.txt
(2).requirements-py37-horovod-pytorch.txt
(3).requirements-py37-horovod-tf.txt
(4).Requirements-colabtf1.txt
(5).Requirements-colabtf2.txt
(6).Requirements-colab-pytorch.txt

2.If you have other requirements besides python, please submit an issue.
3.developer update requirements process

E.g: http://10.112.231.51:18888/job/BigDL-requirements-validation/

4.jenkins slave node update process

[pinggao187]

Another way: create a temporary container: #4