Deployment to gce for continuous integration by liamsi · Pull Request #607 · google/keytransparency

liamsi · 2017-06-12T17:40:22Z

Work in progress PR created early to trigger travis (on the original repo, not my fork)
Will eventually resolve #601

Added kubernetes configuration and a deploy script which:

Builds all docker images
cleans up the default namespace in a particular gcloud project (delete all services, pods/deployments)
pushes all trillian images to the project's container registry
Starts a trillian deployment on gce
pushes all keytransparency docker images to gcloud
creates a log-/map-tree and sets the IDs in the keytransparency kubernetes file as environment vars
runs keytransparency in gce using kubernetes

This script can be used locally or by travis. Travis will run the script every time after pull gets merged into master.

Resolves #601
Also resolves #615

codecov-io · 2017-06-12T19:32:26Z

Codecov Report

Merging #607 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #607   +/-   ##
=======================================
  Coverage   56.76%   56.76%           
=======================================
  Files          35       35           
  Lines        2489     2489           
=======================================
  Hits         1413     1413           
  Misses        787      787           
  Partials      289      289

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 524573a...a5e8424. Read the comment docs.

gdbelvin · 2017-06-19T10:54:28Z

+  directories:
+    - "$HOME/google-cloud-sdk/"
+env:
+  - DOCKER_COMPOSE_VERSION="1.13.0" PATH=$PATH:${HOME}/google-cloud-sdk/bin CLOUDSDK_CORE_DISABLE_PROMPTS=1 PROJECT_NAME=transparent-keyserver


Split the ENV variables onto separate lines if possible

If you split them in several lines/items one build will be triggered per line: https://docs.travis-ci.com/user/environment-variables/#Defining-public-variables-in-.travis.yml
But it seems

env: global: # separate line per VAR

does what we want. ~~Will do.~~ done

gdbelvin · 2017-06-19T10:59:38Z


 dist: trusty
-sudo: false
+sudo: required


From the travis docs, it looks like we can set sudo to false and just get the docker binaries.

Docker is installed. We use the official Docker apt repository, so you can easily install another version with apt if needed. When sudo: required is specified, Docker is installed as a service. When sudo: false is specified, the Docker binaries are available for local use.

docker build needs the docker daemon to be running. Unfortunately, it looks like we can not do without sudo: required.

The only viable alternative I've found: Instead of running docker-compose build on travis (and therefore needing sudo to run the daemon), we could get rid of docker-compose, run only docker build instead and do this in gcloud instead of travis with gcloud docker -- build <image-to-be-build>. Se: https://cloud.google.com/sdk/gcloud/reference/docker
This could be faster, too.

gdbelvin · 2017-06-19T11:04:31Z

+env:
+  - DOCKER_COMPOSE_VERSION="1.13.0" PATH=$PATH:${HOME}/google-cloud-sdk/bin CLOUDSDK_CORE_DISABLE_PROMPTS=1 PROJECT_NAME=transparent-keyserver
+before_install:
+  - sudo rm /usr/local/bin/docker-compose


Could you try using the travis apt addon to install the latest version?

Switched to using pip now which is less hacky (apt doesn't work for docker-compose).

gdbelvin · 2017-06-19T11:05:16Z


 after_success:
  - bash <(curl -s https://codecov.io/bash)
+  # The last two commands are only needed for the deploy step (which will only


If this is only needed in the deploy step, could it moved to the deploy step?

gdbelvin · 2017-06-19T11:05:47Z

+  - openssl aes-256-cbc -K $encrypted_a1eb99cfc21e_key -iv $encrypted_a1eb99cfc21e_iv -in credentials.tar.gz.enc -out credentials.tar.gz -d
+  - if [ ! -d "$HOME/google-cloud-sdk/bin" ]; then rm -rf $HOME/google-cloud-sdk; curl https://sdk.cloud.google.com | bash; fi
+  # needed for envsubst command used in deploy script:
+  - sudo apt-get install gettext


install via apt addon

Switched to sed as you suggested (thanks).

gdbelvin · 2017-06-19T11:06:14Z

 ENV LOG_ID=0 \
    LOG_URL=localhost:8090 
 ENV GOOGLE_APPLICATION_CREDENTIALS=genfiles/service_key.json
-ENV VERBOSE=0


We're deleting the code behind verbose, I think... leave this as 0

gdbelvin · 2017-06-19T11:07:25Z

+  # The last two commands are only needed for the deploy step (which will only
+  # run on master. The first fails on PRs from forks (but the build passes), the
+  # 2nd only does sth. if the cache is empty.
+  - openssl aes-256-cbc -K $encrypted_a1eb99cfc21e_key -iv $encrypted_a1eb99cfc21e_iv -in credentials.tar.gz.enc -out credentials.tar.gz -d


Please pick a more descriptive name for the credential file

Changed to: client_secret-and-service_key.tar.gz

gdbelvin · 2017-06-19T11:10:33Z

      DB_USER: test
      DB_PASSWORD: zaphod
-      GOOGLE_APPLICATION_CREDENTIALS: /genfiles/service_key.json
-      KEY: /kt/server.key


leave these. It's better to have them explicitly set

The trillian-log-signer doesn't need them. Copied from its Dockerfile:

/go/bin/trillian_log_signer \ --mysql_uri="${DB_USER}:${DB_PASSWORD}@tcp(${DB_HOST})/${DB_DATABASE}" \ --http_endpoint="$HOST:$HTTP_PORT" \ --sequencer_guard_window="$SEQUENCER_GUARD_WINDOW" \ --force_master="$FORCE_MASTER" \ --alsologtostderr

gdbelvin · 2017-06-19T11:14:04Z

+    spec:
+      containers:
+      - name: kt-server
+        image: us.gcr.io/transparent-keyserver/keytransparency-server


transparent-keyserver should be keytransparency or key-transparency

I've created a new service-key in this project but authenticating using this key does not grant the permissions to create a repository for the container registry there. Either one of the owners need to create a key-file, download it and encrypt it after we merged this pull or you need to grant me access to do so.

gdbelvin · 2017-06-19T11:15:02Z

+          - containerPort: 8080
+            name: json-grpc
+        env:
+            - name: LOG_URL


can name and value be put on the same line?

This seems to be the default (as in the documentation). But I've tried:
- name: LOG_URL value: trillian-log:8090
and get an error: error: error converting YAML to JSON: yaml: line 24: mapping values are not allowed in this context

gdbelvin

Let's start by cleaning up the travis config

liamsi · 2017-06-19T14:35:43Z

All comments addressed: PTAL

- update gitignore - not yet working kubernetes db deployment

comments, simplified cred. file, fix sed,

…time/#ParseDuration

gdbelvin · 2017-06-20T08:30:59Z

+      curl https://sdk.cloud.google.com | bash; fi
+    - openssl aes-256-cbc -K $encrypted_a1eb99cfc21e_key -iv $encrypted_a1eb99cfc21e_iv -in service_key.tar.gz.enc -out service_key.tar.gz -d
+    - tar -xzf service_key.tar.gz
+    - (cd scripts && ./deploy.sh)


should this be ./scripts/deploy.sh ?

gdbelvin · 2017-06-20T08:32:47Z

+    - docker-compose --version
+    - if [ ! -d "$HOME/google-cloud-sdk/bin" ]; then rm -rf $HOME/google-cloud-sdk;
+      curl https://sdk.cloud.google.com | bash; fi
+    - openssl aes-256-cbc -K $encrypted_a1eb99cfc21e_key -iv $encrypted_a1eb99cfc21e_iv -in service_key.tar.gz.enc -out service_key.tar.gz -d


Double check that service_key.tar.gz.enc is still correct. It doesn't look like it matches with the .enc file in the PR.

Consider naming this travis_secrets.tar.gz.enc

gdbelvin · 2017-06-20T08:33:09Z


 ENV MIN_SIGN_PERIOD=5s \
-    MAX_SIGN_PERIOD=1d
+    MAX_SIGN_PERIOD=24h


gdbelvin · 2017-06-20T08:33:29Z


-ADD . /go/src/github.com/google/keytransparency
+ADD trillian/testdata/* /trillian/
+


remove the newline

gdbelvin · 2017-06-20T08:34:23Z

+[Google Compute Engine](https://cloud.google.com/sdk/gcloud/), run the following 
+command to create, upload, and run all containers:
+```
+cd scripts && ./deploy.sh


Is it possible to remove the cd?

gdbelvin · 2017-06-20T08:57:52Z

@@ -0,0 +1,6 @@
+FROM mysql:5.7


What do you think about moving these files to
deploy/mysql-trillian
deploy/kubernetes

?

gdbelvin · 2017-06-20T08:58:44Z

+################################################################################
+# Following assumptions are made by this script:                               #
+# * gcloud, docker, and docker-compose is installed                            #
+# * it is called from $GOPATH/src/github.com/google/keytransparency/scripts    #


Could we it's assumption to be called from the root directory
$GOPATH/src/github.com/google/keytransparency ?

gdbelvin · 2017-06-20T10:13:07Z

 genfiles/*
 .keytransparency-scts.dat
 .keystore
+credentials.tar.gz


Update filename

gdbelvin · 2017-06-20T10:16:30Z

+  # Work around some git permission issues on linux:
+  chmod a+r ../../trillian/storage/mysql/storage.sql
+  # Create and push a fresh db image (with schema created)
+  docker build -t  us.gcr.io/${PROJECT_NAME}/db -f ../kubernetes/mysql-trillian/Dockerfile ../..


See if you can get docker-compose to do this for you.

Set the tag

it's ok to hardcode the project

See if you can publish the containers

All done besides publishing. gcloud doesn't seem to support docker-compose. What we could do though, is make docker login into gcloud: https://stackoverflow.com/a/29311300
Do you think that's safe and worth trying?

gdbelvin · 2017-06-20T10:22:03Z

+  "trillian_log_signer" "keytransparency-signer")
+  for DOCKER_IMAGE_NAME in "${images[@]}"
+  do
+    docker tag ${DOCKER_IMAGE_NAME} us.gcr.io/${PROJECT_NAME}/${DOCKER_IMAGE_NAME}


If docker-compose tags things we might not need to do this here.

done (docker-compose now tags with us.gcr.io/key-transparency/)

gdbelvin · 2017-06-20T10:34:38Z

+  kubectl apply -f ../kubernetes/keytransparency-deployment.yml
+}
+
+function initGcloud()


Try doing this in before_deploy:
https://medium.com/google-cloud/continuous-delivery-in-a-microservice-infrastructure-with-google-container-engine-docker-and-fb9772e81da7

gdbelvin · 2017-06-20T10:39:41Z

+  do
+    docker tag ${DOCKER_IMAGE_NAME} us.gcr.io/${PROJECT_NAME}/${DOCKER_IMAGE_NAME}
+    # Push the images as we refer to them in the kubernetes config files:
+    gcloud docker -- push us.gcr.io/${PROJECT_NAME}/${DOCKER_IMAGE_NAME}


docker-compose push?
https://docs.docker.com/compose/reference/push/

Doesn't look like docker-compose is supported by gcloud.

gdbelvin · 2017-06-20T10:40:42Z

+  done
+}
+
+function pushKTImgs()


Do we need to have a separate push command for KT?
Is it possible to just push everything together?

As discussed: As we need to wait for the trillian services to be up and running, we can as well spent this time doing sth. we need to do anyways.

gdbelvin · 2017-06-20T10:46:20Z

+    let COUNTER+=1
+    # get the currentl trillian-map pod:
+    MAPSRV=$(kubectl get pods --selector=run=trillian-map -o jsonpath={.items[*].metadata.name});
+    LOG_ID=$(echo 'go run $GOPATH/src/github.com/google/trillian/cmd/createtree/main.go --admin_server=localhost:8090 --pem_key_path=testdata/log-rpc-server.privkey.pem --pem_key_password="towel" --signature_algorithm=ECDSA --tree_type=LOG' | kubectl exec -i $MAPSRV -- /bin/sh )


is localhost going to be a problem?

No, the command will be run directly on the pod (at the very end: kubectl exec -i $MAPSRV -- /bin/sh
But we can change to trillian-map for consistency, if you prefer (which will resolve to the same machine as localhost).

liamsi · 2017-06-20T13:46:42Z

Adressed all comments, and tested the deploy script locally. PTAL

gdbelvin

LGTM!

liamsi added the WIP label Jun 12, 2017

liamsi changed the title CI Deployment to gce for continuous integration Jun 15, 2017

liamsi removed the WIP label Jun 15, 2017

liamsi requested review from cesarghali and gdbelvin June 15, 2017 14:46

liamsi mentioned this pull request Jun 16, 2017

Prometheus #589

Merged

liamsi requested review from RJPercival and removed request for cesarghali June 16, 2017 12:54

liamsi force-pushed the kubernetes branch from 5629e8b to 4e4a006 Compare June 16, 2017 13:05

gdbelvin reviewed Jun 19, 2017

View reviewed changes

gdbelvin suggested changes Jun 19, 2017

View reviewed changes

liamsi force-pushed the kubernetes branch 2 times, most recently from ceb140f to 7483212 Compare June 19, 2017 14:45

liamsi added 6 commits June 19, 2017 15:56

non-interactive version of prepare_server.sh

73aea86

trigger travis build

474d757

deploy dummy script

09c3d30

travis

7a15dcc

Work in progress: add encrypted credentials (and vars to travis)

51df996

- update gitignore - not yet working kubernetes db deployment

Work in progress: updated vars in travis file

adaeb03

remove obsolete lines

d3afac2

liamsi force-pushed the kubernetes branch from 1c8927a to d3afac2 Compare June 19, 2017 16:03

liamsi added 4 commits June 19, 2017 17:14

Updated project-name: s/transparent-keyserver/key-transparency/

9a20557

change zone according to new project (key-transparency), added some

b5533cd

comments, simplified cred. file, fix sed,

d is not a valid unit for time.ParseDuration: https://golang.org/pkg/…

d2d1116

…time/#ParseDuration

Documented assumptions made by the script in a comment

72f9a1d

liamsi force-pushed the kubernetes branch from 8575bd7 to 72f9a1d Compare June 19, 2017 20:29

liamsi requested review from cesarghali and gdbelvin and removed request for cesarghali June 19, 2017 21:38

gdbelvin reviewed Jun 20, 2017

View reviewed changes

Add correct credentials file

8b67bd6

gdbelvin suggested changes Jun 20, 2017

View reviewed changes

liamsi added 2 commits June 20, 2017 14:43

Correct credentials file, review comments

65e8e88

Merge branch 'master' into kubernetes

83b25c2

liamsi added 2 commits June 20, 2017 14:53

Added a line about kubectl proxy

7ca003d

Added prometheus to kubernetes and fixed a typo

a5e8424

liamsi force-pushed the kubernetes branch from 609402f to a5e8424 Compare June 20, 2017 14:13

gdbelvin approved these changes Jun 20, 2017

View reviewed changes

liamsi merged commit 9c324f6 into google:master Jun 20, 2017

This was referenced Jun 20, 2017

Run on Google Cloud #397

Closed

Update Setup Scripts #516

Closed


		ADD . /go/src/github.com/google/keytransparency
		ADD trillian/testdata/* /trillian/

Conversation

liamsi commented Jun 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-io commented Jun 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamsi Jun 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gdbelvin Jun 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamsi Jun 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gdbelvin left a comment

Choose a reason for hiding this comment

Uh oh!

liamsi commented Jun 19, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liamsi commented Jun 12, 2017 •

edited

Loading

codecov-io commented Jun 12, 2017 •

edited

Loading

liamsi Jun 19, 2017 •

edited

Loading

gdbelvin Jun 19, 2017 •

edited

Loading

liamsi Jun 19, 2017 •

edited

Loading

liamsi commented Jun 19, 2017 •

edited

Loading