Download as PDF, PPTX
Uploaded byJorge Hidalgo
Multilanguage Pipelines with Jenkins, Docker and Kubernetes (Oracle Code One San Francisco 2018)
The document discusses the implementation of multilanguage pipelines using Jenkins, Docker, and other tools, emphasizing automation and DevOps principles. It outlines the pipeline process from environment preparation to code inspection, detailing each testing stage and the importance of code quality measures. The authors, Jorge Hidalgo and Julio Palma, encourage collaboration and provide online resources for further exploration.
Multilanguage Pipelines with Jenkins, Docker and Kubernetes (Oracle Code One San Francisco 2018)
- 1. INFINITE POSSIBILITIES MULTILANGUAGE PIPELINES WITH JENKINS, DOCKER ANDFRIENDS JORGE HIDALGO @_deors JULIO PALMA @restalion ORACLE CODE ONE SAN FRANCISCO 2018
- 2. WHO WE ARE JorgeHidalgo @_deors Senior Technology Architect Accenture Technology Spain Global Java Lead Spain Advanced Technology Center Custom Engineering & Architecture Lead MálagaJUG co-lead Julio Palma @restalion Technology Architect Accenture Technology Spain Spain Advanced Technology Center Backend & Cloud Architecture Chapter Lead MálagaJUG member Honorable mentions: Vicente González and the whole Architecture & DevOps Squad at Spain ATC INFINITE POSSIBILITIES
- 3. TL;DR All code andpipeline examples can be found online in these repositories: https://github.com/deors/deors-demos-petclinic https://github.com/restalion/python-jenkins-pipeline Don’t hesitate to ask, log issues or send your pull requests – HACKERS WELCOMED! INFINITE POSSIBILITIES
- 4. AT THE BEGINNING... ...ONLYPAIN AND BREAKING BUILDS
- 5.
- 6.
- 7. ADOP THE DEVOPS PLATFORM Share success Easy accessto tooling platform accenture.github.io/adop-docker-compose/ open source (Apache 2 license)
- 8. ADOP IS... NGINX web /proxy server OpenLDAP directory server ELK log monitoring all tools packaged as Docker containers Sensu - Uchiwa infra/app monitoring Jenkins continuous integration / delivery Gerrit Git repositories / code review SonarQube code inspection / quality dashboard Selenium web browser testing Nexus binary artefact repositories
- 9. ADOP DRIVES OURJOURNEY TO DEVOPS AND A NEW WAY OF WORKING AS TEAMS…
- 10. ...BUT THERE ISSTILL MORE TO DO
- 11. AGILE DEVOPS ARCHITECTURE VALUE PRIORITISED DELIVER–REFLECT–IMPROVE DECOUPLED ARCHITECTURE RESILIENTBY DESIGN AUTOMATED PIPELINES BLENDED TEAMS PRODUCT AND CUSTOMER CENTRIC BUILT-IN QUALITY DECENTRALISED EXECUTION
- 12.
- 13.
- 14.
- 15.
- 16. K8S - HAVEIT YOUR WAY
- 17.
- 18.
- 19.
- 20.
- 21. AND LET THE PLATFORM CAREOF ANYTHING ELSE
- 22. STANDARDIZATION TAKES FORM AS ANAGREED PIPELINE SCHEMA “Behavior-over-tool”
- 23. BRIEF HISTORY OFA PIPELINE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 24. • Prepare environmentfor pipeline execution • Builder image with basic stuff: JDK, Maven, npm, Node.js, Python… • Install all needed dependencies • Python: pip install • Node.js: npm install • Java: Maven, either explicit or along the pipeline ENVIRONMENT PREPARATION ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 25. • Transform sourcecode to binaries • In some cases compilation is replaced by other activities like uglyfication, minification, transpilation… • For interpreted languages, when viable, check for syntax errors COMPILE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 26. • Run projectunit tests • And never forget: gather code coverage metrics • Coverage will highlight which code is not being exercized • “EXERCIZED” does not imply “TESTED” UNIT TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 27. • In anutshell: validates whether your tests are actually testing anything • Mutation tests will uncover: • Untested use cases • Test cases without assertions • Also, test cases with insufficient or wrong assertions MUTATION TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 28. • Package allapplication resources: • Binaries • Static resources • Application configuration which does not depend on the environment, e.g. literals, error codes… • Environment specific configuration must be externalized (package once run everywhere) PACKAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 29. • Put togetherall pieces which constitute our application • Base image • Build artifacts • Dependencies • Base images have their own pipeline • Typically maintained by someone else (vendor, community, other team…) BUILD DOCKER IMAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 30. • The purposeof this step is to ensure that our application runs and to make it available for tests • If application depends on other applications (database, message queue, other services…) they will be executed all together (docker compose, application.yml…) • The test environment is not lasting — will be available only to tests during the life of the pipelines • Should allow for concurrent execution (fast feedback on a continuous integration build-per-commit approach) RUN DOCKER IMAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 31. • Run testsrequiring the application running and its dependencies • Integration tests may include: UI tests, API tests… • Should be user-oriented whenever possible • Test data preparation is performed at this step too • For web applications use vulnerability scan agent/proxy • And remember: gather code coverage metrics INTEGRATION TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 32. • Exercize theapplication under load • Response times are important but will also uncover errors due to concurrency usage • Quality gate should be asserting both metrics • Should be user-oriented too — reuse integration tests whenever possible • Use of dynamic performance monitor agents is also advised PERFORMANCE TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 33. • Assessing applicationsecurity is not negotiable • But usually we forget that most vulnerabilities, if not all, come from external dependencies • Some tools for this step include: • OWASP Dependency Check • Snyk • Black Duck • Nexus Lifecycle • PyUp Safety DEPENDENCY VULNERABILITY TESTS ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 34. • Static codeanalysis: • Adherence to coding standards • Possible bugs • Code smells and best practices • Security vulnerabilities • Documentation and complexity metrics • Code duplication • This step usually happens at the same time that the overall quality gate (SonarQube) • Ideally this should happen earlier in the pipeline — however we assume that this is happening before code is pushed and/or merged CODE INSPECTION ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 35. • After thefinal quality gate, if all pass, the image is pushed to a shared registry • Once pushed it is available to be used by: • Other applications that depends on this image • To be promoted to a stage environment • All the way up to production PUSH DOCKER IMAGE ENVIRONMENT PREPARATION COMPILE UNIT TESTS MUTATION TESTS PACKAGE BUILD DOCKER IMAGE RUN DOCKER IMAGE INTEGRATION TESTS PERFORMANCE TESTS DEPENDENCY VULNERABILITY TESTS CODE INSPECTION PUSH DOCKER IMAGE
- 36.