Downloaded 15 times
![{ “name”: “Padraig”, “job”: “engineer”, “twitter”: “@Podgeypoos79”, “extra”: [ “NodeSchool organiser”, “LeanCoffee organiser”, “Serverlesslab.com founder” ] }](https://image.slidesharecdn.com/buildingaserverlesscompanyonaws-codeeuropewroclaw13dec2017-171213165510/75/Building-a-serverless-company-on-AWS-lambda-and-Serverless-framework-4-2048.jpg)
![{ “name”: “Luciano”, “job”: “engineer”, “twitter”: “@loige”, “Website”: “loige.co” “side-projects”: [ “Node.js Design Patterns”, “Fullstack Bulletin”, “Serverlesslab.com founder” ] }](https://image.slidesharecdn.com/buildingaserverlesscompanyonaws-codeeuropewroclaw13dec2017-171213165510/75/Building-a-serverless-company-on-AWS-lambda-and-Serverless-framework-5-2048.jpg)
![S3 events: filename encoding Space replaced with "+" & URL encodeds3://podge-toys Podge's Unicorn.png { "Records": [{ "s3": { "object": { "key": "Podge%27s+Unicorn.png" } } }] } const middy = require('middy') const { s3KeyNormalizer } = require('middy/middlewares') middy((event, context, cb) => { console.log(event.Records[0].s3.object.key) // Podge's Unicorn }).use(s3KeyNormalizer())](https://image.slidesharecdn.com/buildingaserverlesscompanyonaws-codeeuropewroclaw13dec2017-171213165510/75/Building-a-serverless-company-on-AWS-lambda-and-Serverless-framework-54-2048.jpg)
Uploaded byLuciano Mammino
Building a serverless company on AWS lambda and Serverless framework
The document discusses the experiences of building a serverless company on AWS, focusing on Planet 9 Energy's approach to automating energy trading and transparency in billing. The authors cover their technology stack, organization of code, production paths, and lessons learned throughout the process. They also highlight the advantages of using serverless architectures, including reduced operational overhead and quicker feature development.
Building a serverless company on AWS lambda and Serverless framework
- 1. Building a serverless companyon AWS Padraig O'Brien @Podgeypoos79 Luciano Mammino @loige
- 2. What we willcover - Planet 9 Energy and the problem we are solving - What is serverless? - Our technology stack - How our code is organised - Path to production - Gotchas and things we learned - The Future
- 3.
- 4. { “name”: “Padraig”, “job”: “engineer”, “twitter”:“@Podgeypoos79”, “extra”: [ “NodeSchool organiser”, “LeanCoffee organiser”, “Serverlesslab.com founder” ] }
- 5. { “name”: “Luciano”, “job”: “engineer”, “twitter”:“@loige”, “Website”: “loige.co” “side-projects”: [ “Node.js Design Patterns”, “Fullstack Bulletin”, “Serverlesslab.com founder” ] }
- 6. Electricity suppliers: Do youtrust them?
- 7. Technology adoption byindustry Source: BCG, Boston Consulting Group, 2016
- 8. The numbers ● 17520half hours in a year. ● 30 line items per half hour. ● 6 revisions of that data. ● ~ 3 million data points (year × meter point)
- 9. See your billdown to the half hour
- 10.
- 11. Planet9Energy ● ESB fundedstartup (25 people) ● UK energy supplier. ● Focus on I & C customers.
- 14.
- 15.
- 16. - We arelazy - We want as little manual operational work - We are full stack engineers with T/E profiles AS engineers
- 18. What is aLambda? - Function as a service (FAAS) in AWS - Pay for invocation / processing time - Virtually “infinite” auto-scaling - Focus on business logic, not on servers Daaa!
- 19. Lambdas as micro-services - Eventsare first-class citizens - Every lambda scales independently - Agility (develop features quick and in an isolated fashion) Classic micro-services concerns - Granularity (how to separate features? BDD? Bounded Contexts?) - Orchestration (dependencies between lambdas, service discovery…)
- 20. Anatomy of aLambda in Node.js
- 21. Some use cases -REST over HTTP (API Gateway) - SNS messages, react to a generic message - Schedule/Cron - DynamoDB, react to data changes - S3, react to files changes - IoT
- 22. HTTP REQUEST -API Call POST /path/to/resource?foo=bar { “test”: “body” }
- 23.
- 24.
- 25. Anatomy of Serverless.yml Serverless.yml (2/2) Definingfunctions and events
- 26.
- 27.
- 28. Iteration 1 Review -Dynamodb - low ops overhead but only good for simple read patterns and no good backup solution. - Redshift - epic at aggregation but limited to 50 or so connections. - JAWS - 1.x was completely different so we had to re-write (almost) everything
- 29.
- 30. Iteration 2 Review ●Cassandra replaced redshift. ● Postgres RDS is a lot more flexible than dynamoDB ● Ansible is very good for provisioning VMs. ● Rundeck was used for runbook automation for deploying Lambdas.
- 31.
- 32. Current iteration ● Definedcustom VPC, Yay we are (more) secure. ● Dropped Cassandra. ● Dropped Rundeck, replaced it with parameter store and Jenkins. ● Started using Terraform.
- 33.
- 34. How our services areorganised
- 35. ● A function(not a service) is the natural level of granularity! ● How to identify and structure services? ● How to connect services? ● How many repositories? ● How to deploy? ● Versioning? ● When and how to share code? Iteration 1
- 36. ● Proper servicedesign using methodologies like Domain Driven Design ● Find the bounded context of each service ● Integration through message passing (events / APIs) ● Put everything related to a service into one repo Service 2 Service 3 Service 1 Iteration 2
- 37. ● Terraform code:define infrastructure needed by the service (VPC, database, keys, S3 buckets, etc.) ● Database code: Migrations and seeds (Using knex.js) ● Application code: A Serverless framework project defining Lambdas and events needed by the service Current code layout
- 38.
- 39. Develop locally ● Developlocally on our laptops. ● PostgreSQL on docker. ● Plugins from Serverless to “mimic” API Gateway etc. ● Git commit all the things to branch. ● Pull request. ● Integrate to master. ● Jenkins takes care of everything else (more or less).
- 40. Our CI (Jenkins): ●Run tests ● Build the project ● Updates the infrastructure (Terraform) ● Updates the database (Knex) ● Deploy lambdas (Serverless framework) ● We have a stop-gate with manual approval before it goes to production We we integrate to master
- 42.
- 43. Lots of codeis repeated in every lambda (event, context, callback) => { // decrypt environment variables with KMS // deserialize the content of the event // validate input, authentication, authorization // REAL BUSINESS LOGIC (process input, generate output) // validate output // serialize response // handle errors } BOILERPLATE CODE BOILERPLATE CODE
- 44. middy.js.org The stylish Node.jsmiddleware engine for AWS Lambda
- 45. const middy =require('middy') const { middleware1, middleware2, middleware3 } = require('middy/middlewares') const originalHandler = (event, context, callback) => { /* your pure business logic */ } const handler = middy(originalHandler) handler .use(middleware1()) .use(middleware2()) .use(middleware3()) module.exports = { handler } ● Business logic code is isolated: Easier to understand and test ● Boilerplate code is written as middlewares: ○ Reusable ○ Testable ○ Easier to keep it up to date
- 46. Large services ● serverless-plugin-split-stacks ○migrates the RestApi resource to a nested stack ● Template format error: Number of resources, 214, is greater than the maximum allowed, 200
- 47. API Gateway &Lambda size limits ● 128 K payload for async event invocation ● 10 MB payload for response ● Don’t find these limits when using sls webpack serve
- 48. API Gateways events consthandler = (event, context, callback) { console.log(event.queryStringParameters.name) // … } It will output "me" https://myapi.me?name=me { "requestContext": { … }, "queryStringParameters": { "name": "me" }, "headers": { … } }
- 49. API Gateways events consthandler = (event, context, callback) { console.log(event.queryStringParameters.name) // … } https://myapi.me (no query string!) { "requestContext": { … }, "headers": { … } } (no queryStringParameters key!) TypeError: Cannot read property 'name' of undefined undefined
- 50. API Gateways events consthandler = (event, context, callback) { if (event.queryStringParameters) { console.log(event.queryStringParameters.name) } // or console.log(event.queryStringParameters ? event.queryStringParameters.name : undefined } Api Gateway proxy event normalizer middleware is coming to Middy! MOAR boilerplate!
- 51. API Gateways customdomain. ● Serverless does not provide custom domain name mapping ● Has to be done in cloudformation ● There is a plugin. Serverless-plugin-custom-domain Serverless-domain-manager
- 52. Disk usage matters ●50 MB if deploying directly. ● 250 if going from S3. ● We use Node.js, Webpack and tree shaking help us (serverless webpack plugin) ● 75GB for entire region, covers all lambdas and versions of lambdas, you might need a janitor lambda...
- 53. Node.js Event loop ●We use postgres and connection pooling ● Event loop will never become empty ● Use Middy! :) const middy = require('middy') const {doNotWaitForEmptyEventLoop} = require('middy/middlewares') const handler = middy((event, context, cb) => { // ... }).use(doNotWaitForEmptyEventLoop())
- 54. S3 events: filenameencoding Space replaced with "+" & URL encodeds3://podge-toys Podge's Unicorn.png { "Records": [{ "s3": { "object": { "key": "Podge%27s+Unicorn.png" } } }] } const middy = require('middy') const { s3KeyNormalizer } = require('middy/middlewares') middy((event, context, cb) => { console.log(event.Records[0].s3.object.key) // Podge's Unicorn }).use(s3KeyNormalizer())
- 56. Serverless Aurora Building aserverless company on AWS lambda https://aws.amazon.com/rds/aurora/serverless/
- 57. Blue Green Deploys Buildinga serverless company on AWS lambda http://docs.aws.amazon.com/lambda/latest/dg/automating-updates-to-serverless-apps.html
- 58. A retrospective 2 yearsafter... ● Learning how to do serverless right took a while (as learning any other new tech) ● We never received a call at 2AM! ● Our tech bill is extremely small (apart for RDS!) ● We definitely don't regret the choice :)
- 59. Recap Building a serverlesscompany on AWS lambda We are hiring! @loige @Podgeypoos79 Thank you