Open stack ocata summit enabling aws lambda-like functionality with openstack swift and openwhisk

Enabling AWS Lambda-like functionality with OpenStack Swift and OpenWhisk Shaun Murakami, Senior Cloud Architect, @stmuraka Andrew Bodine, Software Engineer, @bodine_andrew

Development in the Cloud Bare metal Virtual machines Containers Functions Decreasing concern (and control) over stack implementation Increasingfocusonbusinesslogic

Serverless can handle many cloud native 12 Factors app I Codebase Handled by developer (Manage versioning of functions on their own) II Dependencies Handled by developer, facilitated by serverless platform (Runtimes and packages) III Config Handled by platform (Environment variables or injected event parameters) IV Backing services Handled by platform (Connection information injected as event parameters) V Build, release, run Handled by platform (Deployed resources are immutable and internally versioned) VI Processes Handled by platform (Single stateless containers often used) VII Port binding Handled by platform (Actions or functions are automatically discovered) VIII Concurrency Handled by platform (Process model is hidden and scales in response to demand) IX Disposability Handled by platform (Lifecycle is hidden from the user, fast startup and elastic scale is prioritized) X Dev/prod parity Handled by developer (The developer is the deployer. Scope of what differs is narrower) XI Logs Handled by platform (Developer writes to console.log, platform handles log streaming) XII Admin processes Handled by developer (No distinction between one off processes and long running)

Event driven programming Execute app logic in response to database triggers Execute app logic in response to sensor data Execute app logic in response to cognitive trends Execute app logic in response to scheduled tasks Provide easy server-side backend for mobile app

Amazon Lambda • An event-driven compute service where AWS Lambda runs your code in response to events, such as changes to data or in response to HTTP requests. http://docs.aws.amazon.com/lambda/latest/dg/welcome.html http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html

OpenWhisk An open source cloud platform that executes code in response to events https://github.com/openwhisk/openwhisk

OpenWhisk supports many growing workloads OpenWhisk can help power various mobile, web and IoT app use cases by simplifying the programming model of orchestrating various services using events without a dedicated backend. Digital app workloads Big Data/Analytics pipeline Complex data pipelines for Big Data/Analytics tasks can be scripted using changes in data services or streams for near real-time analytics and feedback. DevOps and infrastructure as code OpenWhisk can be used to automate DevOps pipelines based on events triggered from successful builds, or completed staging, or a go-live event. Microservices builder OpenWhisk can be used to easily build microservices given the footprint and programming model desired by microservices

OpenWhisk Programming model Developers use Rules to… associate how Actions handle events emitted by Triggers generated by services R T A:=

riggers A class of events that can occur T Social events Data changes Device readings Location updates User input

ctions Code that runs in response to an event (i.e., an event handler) A Supports a variety of languages such as JavaScript, Python, Java, and Swift ... function main(params) { return { message: 'Hello, ' + params.name + ' from ' + params.place }; }; ... even as a Docker container Actions can be chained together AA := A1 + A2 + A3 AB := A2 + A1 + A3 AC := A3 + A1 + A2

OpenWhisk can implement RESTful microservices Send HTTP request HTTP GET app.com/customers 1 Invoke OpenWhisk action get-customers Browser Mobile App Web App 2 JS Swift Docker … OpenWhisk API Proxy

OpenStack Swift Middleware • Swift uses middleware to add custom behaviors. • The majority of Swift middleware is applied to the Proxy Server. • Python WSGI Middleware (or just “middleware”) can be used to “wrap” the request and response of a Python WSGI application (i.e. a webapp, or REST/HTTP API), like Swift’s WSGI servers (proxy-server, account-server, container-server, object-server) http://docs.openstack.org/developer/swift/development_middleware.html http://docs.openstack.org/admin-guide/objectstorage-arch.html

OpenStack Swift System Metadata • Provides a means to store potentially private custom metadata with associated Swift resources in a safe and secure fashion. • May be set on accounts and containers by including headers with a PUT or POST request. • Maybe set on objects using only PUT requests. • Takes the form of X-<type>-Sysmeta-<key>: <value> • <type> depends on the resources type (i.e. Account, Container, Object) • <key> and <value> are set by trusted code running in a Swift WSGI Server http://docs.openstack.org/developer/swift/development_middleware.html#sysmeta

AWS S3 + Lambda 1. A user uploads an object to the source bucket in Amazon S3 (object-created event). 2. Amazon S3 detects the object-created event. 3. Amazon S3 publishes the s3:ObjectCreated:* event to AWS Lambda by invoking the Lambda function and passing event data as a function parameter. 4. AWS Lambda executes the Lambda function by assuming the execution role that you specified at the time you created the Lambda function. 5. From the event data it receives, the Lambda function knows the source bucket name and object key name. The Lambda function reads the object and creates a thumbnail using graphics libraries, and saves it to the target bucket http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

OpenStack Swift + OpenWhisk JS Swift Docker … OpenWhisk APIProxy 1. A user uploads an object to a OpenStack Swift Container. 2. OpenStack Swift detects an object has been modified in the Container. 3. The Swift Webhook middleware triggers an OpenWhisk Action by invoking the OpenWhisk API specified as the container’s webhook URL. 4. The OpenWhisk credentials are validated against the OpenWhisk authorization service. 5. From the request payload, the OpenWhisk action knows the source container and object name. The OpenWhisk Action creates a thumbnail using graphics libraries and saves it to a target container Swift Middleware Source Container Target Container 2 3 4 Whisk Action 1 5

Let’s take a look at how it’s implemented • OpenStack Swift Middleware Code • OpenWhisk Action Code

OpenStack Swift Middleware (webhook.py) class WebhookMiddleware(object): @wsgify def __call__(self, req): req.headers[SYSMETA_WEBHOOK] = req.headers['x-webhook'] req.headers[SYSMETA_WEBHOOK_AUTH] = req.headers['x-webhook-auth'] resp = req.get_response(self.app) if obj and is_success(resp.status_int) and req.method in [’PUT’,’DELETE’]: webhook_req = urllib2.Request(container_info['sysmeta'].get('webhook')) webhook_req.add_header('Authorization', base64.b64encode(container_info['sysmeta'].get('webhook-auth'))) webhook_req.add_data(json.dumps({ ’url’: swiftUrl, ‘token’: req.headers[‘x-auth-token’], ‘container’: container, ‘object’: obj, ‘method’: req.method })) ... try: webhook_resp = urllib2.urlopen(webhook_req).read() …. return resp

Install Webhook Middleware 1. Place webhook.py in middleware of swift-proxy …/swift/common/middleware/webhook.py 2. Update proxy-server.conf … [pipeline:main] pipeline = catch_errors gatekeeper healthcheck ... webhook proxy-server [filter:webhook] paste.filter_factory = swift.common.middleware.webhook:webhook_factory ...

OpenWhisk Action Dockerfile FROM ubuntu:14.04 ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && apt-get install -y curl # Node.js RUN curl -sL https://deb.nodesource.com/setup_4.x | bash - && apt-get install -y nodejs WORKDIR /root/server COPY server . RUN npm install EXPOSE 8080 CMD node ./app.js

Demo Putting it all together…

Future work Swift Asynchronous Event Notification Framework • Support Multiple Notification Use Cases • OpenWhisk / Serverless computing • Cloud inter-service notifications • Metadata search indexing • Resource / traffic monitoring • Common Notification APIs • Enable / disable notifications on an account or container • Allow for user / system level policies • Desire to become a de-facto standard in Swift for notification management • Deployment Flexibility • Can be adapted to multiple message queue technologies • Initial submission will include Kafka and Zaqar • Patch WIP • https://review.openstack.org/388393

Resources • AWS Lambda • Example: http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html • OpenWhisk • Code: https://github.com/openwhisk/openwhisk • On IBM BlueMix: http://www.ibm.com/cloud-computing/bluemix/openwhisk/ • OpenStack Swift Middleware • http://docs.openstack.org/developer/swift/development_middleware.htm • Demo Code • https://github.ibm.com/stmuraka/OpenStackSwift-OpenWhisk

Open stack ocata summit enabling aws lambda-like functionality with openstack swift and openwhisk

More Related Content

What's hot

Viewers also liked

Similar to Open stack ocata summit enabling aws lambda-like functionality with openstack swift and openwhisk

Recently uploaded

Open stack ocata summit enabling aws lambda-like functionality with openstack swift and openwhisk