k6 is the best load test tool available, but it's pricey. I wanted to start unlimited VUs (virtual users) without paying a lot of money and found a neat way to do it by spawning many short-lived EC2 instances!
Too many templates
I used a simple user-data script with heredocs. Make sure you have an AMI with Docker installed. You will also need S3 access.
Start by setting up AWS credentials.
# load_test.user-data echo "setting up aws-cli" mkdir -p /home/ec2-user/.aws cat <<EOF > /home/ec2-user/.aws/config [default] output=json region=us-west-1 EOF cat <<EOF > /home/ec2-user/.aws/credentials [default] aws_access_key_id={{ aws-creds.access-key-id }} aws_secret_access_key={{ aws-creds.secret-access-key }} EOF Next, you can write out the load-test and make it executable:
# load_test.user-data echo "writing load test: {{load-test}}" mkdir -p /home/ec2-user/load_test chmod a+rw /home/ec2-user/load_test cat <<EOF > /home/ec2-user/load_test/{{load-test}} {{load-test-content|safe}} EOF We pull the k6 docker image and execute it with a single command.
# load_test.user-data echo "running load test" sudo -u ec2-user docker run --rm \ --ulimit nofile=65536:65536 \ --network host \ -v /home/ec2-user/load_test/:/mnt/load_test:rw \ -i loadimpact/k6 \ run --summary-export=/mnt/load_test/out.json \ /mnt/load_test/{{load-test}} > /dev/null Finally, upload the test results to s3.
# load_test.user-data echo "uploading results" sudo -u ec2-user aws s3api put-object \ --bucket load-test-results \ --key {{uuid}}/{{instance-name}} \ --body /home/ec2-user/load_test/out.json > /dev/null IMPORTANT: You should set a trap function to call systemctl halt to shutdown the EC2 instances. Leaving EC2 instances running would be counter-productive to our goal of saving money.
You may have noticed that I have used templates in the user-data. We can use Selmer to supply all of the important stuff.
(defn build-user-data [aws-creds uuid instance-name load-test load-test-template aws-creds] (selmer/render-file "load_test.user-data" {:aws-creds aws-creds :uuid uuid :instance-name instance-name :load-test (.getName (io/file (io/resource load-test))) :load-test-content (selmer/render-file load-test load-test-template)})) aws-creds should be a map with keys :access-key-id and :secret-access-key. These are your S3 credentials.
You may have noticed that load-test-content is itself a template. That is so we can point the load test to an arbitrary address and host our web app from anywhere. Here is an example load-test.
import http from 'k6/http'; import { check, sleep } from 'k6'; export let options = { stages: [ { duration: '30s', target: 5000 }, // ramp up { duration: '30s', target: 5000 }, // sustained load { duration: '30s', target: 0 }, // cool down ], }; export default function () { let res = http.get('{{url-prefix}}/ping') check(res, {'status is 200': (r) => r && r.status === 200}); sleep(1); } load-test-template should be a map with a single key :url-prefix. You may find more exciting uses for using templates in your load tests. 😉
Orchestration
We can use the excellent Cognitect AWS library to kick off our tests.
(defn launch-load-test-instance [ec2 aws-creds uuid instance-name load-test load-test-template] (aws/invoke ec2 {:op :RunInstances :request {:ImageId "my-ami" :InstanceType "my-size" :MinCount 1 :MaxCount 1 :InstanceInitiatedShutdownBehavior "terminate" :UserData (-> (build-user-data aws-creds uuid instance-name load-test load-test-template) byte-streams/to-byte-array base64/encode byte-streams/to-string) :TagSpecifications [{:ResourceType "instance" :Tags [{:Key "Name" :Value instance-name} {:Key "load-test-id" :Value uuid}]}] :SecurityGroupIds [...] :KeyName "my-company"}})) (defn launch-load-test-instances [ec2 aws-creds uuid instance-prefix load-test load-test-template num-instances] (doall (map #(launch-load-test-instance ec2 aws-creds uuid (str instance-prefix %) load-test load-test-template) (range num-instances)))) There are some gaps in the above code that you will need to fill.
I feel like a mad scientist after writing that code. Wreak havoc, my minions!
Gathering test results
In the previous code, you may have noticed that we supplied a uuid to each ec2 instance and used it in the user-data script as a shared bucket for all of our k6 results. I hope you kept track of it. 😈
(def s3-bucket "load-test-results") (defn get-k6-summary [s3 key] (->> {:op :GetObject :request {:Bucket s3-bucket :Key key}} (aws/invoke s3) :Body io/reader json/decode-stream)) (defn get-k6-summaries [s3 uuid] (map (comp (partial get-k6-summary s3) :Key) (->> {:op :ListObjects :request {:Bucket s3-bucket :Prefix uuid}} (aws/invoke s3) :Contents))) Calling get-k6-summaries will return a sequence of all the generated results. Those individual results may be helpful for some analysis, but you should aggregate as many metrics as can be combined.
(defn metric-type [m] (condp = (set (keys m)) #{"count" "rate"} :counter #{"min" "max" "p(90)" "p(95)" "med" "avg"} :trend #{"fails" "value" "passes"} :rate #{"min" "max" "value"} :gauge :unknown)) (defn merge-summaries "summaries is a vector of k6 output summaries converted from JSON to EDN. get-k6-summaries returns this structure precisely. This function works with the various types of k6 metrics (https://k6.io/docs/using-k6/metrics) and does the following: - min values are the minimum in all the maps - max values are the maximum in all the maps - avg values are the average in all the maps - count is the sum in all the maps - rate is the average in all the maps - fails is the sum in all the maps - passes is the sum in all the maps p, med, & 'value' values are not included in the merged summary, but they can still be viewed for individual machines by inspecting the output from get-k6-summary" [summaries] (letfn [(merge-metrics [v1 v2] (condp = (metric-type v2) :counter {"count" (+ (get v1 "count") (get v2 "count")) "rate" (double (/ (+ (get v1 "rate") (get v2 "rate")) 2))} :trend {"min" (min (get v1 "min") (get v2 "min")) "max" (max (get v1 "max") (get v2 "max")) "avg" (double (/ (+ (get v1 "avg") (get v2 "avg")) 2))} :rate {"fails" (+ (get v1 "fails") (get v2 "fails")) "passes" (+ (get v1 "passes") (get v2 "passes"))} :gauge {"min" (min (get v1 "min") (get v2 "min")) "max" (max (get v1 "max") (get v2 "max"))} {})) (merge-checks [v1 v2] {"passes" (+ (get v1 "passes") (get v2 "passes")) "fails" (+ (get v1 "fails") (get v2 "fails"))}) (max-vus [m] (get-in m ["metrics" "vus_max" "max"]))] {:metrics (apply merge-with merge-metrics (map #(get % "metrics") summaries)) :checks (apply merge-with merge-checks (map #(get-in % ["root_group" "checks"]) summaries)) :max-vus (->> summaries (map max-vus) (apply +))})) Call merge-summaries to get aggregate results.
Tradeoffs
A paid k6 plan will provide you with a friendlier interface and better result-sets. But is $25k/year worth that amount of money? I don't know. That's up to each user.
If you're willing to manage some ec2 instances yourself and work with less information in aggregate, then perhaps this approach is better.
One problem I didn't go into any detail about is managing failed EC2 instances. Certain failure states require an external TerminateInstances operation: management of 'dangling' ec2 instances is now your problem.
k6 cloud will also distribute traffic across regions. I think it will also simulate clients with slower bandwidth. Could you provide all of this if you need it?
I believe you could get close to k6's paid feature-set, but only with a lot of effort. Hopefully, I have given you a starting point.
Don't forget to have fun!
Top comments (0)