refactor: cleanup and update README

Author: steve-chavez

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Steve Chavez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,53 +1,223 @@
-# PostgREST benchmark(Work in progress)
+# PostgREST Benchmark
 
-Updated and reproducible benchmark for PostgREST by using [Nix](https://nixos.org/) and [k6](https://k6.io/).
+Repeatable benchmark for PostgREST by using [Nix](https://nixos.org/) and [NixOps](https://github.com/NixOS/nixops).
 
-## Setup
+NixOps provisions AWS EC2 instances on a dedicated VPC and deploys the different components for load testing.
 
-The tests are ran on AWS EC2 instances on a dedicated VPC. The client(k6) and the servers(pg+pgrest) are on different instances. This whole setup will be handled by Nix.
+The default setup includes:
+
+- A `m5a.xlarge` instance which uses [k6](https://k6.io/) for load testing.
+- A `t3a.nano` instance with PostgreSQL.
+- A `t3a.nano` instance with PostgREST + Nginx.
+
+This setup, including the size of the EC2 instances, can be modified with environment variables. As the PostgreSQL server instance size increases, its settings are modified according to [PGTune](https://pgtune.leopard.in.ua/) recommendations.
+
+## Requirements
+
+- [AWS](https://aws.amazon.com) account with an `~/.aws/credentials` file in place (can be created with `aws-cli`). The "default" profile is picked up by default but you can change it by doing:
+ ```
+ export PGRSTBENCH_AWS_PROFILE="another_profile"
+ ```
+
+- [Nix](https://nixos.org/).
+
+## Quickstart
 
 Run `nix-shell`. This will provide an environment where all the dependencies are available.
 
 ```
-nix-shell
+$ nix-shell
 >
 ```
 
-Now create the environment with nixops.
+Deploy with:
+
+```
+$ pgrstbench-deploy
+
+pgrstBenchVpc.....> creating vpc under region us-east-2
+..
+
+pg................> activation finished successfully
+pgrstbench> deployment finished successfully
+
+# this command will take a couple minutes, it will deploy the client and server AWS machines VPC stuff
+```
+
+Run a `k6` test on the client instance and get the output:
+
+```
+$ pgrstbench-k6 20 k6/GETSingle.js
+```
+
+Destroy all the setup and the AWS instances:
+
+```
+$ pgrstbench-destroy
+```
+
+## SSH
+
+To connect to the PostgreSQL instance:
+
+```
+$ pgrstbench-ssh pg
+
+# Check the installed services
+$ systemctl list-units
+
+## connect to postgres
+$ psql -U postgres
+$ \d
+```
+
+The postgresql server comes loaded with the [chinook database](https://github.com/xivSolutions/ChinookDb_Pg_Modified).
+
+To connect to the PostgREST instance:
+
+```
+$ pgrstbench-ssh pgrst
+
+# Check the installed services
+$ systemctl list-units
+
+# Do a request
+$ curl localhost:80/artist
+```
+
+You can also get info (like the IPs) of the instances with:
+
+```
+$ pgrstbench-info
+```
+
+## K6
+
+K6 runs on the client instance, but you can get the output of the load test on your machine:
+
+```
+## k6 will run with 10 VUs on the AWS client instance and load test the t3anano instance with the local k6/GETSingle.js script
+$ pgrbench-k6 10 k6/GETSingle.js
+
+## You will see the k6 logo and runs here
+```
+
+There are different scripts on `k6/` which test different PostgREST requests.
 
+## Pgbench
+
+pgbench also runs on the client instance, you can get its output with:
+
+```
+$ pgrstbench-pgbench pgbench/GETSingle.sql
 ```
-# This assumes there's a `~/.aws/credentials` file(created with aws-cli) with a default profile.
-nixops create ./deploy.nix -d pgrst-bench
 
-# To explore and connect to the ec2 instances
-# nixops ssh -d pgrst-bench t2nano
-# nixops ssh -d pgrst-bench t3anano
-# nixops ssh -d pgrst-bench client
-#
-# you can inspect the db with
-# psql -U postgres
-# \d
+The `GETSingle.sql` runs an equivalent SQL statement to what PostgREST generates for `GETSingle.js`. The motivation for this comparison is to see how much PostgREST performance differs from direct SQL connections.
+
+## Varying Scripts
+
+There are scripts that help with varying the environment while load testing. You can use these to get a report once the command finishes running:
+
+Run pgbench with a different qty of clients:
+
+```
+$ pgrstbench-pgbench-vary-clients pgbench/GETSingle.sql
 ```
 
-## Usage
+Run k6 with a different qty of VUs:
+
+```
+$ pgrstbench-k6-vary-vus k6/GETSingle.js
+```
 
-Run the k6 script:
+Run pgbench with varied clients and with varied pg instances (this will involve reprovisioning/redeploying the pg instance, it will take a while):
 
 ```
-## k6 will run on the aws instance
-nixops ssh -d pgrst-bench client k6 run -e HOST=t3anano - < k6/GETSingle.js
+$ pgrstbench-vary-pg pgrstbench-pgbench-vary-clients pgbench/GETSingle.sql > PGBENCH_GET_SINGLE.txt
+```
+
+Run k6 with varied vus and with varied pg instances and pgrst instances (this will involve reprovisioning/redeploying the pg and pgrst instance, it will take even longer):
+
+```
+$ pgrstbench-vary-pg-pgrst pgrstbench-k6-vary-vus k6/GETSingle.js > K6_GET_SINGLE.txt
+```
+
+## Different Setups
+
+### Nginx included (default)
+
+To load test with nginx included do:
+
+```bash
+export PGRBENCH_WITH_NGINX="true"
+pgrbench-deploy
+```
+
+To only have PostgREST listening directly on port 80:
+
+```bash
+export PGRBENCH_WITH_NGINX="false"
+pgrbench-deploy
 ```
 
-(Steps for collecting the results are pending)
+### Unix socket (default)
+
+To load test connecting pgrest to pg with unix socket, and pgrest to nginx with unix socket.
+
+```bash
+export PGRBENCH_WITH_UNIX_SOCKET="true"
+pgrbench-deploy
+```
+
+To use tcp instead, you can do:
+
+```bash
+export PGRBENCH_WITH_UNIX_SOCKET="false"
+pgrbench-deploy
+```
+
+### Separate PostgreSQL instance (default)
+
+To load test with a pg on a different ec2 instance.
+
+```bash
+export PGRBENCH_SEPARATE_PG="true"
+pgrbench-deploy
+```
+
+To use the same instance for both PostgreSQL and PostgREST.
+
+```bash
+export PGRBENCH_SEPARATE_PG="false"
+pgrbench-deploy
+```
+
+### Two PostgREST instances over load balanced with Nginx
+
+Some experiments indicate that when load testing on big instances (like `m5a.8xlarge` and above), having two postgrest instances instead of one increase throughput. You can try this with:
+
+```bash
+export PGRSTBENCH_PGRST_NGNIX_LBS="true"
+pgrbench-deploy
+```
+
+### Different EC2 instance types
+
+To change pg and PostgREST EC2 instance types(both `t3a.nano` by default):
+
+```bash
+export PGRBENCH_PG_INSTANCE_TYPE="t3a.xlarge"
+export PGRBENCH_PGRST_INSTANCE_TYPE="t3a.xlarge"
+export PGRBENCH_PGRST_INSTANCE_TYPE="t3a.xlarge"
+
+pgrbench-deploy
+```
 
-## Notes
+## Limitations
 
-- Scenarios to test:
- - [x]read heavy workload(with resource embedding)
- - [x]write heavy workload(with and without [specifying-columns](http://postgrest.org/en/v7.0.0/api.html#specifying-columns))
- - [ ]pg + pgrest on the same machine(unix socket and tcp).
- - [ ]pg and pgrest on different machines?
- - [ ]pgrest with `pre-request`?
+- Uses an outdated version of NixOps, 1.7. Newer versions have changed considerably.
+- Don't try changing to ARM-based instances, these don't work with the above NixOps version.
+ + The instances tested for this benchmark are the `t3a` series and the `m5a` series.
 
 ## Other benchmarks