1

Please excuse lengthy write up - I would really appreciate any help in following regard.

I am trying to setup multi tenant subdomain + custom domain with SSL using LetsEncrypt:

(some will use subdomain some will use custom domain)

https://customer1.myapp.com https://customer2.myapp.com https://customer1.com (customer sets up A/CNAME recoreds at his DNS provider)

I am on EC2 instance using Ubuntu OS with username 'ubuntu'.

I learned from following tutorials:

https://sandeep.dev/how-we-generate-and-renew-ssl-certs-for-arbitrary-custom-domains-using-letsencrypt-cjtk0utui000c1cs1f7y9ua5n

https://www.digitalocean.com/community/tutorials/how-to-use-the-openresty-web-framework-for-nginx-on-ubuntu-16-04

https://sandro-keil.de/blog/openresty-nginx-with-auto-generated-ssl-certificate-from-lets-encrypt/

I have successfully done following:

  • Installed build-essential on server
  • Install OpenResty (Comes with its own Nginx & OpenSSL)
  • Install LuaRocks
  • Install lua-resty-auto-ssl

  • Created directory for resty auto ssl

    sudo mkdir /etc/resty-auto-ssl sudo chown -R ubuntu /etc/resty-auto-ssl sudo chown -R www-data /etc/resty-auto-ssl chmod -R 777 /etc/resty-auto-ssl/

  • Created Fallback Self-signed Certificate which expires in 3600 days

This is my starter conf file (/usr/local/openresty/nginx/conf/nginx.conf)

(I would refine it further to suite my redirect & security needs)

#user nginx; error_log /usr/local/openresty/nginx/logs/error.log warn; events { worker_connections 1024; } http { lua_shared_dict auto_ssl 1m; lua_shared_dict auto_ssl_settings 64k; init_by_lua_block { auto_ssl = (require "resty.auto-ssl").new() auto_ssl:set("allow_domain", function(domain) return true end) auto_ssl:set("dir", "/etc/resty-auto-ssl") auto_ssl:init() } init_worker_by_lua_block { auto_ssl:init_worker() } # access_log /usr/local/openresty/nginx/logs/access.log main; server { listen 443 ssl; ssl_certificate_by_lua_block { auto_ssl:ssl_certificate() } ssl_certificate /etc/ssl/resty-auto-ssl-fallback.crt; ssl_certificate_key /etc/ssl/resty-auto-ssl-fallback.key; root /var/www/myapp.com/public; index index.php index.html index.htm; location / { try_files $uri $uri/ /index.php?$query_string; } # location ~ \.php$ { # include snippets/fastcgi-php.conf; # fastcgi_pass unix:/run/php/php7.4-fpm.sock; # fastcgi_read_timeout 600; # } location ~ /\.ht { deny all; } } server { listen 80; server_name *.myapp.com myapp.com; location /.well-known/acme-challenge/ { content_by_lua_block { auto_ssl:challenge_server() } } location / { return 301 https://myapp.com$request_uri; } } server { listen 8999; location / { content_by_lua_block { auto_ssl:hook_server() } } } }

I am facing multiple issues like:

  1. Cant mention user in nginx config - still works without it also

    • Trying to mention user in 1st line of config files gives me error.
    • So i commented it out and tried to caryy on anyways

  2. Dehydrated Failure but certificate is created

    • keep getting following error in my log: lets_encrypt.lua:40: issue_cert(): auto-ssl: dehydrated failed: env HOOK_SECRET=XXXX HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/dehydrated --cron --accept-terms --no-lock --domain myapp.com --challenge http-01 --config /etc/resty-auto-ssl/letsencrypt/config --hook /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks status: 256 out: # INFO: Using main config file /etc/resty-auto-ssl/letsencrypt/config
    • But it still goes on & does create a certificate after which it gives random number generator error.
    • Sometimes, if I delete everything inside /etc/resty-auto-ssl - it dosent give me such errors.

  3. Can't find OpenSSL random number generator

    • I keep getting following error in my log: Can't load ./.rnd into RNG random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:98:Filename=./.rnd curl: (22) The requested URL returned error: 500 Internal Server Error

  4. PHP-FPM on nginx provided with OpenResty

    • I have properly installed php-fpm and have tested it when using nginx standalone.
    • But, now that I am using nginx provided with openresty, it dosent seem to work
    • Error (Shown when tested config using: nginx -t command): "/usr/local/openresty/nginx/conf/snippets/fastcgi-php.conf" failed (2: No such file or directory)

  5. Failed to create certificate

    • Sometimes this error is followed by error in above point number 2:

    auto-ssl: could not get certificate for myapp.com - using fallback - failed to get or issue certificate, context: ssl_certificate_by_lua*, client: 123.201.226.209, server: 0.0.0.0:443

    set_response_cert(): auto-ssl: failed to set ocsp stapling for xxxx.myapp.com - continuing anyway - failed to get ocsp response: OCSP responder query failed (http://ocsp.int-x3.letsencrypt.org): no resolver defined to resolve "ocsp.int-x3.letsencrypt.org", context: ssl_certificate_by_lua*, client: 123.201.226.209, server: 0.0.0.0:443

    connect() to unix:/run/php/php7.4-fpm.sock failed (13: Permission denied) while connecting to upstream, client: 123.201.226.209, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.4-fpm.sock:", host: "xxxx.myapp.com"

  6. When trying to access customer1.com whoes A record points to myapp.com server IP

    "Error creating new order :: Cannot issue for \"X.X.X.X\": The ACME server can not issue a certificate for an IP address"

    ssl_certificate.lua:281: auto-ssl: could not determine domain for request (SNI not supported?) - using fallback - , context: ssl_certificate_by_lua*, client: 45.148.10.72, server: 0.0.0.0:443

    • ... where x.x.x.x is A recored for customer1.com whch was opened from browser

I have following confusions:

  1. Should I get one proper (paid) wildcard positive ssl certificate for myapp.com ? (And use it as fallback)

    • This covers all my subdomain and I won't have to deal with limits on subdomain by letsencrypt.
    • This way I only have to use lets encrypt for custom domains like customer1.com

  2. I am not sure if my users & permission are properly set up - any pointers would help

  3. I would wish my final nginx config to fulfill following needs

Improve this question
3
  • Do you need to stick to openresty? Because I'm running a website-as-a-service company and I went through couple of configurations. From raw nginx + certbot over to kubernetes ingress with certmanager and finally I'm considering to move away from kubernetes and run my custom-domain layer on caddy 2. Commented Apr 26, 2020 at 16:46
  • No such requirement to use openresty. Since I have little knowledge of server configuration, that was the only option I found on internet easily. What route do you suggest I should look into? Commented Apr 27, 2020 at 19:56
  • If you have an existing Kubernetes cluster (or experience) I would go with Ingress and control the resources through the kube-api. The downside: A kubernetes cluster only for ingress seems a bit overkill and frequent reloads might slow down nginx. Caddy2 has also letsencrypt support, provides an REST API and the Caddyfile syntax is pretty handy to use. The only downside which I discovered during testing is the missing HA storage system for certs. As of only file_system is supported. Commented Apr 27, 2020 at 20:30

1 Answer 1

Reset to default
0

It is somewhat hard to answer all these question, so I'll attempt to answer part of 5 & 6. I have setup open resty myself in a prod environment, see link.

  1. I ran into this OCSP stapling issue. I found that it was resolved by adding this to my NGINX config:
# A DNS resolver must be defined for OSCP stapling to function. resolver 172.20.0.10 ipv6=off;

Regarding question 6, I would suggest that customer1.com should be a CNAME to myapp.com.

I would also recommend using as a base the openresty docker image, or at least a reverse engineered version of the docker image into an EC2 instance. Here is my dockerfile:

FROM openresty/openresty:latest-xenial RUN /usr/local/openresty/luajit/bin/luarocks install lua-resty-auto-ssl RUN /usr/local/openresty/luajit/bin/luarocks install lua-resty-http RUN apt-get update RUN apt-get install -y dnsutils RUN openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -subj '/CN=sni-support-required-for-valid-ssl' -keyout /etc/ssl/resty-auto-ssl-fallback.key -out /etc/ssl/resty-auto-ssl-fallback.crt ADD nginx.conf /usr/local/openresty/nginx/conf/nginx.conf

Hopefully this is helpful.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.