This website and its server was rearchitected during the summer of 2015. It is currently hosted on a single Dell PowerEdge R710 in my home, running Arch Linux.
I use Docker and Docker Compose to containerize and manage services, each of which is located in a Docker container. Each request passes through the nginx container, which routes the request to the corresponding backend service.
Below is a non-exhaustive overview of the various components of this server’s architecture.
I use a customized version of nginx known as OpenResty, which provides a lightweight LuaJIT scripting environment to process some requests, among other enhancements. Other customizations to this nginx instance include replacing OpenSSL with BoringSSL and the addition of ngx_pagespeed to help improve page load performance. Previously, patches were applied for SPDY support or for ChaCha20-Poly1305 cipher support, however these are no longer necessary.
nginx acts as a static file server, a TLS terminator (using certificates from Let’s Encrypt), page/script/style/image optimizer (via ngx_pagespeed), rate limiter to limit the impact of minor DoS attacks, and a cache for responses from backend servers. Trivial responses are generated by small Lua scripts, running in LuaJIT embedded within nginx.
This page was generated by WordPress, running on PHP-FPM and MariaDB. It is a WordPress Multisite install, hosting multiple websites. Page caching is performed by nginx, with an additional helper plugin installed in WordPress to automatically purge the cache when necessary.
Data is stored on two 1 TB hard drives and two 4 TB hard drives, managed by ZFS as two sets of mirrors. Additional ZFS datasets were created for database servers (with adjusted caching and record sizes) and compressible and rarely-accessed data. A 200 GB SSD acts as a ZIL for improved synchronous write performance and an L2ARC as a read cache.
Cloudflare provides fast DNS service and caching for static files.
One caveat of Cloudflare’s “Universal SSL” is that it only supports clients that support ECDSA key exchange, which is less computationally expensive for the server compared to RSA. Most notably, this does not include clients such as Chrome on Windows XP, some “security” software, and corporate proxies. This website works around this by loading a “canary” script from Cloudflare. If this fails, an inline script sets a “nocdn” cookie and reloads the page, and nginx is configured to not rewrite images and scripts to the CDN if such a cookie is present.
While Cloudflare’s proxying is not enabled on andrewsun.com for compatibility purposes, there are some cases when it is necessary to use Cloudflare’s DDoS mitigation. During these attacks, I will manually enable Cloudflare proxying.