OpenResty Overview

By jPablo Caballero | February 14, 2020

A brief overview of OpenResty, a web platform that integrates Nginx, LuaJIT, and various Nginx modules. With it, you can build scalable and very high-performance web applications, web services, and dynamic web gateways.

Maybe you have not heard about OpenResty, and indeed, this web platform is not as widely known as others. But in various forms it is used in very demanding environments by very high-profile companies.

Perhaps one of the reasons why it has gone under the radar for many, is because its genesis and initial growth took place in China. In fact, in its early years it was supported by Taobao.com (Alibaba group). Taobao is the biggest e-commerce website, and the eight most visited website (see Wikipedia). Later, it was supported and used by CloudFlare a large and well-known provider of content delivery network, distributed domain name server, DDoS mitigation, and internet security services. Nowadays it’s sponsored by the commercial company established by its creator (Yichun Zhang, aka agentzh). There are also some other interesting opensource/commercial products based on OpenResty, such as Kong.

By the history of the Open Source OpenResty platform, the services provided in the commercial products based on it, and the list of companies that use it (see the commercial site), it’s pretty clear that this platform is appropriate for environments that require robustness, scale and performance.

According to Netcraft, its use seems to be growing (see Netcraft’s November 2019 Web Server Survey).

It is important to understand that OpenResty is not a fork of Nginx. It does use the standard Nginx. You can think of it as carefully integrated bundle that supercharges your web server to make it even more flexible, dynamic and very powerful while remaining performant. A key component of this bundle is LuaJIT, a Just-In-Time compiler for the Lua language. This is what allows you to script Nginx and do pretty amazing things with it.

The first time I ran into OpenResty I really did not pay much attention to it, and dismissed it almost immediately. I did not want to learn Lua, I did not understand what benefits it could bring, so I set it aside. But, as many other technologies I read about but for which I have no use right away, it kept lingering in the background processes of my brain.

At some later point in time, I needed to implement some proxy-like functionality for web requests, concretely, modifying the body of some requests as they reached the web server, and also modifying some responses before they were sent out to the client, mainly for security and privacy purposes. As I was already a user of Nginx, I somehow remembered that OpenResty could be a good fit for this. That’s when I really learned about it and saw what it could do. I then realized that it is a powerful tool to have in your arsenal, so it’s worth to know about it.

This article just provides some general information about OpenResty, the types of uses that are a good fit for it, and some general considerations about developing with it. At the end there is a short selection of links to some interesting resources about OpenResty.

The basics

As mentioned earlier, OpenResty bundles a set of components, one of them being Nginx, some of which are Lua libraries, and others are Nginx 3rd-party modules (most of them developed by the OpenResty team), etc.

Some of the Nginx modules can be used (and are useful) by themselves, for example the Postgres Nginx Module (ngx_postgres) provides configuration variables and directives that allow Nginx to communicate directly with a Postgres database. There is also a similar module for Redis, etc.

However, we could say that the most relevant and powerful core component in OpenResty is the ngx_lua module (ngx_http_lua_module). This module embeds LuaJIT into Nginx. It implements various directives that allow you to script Nginx with Lua. Lua code that runs under these directives has access to the powerful Nginx Lua API. OpenResty also bundles other very useful Lua libraries that can be used from your scripts. The power that all this gives you is pretty impresive.

I know that all this sounds way to abstract and vague, so let’s try to make it more concrete. To do this, I’ll summarize the documentation for a few ngx_lua directives and a few functions in the Lua API and then show a couple of short examples of how they would be used and what they would do.

Example 1

  • The directive content_by_lua_block is like a content handler for every request. It executes the Lua code provided within the pair of curly braces ({})
  • the function ngx.say emits arguments concatenated, and a trailing newline, to the HTTP client (as response body).
  • The directive content_by_lua_file is analogous to content_by_lua_block, but the argument is a path to a file that contains the Lua code to execute.

Now take a look at the following configuration file for Nginx/OpenResty:

worker_processes  1;
error_log logs/error.log;
events {
    worker_connections 1024;
}
http {
    server {
        listen 8000;
        location / {
            default_type text/html;
            content_by_lua_block {
                ngx.say("<h1>Hello from Lua!</h1>")
            }
        }
    }
}

As you can see, it is pretty clear. When location / is accessed, the script will return the body:

pablo@node1:~/Scripts$ curl -v http://10.100.174.172:8000/
*   Trying 10.100.174.172...
* Connected to 10.100.174.172 (10.100.174.172) port 8000 (#0)
> GET / HTTP/1.1
> Host: 10.100.174.172:8000
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.2
< Date: Wed, 12 Feb 2020 10:17:19 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< 
<h1>Hello from a Lua script file!</h1>
* Connection #0 to host 10.100.174.172 left intact

Embedding Lua code in the configuration file may be acceptable when the code is extremely short (1 to 4 lines or so), but can get messy when the scripts are larger. In this case, it’s better to place the Lua code in a separate file, and use the content_by_lua_file directive:

scripts/hello.lua

ngx.say("<h1>Hello from a Lua script file!</h1>")

worker_processes  1;
error_log logs/error.log;
events {
    worker_connections 1024;
}
http {
    server {
        listen 8000;
        location / {
            default_type text/html;
            content_by_lua_file scripts/hello.lua;
        }
    }
}

Obviously, these are extremely simple examples. There are libraries for templating, you can return JSON, you can modify headers, you can read and write Nginx variables, and much more. The Nginx API for Lua documentation is worth a look.

Example 2

  • The Lua CJSON Library is a Lua C module that provides fast JSON parsing and encoding. Bundled by default in OpenResty.
  • The function ngx.location.capture issues a synchronous but still non-blocking Nginx Subrequest using the given uri.
  • The function ngx.location.capture_multi is the same as the previous one, but allows multiple subrequests running in parallel.

Consider this simple configuration file:

worker_processes  1;
error_log logs/error.log;
events {
    worker_connections 1024;
}
http {

    server {
        listen 8000;

        location /proxycat {
           internal;
           proxy_pass https://aws.random.cat/meow;
           proxy_pass_request_headers off;
        }

        location / {
            default_type  text/html;
            content_by_lua_block {
                local CJSON = require "cjson"
                local res = ngx.location.capture('/proxycat')
                local catinfo = CJSON.decode(res.body)
                ngx.say('<img src="' .. catinfo.file .. '"/>')
            }
        }
    }
}

It defines an internal proxy location that calls an external public API that returns a json with the file (url) of a random image of a cat. What we do in our / location is pretty obvious:we have a 3-line Lua script that uses ngx.location.capture to get the result from that external API call. It then decodes the json to extract the file information, and then it uses that info to build and return a body with an image tag (.. is the concatenation operator in Lua). So, when you visit /, you get a random image of a cat. Using the multi version, you could be calling any number of APIs, making any number of subrequests, and then combining the results in any way you wanted.

Of course, this simple example of the cat image could be done this directly from the client, but that’s not the point. The point is that you are ‘hiding’, in a way, how you get the url of the image. This API is public and doesn’t require a key. However, if you were using an API that required a Key, and you didn’t want to expose that key to the clients that call your server, you could use this to enhance your security/privacy. An example of this would be a contact page: your script could get the data from the contact form, and then call a mail API (like MailChimp, MailJet, SendGrid, or any other) to easily send that info through email, without exposing your mail service API key.

Another way to make external http calls that may be more intuitive and convenient is to use a library called lua-resty-http. It basically implements a client for http (http client cosocket driver). So you can make new http requests from within your request-processing scripts.

Additionally, let’s just mention lua-resty-websocket, which implements non-blocking client and server WebSocket functionality in OpenResty/Lua. With this, you could -for example- communicate with other OpenResty servers (or any services that accept websocket connections) through WebSockets.

Phases

So far we have seen the content_by_* directives, which allows you to use Lua code to interfere in the Nginx content phase of request processing. But this is not the only phase where you can use Lua code. OpenResty provides directives for other phases as well. You can use Lua code in the following phases:

  • Initialization
  • Rewrite
  • Access
  • Content
  • Log

Remember that OpenResty directives are the entry points for your Lua code. Also, it’s important to check the Nginx Lua API, since not all functions are available in all contexts.

As you can imagine, being able to use your own dynamic logic (in many cases including calling external services, like APIs or databases) in all those phases gives you considerable power and flexibility.

Concretely, using scripts in the access phase allows you to implement custom (as complex as needed) authorization, access control, session control, etc. Even if you use upstream servers in other languajes/systems (such as Go, Python, Node, etc.) as part of your complete web app, you can use scripts in the access phase to do preliminary filtering or routing/balancing. For example, if your app uses JWT (Json Web Tokens), you can check and validate them right in Nginx, and deny access if necessary. This way, non-authorized or otherwise invalid requests would not even hit your upstream servers.

Do check out the documentation. It is thorough, and contains easy to understand short examples.

Development

Model

What is really cool about OpenResty is the fact that you write synchronous but yet non-blocking code. That is, you don’t have to worry about promises, callbacks, or anything like that. The structure of the code in a handler is fairly linear, simple and easy to understand. Let’s see an example of an access controller for a specific location in a real application:

part of nginx configuration

  location ~ ^/(?P<session_path>[a-zA-Z0-9_-]+)/xapiunit/(?P<unit_path>.*)$ {
        # control access based on the session_path and the cookie, if ok, do ngx.exec (internal redirect to /private)
        access_by_lua_file $appFilePath/unit_access.lua;
    }

unit_access.lua

-- unit access control

local appName = ngx.var.appName
local CONF = require(appName..'/app_conf')

-- sessionPath and unitPath come from the capture groups in ngix's location
local sessionPath = ngx.var.session_path
local unitPath = ngx.var.unit_path

-- check if there's a xpysession going on
local xpysession = require "resty.session".open{ name = CONF.SESSION.XPROXY_COOKIE}

if not xpysession.present then
  ngx.log(ngx.ERR, err)
  ngx.exit(ngx.HTTP_NOT_FOUND)
end

local fpath = '/private_local/' .. unitPath

-- unit access granted, doing internal redirect...

ngx.exec(fpath)

Even if you haven’t used Lua before, this is easy to understand: set up some variables, get information about the session, if there’s no session exit the request processing with a not found code/message, and if there is a valid session, do an internal redirect (thus, granting access to the content). Of course this code make use of a library called resty-session, which is used throughout the app to set up the sessions etc. This session library can store session info in various storage backends, such as Redis. Imagine that this code uses Redis as a backend storage for sessions. This would mean that each time there’s a request for that location, the system will call an external Redis server to get information, and based on the response, it would grant access or not. This whole process is non-blocking. So even though the code is ‘synchronous’ (the call to “resty.session”.open would reach out to the Redis server, return a response, and we expect that response to be available in the next line of code), this whole thing doesn’t block the server at all. Nginx manages its workers, its threads, Lua spawns its light threads etc. And so this instance of Nginx can get many requests at the same time for this location, and it will still perform very well. Yet your code is simple and clear.

One caveat, though, is that to take advantage of this non-blocking operation, you have to use the right libraries (ones that use cosockets appropriately). Not everything goes. For example, to make calls to postgres, use the pgmoon library, which was created for OpenResty and uses cosockets. If you use a different library to access Postgres from Lua, it will result in blocking. As far as I know, cpu-bound calculations will also block your system, so for these types of loads, it would probably be better to perform the operations on a separate upstream server (written maybe in C, Go… maybe Python).

There are many libraries that can be very useful for your OpenResty apps. For example, you could use a template library, if you needed to generate more complex content.

Use Cases

As OpenResty can act as an a solution for advanced, dynamic, programmable configuration of web servers, and manipulation of different phases of request processing, it is used in situations where these types of operations are a key, for example:

  • Access control, routing and load balancing: By examining data from the request, and data from other sources (if needed) you can implement complex and flexible access control, rewrite, and load balancing schemes. This can also help simplify the core responsibilities of upstream applications. Things like dynamic black lists, advanced rate limiting etc. would also be relatively easy to implement.
  • Proxies: You can manipulate the headers or body in requests (before reaching other parts of the system) or the responses (after other parts of the system have processed the request and provided a response). This can be useful for increasing security and/or privacy, for example performing anonymization or pseudunomyzation.
  • API gateways: This is fairly straightforward use of OpenResty. By using subrequests and other features, you can add a smart API layer to the edge of your system. This way, you can present a stable API to external clients, while your internal APIs or (micro) services can evolve, and can interoperate amongst themselves.
  • Monitoring: Being able to analyze requests in the access and content phases allows you to implement all kinds of monitoring / calculations to your APIs or web application, and send that information to other systems.
  • Microservices: Aside from the already mentioned API gateways in microservice environments, with OpenResty you could even implement some aspects of service meshes, such as registration, discovery, etc.

Of course OpenResty would be appropriate for a wide range of tasks. Those are the most typical. Nothing prevents you from using it for complete web applications, if you wish. In fact there is a whole web framework for web applications that works on top of OpenResty.

Lua

One of the things that may discourage people about using OpenResty at first is the need to use the Lua language. Don’t let this stop you from trying OpenResty. Lua is not hard to learn, although it has a few characteristics that will probably trip you several times (like array indexing starting at 1, instead of 0), global variables by default (which you should try to avoid, particularly in OpenResty) but nice block scope for local variables (which I like…), etc. But in general, it’s easy to learn.

Lua is is a compact language that is easily embeddable in other systems/programs. It is used a lot in games and other systems, for example in Redis.

As with many development environments, the language per se, the syntax etc. is not a problem. Lua is nice and readable. It’s more a matter of getting used to the whole environment: your Lua scripts executing in Nginx, being careful that you’re accessing the right data from the requests, etc.

One thing that can be a bit frustrating is to chase errors. Sometimes an error will cause a Server Error (500) in Nginx, so get used to look at the logs. Usually they log lines around the crash have enough information to spot the bug, but you can output custom information to the error log with the ngx.log API function.

Something that can help with this is to use appropriate tooling. The one IDE that I have found particularly useful is ZeroBrane Studio. Wiht ZeroBrane you can set up debugging so that you can step through your code and see what’s going on, as a request is being processed. This is extremely valuable.

Also, there are Lua libraries for unit testing, and there’s a robust test scaffold developed by the main developer of OpenResty. It is used to perform automated test cases for Nginx C modules, Lua libraries and you can even use it for your applications.

OpenResty is a powerful framework that strengthens an already very capable and robust system, Nginx. I believe it is very much worth to learn a bit about it, and have it handy as it can be a very useful tool for web development and web services deployment. With a few lines of easy-to-understand code in the right places, you can solve problems that would be much much harder to solve in other environments. Many high-performance high-scale system, like CDNs, API gateways, and massive eCommerce sites etc. use OpenResty. It gives you a supercharged Nginx, so you may use it to enhance only some parts of your web applications and services, if you are using other languages and frameworks.

As mentioned initially, this is just a very general overview. I’ve sprinkled links so you can get more information. Here are some more links to documentation, articles, presentations, etc. that are interesting and illustrative of OpenResty’s uses and power. Take a look at them to get a deeper vision and understanding of OpenResty. Enjoy.

Tags