Apache Series — Listener subsystem of mpm_event module

In this post, I’d like to focus primarily on the listener subsystem of the mpm_event module.

The listener subsystem is nothing but a thread that listens for incoming connections, accepts the connections and distributes these established connections among the other worker threads that process the requests that arrive on these connections.

Let’s dive in to how this works:

One listener thread and multiple worker threads

The behavior of the listener thread is influenced by the arguments specified for the Listen* directives defined in the Apache HTTPD configuration

The directives that influence the listener thread are –

• Listen directive — tells the server to accept incoming requests on the specified port or address-and-port combination.
• ListenCoresBucketsRatio — when specified on systems with multiple cores, this directive can help scale accept() call.

Let us consider an example where the following snippet is present in the HTTPD configuration file:

Listen 127.0.0.1:5555
ListenCoresBucketRatio 4

In the main/parent HTTPD process,

• Multiple listen buckets equal to num_CPU_cores/ListenCoresBucketRatio are created.
• Assuming, this is a system with 16 online cores, we have 4 listen buckets created.
• For each listen bucket, a listen socket is created using socket() API.
• There is a 1–1 mapping between listen sockets and listen buckets.
• Each socket is bound to the same interface and port. The interface and port used are those specified in the Listen directive.
• The bind() calls succeed as the sockets were created using SO_REUSEPORT option.
• The parent process executes the listen() API on each of the sockets created above.
• The sockets are now called as Listen sockets.
• Child processes are spawned. The number of child processes is determined by the value of StartServers.
• Listen Sockets are assigned to child processes in a round robin fashion.
• If ListenCoresBucketRatio is not specified (default case), a single listen socket is created.
• Depending on the value of ListenCoresBucketRatio, the listener threads in all child process use the same listen socket to listen for incoming connections or have separate aka distinct listen sockets.

System calls executed in the parent HTTPD process.

The listener thread in each child process registers the listen socket/(s) using epoll_ctl() for event notification to their epoll instance. The threads create a unique epoll instance using epoll_create1 system call and do not share the epoll instance.

In the example below,

• Listener thread in process ID 28733 registers listen socket with socket number 3 for EPOLLIN events.
• Listener thread in process ID 28744 registers listen socket number 5 for EPOLLIN events.

28773 epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN, {u32=331335488, u64=94687180343104}} <unfinished ...>

28774 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN, {u32=331332736, u64=94687180340352}} <unfinished ...> 

When ListenCoresBucketRatio is non-zero:

In case ListenCoresBucketRatio is non-zero, it helps to scale out accepting connections as the network stack selects a listen socket among the many sockets waiting on the given interface:port combination as described below:

• The listener threads waits for events using the epoll_wait() API
• When a client connects to the interface:port corresponding to the listen sockets, the network stack selects a listen socket among the many listen sockets that are listening on the given interface:port combination.
• The incoming connection enters the SYN queue of the selected listen socket and waits in the SYN queue until the 3 way handshake completes.
• Once the 3-way handshake is complete, the connection is moved to the accept queue of the socket.
• In our example, either process with PID 28773 (socket with socketfd 3) or 28774 (socket with socketfd 5) may be selected to handle the incoming connection.
• The epoll_wait() API of the corresponding listen socket returns with socket number.
• The listener thread then calls accept() using the socket number returned by epoll_wait()

When ListenCoresBucketRatio is zero

• In this case, the listener thread in every child process adds the same listen socket to its own unique epoll instance.
• When the client connects to the interface:port on which the listener threads are listening, the epoll API selects the thread that must consume the new connection.
• It is observed that epoll does not distribute this load evenly and selects the same thread repeatedly.

An excellent article reading this behavior of epoll can be found at:

https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/