I’ve been asked to help people out with libev example code and usage for a while but I never got around to it since I hadnt used libev in ages. However I have a need for a simple server framework (more on this in another post). So I figured I’d publish a simple libev tutorial. More importantly I will create a C library that is a wrapper on top of libev to handle and maintain multiple connections in a clean and extendible way.

A quick introduction to libev is in order. Back in the day, handling multiple network client connections was either done using multiple threads (one thread per connuction) or via asynchronous apis that multiplexed between several io events across the various connections. Several APIs existed to enable the later (select/poll, epoll, kqueue etc). Each of the methods had their own performance guarantees and worse still had platform dependant compatibility issues. To get around this libevent was developed to provide a uniform interface hiding platform specific details. Libev developed later was designed to simplify and cleanup several aspects of libevent (eg stripped down, threadsafe, low watcher overhead etc).

The core concept in libev is that of an event loop – a loop that runs continuously listening registered events and calling the associated callback functions. Events could be io events like file descriptors ready for reads/writes, socket connections accepted etc.

With that it is time to dive in.

Let us first start with a socket server structure that is the context for all things related to a multi client TCP server:

 1struct LEWSocketServer {
 2    // A pointer to the event loop that is handling event on
 3    // this server socket.
 4    struct ev_loop *eventLoop;
 5
 6    // Host the server socket is bound to (not used for now)
 7    char *host;
 8
 9    // Port the server is running on
10    int port;
11
12    // The socket associated with the server.
13    int serverSocket;
14    struct ev_io acceptWatcher;
15
16    // A listener structure for events on this socket.
17    LEWSocketServerListener *listener;
18};

A simple server socket API would begin with being able to start a server:

 1typedef struct LEWSocketServerListener {
 2    /**
 3     * Called when a connection was accepted.  This is an opportunity
 4     * for the handler of this method to create any connection specific data
 5     * to be created and returned so that any further activities on the connection
 6     * will be invoked on this object.
 7     *
 8     * This method must NOT return NULL.  If it returns NULL, then the connection is refused.
 9     */
10    void *(*createConnectionContext)();
11
12    /**
13     * Called when data has been received for this connection from a client.
14     */
15    void (*processData)(LEWConnection *connection, const char *data, size_t length);
16
17    /**
18     * Called to indicate connection was closed.
19     */
20    void (*connectionClosed)(LEWConnection *connection);
21
22    /**
23     * Called to ask the connection handler for data that can be written.
24     * The buffer is an output parameters to be updated by the listener.
25     * Return the number of bytes available in the buffer.
26     */
27    size_t (*writeDataRequested)(LEWConnection *connection, const char **buffer);
28
29    /**
30     * Called to indicate that nWritten bytes of data has been written and that the connection
31     * object should update its write buffers to discard this data.
32     */
33    void (*dataWritten)(LEWConnection *connection, size_t nWritten);
34} LEWSocketServerListener;

The general structure of a connection follows:

Server socket is in listen state
When a new connection is accepted, a client socket is created and added libev’s eventloop for read events.
write events for the client socket are not enabled. The level-triggered nature of libev (by default) will cause unnecessary write event callbacks even when there is no data to be sent. So a design choice made was to make the api caller responsible to initiate writes when it had data to be sent.
When data is available to be read, it is sent via the listener’s processData callback (along with the connection object associated with the client).
When the caller has data to write it invokes the connection’s writeable attribute.
When the writeable attribute on a connection is set, the write events on the client socket are enabled which invokes the writeDataRequested method on the caller until it return 0 (bytes).
Additionally the library calls the dataWritten callback on the listener so that the client can update its own write data buffers (to pop off the written/sent data).

With this the echo server now looks like:

 1#include "server.h"
 2
 3typedef struct EchoConnection {
 4    char readBuffer[4096];
 5    int length;
 6} EchoConnection;
 7
 8/**
 9 * Called when a connection was accepted.  This is an opportunity
10 * for the handler of this method to create any connection specific data
11 * to be created and returned so that any further activities on the connection
12 * will be invoked on this object.
13 *
14 * This method must NOT return NULL.  If it returns NULL, then the connection is refused.
15 */
16void *createConnectionContextCallback()
17{
18    EchoConnection *out = calloc(1, sizeof(EchoConnection));
19    return out;
20}
21
22/**
23 * Called when data has been received for this connection from a client.
24 */
25void processDataCallback(LEWConnection *connection, const char *data, size_t length)
26{
27    EchoConnection *echoconn = (EchoConnection *)lew_connection_get_context(connection);
28    memcpy(echoconn->readBuffer, data, length);
29    echoconn->length = length;
30    lew_connection_set_writeable(connection);
31}
32
33/**
34 * Called to indicate connection was closed.
35 */
36void connectionClosedCallback(LEWConnection *connection)
37{
38    printf("Connection closed...\n");
39}
40
41/**
42 * Called to ask the connection handler for data that can be written.
43 * The buffer is an output parameters to be updated by the listener.
44 * Return the number of bytes available in the buffer.
45 */
46size_t writeDataRequestedCallback(LEWConnection *connection, const char **buffer)
47{
48    printf("Write data requested...\n");
49    EchoConnection *echoconn = (EchoConnection *)lew_connection_get_context(connection);
50    *buffer = echoconn->readBuffer;
51    return echoconn->length;
52}
53
54/**
55 * Called to indicate that nWritten bytes of data has been written and that the connection
56 * object should update its write buffers to discard this data.
57 */
58void dataWrittenCallback(LEWConnection *connection, size_t nWritten)
59{
60    EchoConnection *echoconn = (EchoConnection *)lew_connection_get_context(connection);
61    echoconn->length -= nWritten;
62}
63
64int main(void)
65{
66    LEWSocketServerListener listener;
67    listener.createConnectionContext = createConnectionContextCallback;
68    listener.processData = processDataCallback;
69    listener.connectionClosed = connectionClosedCallback;
70    listener.writeDataRequested = writeDataRequestedCallback;
71    listener.dataWritten = dataWrittenCallback;
72
73    lew_start_server(NULL, 9999, &listener);
74}

While there is still some work to be done to handle edge cases this would be relegated to the library rather than leaking out to the client code and most the client code is actually to do with connection logic rather than messing about with event loops.

Full source code is available at: https://github.com/panyam/LibEvWrapper