Building an RDMA-capable application with IB verbs, part 1: basics

If you’re looking to build an application that uses InfiniBand natively, now would be a good time to ask yourself if you wouldn’t be better off using one of InfiniBand’s upper-layer protocols (ULPs), such as IP-over-IB/SDP or RDS, or, most obviously, MPI. Writing programs using the verbs library (libibverbs, but I’ll refer to it as ibverbs) isn’t hard, but why reinvent the wheel?

My own reasons for choosing ibverbs rather than MPI or any of the available ULPs had to do with comparative performance advantages over IPoIB and that my target applications are ill-suited to the MPI message-passing model. MPI-2’s one-sided communication semantics would probably have worked, but for reasons irrelevant to this discussion MPI is/was a non-starter anyway.

Before looking at the details of programming with ibverbs, we should cover some prerequisites. I strongly recommend reading though the InfiniBand Trade Association’s introduction — chapters one and four in particular (only thirteen pages!). I’m also going to assume that you’re comfortable programming in C, and have at least passing familiarity with sockets, MPI, and networking in general.

Our goal is to connect two applications such that they can exchange data. With reliable, connection-oriented sockets (i.e., SOCK_STREAM) , this involves setting up a listening socket on the server side, and connecting to it from the client side. Once a connection is established, either side can call send() and recv() to transfer data. This doesn’t change much with ibverbs, but things are done in a much more explicit manner. The significant differences are:

• You’re not limited to send() and recv(). Reading and writing directly from/to remote memory (i.e., RDMA) is enormously useful.
• Everything is asynchronous. Requests are made and notification is received at some point in the future that they have (or have not) completed.
• At the application level, nothing is buffered. Receives have to be posted before sends. Memory used for a send request cannot be modified until the request has completed.
• Memory used for send/receive operations has to be registered, which effectively “pins” it such that it isn’t swapped out.

So in an InfiniBand world, how do we establish connections between applications? If you’ve read the IBTA’s introduction you’ll know that the key components we need to set up are the queue pair (consisting of a send queue and a receive queue on which we post send and receive operations, respectively) and the completion queue, on which we receive notification that our operations have completed. Each side of a connection will have a send-receive queue pair and a completion queue (but note that the mapping between an individual send or receive queue and completion queues within any given application can be many-to-one). I’m going to focus on the reliable, connected service (similar to TCP) for now. In later posts I’ll explore the datagram service.

Building queue pairs and connecting them to each other, such that operations posted on one side are executed on the other, involves the following steps:

1. Create a protection domain (which associates queue pairs, completion queues, memory registrations, etc.), a completion queue, and a send-receive queue pair.
2. Determine the queue pair’s address.
3. Communicate the address to the other node (through some out-of-band mechanism).
4. Transition the queue pair to the ready-to-receive (RTR) state and then the ready-to-send (RTS) state.
5. Post send, receive, etc. operations as appropriate.

Step four in particular isn’t very pleasant, so we’ll use a event-driven connection manager (CM) to connect queue pairs, manage state transitions, and handle errors. We could use the InfiniBand Connection Manager (ib_cm), but the RDMA Connection Manager (available in librdmacm, and also known as the connection manager abstraction) uses a higher-level IP address/port number abstraction that should be familiar to anyone who’s written a sockets program.

This gives us two distinct procedures, one for the passive (responder) side of the connection, and another for the active (initiator) side:

Passive Side

1. Create an event channel so that we can receive rdmacm events, such as connection-request and connection-established notifications.
2. Bind to an address.
3. Create a listener and return the port/address.
4. Wait for a connection request.
5. Create a protection domain, completion queue, and send-receive queue pair.
6. Accept the connection request.
7. Wait for the connection to be established.
8. Post operations as appropriate.

Active Side

1. Create an event channel so that we can receive rdmacm events, such as address-resolved, route-resolved, and connection-established notifications.
2. Create a connection identifier.
3. Resolve the peer’s address, which binds the connection identifier to a local RDMA device.
4. Create a protection domain, completion queue, and send-receive queue pair.
5. Resolve the route to the peer.
6. Connect.
7. Wait for the connection to be established.
8. Post operations as appropriate.

Both sides will share a fair amount of code — steps one, five, seven, and eight on the passive side are roughly equivalent to steps one, four, seven, and eight on the active side. It may or may not be worth pointing out that as with sockets once the connection has been established, both sides are peers. Making use of the connection requires that we post operations on the queue pair. Receive operations are posted (unsurprisingly) on the receive queue. On the send queue, we post send requests, RDMA read/write requests, and atomic operation requests.

The next two posts will describe in detail the construction of two applications: one will act as the passive/server side and the other will act as the active/client side. Once connected, the applications will exchange a simple message and disconnect.

If you haven’t already, download and install the OpenFabrics software stack. You’ll need it to build the sample code provided in the next posts.

Updated, Nov. 6: Fixed link to IBTA introduction.