Programming odds and ends — InfiniBand, RDMA, and low-latency networking for now.


About This Blog
For now, I’ll be writing mainly about building RDMA-enabled applications for InfiniBand fabrics and s3fuse, my pet project. I’ll probably start writing about other, possibly non-technical, topics as time goes on.

About Me
I’m 30-something, live in California, and work in tech.

21 responses

  1. qiang

    HI,I’m a novice for infiniband,I feel depressed because of too less related resources.Could you recommend me some resources or a simple example about communicaton program on ibverbs. thank you

    March 27, 2011 at 3:01 am

    • Sure — have a look at my multi-part tutorial and at the examples that come with librdmacm and libibverbs. You might also want to consider MPI. In most cases, most of the time, it’s the easier/faster/better way to use an InfiniBand or 10 GigE fabric.

      March 27, 2011 at 12:21 pm

  2. Dan


    just as qiang posted I’m doing some work using infiniband and have trouble finding good resources on the verbs API.

    I’ve read your tutorial and found it very helpful, but the amout of documentation to be found online is pretty limited.

    Are there any books that you know of that covers the verbs API in a more accessible way than just trying to read the headers? Headers seldom convers information on *how* to use an API, just what the prototypes are…

    You must get your information from somewhere, right? 🙂



    March 30, 2011 at 3:47 am

    • Yes, IB documentation is pretty scarce. I’ve not come across any recent books on the subject, but there is quite a bit of information available on the web if you’re willing to dig around. There are man pages for all of the ibv_* (e.g., ibv_post_send()) and rdma_* (e.g., rdma_accept()) functions, there’s the IB specification, there are the archives of the linux-rdma list, and of course the numerous open-source projects that use IB, like OpenMPI and the sample programs included with libibverbs and librdmacm. It’s not ideal, but it’s better than nothing. 🙂

      March 30, 2011 at 9:36 pm

      • Hi all.

        I’ve been working for more than a year now to expand the knowledge on IB and RDMA API (at the beginning, especially the libibverbs API).

        This information is available in the blog RDMAmojo:

        Dotan Barak

        June 29, 2013 at 12:51 pm

  3. nic

    If you haven’t already heard about it, here is an awesome use of InfiniBand for data centers:

    February 8, 2013 at 5:53 pm

    • Certainly looks like a very interesting project — thanks for the pointer!

      February 10, 2013 at 12:37 pm

  4. Hi There,

    you cover two topics which I am actively involved in… XCP and Infiniband. I was wondering if i could pick your brain offline please?

    June 26, 2013 at 7:38 am

  5. Hi there,
    I’m currently trying to do some work in the RoCE space , on Linux and i’m still ploughing my way through linux-RDMA. as the RDMA part is new. Find that a lot of modules /libraries being used … Been getting some luck digging through using ltrace and ftrace, but was wondering if you could explain better or point me to some kind of basic High level design doc of the linux RDMA stack rather than me second-guessing it thru some stack traces ??
    Would certainly like to pick your brain offline over email if you wouldn’t object ?


    September 2, 2013 at 7:31 am

  6. SeongJae Park

    Hi, I am Infiniband newbie and this awesome blog was very helpful to me.
    Thank you very much for your effort and sharing! 🙂

    BTW, I encountered some build failure while building your sample codes and fixed them by editing the Makefile and uploaded the change on github(
    I’m not sure it would be helpful but I thought it would be better to let you know.

    If there is something wrong with my change, please let me know.

    September 16, 2013 at 7:47 am

    • Thank you for letting me know! I’ve updated my Makefiles and will post a link to a repository with all my sample code.

      October 4, 2013 at 8:22 pm

  7. Hi Geek, any chance you might have played with SR-IOV, & virtialization with infiniband? I have it working to a point but am running into issues with mkey errors and thought maybe you might know something about them.

    i.e. Oct 01 01:38:43 877836 [4D11B700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
    Oct 01 01:38:43 877855 [4D11B700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID

    October 1, 2013 at 3:45 am

  8. Pingback: Sample Code Repository | The Geek in the Corner

  9. Malek Musleh


    I am having some issues using rdma/ib_verbs with Infiniband within a VM using PCI-passthrough. I can see the IB devices, and ping to/from separate VMs. I am also able to run the basic Mellanox ib_write_lat tests without (-R flag) which specifies rdma_cm. I then tried to run your rdma-server client examples, and I get the following error on the client side:

    [root@b011-vm1 rdma]# ./rdma-client write 57598
    on_event: unknown event.

    the rdma_cm module seems to be loaded:
    [root@b011-vm1 rdma]# lsmod | grep rdma
    rdma_ucm 13178 0
    rdma_cm 34922 1 rdma_ucm
    iw_cm 8579 1 rdma_cm
    ib_addr 5900 1 rdma_cm
    ib_cm 41448 3 ib_ucm,rdma_cm,ib_ipoib
    ib_uverbs 41010 2 rdma_ucm,ib_ucm
    ib_sa 24010 5 rdma_ucm,rdma_cm,ib_ipoib,ib_cm,mlx4_ib
    ib_core 81975 12 rdma_ucm,ib_ucm,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx5_ib,mlx4_ib,ib_sa,ib_mad
    compat 22986 17 rdma_ucm,ib_ucm,rdma_cm,iw_cm,ib_addr,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx5_ib,mlx5_core,mlx4_en,mlx4_ib,ib_sa,ib_mad,ib_core,mlx4_core

    Do you have any ideas? I know it must be something related to the rdma module.



    October 7, 2013 at 2:07 am

    • That’s interesting. I made a small change to the sample code so it’ll now print the event type before aborting. Please try it and report back.

      October 17, 2013 at 9:46 am

  10. Walker

    Hi, I’m new and wonder able to create a simple p2p program with RDMA myself using normal NIC?

    December 9, 2013 at 9:09 pm

  11. chris


    Big thanks for the great code samples and the descriptions concerning RDMA and InfiniBand. This samples really helped us a lot. We used it for our project at university, where we had to implement a data transfer using InfiniBand. Therefore we ported your 3rd sample to Windows, modified it a bit and it works really great. Thanks!

    I’m curious about further papers concerning a speed-improvement as you mentionend in your third paper.


    January 30, 2014 at 8:31 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s