Linux, like most POSIX / System V compatible operating systems prefer processes over threads, as a matter of fact, a POSIX thread is nothing but a process with a layer of abstraction. The main motive of thread usage is the ability to share memory between several running instances and components of the program in a transparent way. Since threads have a heavy performance penalty in Linux, the preferred method of sharing an object is the use of System V SHM (SHared Memory) virtual file.

Basic SHM setup

As mentioned before, our goal is writing a program that shares a struct in C with all running instances of the software, no matter when they run. The implementation will be something in the lines of:

#include <fcntl.h>
#include <sys/stat.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/types.h> //shm_open
#include <stdio.h>  //printf
#include <stdlib.h> //exit
#include <unistd.h> //close
#include <string.h> //strerror

/* This will be created under /dev/shm/ */
#define STATE_FILE "/program.shared" 
#define  NAMESIZE 1024
#define   MAXNAMES 100

/* Define a struct we wish to share. Notice that we will allocate 
 * only sizeof SHARED_VAR, so all sizes are constant              
 */
typedef struct
{
  char name[MAXNAMES][NAMESIZE];
  int flags;
}  SHARED_VAR;

int main (void)
{
int first = 0;
int shm_fd;
static SHARED_VAR *conf;

  /* Try to open the shm instance with  O_EXCL,
   * this tests if the shm is already opened by someone else 
   */
  if((shm_fd = shm_open(STATE_FILE, (O_CREAT | O_EXCL | O_RDWR), 
                       (S_IREAD | S_IWRITE))) > 0 ) {
          first = 1; /* We are the first instance */
  }
  else if((shm_fd = shm_open(STATE_FILE, (O_CREAT | O_RDWR), 
                        (S_IREAD | S_IWRITE))) < 0) {
   /* Try to open the shm instance normally and share it with 
    * existing clients 
    */
    printf("Could not create shm object. %s\n", strerror(errno));
    return errno;
  } 

  /* Set the size of the SHM to be the size of the struct. */
  ftruncate(shm_fd, sizeof(SHARED_VAR));

  /* Connect the conf pointer to set to the shared memory area,
   * with desired permissions 
   */
  if((conf =  mmap(0, sizeof(SHARED_VAR), (PROT_READ | PROT_WRITE), 
                   MAP_SHARED, shm_fd, 0)) == MAP_FAILED) {

    return errno;

  }
  if(first) {
   /* Run a set up for the first time, fill some args */
   printf("First creation of the shm. Setting up default values\n");
   conf->flags = 4;
  }
  else
  {
    printf("Value of flags = %d\n", conf->flags);
  }

/* Do some work... */

  close(shm_fd);
  exit(0);
}

Notice the code should be compiled against librt like so: gcc -lrt -o shm shm.c

Once the shared memory has been set up, every time the program loads (until reboot or deletion of the virtual file), it can connect to the same shared memory location, get and store it’s data.

When running the binary, you will notice the shm object was created in /dev:

ami@codemaestro:~$ ls -l /dev/shm/
-rw------- 1 ami ami 102404 Jul 27 09:34 program.shared

Access Control and Synchronization

It is clear that the shm mechanism provides bare-bones tools for the user. All access control must be taken care of by the programmer. Locking and synchronization is being kindly provided by the kernel, this means the user have less worries about race conditions. Note that this model provides only a symmetric way of sharing data between processes. If a process wishes to notify another process that new data has been inserted to the shared memory, it will have to use signals, message queues, pipes, sockets, or other types of IPC.

Shared Kernel memory

Although it’s frighteningly more dangerous, sharing memory can also be done between kernel space and user space. For those of you wondering why on earth will they want to do this, the answer is simple – Performance ; Why copy memory when you can share it? For example, imagine the performance boost for a network appliance if its network interface doesn’t have to copy each and every packet’s data to user space, but only send its memory address to a user space network application that can read from the shared kernel memory section.

Defining a shared memory space between kernel space and user space is not much different from the usual user space shared memory, but as mentioned before, the possible damage is far greater in case of a bug.

Describing how to write a kernel module that shares some kernel memory is outside the scope of this article. However, a complete detailed example of exposing shared memory from the kernel for writing a network driver can be found in an article “Network Buffers and memory management” [3].

The Registry Model

SHM allows a simple management of a knowledge base to be shared between processes. The facilities supplied with SHM make it easy to implement the following model:
Registry manager process loads the registry from file and maps it to SHM.
Child / sibling processes can read / write entries with managed permissions.
Registry manager saves desired entries to file (on every write for example).

Via SHM we can detect if we are the first running instance and manage permissions. If we combine this with a notification mechanism, we can create an efficient way to share structured data between processes. More on this in the referenced article.

A More Advanced Model

Sachin Agrawal and Swati P. Udas from IBM published an article that provides an infrastructure for shared memory in high level languages like C++. The referenced article [2] shows the use of SHM in object oriented environment, as well as a system for IPC.

References

Posted by Ami Chayun | | [4] comments

4 Comments »

  1. Do you have a source for the “heavy performance penalty”?

    Comment by tom — February 18, 2008 @ 9:53 pm

  2. Thanks for the info, I think this solves a problem for me. I create linked lists [that is chained structure] in a shared memory seg. First program loads the data [lots of it about 10 lists including processes to start into shm. second process [an exec] reads the data in the shm, sets up coms to about 100 remote nodes, fires up processes local and remote , and supervises the “system”. Only under linux the second process cannot see the linked list. The start of a list is a fixed offset from the start of shm but the back and next pointers of the next “link” point to locations with junk. No data, no valid next. I will try this shm file to see if it will get me out of a problem. I ran your test code and it looks good. So again thanks for the info. best regards, (dja.)

    Comment by dave — May 10, 2008 @ 6:59 pm

  3. Hi!
    Dave , I am trying to implement linked list in shared memory , however I am facing problem. If I have a prev and next pointer , what shall it contain the offset from start of the shared memory address ?? this is okay only if my nodes are not same in size , otherwise having next and prev pointers are redundant as incrementing and decrementing by node size can get me the previous and next node. Any help in this regard will be useful. TX

    Comment by Neil — December 15, 2008 @ 11:18 am

  4. It’s not working for me.
    For two different programs, they appear to access different sets of memory.
    ?
    There is only one instance of /dev/shm/mymem

    Comment by John Wallis — September 23, 2010 @ 4:07 pm

RSS feed for comments on this post. TrackBack URI

Leave a comment