8.7.5 Accessing Lisp Arrays

Due to the way CMU Common Lisp manages memory,the amount of memory that can be dynamically allocated by malloc or  make-alien (page ) is limited.

To overcome this limitation, it is possible to access the content of Lisp arrays which are limited only by the amount of physical memory and swap space available. However, this technique is only useful if the foreign function takes pointers to memory instead of allocating memory for itself. In this case, you will have to modify the foreign functions.

This technique takes advantage of the fact that CMU Common Lisp has specialized array types (see section 5.11.8) that match a typical C array. For example, a (simple-array double-float (100)) is laid out in memory in essentially the same way as the C array double x[100] would be. The following function allows us to get the physical address of such a Lisp array:

(defun vector-data-address (vec)
  "Return the physical address of where the actual data of the vector
VEC is stored.

  VEC - must be a specialized array type in CMU Lisp.
Returns
  1   - physical address (unsigned-byte 32)"
  (let ((base-address (the (unsigned-byte 32)
                        (logandc1 7 (kernel:get-lisp-obj-address vec)))))
    (declare (type (unsigned-byte 32) base-address))
    ;; For a simple-array, memory is laid out like this:
    ;;
    ;;   byte offset    Value
    ;;        0         type code (should be 70 for double-float vector)
    ;;        4         4 * number of elements in vector
    ;;        8         1st element of vector
    ;;       12         2nd element of vector
    ;;      ...         ...
    ;;

    (the (unsigned-byte 32) (+ 8 base-address))))

Assume we have the C function below that we wish to use:

double dotprod(double* x, double* y, int n)
  {
    int k;
    double sum = 0;

    for (k = 0; k < n; ++k) {
      sum += x[k] * y[k];
    }
  }

The following example generates two large arrays in Lisp, and calls the C function to do the desired computation. This would not have been possible using malloc or make-alien since we need about 16 MB of memory to hold the two arrays.

(def-alien-routine "dotprod" double
    (x (* double-float) :in)
    (y (* double-float) :in)
    (n int :in))

  (let ((x (make-array 1000000 :element-type 'double-float))
        (y (make-array 1000000 :element-type 'double-float)))
    ;; Initialize X and Y somehow
    (let ((x-addr (system:int-sap (vector-data-address x)))
          (y-addr (system:int-sap (vector-data-address y))))
      (dotprod x-addr y-addr 1000000)))    

In this example, it may be useful to wrap the inner let expression in an unwind-protect that first turns off garbage collection and then turns garbage collection on afterwards. This will prevent garbage collection from moving x and y after we have obtained the (now erroneous) addresses but before the call to dotprod is made.