How to Run Neanderthal With ATLAS BLAS From Ubuntu Repositories

penguin-161418

Neanderthal is an excellent Clojure library that provides ridiculously fast matrix operations. It has two native implementations under the hood: ATLAS BLAS for CPU and OpenCL for GPU computation. Basically, it is a must have for any machine learning application with a data sample larger than a “Hello world” example.

The one catch with a super-fast native library is that we are supposed to compile it from source. At least, if we expect to have the top performance, that is. Of course, it is also possible to cut the line and just apt-get it. You wouldn’t get the top performance, but on the other hand it is likely to be faster than what you would’ve been using anyway.

In my case, I wanted to be able to run the Neanderthal examples on my spare laptop and it didn’t make sense to waste too much effort compiling stuff for inferior hardware. All I needed was a working library, so apt-get it is.

First install the native dependencies:

sudo apt-get install libatlas-base-dev

Then check out the project, start a REPL and try out the example from Neanderthal docs:

(use '[uncomplicate.neanderthal core native])

(def x (dv 1 2 3))
(def y (dv 10 20 30))
(dot x y)

However, instead of the expected 140.0, what we get is the following error:

java: symbol lookup error: /tmp/libneanderthal-atlas-0.3.14576897804109763789.so: undefined symbol: cblas_ddot

The same happened for the other functions – same error, just a different undefined symbol name. Now, what happened there is that for some reason Ubuntu ships ATLAS as two separate library files:

  • /usr/lib/libatlas.so
  • /usr/lib/liblapack_atlas.so

Neanderthal only tries to load the first one, but it appears that the second one contains some important symbols, so it must be loaded too. The fix is to inject it with LD_PRELOAD like this:

LD_PRELOAD=/usr/lib/liblapack_atlas.so lein repl

And then the same example gives the correct result!

To summarize, all you need to run Neanderthal with a pre-compiled Atlas on your Ubuntu machine are the following two lines:

sudo apt-get install libatlas-base-dev
LD_PRELOAD=/usr/lib/liblapack_atlas.so lein repl

Happy hacking!

Written by Goran Jovic

Software developer and founder of Big Solutions. He has been working with Clojure since 2010 on a number of projects usually involving Big Data processing and analytics. His current focus is application of Clojure-fu on IoT integration.

View all author posts ->

6 thoughts on “How to Run Neanderthal With ATLAS BLAS From Ubuntu Repositories

  1. Thanks a lot for your solution, because I suffered from the same problem.

    However, could you tell me how I can get the same result in emacs cider repl, not in raw lein repl? Because I don’t know how to set the LD_PRELOAD=/usr/lib/liblapack_atlas.so before invoking M-x cider-jack-in.

    1. I’m not familiar with cider, but if it is using nrepl like the rest of the tools, I guess you should add it to the place where you start the nrepl server. Now, I’ve no idea where that’s located with cider-jack-in. The only idea, I can offer is to connect cider to a manually started nrepl server, e.g. with lein or boot, where repl server has a pre-loaded library,

      The other idea is to try to prefix the LD_PRELOAD on the command you use to start emacs, but I’ve no clue if that would work.

      In any case, if you do figure it out, please share with the rest of us – others are likely to have similar issues.

      1. I found out the way to set the shell environment variable LD_PRELOAD=/usr/lib/liblapack_atlas.so in project.clj using lein-environ plugin. So I would like to share that way with others and thanks again for the awesome hacking and kind reply of Goran Jovic!

        ;; project.clj
        (defproject linear “0.1.0-SNAPSHOT”
        :dependencies [[org.clojure/clojure “1.9.0-alpha14”]
        [uncomplicate/neanderthal “0.8.0”]]
        :plugins [[lein-environ “1.1.0”]]
        :env {:ld-preload “/usr/lib/liblapack_atlas.so”})

        ;; test sample
        (ns linear.core
        (:use (uncomplicate.neanderthal core native)))

        (def x (dv 1 2 3))
        (def y (dv 10 20 30))

        (dot x y)
        ; => 140.0

        (def a (dge 3 2 [1 2 3 4 5 6]))
        (def b (dge 2 3 [10 20 30 40 50 60]))

        (mm a b)
        ; => #RealGeneralMatrix[double, ord:COL, mxn:3×3, ld:3]
        ; ((90.0 120.0 150.0) (190.0 260.0 330.0) (290.0 400.0 510.0))

          1. I am very sorry for my serious confusion. I had tried the following trick before attempting the previous way.

            ;; ~/.profile
            export LD_PRELOAD=/usr/lib/liblapack_atlas.so

            I forgot about the above line and tried the previous way and it worked.

            However after having realized that I forgot to delete the above line, I removed the above line and tried again the previous way but It unfortunately didn’t work.

            Very sorry about this mistake of mine and I will try another way if it is possible.

  2. The following is another way to set the shell environment variable in ~/.emacs.d/init.el, though it is not elegant in my opinion. However it works anyway.

    ;; ~/.emacs.d/init.el
    (defun my-clojure-mode-init ()
    “Initializes clojure mode.”
    (interactive)

    ;; ……
    (setenv “LD_PRELOAD” “/usr/lib/liblapack_atlas.so”))

    (add-hook ‘clojure-mode-hook ‘my-clojure-mode-init)

Leave a reply

Your email address will not be published. Required fields are marked *