Skip to contents

14 Speed It Up (Why So Slow?)

2026-06-01

Source: vignettes/articles/14_Speed_It_Up.Rmd
14_Speed_It_Up.Rmd

AutoSpectral takes a lot longer to unmix files than standard OLS or WLS unmixing approaches. Most of that is because it’s doing way more calculations. If we consider a 40-color panel with a million cells, there is essentially only a single computationally expensive inversion to produce the unmixing matrix and then a second slow step applying that to the data. In the per-cell optimization part of unmix.autospectral(), AutoSpectral performs up to 100 such calculations per cell per fluorophore. In other words, for that same set of a million cells in 40 color space, as much as 100 x 40 x 1000000 = 4x10^9 times as many computationally intensive steps. The other reason it’s slow is that I am not a computational person.

There are some things that can be done to improve this. Release v1.0.0 implements an approach to reduce the search space, somewhere between 10 and 100x, which will help. AutoSpectralRcpp now integrates the full unmixing pipeline in C++, which also helps.

Go Faster

R is not known for its speed.

For faster processing there are three things you can do, hopefully all fairly easy.

First, upgrade the BLAS and LAPACK libraries used by R. These provide algorithms for linear algebra, which is the heart of spectral unmixing.

On Windows, simply swapping out your .dll files as in this tutorial can give speed ups of 5x. Install OpenBLAS All this involves is downloading the files from the internet, placing them in the right folder and doing a quick restart.

On MacOS Sonoma/Sequoia (most modern Macs), you don’t need to do anything about the BLAS. The Accelerate BLAS that you already have is optimized for your system.

For older MacOS, the following articles may be helpful: BLAS for Mac in R Performance BLAS Feedback from users on Mac who successfully upgrade their BLAS would be appreciated.

Do not set multiple threads for the BLAS as this will conflict with higher level parallelization, either in AutoSpectral or in other multithreaded libraries. That said, AutoSpectral checks for this internally, overriding anything you do in that regard.

After upgrading the BLAS on Windows or Linux, you probably need to re-install Rcpp and RcppArmadillo to have these compiled against the new BLAS. On Mac with Accelerate this is not necessary as Accelerate is a system framework linked at build time.

install.packages( c( "Rcpp", "RcppArmadillo" ) )

Now, install AutoSpectralRcpp. This is fully accessible from R and integrates with AutoSpectral. But, when it gets to the slow bits in the unmixing and several other aspects of the processing, it switches over to calculating in C++, so it can be 10-100x faster. Do this after upgrading the BLAS because the C++ gets compiled with whatever BLAS you’ve got installed.

AutoSpectralRcpp

AutoSpectralRcpp requires a C++ compiler. What you need depends on your platform:

  • Windows: Install Rtools. This provides the compiler and takes around 10 minutes to download and install. OpenMP support is included and enabled automatically.

  • macOS: Apple’s built-in compiler does not include OpenMP support. To get the best performance from AutoSpectralRcpp on a Mac, you need to install two things from the Terminal: Homebrew (a package manager) and libomp (the OpenMP library). Neither requires any programming knowledge — it is just a matter of copying and pasting a couple of commands. AutoSpectralRcpp will detect these automatically and enable OpenMP at install time. If they are not present, the package will still install and work correctly, just without multi-threading.

    Step 1: Open Terminal

    Terminal is a built-in Mac application. You can find it by opening Spotlight (press Cmd + Space), typing “Terminal”, and pressing Enter.

    Step 2: Install the Xcode Command Line Tools

    These provide the C++ compiler that R needs to build packages like AutoSpectralRcpp. Paste the following into Terminal and press Enter:

  xcode-select --install

A dialog box will appear asking you to confirm. Click “Install” and wait for it to finish — this may take a few minutes. If you see a message saying the tools are already installed, you can skip ahead.

Step 3: Install Homebrew

Homebrew is a free, widely-used package manager for macOS that makes it easy to install software from the Terminal. Paste the following into Terminal and press Enter:

  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

You will be asked for your Mac login password. As you type it, nothing will appear on screen — this is normal. Press Enter when done. The installer will then run and may take a few minutes. At the end it will print a message about “Next steps” — follow any instructions it gives about adding Homebrew to your PATH (it will give you two lines to paste into Terminal; paste and run them).

Step 4: Install libomp

Once Homebrew is installed, installing libomp is straightforward. Paste the following into Terminal and press Enter:

  brew install libomp

This should only take a minute or so.

Step 5: Install or reinstall AutoSpectralRcpp

Now install (or reinstall) AutoSpectralRcpp from R. The configure script will find libomp automatically and compile with OpenMP enabled:

  remotes::install_github("DrCytometer/AutoSpectralRcpp")

You should see a line in the output confirming OpenMP was found, e.g.: configure: OpenMP enabled using libomp at /opt/homebrew/opt/libomp

If you do not see this, or if you see a warning that libomp was not found, double-check that Steps 3 and 4 completed successfully and try reinstalling.

  • Linux: GCC is typically already present. OpenMP is enabled automatically. If you’re using Linux, you probably know what you’re doing.

You can install AutoSpectralRcpp like so:

remotes::install_github("DrCytometer/AutoSpectralRcpp")

Third, turn on parallel processing in AutoSpectral. Update: This is now supported via a parallel backend. Importantly, this moves away from future and future_lapply, which were aborting sometimes on Windows. More usefully, this is a bit faster in many cases, and should support forking via mclapply on Mac and Linux systems, which will be much faster. Perhaps most usefully, this appears to allow parallelization of the native R per-cell fluorophore optimization in unmix.autospectral, so you should see a nice speed up there.

The parallel processing in AutoSpectralRcpp operates via OpenMP. On Windows and Linux, OpenMP is always enabled. On macOS, it is enabled if libomp was present at install time (see above). The number of threads can be configured regardless of platform.

To activate parallel processing, check the function arguments for a parallel option and set it to TRUE. Additionally, there is control over the number of threads used, which should be directly in the function call via a threads argument. If you don’t know how many threads to use, check the recommendation for your machine after running get.autospectral.param():

asp$worker.process.n

Multi-threading will always default to this number if you do not explicitly set a number of threads and set parallel=TRUE. This is one less than parallelly::availableCores(), so it is designed to allow you to keep working on minor stuff while AutoSpectral chugs along in the background.

Or just check how many you have available:

parallelly::availableCores()

For unmixing larger data sets, you will do well to use a machine with more CPUs. Version 1.0.0 brings faster processing for the unmixing.

Installation and Runtime

Installation via GitHub should take only a minute or so. It takes less than that on my Dell i7 8-core Windows laptop, but that may also be because I have already installed the dependencies.

Occasionally, the help gets corrupted. Just re-install if that happens. If you know why this happens, let me know.

Installation of AutoSpectralRcpp will take a couple of minutes because the code needs to compile. A C++ compiler is required — see the platform-specific instructions above. Upgrading the BLAS and LAPACK (see above) also takes several minutes if you’re taking the time to read the instructions carefully.

Run-time will vary considerably depending on the size and quantity of files you’re processing. For the benchmark 42-color Aurora dataset using single-stained cell controls with plenty of data, the slow steps in the pre-processing pipeline are define.flow.control() and clean.controls(). For the same Aurora dataset on the i7 Windows laptop:

These should run faster in v1.0.0 for datasets with lots of cells because I have changed the default plotting settings to limit the maximum number of events plotted (plotting lots of points is slow in R, even with the accelerated scattermore package).

Unmixing time depends on the following variable:

  • file size (number of cells/events, including debris)
  • number of detectors (ID7000 data takes a bit longer)
  • number of fluorophores
  • unmixing algorithm

OLS and WLS are fast, per-cell AF extraction will be slower, per-cell fluorophore optimization will be slower still. As of version 1.0.0, the time required is down quite a bit (see below).

If you are not using AutoSpectralRcpp, then the per-cell methods will likely run about 10x slower.

Benchmarks for “C3 Lung_GFP_003_Samples.fcs”, again on the 8-core laptop, using OpenBLAS and AutoSpectralRcpp, where applicable:

  • unmix.fcs() WLS or OLS v0.8.7: 9sec

  • unmix.fcs() WLS or OLS v0.9.0: 9sec

  • unmix.fcs() WLS or OLS v1.0.0: 9-10sec (most of this is handling the FCS file)

  • unmix.fcs() WLS or OLS v1.1.0: 4sec (now with native FCS handling)

  • unmix.fcs() perCell AF extraction v0.8.7: 2min

  • unmix.fcs() perCell AF extraction v0.9.0: 1min

  • unmix.fcs() perCell AF extraction v1.0.0: 11sec (45s in R without C++)

  • unmix.fcs() perCell AF extraction v1.0.3: 10sec

  • unmix.fcs() perCell AF extraction v1.1.0: 4.5sec (now with native FCS handling)

  • unmix.fcs() perCell AF extraction v1.6.0: 3.5sec

  • unmix.fcs() perCell fluorophore optimization “fast” v0.8.7: 9min

  • unmix.fcs() perCell fluorophore optimization “fast” v0.9.0: <2min

  • unmix.fcs() perCell fluorophore optimization “fast” v1.0.0: <2min (note: performance here should now be comparable to the previous “slow”)

  • unmix.fcs() perCell fluorophore optimization “fast” v1.0.2: 1.3min

  • unmix.fcs() perCell fluorophore optimization “fast” v1.6.0: 50sec

  • unmix.fcs() perCell fluorophore optimization “slow” v0.8.7: 62min

  • unmix.fcs() perCell fluorophore optimization “slow” v0.9.0: 19min

  • unmix.fcs() perCell fluorophore optimization “slow” v1.0.0: 11min

  • unmix.fcs() perCell fluorophore optimization “slow” v1.0.2: 9min

  • unmix.fcs() perCell fluorophore optimization “slow” v1.6.0: 3.2min

  • unmix.fcs() perCell fluorophore optimization “joint” v1.6.0: 2.9min (note: new default pipeline to eliminate discontinuities)

  • unmix.folder() WLS or OLS, 6 files, v0.8.7: 67sec sequential, interrupted parallel

  • unmix.folder() WLS or OLS, 6 files, v0.9.0: 62sec sequential, 31sec parallel

If I have not updated the benchmarks, it is because I do not expect much improvement since last check. That said, since v1.0.2, native FCS read/write via C++ has been added, so things should be a bit faster, particularly for large files.

I do not expect more major gains, but I will look into GPU acceleration.