Using Bioconda to streamline software installation for bioinformatics
2018-01-30
Andrew Perry,
Monash Bioinformatics Platform
Follow along: https://github.com/MonashBioinformaticsPlatform/bioconda-tutorial/blob/master/Bioconda_Installation.ipynb
Disclaimer: while I'm talking about it, I don't pretend to be a conda / Bioconda expert.
This my whirlwind reinterpretation based on a workshop delivered by Simon Gladman, Saskia Hiltemann and Eric Rasche "Packaging your bioinformatics tool with Bioconda and Galaxy".
Why conda ?¶
Manages self contained environments, including dependencies. No
sudorequired.Large ecosystem of precompiled packages, organized as 'channels' (eg conda-forge, bioconda)
Language agnostic (not only Python !)
Creating new packages is straightforward compared with many systems (a recipe composed of a short build.sh and meta.yml).
Interoperates gracefully with Python
virtualenvs andpip.Open Source tool (BSD), backed by a company (Continuum Analytics). Some compiled packages in the default 'channel' aren't Open Source.
Woah .. waddayamean 'environment'¶
Conda sets some UNIX shell environment variables, primarily PATH, that determine which copy of a binary is run when you type it's name.
Type:
env
and you will see your environment variables.
Why use Bioconda ?¶
Large ecosystem of pre-packaged software already available, ~250 contributors.
- !!!--> http://bioconda.github.io/recipes.html <---!!!
Conda environments aid reproducibilty - an
environment.ymlfile can record tool versions (+build number) that can be reliably reinstalled elsewhere.Is the supported tool installation method for Galaxy, so existing packages will be maintained and grow.
Quality control - well documented guidelines (http://bioconda.github.io/guidelines.html), automatic testing and package builds (TravisCI / CircleCI), Pull Requests reviewed by core team.
- Fun fact: Bioconda recipes don't blindly trust version numbers on tools, they use an md5 or sha256 checksum of the source tarball.
Docker and Singularity containers are automatically generated too (Biocontainers project)
Let's install Miniconda and some tools from Bioconda¶
Miniconda installs the conda package manager in your home directory, in it's own conda 'environment' (Miniconda is a trimmed down version of 'Anaconda').
Miniconda can be found at: https://conda.io/miniconda.html
cd ~
# macOS
curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
# or Linux
# curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh ./Miniconda3-latest-MacOSX-x86_64.sh --help
sh ./Miniconda3-latest-MacOSX-x86_64.sh
# sh ./Miniconda3-latest-MacOSX-x86_64.sh -b
Read license (press ENTER many times) then type 'yes'.
The installer adds line to ~/.bash_profile ( -s option prevents this ).
export PATH="$HOME/miniconda3/bin:$PATH"
conda --help
# Enable the bioconda 'channel' and some others
conda config --add channels defaults
conda config --add channels conda-forge
conda config --add channels bioconda
# This is the config file that was modified by the conda config command.
# Channels at the bottom take precedence
cat ~/.condarc
# Install some conda packages
conda install -y seqtk
# samtools version 0.1.15, build number 0
conda install -y samtools=0.1.15=0
# conda install kallisto
# conda install salmon=0.9.1 sailfish
# Take a look at where those packages were installed (default environment 'root')
ls ~/miniconda3/bin
Using environments¶
Let's create and switch to a new environment, install some packages, record what we installed, deactivate it.
https://conda.io/docs/user-guide/tasks/manage-environments.html
# Create a new conda environment
conda create -y --name new_env
# Activate it explicitly (actually not required after `conda create`)
source activate new_env
# Install some packages
conda install -y samtools=0.1.14
# By default, environments live under ~/miniconda3/envs
# Take a look
ls ~/miniconda3/envs/new_env/bin
which samtools
# You can export the current environment to an environment.yml
# (For Pythonistas, this functionally similar to a requirements.txt file)
# Keep the environment.yml with your analysis project
conda env export >environment.yml
cat environment.yml
# An alternative, if you've like to lock exact package versions with channel + version + build + md5 checksums
# Note that this is now platform specific, but gives better guarantees of a reproducible environment
conda list --show-channel-urls --md5 --explicit >package-list.txt
cat package-list.txt
# You can use these files to create new environments with those packages like:
# conda create --name reproduced_env --file environment.yml
# conda create --name locked_reproduced_env --file package-list.txt
# You can recreate the environment somewhere else with
# conda env create -f environment.yml
# One step toward reproducability !
# Leave the environment
source deactivate
What about apt, yum, etc¶
Targeted at system-wide packages - typically don't handle non-privileged installations or concurrent versions gracefully, if at all.
Official distributions (Debian, Ubuntu, Centos) software repositories rarely keep up to date with recent versions. Unofficial repositories sometimes fill this void (eg Debian Med).
Creating packages seems needlessly complex for mere mortals.
What about modules, LMOD etc¶
Well suited to using the shell environment (ie
PATH) to manage concurrent versions.LMOD environments can be 'additive' (many 'modules' can be loaded simultaneously, unlike conda environments).
LMOD isn't a package manager (doesn't handle download/compilation/installation)
What about pip and virtualenv ?¶
- Both come with Python, great for Python packages, large ecosystem of existing packages.
- Not well suited to non-Python packages (binaries and dependencies).
What about bio-ansible ?¶
- bio-ansible is awesome because we wrote it.
- Not awesome because we have to write every task to install new tools.
- But! bio-ansible also supports installing
condapackages as LMOD modules (we can add anything from Bioconda easily)
- But! bio-ansible also supports installing
- Allows unprivileged installation of tools in your home directory, and system-wide packages & config.
- Allows concurrent versions and 'stacking' environments via LMOD modules.
- Higher barrier to entry for casual users, per-project installs (IMO).
Things I've noticed conda doesn't do well¶
- Doesn't officially support 'stacking' environments, ala LMOD.
- Do we care ? Software is small, disk space is cheap, just make environments with the combinations of tool versions you need.
- Unofficially, adding
max_shlvl: 16to~/.condarcallows it (YMMV) (https://github.com/conda/conda/issues/3580).
- Recipes for building packages (coming up) specify dependency names but not the 'channel' they come from.
- No streamlined method to install from source (eg
conda install https://github.com/me/my_recipe), pre-compiled binary centric ?- Feature appears to have been considered and rejected: https://github.com/conda/conda/issues/306
Creating (Bio)conda packages¶
Lets look at the Bioconda recipe for seqtk¶
https://github.com/bioconda/bioconda-recipes/tree/master/recipes/seqtk
build.sh¶
#!/bin/bash
export C_INCLUDE_PATH=${PREFIX}/include
export LIBRARY_PATH=${PREFIX}/lib
make all
mkdir -p $PREFIX/bin
cp -f seqtk $PREFIX/bin/
PREFIX is the installation path in the conda build environment (in a configure script you'd do configure --prefix=$PREFIX)
meta.yaml¶
package:
name: seqtk
version: 1.2
source:
fn: v1.2.tar.gz
url: https://github.com/lh3/seqtk/archive/v1.2.tar.gz
md5: 255ffe05bf2f073dc57abcff97f11a37
build:
number: 0
requirements:
build:
- gcc # [not osx]
- llvm # [osx]
- zlib
run:
- zlib
about:
home: https://github.com/lh3/seqtk
license: MIT License
summary: Seqtk is a fast and lightweight tool for processing sequences in the FASTA or FASTQ format
test:
commands:
- seqtk seq
Building a recipe (vanilla conda way)¶
cd ~
mkdir -p recipes/seqtk
# Create your build.sh and meta.yml
# See also: `conda skeleton`, which can help generate templates for specific package types
# vi recipes/seqtk/build.sh
# vi recipes/seqtk/meta.yaml
# We need the conda-build package to ... build packages
conda install -y conda-build
# Once a new recipe is created, we can build it
conda build recipes/seqtk/
# Where is the package archive we just built ?
conda build recipes/seqtk/ --output
# Install the locally built package from the Conda build cache (the 'local' channel)
conda install -y --use-local seqtk
# or just using the package tar.bz2 directly, or an http:// link where it is hosted
# conda install /Users/perry/miniconda3/conda-bld/osx-64/seqtk-1.2-0.tar.bz2
# conda install http://bioinformatics.erc.monash.edu/home/andrewperry/conda/seqtk-1.2-0.tar.bz2
Distributing your build on Anaconda.org¶
(on a personal conda channel, independently of the Bioconda project / channel)
Step 1: Create an account at https://anaconda.org/
# Step 2: Install the `anaconda-client` package
conda install anaconda-client
# Step 3: Login and upload
anaconda login
# use the path to the package reported by `conda build recipes/seqtk/ --output`
anaconda upload <path_to_package_bz2>
# Step 4: Test that it worked - install from the channel you uploaded to
conda uninstall bigdatascript
conda install -c https://conda.anaconda.org/pansapiens bigdatascript
The Bioconda way¶
http://bioconda.github.io/contributing.html
Adding a recipe and testing locally before making a Pull Request.
First step - go to https://github.com/bioconda/bioconda-recipes and fork the repo.
Then clone your fork
export MY_GIT_USERNAME=pansapiens
git clone https://github.com/${MY_GIT_USERNAME}/bioconda-recipes.git
cd bioconda-recipes
Create a new branch for your recipe.
git branch mynewtool
git checkout mynewtool
Edit your recipe. Using a similar existing recipe as a template to modify it is often a good stratgy. (You can also use conda skeleton).
vi recipes/mynewtool/meta.yaml
vi recipes/mynewtool/build.sh
git commit -a -m "Added new recipe for mynewtool."
Simulate the TravisCI environment to do a test build
# This Bioconda script creates a fresh Miniconda install, similar to the Bioconda Travis CI
./simulate-travis.py --bootstrap /tmp/anaconda --overwrite
# Build your recipe and run tests.
# --packages is optional, only recipes where version and build have changed are built (?)
./simulate-travis.py --disable-docker --packages mynewtool --force
If tests passed, you can now squash your commits, push the branch
git push origin
- Go to https://github.com/bioconda/bioconda-recipes and make a Pull Request for your branch
- TravisCI will automatically build your branch.
- If TravisCI tests pass, the Bioconda team will review your PR and if it's okay, merge it
- Recipes in the Bioconda master branch are automatically built on Linux and macOS (via TravisCI) and uploaded to the Anaconda
biocondachannel.