%\VignetteIndexEntry{03. Reads and Alignments}
%\VignettePackage{useR2013}



\section{Reads and Alignments}
\label{sec:readsalign}

The following sections introduce core tools for working with
high-throughput sequence data; key packages for representating reads
and alignments are summarized in Table
\ref{tab:sequence-and-alignment-pkgs}. This section focus on the reads
and alignments that are the raw material for analysis. 
Section \ref{sec:rnaseq} addresses statistical approaches to
assessing differential representation in RNA-seq experiments. Section
\ref{sec:chipseq} outlines ChIP-seq analysis. Section
\ref{sec:annotation} introduces resources for annotating sequences.

\begin{table}
  \centering
  \caption{Selected \Bioconductor{} packages for sequence reads and
    alignments.}
  \label{tab:sequence-and-alignment-pkgs}
  \begin{tabular}{lp{.7\textwidth}}
    Package & Description\\
    \hline

    \Biocpkg{ShortRead} & Defines the \Rclass{ShortReadQ} class and
    functions for manipulating \texttt{fastq} files; these classes
    rely heavily on \Biocpkg{Biostrings}.\\

    \Biocpkg{GenomicRanges} & \Rclass{GappedAlignments},
    \Rclass{GappedAlignmentPairs} (\Rclass{GAlignments}, 
    \Rclass{GAlignmentPairs} in the next version of \Bioconductor), 
    and \Rclass{GAlignmentsList} store single- and paired-end 
    aligned reads.\\

    \Biocpkg{Rsamtools} & Provides access to BAM alignment and other
    large sequence-related files.\\

    \Biocpkg{rtracklayer} & Input and output of \texttt{bed},
    \texttt{wig} and similar files \\

    \hline
  \end{tabular}
\end{table}

\subsection{The \emph{pasilla} Data Set}

As a running example, we use the \emph{pasilla} data set, derived from
\cite{Brooks2010}. The authors investigate conservation of RNA
regulation between \Dmel{} and mammals. Part of their study used RNAi
and RNA-seq to identify exons regulated by Pasilla (\emph{ps}), the
\Dmel{} ortholog of mammalian NOVA1 and NOVA2. Briefly, their
experiment compared gene expression as measured by RNAseq in S2-DRSC
cells cultured with, or without, a 444bp dsRNA fragment corresponding
to the \emph{ps} mRNA sequence. Their assessment investigated
differential exon use, but our worked example will focus on gene-level
differences.

In this section we look at a subset of the \emph{ps} data,
corresponding to reads obtained from lanes of their RNA-seq
experiment, and to the same reads aligned to a \Dmel{} reference
genome. Reads were obtained from GEO and the Short Read Archive (SRA),
and were aligned to the \Dmel{} reference genome \emph{dm3} as
described in the \Rpackage{pasilla} experiment data package.

\subsection{Reads and the \Biocpkg{ShortRead} Package}

\paragraph{Short read formats}

The Illumina GAII and HiSeq technologies generate sequences by
measuring incorporation of florescent nucleotides over successive PCR
cycles. These sequencers produce output in a variety of formats, but
\emph{FASTQ} is ubiquitous. Each read is represented by a record of
four components:
\begin{Schunk}
\begin{Soutput}
@SRR031724.1 HWI-EAS299_4_30M2BAAXX:5:1:1513:1024 length=37
GTTTTGTCCAAGTTCTGGTAGCTGAATCCTGGGGCGC
+SRR031724.1 HWI-EAS299_4_30M2BAAXX:5:1:1513:1024 length=37
IIIIIIIIIIIIIIIIIIIIIIIIIIII+HIIII<IE
\end{Soutput}
\end{Schunk}
The first and third lines (beginning with \verb|@| and \verb|+|
respectively) are unique identifiers. The second and fourth lines of the FASTQ record are the nucleotides
and qualities of each cycle in the read. This information is given in
5' to 3' orientation as seen by the sequencer. A letter \texttt{N} in
the sequence is used to signify bases that the sequencer was not able
to call.  The fourth line of the FASTQ record encodes the quality
(confidence) of the corresponding base call. The quality score is
encoded following one of several conventions, with the general notion
being that letters later in the visible ASCII alphabet
\begin{Schunk}
\begin{Soutput}
!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNO
PQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
\end{Soutput}
\end{Schunk}
%
are of lower quality. Both the sequence and quality scores may span
multiple lines.  Technologies other than Illumina use different
formats to represent sequences; 454 sequence input is supported in
\Biocpkg{R453Plus1Toolbox}; `color space' is not supported.

FASTQ files can be read in to \R{} using the \Rfunction{readFastq}
function from the \Biocpkg{ShortRead} package. Use this function by
providing the path to a FASTQ file. There are sample data files
available in the \Rpackage{useR2013} package, each consisting of 1
million reads from a lane of the Pasilla data set.

\begin{Schunk}
\begin{Sinput}
> fastqDir <- file.path(bigdata(), "fastq")
> fastqFiles <- dir(fastqDir, full=TRUE)
> fq <- readFastq(fastqFiles[1])
> fq
\end{Sinput}
\begin{Soutput}
class: ShortReadQ
length: 1000000 reads; width: 37 cycles
\end{Soutput}
\end{Schunk}
The data are represented as an object of class \Rclass{ShortReadQ}.
\begin{Schunk}
\begin{Sinput}
> head(sread(fq), 3)
\end{Sinput}
\begin{Soutput}
  A DNAStringSet instance of length 3
    width seq
[1]    37 GTTTTGTCCAAGTTCTGGTAGCTGAATCCTGGGGCGC
[2]    37 GTTGTCGCATTCCTTACTCTCATTCGGGAATTCTGTT
[3]    37 GAATTTTTTGAGAGCGAAATGATAGCCGATGCCCTGA
\end{Soutput}
\begin{Sinput}
> head(quality(fq), 3)
\end{Sinput}
\begin{Soutput}
class: FastqQuality
quality:
  A BStringSet instance of length 3
    width seq
[1]    37 IIIIIIIIIIIIIIIIIIIIIIIIIIII+HIIII<IE
[2]    37 IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
[3]    37 IIIIIIIIIIIIIIIIIIIIII'IIIIIGBIIII2I+
\end{Soutput}
\end{Schunk}
%
There are many ways to manipulate these objects; the
\Rfunction{alphabetByCycle} function summarizes use of nucleotides at
each cycle in a (equal width) \Rclass{ShortReadQ} or
\Rclass{DNAStringSet} instance.
\begin{Schunk}
\begin{Sinput}
> abc <- alphabetByCycle(sread(fq))
> abc[1:4, 1:8]
\end{Sinput}
\begin{Soutput}
        cycle
alphabet   [,1]   [,2]   [,3]   [,4]   [,5]   [,6]   [,7]   [,8]
       A  78194 153156 200468 230120 283083 322913 162766 220205
       C 439302 265338 362839 251434 203787 220855 253245 287010
       G 397671 270342 258739 356003 301640 247090 227811 246684
       T  84833 311164 177954 162443 211490 209142 356178 246101
\end{Soutput}
\end{Schunk}

FASTQ files are getting larger. A very common reason for looking at
data at this early stage in the processing pipeline is to explore
sequence quality. In these circumstances it is often not necessary to
parse the entire FASTQ file. Instead create a representative sample
\begin{Schunk}
\begin{Sinput}
> sampler <- FastqSampler(fastqFiles[1], 1000000)
> yield(sampler) # sample of 1000000 reads
\end{Sinput}
\begin{Soutput}
class: ShortReadQ
length: 1000000 reads; width: 37 cycles
\end{Soutput}
\end{Schunk}
A second common scenario is to pre-process reads, e.g., trimming
low-quality tails, adapter sequences, or artifacts of sample
preparation. The \Rclass{FastqStreamer} class can be used to `stream'
over the fastq files in chunks, processing each chunk independently.

\Biocpkg{ShortRead} contains facilities for quality assessment of
FASTQ files. The following generates a report from a sample of 1
million reads from each of our files and display it in a web browser
\begin{Schunk}
\begin{Sinput}
> ## Bioc 2.13 only; see ?qa for Bioc 2.12
> qas <- qa(fastqFiles, type="fastq")
> rpt <- report(qas, dest=tempfile())
> browseURL(rpt)
\end{Sinput}
\end{Schunk}
A report from a larger subset of the experiment is available
\begin{Schunk}
\begin{Sinput}
> rpt <- system.file("GSM461176_81_qa_report", "index.html",
+                    package="useR2013")
> browseURL(rpt)
\end{Sinput}
\end{Schunk}

\begin{Exercise}
  Use the helper function \Rfunction{bigdata} (defined in the
  \Rpackage{useR2013} package) and the \Rfunction{file.path} and
  \Rfunction{dir} functions to locate two fastq files from
  \cite{Brooks2010} (the files were obtained as described in the
  appendix and \Rpackage{pasilla} experiment data package.

  Input one of the fastq files using \Rfunction{readFastq} from the
  \Biocpkg{ShortRead} package.

  Using the helper function \Rfunction{gcFunction} from the
  \Rpackage{useR2013} package, draw a histogram of the distribution of
  GC frequencies across reads.

  Use \Rfunction{alphabetByCycle} to summarize the frequency of each
  nucleotide, at each cycle. Plot the results using
  \Rfunction{matplot}, from the \Rpackage{graphics} package.

  As an advanced exercise, and if on Mac or Linux, use the
  \Rpackage{parallel} package and \Rfunction{mclapply} to read and
  summarize the GC content of reads in two files in parallel.
\end{Exercise}

\begin{Solution}
Discovery:
\begin{Schunk}
\begin{Sinput}
> dir(bigdata())
\end{Sinput}
\begin{Soutput}
[1] "bam"   "fastq"
\end{Soutput}
\begin{Sinput}
> fls <- dir(file.path(bigdata(), "fastq"), full=TRUE)
\end{Sinput}
\end{Schunk}
Input:
\begin{Schunk}
\begin{Sinput}
> fq <- readFastq(fls[1])
\end{Sinput}
\end{Schunk}
GC content:
\begin{Schunk}
\begin{Sinput}
> alf0 <- alphabetFrequency(sread(fq), as.prob=TRUE, collapse=TRUE)
> sum(alf0[c("G", "C")])
\end{Sinput}
\begin{Soutput}
[1] 0.5457237
\end{Soutput}
\end{Schunk}
A histogram of the GC content of individual reads is obtained with
\begin{Schunk}
\begin{Sinput}
> gc <- gcFunction(sread(fq))
> hist(gc)
\end{Sinput}
\end{Schunk}
Alphabet by cycle:
\begin{Schunk}
\begin{Sinput}
> abc <- alphabetByCycle(sread(fq))
> matplot(t(abc[c("A", "C", "G", "T"),]), type="l")
\end{Sinput}
\end{Schunk}
Advanced (Mac, Linux only): processing on multiple cores.
\begin{Schunk}
\begin{Sinput}
> library(parallel)
> gc0 <- mclapply(fls, function(fl) {
+   fq <- readFastq(fl)
+   gc <- gcFunction(sread(fq))
+   table(cut(gc, seq(0, 1, .05)))
+ })
> ## simplify list of length 2 to 2-D array
> gc <- simplify2array(gc0)
> matplot(gc, type="s")
\end{Sinput}
\end{Schunk}
\end{Solution}

\subsection{Alignments and the \Biocpkg{Rsamtools} Package}

Most down-stream analysis of short read sequences is based on reads
aligned to reference genomes. There are many aligners available,
including \href{http://bio-bwa.sourceforge.net/}{BWA}
\cite{pmid20080505,pmid19451168},
\href{http://bowtie-bio.sourceforge.net/}{Bowtie} /
\href{http://bowtie-bio.sourceforge.net/bowtie2/}{Bowtie2}
\cite{pmid19261174}, and
\href{http://research-pub.gene.com/gmap/}{GSNAP}; merits of these are
discussed in the literature. There are also alignment algorithms
implemented in \Bioconductor{} (e.g., \Rfunction{matchPDict} in the
\Biocpkg{Biostrings} package, and the \Biocpkg{Rsubread} package);
\Rfunction{matchPDict} is particularly useful for flexible alignment
of moderately sized subsets of data.

\paragraph{Alignment formats}

Most main-stream aligners produce output in SAM (text-based) or BAM
format. A SAM file is a text file, with one line per aligned read, and
fields separated by tabs. Here is an example of a single SAM line,
split into fields.
\begin{Schunk}
\begin{Sinput}
> fl <- system.file("extdata", "ex1.sam", package="Rsamtools")
> strsplit(readLines(fl, 1), "\t")[[1]]
\end{Sinput}
\begin{Soutput}
 [1] "B7_591:4:96:693:509"                  "73"                                  
 [3] "seq1"                                 "1"                                   
 [5] "99"                                   "36M"                                 
 [7] "*"                                    "0"                                   
 [9] "0"                                    "CACTAGTGGCTCATTGTAAATGTGTGGTTTAACTCG"
[11] "<<<<<<<<<<<<<<<;<<<<<<<<<5<<<<<;:<;7" "MF:i:18"                             
[13] "Aq:i:73"                              "NM:i:0"                              
[15] "UQ:i:0"                               "H0:i:1"                              
[17] "H1:i:0"                              
\end{Soutput}
\end{Schunk}
%
The SAM \href{http://samtools.sourceforge.net/SAM1.pdf}{specification}
summarizes these fields.  We recognize from the FASTQ file the
identifier string, read sequence and quality. The alignment is to a
chromosome `seq1' starting at position 1. The strand of alignment is
encoded in the `flag' field.  The alignment record also includes a
measure of mapping quality, and a CIGAR string describing the nature
of the alignment. In this case, the CIGAR is 36M, indicating that the
alignment consisted of 36 \texttt{M}atches or mismatches, with no
indels or gaps; indels are represented by \texttt{I} and \texttt{D};
gaps (e.g., from alignments spanning introns) by \texttt{N}.  BAM
files encode the same information as SAM files, but in a format that
is more efficiently parsed by software; BAM files are the primary way
in which aligned reads are imported in to \R{}.

\paragraph{Aligned reads in \R{}}

The \Rfunction{readGappedAlignments} function from the
\Biocpkg{GenomicRanges} package reads essential information from a BAM
file in to \R. The result is an instance of the
\Rclass{GappedAlignments} class. The \Rclass{GappedAlignments} class
has been designed to allow useful manipulation of many reads (e.g., 20
million) under moderate memory requirements (e.g., 4 GB).
\begin{Schunk}
\begin{Sinput}
> alnFile <- system.file("extdata", "ex1.bam", package="Rsamtools")
> aln <- readGappedAlignments(alnFile)
> head(aln, 3)
\end{Sinput}
\begin{Soutput}
GappedAlignments with 3 alignments and 0 metadata columns:
      seqnames strand       cigar    qwidth     start       end     width      ngap
         <Rle>  <Rle> <character> <integer> <integer> <integer> <integer> <integer>
  [1]     seq1      +         36M        36         1        36        36         0
  [2]     seq1      +         35M        35         3        37        35         0
  [3]     seq1      +         35M        35         5        39        35         0
  ---
  seqlengths:
   seq1 seq2
   1575 1584
\end{Soutput}
\end{Schunk}
The \Rfunction{readGappedAlignments} function takes an additional
argument, \Rcode{param}, allowing the user to specify regions of
the BAM file (e.g., known gene coordinates) from which to extract
alignments.

A \Rclass{GappedAlignments} instance is like a data frame, but with
accessors as suggested by the column names. It is easy to query, e.g.,
the distribution of reads aligning to each strand, the width of reads,
or the cigar strings
\begin{Schunk}
\begin{Sinput}
> table(strand(aln))
\end{Sinput}
\begin{Soutput}
   +    -    * 
1647 1624    0 
\end{Soutput}
\begin{Sinput}
> table(width(aln))
\end{Sinput}
\begin{Soutput}
  30   31   32   33   34   35   36   38   40 
   2   21    1    8   37 2804  285    1  112 
\end{Soutput}
\begin{Sinput}
> head(sort(table(cigar(aln)), decreasing=TRUE))
\end{Sinput}
\begin{Soutput}
     35M      36M      40M      34M      33M 14M4I17M 
    2804      283      112       37        6        4 
\end{Soutput}
\end{Schunk}

\begin{Exercise}
  Use \Rfunction{bigdata}, \Rfunction{file.path} and \Rfunction{dir}
  to obtain file paths to the BAM files. These are a subset of the
  aligned reads, overlapping just four genes.

  Input the aligned reads from one file using
  \Rfunction{readGappedAlignments}. Explore the reads, e.g., using
  \Rfunction{table} or \Rfunction{xtabs}, to summarize which chromosome
  and strand the subset of reads is from.

  The object \Robject{ex} created earlier contains coordinates of four
  genes. Use \Rfunction{findOverlaps} to first determine the number of
  genes an individual read aligns to, and then the number of uniquely
  aligning reads overlapping each gene. Since the RNAseq protocol was
  not strand-sensitive, set the strand of \Robject{aln} to \Rcode{*}.

  Write the sequence of steps required to calculate counts as a simple
  function, and calculate counts on each file. On Mac or Linux, can
  you easily parallelize this operation?
\end{Exercise}

\begin{Solution}
  We discover the location of files using standard \R{} commands:
\begin{Schunk}
\begin{Sinput}
> fls <- dir(file.path(bigdata(), "bam"), ".bam$", full=TRUE)
> names(fls) <- sub("_.*", "", basename(fls))
\end{Sinput}
\end{Schunk}
Use \Rfunction{readGappedAlignments} to input data from one of the
files, and standard \R{} commands to explore the data.
\begin{Schunk}
\begin{Sinput}
> ## input
> aln <- readGappedAlignments(fls[1])
> xtabs(~seqnames + strand, as.data.frame(aln))
\end{Sinput}
\begin{Soutput}
        strand
seqnames    +    -
   chr3L 5402 5974
   chrX  2278 2283
\end{Soutput}
\end{Schunk}
To count overlaps in regions defined in a previous exercise, load
the regions.
\begin{Schunk}
\begin{Sinput}
> data(ex)             # from an earlier exercise
\end{Sinput}
\end{Schunk}
Many RNA-seq protocols are not strand aware, i.e., reads align to the
plus or minus strand regardless of the strand on which the
corresponding gene is encoded. Adjust the strand of the aligned reads
to indicate that the strand is not known.
\begin{Schunk}
\begin{Sinput}
> strand(aln) <- "*"   # protocol not strand-aware
\end{Sinput}
\end{Schunk}
For simplicity, we are interested in reads that align to only a single
gene. Find the overlaps between genes and reads\ldots
\begin{Schunk}
\begin{Sinput}
> hits <- findOverlaps(aln, ex)
\end{Sinput}
\end{Schunk}
%% 
The \Robject{hits} object has two columns, indicating each query-overlap pair.
We are interested in reads that hit exactly one gene, so tabulate the number of hits of each query element
\begin{Schunk}
\begin{Sinput}
> qhits <- countQueryHits(hits)
> table(qhits)
\end{Sinput}
\begin{Soutput}
qhits
    0     1     2 
  772 15026   139 
\end{Soutput}
\end{Schunk}
then select the uniquely-aligning reads
\begin{Schunk}
\begin{Sinput}
> keep <- which(qhits == 1)
\end{Sinput}
\end{Schunk}
%% 
and reverse the operation to count the number of times each region of
interest aligns to a uniquely overlapping alignment.
\begin{Schunk}
\begin{Sinput}
> cnt <- countSubjectHits(hits[queryHits(hits) %in% keep])
\end{Sinput}
\end{Schunk}
A simple function for counting reads is
\begin{Schunk}
\begin{Sinput}
> counter <-
+     function(filePath, range)
+ {
+     hits <- findOverlaps(aln, ex)
+     keep <- which(countQueryHits(hits) == 1)
+     cnts <- countSubjectHits(hits[queryHits(hits) %in% keep])
+     setNames(cnts, names(ex))
+ }
\end{Sinput}
\end{Schunk}
This can be applied to all files using \Rfunction{sapply}
\begin{Schunk}
\begin{Sinput}
> counts <- sapply(fls, counter, ex)
> counts
\end{Sinput}
\begin{Soutput}
            treated1 treated2 treated3 untreated1 untreated2 untreated3 untreated4
FBgn0002183     8359     8359     8359       8359       8359       8359       8359
FBgn0003360     1006     1006     1006       1006       1006       1006       1006
FBgn0025111     3162     3162     3162       3162       3162       3162       3162
FBgn0036449     2499     2499     2499       2499       2499       2499       2499
\end{Soutput}
\end{Schunk}
The counts in one BAM file are independent of counts in another BAM
file. This encourages us to count reads in each BAM file in parallel,
decreasing the length of time required to execute our program. On
Linux and Mac OS, a straight-forward way to parallelize this operation
is:
\begin{Schunk}
\begin{Sinput}
> if (require(parallel))
+     simplify2array(mclapply(fls, counter, ex))
\end{Sinput}
\end{Schunk}
%% 
The \Rfunction{summarizeOverlaps} function in the
\Biocpkg{GenomicRanges} package implements more appropraite counting
strategies.
\end{Solution}

\begin{Exercise}
  Consult the help page for \Rfunction{ScanBamParam}, and construct an
  object that restricts the information returned by a
  \Rfunction{scanBam} query to the aligned read DNA sequence. Your
  solution will use the \Rcode{what} parameter to the
  \Rfunction{ScanBamParam} function.

  Use the \Rfunction{ScanBamParam} object to query a BAM file, and
  calculate the GC content of all aligned reads. Summarize the
  GC content as a histogram (Figure \ref{fig:gc-read}).
\end{Exercise}
\begin{figure}
  \centering
  \includegraphics[width=.4\textwidth]{ReadsAndAlignments-gc-read}
  \caption{GC content in aligned reads}
  \label{fig:gc-read}
\end{figure}
\begin{Solution}
\begin{Schunk}
\begin{Sinput}
> param <- ScanBamParam(what="seq")
> seqs <- scanBam(fls[[1]], param=param)
> readGC <- gcFunction(seqs[[1]][["seq"]])
> hist(readGC)
\end{Sinput}
\end{Schunk}
\end{Solution}
