Quick Start
From PileLine
PileLine Input Files
PileLine is capable to handle, filter and compare genomic position files (GP) including standard pileup, BED,GFF, or VCF files.
Basically, GP are tabular files where the two first columns contain sequence name (i.e. chromosome name) and position coordinate respectively. Additional optional fields are accepted in PileLine, see an example of GP input file below:
chr10 118829 optional1 optional2 optional3 ... chr10 121207 optional1 optional2 optional3 ... chr10 121337 optional1 optional2 optional3 ... chr10 121636 optional1 optional2 optional3 ...
PileLine Guided Example #1
The code included in this tutorial runs on UNIX systems. For Windows you must use PileLine commands adding .bat extension (i.e. pileline-2smc.bat)
1. Download and uncompress GP files (pileup format) and Annotation files to your working directory:
- GP files-Experiment 1.
- GP files-Experiment 2.
- Annotation files
- Hg18_hgnc_ensembl_genes.bed.zip
- DbSNP_36.3.bed.bgz.zip (to annotate using rfilter --annotate)
- DbSNP_36.3.gpfile.zip (to annotate using fastjoin)
Each Experiment .zip file contains 2 pileup files:
- Whole pileup file (.pileup)
- Variants against reference genome pileup file (.variants.pileup).
2. You may compare 2 samples at variant level using pileline-2smc.sh functionality.
Use this command line to compare file A (Control1) vs file B (Case1), remind that INPUT FILES MUST BE SORTED :
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-2smc.sh -a ./Control1.pileup -b ./Case1.pileup -v ./Control1.variants.pileup -w ./Case1.variants.pileup -o ./myoutput1.txt
Executing this command using by default parameters you will obtain 4 output files containing:
- myoutput1.txt.onlyA: Only variants found in File A. In this example variants found in Control1 but not in Case1 (i.e. germ-line reverted mutations or SNPs)
- myoutput1.txt.onlyB: Only variants found in File B. In this case, variants found in Case1 but not in Control1 (i.e. somatic mutations or SNPs)
- myoutput1.txt.both: Both File A and File B variants present the same allele and both of them are different to the reference genome allele. (i.e. germ-line variants or SNPs)
- myoutput1.txt.AdiscrepantB: Both File A and File B variants present different alleles and both of them are different to the reference genome allele. (i.e. germ-line variants mutated or SNPs).
See an example in this table:
| Ref genome | File A | File B | Output File Name |
|---|---|---|---|
| T | A | T | myoutput1.txt.onlyA |
| T | T | G | myoutput1.txt.onlyB |
| T | A | A | myoutput1.txt.both |
| T | C | G | myoutput1.txt.AdiscrepantB |
pileline-2smc.sh output file format specifications are shown in the next table:
| Chr | Position | Ref Genome Allele | Variant Allele | File ID | Variant Score | FileA_inputdata | FileB_inputdata |
|---|
See an explained example here:
10 116237 G A A 234 ... ... # Variant (G => A) found in File A (Control1 in this example) 10 116402 T C B 424 ... ... # Variant (T => C) found in File B (Case1 in this example) 10 118829 A G AB 125 ... ... # Variant (A => G) found in both A and B files (Control and Case1) 10 118845 A K D 127 ... ... # Discrepant variant (A => K) found in both A and B files (Control and Case1). IUPAC code is used here.
Note: If you apply pileline-2smc.sh to pileup files you'll find a variant score in the output file. This score reports the sum of Phred-scale Consensus Qualities (PCQ) for each position in both conditions (PCQ_Control + PCQ_Case).
Now, run pileline-2smc.sh to compare Case2 vs Control2:
$ sh YOUR_PATH_TO_PILELINE/cmd/pileline-2smc.sh -a ./Control2.pileup -b ./Case2.pileup -v ./Control2.variants.pileup -w ./Case2.variants.pileup -o ./myoutput2.txt
3. You can also compare multiple samples to report consistent variants.
You should use pileline-nsmc.sh command. In the following example we compare 4 samples (Control1 and Control2 variants vs Case1 and Case2 variants) but pileline-nsmc.sh can be applied to n samples (reminder: INPUT FILES MUST BE SORTED):
$ sh YOUR_PATH_TO_PILELINE/cmd/pileline-nsmc.sh -a ./Control1.variants.pileup -a ./Control2.variants.pileup -b ./Case1.variants.pileup -b ./Case2.variants.pileup -o ./mycommonvariants.txt
The output file (mycommonvariants.txt in this tutorial) contains the following information:
| Chr | Position | Ref Genome Allele | Variant Allele | Presence of variant in -a files | Presence of variant in -b files | # of samples containing the variant | Fisher's test p-value | FDR |
|---|
pileline-nsmc.sh performs a Fisher's test to estimate dependency amongst variants presence and samples type. The False Discovery Rate (FDR) is obtained by using Benjamini-Hochberg adjustment.
See next an example of the pileline-nsmc.sh output file:
10 115839 C Y NO NO NO YES 1 0.50000002 1.0 10 116237 G R NO YES NO YES 2 1.00000002 1.0 10 116402 T C YES YES NO YES 3 0.50000002 1.0 10 116699 C M NO YES NO YES 2 1.00000002 1.0 10 118829 A R YES YES YES YES 4 1.00000002 1.0 10 6101971 A M YES YES NO NO 2 0.16666669 1.0 ... 10 42940557 G R NO NO YES YES 2 0.166666667 1
In this case the variant located at position 6101971 has been found in both Control samples (-a files) but not in Case samples (-b files). On the contrary, the variant located in position 42940557 has been found in both Cases samples but not in Controls.
Note that pileline-nsmc.sh command may be particularly useful whether you want to find common variants in biological replicates. You should run pileline-nsmc.sh in this way:
$ sh YOUR_PATH_TO_PILELINE/cmd/pileline-nsmc.sh -a /Case1.variants.pileup -a ./Case2.variants.pileup -o ./mycommonvariants_in_Cases.txt
4. Variants annotation (e.g. SNPs, Ensembl IDs, HGNC Gene Names ...)
To annotate SNPs in variants found between Case1 and Control1, you have two choices:
- Use the dbSNP_36.3.bed.bgz (a BED, that is, an intervals GP file) and execute pileline-rfilter.sh --annotate command as follows:
$ sh YOUR_PATH_TO_PILELINE/cmd/pileline-rfilter --annotate -A ./myoutput1.txt.onlyB.pileup -i ./dbSNP_36.3.gpfile > ./mydbSNPannotation1.txt
- Use the dbSNP_36.3.gpfile (a single-nucleotide GP file) and execute pileline-fastjoin.sh command as follows:
$ sh YOUR_PATH_TO_PILELINE/cmd/pileline-fastjoin.sh -a ./myoutput1.txt.onlyB.pileup -b ./dbSNP_36.3.gpfile --left-outer-join > ./mydbSNPannotation1.txt
The output file (mydbSNPannotation1.txt) generated contains the same information as the input file but adding a new annotation column. SNPs are annotated with their respective dbSNP ID (i.e.rs7906865). Those variants which do not match to reported SNPs are annotated with NULL label.
In order to annotate the variants with Gene Symbols, you need the pileline-rfilter.sh command:
$ sh YOUR_PATH_TO_PILELINE/cmd/pileline-rfilter.sh --annotate -A ./mydbSNPannotation1.txt -b ./hg18_hgnc_ensembl_genes.bed > ./mydbSNPandGeneannotation1.txt
Note here that you can customize your annotations using your own BED file. Combining pileline-rfilter.sh and basic UNIX commands you can even annotate your pileup files with multiple .BED files using a single command line. See the following example:
cat mypileup | pileline-rfilter.sh --annotate -A - -i annotations_file1.bed | pileline-rfilter.sh --annotate -A - -i annotations_file2.bed > myfullyannotated.pileup
5. Generate input for external mutational effects prediction software.
Finally, you may be interested in generate SIFT, Polyphen and Firestar compatible input files to predict the functional consequences of variants found between Case1 and Control1. Use pileline-pileup2sift.sh and pileline-pileup2polyphen.sh commands to this end.
#To generate SIFT-compatible inputs $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-pileup2sift.sh -i ./mydbSNPandGeneannotation1.txt > ./mysiftinput.txt
#To generate PolyPhen-compatible inputs: $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-pileup2polyphen.sh -i ./mydbSNPandGeneannotation1.txt > ./mypolypheninput.txt
#To generate Firestar-compatible inputs: $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-pileup2firestar.sh -i ./mydbSNPandGeneannotation1.txt > ./myfirestarinput.txt
PipeLine Guided Example #2
The code included in this tutorial runs on UNIX systems. For Windows you must use PileLine commands adding .bat extension (i.e. pileline-fastseek.bat)
1. Download GP files (vcf, .bed and .gff format):
This .zip compressed file contains 3 GP files:
- TestVcf.vcf
- TestBed.bed
- TestGff.gff
- My.intervals.gff
- Note: All the guided examples showed here may be also applied to .pileup files.
2. Sort GP files by chromosomic positions:
.vcf
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-sort.sh -i ./TestVcf.vcf -o ./My.sorted.Vcf
.bed
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-sort.sh -i ./TestBed.bed -o ./My.sorted.bed
.gff
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-sort.sh -i ./TestGff.gff --pos-col 4 -o ./My.sorted.gff
3. Search for chromosomic positions within the GP files:
.vcf
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-fastseek.sh -p ./My.sorted.Vcf -s 1:762969:804635
.bed
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-fastseek.sh -p ./My.sorted.bed -s 9:138938078:138940781
.gff
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-fastseek.sh -p ./My.sorted.gff --pos-col 4 -s 7:157540895:157681138
4. Join two sorted GP files by common rows (if any):
.vcf and .bed
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-fastjoin.sh -a My.sorted.Vcf -b My.sorted.bed
.vcf and .gff
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-fastjoin.sh -a My.sorted.Vcf -b My.sorted.gff --pos-col-b 4
bed and .gff
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-fastjoin.sh -a My.sorted.bed -b My.sorted.gff --pos-col-b 4
5. Filters (or annotates) a GP file:
Annotate a .vcf file with Ensembl IDs contained in a .bed file
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-rfilter.sh --annotate -A ./My.sorted.Vcf -b ./hg18_hgnc_ensembl_genes.bed
Filter .bed using a .gff file containing intervals to be filtered
$ cd DOWNLOADED_FILES_DIRECTORY $ sh YOUR_PATH_TO_PILELINE/cmd/pileline-rfilter.sh -A ./My.sorted.bed -g ./My.intervals.gff --start-col-intervals 4 --end-col-intervals 5
