A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Go to file
2018-01-03 12:21:42 +08:00
_config.yml Set theme jekyll-theme-cayman 2017-06-09 12:37:53 +08:00
README.md Update README.md 2018-01-03 12:21:42 +08:00

Bash-Oneliner

Hi bash learners and bioinformaticans, welcome to Bash Oneliner. I started studying bioinformatics data four years ago (recently started working on cloud computing), and was amazed by those single-word bash commands which are much faster than my dull scripts, so much time can be saved through knowing command-line shortcuts and scripting. Not all the code here is oneliner (if the ';' counts..), but i put effort on making them brief and fast. I am mainly using Ubuntu and RedHat, sorry if the commands dont work on your system.

This blog will focus on simple bash commands for parsing data, most of which are for tsv files (tab-separated values); some of them are for Linux system maintenance. I apologize that there won't be any citation for the codes, but they are probably from dear Google and Stackoverflow.

English and bash are not my first language, so... correct me anytime, thank you!
And if you know any cool command that are not included here, Please Teach Me.

In case you would like to check and vote up my questions on Stackoverflow, here's my page: http://stackoverflow.com/users/4290753/once

Here's a more stylish version of Bash-Oneliner~ http://onceupon.github.io/Bash-Oneliner/

Handy Bash oneliner commands for tsv file editing

Grep

Extract text bewteen words (e.g. w1,w2)
grep -o -P '(?<=w1).*(?=w2)'
Grep only integer
grep -o '[0-9]*'
Grep lines without word (e.g. bbo)
grep -v bbo filename
Grep lines not begin with string (e.g. #)
grep -v '^#' file.txt
Grep variables with space within it (e.g. bbo="some strings")
grep "$boo" filename

#remember to quote the variable!

Grep only one/first match (e.g. bbo)
grep -m 1 bbo filename
Grep and count (e.g. bbo)
grep -c bbo filename
Insensitive grep (e.g. bbo/BBO/Bbo)
grep -i "bbo" filename 
Count occurrence (e.g. three times a line count three times)
grep -o bbo filename 
COLOR the match (e.g. bbo)!
grep --color bbo filename 
Grep search all files in a directory(e.g. bbo)
grep -R bbo /path/to/directory 

or

grep -r bbo /path/to/directory 
Search all files in directory, only output file names with matches(e.g. bbo)
grep -Rh bbo /path/to/directory 

or

grep -rh bbo /path/to/directory 

or only list filename with match

grep -rl bbo /path/to/directory
Grep OR (e.g. A or B or C or D)
grep 'A\|B\|C\|D' 
Grep AND (e.g. A and B)
grep 'A.*B' 
Grep all content of a fileA from fileB
grep -f fileA fileB 
Grep a tab
grep $'\t' 
Grep variable from variable
$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi

//grep -q will output 0 if match found
//remember to add space between []!

Grep strings between a bracket()
grep -oP '\(\K[^\)]+'
Grep number of characters with known strings in between(e.g. AAEL000001-RA)
grep -o -w "\w\{10\}\-R\w\{1\}"

// \w word character [0-9a-zA-Z_] \W not word character

A lot examples here

http://www.cyberciti.biz/faq/grep-regular-expressions/

Sed

[back to top]

Remove the 1st line
sed 1d filename
Remove the 100 lines (remove line 1-100)
sed 1,100d filename

remove lines with string (e.g. bbo)

sed "/bbo/d" filename
- case insensitive:
sed "/bbo/Id" filename
Edit infile (edit and save)
sed -i "/bbo/d" filename
When using variable (e.g. $i), use double quotes " "

e.g. add >$i to the first line (to make a FASTA file)

sed "1i >$i"  

//notice the double quotes! in other examples, you can use a single quote, but here, no way!
//'1i' means insert to first line

Delete/remove empty lines
sed '/^\s*$/d' 

or

sed 's/^$/d' 
Delete/remove last line
sed '$d' 
Delete/remove last character from end of file
sed -i '$ s/.$//' filename
Add string to end of file (e.g. "]")
sed '$s/$/]/' filename
Add newline to the end
sed '$a\'
Add string to beginning of every line (e.g. bbo)
sed -e 's/^/bbo/' file
Add string to end of each line (e.g. "}")
sed -e 's/$/\}\]/' filename
Add \n every nth character (e.g. every 4th character)
sed 's/.\{4\}/&\n/g' 
Concatenate/combine/join files with a seperator and next line (e.g seperate by ",")
sed -s '$a,' *.json > all.json
Substitution (e.g. replace A by B)
sed 's/A/B/g' filename 
Substitution with wildcard (e.g. replace a line start with aaa= by aaa=/my/new/path)
sed "s/aaa=.*/aaa=\/my\/new\/path/g"
Select lines start with string (e.g. bbo)
sed -n '/^@S/p' 
Delete lines with string (e.g. bbo)
sed '/bbo/d' filename 
Print/get/trim a range of line (e.g. line 500-5000)
sed -n 500,5000p filename
Print every nth lines
sed -n '0~3p' filename

//catch 0: start; 3: step

Print every odd # lines
sed -n '1~2p' 
Print every third line including the first line
sed -n '1p;0~3p' 
Remove leading whitespace and tabs
sed -e 's/^[ \t]*//'

//notice a whitespace before '\t'!!

Remove only leading whitespace
sed 's/ *//'

//notice a whitespace before '*'!!

Remove ending commas
sed 's/,$//g' 
Add a column to the end
sed "s/$/\t$i/"

//$i is the valuable you want to add
e.g. add the filename to every last column of the file

for i in $(ls);do sed -i "s/$/\t$i/" $i;done
Add extension of filename to last column
for i in T000086_1.02.n T000086_1.02.p;do sed "s/$/\t${i/*./}/" $i;done >T000086_1.02.np
Remove newline\ nextline
sed ':a;N;$!ba;s/\n//g'
Print a particular line (e.g. 123th line)
sed -n -e '123p'
Print a number of lines (e.g. line 10th to line 33 rd)
sed -n '10,33p' <filename
Change delimiter
sed 's=/=\\/=g'
Replace with wildcard (e.g A-1-e or A-2-e or A-3-e....)
sed 's/A-.*-e//g' filename
Remove last character of file
sed '$ s/.$//'
Insert character at specified position of file (e.g. AAAAAA --> AAA#AAA)
sed -r -e 's/^.{3}/&#/' file

Awk

[back to top]

Set tab as field separator
awk -F $'\t'  
Output as tab separated (also as field separator)
awk -v OFS='\t' 
Pass variable
a=bbo;b=obb;
awk -v a="$a" -v b="$b" "$1==a && $10=b" filename 
Print line number and number of characters on each line
awk '{print NR,length($0);}' filename 
Find number of columns
awk '{print NF}' 
Reverse column order
awk '{print $2, $1}' 
Check if there is a comma in a column (e.g. column $1)
awk '$1~/,/ {print}'  
Split and do for loop
awk '{split($2, a,",");for (i in a) print $1"\t"a[i]}' filename 
Print all lines before nth occurence of a string (e.g stop print lines when bbo appears 7 times)
awk -v N=7 '{print}/bbo/&& --N<=0 {exit}'
Print filename and last line of all files in directory
ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' file
Add string to the beginning of a column (e.g add "chr" to column $3)
awk 'BEGIN{OFS="\t"}$3="chr"$3' 
Remove lines with string (e.g. bbo)
awk '!/bbo/' file 
Usage and meaning of NR and FNR

e.g.
fileA:
a
b
c
fileB:
d
e

awk 'print FILENAME, NR,FNR,$0}' fileA fileB 

fileA 1 1 a
fileA 2 2 b
fileA 3 3 c
fileB 4 1 d
fileB 5 2 e

AND gate

e.g.
fileA:
1 0
2 1
3 1
4 0

fileB:
1 0
2 1
3 0
4 1

awk -v OFS='\t' 'NR=FNR{a[$1]=$2;next} NF {print $1,((a[$1]=$2)? $2:"0")}' fileA fileB 

1 0
2 1
3 0
4 0

Round all numbers of file (e.g. 2 significant figure)
awk '{while (match($0, /[0-9]+\[0-9]+/)){
    \printf "%s%.2f", substr($0,0,RSTART-1),substr($0,RSTART,RLENGTH)
    \$0=substr($0, RSTART+RLENGTH)
    \}
    \print
    \}'
Give number/index to every row
awk '{printf("%s\t%s\n",NR,$0)}'
Break combine column data into rows

e.g.
seperate

David cat,dog
into
David cat
David dog

detail here: http://stackoverflow.com/questions/33408762/bash-turning-single-comma-separated-column-into-multi-line-string

awk '{split($2,a,",");for(i in a)print $1"\t"a[i]}' file
Average a file (each line in file contains only one number)
awk '{s+=$1}END{print s/NR}'
Print field start with string (e.g Linux)
awk '$1 ~ /^Linux/'
Sort a row (e.g. 1 40 35 12 23 --> 1 12 23 35 40)
awk ' {split( $0, a, "\t" ); asort( a ); for( i = 1; i <= length(a); i++ ) printf( "%s\t", a[i] ); printf( "\n" ); }'
Subtract previous row values (add column6 which equal to column4 minus last column5)
awk '{$6 = $4 - prev5; prev5 = $5; print;}'

Xargs

[back to top]

Set tab as delimiter (default:space)
xargs -d\t
Display 3 items per line
echo 1 2 3 4 5 6| xargs -n 3

//1 2 3
4 5 6

Prompt before execution
echo a b c |xargs -p -n 3
Print command along with output
xargs -t abcd

///bin/echo abcd
//abcd

With find and rm
find . -name "*.html"|xargs rm -rf
Delete fiels with whitespace in filename (e.g. "hello 2001")
find . -name "*.c" -print0|xargs -0 rm -rf
Show limits
xargs --show-limits
Move files to folder
find . -name "*.bak" -print 0|xargs -0 -I {} mv {} ~/old

or

find . -name "*.bak" -print 0|xargs -0 -I file mv file ~/old
Move first 100th files to a directory (e.g. d1)
ls |head -100|xargs -I {} mv {} d1
Parallel
time echo {1..5} |xargs -n 1 -P 5 sleep

a lot faster than

time echo {1..5} |xargs -n1 sleep
Copy all files from A to B
find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file --target-directory=/path/to/B

//v: verbose|
//p: keep detail (e.g. owner)

With sed
ls |xargs -n1 -I file sed -i '/^Pos/d' filename
Add the file name to the first line of file
ls |sed 's/.txt//g'|xargs -n1 -I file sed -i -e '1 i\>file\' file.txt
Count all files
ls |xargs -n1 wc -l
Turn output into a single line
ls -l| xargs
Count files within directories
echo mso{1..8}|xargs -n1 bash -c 'echo -n "$1:"; ls -la "$1"| grep -w 74 |wc -l' --

// "--" signals the end of options and display further option processing

Download dependencies files and install (e.g. requirements.txt)
cat requirements.txt| xargs -n1 sudo pip install
Count lines in all file, also count total lines
ls|xargs wc -l
Xargs and grep
cat grep_list |xargs -I{} grep {} filename
Xargs and sed (replace all old ip address with new ip address under /etc directory)
grep -rl '192.168.1.111' /etc | xargs sed -i 's/192.168.1.111/192.168.2.111/g'

Find

[back to top]

List all sub directory/file in the current directory
find .
List all files under the current directory
find . -type f
List all directories under the current directory
find . -type d
Edit all files under current directory (e.g. replace 'www' with 'ww')
find . name '*.php' -exec sed -i 's/www/w/g' {} \;

if no subdirectory

replace "www" "w" -- *

//a space before *

Find and output only filename (e.g. "mso")
find mso*/ -name M* -printf "%f\n"
Find and delete file with size less than (e.g. 74 byte)
find . -name "*.mso" -size -74c -delete

//M for MB, etc

Loops

[back to top]

While loop, column subtraction of a file (e.g. a 3 columns file)
while read a b c; do echo $(($c-$b));done < <(head filename)

//there is a space between the two '<'s

While loop, sum up column subtraction
i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < <(head filename)
While loop, keep checking a running process (e.g. perl) and start another new process (e.g. python) immetiately after it. (BETTER use the wait command! Ctrl+F 'wait')
while [[ $(pidof perl) ]];do echo f;sleep 10;done && python timetorunpython.py
If loop
if (($j==$u+2))

//(( )) use for arithmetic operation

if [[$age >21]]

// use for comparison

Test if file exist
if [ -e 'filename' ]
then
  echo -e "file exists!"
fi
if else Test if file exist
 if [ -e $filename ]; then echo -e "file exists!"; else mkdir $filename; fi
For loop
for i in $(ls); do echo file $i;done
for loop, press any key to continue each loop
for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}');do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key;done
for loop, print a file line by line when a key is pressed
for line in $(cat myfile); do echo $line; read -n1; done

Math

[back to top]

Print out the prime factors of a number (e.g. 50)
factor 50
Sum up input list (e.g. seq 10)
seq 10|paste -sd+|bc
Sum up a file (each line in file contains only one number)
awk '{s+=$1} END {print s}' filename
Column subtraction
cat file| awk -F '\t' 'BEGIN {SUM=0}{SUM+=$3-$2}END{print SUM}'
Simple math with expr
expr 10+20 #30
expr 10\*20 #600
expr 30 \> 20 #1 (true)
More math with bc
  • Number of decimal digit/ significant figure
echo "scale=2;2/3" | bc  
#.66
  • Exponent operator
echo "10^2" | bc  
#100
  • Using variables
echo "var=5;--var"| bc  
#4

Download

[back to top]

Download all from a page
wget -r -l1 -H -t1 -nd -N -np -A mp3 -e robots=off http://example.com

//-r: recursive and download all links on page
//-l1: only one level link
//-H: span host, visit other hosts
//-t1: numbers of retries
//-nd: don't make new directories, download to here
//-N: turn on timestamp
//-nd: no parent
//-A: type (seperate by ,)
//-e robots=off: ignore the robots.txt file which stop wget from crashing the site, sorry example.com

Upload a file to web and download (https://transfer.sh/)

--> upload:

curl --upload-file ./filename.txt https://transfer.sh/filename.txt

(the above command will return a URL, e.g: https://transfer.sh/tG8rM/filename.txt)
--> download:

curl https://transfer.sh/tG8rM/filename.txt -o filename.txt
Download file if necessary
data=file.txt
url=http://www.example.com/$data
if [! -s $data];then
    echo "downloading test data..."
    wget $url
fi
Wget to a filename (when a long name)
wget -O filename "http://example.com"
Wget files to a folder
wget -P /path/to/directory "http://example.com"

Random

[back to top]

Random pick 100 lines from a file
shuf -n 100 filename
Random order (lucky draw)
for i in a b c d e; do echo $i; done| shuf
Echo series of random numbers between a range (e.g. shuffle numbers from 0-100, then pick 15 of them randomly)
shuf -i 0-100 -n 15
Echo a random number
echo $RANDOM
Random from 0-9
echo $((RANDOM % 10))
Random from 1-10
echo $(((RANDOM %10)+1))

Xwindow

[back to top]

X11 GUI applications! Here are some GUI tools for you if you get bored by the text-only environment.

Enable X11 forwarding,in order to use graphical application on servers
ssh -X user_name@ip_address

or setting through xhost

--> Install the following for Centos:
xorg-x11-xauth
xorg-x11-fonts-*
xorg-x11-utils

Little xwindow tools
xclock
xeyes
xcowsay
Open pictures/images from ssh server
1. ssh -X user_name@ip_address
2. apt-get install eog
3. eog picture.png
Use gedit on server (GUI editor)
1. ssh -X user_name@ip_address
2. apt-get install gedit
3. gedit filename.txt
Open PDF file from ssh server
1. ssh -X user_name@ip_address
2. apt-get install evince
3. evince filename.pdf
Use google-chrome browser from ssh server
1. ssh -X user_name@ip_address
2. apt-get install libxss1 libappindicator1 libindicator7
3. wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
4. sudo apt-get install -f
5. dpkg -i google-chrome*.deb
6. google-chrome

[back to top]

Others

[back to top]

Remove newline / nextline
tr --delete '\n' <input.txt >output.txt
Replace newline
tr '\n' ' ' <filename
To uppercase/lowercase
tr /a-z/ /A-Z/
Compare files (e.g. fileA, fileB)
diff fileA fileB

//a: added; d:delete; c:changed

or

sdiff fileA fileB

//side-to-side merge of file differences

Number a file (e.g. fileA)
nl fileA

or

nl -nrz fileA

//add leading zeros

or

nl -w1 -s ' '

//making it simple, blank seperated

Combine/ paste two files (e.g. fileA, fileB)
paste fileA fileB

//default tab seperated

Reverse string
echo 12345| rev
Read .gz file without extracting
zmore filename

or

zless filename
Run in background, output error file
some_commands  &>log &

or

some_commands 2>log &

or

some_commands 2>&1| tee logfile

or

some_commands 2>&1 >>outfile

//0: standard input; 1: standard output; 2: standard error

Send mail
echo 'heres the content'| mail -a /path/to/attach_file.txt -s 'mail.subject' me@gmail.com

//use -a flag to set send from (-a "From: some@mail.tld")

.xls to csv
xls2csv filename
Append to file (e.g. hihi)
echo 'hihi' >>filename
Make BEEP sound
speaker-test -t sine -f 1000 -l1
Set beep duration
(speaker-test -t sine -f 1000) & pid=$!;sleep 0.1s;kill -9 $pid
History edit/ delete
~/.bash_history

or

history -d [line_number]
Get last history/record filename
head !$
Clean screen
clear

or

Ctrl+l
Send data to last edited file
cat /directory/to/file
echo 100>!$
Run history number (e.g. 53)
!53
Run last command
!!
Run last command that began with (e.g. cat filename)
!cat

or

!c

//run cat filename again

Extract .xf
1.unxz filename.tar.xz  
2.tar -xf filename.tar
Install python package
pip install packagename
Delete current bash command
Ctrl+U

or

Ctrl+C

or

Alt+Shift+#

//to make it to history

Add something to history (e.g. "addmetohistory")
#addmetodistory

//just add a "#" before~~

Sleep awhile or wait for a moment or schedule a job
sleep 5;echo hi
Count the time for executing a command
time echo hi
Backup with rsync
rsync -av filename filename.bak
rsync -av directory directory.bak
rsync -av --ignore_existing directory/ directory.bak
rsync -av --update directory directory.bak
rsync -av directory user@ip_address:/path/to/directory.bak

//skip files that are newer on receiver (i prefer this one!)

Make all directories at one time!
mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat}

//-p: make parent directory
//this will create project/doc/html/; project/doc/info; project/lib/ext ,etc

Run command only if another command returns zero exit status (well done)
cd tmp/ && tar xvf ~/a.tar
Run command only if another command returns non-zero exit status (not finish)
cd tmp/a/b/c ||mkdir -p tmp/a/b/c
Extract to a path
tar xvf -C /path/to/directory filename.gz
Use backslash "" to break long command
cd tmp/a/b/c \
> || \
>mkdir -p tmp/a/b/c
Get pwd
VAR=$PWD; cd ~; tar xvf -C $VAR file.tar

//PWD need to be capital letter

List file type of file (e.g. /tmp/)
file /tmp/

//tmp/: directory

Bash script
#!/bin/bash
file=${1#*.}

//remove string before a "."

file=${1%.*}

//remove string after a "."

Search from history
Ctrl+r
Python simple HTTP Server
python -m SimpleHTTPServer
Variables
{i/a/,}

e.g. replace all

{i//a/,}

//for variable i, replace all 'a' with a comma e.g. with grep

 test="god the father"
 grep ${test// /\\\|} file.txt
 #turning the space into 'or' (\|) in grep
Read user input
read input
echo $input
Generate sequence 1-10
seq 10
Find average of input list/file
i=`wc -l filename|cut -d ' ' -f1`; cat filename| echo "scale=2;(`paste -sd+`)/"$i|bc
Generate all combination (e.g. 1,2)
echo {1,2}{1,2}

//1 1, 1 2, 2 1, 2 2

Generate all combination (e.g. A,T,C,G)
set = {A,T,C,G}
group= 5
for ((i=0; i<$group; i++));do
    repetition=$set$repetition;done
    bash -c "echo "$repetition""
Read file content to variable
foo=$(<test1)
Echo size of variable
echo ${#foo}
Echo tab
echo -e ' \t '
Array
declare -A array=()
Send a directory
scp -r directoryname user@ip:/path/to/send
Split file into lines (e.g. 1000 lines/smallfile)
$ split -d -l 1000 bigfilename
Rename all files (e.g. remove ABC from all .gz files)
rename 's/ABC//' *.gz
Remove extention (e.g remove .gz from filename.gz)
basename filename.gz .gz

zcat filename.gz> $(basename filename.gz .gz).unpacked
Use the squeeze repeat option (e.g. /t/t --> /t)
tr -s "/t" < filename
Do not print nextline with echo
echo -e 'text here \c'
Use the last argument
!$
Check last exit code
echo $?
View first 50 characters of file
head -c 50 file
Group/combine rows into one row

e.g.
AAAA
BBBB
CCCC
DDDD

cat filename|paste - -
-->
AAAABBBB
CCCCDDDD
cat filename|paste - - - -
-->
AAAABBBBCCCCDDDD
Fastq to fasta
cat file.fastq | paste - - - - | sed 's/^@/>/g'| cut -f1-2 | tr '\t' '\n' >file.fa
Cut and get last column
cat file|rev | cut -d/ -f1 | rev
Add one to variable/increment/ i++ a numeric variable (e.g. $var)
((var++))
or
var=$((var+1))

Some handy environment variables

$0 :name of shell or shell script.
$1, $2, $3, ... :positional parameters.
$# :number of positional parameters.
$? :most recent foreground pipeline exit status.
$- :current options set for the shell.

   :pid of the current shell (not subshell).  
$!   :is the PID of the most recent background command.  

##### Clear the contents of a file (e.g. filename)
```bash
>filename
```
##### Unzip tar.bz2 file (e.g. file.tar.bz2)
```bash
tar xvfj file.tar.bz2
```
##### Output a y/n repeatedly until killed 
'y':
```bash
yes
```
or 'n':
```bash
yes n
```
or 'anything':
```bash
yes anything
```

For example: 
```bash
yes | rm -r large_directory
``` 

##### Create dummy file of certain size instantly (e.g. 200mb)
```bash
dd if=/dev/zero of=//dev/shm/200m bs=1024k count=200
or
dd if=/dev/zero of=//dev/shm/200m bs=1M count=200
``` 

Standard output:  
200+0 records in  
200+0 records out  
209715200 bytes (210 MB) copied, 0.0955679 s, 2.2 GB/s  

##### Cat to a file
```bash
cat >myfile
let me add sth here
exit by control + c
^C
``` 

##### Keep /repeatedly executing the same command (e.g Repeat 'wc -l filename' every 1 second)
```bash
watch -n 1 wc -l filename
``` 

##### Print commands and their arguments when execute (e.g. echo `expr 10 + 20 `)
```bash
set -x; echo `expr 10 + 20 `
``` 

##### Print some meaningful sentences to you (install fortune first)
```bash
fortune
``` 
##### Colorful (and useful) version of top (install htop first)
```bash
htop
``` 
##### Press any key to continue
```bash
read -rsp $'Press any key to continue...\n' -n1 key
``` 

##### Run sql-like command on files from terminal
download:  
https://github.com/harelba/q  
example:
```bash
q -d "," "select c3,c4,c5 from /path/to/file.txt where c3='foo' and c5='boo'"
``` 

##### Sreen and tmux
create session and attach:
```bash
screen
or  
tmux
``` 

create detached session foo:  
```bash
screen -S foo -d -m	
or  
tmux new -s foo -d
``` 

detached session foo:  
```bash
screen: ^a^d
or  
tmux: ^bd
``` 


list sessions:  
```bash
screen -ls	
or  
tmux ls
``` 

attach:  
```bash
screen -r	
or  
tmux attach
``` 

attach to session foo:  
```bash
screen -r foo	
or 
tmux attach -t foo
``` 
kill session foo:
```bash
screen -r foo -X quit
or  
tmux kill-session -t foo
``` 
scroll:
Hit your screen prefix combination (C-a / control+A), then hit Escape.
Move up/down with the arrow keys (↑ and ↓).

Send command to all panes in tmux:
```bash
Ctrl-B :
setw synchronize-panes
``` 
##### Cut the last column
```bash
cat filename|rev|cut -f1|rev
``` 
##### pass password to ssh
```bash
sshpass -p mypassword ssh root@10.102.14.88 "df -h"
```
##### wait for a pid (job) to complete
```bash
wait %1 
or 
wait $PID
wait ${!}
#wait ${!} to wait till the last background process ($! is the PID of the last background process)
```
##### pdf to txt
```bash
sudo apt-get install poppler-utils
pdftotext example.pdf example.txt
```
##### list only directory
```bash
ls -ld -- */
```

##### Capture/record/save terminal output (capture everything you type and output)
```bash
script output.txt
#start using terminal 
#to logout the screen session (stop saving the contents), type exit.
```


## System
[[back to top](#handy-bash-oneliner-commands-for-tsv-file-editing)]

##### Snapshot of the current processes

```bash
ps 
```

##### Check graphics card

```bash
lspci
```

##### Show IP address
    
```bash
$ip add show
```
or
    
```bash
ifconfig
```

##### Check system version
    
```bash
cat /etc/*-release
```

##### Linux Programmer's Manuel: hier- description of the filesystem hierarchy
    
```bash
man hier
```

##### List job
    
```bash
jobs -l
```

##### Export PATH
    
```bash
export PATH=$PATH:~/path/you/want
```

##### Make file execuable
    
```bash
chmod +x filename
```
//you can now ./filename to execute it

##### List screen
    
```bash
screen -d -r
```

##### Echo screen name
    
```bash
screen -ls
```

##### Check system (x86-64)
    
```bash
uname -i
```

##### Surf the net

```bash
links www.google.com
```

##### Add user, set passwd
    
```bash
useradd username
passwd username
```

##### Edit variable for bash, (e.g. displaying the whole path)
    
```bash
1. joe ~/.bash_profile 
2. export PS1='\u@\h:\w\$' 
```
//$PS1 is a variable that defines the makeup and style of the command prompt 
```bash
3. source ~/.bash_profile
```

##### Edit environment setting (e.g. alias)
    
```bash
1. joe ~/.bash_profile
2. alias pd="pwd" //no more need to type that 'w'!
3. source ~/.bash_profile
```

##### List environment variables (e.g. PATH)
    
```bash
$echo $PATH
```
//list of directories separated by a colon

##### List all environment variables for current user
    
```bash
$env
```

##### Show partition format
    
```bash
lsblk
```

##### Soft link program to bin
    
```bash
ln -s /path/to/program /home/usr/bin
```
//must be the whole path to the program

##### Show hexadecimal view of data
    
```bash
hexdump -C filename.class
```

##### Jump to different node
    
```bash
rsh node_name
```

##### Check port (active internet connection)
    
```bash
netstat -tulpn
```

##### Find whick link to a file
    
```bash
readlink filename
```

##### Check where a command link to (e.g. python)
    
```bash
which python
```

##### List total size of a directory
    
```bash
du -hs .
```
or
    
```bash
du -sb
```

##### Copy directory with permission setting
    
```bash
cp -rp /path/to/directory
```

##### Store current directory
    
```bash
pushd . $popd ;dirs -l 
```

##### Show disk usage
    
```bash
df -h 
```
or
   
```bash
du -h 
```
or
    
```bash
du -sk /var/log/* |sort -rn |head -10
```

##### Show current runlevel
    
```bash
runlevel
```

##### Switch runlevel
    
```bash
init 3 
```
or
```bash
telinit 3 
```
##### Permanently modify runlevel
    
```bash
1. edit /etc/init/rc-sysinit.conf 
2. env DEFAULT_RUNLEVEL=2 
```

##### Become root
    
```bash
su
```

##### Become somebody
    
```bash
su somebody
```

##### Report user quotes on device
    
```bash
requota -auvs
```

##### Get entries in a number of important databases
    
```bash
getent database_name
```
(e.g. the 'passwd' database)
    
```bash
getent passwd
```
//list all user account (all local and LDAP)  
(e.g. fetch list of grop accounts)
    
```bash
getent group
```
//store in database 'group'

##### Change owner of file
    
```bash
chown user_name filename
chown -R user_name /path/to/directory/
```
//chown user:group filename

##### List current mount detail
    
```bash
df
```

##### List current usernames and user-numbers
    
```bash
cat /etc/passwd
```
##### Get all username
    
```bash
getent passwd| awk '{FS="[:]"; print $1}'
```

##### Show all users
    
```bash
compgen -u
```

##### Show all groups
    
```bash
compgen -g
```

##### Show group of user
    
```bash
group username
```

##### Show uid, gid, group of user
    
```bash
id username
```

##### Check if it's root
    
```bash
if [$(id -u) -ne 0];then
    echo "You are not root!"
    exit;
fi
```
//'id -u' output 0 if it's not root

##### Find out CPU information
    
```bash
more /proc/cpuinfo
```
or
    
```bash
lscpu
```

##### Set quota for user (e.g. disk soft limit: 120586240; hard limit: 125829120)
    
```bash
setquota username 120586240 125829120 0 0 /home
```

##### Show quota for user
    
```bash
quota -v username
```

##### Fork bomb
    
```bash
:(){:|:&};:
```
//dont try this at home

##### Check user login
    
```bash
lastlog
```

##### Edit path for all users
    
```bash
joe /etc/environment
```
//edit this file

##### Show running processes
    
```bash
ps aux
```

##### Find maximum number of processes
    
```bash
cat /proc/sys/kernal/pid_max
```

##### Show and set user limit
    
```bash
ulimit -u
```
##### Which ports are listening for TCP connections from the network

```bash
nmap -sT -O localhost
```

##### Print out number of cores/ processors

```bash
nproc --all
```
##### Check status of each core
1. top  
2. press '1'

##### Show jobs and PID

```bash
jobs -l
```

##### List all running services

```bash
service --status-all
```

##### Schedule shutdown server
```bash
shutdown -r +5 "Server will restart in 5 minutes. Please save your work."
```
##### Cancel scheduled shutdown
```bash
shutdown -c
```

##### Boardcast to all users
```bash
wall -n hihi
```
##### Kill all process of a user
```bash
pkill -U user_name
```
##### Set gedit preference on server

-->you might have to install the following:

apt-get install libglib2.0-bin;  

yum install dconf dconf-editor;  
yum install dbus dbus-x11;  

-->Check list  
```bash
gsettings list-recursively
```
-->Change setting  
e.g.
```bash
gsettings set org.gnome.gedit.preferences.editor highlight-current-line true
gsettings set org.gnome.gedit.preferences.editor scheme 'cobalt'
gsettings set org.gnome.gedit.preferences.editor use-default-font false
gsettings set org.gnome.gedit.preferences.editor editor-font 'Cantarell Regular 12'
```
##### Find out who has logged in on your system
--> [Quick] Printing out only the names:
```bash
users
```

--> [Detail] Printing out login time, load average, etc
```bash
w
```
##### Add user to a group (e.g add user 'nice' to the group 'docker', so that he can run docker without sudo) 
```bash
sudo gpasswd -a nice docker
```

##### pip install python package without root 
```bash
1. pip install --user package_name
2. You might need to export ~/.local/bin/ to PATH: export PATH=$PATH:~/.local/bin/
```

##### Removing old linux kernels (when /boot almost full...)
```bash
1. uname -a  #check current kernel, which should NOT be removed
2. sudo apt-get purge linux-image-X.X.X-X-generic  #replace old version
```


##### Change hostname
```bash
sudo hostname your-new-name
```
if not working, do also:  
```bash
hostnamectl set-hostname your-new-hostname
```
then run:  
hostnamectl

check /etc/hostname  

if still not working..., edit:  
/etc/sysconfig/network  
/etc/sysconfig/network-scripts/ifcfg-ensxxx  
add HOSTNAME="your-new-hostname"  
  

##### List installed packages
```bash
apt list --installed
```
or Red Hat: 
```bash
yum list installed
```

##### Check which file make the device busy on umount
```bash
lsof /mnt/dir
```


##### When sound not working
```bash
killall pulseaudio
```
then press Alt-F2 and type in pulseaudio  

##### When sound not working
```bash
killall pulseaudio
```

##### Finding Out Hardware Details Without Opening The Computer Case (e.g. memory device detail)
```bash
sudo dmidecode -t memory
```
 ##### lists information about SCSI devices in Linux
```bash
lsscsi
```

##### Tutorial for setting up your own DNS server
http://onceuponmine.blogspot.tw/2017/08/set-up-your-own-dns-server.html

##### Tutorial for creating a simple daemon
http://onceuponmine.blogspot.tw/2017/07/create-your-first-simple-daemon.html

##### Tutorial for using your gmail to send email
http://onceuponmine.blogspot.tw/2017/10/setting-up-msmtprc-and-use-your-gmail.html

 ##### Using telnet to test open ports, test if you can connect to a port (e.g 53) of a server (e.g 192.168.2.106)
```bash
telnet 192.168.2.106 53
```
##### change network maximum transmission unit (mtu) (e.g. change to 9000) 
```bash
ifconfig eth0 mtu 9000
```
##### get pid of a running process (e.g python) 
```bash
pidof python
```
or  
```bash
ps aux|grep python
```
##### ntp
start ntp:  
```bash
ntpd
```
check ntp:  
```bash
ntpq -p
```
##### remove unnecessary files to clean your server
```bash
sudo apt-get autoremove
sudo apt-get clean
sudo rm -rf ~/.cache/thumbnails/*
```
Remove old kernal:
```bash
sudo dpkg --list 'linux-image*'
sudo apt-get remove linux-image-OLDER_VERSION
```

##### Increase/ resize root partition (root partition is an LVM logical volume)
```bash
pvscan
lvextend -L +130G /dev/rhel/root -r
```
#Adding -r will grow filesystem after resizing the volume.  


##### Create a UEFI Bootable USB drive (e.g. /dev/sdc1)
```bash
sudo dd if=~/path/to/isofile.iso of=/dev/sdc1 oflag=direct bs=1048576
```
##### Locate and remove a package
```bash
sudo dpkg -l | grep <package_name>
sudo dpkg --purge <package_name>
```

##### Create a ssh tunnel
```bash
ssh -f -L 9000:targetservername:8088 root@192.168.14.72 -N
```
#-f: run in background; -L: Listen; -N: do nothing  
#the 9000 of your computer is now connected to the 8088 port of the targetservername through 192.168.14.72  
#so that you can see the content of targetservername:8088 by entering localhost:9000 from your browser.  

##### Count the number of Segate hard disks
```bash
lsscsi|grep SEAGATE|wc -l
or
sg_map -i -x|grep SEAGATE|wc -l
```

##### Print detail of memory hardware
```bash
dmidecode -t memory
```
##### Log out your account after a certain period of time (e.g 10 seconds)
```bash
TMOUT=10
#once you set this variable, logout timer start running!
```




=-=-=-=-=-A lot more coming!! =-=-=-=-=-=-=-=-=-=waitwait-=-=-=-=-=-=-=-=-=-