1 Commits

408
README.md
View File

@ -1,7 +1,7 @@
# Bash-Oneliner # Bash-Oneliner
I am glad that you are here! I was working on bioinformatics a few years ago and was amazed by those single-word bash commands which are much faster than my dull scripts, time saved through learning command-line shortcuts and scripting. Recent years I am working on cloud computing and I keep recording those useful commands here. Not all of them is oneliner, but i put effort on making them brief and swift. I am mainly using Ubuntu, Amazon Linux, RedHat, Linux Mint, Mac and CentOS, sorry if the commands don't work on your system. I am glad that you are here! I was working on bioinformatics a few years ago and was amazed by those single-word bash commands which are much faster than my dull scripts, time saved through learning command-line shortcuts and scripting. Recent years I am working on cloud computing and I keep recording those useful commands here. Not all of them is oneliner, but i put effort on making them brief and swift. I am mainly using Ubuntu, Amazon Linux, RedHat, Linux Mint, Mac and CentOS, sorry if the commands don't work on your system.
This blog will focus on simple bash commands for parsing data and Linux system maintenance that i acquired from work and LPIC exam. I apologize that there are no detailed citation for all the commands, but they are probably from dear Google and Stack Overflow. This blog will focus on simple bash commands for parsing data and Linux system maintenance that i acquired from work and LPIC exam. I apologize that there are no detailed citation for all the commands, but they are probably from dear Google and Stackoverflow.
English and bash are not my first language, please correct me anytime, thank you. English and bash are not my first language, please correct me anytime, thank you.
If you know other cool commands, please teach me! If you know other cool commands, please teach me!
@ -12,13 +12,13 @@ Here's a more stylish version of [Bash-Oneliner](https://onceupon.github.io/Bash
- [Terminal Tricks](#terminal-tricks) - [Terminal Tricks](#terminal-tricks)
- [Variable](#variable) - [Variable](#variable)
- [Math](#math)
- [Grep](#grep) - [Grep](#grep)
- [Sed](#sed) - [Sed](#sed)
- [Awk](#awk) - [Awk](#awk)
- [Xargs](#xargs) - [Xargs](#xargs)
- [Find](#find) - [Find](#find)
- [Condition and Loop](#condition-and-loop) - [Condition and Loop](#condition-and-loop)
- [Math](#math)
- [Time](#time) - [Time](#time)
- [Download](#download) - [Download](#download)
- [Random](#random) - [Random](#random)
@ -33,23 +33,22 @@ Here's a more stylish version of [Bash-Oneliner](https://onceupon.github.io/Bash
##### Using Ctrl keys ##### Using Ctrl keys
``` ```
Ctrl + a : move to the beginning of line.
Ctrl + d : if you've type something, Ctrl + d deletes the character under the cursor, else, it escapes the current shell.
Ctrl + e : move to the end of line.
Ctrl + k : delete all text from the cursor to the end of line.
Ctrl + l : equivalent to clear.
Ctrl + n : same as Down arrow. Ctrl + n : same as Down arrow.
Ctrl + p : same as Up arrow. Ctrl + p : same as Up arrow.
Ctrl + q : to resume output to terminal after Ctrl + s.
Ctrl + r : begins a backward search through command history.(keep pressing Ctrl + r to move backward) Ctrl + r : begins a backward search through command history.(keep pressing Ctrl + r to move backward)
Ctrl + s : to stop output to terminal. Ctrl + s : to stop output to terminal.
Ctrl + t : transpose the character before the cursor with the one under the cursor, press Esc + t to transposes the two words before the cursor. Ctrl + q : to resume output to terminal after Ctrl + s.
Ctrl + u : cut the line before the cursor; then Ctrl + y paste it Ctrl + a : move to the beginning of line.
Ctrl + w : cut the word before the cursor; then Ctrl + y paste it Ctrl + e : move to the end of line.
Ctrl + d : if you've type something, Ctrl + d deletes the character under the cursor, else, it escapes the current shell.
Ctrl + k : delete all text from the cursor to the end of line.
Ctrl + x + backspace : delete all text from the beginning of line to the cursor. Ctrl + x + backspace : delete all text from the beginning of line to the cursor.
Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands. Ctrl + t : transpose the character before the cursor with the one under the cursor, press Esc + t to transposes the two words before the cursor.
Ctrl + z : stop current running process and keep it in background. You can use `fg` to continue the process in the foreground, or `bg` to continue the process in the background. Ctrl + w : cut the word before the cursor; then Ctrl + y paste it
Ctrl + u : cut the line before the cursor; then Ctrl + y paste it
Ctrl + _ : undo typing. Ctrl + _ : undo typing.
Ctrl + l : equivalent to clear.
Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands.
``` ```
##### Change case ##### Change case
```bash ```bash
@ -58,9 +57,8 @@ Esc + u
Esc + l Esc + l
# converts text from cursor to the end of the word to lowercase. # converts text from cursor to the end of the word to lowercase.
Esc + c Esc + c
# converts letter under the cursor to uppercase, rest of the word to lowercase. # converts letter under the cursor to uppercase.
``` ```
##### Run history number (e.g. 53) ##### Run history number (e.g. 53)
```bash ```bash
!53 !53
@ -71,6 +69,7 @@ Esc + c
!! !!
# run the previous command using sudo # run the previous command using sudo
sudo !! sudo !!
# of course you need to enter your password
``` ```
##### Run last command and change some parameter using caret substitution (e.g. last command: echo 'aaa' -> rerun as: echo 'bbb') ##### Run last command and change some parameter using caret substitution (e.g. last command: echo 'aaa' -> rerun as: echo 'bbb')
@ -99,16 +98,16 @@ sudo !!
##### Bash globbing ##### Bash globbing
```bash ```bash
# '*' serves as a "wild card" for filename expansion. # '*' serves as a "wild card" for filename expansion.
/etc/pa*wd #/etc/passwd
# '?' serves as a single-character "wild card" for filename expansion.
/b?n/?at #/bin/cat /b?n/?at #/bin/cat
# '[]' serves to match the character from a range. # '?' serves as a single-character "wild card" for filename expansion.
/etc/pa*wd #/etc/passwd
# [] serves to match the character from a range.
ls -l [a-z]* #list all files with alphabet in its filename. ls -l [a-z]* #list all files with alphabet in its filename.
# '{}' can be used to match filenames with more than one patterns # {} can be used to match filenames with more than one patterns
ls *.{sh,py} #list all .sh and .py files ls {*.sh,*.py} #list all .sh and .py files
``` ```
##### Some handy environment variables ##### Some handy environment variables
@ -120,7 +119,6 @@ $? :most recent foreground pipeline exit status.
$- :current options set for the shell. $- :current options set for the shell.
$$ :pid of the current shell (not subshell). $$ :pid of the current shell (not subshell).
$! :is the PID of the most recent background command. $! :is the PID of the most recent background command.
$_ :last argument of the previously executed command, or the path of the bash script.
$DESKTOP_SESSION current display manager $DESKTOP_SESSION current display manager
$EDITOR preferred text editor. $EDITOR preferred text editor.
@ -132,49 +130,14 @@ $USER current username
$HOSTNAME current hostname $HOSTNAME current hostname
``` ```
##### Using vi-mode in your shell
```bash
set -o vi
# change bash shell to vi mode
# then hit the Esc key to change to vi edit mode (when `set -o vi` is set)
k
# in vi edit mode - previous command
j
# in vi edit mode - next command
0
# in vi edit mode - beginning of the command
R
# in vi edit mode - replace current characters of command
2w
# in vi edit mode - next to 2nd word
b
# in vi edit mode - previous word
i
# in vi edit mode - go to insert mode
v
# in vi edit mode - edit current command in vi
man 3 readline
# man page for complete readline mapping
```
## Variable ## Variable
[[back to top](#handy-bash-one-liners)] [[back to top](#handy-bash-one-liners)]
##### Variable substitution within quotes ##### Variable substitution within quotes
```bash ```bash
# foo=bar # foo=bar
echo $foo echo "'$foo'"
# bar #'bar'
echo "$foo" # double/single quotes around single quotes make the inner single quotes expand variables
# bar
# single quotes cause variables to not be expanded
echo '$foo'
# $foo
# single quotes within double quotes will not cancel expansion and will be part of the output
echo "'$foo'"
# 'bar'
# doubled single quotes act as if there are no quotes at all
echo ''$foo''
# bar
``` ```
##### Get the length of variable ##### Get the length of variable
```bash ```bash
@ -215,13 +178,11 @@ echo ${var[@]#0}
```bash ```bash
{var//a/,} {var//a/,}
``` ```
##### Grep lines with strings from a file (e.g. lines with 'stringA or 'stringB' or 'stringC')
```bash ```bash
#with grep #with grep
test="stringA stringB stringC" test="god the father"
grep ${test// /\\\|} file.txt grep ${test// /\\\|} file.txt
# turning the space into 'or' (\|) in grep # turning the space into 'or' (\|) in grep
``` ```
##### To change the case of the string stored in the variable to lowercase (Parameter Expansion) ##### To change the case of the string stored in the variable to lowercase (Parameter Expansion)
@ -244,9 +205,9 @@ echo "$bar" # foo
```bash ```bash
echo $(( 10 + 5 )) #15 echo $(( 10 + 5 )) #15
x=1 x=1
echo $(( x++ )) #1 , notice that it is still 1, since it's post-increment echo $(( x++ )) #1 , notice that it is still 1, since it's post-incremen
echo $(( x++ )) #2 echo $(( x++ )) #2
echo $(( ++x )) #4 , notice that it is not 3 since it's pre-increment echo $(( ++x )) #4 , notice that it is not 3 since it's pre-incremen
echo $(( x-- )) #4 echo $(( x-- )) #4
echo $(( x-- )) #3 echo $(( x-- )) #3
echo $(( --x )) #1 echo $(( --x )) #1
@ -304,10 +265,10 @@ echo "var=5;--var"| bc
##### Type of grep ##### Type of grep
```bash ```bash
grep = grep -G # Basic Regular Expression (BRE) grep = grep -G # Basic Regular Expression (BRE)
fgrep = grep -F # fixed text, ignoring meta-characters fgrep = grep -F # fixed text, ignoring meta-charachetrs
egrep = grep -E # Extended Regular Expression (ERE) egrep = grep -E # Extended Regular Expression (ERE)
pgrep = grep -P # Perl Compatible Regular Expressions (PCRE)
rgrep = grep -r # recursive rgrep = grep -r # recursive
grep -P # Perl Compatible Regular Expressions (PCRE)
``` ```
##### Grep and count number of empty lines ##### Grep and count number of empty lines
@ -319,15 +280,15 @@ grep -c "^$"
```bash ```bash
grep -o '[0-9]*' grep -o '[0-9]*'
#or #or
grep -oP '\d*' grep -oP '\d'
``` ```
##### Grep integer with certain number of digits (e.g. 3) ##### Grep integer with certain number of digits (e.g. 3)
```bash ```bash
grep '[0-9]\{3\}' grep [0-9]\{3\}
# or # or
grep -E '[0-9]{3}' grep -E [0-9]{3}
# or # or
grep -P '\d{3}' grep -P \d{3}
``` ```
##### Grep only IP address ##### Grep only IP address
@ -434,14 +395,14 @@ grep 'A\|B\|C\|D'
grep 'A.*B' grep 'A.*B'
``` ```
##### Regex any single character (e.g. ACB or AEB) ##### Regex any singer character (e.g. ACB or AEB)
```bash ```bash
grep 'A.B' grep 'A.B'
``` ```
##### Regex with or without a certain character (e.g. color or colour) ##### Regex with or without a certain character (e.g. color or colour)
```bash ```bash
grep 'colou\?r' grep colou?r
``` ```
##### Grep all content of a fileA from fileB ##### Grep all content of a fileA from fileB
@ -495,10 +456,9 @@ sed 1,100d filename
##### Remove lines with string (e.g. 'bbo') ##### Remove lines with string (e.g. 'bbo')
```bash ```bash
sed "/bbo/d" filename sed "/bbo/d" filename
# case insensitive: - case insensitive:
sed "/bbo/Id" filename sed "/bbo/Id" filename
``` ```
##### Remove lines whose nth character not equal to a value (e.g. 5th character not equal to 2) ##### Remove lines whose nth character not equal to a value (e.g. 5th character not equal to 2)
```bash ```bash
sed -E '/^.{5}[^2]/d' sed -E '/^.{5}[^2]/d'
@ -550,7 +510,7 @@ sed -i '1s/^/[/' file
##### Add string at certain line number (e.g. add 'something' to line 1 and line 3) ##### Add string at certain line number (e.g. add 'something' to line 1 and line 3)
```bash ```bash
sed -e '1isomething' -e '3isomething' sed -e '1isomething -e '3isomething'
``` ```
##### Add string to end of file (e.g. "]") ##### Add string to end of file (e.g. "]")
@ -564,7 +524,7 @@ sed '$a\'
##### Add string to beginning of every line (e.g. 'bbo') ##### Add string to beginning of every line (e.g. 'bbo')
```bash ```bash
sed -e 's/^/bbo/' filename sed -e 's/^/bbo/' file
``` ```
##### Add string to end of each line (e.g. "}") ##### Add string to end of each line (e.g. "}")
@ -577,14 +537,7 @@ sed -e 's/$/\}\]/' filename
sed 's/.\{4\}/&\n/g' sed 's/.\{4\}/&\n/g'
``` ```
##### Add a line after the line that matches the pattern (e.g. add a new line with "world" after the line with "hello") ##### Concatenate/combine/join files with a seperator and next line (e.g separate by ",")
```bash
sed '/hello*/a world' filename
# hello
# world
```
##### Concatenate/combine/join files with a separator and next line (e.g separate by ",")
```bash ```bash
sed -s '$a,' *.json > all.json sed -s '$a,' *.json > all.json
``` ```
@ -652,12 +605,12 @@ sed "s/$/\t$i/"
# $i is the valuable you want to add # $i is the valuable you want to add
# To add the filename to every last column of the file # To add the filename to every last column of the file
for i in $(ls); do sed -i "s/$/\t$i/" $i; done for i in $(ls);do sed -i "s/$/\t$i/" $i;done
``` ```
##### Add extension of filename to last column ##### Add extension of filename to last column
```bash ```bash
for i in T000086_1.02.n T000086_1.02.p; do sed "s/$/\t${i/*./}/" $i; done >T000086_1.02.np for i in T000086_1.02.n T000086_1.02.p;do sed "s/$/\t${i/*./}/" $i;done >T000086_1.02.np
``` ```
##### Remove newline\ nextline ##### Remove newline\ nextline
@ -821,7 +774,7 @@ awk '{printf("%s\t%s\n",NR,$0)}'
##### Break combine column data into rows ##### Break combine column data into rows
```bash ```bash
# For example, separate the following content: # For example, seperate the following content:
# David cat,dog # David cat,dog
# into # into
# David cat # David cat
@ -941,7 +894,7 @@ find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file
##### With sed ##### With sed
```bash ```bash
ls |xargs -n1 -I file sed -i '/^Pos/d' file ls |xargs -n1 -I file sed -i '/^Pos/d' filename
``` ```
##### Add the file name to the first line of file ##### Add the file name to the first line of file
@ -1029,11 +982,6 @@ find . -type f -empty
find . -type f -empty -delete find . -type f -empty -delete
``` ```
##### Recursively count all the files in a directory
```bash
find . -type f | wc -l
```
## Condition and loop ## Condition and loop
[[back to top](#handy-bash-one-liners)] [[back to top](#handy-bash-one-liners)]
@ -1043,7 +991,7 @@ find . -type f | wc -l
if [[ "$c" == "read" ]]; then outputdir="seq"; else outputdir="write" ; fi if [[ "$c" == "read" ]]; then outputdir="seq"; else outputdir="write" ; fi
# Test if myfile contains the string 'test': # Test if myfile contains the string 'test':
if grep -q hello myfile; then echo -e "file contains the string!" ; fi if grep -q hello myfile; then
# Test if mydir is a directory, change to it and do other stuff: # Test if mydir is a directory, change to it and do other stuff:
if cd mydir; then if cd mydir; then
@ -1053,12 +1001,9 @@ else
fi fi
# if variable is null # if variable is null
if [ ! -s "myvariable" ]; then echo -e "variable is null!" ; fi if [ ! -s "myvariable" ]
#True of the length if "STRING" is zero. #True of the length if "STRING" is zero.
# Using test command (same as []), to test if the length of variable is nonzero
test -n "$myvariable" && echo myvariable is "$myvariable" || echo myvariable is not set
# Test if file exist # Test if file exist
if [ -e 'filename' ] if [ -e 'filename' ]
then then
@ -1072,17 +1017,16 @@ then
fi fi
# Test if the value of x is greater or equal than 5 # Test if the value of x is greater or equal than 5
if [ "$x" -ge 5 ]; then echo -e "greater or equal than 5!" ; fi if [ "$x" -ge 5 ]; then
# Test if the value of x is greater or equal than 5, in bash/ksh/zsh: # Test if the value of x is greater or equal than 5, in bash/ksh/zsh:
if ((x >= 5)); then echo -e "greater or equal than 5!" ; fi if ((x >= 5)); then
# Use (( )) for arithmetic operation # Use (( )) for arithmetic operation
if ((j==u+2)); then echo -e "j==u+2!!" ; fi if ((j==u+2))
# Use [[ ]] for comparison # Use [[ ]] for comparison
if [[ $age -gt 21 ]]; then echo -e "forever 21!!" ; fi if [[ $age -gt 21 ]]
``` ```
[More if commands](http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html) [More if commands](http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html)
@ -1090,7 +1034,7 @@ if [[ $age -gt 21 ]]; then echo -e "forever 21!!" ; fi
##### For loop ##### For loop
```bash ```bash
# Echo the file name under the current directory # Echo the file name under the current directory
for i in $(ls); do echo file $i; done for i in $(ls); do echo file $i;done
#or #or
for i in *; do echo file $i; done for i in *; do echo file $i; done
@ -1098,7 +1042,7 @@ for i in *; do echo file $i; done
for dir in $(<myfile); do mkdir $dir; done for dir in $(<myfile); do mkdir $dir; done
# Press any key to continue each loop # Press any key to continue each loop
for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}'); do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key; done for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}');do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key;done
# Print a file line by line when a key is pressed, # Print a file line by line when a key is pressed,
oifs="$IFS"; IFS=$'\n'; for line in $(cat myfile); do ...; done oifs="$IFS"; IFS=$'\n'; for line in $(cat myfile); do ...; done
@ -1108,21 +1052,21 @@ while read -r line; do ...; done <myfile
for line in $(cat myfile); do echo $line; read -n1; done for line in $(cat myfile); do echo $line; read -n1; done
#Loop through an array #Loop through an array
for i in "${arrayName[@]}"; do echo $i; done for i in "${arrayName[@]}"; do echo $i;done
``` ```
##### While loop, ##### While loop,
```bash ```bash
# Column subtraction of a file (e.g. a 3 columns file) # Column subtraction of a file (e.g. a 3 columns file)
while read a b c; do echo $(($c-$b)); done < <(head filename) while read a b c; do echo $(($c-$b));done < <(head filename)
#there is a space between the two '<'s #there is a space between the two '<'s
# Sum up column subtraction # Sum up column subtraction
i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < <(head filename) i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < <(head filename)
# Keep checking a running process (e.g. perl) and start another new process (e.g. python) immediately after it. (BETTER use the wait command! Ctrl+F 'wait') # Keep checking a running process (e.g. perl) and start another new process (e.g. python) immediately after it. (BETTER use the wait command! Ctrl+F 'wait')
while [[ $(pidof perl) ]]; do echo f; sleep 10; done && python timetorunpython.py while [[ $(pidof perl) ]];do echo f;sleep 10;done && python timetorunpython.py
``` ```
##### switch (case in bash) ##### switch (case in bash)
@ -1161,46 +1105,7 @@ date +%F
# or # or
date +'%d-%b-%Y-%H:%M:%S' date +'%d-%b-%Y-%H:%M:%S'
# 10-Apr-2020-21:54:40 #10-Apr-2020-21:54:40
# Returns the current time with nanoseconds.
date +"%T.%N"
# 11:42:18.664217000
# Get the seconds since epoch (Jan 1 1970) for a given date (e.g Mar 16 2021)
date -d "Mar 16 2021" +%s
# 1615852800
# or
date -d "Tue Mar 16 00:00:00 UTC 2021" +%s
# 1615852800
# Convert the number of seconds since epoch back to date
date --date @1615852800
# Tue Mar 16 00:00:00 UTC 2021
```
##### Print current time point for N days ago or N days after
```bash
# print current date first (for the following example)
date +"%F %H:%M:%S"
# 2023-03-11 16:17:09
# print the time that is 1 day ago
date -d"1 day ago" +"%F %H:%M:%S"
# 2023-03-10 16:17:09
# print the time that is 7 days ago
date -d"7 days ago" +"%F %H:%M:%S"
# 2023-03-04 16:17:09
# print the time that is a week ago
date -d"1 week ago" +"%F %H:%M:%S"
# 2023-03-04 16:17:09
# add 1 day to date
date -d"-1 day ago" +"%F %H:%M:%S"
# 2023-03-12 16:17:09
``` ```
##### wait for random duration (e.g. sleep 1-5 second, like adding a jitter) ##### wait for random duration (e.g. sleep 1-5 second, like adding a jitter)
@ -1313,7 +1218,7 @@ shuf -n 100 filename
##### Random order (lucky draw) ##### Random order (lucky draw)
```bash ```bash
for i in a b c d e; do echo $i; done | shuf for i in a b c d e; do echo $i; done| shuf
``` ```
##### Echo series of random numbers between a range (e.g. shuffle numbers from 0-100, then pick 15 of them randomly) ##### Echo series of random numbers between a range (e.g. shuffle numbers from 0-100, then pick 15 of them randomly)
@ -1802,18 +1707,6 @@ du -h
du -sk /var/log/* |sort -rn |head -10 du -sk /var/log/* |sort -rn |head -10
``` ```
##### check the Inode utilization
```
df -i
# Filesystem Inodes IUsed IFree IUse% Mounted on
# devtmpfs 492652 304 492348 1% /dev
# tmpfs 497233 2 497231 1% /dev/shm
# tmpfs 497233 439 496794 1% /run
# tmpfs 497233 16 497217 1% /sys/fs/cgroup
# /dev/nvme0n1p1 5037976 370882 4667094 8% /
# tmpfs 497233 1 497232 1% /run/user/1000
```
##### Show all file system type ##### Show all file system type
```bash ```bash
df -TH df -TH
@ -2109,6 +2002,11 @@ killall pulseaudio
# then press Alt-F2 and type in pulseaudio # then press Alt-F2 and type in pulseaudio
``` ```
##### When sound not working
```bash
killall pulseaudio
```
##### List information about SCSI devices ##### List information about SCSI devices
```bash ```bash
lsscsi lsscsi
@ -2123,7 +2021,7 @@ http://onceuponmine.blogspot.tw/2017/07/create-your-first-simple-daemon.html
##### Tutorial for using your gmail to send email ##### Tutorial for using your gmail to send email
http://onceuponmine.blogspot.tw/2017/10/setting-up-msmtprc-and-use-your-gmail.html http://onceuponmine.blogspot.tw/2017/10/setting-up-msmtprc-and-use-your-gmail.html
##### Using telnet to test open ports, test if you can connect to a port (e.g 53) of a server (e.g 192.168.2.106) ##### Using telnet to test open ports, test if you can connect to a port (e.g 53) of a server (e.g 192.168.2.106)
```bash ```bash
telnet 192.168.2.106 53 telnet 192.168.2.106 53
``` ```
@ -2265,7 +2163,6 @@ sar -n ALL
# reading SAR log file using -f # reading SAR log file using -f
sar -f /var/log/sa/sa31|tail sar -f /var/log/sa/sa31|tail
```
##### Reading from journal file ##### Reading from journal file
```bash ```bash
@ -2483,7 +2380,7 @@ nc -vw5 google.com 22
# From server A: # From server A:
$ sudo nc -l 80 $ sudo nc -l 80
# then you can connect to the 80 port from another server (e.g. server B): # then you can connect to the 80 port from another server (e.g. server B):
# e.g. telnet <server A IP address> 80 # e.g. telent <server A IP address> 80
# then type something in server B # then type something in server B
# and you will see the result in server A! # and you will see the result in server A!
``` ```
@ -2591,16 +2488,6 @@ curl -I http://example.com/
# Vary: Accept-Encoding # Vary: Accept-Encoding
``` ```
##### Find out the http status code of a URL
```bash
curl -s -o /dev/null -w "%{http_code}" https://www.google.com
```
##### Unshorten a shortended URL
```bash
curl -s -o /dev/null -w "%{redirect_url}" https://bit.ly/34EFwWC
```
##### Perform network throughput tests ##### Perform network throughput tests
```bash ```bash
# server side: # server side:
@ -2625,16 +2512,16 @@ sudo iptables A INPUT s <IP> -p tcp dport 80 j DROP
```bash ```bash
# If file is not specified, the file /usr/share/dict/words is used. # If file is not specified, the file /usr/share/dict/words is used.
look phy|head -n 10 look phy|head -n 10
# phycic # Phil
# Phyciodes # Philadelphia
# phycite # Philadelphia's
# Phycitidae # Philby
# phycitol # Philby's
# phyco- # Philip
# phycochrom # Philippe
# phycochromaceae # Philippe's
# phycochromaceous # Philippians
# phycochrome # Philippine
``` ```
##### Repeat printing string n times (e.g. print 'hello world' five times) ##### Repeat printing string n times (e.g. print 'hello world' five times)
@ -2689,41 +2576,7 @@ sdiff fileA fileB
##### Compare two files, strip trailing carriage return/ nextline (e.g. fileA, fileB) ##### Compare two files, strip trailing carriage return/ nextline (e.g. fileA, fileB)
```bash ```bash
diff fileA fileB --strip-trailing-cr diff fileA fileB --strip-trailing-cr
```
##### Find common/differing lines
```bash
# having two sorted and uniqed files (for example after running `$ sort -uo fileA fileA` and same for fileB):
# ------
# fileA:
# ------
# joey
# kitten
# piglet
# puppy
# ------
# fileB:
# ------
# calf
# chick
# joey
# puppy
#
# Find lines in both files
comm -12 fileA fileB
# joey
# puppy
#
# Find lines in fileB that are NOT in fileA
comm -13 fileA fileB
# calf
# chick
#
# Find lines in fileA that are NOT in fileB
comm -23 fileA fileB
# kitten
# piglet
``` ```
##### Number a file (e.g. fileA) ##### Number a file (e.g. fileA)
@ -2799,8 +2652,8 @@ echo {1,2}{1,2}
```bash ```bash
set = {A,T,C,G} set = {A,T,C,G}
group= 5 group= 5
for ((i=0; i<$group; i++)); do for ((i=0; i<$group; i++));do
repetition=$set$repetition; done repetition=$set$repetition;done
bash -c "echo "$repetition"" bash -c "echo "$repetition""
``` ```
@ -2890,28 +2743,12 @@ var=$((var+1))
cat filename|rev|cut -f1|rev cat filename|rev|cut -f1|rev
``` ```
##### Create or replace a file with contents ##### Cat to a file
```bash ```bash
cat >myfile cat >myfile
let me add sth here let me add sth here
# exit with ctrl+d exit by control + c
^C
# or using tee
tee myfile
let me add sth else here
# exit with ctrl+d
```
##### Append to a file with contents
```bash
cat >>myfile
let me add sth here
# exit with ctrl+d
# or using tee
tee -a myfile
let me add sth else here
# exit with ctrl+d
``` ```
##### Clear the contents of a file (e.g. filename) ##### Clear the contents of a file (e.g. filename)
@ -2994,7 +2831,7 @@ tac filename
##### Reverse the result from `uniq -c` ##### Reverse the result from `uniq -c`
```bash ```bash
while read a b; do yes $b |head -n $a ; done <test.txt while read a b; do yes $b |head -n $a ;done <test.txt
``` ```
@ -3030,13 +2867,6 @@ cal
# only display November # only display November
cal -m 11 cal -m 11
``` ```
##### Convert the hexadecimal MD5 checksum value into its base64-encoded format.
```bash
openssl md5 -binary /path/to/file| base64
# NWbeOpeQbtuY0ATWuUeumw==
```
##### Forces applications to use the default language for output ##### Forces applications to use the default language for output
```bash ```bash
export LC_ALL=C export LC_ALL=C
@ -3174,40 +3004,11 @@ rsync -av directory user@ip_address:/path/to/directory.bak
# skip files that are newer on receiver (i prefer this one!) # skip files that are newer on receiver (i prefer this one!)
``` ```
##### Create a temporary directory and `cd` into it
```bash
cd $(mktemp -d)
# for example, this will create a temporary directory "/tmp/tmp.TivmPLUXFT"
```
##### Make all directories at one time! ##### Make all directories at one time!
```bash ```bash
mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat} mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat}
# -p: make parent directory # -p: make parent directory
# this will create: # this will create project/doc/html/; project/doc/info; project/lib/ext ,etc
# project/
# project/bin/
# project/demo/
# project/demo/stat/
# project/doc/
# project/doc/html/
# project/doc/info/
# project/doc/pdf/
# project/lib/
# project/lib/ext/
# project/src/
#
# project/
# ├── bin
# ├── demo
# │ └── stat
# ├── doc
# │ ├── html
# │ ├── info
# │ └── pdf
# ├── lib
# │ └── ext
# └── src
``` ```
##### Run command only if another command returns zero exit status (well done) ##### Run command only if another command returns zero exit status (well done)
@ -3287,7 +3088,7 @@ scp -r directoryname user@ip:/path/to/send
echo $? echo $?
``` ```
##### Extract .xz ##### Extract .xf
``` ```
unxz filename.tar.xz unxz filename.tar.xz
# then # then
@ -3327,7 +3128,8 @@ yes n
# or 'anything': # or 'anything':
yes anything yes anything
# pipe yes to other command # For example:
```bash
yes | rm -r large_directory yes | rm -r large_directory
``` ```
@ -3338,9 +3140,9 @@ fallocate -l 10G 10Gigfile
##### Create dummy file of certain size (e.g. 200mb) ##### Create dummy file of certain size (e.g. 200mb)
```bash ```bash
dd if=/dev/zero of=/dev/shm/200m bs=1024k count=200 dd if=/dev/zero of=//dev/shm/200m bs=1024k count=200
# or # or
dd if=/dev/zero of=/dev/shm/200m bs=1M count=200 dd if=/dev/zero of=//dev/shm/200m bs=1M count=200
# Standard output: # Standard output:
# 200+0 records in # 200+0 records in
@ -3353,29 +3155,9 @@ dd if=/dev/zero of=/dev/shm/200m bs=1M count=200
watch -n 1 wc -l filename watch -n 1 wc -l filename
``` ```
##### Use Bash Strict Mode
```bash
# These options can make your code safer but, depending on how your pipeline is written, it might be too aggressive
# or it might not catch the errors that you are interested in
# for reference see https://gist.github.com/mohanpedala/1e2ff5661761d3abd0385e8223e16425
# and https://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail
set -o errexit # exit immediately if a pipeline returns a non-zero status
set -o errtrace # trap ERR from shell functions, command substitutions, and commands from subshell
set -o nounset # treat unset variables as an error
set -o pipefail # pipe will exit with last non-zero status, if applicable
set -Eue -o pipefail # shorthand for above (pipefail has no short option)
```
##### Print commands and their arguments when execute (e.g. echo `expr 10 + 20 `) ##### Print commands and their arguments when execute (e.g. echo `expr 10 + 20 `)
```bash ```bash
set -x; echo `expr 10 + 20 ` set -x; echo `expr 10 + 20 `
# or
set -o xtrace; echo `expr 10 + 20 `
# to turn it off..
set +x
``` ```
##### Print some meaningful sentences to you (install fortune first) ##### Print some meaningful sentences to you (install fortune first)
@ -3406,9 +3188,6 @@ q -d "," "select c3,c4,c5 from /path/to/file.txt where c3='foo' and c5='boo'"
# Create session and attach: # Create session and attach:
screen screen
# Create a screen and name it 'test'
screen -S test
# Create detached session foo: # Create detached session foo:
screen -S foo -d -m screen -S foo -d -m
@ -3427,17 +3206,16 @@ screen -r foo
# Kill session foo: # Kill session foo:
screen -r foo -X quit screen -r foo -X quit
# Scroll: # Scroll:
# Hit your screen prefix combination (C-a / control+A), then hit Escape. Hit your screen prefix combination (C-a / control+A), then hit Escape.
# Move up/down with the arrow keys (↑ and ↓). Move up/down with the arrow keys (↑ and ↓).
# Redirect output of an already running process in Screen: # Redirect output of an already running process in Screen:
# (C-a / control+A), then hit 'H' (C-a / control+A), then hit 'H'
# Store screen output for Screen: # Store screen output for Screen:
# Ctrl+A, Shift+H Ctrl+A, Shift+H
# You will then find a screen.log file under current directory. # You will then find a screen.log file under current directory.
``` ```
##### Using Tmux for multiple terminal sessions ##### Using Tmux for multiple terminal sessions
@ -3563,8 +3341,6 @@ source .venv/bin/activate
type pip type pip
# 5. Now you can install your pip package, here requirements.txt is simply a txt file containing all the packages you want. (e.g tornado==4.5.3). # 5. Now you can install your pip package, here requirements.txt is simply a txt file containing all the packages you want. (e.g tornado==4.5.3).
pip install -r requirements.txt pip install -r requirements.txt
# 6. Exit virtual environment
deactivate
``` ```