Bash-Oneliner/README.md

3221 lines
66 KiB
Markdown
Raw Normal View History

2019-05-14 17:39:35 +00:00
# Bash-Oneliner
2020-07-18 16:08:43 +00:00
I am glad that you are here! I was working on bioinformatics a few years ago and was amazed by those single-word bash commands which are much faster than my dull scripts, time saved through learning command-line shortcuts and scripting. Recent years I am working on cloud computing and I keep recording those useful commands here. Not all of them is oneliner, but i put effort on making them brief and swift. I am mainly using Ubuntu, Amazon Linux, RedHat, Linux Mint, Mac and CentOS, sorry if the commands don't work on your system.
2016-06-14 06:53:59 +00:00
2019-04-17 08:02:16 +00:00
This blog will focus on simple bash commands for parsing data and Linux system maintenance that i acquired from work and LPIC exam. I apologize that there are no detailed citation for all the commands, but they are probably from dear Google and Stackoverflow.
2016-06-14 06:53:59 +00:00
English and bash are not my first language, please correct me anytime, thank you.
2019-04-17 08:02:16 +00:00
If you know other cool commands, please teach me!
2016-06-14 06:53:59 +00:00
Here's a more stylish version of [Bash-Oneliner](https://onceupon.github.io/Bash-Oneliner/)~
2016-06-16 06:55:27 +00:00
2019-05-14 17:40:01 +00:00
## Handy Bash one-liners
2016-06-14 07:11:00 +00:00
2018-10-24 01:44:11 +00:00
- [Terminal Tricks](#terminal-tricks)
- [Variable](#variable)
2016-06-14 07:22:55 +00:00
- [Grep](#grep)
- [Sed](#sed)
- [Awk](#awk)
- [Xargs](#xargs)
- [Find](#find)
- [Condition and Loop](#condition-and-loop)
2017-09-12 09:38:43 +00:00
- [Math](#math)
- [Time](#time)
2016-06-16 06:44:57 +00:00
- [Download](#download)
2016-06-27 05:54:48 +00:00
- [Random](#random)
2017-06-09 04:26:34 +00:00
- [Xwindow](#xwindow)
2016-06-16 06:44:57 +00:00
- [System](#system)
- [Hardware](#hardware)
2018-12-18 08:34:40 +00:00
- [Networking](#networking)
- [Data Wrangling](#data-wrangling)
2018-02-08 02:05:09 +00:00
- [Others](#others)
2018-10-24 01:44:11 +00:00
## Terminal Tricks
2019-05-10 09:10:05 +00:00
##### Using Ctrl keys
2019-05-10 07:22:11 +00:00
```
2018-10-23 17:38:36 +00:00
Ctrl + n : same as Down arrow.
Ctrl + p : same as Up arrow.
2019-05-09 07:39:05 +00:00
Ctrl + r : begins a backward search through command history.(keep pressing Ctrl + r to move backward)
Ctrl + s : to stop output to terminal.
Ctrl + q : to resume output to terminal after Ctrl + s.
Ctrl + a : move to the beginning of line.
Ctrl + e : move to the end of line.
Ctrl + d : if you've type something, Ctrl + d deletes the character under the cursor, else, it escapes the current shell.
Ctrl + k : delete all text from the cursor to the end of line.
Ctrl + x + backspace : delete all text from the beginning of line to the cursor.
Ctrl + t : transpose the character before the cursor with the one under the cursor, press Esc + t to transposes the two words before the cursor.
Ctrl + w : cut the word before the cursor; then Ctrl + y paste it
Ctrl + u : cut the line before the cursor; then Ctrl + y paste it
2019-05-11 14:42:07 +00:00
Ctrl + _ : undo typing.
Ctrl + l : equivalent to clear.
Ctrl + x + Ctrl + e : launch editor defined by $EDITOR to input your command. Useful for multi-line commands.
```
2018-10-24 01:44:11 +00:00
##### Change case
```bash
Esc + u
# converts text from cursor to the end of the word to uppercase.
2018-10-24 01:44:11 +00:00
Esc + l
# converts text from cursor to the end of the word to lowercase.
2018-10-24 01:44:11 +00:00
Esc + c
# converts letter under the cursor to uppercase.
```
##### Run history number (e.g. 53)
2018-10-24 01:44:11 +00:00
```bash
!53
```
##### Run last command
2018-10-24 01:44:11 +00:00
```bash
!!
# run the previous command using sudo
sudo !!
# of course you need to enter your password
2018-10-24 01:44:11 +00:00
```
##### Run last command and change some parameter using caret substitution (e.g. last command: echo 'aaa' -> rerun as: echo 'bbb')
2018-10-24 01:44:11 +00:00
```bash
#last command: echo 'aaa'
^aaa^bbb
#echo 'bbb'
#bbb
#Notice that only the first aaa will be replaced, if you want to replace all 'aaa', use ':&' to repeat it:
^aaa^bbb^:&
#or
!!:gs/aaa/bbb/
2018-10-24 01:44:11 +00:00
```
##### Run past command that began with (e.g. cat filename)
```bash
!cat
# or
2018-10-24 01:44:11 +00:00
!c
# run cat filename again
```
2016-06-14 07:20:16 +00:00
2019-05-14 04:21:42 +00:00
##### Bash globbing
```bash
2019-05-14 04:30:18 +00:00
# '*' serves as a "wild card" for filename expansion.
/b?n/?at #/bin/cat
# '?' serves as a single-character "wild card" for filename expansion.
/etc/pa*wd #/etc/passwd
2019-05-14 04:30:18 +00:00
# [] serves to match the character from a range.
ls -l [a-z]* #list all files with alphabet in its filename.
# {} can be used to match filenames with more than one patterns
ls {*.sh,*.py} #list all .sh and .py files
```
2019-05-10 08:20:40 +00:00
##### Some handy environment variables
```
$0 :name of shell or shell script.
$1, $2, $3, ... :positional parameters.
$# :number of positional parameters.
$? :most recent foreground pipeline exit status.
$- :current options set for the shell.
$$ :pid of the current shell (not subshell).
$! :is the PID of the most recent background command.
2019-06-11 06:22:34 +00:00
$DESKTOP_SESSION current display manager
$EDITOR preferred text editor.
$LANG current language.
$PATH list of directories to search for executable files (i.e. ready-to-run programs)
$PWD current directory
$SHELL current shell
2019-06-11 06:22:34 +00:00
$USER current username
$HOSTNAME current hostname
2019-05-10 08:20:40 +00:00
```
2020-07-19 13:15:06 +00:00
## Variable
[[back to top](#handy-bash-one-liners)]
##### Variable substitution within quotes
```bash
# foo=bar
echo "'$foo'"
#'bar'
# double/single quotes around single quotes make the inner single quotes expand variables
```
##### Get the length of variable
```bash
var="some string"
echo ${#var}
# 11
```
##### Get the first character of the variable
```bash
var=string
echo "${var:0:1}"
#s
# or
echo ${var%%"${var#?}"}
```
##### Remove the first or last string from variable
```bash
var="some string"
echo ${var:2}
#me string
```
##### Replacement (e.g. remove the first leading 0 )
```bash
var="0050"
echo ${var[@]#0}
#050
```
##### Replacement (e.g. replace 'a' with ',')
```bash
{var/a/,}
```
##### Replace all (e.g. replace all 'a' with ',')
```bash
{var//a/,}
```
```bash
#with grep
test="god the father"
grep ${test// /\\\|} file.txt
# turning the space into 'or' (\|) in grep
```
##### To change the case of the string stored in the variable to lowercase (Parameter Expansion)
```bash
var=HelloWorld
echo ${var,,}
helloworld
```
##### Expand and then execute variable/argument
```bash
cmd="bar=foo"
eval "$cmd"
echo "$bar" # foo
```
## Math
[[back to top](#handy-bash-one-liners)]
##### Arithmetic Expansion in Bash (Operators: +, -, *, /, %, etc)
```bash
echo $(( 10 + 5 )) #15
x=1
echo $(( x++ )) #1 , notice that it is still 1, since it's post-incremen
echo $(( x++ )) #2
echo $(( ++x )) #4 , notice that it is not 3 since it's pre-incremen
echo $(( x-- )) #4
echo $(( x-- )) #3
echo $(( --x )) #1
x=2
y=3
echo $(( x ** y )) #8
```
##### Print out the prime factors of a number (e.g. 50)
```bash
factor 50
# 50: 2 5 5
```
##### Sum up input list (e.g. seq 10)
```bash
seq 10|paste -sd+|bc
```
##### Sum up a file (each line in file contains only one number)
```bash
awk '{s+=$1} END {print s}' filename
```
##### Column subtraction
```bash
cat file| awk -F '\t' 'BEGIN {SUM=0}{SUM+=$3-$2}END{print SUM}'
```
##### Simple math with expr
```bash
expr 10+20 #30
expr 10\*20 #600
expr 30 \> 20 #1 (true)
```
##### More math with bc
```bash
# Number of decimal digit/ significant figure
echo "scale=2;2/3" | bc
#.66
# Exponent operator
echo "10^2" | bc
#100
# Using variables
echo "var=5;--var"| bc
#4
```
2017-03-29 08:52:59 +00:00
## Grep
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2018-09-06 06:31:00 +00:00
2018-10-24 09:23:35 +00:00
##### Type of grep
```bash
grep = grep -G # Basic Regular Expression (BRE)
2018-10-24 09:25:32 +00:00
fgrep = grep -F # fixed text, ignoring meta-charachetrs
2018-10-24 09:23:35 +00:00
egrep = grep -E # Extended Regular Expression (ERE)
pgrep = grep -P # Perl Compatible Regular Expressions (PCRE)
rgrep = grep -r # recursive
```
2019-05-10 09:10:05 +00:00
##### Grep and count number of empty lines
```bash
grep -c "^$"
```
2018-10-24 09:23:35 +00:00
##### Grep and return only integer
```bash
grep -o '[0-9]*'
#or
grep -oP '\d'
```
##### Grep integer with certain number of digits (e.g. 3)
```bash
grep [0-9]\{3\}
# or
grep -E [0-9]{3}
# or
grep -P \d{3}
```
2018-09-06 06:31:00 +00:00
##### Grep only IP address
```bash
2018-09-06 06:31:16 +00:00
grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
# or
grep -Po '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}'
```
##### Grep whole word (e.g. 'target')
```bash
grep -w 'target'
#or using RE
grep '\btarget\b'
```
##### Grep returning lines before and after match (e.g. 'bbo')
```bash
# return also 3 lines after match
grep -A 3 'bbo'
# return also 3 lines before match
grep -B 3 'bbo'
# return also 3 lines before and after match
grep -C 3 'bbo'
2018-09-06 06:31:00 +00:00
```
##### Grep string starting with (e.g. 'S')
```bash
grep -o 'S.*'
```
##### Extract text between words (e.g. w1,w2)
2016-07-01 10:52:19 +00:00
```bash
grep -o -P '(?<=w1).*(?=w2)'
```
2016-06-14 07:20:16 +00:00
##### Grep lines without word (e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
2016-07-20 04:08:30 +00:00
grep -v bbo filename
```
##### Grep lines not begin with string (e.g. #)
```bash
grep -v '^#' file.txt
```
##### Grep variables with space within it (e.g. myvar="some strings")
```bash
grep "$myvar" filename
#remember to quote the variable!
2019-05-10 09:10:05 +00:00
```
##### Grep only one/first match (e.g. 'bbo')
2016-07-20 04:08:30 +00:00
```bash
grep -m 1 bbo filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Grep and return number of matching line(e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
grep -c bbo filename
```
2016-06-14 06:57:38 +00:00
##### Count occurrence (e.g. three times a line count three times)
2016-07-01 10:52:19 +00:00
```bash
grep -o bbo filename |wc -l
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Case insensitive grep (e.g. 'bbo'/'BBO'/'Bbo')
2016-07-01 10:52:19 +00:00
```bash
grep -i "bbo" filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### COLOR the match (e.g. 'bbo')!
2016-07-01 10:52:19 +00:00
```bash
grep --color bbo filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Grep search all files in a directory(e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
grep -R bbo /path/to/directory
# or
grep -r bbo /path/to/directory
2016-07-01 10:52:19 +00:00
```
##### Search all files in directory, do not ouput the filenames (e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
grep -rh bbo /path/to/directory
```
##### Search all files in directory, output ONLY the filenames with matches(e.g. 'bbo')
2017-10-11 07:49:28 +00:00
```bash
grep -rl bbo /path/to/directory
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep OR (e.g. A or B or C or D)
2016-07-01 10:52:19 +00:00
```
grep 'A\|B\|C\|D'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep AND (e.g. A and B)
2016-07-01 10:52:19 +00:00
```bash
grep 'A.*B'
2016-07-01 10:52:19 +00:00
```
##### Regex any singer character (e.g. ACB or AEB)
```bash
grep 'A.B'
```
2019-05-10 09:10:05 +00:00
##### Regex with or without a certain character (e.g. color or colour)
```bash
grep colou?r
```
##### Grep all content of a fileA from fileB
2016-07-01 10:52:19 +00:00
```bash
grep -f fileA fileB
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep a tab
2016-07-01 10:52:19 +00:00
```bash
grep $'\t'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Grep variable from variable
2016-08-25 11:40:18 +00:00
```bash
$echo "$long_str"|grep -q "$short_str"
if [ $? -eq 0 ]; then echo 'found'; fi
#grep -q will output 0 if match found
#remember to add space between []!
2016-08-25 11:40:18 +00:00
```
2017-06-09 08:53:36 +00:00
##### Grep strings between a bracket()
2016-09-05 07:26:20 +00:00
```bash
grep -oP '\(\K[^\)]+'
```
2017-06-09 08:53:36 +00:00
##### Grep number of characters with known strings in between(e.g. AAEL000001-RA)
2016-09-05 07:26:20 +00:00
```bash
grep -o -w "\w\{10\}\-R\w\{1\}"
# \w word character [0-9a-zA-Z_] \W not word character
```
##### Skip directory (e.g. 'bbo')
```bash
grep -d skip 'bbo' /path/to/files/*
```
2016-09-05 07:37:30 +00:00
2017-03-29 08:52:59 +00:00
## Sed
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2017-10-02 02:38:44 +00:00
##### Remove the 1st line
```bash
sed 1d filename
```
2018-03-07 03:16:18 +00:00
##### Remove the first 100 lines (remove line 1-100)
2017-10-02 02:38:44 +00:00
```bash
sed 1,100d filename
```
##### Remove lines with string (e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
sed "/bbo/d" filename
- case insensitive:
sed "/bbo/Id" filename
2016-07-01 10:52:19 +00:00
```
##### Remove lines whose nth character not equal to a value (e.g. 5th character not equal to 2)
```bash
sed -E '/^.{5}[^2]/d'
#aaaa2aaa (you can stay)
#aaaa1aaa (delete!)
```
2016-06-14 06:57:38 +00:00
##### Edit infile (edit and save to file), (e.g. deleting the lines with 'bbo' and save to file)
2016-07-01 10:52:19 +00:00
```bash
sed -i "/bbo/d" filename
```
##### When using variable (e.g. $i), use double quotes " "
2016-07-01 10:52:19 +00:00
```bash
# e.g. add >$i to the first line (to make a bioinformatics FASTA file)
sed "1i >$i"
# notice the double quotes! in other examples, you can use a single quote, but here, no way!
# '1i' means insert to first line
2016-07-01 10:52:19 +00:00
```
2019-04-17 07:57:50 +00:00
##### Using environment variable and end-of-line pattern at the same time.
```bash
# Use backslash for end-of-line $ pattern, and double quotes for expressing the variable
sed -e "\$s/\$/\n+--$3-----+/"
```
2017-10-02 02:38:44 +00:00
##### Delete/remove empty lines
2016-07-01 10:52:19 +00:00
```bash
sed '/^\s*$/d'
# or
sed '/^$/d'
2016-07-01 10:52:19 +00:00
```
2017-10-02 02:38:44 +00:00
##### Delete/remove last line
2016-07-01 10:52:19 +00:00
```bash
sed '$d'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-10-02 02:38:44 +00:00
##### Delete/remove last character from end of file
2016-07-13 08:37:37 +00:00
```bash
sed -i '$ s/.$//' filename
```
##### Add string to beginning of file (e.g. "\[")
```bash
sed -i '1s/^/[/' file
```
2019-04-17 07:57:50 +00:00
##### Add string at certain line number (e.g. add 'something' to line 1 and line 3)
```bash
sed -e '1isomething -e '3isomething'
```
2017-06-09 08:53:36 +00:00
##### Add string to end of file (e.g. "]")
2016-07-13 08:37:37 +00:00
```bash
sed '$s/$/]/' filename
```
2017-06-09 08:53:36 +00:00
##### Add newline to the end
2017-05-16 07:52:33 +00:00
```bash
2017-05-10 08:18:56 +00:00
sed '$a\'
```
2016-07-13 08:37:37 +00:00
##### Add string to beginning of every line (e.g. 'bbo')
2016-09-09 10:14:09 +00:00
```bash
sed -e 's/^/bbo/' file
```
2017-06-09 08:53:36 +00:00
##### Add string to end of each line (e.g. "}")
2016-07-13 08:29:31 +00:00
```bash
2016-07-13 08:37:37 +00:00
sed -e 's/$/\}\]/' filename
2016-07-13 08:29:31 +00:00
```
##### Add \n every nth character (e.g. every 4th character)
2016-07-01 10:52:19 +00:00
```bash
sed 's/.\{4\}/&\n/g'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Concatenate/combine/join files with a seperator and next line (e.g separate by ",")
2016-07-27 02:34:04 +00:00
```bash
sed -s '$a,' *.json > all.json
```
2017-06-09 08:53:36 +00:00
##### Substitution (e.g. replace A by B)
2016-07-01 10:52:19 +00:00
```bash
sed 's/A/B/g' filename
2016-07-01 10:52:19 +00:00
```
##### Substitution with wildcard (e.g. replace a line start with aaa= by aaa=/my/new/path)
```bash
sed "s/aaa=.*/aaa=\/my\/new\/path/g"
```
##### Select lines start with string (e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
sed -n '/^@S/p'
2016-07-01 10:52:19 +00:00
```
##### Delete lines with string (e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
sed '/bbo/d' filename
2016-07-01 10:52:19 +00:00
```
2017-11-24 08:50:22 +00:00
##### Print/get/trim a range of line (e.g. line 500-5000)
```bash
sed -n 500,5000p filename
```
2017-06-09 08:53:36 +00:00
##### Print every nth lines
2016-07-01 10:52:19 +00:00
```bash
sed -n '0~3p' filename
# catch 0: start; 3: step
2016-07-01 10:52:19 +00:00
```
2017-06-09 08:53:36 +00:00
##### Print every odd # lines
2016-07-01 10:52:19 +00:00
```bash
sed -n '1~2p'
2016-07-01 10:52:19 +00:00
```
2017-06-09 08:53:36 +00:00
##### Print every third line including the first line
2016-07-01 10:52:19 +00:00
```bash
sed -n '1p;0~3p'
2016-07-01 10:52:19 +00:00
```
2017-06-09 08:53:36 +00:00
##### Remove leading whitespace and tabs
2016-07-01 10:52:19 +00:00
```bash
sed -e 's/^[ \t]*//'
# Notice a whitespace before '\t'!!
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### Remove only leading whitespace
2016-07-01 10:52:19 +00:00
```bash
sed 's/ *//'
2016-06-14 08:03:32 +00:00
# notice a whitespace before '*'!!
```
2017-06-09 08:53:36 +00:00
##### Remove ending commas
2016-07-01 10:52:19 +00:00
```bash
sed 's/,$//g'
2016-07-01 10:52:19 +00:00
```
2017-06-09 08:53:36 +00:00
##### Add a column to the end
2016-07-01 10:52:19 +00:00
```bash
sed "s/$/\t$i/"
# $i is the valuable you want to add
# To add the filename to every last column of the file
2016-07-01 10:52:19 +00:00
for i in $(ls);do sed -i "s/$/\t$i/" $i;done
```
2016-08-17 10:05:54 +00:00
##### Add extension of filename to last column
2016-08-17 10:05:54 +00:00
```bash
for i in T000086_1.02.n T000086_1.02.p;do sed "s/$/\t${i/*./}/" $i;done >T000086_1.02.np
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Remove newline\ nextline
2016-07-01 10:52:19 +00:00
```bash
sed ':a;N;$!ba;s/\n//g'
```
2016-06-14 06:57:38 +00:00
2017-09-28 10:10:55 +00:00
##### Print a particular line (e.g. 123th line)
```bash
sed -n -e '123p'
```
##### Print a number of lines (e.g. line 10th to line 33 rd)
2016-07-01 10:52:19 +00:00
```bash
sed -n '10,33p' <filename
```
2016-07-01 06:47:42 +00:00
##### Change delimiter
2016-09-01 09:24:27 +00:00
```bash
sed 's=/=\\/=g'
```
2017-06-09 08:53:36 +00:00
##### Replace with wildcard (e.g A-1-e or A-2-e or A-3-e....)
2016-10-13 09:57:34 +00:00
```bash
sed 's/A-.*-e//g' filename
```
2017-06-09 08:53:36 +00:00
##### Remove last character of file
2018-01-03 04:21:42 +00:00
```bash
sed '$ s/.$//'
```
2016-10-13 09:57:34 +00:00
2017-06-09 08:53:36 +00:00
##### Insert character at specified position of file (e.g. AAAAAA --> AAA#AAA)
2018-01-03 04:21:42 +00:00
```bash
sed -r -e 's/^.{3}/&#/' file
```
2016-06-14 06:57:38 +00:00
2017-05-10 08:18:56 +00:00
2017-03-29 08:52:59 +00:00
## Awk
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2016-06-14 07:55:08 +00:00
##### Set tab as field separator
2016-07-01 10:52:19 +00:00
```bash
awk -F $'\t'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Output as tab separated (also as field separator)
2016-07-01 10:52:19 +00:00
```bash
awk -v OFS='\t'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Pass variable
2016-07-01 10:52:19 +00:00
```bash
a=bbo;b=obb;
awk -v a="$a" -v b="$b" "$1==a && $10=b" filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2017-06-09 08:53:36 +00:00
##### Print line number and number of characters on each line
2016-07-01 10:52:19 +00:00
```bash
awk '{print NR,length($0);}' filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Find number of columns
2016-07-01 10:52:19 +00:00
```bash
awk '{print NF}'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Reverse column order
2016-07-01 10:52:19 +00:00
```bash
awk '{print $2, $1}'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Check if there is a comma in a column (e.g. column $1)
2016-07-01 10:52:19 +00:00
```bash
awk '$1~/,/ {print}'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Split and do for loop
2016-07-01 10:52:19 +00:00
```bash
awk '{split($2, a,",");for (i in a) print $1"\t"a[i]}' filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Print all lines before nth occurrence of a string (e.g stop print lines when 'bbo' appears 7 times)
2016-07-01 10:52:19 +00:00
```bash
awk -v N=7 '{print}/bbo/&& --N<=0 {exit}'
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Print filename and last line of all files in directory
2016-07-15 09:12:52 +00:00
```bash
2017-09-15 03:07:16 +00:00
ls|xargs -n1 -I file awk '{s=$0};END{print FILENAME,s}' file
2016-07-15 09:12:52 +00:00
```
2017-06-09 08:53:36 +00:00
##### Add string to the beginning of a column (e.g add "chr" to column $3)
2016-07-01 10:52:19 +00:00
```bash
awk 'BEGIN{OFS="\t"}$3="chr"$3'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Remove lines with string (e.g. 'bbo')
2016-07-01 10:52:19 +00:00
```bash
awk '!/bbo/' file
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Remove last column
2018-02-09 02:36:52 +00:00
```bash
awk 'NF{NF-=1};1' file
```
2017-06-09 08:53:36 +00:00
##### Usage and meaning of NR and FNR
2016-07-01 10:52:19 +00:00
```bash
# For example there are two files:
# fileA:
# a
# b
# c
# fileB:
# d
# e
awk 'print FILENAME, NR,FNR,$0}' fileA fileB
# fileA 1 1 a
# fileA 2 2 b
# fileA 3 3 c
# fileB 4 1 d
# fileB 5 2 e
2016-07-01 10:52:19 +00:00
```
2017-06-09 08:53:36 +00:00
##### AND gate
2016-07-01 10:52:19 +00:00
```bash
# For example there are two files:
# fileA:
# 1 0
# 2 1
# 3 1
# 4 0
# fileB:
# 1 0
# 2 1
# 3 0
# 4 1
awk -v OFS='\t' 'NR=FNR{a[$1]=$2;next} NF {print $1,((a[$1]=$2)? $2:"0")}' fileA fileB
# 1 0
# 2 1
# 3 0
# 4 0
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### Round all numbers of file (e.g. 2 significant figure)
2016-07-01 10:52:19 +00:00
```bash
awk '{while (match($0, /[0-9]+\[0-9]+/)){
2016-06-14 07:55:08 +00:00
\printf "%s%.2f", substr($0,0,RSTART-1),substr($0,RSTART,RLENGTH)
\$0=substr($0, RSTART+RLENGTH)
\}
\print
\}'
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
2017-06-09 08:53:36 +00:00
##### Give number/index to every row
2016-07-01 10:52:19 +00:00
```bash
awk '{printf("%s\t%s\n",NR,$0)}'
```
2017-06-09 08:53:36 +00:00
##### Break combine column data into rows
2016-08-18 03:03:45 +00:00
```bash
# For example, seperate the following content:
# David cat,dog
# into
# David cat
# David dog
2016-08-18 03:03:45 +00:00
awk '{split($2,a,",");for(i in a)print $1"\t"a[i]}' file
# Detail here: http://stackoverflow.com/questions/33408762/bash-turning-single-comma-separated-column-into-multi-line-string
2016-08-18 03:03:45 +00:00
```
2017-06-09 08:53:36 +00:00
##### Average a file (each line in file contains only one number)
2016-09-05 07:37:30 +00:00
```bash
awk '{s+=$1}END{print s/NR}'
```
2017-06-09 08:53:36 +00:00
##### Print field start with string (e.g Linux)
2016-09-01 09:24:27 +00:00
```bash
awk '$1 ~ /^Linux/'
```
2017-06-09 08:53:36 +00:00
##### Sort a row (e.g. 1 40 35 12 23 --> 1 12 23 35 40)
2016-09-02 03:29:05 +00:00
```bash
awk ' {split( $0, a, "\t" ); asort( a ); for( i = 1; i <= length(a); i++ ) printf( "%s\t", a[i] ); printf( "\n" ); }'
```
2016-09-01 09:24:27 +00:00
##### Subtract previous row values (add column6 which equal to column4 minus last column5)
2017-06-16 09:53:05 +00:00
```bash
awk '{$6 = $4 - prev5; prev5 = $5; print;}'
```
2017-03-29 08:52:59 +00:00
## Xargs
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2016-06-28 02:59:23 +00:00
2017-06-09 08:53:36 +00:00
##### Set tab as delimiter (default:space)
2016-07-01 10:52:19 +00:00
```bash
xargs -d\t
```
2016-06-14 06:57:38 +00:00
##### Prompt commands before running commands
```bash
ls|xargs -L1 -p head
```
2017-06-09 08:53:36 +00:00
##### Display 3 items per line
2016-07-01 10:52:19 +00:00
```bash
echo 1 2 3 4 5 6| xargs -n 3
# 1 2 3
# 4 5 6
2016-06-14 08:03:32 +00:00
```
2017-06-09 08:53:36 +00:00
##### Prompt before execution
2016-07-01 10:52:19 +00:00
```bash
echo a b c |xargs -p -n 3
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Print command along with output
2016-07-01 10:52:19 +00:00
```bash
xargs -t abcd
# bin/echo abcd
# abcd
2016-06-14 08:03:32 +00:00
```
2017-06-09 08:53:36 +00:00
##### With find and rm
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.html"|xargs rm
# when using a backtick
rm `find . -name "*.html"`
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Delete files with whitespace in filename (e.g. "hello 2001")
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.c" -print0|xargs -0 rm -rf
```
2016-06-14 06:57:38 +00:00
##### Show limits on command-line length
2016-07-01 10:52:19 +00:00
```bash
xargs --show-limits
# Output from my Ubuntu:
# Your environment variables take up 3653 bytes
# POSIX upper limit on argument length (this system): 2091451
# POSIX smallest allowable upper limit on argument length (all systems): 4096
# Maximum length of command we could actually use: 2087798
# Size of command buffer we are actually using: 131072
# Maximum parallelism (--max-procs must be no greater): 2147483647
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Move files to folder
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.bak" -print 0|xargs -0 -I {} mv {} ~/old
2016-06-14 07:55:08 +00:00
# or
2016-07-01 10:52:19 +00:00
find . -name "*.bak" -print 0|xargs -0 -I file mv file ~/old
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Move first 100th files to a directory (e.g. d1)
2016-07-01 10:52:19 +00:00
```bash
ls |head -100|xargs -I {} mv {} d1
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Parallel
2016-07-01 10:52:19 +00:00
```bash
time echo {1..5} |xargs -n 1 -P 5 sleep
# a lot faster than:
2016-07-01 10:52:19 +00:00
time echo {1..5} |xargs -n1 sleep
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Copy all files from A to B
2016-07-01 10:52:19 +00:00
```bash
find /dir/to/A -type f -name "*.py" -print 0| xargs -0 -r -I file cp -v -p file --target-directory=/path/to/B
# v: verbose|
# p: keep detail (e.g. owner)
2016-06-14 06:57:38 +00:00
```
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### With sed
2016-07-01 10:52:19 +00:00
```bash
ls |xargs -n1 -I file sed -i '/^Pos/d' filename
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Add the file name to the first line of file
2016-07-01 10:52:19 +00:00
```bash
ls |sed 's/.txt//g'|xargs -n1 -I file sed -i -e '1 i\>file\' file.txt
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Count all files
2016-07-01 10:52:19 +00:00
```bash
ls |xargs -n1 wc -l
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Turn output into a single line
2016-07-01 10:52:19 +00:00
```bash
ls -l| xargs
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Count files within directories
2016-07-01 10:52:19 +00:00
```bash
echo mso{1..8}|xargs -n1 bash -c 'echo -n "$1:"; ls -la "$1"| grep -w 74 |wc -l' --
# "--" signals the end of options and display further option processing
```
2016-06-14 08:03:32 +00:00
2017-06-09 08:53:36 +00:00
##### Count lines in all file, also count total lines
2016-07-01 10:52:19 +00:00
```bash
ls|xargs wc -l
```
2017-06-09 08:53:36 +00:00
##### Xargs and grep
2016-09-29 09:17:45 +00:00
```bash
cat grep_list |xargs -I{} grep {} filename
```
2016-06-14 06:57:38 +00:00
2017-10-11 07:49:28 +00:00
##### Xargs and sed (replace all old ip address with new ip address under /etc directory)
```bash
grep -rl '192.168.1.111' /etc | xargs sed -i 's/192.168.1.111/192.168.2.111/g'
```
## Find
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
##### List all sub directory/file in the current directory
2016-07-01 10:52:19 +00:00
```bash
find .
```
2016-06-14 06:57:38 +00:00
##### List all files under the current directory
2016-07-01 10:52:19 +00:00
```bash
find . -type f
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### List all directories under the current directory
2016-07-01 10:52:19 +00:00
```bash
find . -type d
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Edit all files under current directory (e.g. replace 'www' with 'ww')
2016-07-01 10:52:19 +00:00
```bash
find . -name '*.php' -exec sed -i 's/www/w/g' {} \;
# if there are no subdirectory
2016-07-01 10:52:19 +00:00
replace "www" "w" -- *
# a space before *
2016-07-01 10:52:19 +00:00
```
2017-06-09 08:53:36 +00:00
##### Find and output only filename (e.g. "mso")
2016-07-01 10:52:19 +00:00
```bash
find mso*/ -name M* -printf "%f\n"
```
2016-06-14 06:57:38 +00:00
2017-06-09 08:53:36 +00:00
##### Find and delete file with size less than (e.g. 74 byte)
2016-07-01 10:52:19 +00:00
```bash
find . -name "*.mso" -size -74c -delete
# M for MB, etc
2016-07-01 10:52:19 +00:00
```
2020-07-18 14:37:25 +00:00
##### Find empty (0 byte) files
```bash
2020-07-18 14:37:25 +00:00
find . -type f -empty
# to further delete all the empty files
find . -type f -empty -delete
```
2016-06-14 06:57:38 +00:00
## Condition and loop
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
##### If statement
2016-07-01 10:52:19 +00:00
```bash
2018-05-31 08:07:50 +00:00
# if and else loop for string matching
if [[ "$c" == "read" ]]; then outputdir="seq"; else outputdir="write" ; fi
2018-05-31 08:07:50 +00:00
# Test if myfile contains the string 'test':
if grep -q hello myfile; then …
2017-01-05 07:44:41 +00:00
# Test if mydir is a directory, change to it and do other stuff:
if cd mydir; then
echo 'some content' >myfile
else
echo >&2 "Fatal error. This script requires mydir."
fi
2018-12-19 09:59:27 +00:00
# if variable is null
if [ ! -s "myvariable" ]
#True of the length if "STRING" is zero.
# Test if file exist
2017-06-09 08:53:36 +00:00
if [ -e 'filename' ]
2017-01-05 07:44:41 +00:00
then
2017-01-05 07:45:36 +00:00
echo -e "file exists!"
2017-01-05 07:44:41 +00:00
fi
# Test if file exist but also including symbolic links:
if [ -e myfile ] || [ -L myfile ]
then
echo -e "file exists!"
fi
# Test if the value of x is greater or equal than 5
if [ "$x" -ge 5 ]; then …
# Test if the value of x is greater or equal than 5, in bash/ksh/zsh:
if ((x >= 5)); then …
# Use (( )) for arithmetic operation
if ((j==u+2))
# Use [[ ]] for comparison
if [[ $age -gt 21 ]]
2017-10-25 09:59:22 +00:00
```
2017-01-05 07:44:41 +00:00
2018-12-20 04:11:41 +00:00
[More if commands](http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html)
2017-06-09 08:53:36 +00:00
##### For loop
2016-07-01 10:52:19 +00:00
```bash
for i in $(ls); do echo file $i;done
#or
for i in *; do echo file $i; done
2016-06-16 06:44:57 +00:00
# Press any key to continue each loop
2017-09-26 01:16:16 +00:00
for i in $(cat tpc_stats_0925.log |grep failed|grep -o '\query\w\{1,2\}');do cat ${i}.log; read -rsp $'Press any key to continue...\n' -n1 key;done
# Print a file line by line when a key is pressed,
oifs="$IFS"; IFS=$'\n'; for line in $(cat myfile); do ...; done
while read -r line; do ...; done <myfile
#If only one word a line, simply
for line in $(cat myfile); do echo $line; read -n1; done
#Loop through an array
for i in "${arrayName[@]}"; do echo $i;done
```
##### While loop,
```bash
# Column subtraction of a file (e.g. a 3 columns file)
while read a b c; do echo $(($c-$b));done < <(head filename)
#there is a space between the two '<'s
# Sum up column subtraction
i=0; while read a b c; do ((i+=$c-$b)); echo $i; done < <(head filename)
2019-05-09 07:39:05 +00:00
# Keep checking a running process (e.g. perl) and start another new process (e.g. python) immediately after it. (BETTER use the wait command! Ctrl+F 'wait')
while [[ $(pidof perl) ]];do echo f;sleep 10;done && python timetorunpython.py
```
2018-04-12 06:49:54 +00:00
##### switch (case in bash)
```bash
read type;
case $type in
'0')
echo 'how'
;;
'1')
echo 'are'
;;
'2')
echo 'you'
;;
esac
```
2017-09-12 09:38:43 +00:00
## Time
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
##### Find out the time require for executing a command
```bash
time echo hi
```
##### Wait for some time (e.g 10s)
```bash
sleep 10
2020-07-19 12:36:59 +00:00
```
##### Print date in %Y-%m-%d
```bash
date +%F
# 2020-07-19
```
# wait for random duration (e.g. sleep 1-5 second, like adding a jitter)
sleep $[ ( $RANDOM % 5 ) + 1 ]
```
##### Log out your account after a certain period of time (e.g 10 seconds)
```bash
TMOUT=10
#once you set this variable, logout timer start running!
```
##### Set how long you want to run a command
```bash
2019-05-09 07:39:05 +00:00
#This will run the command 'sleep 10' for only 1 second.
timeout 1 sleep 10
```
##### Set when you want to run a command (e.g 1 min from now)
```bash
at now + 1min #time-units can be minutes, hours, days, or weeks
warning: commands will be executed using /bin/sh
2018-04-18 03:30:24 +00:00
at> echo hihigithub >~/itworks
at> <EOT> # press Ctrl + D to exit
job 1 at Wed Apr 18 11:16:00 2018
```
2017-09-12 09:38:43 +00:00
2017-03-29 08:52:59 +00:00
## Download
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
##### Download the content of this README.md (the one your are viewing now)
```bash
curl https://raw.githubusercontent.com/onceupon/Bash-Oneliner/master/README.md | pandoc -f markdown -t man | man -l -
# or w3m (a text based web browser and pager)
curl https://raw.githubusercontent.com/onceupon/Bash-Oneliner/master/README.md | pandoc | w3m -T text/html
# or using emacs (in emac text editor)
emacs --eval '(org-mode)' --insert <(curl https://raw.githubusercontent.com/onceupon/Bash-Oneliner/master/README.md | pandoc -t org)
# or using emacs (on terminal, exit using Ctrl + x then Ctrl + c)
emacs -nw --eval '(org-mode)' --insert <(curl https://raw.githubusercontent.com/onceupon/Bash-Oneliner/master/README.md | pandoc -t org)
```
##### Download all from a page
2016-07-01 10:52:19 +00:00
```bash
wget -r -l1 -H -t1 -nd -N -np -A mp3 -e robots=off http://example.com
# -r: recursive and download all links on page
# -l1: only one level link
# -H: span host, visit other hosts
# -t1: numbers of retries
# -nd: don't make new directories, download to here
# -N: turn on timestamp
# -nd: no parent
# -A: type (separate by ,)
# -e robots=off: ignore the robots.txt file which stop wget from crashing the site, sorry example.com
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
2017-06-09 08:53:36 +00:00
##### Upload a file to web and download (https://transfer.sh/)
```bash
# Upload a file (e.g. filename.txt):
2017-06-09 08:53:36 +00:00
curl --upload-file ./filename.txt https://transfer.sh/filename.txt
# the above command will return a URL, e.g: https://transfer.sh/tG8rM/filename.txt
# Next you can download it by:
2017-06-09 08:53:36 +00:00
curl https://transfer.sh/tG8rM/filename.txt -o filename.txt
```
##### Download file if necessary
```bash
data=file.txt
url=http://www.example.com/$data
if [ ! -s $data ];then
2017-06-09 08:53:36 +00:00
echo "downloading test data..."
wget $url
fi
```
##### Wget to a filename (when a long name)
```bash
wget -O filename "http://example.com"
```
##### Wget files to a folder
```bash
wget -P /path/to/directory "http://example.com"
```
2016-06-16 06:44:57 +00:00
##### Instruct curl to follow any redirect until it reaches the final destination:
```bash
curl -L google.com
```
2016-06-27 05:54:48 +00:00
2017-03-29 08:52:59 +00:00
## Random
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
##### Random generate password (e.g. generate 5 password each of length 13)
```bash
sudo apt install pwgen
pwgen 13 5
#sahcahS9dah4a xieXaiJaey7xa UuMeo0ma7eic9 Ahpah9see3zai acerae7Huigh7
```
##### Random pick 100 lines from a file
2016-07-01 10:52:19 +00:00
```bash
shuf -n 100 filename
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Random order (lucky draw)
2016-07-01 10:52:19 +00:00
```bash
for i in a b c d e; do echo $i; done| shuf
```
2016-06-27 05:54:48 +00:00
##### Echo series of random numbers between a range (e.g. shuffle numbers from 0-100, then pick 15 of them randomly)
2016-07-01 10:52:19 +00:00
```bash
shuf -i 0-100 -n 15
2016-07-01 10:52:19 +00:00
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Echo a random number
2016-07-01 10:52:19 +00:00
```bash
echo $RANDOM
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Random from 0-9
2016-07-01 10:52:19 +00:00
```bash
echo $((RANDOM % 10))
```
2016-06-27 05:54:48 +00:00
2017-06-09 08:53:36 +00:00
##### Random from 1-10
2016-07-01 10:52:19 +00:00
```bash
echo $(((RANDOM %10)+1))
```
2016-06-27 05:54:48 +00:00
2017-06-09 04:26:34 +00:00
## Xwindow
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2017-09-28 05:40:02 +00:00
X11 GUI applications! Here are some GUI tools for you if you get bored by the text-only environment.
2017-06-09 04:26:34 +00:00
2017-06-09 08:53:36 +00:00
##### Enable X11 forwarding,in order to use graphical application on servers
2017-06-09 04:26:34 +00:00
```bash
ssh -X user_name@ip_address
# or setting through xhost
# --> Install the following for Centos:
# xorg-x11-xauth
# xorg-x11-fonts-*
# xorg-x11-utils
```
2017-06-09 04:26:34 +00:00
2017-06-09 08:53:36 +00:00
##### Little xwindow tools
2017-06-09 04:26:34 +00:00
```bash
xclock
xeyes
2017-09-14 04:20:58 +00:00
xcowsay
2017-06-09 04:26:34 +00:00
```
2017-06-09 08:53:36 +00:00
##### Open pictures/images from ssh server
2017-06-09 04:26:34 +00:00
```bash
1. ssh -X user_name@ip_address
2. apt-get install eog
3. eog picture.png
```
##### Watch videos on server
```bash
1. ssh -X user_name@ip_address
2. sudo apt install mpv
3. mpv myvideo.mp4
```
##### Use gedit on server (GUI editor)
2017-06-09 04:26:34 +00:00
```bash
1. ssh -X user_name@ip_address
2. apt-get install gedit
3. gedit filename.txt
```
##### Open PDF file from ssh server
2017-06-09 04:26:34 +00:00
```bash
1. ssh -X user_name@ip_address
2. apt-get install evince
3. evince filename.pdf
```
##### Use google-chrome browser from ssh server
2017-06-09 04:26:34 +00:00
```bash
1. ssh -X user_name@ip_address
2. apt-get install libxss1 libappindicator1 libindicator7
3. wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
4. sudo apt-get install -f
5. dpkg -i google-chrome*.deb
6. google-chrome
```
## System
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2019-11-03 16:08:19 +00:00
##### Audit files to see who made changes to a file [RedHat based system only]
```bash
# To audit a directory recursively for changes (e.g. myproject)
auditctl -w /path/to/myproject/ -p wa
# If you delete a file name "VIPfile", the deletion is recorded in /var/log/audit/audit.log
sudo grep VIPfile /var/log/audit/audit.log
#type=PATH msg=audit(1581417313.678:113): item=1 name="VIPfile" inode=300115 dev=ca:01 mode=0100664 ouid=1000 ogid=1000 rdev=00:00 nametype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
```
##### Check out whether SELinux is enabled
```bash
sestatus
# SELinux status: enabled
# SELinuxfs mount: /sys/fs/selinux
# SELinux root directory: /etc/selinux
# Loaded policy name: targeted
# Current mode: enforcing
# Mode from config file: enforcing
# Policy MLS status: enabled
# Policy deny_unknown status: allowed
# Max kernel policy version: 31
```
2019-11-03 16:08:19 +00:00
##### Generate public key from private key
```bash
ssh-keygen -y -f ~/.ssh/id_rsa > ~/.ssh/id_rsa.pub
```
##### Copy your default public key to remote user
```bash
ssh-copy-id <user_name>@<server_IP>
# then you need to enter the password
# and next time you won't need to enter password when ssh to that user
```
##### Copy default public key to remote user using the required private key (e.g. use your mykey.pem key to copy your id_rsa.pub to the remote user)
```bash
# before you need to use mykey.pem to ssh to remote user.
ssh-copy-id -i ~/.ssh/id_rsa.pub -o "IdentityFile ~/Downloads/mykey.pem" <user_name>@<server_IP>
# now you don't need to use key to ssh to that user.
```
2020-01-09 16:58:15 +00:00
##### SSH Agent Forwarding
```bash
# To bring your key with you when ssh to serverA, then ssh to serverB from serverA using the key.
ssh-agent
ssh-add /path/to/mykey.pem
ssh -A <username>@<IP_of_serverA>
# Next you can ssh to serverB
ssh <username>@<IP_of_serverB>
```
##### Follow the most recent logs from service
```bash
journalctl -u <service_name> -f
```
##### Eliminate the zombie
```bash
# A zombie is already dead, so you cannot kill it. You can eliminate the zombie by killing its parent.
# First, find PID of the zombie
ps aux| grep 'Z'
# Next find the PID of zombie's parent
pstree -p -s <zombie_PID>
# Then you can kill its parent and you will notice the zombie is gone.
sudo kill 9 <parent_PID>
```
###### Show memory usage
```bash
free -c 10 -mhs 1
# print 10 times, at 1 second interval
```
##### Display CPU and IO statistics for devices and partitions.
```bash
# refresh every second
iostat -x -t 1
```
##### Display bandwidth usage on an network interface (e.g. enp175s0f0)
```bash
iftop -i enp175s0f0
```
##### Tell how long the system has been running and number of users
```bash
uptime
```
##### Check if it's root running
```bash
2019-02-21 09:08:23 +00:00
if [ "$EUID" -ne 0 ]; then
echo "Please run this as root"
exit 1
fi
```
##### Change shell of a user (e.g. bonnie)
```bash
chsh -s /bin/sh bonnie
# /etc/shells: valid login shells
```
##### Change root / fake root / jail (e.g. change root to newroot)
```bash
chroot /home/newroot /bin/bash
# To exit chroot
exit
```
##### Display file status (size; access, modify and change time, etc) of a file (e.g. filename.txt)
```bash
stat filename.txt
```
2019-02-21 09:08:23 +00:00
##### Snapshot of the current processes
2016-07-01 10:52:19 +00:00
```bash
ps aux
```
##### Display a tree of processes
```bash
pstree
```
##### Find maximum number of processes
```bash
cat /proc/sys/kernel/pid_max
```
##### Print or control the kernel ring buffer
```bash
dmesg
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Show IP address
2016-07-01 10:52:19 +00:00
```bash
$ip add show
# or
ifconfig
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
##### Print previous and current SysV runlevel
```bash
runlevel
# or
who -r
```
##### Change SysV runlevel (e.g. 5)
```bash
init 5
#or
telinit 5
```
##### Display all available services in all runlevels,
```bash
chkconfig --list
# update-rc.d equivalent to chkconfig in ubuntu
```
##### Check system version
2016-07-01 10:52:19 +00:00
```bash
cat /etc/*-release
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
##### Linux Programmer's Manuel: hier- description of the filesystem hierarchy
2016-07-01 10:52:19 +00:00
```bash
man hier
2016-07-01 10:52:19 +00:00
```
##### Control the systemd system and service manager
```bash
# e.g. check the status of cron service
systemctl status cron.service
# e.g. stop cron service
systemctl stop cron.service
```
##### List job
2017-12-29 02:14:00 +00:00
```bash
jobs -l
2017-12-29 02:14:00 +00:00
```
##### Run a program with modified priority (e.g. ./test.sh)
```bash
# nice value is adjustable from -20 (most favorable) to +19
# the nicer the application, the lower the priority
# Default niceness: 10; default priority: 80
nice -10 ./test.sh
```
##### Export PATH
2016-07-01 10:52:19 +00:00
```bash
export PATH=$PATH:~/path/you/want
2016-07-01 10:52:19 +00:00
```
2019-05-09 07:39:05 +00:00
##### Make file executable
2016-07-01 10:52:19 +00:00
```bash
chmod +x filename
# you can now ./filename to execute it
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Print system information
2016-07-01 10:52:19 +00:00
```bash
uname -a
# Check system hardware-platform (x86-64)
uname -i
2016-09-08 09:31:55 +00:00
```
##### Surf the net
2016-09-08 09:31:55 +00:00
```bash
links www.google.com
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
##### Add user, set passwd
2016-07-01 10:52:19 +00:00
```bash
useradd username
passwd username
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
##### Edit variable for bash, (e.g. displaying the whole path)
```bash
1. joe ~/.bash_profile
2. export PS1='\u@\h:\w\$'
# $PS1 is a variable that defines the makeup and style of the command prompt
3. source ~/.bash_profile
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
##### Edit environment setting (e.g. alias)
2016-07-01 10:52:19 +00:00
```bash
1. joe ~/.bash_profile
2. alias pd="pwd" //no more need to type that 'w'!
3. source ~/.bash_profile
2016-07-01 10:52:19 +00:00
```
##### Print all alias
```bash
alias -p
```
##### Unalias (e.g. after alias ls='ls --color=auto')
```bash
unalias ls
```
##### Set and unset shell options
```bash
# print all shell options
shopt
2019-05-17 07:59:33 +00:00
# to unset (or stop) alias
shopt -u expand_aliases
2019-05-17 07:59:33 +00:00
# to set (or start) alias
shopt -s expand_aliases
```
2016-06-14 08:03:32 +00:00
##### List environment variables (e.g. PATH)
2016-07-01 10:52:19 +00:00
```bash
2019-06-11 06:22:34 +00:00
echo $PATH
# list of directories separated by a colon
2016-07-01 10:52:19 +00:00
```
##### List all environment variables for current user
2016-07-01 10:52:19 +00:00
```bash
2019-05-16 03:52:54 +00:00
env
2016-07-01 10:52:19 +00:00
```
##### Unset environment variable (e.g. unset variable 'MYVAR')
```bash
unset MYVAR
```
2016-06-14 06:57:38 +00:00
##### Show partition format
2016-07-01 10:52:19 +00:00
```bash
lsblk
2016-07-01 10:52:19 +00:00
```
##### Inform the OS of partition table changes
```bash
partprobe
```
##### Soft link program to bin
2016-07-01 10:52:19 +00:00
```bash
ln -s /path/to/program /home/usr/bin
# must be the whole path to the program
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Show hexadecimal view of data
2016-07-01 10:52:19 +00:00
```bash
hexdump -C filename.class
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
##### Jump to different node
2016-07-01 10:52:19 +00:00
```bash
rsh node_name
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Check port (active internet connection)
2016-07-01 10:52:19 +00:00
```bash
netstat -tulpn
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Print resolved symbolic links or canonical file names
2016-07-01 10:52:19 +00:00
```bash
readlink filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
##### Find out the type of command and where it link to (e.g. python)
```bash
type python
# python is /usr/bin/python
# There are 5 different types, check using the 'type -f' flag
# 1. alias (shell alias)
# 2. function (shell function, type will also print the function body)
# 3. builtin (shell builtin)
# 4. file (disk file)
# 5. keyword (shell reserved word)
# You can also use `which`
which python
# /usr/bin/python
```
##### List all functions names
```bash
declare -F
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### List total size of a directory
2016-07-01 10:52:19 +00:00
```bash
du -hs .
# or
du -sb
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Copy directory with permission setting
2016-07-01 10:52:19 +00:00
```bash
cp -rp /path/to/directory
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Store current directory
2016-07-01 10:52:19 +00:00
```bash
pushd .
# then pop
popd
#or use dirs to display the list of currently remembered directories.
dirs -l
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
##### Show disk usage
```bash
df -h
# or
du -h
#or
du -sk /var/log/* |sort -rn |head -10
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Show all file system type
```bash
df -TH
```
##### Show current runlevel
2016-07-01 10:52:19 +00:00
```bash
runlevel
2016-07-01 10:52:19 +00:00
```
2016-06-14 07:55:08 +00:00
##### Switch runlevel
2016-07-01 10:52:19 +00:00
```bash
init 3
#or
telinit 3
2016-07-01 10:52:19 +00:00
```
##### Permanently modify runlevel
2016-07-01 10:52:19 +00:00
```bash
1. edit /etc/init/rc-sysinit.conf
2. env DEFAULT_RUNLEVEL=2
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
##### Become root
2016-07-01 10:52:19 +00:00
```bash
su
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Become somebody
2016-07-01 10:52:19 +00:00
```bash
su somebody
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Report user quotes on device
2016-07-01 10:52:19 +00:00
```bash
2019-05-11 04:13:49 +00:00
repquota -auvs
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
##### Get entries in a number of important databases
2016-07-01 10:52:19 +00:00
```bash
getent database_name
# (e.g. the 'passwd' database)
getent passwd
# list all user account (all local and LDAP)
# (e.g. fetch list of grop accounts)
getent group
# store in database 'group'
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Change owner of file
2016-07-01 10:52:19 +00:00
```bash
chown user_name filename
chown -R user_name /path/to/directory/
# chown user:group filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Mount and unmount
```bash
# e.g. Mount /dev/sdb to /home/test
mount /dev/sdb /home/test
# e.g. Unmount /home/test
umount /home/test
```
##### List current mount detail
2016-07-01 10:52:19 +00:00
```bash
mount
# or
df
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### List current usernames and user-numbers
2016-07-01 10:52:19 +00:00
```bash
cat /etc/passwd
2016-07-01 10:52:19 +00:00
```
##### Get all username
2016-07-01 10:52:19 +00:00
```bash
getent passwd| awk '{FS="[:]"; print $1}'
2016-07-01 10:52:19 +00:00
```
##### Show all users
2016-07-01 10:52:19 +00:00
```bash
compgen -u
2016-07-01 10:52:19 +00:00
```
2016-06-14 08:03:32 +00:00
##### Show all groups
2016-07-01 10:52:19 +00:00
```bash
compgen -g
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### Show group of user
2016-07-01 10:52:19 +00:00
```bash
group username
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
##### Show uid, gid, group of user
2016-07-01 10:52:19 +00:00
```bash
id username
# variable for UID
echo $UID
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
##### Check if it's root
2016-07-01 10:52:19 +00:00
```bash
if [ $(id -u) -ne 0 ];then
echo "You are not root!"
exit;
fi
# 'id -u' output 0 if it's not root
2016-07-01 10:52:19 +00:00
```
##### Find out CPU information
2016-07-01 10:52:19 +00:00
```bash
more /proc/cpuinfo
# or
lscpu
2017-10-27 03:13:54 +00:00
```
2016-06-16 06:44:57 +00:00
##### Set quota for user (e.g. disk soft limit: 120586240; hard limit: 125829120)
2016-07-01 10:52:19 +00:00
```bash
setquota username 120586240 125829120 0 0 /home
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
##### Show quota for user
2016-07-01 10:52:19 +00:00
```bash
quota -v username
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
##### Display current libraries from the cache
2019-05-14 09:27:13 +00:00
```bash
ldconfig -p
```
##### Print shared library dependencies (e.g. for 'ls')
```bash
ldd /bin/ls
```
##### Check user login
2016-07-01 10:52:19 +00:00
```bash
lastlog
2016-07-01 10:52:19 +00:00
```
##### Check last reboot history
```bash
last reboot
```
2016-06-16 06:44:57 +00:00
##### Edit path for all users
2016-07-01 10:52:19 +00:00
```bash
joe /etc/environment
# edit this file
2016-07-01 10:52:19 +00:00
```
##### Show and set user limit
```bash
ulimit -u
```
##### Which ports are listening for TCP connections from the network
2016-07-01 10:52:19 +00:00
```bash
nmap -sT -O localhost
2019-05-09 07:39:05 +00:00
#notice that some companies might not like you using nmap
2016-07-01 10:52:19 +00:00
```
##### Check if a host is up and scan for open ports, also skip host discovery.
#skips checking if the host is alive which may sometimes cause a false positive and stop the scan.
$ nmap google.com -Pn
# Example output:
# Starting Nmap 7.01 ( https://nmap.org ) at 2020-07-18 22:59 CST
# Nmap scan report for google.com (172.217.24.14)
# Host is up (0.013s latency).
# Other addresses for google.com (not scanned): 2404:6800:4008:802::200e
# rDNS record for 172.217.24.14: tsa01s07-in-f14.1e100.net
# Not shown: 998 filtered ports
# PORT STATE SERVICE
# 80/tcp open http
# 443/tcp open https
#
# Nmap done: 1 IP address (1 host up) scanned in 3.99 seconds
2016-06-16 06:44:57 +00:00
##### Print out number of cores/ processors
2016-08-25 11:57:18 +00:00
```bash
nproc --all
2016-08-25 11:57:18 +00:00
```
##### Check status of each core
```
1. top
2. press '1'
```
##### Show jobs and PID
2016-07-01 10:52:19 +00:00
```bash
jobs -l
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
##### List all running services
2016-07-01 10:52:19 +00:00
```bash
service --status-all
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:44:57 +00:00
##### Schedule shutdown server
2016-09-02 03:56:37 +00:00
```bash
shutdown -r +5 "Server will restart in 5 minutes. Please save your work."
2016-09-02 03:56:37 +00:00
```
##### Cancel scheduled shutdown
```bash
shutdown -c
```
2016-06-14 06:57:38 +00:00
2019-05-09 07:39:05 +00:00
##### Broadcast to all users
2016-09-23 03:15:49 +00:00
```bash
wall -n hihi
2016-09-23 03:15:49 +00:00
```
##### Kill all process of a user
2016-10-06 08:39:20 +00:00
```bash
pkill -U user_name
2016-10-06 08:39:20 +00:00
```
2018-05-28 02:15:48 +00:00
##### Kill all process of a program
```bash
kill -9 $(ps aux | grep 'program_name' | awk '{print $2}')
```
##### Set gedit preference on server
```
# You might have to install the following:
2016-10-06 08:39:20 +00:00
apt-get install libglib2.0-bin;
# or
yum install dconf dconf-editor;
yum install dbus dbus-x11;
# Check list
gsettings list-recursively
# Change some settings
gsettings set org.gnome.gedit.preferences.editor highlight-current-line true
gsettings set org.gnome.gedit.preferences.editor scheme 'cobalt'
gsettings set org.gnome.gedit.preferences.editor use-default-font false
gsettings set org.gnome.gedit.preferences.editor editor-font 'Cantarell Regular 12'
2017-01-05 07:41:58 +00:00
```
##### Add user to a group (e.g add user 'nice' to the group 'docker', so that he can run docker without sudo)
```bash
sudo gpasswd -a nice docker
```
2017-01-23 06:27:30 +00:00
##### Pip install python package without root
2017-01-23 06:26:35 +00:00
```bash
1. pip install --user package_name
2. You might need to export ~/.local/bin/ to PATH: export PATH=$PATH:~/.local/bin/
2017-01-23 06:26:35 +00:00
```
##### Removing old linux kernels (when /boot almost full...)
2017-01-23 06:26:35 +00:00
```bash
1. uname -a #check current kernel, which should NOT be removed
2. sudo apt-get purge linux-image-X.X.X-X-generic #replace old version
2017-01-23 06:26:35 +00:00
```
##### Change hostname
2017-02-16 12:25:13 +00:00
```bash
sudo hostname your-new-name
# if not working, do also:
hostnamectl set-hostname your-new-hostname
# then check with:
hostnamectl
# Or check /etc/hostname
2017-01-23 06:26:35 +00:00
# If still not working..., edit:
/etc/sysconfig/network
/etc/sysconfig/network-scripts/ifcfg-ensxxx
#add HOSTNAME="your-new-hostname"
```
##### List installed packages
```bash
apt list --installed
# or on Red Hat:
yum list installed
2017-06-16 09:01:53 +00:00
```
##### Check which file make the device busy on umount
2018-01-23 08:41:00 +00:00
```bash
lsof /mnt/dir
2018-01-23 08:41:00 +00:00
```
##### When sound not working
```bash
killall pulseaudio
# then press Alt-F2 and type in pulseaudio
```
##### When sound not working
```bash
killall pulseaudio
2017-08-16 06:28:50 +00:00
```
2019-05-10 09:10:05 +00:00
##### List information about SCSI devices
2017-08-16 06:20:57 +00:00
```bash
lsscsi
```
##### Tutorial for setting up your own DNS server
http://onceuponmine.blogspot.tw/2017/08/set-up-your-own-dns-server.html
2017-09-08 01:22:48 +00:00
##### Tutorial for creating a simple daemon
http://onceuponmine.blogspot.tw/2017/07/create-your-first-simple-daemon.html
2017-09-08 01:22:48 +00:00
##### Tutorial for using your gmail to send email
http://onceuponmine.blogspot.tw/2017/10/setting-up-msmtprc-and-use-your-gmail.html
##### Using telnet to test open ports, test if you can connect to a port (e.g 53) of a server (e.g 192.168.2.106)
2017-09-12 09:38:43 +00:00
```bash
telnet 192.168.2.106 53
```
##### Change network maximum transmission unit (mtu) (e.g. change to 9000)
2017-09-12 09:38:43 +00:00
```bash
ifconfig eth0 mtu 9000
```
##### Get pid of a running process (e.g python)
```bash
pidof python
# or
ps aux|grep python
```
##### NTP
2017-09-28 05:34:16 +00:00
```bash
# Start ntp:
ntpd
# Check ntp:
ntpq -p
```
##### Remove unnecessary files to clean your server
2017-10-24 05:46:22 +00:00
```bash
sudo apt-get autoremove
sudo apt-get clean
sudo rm -rf ~/.cache/thumbnails/*
# Remove old kernal:
sudo dpkg --list 'linux-image*'
sudo apt-get remove linux-image-OLDER_VERSION
```
2017-09-28 05:28:23 +00:00
##### Increase/ resize root partition (root partition is an LVM logical volume)
2017-09-28 05:28:23 +00:00
```bash
pvscan
lvextend -L +130G /dev/rhel/root -r
# Adding -r will grow filesystem after resizing the volume.
```
2017-09-28 10:10:55 +00:00
##### Create a UEFI Bootable USB drive (e.g. /dev/sdc1)
```bash
sudo dd if=~/path/to/isofile.iso of=/dev/sdc1 oflag=direct bs=1048576
```
##### Locate and remove a package
```bash
sudo dpkg -l | grep <package_name>
sudo dpkg --purge <package_name>
```
##### Create a ssh tunnel
2017-11-29 06:00:46 +00:00
```bash
ssh -f -L 9000:targetservername:8088 root@192.168.14.72 -N
#-f: run in background; -L: Listen; -N: do nothing
#the 9000 of your computer is now connected to the 8088 port of the targetservername through 192.168.14.72
#so that you can see the content of targetservername:8088 by entering localhost:9000 from your browser.
2017-11-29 06:00:46 +00:00
```
##### Get process ID of a process (e.g. sublime_text)
```bash
#pidof
pidof sublime_text
2019-05-09 07:39:05 +00:00
#pgrep, you don't have to type the whole program name
pgrep sublim
#pgrep, echo 1 if process found, echo 0 if no such process
pgrep -q sublime_text && echo 1 || echo 0
#top, takes longer time
top|grep sublime_text
```
2018-04-27 08:09:36 +00:00
##### Some benchmarking tools for your server
[aio-stress](https://openbenchmarking.org/test/pts/aio-stress) - AIO benchmark.
[bandwidth](https://zsmith.co/bandwidth.html) - memory bandwidth benchmark.
[bonnie++](https://www.coker.com.au/bonnie++/) - hard drive and file system performance benchmark.
[dbench](https://dbench.samba.org/) - generate I/O workloads to either a filesystem or to a networked CIFS or NFS server.
[dnsperf](https://www.dnsperf.com/) - authorative and recursing DNS servers.
[filebench](https://github.com/filebench/filebench) - model based file system workload generator.
[fio](https://linux.die.net/man/1/fio) - I/O benchmark.
[fs_mark](https://github.com/josefbacik/fs_mark) - synchronous/async file creation benchmark.
[httperf](https://github.com/httperf/httperf) - measure web server performance.
[interbench](https://github.com/ckolivas/interbench) - linux interactivity benchmark.
[ioblazer](https://labs.vmware.com/flings/ioblazer) - multi-platform storage stack micro-benchmark.
[iozone](http://www.iozone.org/) - filesystem benchmark.
[iperf3](https://iperf.fr/iperf-download.php) - measure TCP/UDP/SCTP performance.
[kcbench](https://github.com/knurd/kcbench) - kernel compile benchmark, compiles a kernel and measures the time it takes.
[lmbench](http://www.bitmover.com/lmbench/) - Suite of simple, portable benchmarks.
[netperf](https://github.com/HewlettPackard/netperf) - measure network performance, test unidirectional throughput, and end-to-end latency.
[netpipe](https://linux.die.net/man/1/netpipe) - network protocol independent performance evaluator.
[nfsometer](http://wiki.linux-nfs.org/wiki/index.php/NFSometer) - NFS performance framework.
[nuttcp](https://www.nuttcp.net/Welcome%20Page.html) - measure network performance.
[phoronix-test-suite](https://www.phoronix-test-suite.com/) - comprehensive automated testing and benchmarking platform.
[seeker](https://github.com/fidlej/seeker) - portable disk seek benchmark.
[siege](https://github.com/JoeDog/siege) - http load tester and benchmark.
[sockperf](https://github.com/Mellanox/sockperf) - network benchmarking utility over socket API.
[spew](https://linux.die.net/man/1/spew) - measures I/O performance and/or generates I/O load.
[stress](https://people.seas.harvard.edu/~apw/stress/) - workload generator for POSIX systems.
[sysbench](https://github.com/akopytov/sysbench) - scriptable database and system performance benchmark.
[tiobench](https://github.com/mkuoppal/tiobench) - threaded IO benchmark.
[unixbench](https://github.com/kdlucas/byte-unixbench) - the original BYTE UNIX benchmark suite, provide a basic indicator of the performance of a Unix-like system.
[wrk](https://github.com/wg/wrk) - HTTP benchmark.
2018-04-27 09:01:11 +00:00
##### Performance monitoring tool - sar
```bash
# installation
# It collects the data every 10 minutes and generate its report daily. crontab file (/etc/cron.d/sysstat) is responsible for collecting and generating reports.
yum install sysstat
systemctl start sysstat
systemctl enable sysstat
# show CPU utilization 5 times every 2 seconds.
sar 2 5
# show memory utilization 5 times every 2 seconds.
sar -r 2 5
# show paging statistics 5 times every 2 seconds.
sar -B 2 5
# To generate all network statistic:
sar -n ALL
# reading SAR log file using -f
2020-07-18 14:53:12 +00:00
sar -f /var/log/sa/sa31|tail
2020-07-18 14:53:12 +00:00
##### Reading from journal file
```bash
journalctl --file ./log/journal/a90c18f62af546ccba02fa3734f00a04/system.journal --since "2020-02-11 00:00:00"
```
2018-04-27 08:09:36 +00:00
##### Show a listing of last logged in users.
```bash
lastb
```
##### Show a listing of current logged in users, print information of them
```bash
who
```
2018-10-22 03:00:12 +00:00
##### Show who is logged on and what they are doing
```bash
w
```
##### Print the user names of users currently logged in to the current host.
```bash
users
```
##### Stop tailing a file on program terminate
```bash
tail -f --pid=<PID> filename.txt
2018-10-24 17:30:35 +00:00
# replace <PID> with the process ID of the program.
```
2018-04-27 08:09:36 +00:00
2018-11-16 08:34:57 +00:00
##### List all enabled services
```bash
systemctl list-unit-files|grep enabled
```
## Hardware
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2019-01-23 07:52:35 +00:00
##### Collect and summarize all hardware info of your machine
```bash
lshw -json >report.json
# Other options are: [ -html ] [ -short ] [ -xml ] [ -json ] [ -businfo ] [ -sanitize ] ,etc
```
##### Finding Out memory device detail
```bash
sudo dmidecode -t memory
2018-01-02 02:17:50 +00:00
```
2017-09-28 05:28:23 +00:00
##### Print detail of CPU hardware
```bash
dmidecode -t 4
# Type Information
# 0 BIOS
# 1 System
# 2 Base Board
# 3 Chassis
# 4 Processor
# 5 Memory Controller
# 6 Memory Module
# 7 Cache
# 8 Port Connector
# 9 System Slots
# 11 OEM Strings
# 13 BIOS Language
# 15 System Event Log
# 16 Physical Memory Array
# 17 Memory Device
# 18 32-bit Memory Error
# 19 Memory Array Mapped Address
# 20 Memory Device Mapped Address
# 21 Built-in Pointing Device
# 22 Portable Battery
# 23 System Reset
# 24 Hardware Security
# 25 System Power Controls
# 26 Voltage Probe
# 27 Cooling Device
# 28 Temperature Probe
# 29 Electrical Current Probe
# 30 Out-of-band Remote Access
# 31 Boot Integrity Services
# 32 System Boot
# 34 Management Device
# 35 Management Device Component
# 36 Management Device Threshold Data
# 37 Memory Channel
# 38 IPMI Device
# 39 Power Supply
```
##### Count the number of Segate hard disks
2018-01-17 05:36:56 +00:00
```bash
lsscsi|grep SEAGATE|wc -l
# or
sg_map -i -x|grep SEAGATE|wc -l
2018-01-17 05:36:56 +00:00
```
2018-05-31 02:55:20 +00:00
##### Get UUID of a disk (e.g. sdb)
```bash
blkid /dev/sdb
```
2019-11-04 15:01:14 +00:00
##### Generate an UUID
```bash
uuidgen
```
##### Print detail of all hard disks
2016-07-01 10:52:19 +00:00
```bash
lsblk -io KNAME,TYPE,MODEL,VENDOR,SIZE,ROTA
#where ROTA means rotational device / spinning hard disks (1 if true, 0 if false)
```
##### List all PCI (Peripheral Component Interconnect) devices
```bash
lspci
# List information about NIC
lspci | egrep -i --color 'network|ethernet'
```
##### List all USB devices
```bash
lsusb
```
##### Linux modules
```bash
# Show the status of modules in the Linux Kernel
lsmod
# Add and remove modules from the Linux Kernel
modprobe
# or
# Remove a module
rmmod
# Insert a module
insmod
```
##### Controlling IPMI-enabled devices (e.g. BMC)
```bash
# Remotely finding out power status of the server
ipmitool -U <bmc_username> -P <bmc_password> -I lanplus -H <bmc_ip_address> power status
# Remotely switching on server
ipmitool -U <bmc_username> -P <bmc_password> -I lanplus -H <bmc_ip_address> power on
# Turn on panel identify light (default 15s)
ipmitool chassis identify 255
# Found out server sensor temperature
ipmitool sensors |grep -i Temp
# Reset BMC
ipmitool bmc reset cold
# Prnt BMC network
ipmitool lan print 1
# Setting BMC network
ipmitool -I bmc lan set 1 ipaddr 192.168.0.55
ipmitool -I bmc lan set 1 netmask 255.255.255.0
ipmitool -I bmc lan set 1 defgw ipaddr 192.168.0.1
2016-07-01 10:52:19 +00:00
```
2018-12-18 08:34:40 +00:00
## Networking
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
2018-12-18 08:34:40 +00:00
##### Resolve a domain to IP address(es)
```bash
dig +short www.example.com
# or
host www.example.com
```
##### Send a ping with a limited TTL to 10 (TTL: Time-To-Live, which is the maximum number of hops that a packet can travel across the Internet before it gets discarded.)
```bash
ping 8.8.8.8 -t 10
```
##### Print the route packets trace to network host
```bash
traceroute google.com
```
2020-07-18 15:42:09 +00:00
##### Check connection to host (e.g. check connection to port 80 and 22 of google.com)
```bash
nc -vw5 google.com 80
# Connection to google.com 80 port [tcp/http] succeeded!
nc -vw5 google.com 22
# nc: connect to google.com port 22 (tcp) timed out: Operation now in progress
# nc: connect to google.com port 22 (tcp) failed: Network is unreachable
```
##### Look up website information (e.g. name server), searches for an object in a RFC 3912 database.
```bash
whois google.com
```
##### Show the SSL certificate of a domain
```bash
openssl s_client -showcerts -connect www.example.com:443
```
2018-12-18 08:34:40 +00:00
##### Display IP address
```bash
ip a
```
##### Display route table
```bash
ip r
```
2019-05-09 07:39:05 +00:00
##### Display ARP cache (ARP cache displays the MAC addresses of device in the same network that you have connected to)
2018-12-18 08:34:40 +00:00
```bash
ip n
```
##### Add transient IP addres (reset after reboot) (e.g. add 192.168.140.3/24 to device eno16777736)
```bash
ip address add 192.168.140.3/24 dev eno16777736
```
##### Persisting network configuration changes
```bash
sudo vi /etc/sysconfig/network-scripts/ifcfg-enoxxx
# then edit the fields: BOOTPROT, DEVICE, IPADDR, NETMASK, GATEWAY, DNS1 etc
```
##### Refresh NetworkManager
2018-12-18 08:34:40 +00:00
```bash
sudo nmcli c reload
```
##### Restart all interfaces
```bash
sudo systemctl restart network.service
```
##### To view hostname, OS, kernal, architecture at the same time!
```bash
hostnamectl
```
##### Set hostname (set all transient, static, pretty hostname at once)
```bash
hostnamectl set-hostname "mynode"
```
##### Find out the web server (e.g Nginx or Apache) of a website
2020-01-02 16:16:43 +00:00
```bash
curl -I http://example.com/
# HTTP/1.1 200 OK
# Server: nginx
# Date: Thu, 02 Jan 2020 07:01:07 GMT
# Content-Type: text/html
# Content-Length: 1119
# Connection: keep-alive
# Vary: Accept-Encoding
# Last-Modified: Mon, 09 Sep 2019 10:37:49 GMT
# ETag: "xxxxxx"
# Accept-Ranges: bytes
# Vary: Accept-Encoding
2020-01-02 16:16:43 +00:00
```
## Data wrangling
2019-05-16 15:53:13 +00:00
[[back to top](#handy-bash-one-liners)]
##### Print some words that start with a particular string (e.g. words start with 'phy')
```bash
# If file is not specified, the file /usr/share/dict/words is used.
look phy|head -n 10
# Phil
# Philadelphia
# Philadelphia's
# Philby
# Philby's
# Philip
# Philippe
# Philippe's
# Philippians
# Philippine
```
##### Repeat printing string n times (e.g. print 'hello world' five times)
```bash
printf 'hello world\n%.0s' {1..5}
```
##### Do not echo the trailing newline
```bash
username=`echo -n "bashoneliner"`
```
2019-04-17 07:57:50 +00:00
##### Copy a file to multiple files (e.g copy fileA to file(B-D))
```bash
tee <fileA fileB fileC fileD >/dev/null
```
##### Delete all non-printing characters
```bash
tr -dc '[:print:]' < filename
```
##### Remove newline / nextline
2016-07-01 10:52:19 +00:00
```bash
tr --delete '\n' <input.txt >output.txt
2016-07-01 10:52:19 +00:00
```
##### Replace newline
2016-07-01 10:52:19 +00:00
```bash
tr '\n' ' ' <filename
2016-07-01 10:52:19 +00:00
```
2016-06-14 06:57:38 +00:00
##### To uppercase/lowercase
2016-07-01 10:52:19 +00:00
```bash
tr /a-z/ /A-Z/
2016-07-01 10:52:19 +00:00
```
##### Translate a range of characters (e.g. substitute a-z into a)
```bash
echo 'something' |tr a-z a
# aaaaaaaaa
```
2016-06-16 06:52:03 +00:00
##### Compare two files (e.g. fileA, fileB)
2016-07-01 10:52:19 +00:00
```bash
diff fileA fileB
# a: added; d:delete; c:changed
# or
sdiff fileA fileB
# side-to-side merge of file differences
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Compare two files, strip trailing carriage return/ nextline (e.g. fileA, fileB)
```bash
diff fileA fileB --strip-trailing-cr
```
##### Number a file (e.g. fileA)
2016-07-01 10:52:19 +00:00
```bash
nl fileA
2016-06-16 06:52:03 +00:00
#or
nl -nrz fileA
# add leading zeros
#or
nl -w1 -s ' '
# making it simple, blank separate
```
##### Join two files field by field with tab (default join by the first column of both file, and default separator is space)
```bash
# fileA and fileB should have the same ordering of lines.
join -t '\t' fileA fileB
# Join using specified field (e.g. column 3 of fileA and column 5 of fileB)
join -1 3 -2 5 fileA fileB
```
##### Combine/ paste two or more files into columns (e.g. fileA, fileB, fileC)
2016-07-01 10:52:19 +00:00
```bash
paste fileA fileB fileC
# default tab separate
2016-07-01 10:52:19 +00:00
```
2019-10-19 16:15:38 +00:00
##### Group/combine rows into one row
```bash
# e.g.
# AAAA
# BBBB
# CCCC
# DDDD
cat filename|paste - -
# AAAABBBB
# CCCCDDDD
cat filename|paste - - - -
# AAAABBBBCCCCDDDD
```
##### Fastq to fasta (fastq and fasta are common file formats for bioinformatics sequence data)
```bash
cat file.fastq | paste - - - - | sed 's/^@/>/g'| cut -f1-2 | tr '\t' '\n' >file.fa
```
##### Reverse string
2016-07-01 10:52:19 +00:00
```bash
echo 12345| rev
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Generate sequence 1-10
```bash
seq 10
```
##### Find average of input list/file of integers
```bash
i=`wc -l filename|cut -d ' ' -f1`; cat filename| echo "scale=2;(`paste -sd+`)/"$i|bc
```
##### Generate all combination (e.g. 1,2)
```bash
echo {1,2}{1,2}
# 1 1, 1 2, 2 1, 2 2
```
##### Generate all combination (e.g. A,T,C,G)
```bash
set = {A,T,C,G}
group= 5
for ((i=0; i<$group; i++));do
repetition=$set$repetition;done
bash -c "echo "$repetition""
```
##### Read file content to variable
```bash
foo=$(<test1)
```
##### Echo size of variable
```bash
echo ${#foo}
```
##### Echo a tab
```bash
echo -e ' \t '
```
##### Split file into smaller file
```bash
# Split by line (e.g. 1000 lines/smallfile)
split -d -l 1000 largefile.txt
# Split by byte without breaking lines across files
split -C 10 largefile.txt
```
##### Create a large amount of dummy files (e.g 100000 files, 10 bytes each):
```bash
#1. Create a big file
dd if=/dev/zero of=bigfile bs=1 count=1000000
#2. Split the big file to 100000 10-bytes files
split -b 10 -a 10 bigfile
```
##### Rename all files (e.g. remove ABC from all .gz files)
```bash
rename 's/ABC//' *.gz
```
##### Remove file extension (e.g remove .gz from filename.gz)
```bash
basename filename.gz .gz
zcat filename.gz> $(basename filename.gz .gz).unpacked
```
##### Add file extension to all file(e.g add .txt)
```bash
rename s/$/.txt/ *
# You can use rename -n s/$/.txt/ * to check the result first, it will only print sth like this:
# rename(a, a.txt)
# rename(b, b.txt)
# rename(c, c.txt)
```
##### Squeeze repeat patterns (e.g. /t/t --> /t)
```bash
tr -s "/t" < filename
```
##### Do not print nextline with echo
```bash
echo -e 'text here \c'
```
##### View first 50 characters of file
```bash
head -c 50 file
```
##### Cut and get last column of a file
```bash
cat file|rev | cut -d/ -f1 | rev
```
##### Add one to variable/increment/ i++ a numeric variable (e.g. $var)
```bash
((var++))
# or
var=$((var+1))
```
##### Cut the last column
```bash
cat filename|rev|cut -f1|rev
```
##### Cat to a file
```bash
cat >myfile
let me add sth here
exit by control + c
^C
```
##### Clear the contents of a file (e.g. filename)
```bash
>filename
```
##### Append to file (e.g. hihi)
```bash
echo 'hihi' >>filename
```
##### Working with json data
```bash
#install the useful jq package
#sudo apt-get install jq
#e.g. to get all the values of the 'url' key, simply pipe the json to the following jq command(you can use .[]. to select inner json, i.e jq '.[].url')
cat file.json | jq '.url'
```
##### Decimal to Binary (e.g get binary of 5)
```bash
D2B=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1})
echo -e ${D2B[5]}
#00000101
echo -e ${D2B[255]}
#11111111
```
##### Wrap each input line to fit in specified width (e.g 4 integers per line)
```bash
echo "00110010101110001101" | fold -w4
# 0011
# 0010
# 1011
# 1000
# 1101
```
##### Sort a file by column and keep the original order
```bash
sort -k3,3 -s
```
##### Right align a column (right align the 2nd column)
```bash
cat file.txt|rev|column -t|rev
```
##### To both view and store the output
```bash
echo 'hihihihi' | tee outputfile.txt
# use '-a' with tee to append to file.
```
##### Show non-printing (Ctrl) characters with cat
```bash
cat -v filename
```
##### Convert tab to space
```bash
expand filename
```
##### Convert space to tab
```bash
unexpand filename
```
##### Display file in octal ( you can also use od to display hexadecimal, decimal, etc)
```bash
od filename
```
##### Reverse cat a file
```bash
tac filename
```
##### Reverse the result from `uniq -c`
```bash
while read a b; do yes $b |head -n $a ;done <test.txt
```
## Others
[[back to top](#handy-bash-one-liners)]
##### Describe the format and characteristics of image files.
```bash
identify myimage.png
#myimage.png PNG 1049x747 1049x747+0+0 8-bit sRGB 1.006MB 0.000u 0:00.000
```
##### Bash auto-complete (e.g. show options "now tomorrow never" when you press'tab' after typing "dothis")
[More examples](https://iridakos.com/tutorials/2018/03/01/bash-programmable-completion-tutorial.html)
```bash
complete -W "now tomorrow never" dothis
# ~$ dothis
# never now tomorrow
# press 'tab' again to auto-complete after typing 'n' or 't'
```
##### Displays a calendar
```bash
# print the current month, today will be highlighted.
cal
# October 2019
# Su Mo Tu We Th Fr Sa
# 1 2 3 4 5
# 6 7 8 9 10 11 12
# 13 14 15 16 17 18 19
# 20 21 22 23 24 25 26
# 27 28 29 30 31
# only display November
cal -m 11
```
##### Forces applications to use the default language for output
```bash
export LC_ALL=C
# to revert:
unset LC_ALL
```
##### Encode strings as Base64 strings
```bash
echo test|base64
#dGVzdAo=
```
##### Get parent directory of current directory
```bash
dirname `pwd`
```
##### Read .gz file without extracting
2016-07-01 10:52:19 +00:00
```bash
zmore filename
# or
zless filename
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Run command in background, output error file
2016-07-01 10:52:19 +00:00
```bash
some_commands &>log &
# or
some_commands 2>log &
2016-06-16 06:52:03 +00:00
# or
some_commands 2>&1| tee logfile
# or
some_commands |& tee logfile
2016-06-16 06:52:03 +00:00
# or
some_commands 2>&1 >>outfile
#0: standard input; 1: standard output; 2: standard error
2016-07-01 10:52:19 +00:00
```
##### Run multiple commands in background
```bash
# run sequentially
(sleep 2; sleep 3) &
# run parallelly
sleep 2 & sleep 3 &
```
##### Run process even when logout (immune to hangups, with output to a non-tty)
```bash
# e.g. Run myscript.sh even when log out.
nohup bash myscript.sh
```
##### Send mail
2016-07-01 10:52:19 +00:00
```bash
echo 'heres the content'| mail -a /path/to/attach_file.txt -s 'mail.subject' me@gmail.com
# use -a flag to set send from (-a "From: some@mail.tld")
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
2019-05-10 09:10:05 +00:00
##### Convert .xls to csv
2016-07-01 10:52:19 +00:00
```bash
xls2csv filename
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Make BEEP sound
2016-07-01 10:52:19 +00:00
```bash
speaker-test -t sine -f 1000 -l1
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Set beep duration
2016-07-01 10:52:19 +00:00
```bash
(speaker-test -t sine -f 1000) & pid=$!;sleep 0.1s;kill -9 $pid
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Editing your history
2016-07-01 10:52:19 +00:00
```bash
history -w
vi ~/.bash_history
history -r
2016-06-16 06:52:03 +00:00
#or
history -d [line_number]
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Interacting with history
```bash
# list 5 previous command (similar to `history |tail -n 5` but wont print the history command itself)
fc -l -5
```
##### Delete current bash command
2016-07-01 10:52:19 +00:00
```bash
Ctrl+U
# or
Ctrl+C
# or
Alt+Shift+#
# to make it to history
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Add something to history (e.g. "addmetohistory")
2016-07-01 10:52:19 +00:00
```bash
# addmetodistory
# just add a "#" before~~
2016-07-01 10:52:19 +00:00
```
2016-06-16 06:52:03 +00:00
##### Get last history/record filename
```bash
head !$
```
##### Clean screen
```bash
clear
# or simply Ctrl+l
```
##### Backup with rsync
2016-07-01 10:52:19 +00:00
```bash
rsync -av filename filename.bak
rsync -av directory directory.bak
rsync -av --ignore_existing directory/ directory.bak
rsync -av --update directory directory.bak
rsync -av directory user@ip_address:/path/to/directory.bak
# skip files that are newer on receiver (i prefer this one!)
2016-07-01 10:52:19 +00:00
```
##### Make all directories at one time!
2016-07-01 10:52:19 +00:00
```bash
mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat}
# -p: make parent directory
# this will create project/doc/html/; project/doc/info; project/lib/ext ,etc
2016-07-01 10:52:19 +00:00
```
##### Run command only if another command returns zero exit status (well done)
2016-07-01 10:52:19 +00:00
```bash
cd tmp/ && tar xvf ~/a.tar
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
##### Run command only if another command returns non-zero exit status (not finish)
2016-07-01 10:52:19 +00:00
```bash
cd tmp/a/b/c ||mkdir -p tmp/a/b/c
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
##### Use backslash "\" to break long command
2016-07-01 10:52:19 +00:00
```bash
cd tmp/a/b/c \
> || \
>mkdir -p tmp/a/b/c
2016-07-01 10:52:19 +00:00
```
##### List file type of file (e.g. /tmp/)
2016-07-01 10:52:19 +00:00
```bash
file /tmp/
# tmp/: directory
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
##### Writing Bash script ('#!'' is called shebang )
2016-07-01 10:52:19 +00:00
```bash
#!/bin/bash
file=${1#*.}
# remove string before a "."
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
##### Python simple HTTP Server
2016-07-01 10:52:19 +00:00
```bash
python -m SimpleHTTPServer
# or when using python3:
python3 -m http.server
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
##### Read user input
2016-07-01 10:52:19 +00:00
```bash
read input
echo $input
2016-07-01 10:52:19 +00:00
```
2016-06-17 10:55:54 +00:00
##### Array
2016-09-10 09:46:30 +00:00
```bash
declare -a array=()
# or
declare array=()
# or associative array
declare -A array=()
2016-09-10 09:46:30 +00:00
```
##### Send a directory
2017-01-06 04:32:31 +00:00
```bash
scp -r directoryname user@ip:/path/to/send
2017-01-06 04:32:31 +00:00
```
2016-09-10 09:46:30 +00:00
##### Fork bomb
```bash
# Don't try this at home!
# It is a function that calls itself twice every call until you run out of system resources.
# A '# ' is added in front for safety reason, remove it when seriously you are testing it.
# :(){:|:&};:
```
##### Use the last argument
```bash
!$
```
2017-01-06 08:16:43 +00:00
##### Check last exit code
```bash
echo $?
```
2017-08-28 01:51:29 +00:00
##### Extract .xf
```
unxz filename.tar.xz
# then
tar -xf filename.tar
2017-08-23 09:17:47 +00:00
```
##### Unzip tar.bz2 file (e.g. file.tar.bz2)
2017-08-23 09:17:47 +00:00
```bash
tar xvfj file.tar.bz2
2017-08-23 09:17:47 +00:00
```
##### Unzip tar.xz file (e.g. file.tar.xz)
```bash
unxz file.tar.xz
tar xopf file.tar
```
##### Extract to a path
```bash
tar xvf -C /path/to/directory filename.gz
```
##### Output a y/n repeatedly until killed
2017-09-20 07:33:16 +00:00
```bash
# 'y':
yes
# or 'n':
yes n
# or 'anything':
yes anything
# For example:
2017-10-06 07:48:47 +00:00
```bash
yes | rm -r large_directory
```
2020-01-09 17:05:47 +00:00
##### Create large dummy file of certain size instantly (e.g. 10GiB)
```bash
fallocate -l 10G 10Gigfile
```
##### Create dummy file of certain size (e.g. 200mb)
2018-02-07 10:23:31 +00:00
```bash
dd if=/dev/zero of=//dev/shm/200m bs=1024k count=200
# or
dd if=/dev/zero of=//dev/shm/200m bs=1M count=200
2017-10-06 07:48:47 +00:00
# Standard output:
# 200+0 records in
# 200+0 records out
# 209715200 bytes (210 MB) copied, 0.0955679 s, 2.2 GB/s
```
2017-09-01 02:46:34 +00:00
##### Keep /repeatedly executing the same command (e.g Repeat 'wc -l filename' every 1 second)
```bash
watch -n 1 wc -l filename
```
2017-09-20 07:33:16 +00:00
##### Print commands and their arguments when execute (e.g. echo `expr 10 + 20 `)
2017-10-19 05:38:40 +00:00
```bash
set -x; echo `expr 10 + 20 `
```
##### Print some meaningful sentences to you (install fortune first)
```bash
fortune
```
##### Colorful (and useful) version of top (install htop first)
```bash
htop
```
##### Press any key to continue
```bash
read -rsp $'Press any key to continue...\n' -n1 key
```
##### Run sql-like command on files from terminal
2017-10-31 02:15:07 +00:00
```bash
# download:
# https://github.com/harelba/q
# example:
q -d "," "select c3,c4,c5 from /path/to/file.txt where c3='foo' and c5='boo'"
```
##### Using Screen for multiple terminal sessions
2017-10-31 02:15:07 +00:00
```bash
2019-05-10 08:20:40 +00:00
# Create session and attach:
screen
# Create detached session foo:
screen -S foo -d -m
# Detached session foo:
screen: ^a^d
# List sessions:
screen -ls
# Attach last session:
screen -r
2019-05-10 08:20:40 +00:00
# Attach to session foo:
screen -r foo
2019-05-10 08:20:40 +00:00
# Kill session foo:
screen -r foo -X quit
2019-05-10 08:20:40 +00:00
# Scroll:
Hit your screen prefix combination (C-a / control+A), then hit Escape.
Move up/down with the arrow keys (↑ and ↓).
# Redirect output of an already running process in Screen:
(C-a / control+A), then hit 'H'
2018-05-31 02:55:20 +00:00
# Store screen output for Screen:
Ctrl+A, Shift+H
2019-05-10 08:20:40 +00:00
# You will then find a screen.log file under current directory.
```
2018-09-13 08:26:09 +00:00
##### Using Tmux for multiple terminal sessions
```bash
2019-05-10 08:20:40 +00:00
# Create session and attach:
tmux
# Attach to session foo:
tmux attach -t foo
# Detached session foo:
2019-05-10 08:20:40 +00:00
^bd
# List sessions:
2019-05-10 08:20:40 +00:00
tmux ls
# Attach last session:
2019-05-10 08:20:40 +00:00
tmux attach
# Kill session foo:
tmux kill-session -t foo
# Create detached session foo:
2019-05-10 08:20:40 +00:00
tmux new -s foo -d
# Send command to all panes in tmux:
2018-05-04 09:49:07 +00:00
Ctrl-B
:setw synchronize-panes
2018-05-04 09:46:44 +00:00
2019-05-10 08:20:40 +00:00
# Some tmux pane control commands:
Ctrl-B
2018-05-04 09:46:44 +00:00
# Panes (splits), Press Ctrl+B, then input the following symbol:
# % horizontal split
# " vertical split
# o swap panes
# q show pane numbers
# x kill pane
# space - toggle between layouts
2018-11-08 07:00:32 +00:00
# Distribute Vertically (rows):
2018-11-08 07:00:32 +00:00
select-layout even-vertical
# or
Ctrl+b, Alt+2
# Distribute horizontally (columns):
2018-11-08 07:00:32 +00:00
select-layout even-horizontal
# or
Ctrl+b, Alt+1
2019-05-10 08:20:40 +00:00
# Scroll
Ctrl-b then \[ then you can use your normal navigation keys to scroll around.
Press q to quit scroll mode.
```
2019-05-10 08:20:40 +00:00
2019-05-10 09:10:05 +00:00
##### Pass password to ssh
```bash
sshpass -p mypassword ssh root@10.102.14.88 "df -h"
```
2019-05-10 09:10:05 +00:00
##### Wait for a pid (job) to complete
```bash
wait %1
# or
wait $PID
wait ${!}
#wait ${!} to wait till the last background process ($! is the PID of the last background process)
```
2019-05-10 09:10:05 +00:00
##### Convert pdf to txt
```bash
sudo apt-get install poppler-utils
pdftotext example.pdf example.txt
2018-01-10 01:48:50 +00:00
```
2019-05-10 09:10:05 +00:00
##### List only directory
```bash
2020-07-19 12:43:22 +00:00
ls -d */
```
##### Capture/record/save terminal output (capture everything you type and output)
2018-01-25 03:27:35 +00:00
```bash
script output.txt
# start using terminal
# to logout the screen session (stop saving the contents), type exit.
```
##### List contents of directories in a tree-like format.
2017-12-14 01:43:55 +00:00
```bash
tree
# go to the directory you want to list, and type tree (sudo apt-get install tree)
# output:
# home/
# └── project
# ├── 1
# ├── 2
# ├── 3
# ├── 4
# └── 5
#
# set level directories deep (e.g. level 1)
tree -L 1
# home/
# └── project
2017-12-14 01:43:55 +00:00
```
2019-05-10 09:10:05 +00:00
##### Set up virtualenv(sandbox) for python
```bash
2019-05-10 08:20:40 +00:00
# 1. install virtualenv.
sudo apt-get install virtualenv
2019-05-10 08:20:40 +00:00
# 2. Create a directory (name it .venv or whatever name your want) for your new shiny isolated environment.
virtualenv .venv
2019-05-10 08:20:40 +00:00
# 3. source virtual bin
source .venv/bin/activate
2019-05-10 08:20:40 +00:00
# 4. you can check check if you are now inside a sandbox.
type pip
2019-05-10 08:20:40 +00:00
# 5. Now you can install your pip package, here requirements.txt is simply a txt file containing all the packages you want. (e.g tornado==4.5.3).
pip install -r requirements.txt
2018-04-10 06:38:09 +00:00
```
2019-02-22 07:26:39 +00:00
> More coming!!