Linux 常用指令 (updating)

2016-06-17 John Tech 863

Other

Remove all files exclude specific file

find . ! -name 'file.txt' -type f -exec rm -f {} +

find . ! -name '2020*' -exec rm -f {} +

Linux file sharing

sudo npm install -g http-server
http-server -o

Filter string and column

| awk '/pattern/{ print $0 }'
# $0 all, $1 column 1, $2 column 2, ...

Download GitHub release

wget --no-check-certificate --content-disposition https://github.com/joyent/node/tarball/v0.7.1

# --no-check-cerftificate was necessary for me to have wget not puke about https
curl -LJO https://github.com/joyent/node/tarball/v0.7.1

# Just download file
curl -OJ https://xxx.com/xxx.zip

Install mkpasswd

yum install expect

Install

yum install httpd-tools

Generate self-signed SSL cert

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mykey.key -out mycert.pem

Check if script not existed and run

ps aux | grep "app/main.py" | grep -v grep
if [ $? != 0 ]; then
  python3 /srv/app/main.py
fi

Check if process existed

ps aux | grep "your_running_command" | grep -v grep

Append string to file

echo $'first line\nsecond line\nthirdline' >> foo

Get http status code

curl -I http://www.example.org
curl -v http://www.example.org    # -v == --verbose

Local mail box

/var/mail/{USER}
/var/spool/mail/{USER}

While loop infinity

while true; do echo hi; sleep 5; done

Print log to file by datetime

ls -al > output_$(date +"%m_%d_%Y")

Check the collected
yum 列出所有 package 可裝版本

yum list {{package}} --showduplicates | sort -r

yum 搜尋 package by keyword

yum search {{keyword}}

Install nodeJS

#delete if needed

sudo apt-get –purge remove node

sudo apt-get –purge remove nodejs

sudo apt-get install nodejs

#Use 'node' command rather than 'nodejs'

update-alternatives –install /usr/bin/node node /usr/bin/nodejs 10

Install pip3

sudo apt-get install python3-pip

Install htop

sudo yum install epel-release
sudo yum update

sudo yum install htop

Jupyter from .ipynb to .py

On the command line, you can use nbconvert:

$ jupyter nbconvert –to script [YOUR_NOTEBOOK].ipynb

As a bit of a hack, you can even call the above command in an IPython notebook by pre-pending ! (used for any command line argument). Inside a notebook:

!jupyter nbconvert –to script config_template.ipynb

發 url request and show the time

curl -o /dev/null -s -w "%{time_connect} + %{time_starttransfer} = %{time_total}\n" <YOUR_URI>

"No package top available.

Error: Nothing to do"

# htop seems to be available in the EPEL repo. Try yum install epel-release, then yum install htop again.

yum install epel-release

About compressing: Zip, Unzip

# zip with hidden file (.htaccess)
sudo find . -type f -name '.*' -print | sudo zip newZipFile.zip -@


# zip all in folder
sudo zip -r  {ZIPNAME}.zip {FOLDER_PATH1} {FOLDER_PATH2} {FOLDER_PATH3} ...


# unzip to specific folder
sudo unzip {ZIPNAME}.zip -d {FOLDER_PATH}

# browser into zip file
unzip -l {{zipfile}}
less {{zipfile}}
vim {{zipfile}}

Install ssh for ubuntu

sudo apt-get install openssh-server

# Enable/Disable root
vi /etc/ssh/sshd_config
--
PermitRootLogin Yes
--

# (optional) Set allowed IP
vi /etc/hosts.allow
--
sshd:192.168.1.88:allow
--
vi /etc/hosts.deny
--
sshd:all:deny
--
-> then restart


# (optional) restart
sudo /etc/init.d/ssh restart

Install Remote GUI for Ubuntu

# xfce4
sudo apt-get install xfce4
## additional goodies
sudo apt-get install xfce4-goodies

Make shortcut to command

alias lc=long-command

About System

Find regex string not include by awk

XXXstring | awk '$0 !~/domain/{print $0}'

Cron Job will lost PATH

# Set the path back again
export PATH={$PATH}

Add user to root group

$ sudo usermod -a -G root john

Remote by ssh with rsa key (private/public/pem)

# Generate ssh key
$ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
$ssh-keygen -t rsa

# Use public(.pub) rsa key to login 
# Put public key into `authorized_keys` (each key start with "ssh-rsa")
$vi ~/.ssh/authorized_keys
--- authorized_keys ---
ssh-rsa XXXXX
ssh-rsa XXXXX
------


# Restrict logining with key only
$sudo vi /etc/ssh/sshd_config   
--- sshd_config ---
PasswordAuthentication no
------

# Restart to apply settings (choose one)
$sudo service ssh restart 
$sudo systemctl restart ssh

Get current IP by web service

import commands
commands.getstatusoutput("wget -qO- http://whatismyip.host/ | grep -oP '(?<=ipaddress\"\>)[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+(?=\<\/p\>)'")

Show swap

sudo swapon –show

Lock account

passwd -l testuser          # lock
passwd --status testuser    # Verify

Switch users

su -   # Switch to root
sudo su   # Run as sudoer

檢查 users and groups

cat /etc/passwd

檢查網路連接狀況

ss -l

檢查 OS 版本 version

uname -a

hostnamectl

cat /etc/os-release

登入成 root

su -

Show all environment variables

env

檢查所有 disk

lsblk
sudo fdisk -l
sudo parted -l
sudo blkid

檢查 disk info and usage

df -Th

檢查 disk usage per folder

    sudo du -hs *
    
    # all current folder

設定 proxy

vi ~/.bashrc
export http_proxy="http://{USER}:{PASSWORD}@{PROXY_SERVER}:{PORT}"
export https_proxy="https://{USER}:{PASSWORD}@{PROXY_SERVER}:{PORT}"
export ftp_proxy="http://{USER}:{PASSWORD}@{PROXY_SERVER}:{PORT}"

for yum

$ sudo vim /etc/yum.conf
proxy={{http://IP:port}}

Ref: http://marcustsai.blogspot.com/2012/07/linux-proxy.html

for docker

1. Build service file.
$ sudo mkdir -p /etc/systemd/system/docker.service.d

HTTP:
$ sudo vim /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="http_proxy={{http://IP:port}}/"

HTTPS:
$ sudo vim /etc/systemd/system/docker.service.d/https-proxy.conf
[Service]
Environment="https_proxy={{http://IP:port}}/"

sudo systemctl daemon-reload
sudo systemctl restart docker
systemctl show --property=Environment docker

Ref: https://github.com/moby/moby/issues/32270

The best 進程監視器

apt-get install htop

If you just to ask "politely" a process to quit use 3 SIGQUIT.

If you want to make sure the process quits use 9 SIGKILL.

重新連結突然斷線的 session (如果有被自動存的話)

yum load-transaction /tmp/yum_save_tx.2018-02-12.08-39.gOr6vZ.yumtx

檢查 memory

cat /proc/meminfo

free -h

檢查 listening port

sudo netstat -tulpn

#列出 occupy port 的 ap
sudo fuser 80/tcp

Clear memory cache

要強制手動釋放或清除Linux中的Cache Memory可以使用下面的指令

echo 3 > /proc/sys/vm/drop_caches

(3 是指釋放pagecache、dentries與inodes，也就是釋放所有的cache)

其他也可以下：

#釋放pagecache

echo 1 > /proc/sys/vm/drop_caches

#釋放dentries與inodes

echo 2 > /proc/sys/vm/drop_caches

持續看著 log 更新

less -S +F your.log

tail -f your.log (like advance cat)

列出所有系統變數

printenv
env

設定開機預載入磁碟區

vi /etc/fstab

加入:

/dev/xvdb /data ext4 defaults 0 0

Customize creating SWAP partition

1. 建立適當大小的檔案(ex. 512MB)

sudo dd if=/dev/zero of=/etc/swap.img bs=1M count=512

2. 將檔案格式化為swap格式

sudo mkswap /etc/swap.img

3. 啟用該swap

sudo swapon /etc/swap.img

如果希望該檔案在每次開機都可以自動掛載，可以透過設定/etc/fstab檔案來讓swap可以自動掛載

vi /etc/fstab

加入:

/etc/swap.img swap swap defaults 0 0

nohup command :

nohup does not disconnect from terminal, it makes your app ignore SIGHUP, and (usually) redirects stdout/stderr. Run jobs in your terminal. It may just be a background job, and you can bring it back with fg. I don't know how to get stderr/stdout once you redirect it.

多任務管理

screen command:

TL;DR;

screen -S paportal -c /var/socketlog/pa.screenrc -L

Ctrl + a, Ctrl + d

screen => 開啟一個新 terminal socket

screen -L => 開啟新 screen + 畫面紀錄功能

screen -S [Socket Name] => 自訂 Socket 名稱

echo $STY => 顯示當前 screen

Ctrl + a, Ctrl + H => 從現有 screen + 畫面紀錄功能

screen -ls(-list) => 看所有 jobs

screen -r {optional: id or keyword} => 喚回 screen

Ctrl + a, Ctrl + d => detach 切到背景執行

Ctrl + a, :, sessionname [nameyouwant] => 修改 screen 名稱

screen -S {id or keyword} -X quit => 刪除指定 screen

PS. .screenrc sample:

https://gist.github.com/MagicJohnJang/aa4cc892ffe8c23df06f396356560d79
背景工作 by Jobs

將工作切到背景執行+暫停

[Ctrl] + z

將工作從背景喚回執行

fg [jobnumber]

將任務放到後台中去處理

bg [jobnumber]

查看後台的工作狀態

jobs -l

管理後台的任務

kill [pid]

& 將指令丟到後台中去執行

變更群組

chgrp [-R] [GroupName] [dirname/filename]

ex.

chgrp -R pls-rd-rw /data

變更擁有者

chown [-R] [帳號名稱] [檔案或目錄]

變更權限

chmod u=rwx,go+rx filename > (1)user (2)group (3)others (4)all ; +(加入) -(除去) =(設定)

chmod [-R] 754 [檔案或目錄] > [4+2+1][4+0+1][4+0+0]

ex.

sudo chgrp -R pls-rd-rw /data/notebooks/

sudo chmod -R g+rwx /data/notebooks/

Check occupied port (like 80 port) and kill it

# Choice one

netstat -an | grep ":80"

lsof -i tcp:80

# And Kill

kill -9 <PID> #where <PID> is the process id returned by lsof

Check remote port connection

telnet {ip/hostname} {port}
curl {ip/hostname}:{port}
wget -qS -O- {ip/hostname}:{port}

尋找資料夾或檔案

find / -type d -name 'YOUR_FOLDER_NAME'

尋找資料夾或檔案並計算容量

sudo du -hs $(sudo find / -type d -name "{folderName}")

Cron Job log

cat /var/log/syslog
01 14 * * * /home/joe/myscript >> /home/log/myscript.log 2>&1   # stderr to stdout

Cron Job setting

crontab -l
crontab -e

#/var/spool/cron/crontabs

Keep typing repeatedly

watch -n 5 {your_command}

curl 測試 connection

# test https
curl --insecure -v https://www.google.com 2>&1 | awk 'BEGIN { cert=0 } /^\* Server certificate:/ { cert=1 } /^\*/ { if (cert) print }'

Check server name by IP

nslookup {IP}

Prompt any key to close the screen (readline, read input)

read -n 1 -s -r -p "Press any key to close"

# input to varible
read varname

Jupyter

Enable interaction widgets

pip3 install ipywidgets
jupyter nbextension enable --py widgetsnbextension

Shortcut Key:

Command Mode (press Esc to enable)

Enter	enter edit mode
Shift-Enter	run cell, select below
Ctrl-Enter	run cell
Alt-Enter	run cell, insert below
Y	to code
M	to markdown
R	to raw
1	to heading 1
2,3,4,5,6	to heading 2,3,4,5,6
Up/K	select cell above
Down/J	select cell below
A/B	insert cell above/below
X	cut selected cell
C	copy selected cell
Shift-V	paste cell above
V	paste cell below
Z	undo last cell deletion
D,D	delete selected cell
Shift-M	merge cell below
Ctrl-S	Save and Checkpoint
L	toggle line numbers
O	toggle output
Shift-O	toggle output scrolling
Esc	close pager
H	show keyboard shortcut help dialog
I,I	interrupt kernel
0,0	restart kernel
Space	scroll down
Shift-Space	scroll up
Shift	ignore

Edit Mode (press Enter to enable)

Tab	code completion or indent
Shift-Tab	tooltip
Ctrl-]	indent
Ctrl-[	dedent
Ctrl-A	select all
Ctrl-Z	undo
Ctrl-Shift-Z	redo
Ctrl-Y	redo
Ctrl-Home	go to cell start
Ctrl-Up	go to cell start
Ctrl-End	go to cell end
Ctrl-Down	go to cell end
Ctrl-Left	go one word left
Ctrl-Right	go one word right
Ctrl-Backspace	delete word before
Ctrl-Delete	delete word after
Esc	command mode
Ctrl-M	command mode
Shift-Enter	run cell, select below
Ctrl-Enter	run cell
Alt-Enter	run cell, insert below
Ctrl-Shift-Subtract	split cell
Ctrl-Shift–	split cell
Ctrl-S	Save and Checkpoint
Up	move cursor up or previous cell
Down	move cursor down or next cell
Ctrl-/	toggle comment on current or selected lines