Basic Linux

Basic Linux

Introduction

Scope of this Manual

This manual provides a brief introduction how to get started using powerful Linux command-line utilities.

 

How to Get Access?

  • Install your preferred GNU/Linux distribution on your local machine (not required!!!)
  • Users at UC Riverside can apply for an account on our Linux clusters by sending an account request to Thomas Girke (tgirke@citrus.ucr.edu).

 

 

 

Logging-In

 

Mac or LINUX

  • Open terminal and type: 
    • ssh -X <your_username>@<host_name>

 

host name is the remote server's domain name (e.g. server123.ucr.edu)
You will be asked to enter the password, simply type it and press enter.

Windows

  1. Open Putty and select ssh. Download PuTTY if you do not have it.

  2. Provide the host name (the remote server's domain name) and session name 

    • hostname:

      server123.ucr.edu
  3. Enter your identity information

    • username:

      your username

      password:

      your password 

      Nothing will show-up,
      simply type the password and press enter.

  4. Setup for graphics emulation. Download and install Xming if you do not have it.

  5. Use WinSCP or FileZilla for file exchange.Download and install WinSCP or FileZilla if you do not have it.

Table of Contents

 

 


Change Password

Please see https://manuals.bioinformatics.ucr.edu/home/hpc#TOC-Password-Change

 

Why GNU/Linux?

  • Software costs $0
  • Advanced Multitasking 
  • Remote tasking ("real networking")
  • Multiuser 
  • Easy access to programming languages, databases, open-source projects 
  • Software freedoms 
    • Free to use for any purpose 
    • Free to study the source code
    • Free to share 
    • Free to modify
  • No dependence on vendors
  • Better performance
  • More up-to-date
  • Many more reasons ...

 

Basics

 

Command-Line Syntax for this Manual

  • Remember the UNIX/LINUX command line is case sensitive!
  • All commands in this manual are printed in gray code boxes.
  • Commands given in red are considered more important for beginners than commands given in black.
  • The hash (pound) sign "#" indicates end of a command and the start of a comment.
  • The notation <...> refers to variables and file names that need to be specified by the user. The symbols < and > need to be excluded.

 

Orientation

Viewing and changing the present working directory:

pwd               # Get full path of the present working directory (same as "echo $HOME")

ls                # Content of pwd
ls -l             # Similar as ls, but provides additional info on files and directories
ls -a             # Includes hidden files (.name) as well
ls -R             # Lists subdirectories recursively
ls -t             # Lists files in chronological order

cd <dir_name>     # Switches into specified directory.
cd                # Brings you to the highest level of your home directory.
cd ..             # Moves one directory up
cd ../../         # Moves two directories up (and so on)
cd -                              #  Go back to you were previously (before the last directory change)
 

The tilde symbol (~) gets interpreted as the path to your home directory. This will happen anywhere on the command line:

echo ~            # View the full (complete) path of your home
find ~            # List all your files (including everything in sub-directories)
ls ~              # List the top level files of your home directory
du -sch ~/*       # Calculate the file sizes in your home

Viewing file info, user, and host:

stat <file-name>  # Last modification time stamps, permissions, and size of a file

whoami            # Shows your user name (same as "echo $USER")
hostname          # Shows on which machine you are (same as "echo $HOSTNAME")

Files and directories


mkdir <dir_name>   # Creates specified directory
rmdir <dir_name>   # Removes empty directory
rm <file_name>     # Removes file name
rm -r <dir_name>   # Removes directory including its content, but asks for confirmation,
                   # 'f' argument turns confirmation off
cp <name> <path>   # Copy file/directory as specified in path (-r to include content in directories)
mv <name1> <name2> # Renames directories or files
mv <name> <path>   # Moves file/directory as specified in path

Copy and paste

The methods differ depending where you are.

 

In a command line environment:
  • Cut last word with keyboard only
    Ctrl+w
      Press multiple times to cut more than one word

    Paste with keyboard only
    Ctrl+y

 

In a non-command line desktop environment:
  • Copy
    Ctrl+c
    Paste

    Ctrl+v
In any Linux desktop environment:
  • Copy with mouse only
    Simply select the text with the mouse
    Paste with mouse only
    Click the middle mouse button or both left/right buttons simltaniously

Handy shortcuts

Anywhere in Command Line:
    • up(down)_key                 - scrolls through command history
history   # shows all commands you have used recently

Auto Completion:
  • <something-incomplete> TAB   - completes program_path/file_name
Taking control over the cursor (the pointer on the command line):
Ctrl+a    # cursor to beginning of command line
Ctrl+e    # cursor to end of command line
Ctrl-w    # Cut last word 
Ctrl+k    # cut to the end of the line
Ctrl+y    # paste content that was cut earlier (by Ctrl-w or Ctrl-k)

When specifying file names:
    • "." (dot)            - refers to the present working directory
    • "~" (Tilda) or "~/"  - refers to user's home directory

Unix Help

man <something> # general help (press the 'q' key to exit) 
man wc          # manual on program 'word count' wc
wc --help       # short help on wc

soap -h         # for less standard programs 

Finding Things

Finding files, directories and applications

find -name "*pattern*"            # searches for *pattern* in and below current directory
find /usr/local -name "*blast*"   # finds file names *blast* in specfied directory
find /usr/local -iname "*blast*"  # same as above, but case insensitive
    • additional useful arguments: -user <user name>, -group <group name>, -ctime <number of days ago changed>
find ~ -type f -mtime -2   # finds all files you have modified in the last two days
locate <pattern>           # finds files and dirs that are written into update file
which <application_name>   # location of application
whereis <application_name> # searches for executeables in set of directories
dpkg -l | grep mypattern   # find Debian packages and refine search with grep pattern


Finding things in files

grep pattern file           # provides lines in 'file' where pattern 'appears', if pattern is shell function use single-quotes: '>'
grep -H pattern             # -H prints out file name in front of pattern
grep 'pattern' file | wc    # pipes lines with pattern into word count wc (see chapter 8); wc arguments: -c: show only bytes, -w: show only words, -l: show only lines; help on regular expressions: $ man 7 regex or man perlre
find /home/my_dir -name '*.txt' | xargs grep -c ^.*  # counts line numbers on many
                            # files and records each count along with individual file
                            # name; find and xargs are used to circumvent the Linux
                            # wildcard limit to apply this function on thousands of files.

Permissions and Ownership

List directories and files
$ ls -al # shows something like this for each file/dir: drwxrwxrwx
         # d: directory
         # rwx: read write execute
         # first triplet: user permissions (u)
         # second triplet: group permissions (g)
         # third triplet: world permissions (o)
Assign write and execute permissions to user and group
chmod ug+rx my_file

To remove all permissions from all three user groups
chmod ugo-rwx my_file
            # '+' causes the permissions selected to be added
            # '-' causes them to be removed
            # '=' causes them to be the only permissions that the file has.
   
chmod +rx public_html/ or $ chmod 755 public_html/ # Example for number system:
Change ownership

chown <user> <file or dir>         # changes user ownership
chgrp <group> <file or dir>        # changes group ownership
chown <user>:<group> <file or dir> # changes user & group ownership

Useful Unix Commands

df          # disk space
free -g     # memory info in Megabytes
uname -a    # shows tech info about machine
bc          # command-line calculator (to exit type 'quit')
wget ftp://ftp.ncbi.nih.... # file download from web
/sbin/ifconfig # give IP and other network info
ln -s original_filename new_filename # creates symbolic link to file or directory
du -sh      # displays disk space usage of current directory
du -sh *    # displays disk space usage of individual files/directories
du -s * | sort -nr # shows disk space used by different directories/files sorted by size

Process Management

General
top               # view top consumers of memory and CPU (press 1 to see per-CPU statistics)
who               # Shows who is logged into system
w                 # Shows which users are logged into system and what they are doing
ps                # Shows processes running by user
ps -e             # Shows all processes on system; try also '-a' and '-x' arguments
ps aux | grep <user_name> # Shows all processes of one user
ps ax --tree      # Shows the child-parent hierarchy of all processes
ps -o %t -p <pid> # Shows how long a particular process was running.
                  # (E.g. 6-04:30:50 means 6 days 4 hours ...)
Ctrl z <enter>    # Suspend (put to sleep) a process
fg                # Resume (wake up) a suspended process and brings it into foreground
bg                # Resume (wake up) a suspended process but keeps it running
                  # in the background.
Ctrl c            # Kills the process that is currently running in the foreground
kill <process-ID> # Kills a specific process
kill -9 <process-ID> # NOTICE: "kill -9" is a very violent approach.
                     # It does not give the process any time to perform cleanup procedures.
kill -l           # List all of the signals that can be sent to a proccess
kill -s SIGSTOP <process-ID> # Suspend (put to sleep) a specific process
kill -s SIGCONT <process-ID> # Resume (wake up) a specific process
renice -n <priority_value> # Changes the priority value, which range from 1-19,
                           # the higher the value the lower the priority, default is 10.
More on Terminating Processes

Text Viewing

more <my_file>  # views text, use space bar to browse, hit 'q' to exit
less <my_file>  # a more versatile text viewer than 'more', 'q' exits, 'G' moves to end of text, 'g' to beginning, '/' find forward, '?' find backwards
cat  <my_file>  # concatenates files and prints content to standard output

Text Editors

Vi and Vim
Non-graphical (terminal-based) editor. Vi is guaranteed to be available on any system. Vim is the improved version of vi.
Emacs
Non-graphical or window-based editor. You still need to know keystroke commands to use it. Installed on all Linux distributions and on most other Unix systems.
XEmacs
More sophisticated version of emacs, but usually not installed by default. All common commands are available from menus. Very powerful editor, with built-in syntax checking, Web-browsing, news-reading, manual-page browsing, etc.

Pico
Simple terminal-based editor available on most versions of Unix. Uses keystroke commands, but they are listed in logical fashion at bottom of screen.
Nano
A simple terminal-based editor which is default on modern Debian systems.

Vim Manual

Basics

vim my_file_name # open/create file with vim

Once you are in Vim the most important commands are i ,  :  and ESC. The i key brings you into the insert mode for typing. The ESC brings you out of there. And the : key starts the command mode at the bottom of the screen. In the following text, all commands starting with : need to be typed in the command mode. All other commands are typed in the normal mode after hitting the ESC key.

Modifier Keys to Control Vim
  • i # INSERT MODE
  • ESC # NORMAL (NON-EDITING) MODE
  • : # commands start with ':'
  • :w # save command; if you are in editing mode you have to hit ESC first!!
  • :q # quit file, don't save
  • :q! # exits WITHOUT saving any changes you have made
  • :wq # save and quit
  • R # replace MODE
  • r # replace only one character under cursor
  • q: # history of commands (from NORMAL MODE!), to reexecute one of them, select and hit enter!
  • :w new_filename # saves into new file
  • :#,#w new_filename # saves specific lines (#,#) to new file
  • :# go to specified line number

Help

Online Help
Help from Command Line

vimtutor # open vim tutorial from shell

Help in Vim
  • :help # opens help within vim, hit :q to get back to your file
  • :help <topic> # opens help on specified topic
  • :help_topic| CTRL-] # when you are in help this command opens help topic specified between |...|, CTRL-t brings you back to last topic
  • $ :help <topic> CTRL-D # gives list of help topics that contain key word
  • $ : <up-down keys> # like in shell you get recent commands!!!!

Moving Around in Files

  • $ # moves cursor to end of line
  • A # same as $, but switches to insert mode
  • 0 (zero) # moves cursor to beginning of line
  • CTRL-g # shows at status line filename and the line you are on
  • SHIFT-G # brings you to bottom of file, type line number (isn't displayed) then SHIFT-G # brings you to specified line#

Line Wrapping and Line Numbers

  • :set nowrap # no word wrapping, :set wrap # back to wrapping
  • :set number # shows line numbers, :set nonumber # back to no-number mode

Working with Many Files & Splitting Windows

vim -o *.txt # opens many files at once and displays them with horizontal
             # split, '-O' does vertical split
vim *.txt    # opens many files at once; ':n' switches between files

  • :wall or :qall # write or quit all open files
  • :args *.txt # places all the relevant files in the argument list
  • :all # splits all files in the argument list (buffer) horizontally
  • CTRL-w # switch between windows
  • :split # shows same file in two windows
  • :split <file-to-open> # opens second file in new window
  • :vsplit # splits windows vertically, very useful for tables, ":set scrollbind" let's you scroll all open windows simultaneously
  • :close # closes current window
  • :only # closes all windows except current one

Spell Checking & Dictionary

  • :set spell # turns on spell checking
  • :set nospell # turns spell checking off
  • :! dict <word> # meaning of word
  • :! wn 'word' -over # synonyms of word

Enabling Syntax Highlighting

  • :set filetype=perl # Turns on syntax coloring for a chosen programming language.
  • :set syntax on # Turns syntax highlighting on
  • :set syntax off # Turns syntax highlighting off

Undo and Redo

  • u # undo last command
  • U # undo all changes on current line
  • CTRL-R # redo one change which was undone

Deleting Things

  • x # deletes what is under cursor
  • dw # deletes from curser to end of word including the space
  • de # deletes from curser to end of word NOT including the space
  • cw # deletes rest of word and lets you then insert, hit ESC to continue with NORMAL mode
  • c$ # deletes rest of line and lets you then insert, hit ESC to continue with with NORMAL mode
  • d$ # deletes from cursor to the end of the line
  • dd # deletes entire line
  • 2dd # deletes next two lines, continues: 3dd, 4dd and so on.

Copy & Paste

  • yy # copies line, for copying several lines do 2yy, 3yy and so on
  • p # pastes clipboard behind cursor

Search in Files

  • /my_pattern # searches for my_pattern downwards, type n for next match
  • ?my_pattern # seraches for my_pattern upwards, type n for next match
  • :set ic # switches to ignore case search (case insensitive)
  • :set hls # switches to highlight search (highlights search hits)

Replacements with Regular Expression Support

Great intro: A Tao of Regular Expressions
  • :s/old_pat/new_pat/ # replaces first occurrence in a line
  • :s/old_pat/new_pat/g # replaces all occurrence in a line
  • :s/old_pat/new_pat/gc # add 'c' to ask for confirmation
  • :#,#s/old_pat/new_pat/g # replaces all occurrence between line numbers: #,#
  • :%s/old_pat/new_pat/g # replaces all occurrence in file
  • :%s/\(pattern1\)\(pattern2\)/\1test\2/g # regular expression to insert, you need here '\' in front of parentheses (<# Perl)
  • :%s/\(pattern.*\)/\1 my_tag/g # appends something to line containing pattern (<# .+ from Perl is .* in VIM)
  • :%s/\(pattern\)\(.*\)/\1/g # removes everything in lines after pattern
  • :%s/\(At\dg\d\d\d\d\d\.\d\)\(.*\)/\1\t\2/g # inserts tabs between At1g12345.1 and Description
  • :%s/\n/new_pattern/g #Replaces return signs
  • :%s/pattern/\r/g #Replace pattern with return signs!!
  • :%s/\(\n\)/\1\1/g # insert additional return signs
  • :%s/\(^At\dg\d\d\d\d\d.\d\t.\{-}\t.\{-}\t.\{-}\t.\{-}\t\).\{-}\t/\1/g # replaces content between 5th and 6th tab (5th column), '{-}' turns off 'greedy' behavior
  • :#,#s/\( \{-} \|\.\|\n\)/\1/g # performs simple word count in specified range of text
  • :%s/\(E\{6,\}\)/<font color="green">\1<\/font>/g # highlight pattern in html colors, here highlighting of >= 6 occurences of Es
  • :%s/\([A-Z]\)/\l\1/g # change uppercase to lowercase, '%s/\([A-Z]\)/\u\1/g' does the opposite
  • :g/my_pattern/ s/\([A-Z]\)/\l\1/g | copy $ # uses 'global' command to apply replace function only on those lines that match a certain pattern. The 'copy $' command after the pipe '|' prints all matching lines at the end of the file.
  • :args *.txt | all | argdo %s/\old_pat/new_pat/ge | update # Command 'args' places all relevant files in the argument list (buffer); 'all' displays each file in separate split window; command 'argdo' applies replacement to all files in argument list (buffer); flag 'e' is necessary to avoid stop at error messages for files with no matches; command 'update' saves all changes to files that were updated.

Useful Utilities in Vim

Matching Parentheses
    • Place curser on (, [ or { and type % # curser moves to matching parentheses
Printing and Inserting Files
  • :ha # prints entire file
  • :#,#ha # prints specified lines: #,#
  • :r <filename> # inserts content of specified file after cursor

Convert Text File to HTML Format
  • :runtime! syntax/2html.vim # run this command with open file in Vim
Shell Commands in Vim
  • :!<SHELL_COMMAND> <ENTER> # executes any shell command, hit <enter> to return
  • :sh # switches window to shell, 'exit' switches back to vim
Using Vim as Table Editor
  • v # starts visual mode for selecting characters
  • V # starts visual mode for selecting lines
  • CTRL-V # starts visual mode for selecting blocks (use CTRL-q in gVim under Windows). This allows column-wise selections and operations like inserting and deleting columns. To restrict substitude commands to a column, one can select it and switch to the command-line by typing ':'. After this the substitution sytax for a selected block looks like this: '<,'>s///.
  • :set scrollbind # starts simultaneous scrolling of 'vsplitted' files. To set to horizontal binding of files, use command ':set scrollopt=hor' (after first one). Run all these commands before the ':split' command.
  • :AlignCtrl I= \t then :%Align # This allows to align tables by column separators (here '\t') when the Align utility from Charles Campbell's is installed. To sort table rows by selected lines or block, perform the visual select and then hit F3 key. The rest is interactive. To enable this function one has to include in the .vimrc file from Gerald Lai the Vim sort script.

Modify Vim Settings

The default settings in Vim are controlled by the .vimrc file in your home directory.
  •  see last chapter of vimtutor (start from shell)
  •  useful .vimrc sample
  •  when vim starts to respond very slowly then one may need to delete the .viminf* files in home directory

The Unix Shell

When you log into UNIX/LINUX system, then is starts a program called the Shell. It provides you with a working environment and interface to the operating system. Usually there are several different shell programs installed. The shell program bash is one of the most common ones.
finger <user_name> # shows which shell you are using
chsh -l # gives list of shell programs available on your system (does not work on all UNIX variants)
<shell_name> # switches to different shell

STDIN, STDOUT, STDERR, Redirections, and Wildcards

See LINUX HOWTOs


By default, UNIX commands read from standard input (STDIN) and send their output to standard out (STDOUT).


You can redirect them by using the following commands:
<beginning-of-filename>*         # * is wildcard to specify many files
ls > file                        # prints ls output into specified file
command < my_file                # uses file after '<' as STDIN
command >> my_file               # appends output of one command to file
command | tee my_file            # writes STDOUT to file and prints it to screen
command > my_file; cat my_file   # writes STDOUT to file and prints it to screen
command > /dev/null              # turns off progress info of applications by redirecting
                                 # their output to /dev/null
grep my_pattern my_file | wc     # Pipes (|) output of 'grep' into 'wc'
grep my_pattern my_non_existing_file 2 > my_stderr # prints STDERR to file

Useful shell commands

cat <file1> <file2> > <cat.out>      # concatenate files in output file 'cat.out'
paste <file1> <file2> > <paste.out>  # merges lines of files and separates them by tabs (useful for tables)
cmp <file1> <file2>                  # tells you whether two files are identical
diff <fileA> <fileB>                 # finds differences between two files
head -<number> <file>                # prints first lines of a file
tail -<number> <file>                # prints last lines of a file
split -l <number> <file>             # splits lines of file into many smaller ones
csplit -f out fasta_batch "%^>%" "/^>/" "{*}" # splits fasta batch file into many files
                                     # at '>'
sort <file>                          # sorts single file, many files and can merge (-m)
                                     # them, -b ignores leading white space, ...
sort -k 2,2 -k 3,3n input_file > output_file # sorts in table column 2 alphabetically and
                                     # column 3 numerically, '-k' for column, '-n' for
                                     # numeric
sort input_file | uniq > output_file # uniq command removes duplicates and creates file/table
                                     # with unique lines/fields
join -1 1 -2 1 <table1> <table2>     # joins two tables based on specified column numbers
                                     # (-1 file1, 1: col1; -2: file2, col2). It assumes
                                     # that join fields are sorted. If that is not the case,
                                     # use the next command:
sort table1 > table1a; sort table2 > table2a; join -a 1 -t "`echo -e '\t'`" table1a table2a > table3                               # '-a <table>' prints all lines of specified table!
                                     # Default prints only all lines the two tables have in
                                     # common. '-t "`echo -e '\t'`" ->' forces join to
                                     # use tabs as field separator in its output. Default is
                                     # space(s)!!!
cat my_table | cut -d , -f1-3        # cut command prints only specified sections of a table,
                                     # -d specifies here comma as column separator (tab is
                                     # default), -f specifies column numbers.
grep                                 # see chapter 4
egrep                                # see chapter 4

Screen

Starting a New Screen Session

screen                 # Start a new session
screen -S <some-name>  # Start a new session and gives it a name
Commands to Control Screen
  • Ctrl-a d #  Detach from the screen session
  • Ctrl-a c # Create a new window inside the screen session
  • Ctrl-a Space # Switch to the next window
  • Ctrl-a a # Switch to the window that you were previously on
  • Ctrl-a " # List all open windows. Double-quotes " are typed with the Shift key
  • Ctrl-d or type exit # Exit out of the current window. Exiting form the last window will end the screen session
  • Ctrl-a [ # Enters the scrolling mode. Use Page Up and Page Down keys to scroll through the window. Hit the Enter key twice to return to normal mode.

Attaching to Screen Sessions

From any computer, you can attach to a screen session after SSH-ing into a server.
screen -r              # Attaches to an existing session, if there is only one
screen -r              # Lists available sessions and their names, if there are more then one session running
screen -r <some-name>  # Attaches to a specific session
screen -r <first-few-letters-of-name> # Type just the first few letters of the name and you will be attached to the session you need

Destroying Screen Sessions

1. Terminate all programs that are running in the screen session. The standard way to do that is:
Ctrl-c

2. Exit out of your shell.
exit
3. Repeat steps 1 and 2 until you see the message:
[screen is terminating]
There may be programs running in different windows of the same screen session. That's why you may need to terminate programs and exit shells multiple time.

Tabs and a Reasonably Large History Buffer

For a better experience with screen, run
cp ~/.screenrc ~/.screenrc.backup 2> /dev/null
echo 'startup_message off
defscrollback 10240
caption always "%{=b dy}{ %{= dm}%H %{=b dy}}%={ %?%{= dc}%-Lw%?%{+b dy}(%{-b r}%n:%t%{+b dy})%?(%u)%?%{-dc}%?%{= dc}%+Lw%? %{=b dy}}"
' > ~/.screenrc

More on Terminating Processes



Tags:

Back