Streams & File Manipulation
Data streams in Linux have two ends. A Linux command that you provide to terminal is treated as one end(input). The other end is determined by the terminal that executed the command. That end will be connected to, may be another terminal window, connected to a pipe or redirected to a file or other command.
Three standard streams are :
- stdin Standard Input : accepts text as input.
- stdout Standard Output : Text output from the command to the terminal is delivered through this stream.
- stderr Standard Error : Error messages from the command are sent through the this stream.
These are the 3 standard streams that are created when a Linux command is executed.
In computing, stream is something that transfer data. In case of these streams, the data is text.
Streams in Linux - like almost everything else are treated as files.
Each file associated with a process is allocated a unique number to identify it. This is known as the file descriptor. Whenever an action is required to be performed on a file, the file descriptor is used to identify the file.
File descriptors for these streams:
- Standard input - 0.
- Standard output - 1.
- Standard error - 2.
To redirect an output of a command > can be used.
ls > out.txt
cat out.txt
This command doesn't show the output in the terminal. It is send to out.txt file.
cat command displays the content in out.txt file.
Now, if you try to add new text to out.txt using > the existing information will be replaced with new text. But what if you want to append text to out.txt file ?
To overcome this problem >> is used instead of > to append data to existing data.
From the above image it can be observed that new data is appended instead of replacing the existing data.
In the above image I tried to store the error message of command in a file using > but it's not working. This problem can be solved using file descriptors.
To store an output error message to a file then use file descriptor 2 before >.
lg 2> [filename]
Now, the output error message will be stored in file. It can be viewed using cat command.
- less
It can be used to view the content of a text file one page at a time.
less /var/log/syslog
This will open system logs file in a new terminal window and can be collapsed on clicking q key.
|- Pipe
Pipe is used to combine two or more commands, and in this output of one command acts as input to another command, and this output can be acts as input to next command ans so on.
ls | cat > out.txt
ls command enlists all the directories and this is given as input to cat command using pipe (|).
- Environment variables
In Linux and Unix based systems environment variables are a set of dynamic values, stored within the system that are used by applications launched in shells or subshells. In simple words, an environment variable is a variable with a name and an associated value.
It allow you to customise how the system works. For example, the environment variable can store information about the default text editor or browser, the path to executable files.
Variables have the following format:
- KEY=value
- KEY="Some other value"
- KEY=value1:value2
Environment variables are case-sensitive and should have UPPER CASE names.
When assigning multiple values to the variable they must be separated by the colon :
There is no space around the equals = symbol.
Variables can be classified into two main categories, environment variables, and shell variables.
Environment variables are variables that are available system-wide and are inherited by all spawned child processes and shells.
Shell variables are variables that apply only to the current shell instance. Each shell such as zsh and bash, has its own set of internal shell variables.
env – The command allows you to run another program in a custom environment without modifying the current one. When used without an argument it will print a list of the current environment variables.
printenv – The command prints all or the specified environment variables.
set – Sets or unset shell variables. When used without an argument it will print a list of all variables including environment and shell variables, and functions.
unset – Deletes shell and environment variables.
export – Sets environment variables.
The printenv and env commands print only the environment variables. If you want to get a list of all variables, including environment, shell and variables, and shell functions you can use the set command:
- head
head command is used to see the first 10 lines of a file.
head /var/log/syslog
head -n 5 /var/log/syslog
-n flag is used to view only specific number of lines of a file.
- tail
tail command is used to view last 10 lies of a file.
tail /var/log/syslog
tail -n 5 /var/log/syslog
tail -f /var/log/syslog
-f flag is used to track the live log.
- sort
sort command returns the sorted version of the file.
-r flag is used with sort to return in items in descending order.
sort [file]
sort -r [file]
tr- Translate
tr command is used for translating or deleting characters. It can be used with pipes to support more complex translations.
c : complements the set of characters in string.i.e., operations apply to characters not in the given set.
d : delete characters in the first set from the output.
s : replaces repeated characters listed in the set1 with single occurrence.
t : truncates set1.
Syntax : tr [flag] [set1] [set2]
cat out.txt | tr a-z A-Z
Output of cat command will be feeded as input to tr command which will convert.
- uniq
uniq command is used to give unique values in a file.
c : This flag is used to get count of every unique value.
u : Returns the values that occur only one time in the file.
d : Returns only the duplicated values.
uniq phanee.txt
If there are duplicated words in a file which are not adjacent to each other then they are treated as unique values. To tackle this issue we sort the file and the output will be feeded as input to the uniq command using pipe to find the unique values.
In the above image you can observe this problem.
- wc
wc: used to print lines count, word count and size of file (byte count) for each file.
wc phanee.txt
l : prints number of lines in a file.
w : prints the number of words in a file.
m : prints the number of characters in a file.
c : prints the number of characters in a file.
grep- Pattern Matching
The grep filter searches a file for a particular pattern of characters, and displays all lines that contain that pattern.
env | grep PWD
This command returns the patterns that matched with PWD.
More on grep in future blogs.