Shell, I/O redirections, and filters

prerequisites

Allowed editors: vi, vim, emacs

• All your scripts will be tested on Ubuntu 20.04 LTS

• All your scripts should be exactly two lines long ($ wc -l file should print 2)

• All your files should end with a new line (why?)

• The first line of all your files should be exactly #!/bin/bash

• A README.md file, at the root of the folder of the project, describing what each script is doing

• You are not allowed to use backticks, &&, || or ;

• All your files must be executable

• You are not allowed to use sed or awk

overview

The "Shell, I/O filters, and Redirection" project builds on the foundational knowledge of shell permissions and delves deeper into the features and capabilities of the shell. The project explores the concept of standard input, output, and error, as well as the various filters and commands that can be used to manipulate input and output streams.

In addition to basic command line tools like head, tail, find, wc, sort, uniq, grep, tr, and sed, this project covers more advanced techniques like shell scripting, redirection of input and output, and how to send the output of one command as input to another command.

The project also delves into the use of special characters like white spaces, single and double quotes, backslash, comment, pipe, command separator, and tilde, and when to use them in different scenarios.

Moreover, this project covers several exercises to help you practice what you have learned, including how to display a line of text, concatenate files, reverse a string, and remove sections from each line of files.

Finally, this project sets the foundation for learning about shell initialization files, variables, and expansions, which will be covered in the next phase of the project. Overall, "Shell, I/O filters, and Redirection" is an essential project for anyone looking to build their skills in shell programming and automation.

introduction

In computing, a shell is a user interface for accessing an operating system's services. It allows users to interact with the operating system by entering commands and receiving feedback in the form of text output. The shell is typically a command-line interface, where users type commands and arguments to perform tasks.

I/O redirection is a mechanism in shell scripting that allows input and output to be redirected from its default sources and destinations to other sources and destinations. This means that instead of input coming from the keyboard and output going to the screen, input can come from a file and output can be redirected to a file or another command. This allows for more flexibility and control over how a command's input and output are handled. For example, you can use ">" to redirect standard output to a file, ">>" to append to a file, "<" to read input from a file, and "|" to pipe output from one command to another as input.

What do the commands head, tail, find, wc, sort, uniq, grep, tr do ?

1. head: This command displays the first few lines of a file. By default, it shows the first 10 lines of a file, but you can specify a different number of lines using the -n option. For example, to display the first 5 lines of a file called myfile.txt, you can use the following command:

head -n 5 myfile.txt

1. tail: This command displays the last few lines of a file. By default, it shows the last 10 lines of a file, but you can specify a different number of lines using the -n option. For example, to display the last 5 lines of a file called myfile.txt, you can use the following command:

tail -n 5 myfile.txt

1. find: This command is used to search for files and directories in a specified location. For example, to find all files with the .txt extension in the current directory and its subdirectories, you can use the following command:

find . -name "*.txt"

1. wc: This command is used to count the number of lines, words, and characters in a file. For example, to count the number of lines in a file called myfile.txt, you can use the following command:

wc -l myfile.txt

1. sort: This command is used to sort the lines in a file alphabetically or numerically. For example, to sort the lines in a file called myfile.txt in alphabetical order, you can use the following command:

sort myfile.txt

1. uniq: This command is used to remove duplicate lines from a file. For example, to remove duplicate lines from a file called myfile.txt, you can use the following command:

uniq myfile.txt

1. grep: This command is used to search for a pattern in a file. For example, to search for the word "apple" in a file called myfile.txt, you can use the following command:

grep "apple" myfile.txt

1. tr: This command is used to translate or delete characters in a file. For example, to replace all occurrences of the letter "a" with the letter "b" in a file called myfile.txt, you can use the following command:

tr 'a' 'b' < myfile.txt

How to redirect standard output to a file

To redirect standard output to a file, you can use the > operator followed by the filename. For example, to redirect the output of the ls command to a file called myfiles.txt, you can use the following command:

ls > myfiles.txt

This will create a new file called myfiles.txt and write the output of the ls command to it. If the file already exists, its contents will be overwritten. If you want to append the output to an existing file, you can use the >> operator instead:

ls >> myfiles.txt

This will append the output of the ls command to the end of the file myfiles.txt without overwriting its existing contents.

How to get standard input from a file instead of the keyboard

To get standard input from a file instead of the keyboard, you can use the < operator followed by the filename. For example, to get standard input from a file called input.txt, you can use the following command:

myprogram < input.txt

This will redirect the contents of input.txt to the standard input of myprogram.

How to send the output from one program to the input of another program

To send the output from one program to the input of another program, you can use the | operator. For example, to send the output of program1 to the input of program2, you can use the following command:

program1 | program2

This will take the standard output of program1 and send it as standard input to program2.

How to combine commands and filters with redirections

You can combine commands and filters with redirections to perform complex operations on files and data. Here are some examples:

To count the number of lines in a file and save the result to a new file called output.txt, you can use the following command:

wc -l myfile.txt > output.txt

This will redirect the output of wc -l myfile.txt to a new file called output.txt.

To find all files in the current directory and its subdirectories that have the .txt extension, sort the results alphabetically, and save them to a new file called output.txt, you can use the following command:

find . -name "*.txt" | sort > output.txt

This will use the find command to search for files with the .txt extension, use the sort command to sort the results, and redirect the output to a new file called output.txt.

To search for the word "apple" in a file called myfile.txt, remove any duplicate lines, and save the result to a new file called output.txt, you can use the following command:

grep "apple" myfile.txt | uniq > output.txt

This will use the grep command to search for the word "apple" in myfile.txt, use the uniq command to remove duplicate lines, and redirect the output to a new file called output.txt.

What are special characters

Special characters are characters that have a specific meaning and function in the Linux/Unix command line environment.

Understand what do the white spaces, single quotes, double quotes, backslash, comment, pipe, command separator, tilde and how and when to use them

1. White spaces: Spaces and tabs are used to separate command line arguments and options. For example, to list the contents of a directory, you can use the ls command followed by a space and the name of the directory:

ls /home/user/documents

1. Single quotes (''): Single quotes are used to enclose a string literal, which tells the shell to interpret the enclosed characters literally. This means that any special characters within the single quotes will be treated as normal characters. For example:

echo 'Hello, World!'

will output Hello, World! literally.

1. Double quotes (""): Double quotes are used to enclose a string literal, which tells the shell to interpret some special characters within the quotes. For example, variable names enclosed within double quotes will be expanded to their values:

name='Alice'
echo "Hello, $name!"

will output Hello, Alice!.

1. Backslash (): The backslash is used to escape special characters, so they are interpreted literally. For example:

echo "The file is called \"file.txt\""

will output The file is called "file.txt".

1. Comment (#): The pound or hash sign is used to indicate a comment in a command line. Any text that follows the # symbol will be ignored by the shell. For example:

# This is a comment
echo 'Hello, World!' # This is another comment

will only output Hello, World!.

1. Pipe (|): The pipe symbol is used to connect the output of one command to the input of another command. For example:

ls -la | grep .txt

will list all files in the current directory and then filter the results to show only files that end with .txt.

1. Command separator (;): The semicolon is used to separate multiple commands on a single line. For example:

echo 'Hello'; echo 'World'

will output Hello and World on separate lines.

1. Tilde (~): The tilde is used as a shorthand for the home directory of the current user. For example:

cd ~

will change the current directory to the home directory of the current user.

It's important to understand how and when to use special characters, as they can significantly affect the behavior and output of your command line operations.

How to display a line of text

To display a line of text, you can use the echo command. For example:

bashCopy codeecho "Hello, World!"

will output Hello, World!.

How to concatenate files and print on the standard output

To concatenate files and print on the standard output, you can use the cat command followed by the names of the files you want to concatenate. For example:

cat file1.txt file2.txt

will concatenate the contents of file1.txt and file2.txt and print the result on the standard output.

how to reverse a string

To reverse a string, you can use the rev command. For example:

echo "Hello, World!" | rev

will output !dlroW ,olleH.

How to remove sections from each line of files

To remove sections from each line of files, you can use the cut command. For example:

cut -d',' -f1 file.csv

will remove everything after the first comma on each line of the file.csv.

What is the /etcHow /passwd file and what is its format

The /etc/passwd file is a system file that stores information about user accounts on a Linux or Unix system. It is a plain text file that contains one line for each user account, with seven fields separated by colons: username, password (encrypted), user ID, group ID, user's full name, home directory, and login shell.

What is the /etc/shadow file and what is its format

The /etc/shadow file is a system file that stores the encrypted passwords and other authentication information for user accounts on a Linux or Unix system. It is a plain text file that contains one line for each user account, with nine fields separated by colons: username, encrypted password, date of last password change, minimum password age, maximum password age, password warning period, password inactivity period, account expiration date, and a reserved field. Only the root user has read and writes access to this file, which helps to keep the passwords and other authentication information secure.

Exercises

1. Write a command that prints the number of lines in a file called example.txt.

   To print the number of lines in a file called example.txt, you can use the following command:

    wc -l example.txt
   

   This will display the number of lines in the file example.txt.

2. Write a command that concatenates the contents of two files file1.txt and file2.txt and writes the result to a new file output.txt.

   To concatenate the contents of two files file1.txt and file2.txt and write the result to a new file output.txt, you can use the following command:

    cat file1.txt file2.txt > output.txt
   

   This will concatenate the contents of file1.txt and file2.txt and write the result to a new file called output.txt.

3. Write a command that reverses the order of characters in a string "Hello, World!" and writes the result to a new file reversed.txt.

   To reverse the order of characters in a string "Hello, World!" and write the result to a new file reversed.txt, you can use the following command:

    echo "Hello, World!" | rev > reversed.txt
   

   This will reverse the order of characters in the string "Hello, World!" using the rev command and write the result to a new file called reversed.txt.

4. To remove the first two fields (separated by commas) from each line of a CSV file called data.csv and write the result to a new file processed.csv, you can use the following command:

    cut -d ',' -f 3- data.csv > processed.csv
   

   This will remove the first two fields (separated by commas) from each line of the CSV file data.csv using the cut command and write the result to a new file called processed.csv.

5. write a command that shows the first 10 lines of a file called largefile.log.

   To show the first 10 lines of a file called largefile.log, you can use the following command:

    head -n 10 largefile.log
   

   This will display the first 10 lines of the file largefile.log using the head Write a command that removes the first two fields (separated by commas) from each line of a CSV file called data.csv and writes the result to a new file processed.csv.

   more exercises for readers: readers are expected to practice the below exercise at their one space.

6. Write a command that finds all files in the current directory and its subdirectories that have the word "example" in their names, and prints the file names and their sizes in a human-readable format.

7. Write a command that shows the total number of words in all files with the .txt extension in the current directory.

8. Write a command that creates a new directory called backup in the home directory of the current user, and copies all files with the .docx extension in the current directory to the backup directory.

9. Write a command that changes the permission of a file secret.txt so that only the owner can read and write to it.

10. Write a command that lists all running processes on the system and sorts them by the amount of memory they are using, in descending order.

These exercises will help you practice and reinforce your knowledge of the Linux/Unix command line. Good luck!

Thank you for reading this tutorial. Today, I hope you learned something new! If you found this article informative, please like, share, and follow me for more blog posts like this in the future.

Let's meet up. I discuss web development, content creation, open source, career counseling, and IT jobs on these platforms.

Linkedin

Twitter