Daily Archives: August 19, 2013
C++ || Serial & Parallel Multi Process File Downloader Using Fork & Execlp
The following is another homework assignment which was presented in an Operating Systems Concepts class. The following are two multi-process programs using commandline arguments, which demonstrates more practice using the fork() and execlp() system calls on Unix based systems.
==== 1. OVERVIEW ====
File downloaders are programs used for downloading files from the Internet. The following programs listed on this page implement two distinct type of multi-process downloaders:
1. a serial file downloader which downloads files one by one.
2. a parallel file downloader which dowloads multiple files in parallel.
In both programs, the parent process first reads a file via the commandline. This file which is read is the file that contains the list of URLs of the files to be downloaded. The incoming url file that is read has the following format:
[URL1]
[URL2]
.
.
.
[URLN]
Where [URL] is an http internet link with a valid absolute file path extension.
(i.e: http://newsimg.ngfiles.com/270000/270173_0204618900-cc-asmbash.jpg)
After the url file is parsed, next the parent process forks a child process. Each created child process uses the execlp() system call to replace its executable image with that of the “wget” program. The use of the wget program performs the actual file downloading.
==== 2. SERIAL DOWNLOADER ====
The serial downloader downloads files one at a time. After the parent process has read and parsed the incoming url file from the commandline, the serial downloader proceeds as follows:
1. The parent forks off a child process.
2. The child uses execlp("/usr/bin/wget", "wget", [URL STRING1], NULL) system call in order to replace its program with wget program that will download the first file in urls.txt (i.e. the file at URL).
3. The parent executes a wait() system call until the child exits.
4. The parent forks off another child which downloads the next file specified in url.txt.
5. Repeat the same process until all files are downloaded.
The following is implemented below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 |
// ============================================================================ // Author: Kenneth Perkins // Date: Aug 19, 2013 // Taken From: http://programmingnotes.org/ // File: Serial.cpp // Description: File downloaders are programs used for downloading files // from the Internet. Using the fork() & execlp("wget") command, the // following is a multi-process serial file downloader which reads an // input file containing url file download links as a commandline // argument and downloads the files located on the internet one by one. // ============================================================================ #include <iostream> #include <cstring> #include <fstream> #include <cstdlib> #include <unistd.h> #include <sys/wait.h> using namespace std; // compile & run // g++ Serial.cpp -o Serial // ./Serial urls.txt int main(int argc, char* argv[]) { // declare variables pid_t pid = -1; int urlNumber = 0; char urlName[256]; ifstream infile; // check if theres enough command line args if(argc < 2) { cout <<"\nERROR -- NOT ENOUGH ARGS!" <<"\n\nUSAGE: "<<argv[0]<<" <file containing url downloads>\n\n"; exit(1); } // try to open the file containing the download url links // exit if the url file is not found infile.open(argv[1]); if(infile.fail()) { cout <<"\nERROR -- "<<argv[1]<<" NOT FOUND!\n\n"; exit(1); } // get download url links from the file while(infile.getline(urlName, sizeof(urlName))) { ++urlNumber; // fork another process pid = fork(); if(pid < 0) { // ** error occurred perror("fork"); exit(1); } else if(pid == 0) { // ** child process cout <<endl<<"** URL #"<<urlNumber <<" is currently downloading... **\n\n"; execlp("/usr/bin/wget", "wget", urlName, NULL); } else { // ** parent process // parent will wait for the child to complete wait(NULL); cout <<endl<<"-- URL #"<<urlNumber<<" is complete! --\n"; } } infile.close(); cout <<endl<<"The parent process is now exiting...\n"; return 0; }// http://programmingnotes.org/ |
Since the serial downloader downloads files one at a time, that can become very slow. That is where the parallel downloader comes in handy!
==== 3. PARALLEL DOWNLOADER ====
The parallel downloader downloads files all at once and is implemented much like the serial downloader. The parallel downloader proceeds as follows:
1. The parent forks off n children, where n is the number of URLs in url.txt.
2. Each child executes execlp("/usr/bin/wget", "wget", [URL STRING], NULL) system call where eachis a distinct URL in url.txt.
3. The parent calls wait() (n times in a row) and waits for all children to terminate.
4. The parent exits.
The following is implemented below:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
// ============================================================================ // Author: Kenneth Perkins // Date: Aug 19, 2013 // Taken From: http://programmingnotes.org/ // File: Parallel.cpp // Description: File downloaders are programs used for downloading files // from the Internet. Using the fork() & execlp("wget") command, the // following is a multi-process parallel file downloader which reads an // input file containing url file download links as a commandline // argument and downloads the files located on the internet all at once. // ============================================================================ #include <iostream> #include <cstring> #include <fstream> #include <cstdlib> #include <unistd.h> #include <sys/wait.h> using namespace std; // compile & run // g++ Parallel.cpp -o Parallel // ./Parallel urls.txt int main(int argc, char* argv[]) { // declare variables pid_t pid = -1; int urlNumber = 0; char urlName[256]; ifstream infile; // check if theres enough command line args if(argc < 2) { cout <<"\nERROR -- NOT ENOUGH ARGS!" <<"\n\nUSAGE: "<<argv[0]<<" <file containing url downloads>\n\n"; exit(1); } // try to open the file containing the download url links // exit if the url file is not found infile.open(argv[1]); if(infile.fail()) { cout <<"\nERROR -- "<<argv[1]<<" NOT FOUND!\n\n"; exit(1); } // get download url links from the file while(infile.getline(urlName, sizeof(urlName))) { ++urlNumber; // fork another process pid = fork(); if(pid < 0) { // ** error occurred perror("fork"); exit(1); } else if(pid == 0) { // ** child process cout <<endl<<"** URL #"<<urlNumber <<" is currently downloading... **\n\n"; execlp("/usr/bin/wget", "wget", urlName, NULL); } } infile.close(); while(urlNumber > 0) { // ** parent process // parent will wait for the child to complete wait(NULL); cout <<endl<<"-- URL #"<<urlNumber<<" is complete! --\n"; --urlNumber; } cout <<endl<<"The parent process is now exiting...\n"; return 0; }// http://programmingnotes.org/ |
QUICK NOTES:
The highlighted lines are sections of interest to look out for.
The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.
Also note, while the parallel downloader executes, the outputs from different children may intermingle.
C++ || Snippet – How To Use Fork & Pipe For Interprocess Communication
The following is sample code which demonstrates the use of the fork, read, and write function calls for use with pipes on Unix based systems.
A pipe is a mechanism for interprocess communication. Data written to a pipe by one process can be read by another process. Creating a pipe is achieved by using the pipe function, which creates both the reading and writing ends of the pipe file descriptor.
In typical use, a parent process creates a pipe just before it forks one or more child processes. The pipe is then used for communication between either the parent or child processes, or between two sibling processes.
A real world example of this kind of communication can be seen in all operating system terminal shells. When you type a command in a shell, it will spawn the executable represented by that command with a call to fork. A pipe is opened to the new child process, and its output is read and printed by the terminal.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
// ============================================================================ // Author: Kenneth Perkins // Date: Aug 19, 2013 // Taken From: http://programmingnotes.org/ // File: ForkPipeExample.cpp // Description: Demonstrate the use of the fork() and pipe() command for // parent and child interprocess communication. // ============================================================================ #include <iostream> #include <cstdlib> #include <cstring> #include <sys/wait.h> #include <unistd.h> using namespace std; // global variables const int BUFFER_SIZE = 256; const int READ_END = 0; const int WRITE_END = 1; int main() { // declare variables int fd[2]; // pipe file descriptor pid_t pid = -1; char writeMsg[BUFFER_SIZE] = "Greetings From Your Parent!"; char readMsg[BUFFER_SIZE]; // create a pipe if(pipe(fd) < 0) { perror("pipe"); exit(1); } cout <<"\nParent is forking a child." << endl; // create a duplicate process of this current program pid = fork(); // exit if something went wrong with the fork if(pid < 0) { perror("fork"); exit(1); } // this code only gets executed by the child process else if(pid == 0) { cout <<"\nStarting the child process..\n\n"; // close the "write" end b/c we arent using it close(fd[WRITE_END]); // get data from the pipe using the read() function read(fd[READ_END], readMsg, sizeof(readMsg)); // close the "read" end close(fd[READ_END]); cout <<"Message from the parent via the pipe: "<<readMsg<<endl; } // the parent process executes here else { cout <<"\nParent is now waiting for child id #"<<pid<<" to complete..\n"; // close the "read" end b/c we arent using it close(fd[READ_END]); // write data on pipe write(fd[WRITE_END], writeMsg, strlen(writeMsg)+1); // close the "write" end close(fd[WRITE_END]); // use the "wait" function to wait for the child to complete wait(NULL); cout <<"\nThe child process is complete and has terminated!\n"; } // NOTE: this message gets displayed twice - Why?! cout <<"\nProgram is now exiting...\n"; return 0; }// http://programmingnotes.org/ |
QUICK NOTES:
The highlighted lines are sections of interest to look out for.
The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.
The following is sample output:
Parent is forking a child.
Parent is now waiting for child id #12776 to complete..Starting the child process..
Message from the parent via the pipe: Greetings From Your Parent!
Program is now exiting...
The child process is complete and has terminated!
Program is now exiting...
C++ || Snippet – How To Use Fork & Execlp For Interprocess Communication
The following is sample code which demonstrates the use of the “fork” and “execlp” function calls on Unix based systems.
The “fork” function call creates a new process by duplicating the calling process; or in more simpler terms, it creates a duplicate process (a child) of the calling (parent) process.
This new process, referred to as the child, is an exact duplicate of the calling process, referred to as the parent.
The “execlp” function call is part of a family of functions which replaces a current running process image with a new process image. That means by using the “execlp” function call, you are basically replacing the entire current running process with a new user defined program.
Though fork and execlp are not required to be used together, they are often used in conjunction with one another as a way of creating a new program running as a child of another process.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
// ============================================================================ // Author: Kenneth Perkins // Date: Aug 19, 2013 // Taken From: http://programmingnotes.org/ // File: ForkExample.cpp // Description: Demonstrate the use of the fork() command for parent and // child process manipulation. // ============================================================================ #include <iostream> #include <cstdlib> #include <sys/wait.h> #include <unistd.h> using namespace std; int main() { // declare variable pid_t pid = -1; cout <<"\nParent is forking a child." << endl; // create a duplicate process of this current program pid = fork(); // exit if something went wrong with the fork if(pid < 0) { perror("fork"); exit(1); } // this code only gets executed by the child process else if(pid == 0) { cout <<"\nStarting the child process..\n" << endl; // NOTE: the "execlp" function replaces the current // child process with the terminal "ls" command. // the child process finally dies after the "execlp" // function is complete execlp("/bin/ls", "ls", "-l", NULL); // "ls -l" command cout <<"\nNOTE: This never gets executed - Why?!"; } // the parent process executes here else { cout <<"\nParent is now waiting for child id #"<<pid<<" to complete.." << endl; // use the "wait" function to wait for the child to complete wait(NULL); cout <<"\nThe child process is complete and has terminated!\n"; } cout <<"\nParent is now exiting...\n"; return 0; }// http://programmingnotes.org/ |
QUICK NOTES:
The highlighted lines are sections of interest to look out for.
The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.
The following is sample output:
Parent is forking a child.
Parent is now waiting for child id #10872 to complete..Starting the child process..
total 1466
-rw-r--r-- 1 admin admin 468 Apr 27 2012 nautilus-computer.desktop
-rwxrwxr-x 1 admin admin 9190 Aug 19 15:17 ForkExample
-rw-rw-r-- 1 admin admin 1640 Aug 19 15:17 ForkExample.cppThe child process is complete and has terminated!
Parent is now exiting...
Hi, I'm a software engineer. I write articles about programming and design in my spare time.
