Tag Archives: unix

C++ || Multi Process Server & Client Hash Table Using Thread Pools, Message Queues, & Signal Handlers

The following is another homework assignment which was presented in an Operating Systems Concepts class. Using commandline arguments, the following is a program which implements a multi threaded hash table, utilizing message queues to pass text from a client program to a server program and vice versa. This program makes use of multiple interprocess communication function calls provided on Unix based systems.

REQUIRED KNOWLEDGE FOR THIS PROGRAM

How To Override The Default Signal Handler (CTRL-C)
How To Create And Use Pthreads For Interprocess Communication
How To Use Message Queues For Interprocess Communication
Multi Process Synchronization Producer Consumer Problem Using Pthreads
Sample Input Server Records File - Download Here

==== 1. OVERVIEW ====

Hash tables are widely used for efficient storage and retrieval of data. When using hash tables in multi threaded applications, you must ensure that concurrent accesses into the hash table is free from race conditions, dead locks, and other problems that arise in program synchronization.

One solution to overcome this problem is to prevent concurrent accesses into the hash table altogether; i.e. prior to accessing the hash table, a thread acquires a lock, and then releases the lock after the access is complete. Such approach, although simple, is inefficient. The program demonstrated on this page implements an alternative solution, one which permits safe concurrent accesses into the hash table. In the approach implemented on this page, each hash location within a hash table is protected with a separate lock. Hence, multiple threads access the hash table concurrently as long as they are accessing different hash locations. For greater efficiency, this program also makes use of a thread pool.

==== 2. TECHNICAL DETAILS ====

This program was implemented in two parts; a server program and a client program. The server side of the program maintains a hash table of records and a pool of threads. The client program requests from the server program search records by sending record ids over a message queue. The server program then retrieves a request from the message queue, and wakes up a thread in the thread pool. That awakened thread then uses the id (sent from the client program) to retrieve the corresponding record from the hash table, and sends the found record from the server program to the client program over the message queue.

The server also reads a specified file from the commandline, which stores initial user data that is to be inserted and stored into the hash table. The incoming text file has the following format:

a unique numerical id 1 (an integer)
the first name 1 (a string)
the last name 1 (a string)
.
.
.
a unique numerical id N (an integer)
the first name N (a string)
the last name N (a string)

These three fields make up one single record for one individual. More than one record may be present in the incoming text file.

==== 3. SERVER ====

The server has the following structure and function:

Multi Threaded Process Server Flow Control SelectShow



The server program is invoked with the following commandline structure:

./Server [FILE NAME] [NUMBER OF THREADS] (e.g. ./Server database.dat 10)

The server is implemented below.

The client program has a much easier flow of control. It is implemented below.

==== 4. CLIENT ====

The client has the following structure and function:

Multi Threaded Process Client Flow Control SelectShow


The client program was designed to sleep for 1 second every time a new record is obtained from the server. This makes it so its easier for the user to see what is being displayed on the screen.

The client program is invoked with the following commandline structure:

./client

The client is implemented below.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

See sample files named server.cpp and client.cpp which illustrate interprocess communications using message queues. See file condvar.cpp which illustrates the use of condition variables. Finally, see file signal.cpp which illustrates the overriding of default signal handlers.

The following is sample output:
(Note: remember to include the initial records input file!)

SERVER OUTPUT:

./Server INPUT_Records_programmingnotes_freeweq_com.txt 26

** SERVER ID #1015808 SUCCESSFULLY ESTABLISHED

^C

Caught the CTRL-C
Shutting down the server connection..

CLIENT OUTPUT:

./Client

** CONNECTION TO SERVER ID #1015808 SUCCESS

ID = 243 Record = Graham Basil
ID = 7943 Record = Tobias Arie
ID = 3607 Record = Claire Amina
ID = 849 Record = Jetta Victoria
ID = 126 Record = Jeramy Tod
ID = 7483 Record = Vivan Krystal
ID = 8036 Record = Lilliam Harley
ID = 1901 Record = Kati Basil
ID = 3524 Record = Kenneth Perkins
ID = 5256 Record = Jodee Albertina
ID = 7065 Record = Marylou Donn
ID = 3951 Record = Ula Domitila
ID = 395 Record = Jaime Lilliam
ID = 9234 Record = Nigel Gene
ID = 4148 Record = Carmella Evelia
ID = 9340 Record = Sang Cherilyn
ID = 3834 Record = Jessica Freddy

** SERVER CONNECTION CLOSED..

C++ || File Copier Using Memory Mapped Files

The following is another homework assignment which was presented in an Operating Systems Concepts class. Using commandline arguments, the following is a program which implements a file copier using memory mapped files. This program makes use of the “mmap” function call provided on Unix based systems.

REQUIRED KNOWLEDGE FOR THIS PROGRAM

How To Use Memory Mapped Files
How To Get The Size Of A File
How To Create A File Of Any Size

==== 1. OVERVIEW ====

A memory mapped file is a segment of virtual memory which has been assigned a direct byte for byte correlation with some portion of a file or file like resource. The primary benefit of memory mapping a file is increasing I/O performance, especially when used on large files. Accessing memory mapped files is faster than using direct read and write operations. Memory mapped files are designed to simplify and optimize file access.

The program demonstrated on this page works with any file type (i.e: txt, cpp, jpg, png, mp4, flv, etc.) and copies the contents of one file, and saves it into another separate output file.

==== 2. TECHNICAL DETAILS ====

This program has the following flow of control:

1. The program is invoked with the source file name and the destination file name as commandline arguments.
2. The program uses the mmap() system call to map both files into memory.
3. The program uses the memory-mapped file memory to to copy the source file to the destination file.

This program is also modified in such a way that you cannot do an mmap on the same file. For example, if you have an input file “A” as the source file, the destination file “B” needs to have a different filename. If both the source and the destination files have the same filename, an error message is displayed, and the program exits. In order for this program to work, the source and the destination files must have different names.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

./mcp 1.png 2.png

The source file "1.png" is 42004 bytes

The destination file "2.png" has been created and is 42004 bytes

C++ || Snippet – How To Create A File Of Any Size

The following is sample code which demonstrates how to create a file of any size using the fseek and fwrite function calls.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

./createfileN createfile.txt 12

[FILE "createfile.txt" IS CREATED AND IS 12 BYTES IN SIZE]

C++ || Snippet – How To Display The Size Of A File Using Stat

The following is sample code which demonstrates the use of the “stat” function calls on Unix based systems.

The stat function returns information about a file. No permissions are required on the file itself, but-in the case of stat() and lstat() – execute (search) permission is required on all of the directories in path that lead to the file.

The “stat” system call returns a stat structure, which contains the following fields:

The following example demonstrates the use of the “st_size” struct parameter to display the size of a file.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

./Stat Stat.cpp

The file "Stat.cpp" is 1170 bytes

C++ || Snippet – How To Use Message Queues For Interprocess Communication

The following is sample code which demonstrates the use of the msgsnd, msgrcv, and msgget function calls for use with message queues on Unix based systems.

Message queues are used for interprocess communication. They provide two (or more) separate programs (i.e: sender and receiver) with the ability to interact with eachother. This is ideal because the sender and receiver do not need to interact with the message queue at the same time. Messages can be sent at one point in time, be placed on the queue until the receiver is ready for it, and then be removed at another point in time when the program(s) that service the queue are finally ready to receive a message.

This concept is different from threaded and forked process IPC procedures, which often times process data in a more streamlined manner. Rather than following a strict pattern as to when data is to be sent and received, queuing is a mechanism in which messages are held until the receiving application is ready to process them.

The example on this page demonstrates the above use of a message queue to transfer data between two separate programs. The sending program (client.cpp) sends an integer and string variable to the receiving program (server.cpp), which then displays that received data to the screen.

NOTE: The server program must be ran before the client program!

=== 1. Server.cpp (Receiver) ===

=== 2. Client.cpp (Sender) ===

QUICK NOTES:
The highlighted lines are sections of interest to look out for.

You can view all allocated message queues using the ipcs command. You can delete a message queue from command line using ipcrm -q [KEY SHOWN BY IPCS]. Message queues are a finite resource. If something goes wrong during the execution of your program, you must manually delete all your queues.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output.

SERVER OUTPUT:

The server has started!

Waiting for someone to connect to server id #753664 with the key 1258295474

someNumber = 0 buff = Message queues are awesome!
someNumber = 1 buff = Message queues are awesome!
someNumber = 2 buff = Message queues are awesome!
someNumber = 3 buff = Message queues are awesome!
someNumber = 4 buff = Message queues are awesome!
someNumber = 5 buff = Message queues are awesome!
someNumber = 6 buff = Message queues are awesome!
someNumber = 7 buff = Message queues are awesome!
someNumber = 8 buff = Message queues are awesome!
someNumber = 9 buff = Message queues are awesome!

Server is now shutting down!

CLIENT OUTPUT:

Successfully connected to server id #753664 with the key 1258295474

Now sending messages....Sending complete!

C++ || Snippet – Multi-Process Synchronization Producer Consumer Problem Using Threads

The following is sample code which demonstrates the use of POSIX threads (pthreads), aswell as pthread mutex and condition variables for use on Unix based systems.

The use of mutual exclusion (mutex) variables refers to the requirement of ensuring that no two processes or threads are in their critical section at the same time. Here, a critical section refers to the period of time in which a process accesses a shared resource, such as shared memory. This procedure highlights a classic example of a multi-process synchronization problem known as the producer–consumer problem.

The producer–consumer problem describes a scenario in which two processes (the producer and the consumer) share a common resource (i.e: a string buffer). In this scenario, the producer’s job is to generate a piece of data, update that data with the shared resource (the buffer), and repeat. At the same time, the consumer is consuming that shared resource (i.e: removing something from that same buffer) one piece at a time. The problem is to make sure that the producer wont try to manipulate that shared resource (i.e: add data into the buffer) while the consumer is accessing it; and that the consumer wont try to remove data from that shared resource (the buffer) while the producer is updating it.

In this instance, a solution for the producer can be to go to sleep if a synchronization conflict occurs, and the next time the consumer removes an item from the buffer, it can notify the producer to wake up and start to fill the buffer again. In the same way, the consumer can go to sleep when the producer is accessing it. The next time the producer puts data into the buffer, it wakes up the sleeping consumer and the process continues.

The example on this page demonstrates the use of the above solution using the “pthread_mutex_lock()” function call to limit access to the critical section, and the “pthread_cond_wait()” function call to simulate the sleeping process.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

To use pthreads, the compiler flag “-lpthread” must be set.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

The Parent is creating the producer and consumer threads..

Produced: 1
Consumed: 1

Produced: 2
Consumed: 2

Produced: 3
Consumed: 3

Produced: 4
Consumed: 4

Produced: 5
Consumed: 5

Produced: 6
Consumed: 6

Both threads have completed and have terminated!

The Parent is now exiting...

C++ || Snippet – How To Create And Use Threads For Interprocess Communication

The following is sample code which demonstrates the use of POSIX threads (pthreads), aswell as the pthread_create, and pthread_join function calls on Unix based systems.

Much like the fork() function call which is used to create new processes, threads are similar in that they too are used for interprocess communication. Threads allow for multi-threading, which is a widespread programming and execution model that allows for multiple threads to exist within the same context of a single program.

Threads share the calling parent process’ resources. Each process has it’s own address space, but the threads within the same process share that address space. Threads also share any other resources within that process. This means that it’s very easy to share data amongst threads, but it’s also easy for the threads to step on each other, which can lead to bad things.

The example on this page demonstrates the use of multiple pthreads to display shared data (an integer variable) to the screen.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

So, what is the difference between fork() processes and thread processes? Threads differ from traditional multitasking operating system processes in that:

• processes are typically independent, while threads exist as subsets of a process
• processes carry considerable state information, whereas multiple threads within a process share state as well as memory and other resources
• processes have separate address spaces, whereas threads share their address space
• processes interact only through system-provided inter-process communication mechanisms.
• Context switching between threads in the same process is typically faster than context switching between processes.

To use pthreads, the compiler flag “-lpthread” must be set.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

./Pthread 5

The Parent is creating 5 threads!
The Parent is now waiting for the thread(s) to complete...

Hi, Im thread #1 and this is my id number: 1664468736
Hi, Im thread #2 and this is my id number: 1647683328
Hi, Im thread #3 and this is my id number: 1656076032
Hi, Im thread #4 and this is my id number: 1672861440
Hi, Im thread #5 and this is my id number: 1681254144

All thread(s) are complete and have terminated!

The Parent is now exiting...

C++ || Snippet – How To Override The Default Signal Handler (CTRL-C)

The following is sample code which demonstrates the use of the “signal” function call on Unix based systems.

Signals are interrupts delivered to a process by the operating system which can terminate a program prematurely. You can generate interrupts by pressing Ctrl+C. The “signal” function call receives two arguments. The first argument is an integer which represents the signal number, and the second argument is a pointer to the user defined signal handling function.

The following program catches the “SIGINT” signal number using the signal() function.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

Please press CTRL-C

^C Haha I have 10 lives!
^C Haha I have 9 lives!
^C Haha I have 8 lives!
^C Haha I have 7 lives!
^C Haha I have 6 lives!
^C Haha I have 5 lives!
^C Haha I have 4 lives!
^C Haha I have 3 lives!
^C Haha I have 2 lives!
^C Haha I have 1 lives!
^C ** Ahh you got me...

C++ || Multi-Hash Interprocess Communication Using Fork, Popen, & Pipes

The following is another homework assignment which was presented in an Operating Systems Concepts class. Using two pipes, the following is a program which implements the computing of hash values on a file using the MD5, SHA1, SHA224, SHA256, SHA384, and SHA512 hashing algorithms provided on Unix based systems.

REQUIRED KNOWLEDGE FOR THIS PROGRAM

How To Use Fork
How To Use Pipe
How To Use Popen

==== 1. OVERVIEW ====

Hash algorithms map large data sets of variable length (e.g. files), to data sets of a fixed length. For example, the contents of a 1GB file may be hashed into a single 128-bit integer. Many hash algorithms exhibit an important property called an avalanche effect – slight changes in the input data trigger significant changes in the hash value.

Hash algorithms are often used for verifying the integrity of files downloaded from the WEB. For example, websites hosting a file usually post the hash value of the file using the MD5 hash algorithm. By doing this, the user can then verify the integrity of the downloaded file by computing the MD5 algorithm on their own, and compare their hash value against the hash value posted on the website. The user will know if the download was valid only if the two hash values match.

==== 2. TECHNICAL DETAILS ====

The following implements a program for computing the hash value of a file using the MD5, SHA1, SHA224, SHA256, SHA384, and SHA512 hashing algorithms provided on Unix based systems.

This program takes the name of the target file being analyzed as a command line argument, and does the following:

1. Check to make sure the file exists.
2. Create two pipes.
3. Create a child process.
4. The parent transmits the name of the file to the child (over the first pipe).
5. The child receives the name of the file and computes the hash of the file using the MD5 algorithm (using Linux program md5sum).
6. The child transmits the computed hash to the parent (over the second pipe) and terminates.
7. The parent receives the hash, prints it, and calls wait().
8. Repeat the same process starting with step 3, but using algorithms SHA1...SHA512.
9. The parent terminates after all hashes have been computed.

The use of the popen function is used in order to launch the above programs and capture their output into a character array buffer.

This program also uses two pipes. The two pipes created are the following:

(1) Parent to child pipe: Used by the parent to transfer the name of the file to the child. The parent writes to this pipe and the child reads it.

(2) Child to parent pipe: Used by the child to transfer the computed hashes to the parent. The child writes to this pipe and the parent reads it.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

Using the following example input file located here, the following is sample output:

C++ || Snippet – How To Use Popen & Save Results Into A Character Array

The following is sample code which demonstrates the use of the “popen” function call on Unix based systems.

The “popen” function call opens a process by creating a pipe, forking, and invoking the shell. It does this by executing the command specified by the incoming string function parameter. It creates a pipe between the calling program and the executed command, and returns a pointer to a stream that can be used to either read from or write to the pipe.

The following example demonstrates how to save the results of the popen command into a char array.

QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:


Program output: fa97c59cc414e42d4e0e853ddf5b4745 /bin/ls

C++ || Serial & Parallel Multi Process File Downloader Using Fork & Execlp

The following is another homework assignment which was presented in an Operating Systems Concepts class. The following are two multi-process programs using commandline arguments, which demonstrates more practice using the fork() and execlp() system calls on Unix based systems.

==== 1. OVERVIEW ====

File downloaders are programs used for downloading files from the Internet. The following programs listed on this page implement two distinct type of multi-process downloaders:

1. a serial file downloader which downloads files one by one.
2. a parallel file downloader which dowloads multiple files in parallel.

In both programs, the parent process first reads a file via the commandline. This file which is read is the file that contains the list of URLs of the files to be downloaded. The incoming url file that is read has the following format:

[URL1]
[URL2]
.
.
.
[URLN]

Where [URL] is an http internet link with a valid absolute file path extension.
(i.e: http://newsimg.ngfiles.com/270000/270173_0204618900-cc-asmbash.jpg)

After the url file is parsed, next the parent process forks a child process. Each created child process uses the execlp() system call to replace its executable image with that of the “wget” program. The use of the wget program performs the actual file downloading.

==== 2. SERIAL DOWNLOADER ====

The serial downloader downloads files one at a time. After the parent process has read and parsed the incoming url file from the commandline, the serial downloader proceeds as follows:

1. The parent forks off a child process.
2. The child uses execlp("/usr/bin/wget", "wget", [URL STRING1], NULL) system call in order to replace its program with wget program that will download the first file in urls.txt (i.e. the file at URL ).
3. The parent executes a wait() system call until the child exits.
4. The parent forks off another child which downloads the next file specified in url.txt.
5. Repeat the same process until all files are downloaded.

The following is implemented below:

Since the serial downloader downloads files one at a time, that can become very slow. That is where the parallel downloader comes in handy!

==== 3. PARALLEL DOWNLOADER ====

The parallel downloader downloads files all at once and is implemented much like the serial downloader. The parallel downloader proceeds as follows:

1. The parent forks off n children, where n is the number of URLs in url.txt.
2. Each child executes execlp("/usr/bin/wget", "wget", [URL STRING], NULL) system call where each is a distinct URL in url.txt.
3. The parent calls wait() (n times in a row) and waits for all children to terminate.
4. The parent exits.

The following is implemented below:


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

Also note, while the parallel downloader executes, the outputs from different children may intermingle.

C++ || Snippet – How To Use Fork & Pipe For Interprocess Communication

The following is sample code which demonstrates the use of the fork, read, and write function calls for use with pipes on Unix based systems.

A pipe is a mechanism for interprocess communication. Data written to a pipe by one process can be read by another process. Creating a pipe is achieved by using the pipe function, which creates both the reading and writing ends of the pipe file descriptor.

In typical use, a parent process creates a pipe just before it forks one or more child processes. The pipe is then used for communication between either the parent or child processes, or between two sibling processes.

A real world example of this kind of communication can be seen in all operating system terminal shells. When you type a command in a shell, it will spawn the executable represented by that command with a call to fork. A pipe is opened to the new child process, and its output is read and printed by the terminal.


QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

Parent is forking a child.
Parent is now waiting for child id #12776 to complete..

Starting the child process..

Message from the parent via the pipe: Greetings From Your Parent!

Program is now exiting...

The child process is complete and has terminated!

Program is now exiting...

C++ || Snippet – How To Use Fork & Execlp For Interprocess Communication

The following is sample code which demonstrates the use of the “fork” and “execlp” function calls on Unix based systems.

The “fork” function call creates a new process by duplicating the calling process; or in more simpler terms, it creates a duplicate process (a child) of the calling (parent) process.

This new process, referred to as the child, is an exact duplicate of the calling process, referred to as the parent.

The “execlp” function call is part of a family of functions which replaces a current running process image with a new process image. That means by using the “execlp” function call, you are basically replacing the entire current running process with a new user defined program.

Though fork and execlp are not required to be used together, they are often used in conjunction with one another as a way of creating a new program running as a child of another process.

QUICK NOTES:
The highlighted lines are sections of interest to look out for.

The code is heavily commented, so no further insight is necessary. If you have any questions, feel free to leave a comment below.

The following is sample output:

Parent is forking a child.
Parent is now waiting for child id #10872 to complete..

Starting the child process..

total 1466
-rw-r--r-- 1 admin admin 468 Apr 27 2012 nautilus-computer.desktop
-rwxrwxr-x 1 admin admin 9190 Aug 19 15:17 ForkExample
-rw-rw-r-- 1 admin admin 1640 Aug 19 15:17 ForkExample.cpp

The child process is complete and has terminated!

Parent is now exiting...