Linux World: July 2012

offsetof: Using the offsetof funcion in c

The "offsetof" function in c is useful in determining the offset of members with in a structure or union.
It is defined in the header file "stddef.h".
The syntax is

Arguments:

Example :

Let say we have a structure foo

The structure "foo" has a integer member which generally would of 4 bytes on a 32 bit system and a chat member which would be one byte.
Thus the offset of the member "ch" in the structure "foo" would be the number of bytes occupied by "int" which is "4".

Here is a example code which uses "offsetof" to find the offset of ch in the structre foo.

Compile the code and execute it

Try changing adding new members to the structure foo and see how the result changes depending on the members added.

Try with the structure :

The offset is the size of the array which is 4 X 10 = 40.

Answers to linux history quiz

1.In which year was the linux kernel released under GNU GPL license for the first time?
Ans.1992

2.The penguin was chosen as the official mascot for linux in
Ans.1996

3. The linux foundation is the merger of
Ans. OSDL and Free standards group

4. Which famous computer scientist said this in 1992 "Linux is obsolete"?
Ans. Andrew S. Tanenbaum

5. Linux 1.0 was released in
Ans. 1994

6. When did microsoft first contribute to linux source code ?
Ans. 2009

7. Which of the following is a linux based company and was started in 1994 ?
Ans. Redhat

8. Which of the following companies invested a billion dollars in to linux in the year 2001 ?
Ans. IBM

9. In which year did Linus Torvalds appear on the cover of Business week ?
Ans. 2005

10. Which of the following Linux Desktop environment (GUI) was started in the year 1997 ?
Ans. GNOME

expand: To convert a tab space to space characters

expand: Command used to convert the tab space to separate space characters.

Example:

Let us say we we have a file "temp" with the following contents.

temp:

In each line we have one tab space, which is equivalent to 8 space characters. To convert this one tab space to 8 space characters we can run the command expand on the file.

The new file "temp_1" looks the same as the file "temp" but if we open the file to edit it, we will encounter 8 space characters instead of tab on every line.

To change the number of space characters from 8 to any number of our choice we can use the option "-t".

We can see that the tab has been replaced by single space. The number that you specify after "-t" will be the number of spaces that will put instead of the tab space.

We can also restrict the command to work only at the beginning of a file by passing the option "-i"

Example :

Let us modify the file temp and add a tab at the beginning of the first line.

temp:

Now use expand on the file along with option "-i"

We can see that only the tab at the beginning of the first line has been removed,but the rest of the tabs are intact.

tr: To translate characters

The command "tr" can be used to translate/change or delete characters from the input.

By default tr takes two arguments, the first one being the set of characters to be searched for and the second argument being the set of characters to be replaced with . The default input is the standard input.

Example:

The command waits for user to enter a string and then hit the return key. In the above example on entering "xyx" the character x has been replaced by character y which was what was passed as arguments to the command.

We can also pass multiple characters as arguments and each characters will be considered separately and not as a string and each of the characters in argument1 will be replaced by the corresponding character in the argument2

Example:

If the string length of argument1 is greater than that of argument2 then the last character of argument2 is applied to all the characters of argument1 that are after the string length of argument2.

Example :

As we can see both y and z are replaced by b.

We can change this behavior by passing the option "-t" which will truncate the characters of first argument to the same as the length of argument2.

Thus we can see, "y" and "z" both are not replaced by character.

The command can be applied on files by using the input redirection.

Example:

Let say we have a file,temp, with following contents

temp:

We can see that all o have been replaced with O.

Instead of translating, we can delete characters too by passing the option "-d" .

Example:

Thus we can see that all the digits in the file have been deleted.

Other examples:

We can user "tr" to convert all the characters in a file to upper case too.

Using tr over a range of characters

Deleting repetitive characters

gzip: Compressing files using gzip

gzip is a command that can be used to compress and decompress files. The command takes one or more files as input and compresses each of them separately.

The new compressed file is created with a ".gz" extension, i.e. if we compressed the file "temp", the compressed file will be named temp.gz and the compressed file will replace the original file.

The command works only on files and if we want to compress folders using gzip , we can convert a folder into a single "tar" archive and use the gzip command on the archive.

Example :

Let us say we have a folder called temp. The size of the folder can be found using the command du.

We can convert the folder into a ".tar" file using the command tar

Note : Using tar command is given in the post :Using tar to archive files

We can see that the "tar" command only archives but does not compress the file.Now we can use the gzip command on the tar file to compress it.

We can see that the original file "temp.tar" has been replaced by the compressed file.

The size of the compressed file is much smaller than the original file, thus clear that gzip compresses the file
The files compressed using gzip can be decompressed using gzip with option "-d" or with the command guzip. By default gzip creates the decompressed file with the name of the original file with ".gz" removed i.e temp.tar.gz will become temp.tar and will replace the original temp.tar.gz Example

If temp.tar already exists in the folder gzip prompts the uses to confirm ifthe previous file has to be overwritten, and only on reciveing the confirmation will it overwrite the existing file

We can also change the suffix from ".gz" to any other characters we wish by using the option "-S"

Example :

While decompressing the files that are compressed with custom suffix we need to pass the suffix also.

If we want to find out the size of the decompressed file before decompressing we can use the option "-l".

bzip2: Compressing files using bzip2

The command bzip2 can be used to compress files and reduce their size. The command accepts one or more files as input on the command line and compresses each of them separately.

Each compressed file is stored with the same name along with ".bz2" added as the extension. By default the original file is replaced unless explicitly specified to retain the original file.

The command works of files only, thus if we want to compress a folder using bzip2 we need to convert it to a file using the tar command and then compress it using bzip2.

Example:
Let us say we have a folder called temp. The size of the folder can be found using the command du.

We can convert the folder into a ".tar" file using the command tar

Note: Using tar command is given in the post : Using tar to create an archive

We can see that the "tar" command only archives but does not compress the file.Now we can use the bzip2 command on the tar file to compress it.

We can see that the original file "temp.tar" has been replaced by the compressed file.

The size of the compressed file is much smaller than the original file, thus clear that bzip2 compresses the file.s

To retain the original files while compressing using bzip2 we need to use the option "-k"

The files compressed using bzip2 can be decompressed using bzip2 with option "-d" or with the command bunzip2. By default bzip2 creates the decompress file with the name of the original file with ".bz2" removed i.e temp.tar.bz2 will become temp.tar.

Example

If temp.tar already exists in the folder bzip2 throws an error and stops with out overwriting the file. We can force it to overwrite the existing file by using the option "-f".

Again the original ".bz2" file is replaced with the decompressed file, and the option "-k" can be used to retain the compressed file.

tar: Using the tar command to create an archive.

The tar command can be used to create a archive of multiple file and folders, making it easier to move or copy the files in one go.

Example:

Let us say we have three files, file1,file2,file3. We can create one tar file by combining all three using the command tar.

The options mean :

c is to create
v is to make the command verbose, that is print what ever files are being archived
f is the use the file name being passed on the command line.

We can list the contents of the archive using. the "-t" option.

The option v and f mean the same as above.
The option "t" is to list or you can read it as tell the contents of the archive.
If we create a tar archive a of a directory, it maintains the folder structure intact, will produce the same folder structure when the archive is untared.

Example :

If we have a folder, temp , with three files, file1,file2,file3 and a sub folder temp1. We can create a tar archive of the same as follows.

Once an archive is created, if we update the folder which we archived we need to create a new archive but instead just update the already created archive.

Example:

In the previous example we created an archive for folder temp. If we add a new file,file4, to the folder we can update the archive a follows.

The tar command only updates the new file that was added to the folder.
If folder to be archived has symbolic links, then by default tar, copies the link only and not the original contents.

Example:

Let us create a soft link to the folder temp, used in the previous examples, and try to tar the link.

After creating the softlink, we used the tar command on the soft link to create the archive. And when we list the contents of the archive created, we can see it does not have the files and folders of the original file but only the link file.

To make sure that tar copies the original folder that the link is pointing to we need to pass the option "-h" , i.e.

We can see that after passing the option "-h", the archive contains all the original files and folders.
Once "tar" archive is created, we can extract the contents of the archive by using the option "-x"

Example:

To get the files archived under temp.tar in the above example we need to run

The name of the folder created will the same as that of the original folder that was archived.

using printk_ratelimit

printk is used in kernel programming to log messages and is very useful while debugging the kernel programs.
But the kernel log being though big, is restricted and unnecessary logging at times can lead to loosing relevant messages .

Thus kernel provides a function, printk_ratelimit , to restrict the logging using which we can set a limit on the number of prints that we want our program to do.

The limit on the number of prints is set in the file /proc/sys/kernel/printk_ratelimit_burst

The printk_ratelimit function will allow 10 prints before it starts blocking the further prints.
The printk_ratelimit, returns 1 as long as the number of prints do not exceed the limit. Once the limit is reached it starts returning 0.
Thus the function can be used as a condition for an "if" statement to decide whether to print a message or not.
The printk will be enabled again after a time interval in seconds mentioned in the file /proc/sys/kernel/printk_ratelimit

Which means, printk will be disabled for 5 seconds after 10 messages by the code which uses the printk_ratelimit function.

In the example module below we put 20 prints in the init function using a for loop.
Before each print, printk_ratelimit is called to decide whether to print or not.
When the module gets inserted, we will notice that only the first 10 prints are printed and the rest gets suppressed.

printk_limit.c :

Makefile:

Compile and load the module

We can see that only the first 10 prints have been done, the rest have been suppressed.

paste: Join multiple files into one

paste command can be used to join multiple files and display on the standard output or redirect it to a new file.

For e.g. let us take the following two example files

file1

file2

These two files can be joined as follows

The files are joined line by line and the default delimiter between the lines is a TAB space. This can be changed using the option "-d".

The output can be sent to anothe file using output redirection.

Instead of joining the files line by line it can be made to join seuentially i.e. one file after another using the option "-s"

The number of input files can be more than too also, the same rules apply to all of them .

printk and console log level.

printk is used in kernel programming to print messages into the kernel logs.

The syntax of printk is

The log levels decide the importance of the message being printed, kernel defines 8 log levels in the file printk.h

We can see each log level corresponds to a number and the lower the number higher the importance of the message.
The levels are useful in deciding what should be displayed to the user on the console and what should not be.

Every console has log level called as the the console log level and any message with a log level number lesser than the console log level gets displayed on the console, and other messages which have a log level number higher or equal to the console log level are logged in the kernel log which can be looked into using the command "dmesg".

The console loglevel can be found by looking into the file /proc/sys/kernel/printk

The first number in the output is the console log level, the second is the default log level, third is the minimum log level and fourth is the maximum log level.

Log level 4 corresponds to KERN_WARNING. Thus all the messages with log levels 3,2,1 and 0 will get displayed on the screen as well as logged and the messages with log level 4,5,6,7 only get logged and can be viewed using "dmesg".

The console log level can be changed by writing into the proc entry

Now the console log level is set to 6, which is KERN_INFO.

We can test logging by using the following module

hello.c:

The printk called in the init function uses KERN_WARNING which is log level and lesser than 6 which is the console log level and hence should be seen on the screen.

The printk used in the exit function is KERN_INFO which is log level 6,same as the console log level, and hence should not be visible on the screen.

Note: We can test the operation of the code only by logging into a text mode as none of the messages are displayed on a terminal of GUI.

Makefile:

Compile and insert

We can see the hello world being printed on the screen.

The good bye world message gets logged but is not printed on the screen but can be see in the logs.

Thus using printk and the console log levels we can control the kernel messages visible to the user.

cut: To pick parts of lines from a file.

The command "cut" can be used to extract parts of a lines from a file.
The syntax of the command is

Let us take the following example file and see how to use the various options of cut.

temp:

The option "-b" allows us to pick specific byte or a number of bytes from each line.

example :

The range need not start from the beginning i.e. "1", it could be any range of bytes or even a single byte from the line.

The -c option allows us to pick a specific character or range of characters from every line.

example

We can also specify a range of characters

The default delimiter for the command "cut" is a tab space, i.e any sequence of characters separated by one tab space is treated as a field and different fields can be selected using the option -f.
The default delimiter can be changed using the option -d.

example:

In the above example, we have passed a single space as an argument to the option "-d" thus the command treat the single space as delimiter and every sequence of characters which are separated by a space are numbered starting from 1, in veery line . Thus when we pass the number 2 to the option "-f" we are requesting for the second field which is what gets displayed on the screen.

We can change the output to characters that don't, match the options we are passing by using the options --complement.

example

Thus the parts of the line which did not match the options passed to the command are printed on the screen.
The output delimiter is treated as same as the input delimiter unless specified using the option --output-demlimiter

example

The fields are separated by an ":" instead of a space.

Unknown symbol find_task_by_pid_ns

You will come across this error

if you are trying to write a module using the function find_task_by_pid_ns and your kernel version is above 2.6.30.
This is because in kernels after 2.6.30 this symbol is no longer exported and instead of this the function pid_task needs to be used.

Ref: http://kerneltrap.org/mailarchive/git-commits-head/2009/6/18/6028583

Here is an example module for using the pid_task to find out a process from its pid. "Module to find a task from its pid"

Module to find a task from its pid

Here is a module which find the task_struct of a process from its pid. To find the task_struct of a process we can make use of the function pid_task defined in kernel/pid.c .

Arguments :

To find the pid structure if we have the pid of a process we can use the functionfind_get_pid which is also defined in kernel/pid.c

In the below module we create a read/write proc entry named task_by_pid. Which ever process we want to find the task_struct of using its pid we can write to number into the proc entry.

When we read the proc entry, it will display the name of the process corresponding to the pid we wrote into it.

proc_task_pid:

#include <linux/module.h>
#include <linux/kernel.h>
#include <asm/uaccess.h>
#include <linux/fs.h> 
#include <linux/cdev.h>
#include <linux/proc_fs.h>
#include <linux/pid.h>

#include <linux/pid_namespace.h>
int p_id;
struct pid *pid_struct;
struct task_struct *task;
static struct proc_dir_entry *proc_write_entry;
char *proc_name="task_by_pid";

int read_proc(char *buf,char **start,off_t offset,int count,int *eof,void *data )
{
int len=0;
pid_struct = find_get_pid(p_id);
task = pid_task(pid_struct,PIDTYPE_PID);

len = sprintf(buf,"\nname %s\n ",task->comm);

return len;
}

int write_proc(struct file *file,const char *buf,int count,void *data )
{
int ret;
char *id;
id = (char *)kmalloc(1000*sizeof(char),GFP_KERNEL);
printk(KERN_INFO "buf passed %s",buf);
if(copy_from_user(id,buf,count))
    return -EFAULT;
printk(KERN_INFO "id passed %s",id);
p_id = simple_strtoul(id,NULL,0);
printk(KERN_INFO "pid %d ret %d",p_id,ret);
return sizeof(buf);
}

void create_new_proc_entry()
{
proc_write_entry = create_proc_entry(proc_name,0666,NULL);
if(!proc_write_entry)
      {
    printk(KERN_INFO "Error creating proc entry");
    return -ENOMEM;
    }
proc_write_entry->read_proc = read_proc ;
proc_write_entry->write_proc = write_proc;
printk(KERN_INFO "proc initialized");

}

int proc_init (void) {
    create_new_proc_entry();
    return 0;
}

void proc_cleanup(void) {
    printk(KERN_INFO " Inside cleanup_module\n");
    remove_proc_entry(proc_name,NULL);
}
MODULE_LICENSE("GPL");   
module_init(proc_init);
module_exit(proc_cleanup);

Use the following make file to compile it:

Insert it into the kernel

Now let us try to find the name of the process with pid "1", which is always init.

As expected the output is "init". Thus we can find the task_struct of any process using its pid.

csplit : Split a file based on patterns

The command "csplit" can be used to split a file into different files based on certain pattern in the file or line numbers.

For example let us say we have the file, temp, with the following contents

temp:

we can split the file into two new files ,each having part of the contents of the original file, using csplit.

The syntax of csplit is

Pattern as integer number:

When the pattern is an integer number it makes cplit to copy line upto that line number,no including the line, into a new file and contents after that into a new file.

Example:

The numbers printed after the command is executed is the number of bytes written into each new file that got created. After the execution of the command we can see that we have two new files "xx00" and "xx01", these are the files created by csplit.

By looking at the contents of xx00 and xx01 we can observe that csplit has split the first 4 lines of file temp into one file and lines 5 till end of temp to another file .

The names of the new files created are by default of the format "xx00", "xx01"... etc.

Pattern as a Regular expression:

We can also pass regular expressions as patterns to split the file.

Example

We have passed the pattern as the string "two", which means we want to split the file temp at the first occurrence of the the string two.

As we can see, the file xx00 only has the first line which is before the line containing the string "two" .

Repeating a pattern:

A pattern can be made to repeat any number of times by passing the integer after the pattern with in {} i.e

csplit will then split the original file into integer number of new files, if the pattern repeats integer number of times else it will split maximum number of times possible which is less than the integer passed.

Examples:

In the above command we passed "1" as the pattern and 7 as integer for repeat. Thus the file was split 7 times with one line in each file.
The repetition can be made to occur as many times as possible by passing "*" instead of a number.

Example:

The above command split the file temp into a new file on each occurrence of the string /Line/ .

The output file prefix can be changed by using the option -f

As we can see what ever string is passed after -f will be used as the prefix.
The number of digits used in the suffix can be changed using the option -n

As we can see the number of digits after prefix "file" is only one, which was passed with the option "-n".

Finding the kernel version (release)

To find out the kernel version or the release that we are currently running we can make use of the command "uname".

uname prints various system related information and to get the kernel version specifically we need to pass the options "-r".

If we pass the option "-a" the command gives the following inmformations Kernel name,Node name on network,kernel release,kernel version,machine,processorand operating system.

Each of these can also be obtaned individually just like the kernel-release by passing the correct options, which are listed in the man pages.

split: To split a file based on line or byte size

The command "split" can be used to split a file into parts or pieces of various sizes based on byte size or number of lines.
Let us say we have a file, temp, with the following contents

temp:

We can use the command split to create new files with parts of the above file in it, by default split breaks a file into different files in the size of 1000 bytes, i.e. one new file for every 1000 bytes of data in the original file.

The new files created by split are named by default with a prefix "x" followed by the alphabets in the format xaa,xab etc.

example:

We can see that along with temp, we have new file xaa. As the file temp does not have more than 1000 bytes of data it was not broken up into multiple files.

We can reduce the size of number of bytes per file by using the option -b.

Example:

We can see that there are 9 files created, each having 10 bytes of data from the original file temp.

By using byte size as the limiting factor we can not be sure as to how much of each line gets split. As we can see in the above example,file xaa has first line and one character of the next line.

To make sure that the splitting happens at exact line boundaries we can use the option -l along with the number of lines per file.

example:

Each file created will have exactly one line of the file "temp".

If we don't like using alphabets to be used with "x" for the names of the files, we can use numbers by using the options -d

The prefix "x" can also be changed to whatever prefix we need by passing the prefix after the file name.

example :

We can see that by passing the string "file" after the filename "temp" we could create the new files with prefix as file.

readonly arguments in a script

The read command is used in scripting to accept an input from the user and store it in a variable. The value assigned to the variable read using the command read can be changed any number of times in the script.

If we want to have a variable whose value remains fixed through out the script and which can not be modified by any assignments once a value is assigned to it, we can make use of command "readonly".

For example in the script below we are reading the variable "var", as readonly and then try to change the value by assigning a new value to it.

Save it,give execute permissions and execute the script.

When the script reaches the fourth line, it throws an error

Clearly indicating that we are not allowed the change the value of "var".

Linux World

Pages