Saturday, 1 March 2014
Procedure to upgrade Linux 6.1 to 6.4 without CD from a local repository
Tuesday, 4 February 2014
How to configure the right amount of swap space in Linux
Linux and other Unix-like operating systems use the term "swap" to describe both the act of moving memory pages between RAM and disk, and the region of a disk the pages are stored on. It is common to use a whole partition of a hard disk for swapping. However, with the 2.6 Linux kernel, swap files are just as fast as swap partitions. Now, many admins (both Windows and Linux/UNIX) follow an old rule of thumb that your swap partition should be twice the size of your main system RAM. Let us say I've 32GB RAM, should I set swap space to 64 GB? Is 64 GB of swap space really required? How big should your Linux / UNIX swap space be?
Old dumb memory managers
I think the '2x swap space' rule came from Old Solaris and Windows admins. Also, earlier memory mangers were very badly designed. There were not very smart. Today, we have very smart and intelligent memory manager for both Linux and UNIX.
Nonsense rule: Twice the size of your main system RAM for Servers
According to OpenBSD FAQ:
Many people follow an old rule of thumb that your swap partition should be twice the size of your main system RAM. This rule is nonsense. On a modern system, that's a LOT of swap, most people prefer that their systems never swap. You don't want your system to ever run out of RAM+swap, but you usually would rather have enough RAM in the system so it doesn't need to swap.
Select right size for your setup
Here is my rule for normal server (Web / Mail etc):
Swap space == Equal RAM size (if RAM < 2GB)
Swap space == 2GB size (if RAM > 2GB)
Many Oracle experts recommend something as follows for heavy duty Oracle server with fast storage such as RAID 10:
Swap space == Equal RAM size (if RAM < 8GB)
Swap space == 0.50 times the size of RAM (if RAM > 8GB)
However, newer servers can have
Red Hat Recommendation
Red hat recommends setting as follows for RHEL 5:
The reality is the amount of swap space a system needs is not really a function of the amount of RAM it has but rather the memory workload that is running on that system. A Red Hat Enterprise Linux 5 system will run just fine with no swap space at all as long as the sum of anonymous memory and system V shared memory is less than about 3/4 the amount of RAM. In this case the system will simply lock the anonymous and system V shared memory into RAM and use the remaining RAM for caching file system data so when memory is exhausted the kernel only reclaims pagecache memory.
Considering that 1) At installation time when configuring the swap space there is no easy way to predetermine the memory a workload will require, and 2) The more RAM a system has the less swap space it typically needs, a better swap space
Systems with 4GB of ram or less require a minimum of 2GB of swap space
Systems with 4GB to 16GB of ram require a minimum of 4GB of swap space
Systems with 16GB to 64GB of ram require a minimum of 8GB of swap space
Systems with 64GB to 256GB of ram require a minimum of 16GB of swap space
Swap will just keep running servers...
Swap space will just keep operation running for a while on heavy duty servers by swapping process. You can always find out swap space utilization using any one of the following command:
cat /proc/swaps
swapon -s
free -m
top
See how to find out disk I/O and related information under Linux. In the end, you need to add more RAM, adjust software (like controlling Apache workers or using lighttpd web server to save RAM) or use some sort of load balancing.
Also, refer Linux kernel documentation for /proc/sys/vm/swappiness. With this you can fine tune swap space.
A note about Desktop and Laptop
If you are going to suspend to disk, then you need swap space more than actual RAM. For example, my laptop has 1GB RAM and swap is setup to 2GB. This only applies to Laptop or desktop but not to servers.
Kernel hackers need more swap space
If you are a kernel hacker (debugging and fixing kernel issues) and generating core dumps, you need twice the RAM swap space.
Conclusion
If Linux kernel is going to use more than 2GB swap space at a time, all users will feel the heat. Either, you get more RAM (recommend) and move to faster storage to improve disk I/O. There are no rules, each setup and configuration is unique. Adjust values as per your requirements. Select amount of swap that is right for you.
Monday, 6 May 2013
Setting up tftp server on Redhat Linux and allow write access to tftpboot
Thursday, 21 July 2011
How to kill the process running on specific port in Linux
'lsof -w -n -i tcp:8080'
or
'fuser -n tcp 8080'
or
'netstat -anp | grep :8080[[:blank:]]'
should show you the process ID.
If there is no (init) script to use to shut down the offending service (since TCP/8080 means proxy) to kill you can by PID 'kill -9 pidnumber'.
How to send email from the Linux command line
First run a quick test to make sure the “sendmail” application is installed and working correctly. Execute the following command, replacing “you@youremailid.com” with your e-mail address.
# mail -s “Hello world” you@youremailid.com
Hit the return key and you will come to a new line. Enter the text “This is a test from my server”. Follow up the text by hitting the return key again. Then hit the key combination of Control+D to continue. The command prompt will ask you if you want to mark a copy of the mail to any other address, hit Control+D again. Check your mailbox. This command will send out a mail to the email id mentioned with the subject, “Hello world”.
To add content to the body of the mail while running the command you can use the following options. If you want to add text on your own:
# echo “This will go into the body of the mail.”
mail -s “Hello world” you@youremailid.com
And if you want mail to read the content from a file:
# mail -s “Hello world” you@youremailid.com < /home/calvin/application.log
Some other useful options in the mail command are:
-s subject (The subject of the mail)
-c email-address (Mark a copy to this “email-address”, or CC)
-b email-address (Mark a blind carbon copy to this “email-address”, or BCC)
Here’s how you might use these options:
# echo “Welcome to the world of Calvin n Hobbes”
mail -s “Hello world” calvin@cnh.com -c hobbes@cnh.com -b susie.derkins@cnh.com
MUTT
One of major drawbacks of using the mail command is that it does not support the sending of attachments. mutt, on the other hand, does support it. I’ve found this feature particularly useful for scripts that generate non-textual reports or backups which are relatively small in size which I’d like to backup elsewhere. Of course, mutt allows you to do a lot more than just send attachments. It is a much more complete command line mail client than the “mail” command. Right now we’ll just explore the basic stuff we might need often. Here’s how you would attach a file to a mail:
# echo “Sending an attachment.”
mutt -a backup.zip -s “attachment” calvin@cnh.com
This command will send a mail to calvin@cnh.com with the subject (-s) “attachment”, the body text “Sending an attachment.”, containing the attachment (-a) backup.zip. Like with the mail command you can use the “-c” option to mark a copy to another mail id.
SENDING MAIL FROM A SHELL SCRIPT
Now, with the basics covered you can send mails from your shell scripts. Here’s a simple shell script that gives you a reading of the usage of space on your partitions and mails the data to you.
#!/bin/bash
df -h
mail -s “disk space report” calvin@cnh.com
Save these lines in a file on your Linux server and run it. You should receive a mail containing the results of the command. If, however, you need to send more data than just this you will need to write the data to a text file and enter it into the mail body while composing the mail. Here’s and example of a shell script that gets the disk usage as well as the memory usage, writes the data into a temporary file, and then enters it all into the body of the mail being sent out:
#!/bin/bash
df -h > /tmp/mail_report.log
free -m >> /tmp/mail_report.log
mail -s “disk and RAM report” calvin@cnh.com < /tmp/mail_report.log
Now here’s a more complicated problem. You have to take a backup of a few files and mail then out. First the directory to be mailed out is archived. Then it is sent as an email attachment using mutt. Here’s a script to do just that:
#!/bin/bash
tar -zcf /tmp/backup.tar.gz /home/calvin/files
echo
mutt -a /tmp/backup.tar.gz -s “daily backup of data” calvin@cnh.com
The echo at the start of the last line adds a blank into the body of the mail being set out.
This should get you started with sending mails form the Linux command line and from shell scripts. Read up the “man page” for both mail and mutt for more options.
Friday, 24 September 2010
sendmail and sm-client very slow to start
After changing the IP address of a server, at the time of booting up sendmail and sm-client took a very long time (about 5 minutes each) to start.
Usually when sendmail is sluggish starting - it's because it can't figure out hostname details.
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
above needs to be the very first line of /etc/hosts
if you are accepting email for a domain then it would help if you have
the fully qualified domain name set in /etc/sysconfig/network and that
name resolves via dns or at least is listed with an ip address
in /etc/hosts...
i.e.
# hostname
linuxserver.mydomain.com
# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=linuxserver.mydomain.com
# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.0.1 linuxserver.mydomain.com linuxserver
# host linuxserver.mydomain.com
linuxserver.mydomain.com has address 192.168.0.1
# cat /etc/mail/local-host-names
mydomain.com
linuxserver.mydomain.com
www.mydomain.com
localhost
localhost.localdomain
That's it !
Tuesday, 21 September 2010
Linux LVM tips
LVM Tool | Description |
pvcreate | Create physical volume from a hard drive |
vgcreate | Create logical volume group from one or more physical volumes |
vgextend | Add a physical volume to an existing volume group |
vgreduce | Remove a physical volume from a volume group |
lvcreate | Create a logical volume from available space in the volume group |
lvextend | Extend the size of a logical volume from free physical extents in the logical volume group |
lvremove | Remove a logical volume from a logical volume group, after unmounting it |
vgdisplay | Show properties of existing volume group |
lvdisplay | Show properties of existing logical volumes |
pvscan | Show properties of existing physical volumes |
Among typical operations:
A Walkthrough of the LVM for Linux
Adding a disk to the Volume Group
Next, we'll add /dev/hda6 to the Volume Group. Just type vgextend vg01 /dev/hda6 and you're done! You can check this out by using vgdisplay -v vg01. Note that there are now a lot more PEs available!
Moving Creating a striped Logical Volume
Note that LVM created your whole Logical Volume on one Physical Volume within the Volume Group. You can also stripe an LV across two Physical Volumes with the -i flag in lvcreate. We'll create a new LV, lv02, striped across hda5 and hda6. Type lvcreate -l4 -nlv02 -i2 vg01 /dev/hda5 /dev/hda6. Specifying the PV on the command line tells LVM which PEs to use, while the -i2 command tells it to stripe it across the two.
You now have an LV striped across two PVs!
Moving data within a Volume Group
Up to now, PEs and LEs were pretty much interchangable. They are the same size and are mapped automatically by LVM. This does not have to be the case, though. In fact, you can move an entire LV from one PV to another, even while the disk is mounted and in use! This will impact your performance, but it can prove useful.
Let's move lv01 to hda6 from hda5. Type pvmove -n/dev/vg01/lv01 /dev/hda5 /dev/hda6. This will move all LEs used by lv01 mapped to PEs on /dev/hda5 to new PEs on /dev/hda6. Effectively, this migrates data from hda5 to hda6. It takes a while, but when it's done, take a look with lvdisplay -v /dev/vg01/lv01 and notice that it now resides entirely on /dev/hda6!
Removing a Logical Volume from a Volume Group
Let's say we no longer need lv02. We can remove it and place its PEs back in the empty pool for the Volume Group. First, unmounting its filesystem. Next, deactivate it with lvchange -a n /dev/vg01/lv02. Finally, delete it by typing lvremove /dev/vg01/lv02. Look at the Volume Group and notice that the PEs are now unused.
Removing a disk from the Volume Group
You can also remove a disk from a volume group. We aren't using hda5 anymore, so we can remove it from the Volume Group. Just type vgreduce vg01 /dev/hda5 and it's gone!
A file system on logical volume may be extended. Also more space may be added to a VG by adding new partitions or devices with the command: vgextend. For example:
lvextend -L +4G /dev/VolGroup00/LogVol04
The command pvmove can be used in several ways to move any LV elsewhere. There are also many more commands to rename, remove, split, merge, activate, deactivate and get extended information about current PV's, VG's and LV's.
Here is a typical du map of a server with volume manager installed. As you can see all partitions except /boot partition are referred vi path /dev/mapper/VolGroup00-LogVolxx where xx is two digit number:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
4128448 316304 3602432 9% /
/dev/sda3 194449 22382 162027 13% /boot
none 2020484 0 2020484 0% /dev/shm
/dev/mapper/VolGroup00-LogVol05
4128448 42012 3876724 2% /home
/dev/mapper/VolGroup00-LogVol03
4128448 41640 3877096 2% /tmp
/dev/mapper/VolGroup00-LogVol02
8256952 3189944 4647580 41% /usr
/dev/mapper/VolGroup00-LogVol04
8256952 174232 7663344 3% /var
/dev/hde 594366 594366 0 100% /media/cdrecorder
Resiliency to renumbering of physical hard disks
LVM identifies PVs by UUID, not by device name.
Each disk (PV) is labeled with a UUID, which uniquely identifies it to the system. 'vgscan' identifies this after a new disk is added that changes your drive numbering. Most distros run vgscan in the lvm startup scripts to cope with this on reboot after a hardware addition. If you're doing a hot-add, you'll have to run this by hand I think. On the other hand, if your vg is activated and being used, the renumbering should not affect it at all. It's only the activation that needs the identifier, and the worst case scenario is that the activation will fail without a vgscan with a complaint about a missing PV.
The failure or removal of a drive that LVM is currently using will cause problems with current use and future activations of the VG that was using it.
How to get information about free space
vgdisplay shows logical volumes one by one and provides the information about free disk space on each:
vgdisplay volume_group_one | grep "Total PE"
# vgcreate vg01 /dev/hda2 /dev/hda10
Volume group "vg01" successfully created
How to create and mount a partition
- Create the partition with lvcreate
# lvcreate -L 5G -n data vg02
Logical volume "data" created
- Format partition
# mkfs -t ext3 /dev/vg02/data
- Make mount point and mount it
# mkdir /data
# mount /dev/vg02/data /data/
- Check results
# df -h /data
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/test--volume-data
50.0G 33M 5.0G 1% /data
- Add it to /etc/fstab
If one wishes to use all the free physical extents on the volume group one can achieve this using the lvm
lvextend command :
lvm lvextend -L +4G /dev/VolGroup00/LogVol04 # extend /var
ext2online /dev/VolGroup00/LogVol04
Option -l operates with free extents . This adds the 7153 free extents to the logical volume:
# lvm lvextend -l+7153 /dev/TestVG/TestLV
Extending logical volume TestLV to 30.28 GB
Logical volume TestLV successfully resized
"lvextend -L +54 /dev/vg01/lvol10 /dev/sdk3" tries to extend the size of that logical volume by 54MB on physical volume /dev/sdk3. This is only possible if /dev/sdk3 is a member of volume group vg01.
Then the pvcreate command is used to create the new physical volume using the new partition, and the pvs again to verify the new physical volume. See redhat.com Knowledgebase
How to remove LVM partionon
Use lvremove to Remove a logical volume from a logical volume group, after unmounting it
syntax:
lvremove [-A/--autobackup y/n] [-d/--debug] [-f/--force] [-h/-?/--help] [-t/--test] [-v/--verbose] LogicalVolumePath [LogicalVolumePath...]
lvremove removes one or more logical volumes. Confirmation will be requested before deactivating any active logical volume prior to removal. Logical volumes cannot be deactivated or removed while they are open (e.g. if they contain a mounted filesystem).
Options.
-f, --force
Remove active logical volumes without confirmation.
EXAMPLES
Remove the active logical volume lvol1 in volume group vg00 without
asking for confirmation:
lvremove -f vg00/lvol1
Remove all logical volumes in volume group vg00:
lvremove vg00
TIPS:
Missing drives in a volume group have caused me a lot of problems. If you just want to excommunicate the drive from your vg, the following works:
vgreduce --removemissing vg00
# pvscan
Couldn't find device with uuid 'pQMQJ3-Rq3M-kQD9-M6DB-UWLi-Dyk4-3pWoa7'.
Couldn't find device with uuid 'pQMQJ3-Rq3M-kQD9-M6DB-UWLi-Dyk4-3pWoa7'.
PV unknown device VG milan lvm2 [30.00 GB / 29.90 GB free]
PV /dev/sdg VG milan lvm2 [30.00 GB / 29.90 GB free]
# vgscan
Reading all physical volumes. This may take a while...
Couldn't find device with uuid 'pQMQJ3-Rq3M-kQD9-M6DB-UWLi-Dyk4-3pWoa7'.
Couldn't find all physical volumes for volume group milan.
Couldn't find device with uuid 'pQMQJ3-Rq3M-kQD9-M6DB-UWLi-Dyk4-3pWoa7'.
Couldn't find all physical volumes for volume group milan.
Volume group "milan" not found
# lvscan
Couldn't find device with uuid 'pQMQJ3-Rq3M-kQD9-M6DB-UWLi-Dyk4-3pWoa7'.
Couldn't find all physical volumes for volume group milan.
Couldn't find device with uuid 'pQMQJ3-Rq3M-kQD9-M6DB-UWLi-Dyk4-3pWoa7'.
Couldn't find all physical volumes for volume group milan.
Volume group "milan" not found
LINKS:
More information about LVM at http://www.softpanorama.org/Commercial_linuxes/logical_volume_manager.shtml
Friday, 14 May 2010
How to delete old files in Linux
find /opt/app/logs -mtime +XX -exec rm {} \;
For example, to delete files older than 90 days:
find /opt/app/logs -mtime +90 -exec rm {} \;
If the same directory has multiple type of files, and want to delete ONLY those with .log extension:
find /opt/app/logs -name '*.log' -mtime +90 -exec rm {} \;
If you want to see the files to be deleted, run the same command removing "-exec rm {} \;"
Tuesday, 20 April 2010
MOUNT USB STICK WITH READ AND WRITE PERMISSIONS FOR A USER
If you just mount the usb stick without special options it is only read/writable for root.
To read/write enable the stick for a different user use the following:
mount /dev/sda1 /mnt/usbstick/ -o uid=500
User ID 500 is on many systems the default ID for your first user. You might want to switch it to another user (take a look at the ID in /etc/passwd) or set permissions for a group. For a group use "gid" instead of "uid".
It also might be handy to add a line like the following to your /etc/fstab file (where you can specify some default mount information):
/dev/sda1 /mnt/usbstick vfat noauto,users,exec,rw,umask=000 0 0
This allows the members of the group "users" to mount the device /dev/sda1 (which represents the usb stick on my system) with read and write access. Another advantage of this line in your /etc/fstab is that for some graphical environments (e.g. KDE) read the file at startup and present you a corresponding icon to mount and unmount the device automatically on your desktop.
Friday, 11 December 2009
Bulk delete from Postfix queue
# postsuper -d BA4491827DE
If there is only one message to delete, I can live with that. However, when there’s a bunch of messages (e.g. from a particular domain) you need to delete from the queue, the above method simply too much of a hassle (well, unless you want to delete *everything*, which would be #postsuper -d ALL). Postfix does not have a function for doing that. Luckily, a search on Google yielded this Perl script that does exactly what I want, removing message(s) from queue based on my keyword. Here is the content of that Perl script called “delete-from-mailq”:
#!/usr/bin/perl
$REGEXP = shift || die “no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!”;
@data = qx;
for (@data) {
if (/^(\w+)(\*|\!)?\s/) {
$queue_id = $1;
}
if($queue_id) {
if (/$REGEXP/i) {
$Q{$queue_id} = 1;
$queue_id = “”;
}
}
}
open(POSTSUPER,”|postsuper -d -”) || die “couldn’t open postsuper” ;
foreach (keys %Q) {
print POSTSUPER “$_\n”;
};
close(POSTSUPER);
Save the above script to a file say “delete-queue” in your home directory, and make it excutable:
# chmod 755 delete-queue
Usage
Delete all queued messages from or to the domain “iamspammer.com”
./delete-queue iamspammer.com
Delete all queued messages to specific address “bogususer@mydomain.com”
./delete-queue bogususer@mydomain.com
Delete all queued messages that begin with the word “bush” in the e-mail address:
./delete-queue bush*\@whateverdomain.com
Delete all queued messages that contain the word “biz” in the e-mail address:
./delete-queue biz
That's it.
Thursday, 10 December 2009
Self-Signed IIS SSL Certificates using OpenSSL
This tutorial assumes that you have a Linux box with OpenSSL installed,and that you want to create a self-signed certificate for IIS5.0 / 6.0
Set up your CA (you only have to do this once)
ON THE LINUX BOX…
Create a private key
openssl genrsa -des3 -out CA.key 1024
(You’ll need to supply a passphrase. DON’T FORGET THIS!!)
Set this to read-only for root for security
chmod 400 CA.key
Create the CA certificate
openssl req -new -key CA.key -x509 -days 1095 -out CA.crt
(Provide appropriate responses to the prompts…for Common Name, you might want to use something like “OurCompany CA”)
Set the certificate to read-only for root for security
chmod 400 CA.crt
Obtain a CSR
ON THE IIS BOX…
Open the Internet Manager
Select the site for which you want to create a key
Right-click and choose Properties
Select the “Directory Security” tab
Click the “Server Certificate” button
Follow the prompts to create a CSR
Save
your CSR, then transfer it to the Linux box for further processing.
(For the following steps, we’ll refer to your CSR as “new.csr”)
Sign the CSR
ON THE LINUX BOX…
Sign the CSR (all of this on one line)
openssl x509 -req -days 365 -in new.csr -CA CA.crt
-CAkey CA.key -CAcreateserial -out new.crt
Transfer the new.crt file back to the IIS box
Install self-signed certificate
ON THE IIS BOX…
Open the Internet Manager
Select the site to install the key
Right-click and choose properties
Select the “Directory Security” tab
Click the “Server Certificate” button
Specify that you want to complete the pending request
Select the .crt file that you just transferred
That’s it!
PS: If you have problems with the certification path (”does not chain up to a
trusted root certificate” in System Log), the following seems to fix it:
1. Internet Information Services -> select the site -> right click -> Properties
2. Directory Security -tab -> Secure communications -frame -> Edit…
3. Select “Enable certificate trust list”, click New… -> Next
4. Add from file -> use CA.crt -> Next
5. Write something to the Name and Description, if you like. -> Next -> Finish
Tuesday, 17 November 2009
Tripwire generating too big report files
I found the problem is that there were many changes applied to this server, and files copied/moved,
so the tripwire DB holding the changes grew up a lot.
The only command to run and update the tripwire DB is:
tripwire --update -Z low
This command will compare your database against your current file system and then launch an editor so that you can choose to make changes to your database.
If you try this command but get an error message about a missing report file, the reason is most likely that the last check was not run immediately prior to the update. The report file in the /var/lib/tripwire/report directory is named by hostname, then date (yyyymmdd) then time. If you have recently run a check and want the update to proceed using your most recent report file, then use the -r option and provide the report filename that you want the update to use.
tripwire --update -Z low --twrfile host-yyyymmdd-tttttt.twr
If it asks for a password, you'll have to set it up first, unless you already know the password:
tripwire --local-passphrase mypassword
Then, run the sync again, now you know the password.
Converting Ext2 Filesystems to Ext3
When you see the partition type, 83 is used for a Linux partition, no matter if it is ext2 or ext3.
The Ext3 filesystem is an Ext2 filesystem with a journal file and some filesystem driver additions making the filesystem journalized.
Converting from Ext2 to Ext3
The conversion procedure is simple enough. Imagine /dev/sdb1 mounted as /data – the procedure would be as follows:
Log in as root
Make sure /etc/fstab has /dev/sdb1 mounted to /data as ext2, read write
umount /dev/sdb1
If you can't unmount it, then remount it read only (mount -o remount,ro /dev/sdb1)
tune2fs -j /dev/sdb1
Edit /etc/fstab, and for /dev/sdb1, change ext2 to ext3
mount /dev/sdb1 /data, or mount -a
Check if the partition was correctly mounted:
mount | grep /dev/sdb1
If it's not shown as ext3, reboot (shutdown -r now)
if still not, troubleshoot ...
Otherwise, you're done.
A few explanations are in order.
The tune2fs command creates the journal file, which is kept in a special inode on the device (by default). You then must change the /etc/fstab entry to reflect it's a journalling filesystem, and then mount it.
You can check a full article describing other procedures in:
http://www.troubleshooters.com/linux/ext2toext3.htm
Monday, 16 November 2009
Accurate Date and Time in Linux using ntp step by step instructions
Why do We Need a Precise Clock?
If our computer never connects to other computers (or other devices that use a clock), the precision of the clock is not critical itself, it depends on the need of the user. However, programs that some way use the net are dependent on a precise date and time. Some examples, when you may need precise clock:
Softwares that deal with transactions
Commercial applications (e.g. eBay)
Mail and messaging-related client and servers
Websites that use cookies
Distributed web applications
Web services
Distributed component-based applications as J2EE, .NET, etc
Advanced modern and paralel filesystems, as AFS, DFS, GFS, GPFS, etc
And of course, to use the computer to adjust our wristwatch clock.
Computer Global Date and Time Concept
To determine the current time for some planet region, a computer needs exactly these two pieces of information:
Correct UTC (universal time as in Greenwich, but not GMT) time
Region's current Time Zone
For computers, there is also the hardware clock, which is used as a base by the OS to set its time.
OS date and time (we'll use only date or time from now on) is set on boot, by some script that reads the hardware clock, makes Time Zone calculations (there is no time zone data stored in BIOS) and sets the OS. After this synchronization, BIOS
and OS time are independent from each other. So after a while they may have some seconds of difference. Which one is correct?
If you don't make any special configuration, none of them.
We'll discuss here how to make them both globally 100% accurate.
Time Zones
Time Zones are a geographical world globe division of 15o each, starting at Greenwich, in England, created to help people know what time is it now in another part of the world.
Nowadays it is much more a political division than geographical, because sometimes people needs to have the same time as other people in not-so-far locations. And for energy savings reasons, we have today the Daylight Savings Time, that are also a Time
Zone variation.
Time Zones are usually defined by your country government or some astronomical institute, and is represented by 3 or 4 letters.
Use the timezoneconverter.com to know what time is it now at any part of the globe.
Daylight Savings Time
For energy savings reasons, governments created the Daylight Savings Time. Our clocks are forwarded one hour, and this makes our days look longer. In fact, what really happens is only a Time Zone change. The primitive time (UTC) is still, and will always be, the same.
Time Zone Mechanism on Linux
Linux systems uses the GLIBC dynamic Time Zones, based on /etc/localtime. This file is a link to (or a copy of) a zone information file, usually located under /usr/share/zoneinfo directory.
To make it effective, you only have to link (or copy) the zoneinfo file to /etc/localtime. In some distributions, there is a higher level (and preferred) way to set the Time Zone, described later.
After making /etc/localtime pointing to the correct zoneinfo file, you are already under that zone rules and DST changes are automatic -- you don't have to change time manually.
Accurate Global Time Synchronization
To have accurate time in all your systems is as important as having a solid network security strategy (achieved by much more than simple firewall boxes). It is one of the primary components of a system administration based on good practices, which leads to organization and security. Specially when administering distributed applications, web-services, or even a distributed security monitoring tool, accurate time is a must.
NTP: The Network Time Protocol
We won't discuss here the protocol, but how this wonderful invention, added to the pervasivenes of the Internet, can be useful for us. You can find more about it at www.ntp.org.
Once your system is properly setup, NTP will manage to keep its time accurate, making very small adjustments to not impact the running applications.
People can get exact time using hardware based on atom's electrons frequency. There is also a method based on GPS (Global Positioning System). The first is more accurate, but the second is pretty good also. Atomic clocks require very special and
expensive equipment, but their maintainers (usually universities and research labs) connect them to computers, that run a NTP daemon, and some of them are connected to the Internet, that finally let us access them for free. And this is how we'll
synchronize our systems.
Building a Simple Time Synchronization Architecture
You will need:
1) A direct or indirect (through a firewall) connection to the Internet, to syncrhonize our servers with a public accurate NTP server.
2) Choose some NTP servers. You can use the public server pool.ntp.org, or choose some from the stratum 2 public time servers on NTP website. If you don't have an Internet access, your WAN administrator (must be a clever guy) can provide you some internal addresses.
3) Have the NTP package installed in all systems you want to synchronize. You can find RPMs in your favorite Linux distribution CD, or make a search on rpmfind.net.
Local Relay Servers for NTP
If you have several machines to synchronize, do not make them all access the remote NTP servers you chose. Only 2 of your server farm's machines need access remote NTP servers, and the other machines will sync with these 2. We will call them the Relay Servers.
Your Relay Servers can be any machine already available in your network. NTP consumes low memory and CPU. You don't need a dedicated machine for it.
The Correct Settings for Your Linux Box
For any OS installation, you must know your Time Zone. This is expressed in terms of a city, a state or a country. You must also decide how to set BIOS time, and we may follow two strategies here:
Linux Only Machine
In this case you should set BIOS time to UTC time. DST changes will be dynamically managed by Time Zone configurations.
Dual Boot Linux and MS Windows Machine
Windows handles time in a more primitive way than Linux. For Windows, BIOS time is allways your local time, so DST changes are more aggressive because they directly change hardware clock. And since both Linux and Windows initially get and set time from the hardware, when they are together, Linux must handle it in the same way. So set BIOS time to your localtime.
Step 1, Setting the Zone Info file:
Your time zone is defined by /etc/localtime
If you are unsure of the file you have is the correct one, you can check if its the same as your time zone:
diff -b /usr/share/zoneinfo/America/Vancouver /etc/localtime
If it doesn't, you will have to remove it and create the link:
#rm /etc/localtime
#ln -s /usr/share/zoneinfo/America/Vancouver /etc/localtime
Alternatively, instead of creting the symbolic link, you can just copy the file:
#rm /etc/localtime
#cp /usr/share/zoneinfo/America/Vancouver /etc/localtime
Step 2, Setting the Time Zone:
On Red Hat Linux and derived systems, you can set the hardware clock strategy and Time Zone using the timeconfig command:
#timeconfig
that shows a user-friendly dialog to select your time zone, and apply the changes.
You can also use it non-interactively:
#timeconfig "America/Vancouver" # set HC to localtime, and TZ to America/Vancouver
or
#timeconfig --utc "America/Vancouver" # set HC to UTC, and TZ to America/Vancouver
In any case, this utility changes /etc/sysconfig/clock file that is read at boot time.
You can edit it by hand, and that is how it looks:
#cat /etc/sysconfig/clock
ZONE="America/Vancouver"
UTC=true
ARC=false
Step 3, Setting the Hardware Clock
I encourage you to set your hardware clock only after understanding how to get accurate time.
The hwclock command reads and sets the hardware clock, based on several options you give to it, documented in its man page.
But you don't have to use it if you have a modern Linux distribution. After defining your hardware clock strategy and Time Zone, you can use the high level setclock command to correctly set your hardware clock.
You don't need to pass any parameters because setclock intelligently calls hwclock to set the BIOS based on your OS current date and time. So you should always use the setclock command.
#hwclock --systohc --utc # set HC with UTC time based on OS current time (I personally use this one)
You can also use any of these options:
#setclock # The easy way to set HC
#hwclock # reads HC
#hwclock --systohc # set HC with local time based on OS current time
#hwclock --set --date "21 Oct 2004 21:17" # set HC with time specified on stringSince the OS time is independent from the hardware clock, any BIOS change we make will take place in the next boot.
Another option to change HC is rebooting and accessing your computer BIOS screens, but as you see, there is no need to do it in Linux!
Step 4, Configure NTP protocol
Of course, as a prerequisite, you need to install the ntp package.
The only file to configure is /etc/ntp.conf, doesn't matter if you are configuring a client, or a server (like a local relay server to serve your local network), you will only need an additional keyword.
Change the following parameters in /etc/ntp.conf:
To make it a local relay server:
1) First we specify the servers you're interested in:
server otherntp.server.org # A stratum 1 server at server.org
server ntp.research.gov # A stratum 2 server at research.gov
2) Restrict the type of access you allow these servers. In this example the servers are not allowed to modify the run-time configuration or query your Linux NTP server.
restrict otherntp.server.org mask 255.255.255.255 nomodify notrap noquery
restrict ntp.research.gov mask 255.255.255.255 nomodify notrap noquery
The mask 255.255.255.255 statement is really a subnet mask limiting access to the single IP address of the remote NTP servers.
3) As this server is also going to provide time for other computers, such as PCs, other Linux servers and networking devices, then you'll have to define the networks from which this server will accept NTP synchronization requests. You do so with a
modified restrict statement removing the noquery keyword to allow the network to query your NTP server. The syntax is:
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
In this case the mask statement has been expanded to include all 255 possible IP addresses on the local network.
4) We also want to make sure that localhost (the universal IP address used to refer to a Linux server itself) has full access without any restricting keywords:
restrict 127.0.0.1
5) Save the file and restart NTP for these settings to take effect.
6) To get NTP configured to start at boot, use the line:
# chkconfig ntpd on
You can now configure other Linux hosts on your network to synchronize with this new master NTP server in a similar fashion.
To make it a client accessing your brand new local relay server:
1) First we specify the servers you're interested in, you can use a host name or an ip address; it is recommended to use a host name, if later you have to change their ip address, you don't have to change anything in your client computers:
server ntp1.mynetwork.com # Your first local ntp server
server ntp2.mynetwork.com # Your second local ntp server
2) Restrict the type of access you allow these servers. In this example the servers are not allowed to modify the run-time configuration or query your Linux NTP server.
restrict ntp1.mynetwork.com mask 255.255.255.255 nomodify notrap noquery
restrict ntp2.mynetwork.com mask 255.255.255.255 nomodify notrap noquery
The mask 255.255.255.255 statement is really a subnet mask limiting access to the single IP address of the remote NTP servers.
3) As this server is NOT going to provide time for other computers, there is no need to allow others to query it, then include the noquery statement:
restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap noquery
In this case the mask statement has been expanded to include all 255 possible IP addresses on the local network.
4) We also want to make sure that localhost (the universal IP address used to refer to a Linux server itself) has full access without any restricting keywords:
restrict 127.0.0.1
5) Save the file and restart NTP for these settings to take effect.
6) To get NTP configured to start at boot, use the line:
# chkconfig ntpd on
Step 5: First sync:
It's a good practice to syncronize manually the first time, because if the time difference is too big, it will not be synchronized automatically.
For your local ntp server:
#service ntpd stop
#ntpdate otherntp.research.gov
#service ntpd start
For the rest of your servers/guests ntp clients:
#service ntpd stop
#ntpdate ntp1.mynetwork.com
#service ntpd start
To check the sync status, you can use:
#ntpq -p
Enjoy, and have A GOOD TIME !!!
Monday, 6 April 2009
Rescan dynamically the scsi bus (applicable to CX Clariion SAN infrastructure)
I've been working for a while with a Dell - Clariion CX-300, and the best way to add new attached LUNs was always to reboot the server.
However, that procedure is not always the most acceptable if you're in a hurry or if just want to do some tests.
I found the procedure described above, in an outdated website, but worked very well in my case.
I also recommend to use rescan-scsi-bus.sh script with the options -lwc. Type rescan-scsi-bus.sh --help to see the description of each option.
The original link is: http://www.it-sudparis.eu/s2ia/user/procacci/Doc/AX100/AX100-en008.html
After initialization ends, the server doesn't see the new devices :-( I tried a script from http://www.linux1394.org/scripts/rescan-scsi-bus.sh that should dynamically rescan the bus, but with no sucess.
$ /root/rescan-scsi-bus.sh
Host adapter 1 (qla2xxx) found.
Host adapter 2 (qla2xxx) found.
Scanning for device 1 0 0 0 ...
OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: LUNZ Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 2 0 0 0 ...
OLD: Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: LUNZ Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
0 new device(s) found.
0 device(s) removed.
So I stoped powerpath and unload qla modules in order to restart the whole thing.
$ /etc/init.d/PowerPath stop
Stopping PowerPath: done
$ lsmod | grep qla
qla6312 119233 0
qla2xxx 165733 1 qla6312
scsi_transport_fc 12225 1 qla2xxx
scsi_mod 116941 5 sg,qla2xxx,scsi_transport_fc,megaraid_mbox,sd_mod
[root@pasargades /opt/Navisphere/bin]
$ modprobe -r qla6312 qla2xxx
[root@pasargades /opt/Navisphere/bin]
$ lsmod | grep qla
then reload the whole thing:
$ modprobe qla2xxx qla6312
[root@pasargades /opt/Navisphere/bin]
$ /etc/init.d/PowerPath start
Starting PowerPath: done
then it works, the kernel does see the new devices
$ cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 06 Lun: 00
Vendor: PE/PV Model: 1x2 SCSI BP Rev: 1.0
Type: Processor ANSI SCSI revision: 02
Host: scsi0 Channel: 01 Id: 00 Lun: 00
Vendor: MegaRAID Model: LD 0 RAID1 69G Rev: 521S
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: RAID 5 Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 01
Vendor: DGC Model: RAID 5 Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi4 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: RAID 5 Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
Host: scsi4 Channel: 00 Id: 00 Lun: 01
Vendor: DGC Model: RAID 5 Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
[root@pasargades /opt/Navisphere/bin]
$ fdisk -l
Disk /dev/sda: 73.2 GB, 73274490880 bytes
255 heads, 63 sectors/track, 8908 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 4 32098+ de Dell Utility
/dev/sda2 * 5 583 4650817+ 83 Linux
/dev/sda3 584 1220 5116702+ 83 Linux
/dev/sda4 1221 8908 61753860 5 Extended
/dev/sda5 1221 3770 20482843+ 83 Linux
/dev/sda6 3771 5682 15358108+ 83 Linux
/dev/sda7 5683 6192 4096543+ 82 Linux swap
Disk /dev/sdb: 676.4 GB, 676457349120 bytes
255 heads, 63 sectors/track, 82241 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc: 1395.8 GB, 1395864371200 bytes
255 heads, 63 sectors/track, 169704 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 676.4 GB, 676457349120 bytes
255 heads, 63 sectors/track, 82241 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/sde: 1395.8 GB, 1395864371200 bytes
255 heads, 63 sectors/track, 169704 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/emcpowera: 676.4 GB, 676457349120 bytes
255 heads, 63 sectors/track, 82241 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/emcpowera doesn't contain a valid partition table
Disk /dev/emcpowerb: 1395.8 GB, 1395864371200 bytes
255 heads, 63 sectors/track, 169704 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/emcpowerb doesn't contain a valid partition table
REmarque: We can see that fdisk sees double path 'raw' devices ( /dev/sdb and /dev/sdd ) to a same device, which finnaly is presented by powerpath as /dev/emcpowera . All disk system command (fdisk etc ...) should now use that device in order to benefit the use of powerpath (load balancing and failover on our double attached FC ).//
The 'rescan' script shows that now:
$ /root/rescan-scsi-bus.sh
Host adapter 3 (qla2xxx) found.
Host adapter 4 (qla2xxx) found.
Scanning for device 3 0 0 0 ...
OLD: Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: RAID 5 Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
Scanning for device 4 0 0 0 ...
OLD: Host: scsi4 Channel: 00 Id: 00 Lun: 00
Vendor: DGC Model: RAID 5 Rev: 0208
Type: Direct-Access ANSI SCSI revision: 04
0 new device(s) found.
0 device(s) removed.
Monday, 23 March 2009
How to run an application a daemon? Example with nsca Nagios utility
I also did it some time ago, to make another utility work as a daemon, but today I want to share the info.
I’ll have to write a utility application that would start the process, store it’s PID in some file and somehow “daemonize” the forked process.
Let’s assume that the application we want to run is /usr/local/nagios/nsca
First, we need to create a script in /etc/init.d/ Let us name the file /etc/init.d/nsca
The file contents would look something like this:
#!/bin/bash
#
# /etc/rc.d/init.d/nsca
#
# Control to start/stop nsca utility as a daemon
# chkconfig: 345 99 01
# description: NSCA passive alerts writer for Nagios
#
# Author: Edwin Salvador (edwin.salvador@gmail.com)
#
# Changelog:
# 2009-03-23
# - First version of the script
#
# processname: nsca
# Source function library.
. /etc/init.d/functions
test -x /usr/local/nagios/nsca || exit 0
RETVAL=0
prog="NSCA passive alerts writer for Nagios"
start() {
echo -n $"Starting $prog: "
daemon /usr/local/nagios/nsca -c /usr/local/nagios/nsca.cfg
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/nsca
echo
}
stop() {
echo -n $"Stopping $prog: "
killproc /usr/local/nagios/nsca
RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/nsca
echo
}
#
# See how we were called.
#
case "$1" in
start)
start
;;
stop)
stop
;;
reload|restart)
stop
start
RETVAL=$?
;;
condrestart)
if [ -f /var/lock/subsys/nsca ]; then
stop
start
fi
;;
status)
status /usr/local/nagios/nsca
RETVAL=$?
;;
*)
echo $"Usage: $0 {condrestart|start|stop|restart|reload|status}"
exit 1
esac
exit $RETVAL
This is a pretty standard service start/stop/restart file. This small script will take care of controlling the PID and work as any other standard service, you don't need to remove any PID file manually, nor event to kill the process.
It can also be easyly modified to match any other program / application / script under Linux you want to daemonize !!.
Remember to assign user and group "root", and chmod 755
Use a different and unique name to each new startup script you create.
You're all set. Just use it as any usual service you run:
START:
#service start nsca
STOP:
#service stop nsca
RESTART:
#service restart nsca
If you need it to startup automatically on system boot-up:
#chkconfig nsca on
Enjoy it !
Thursday, 12 March 2009
Sudo: allow a normal user to run commands as root under Linux / UNIX operating systems
You need to use sudo command which is use to execute a command as another user. It allows a permitted user to execute a command as the superuser or another user, as specified in the /etc/sudoers (config that defines or list of who can run what) file. i.e. the sudo command allows users to do tasks on a Linux system as another user.
sudo is more more secure then su command. By default it logs sudo usage, command and arguments in /var/log/secure (Red Hat/Fedora / CentOS Linux) or /var/log/auth.log (Ubuntu / Debian Linux).
If the invoking user is root or if the target user is the same as the invoking user, no password is required. Otherwise, sudo requires that users authenticate themselves with a password by default (NOTE: in the default configuration this is the user's password, not the root password). Once a user has been authenticated, a timestamp is updated and the user may then use sudo without a password for a short period of time (15 minutes unless overridden in sudoers).
/etc/sudoers Syntax
Following is general syntax used by /etc/sudoers file:
USER HOSTNAME=COMMAND
Where,
USER: Name of normal user
HOSTNAME: Where command is allowed to run. It is the hostname of the system where this rule applies. sudo is designed so you can use one sudoers file on all of your systems. This space allows you to set per-host rules.
COMMAND: A simple filename allows the user to run the command with any arguments he/she wishes. However, you may also specify command line arguments (including wildcards). Alternately, you can specify "" to indicate that the command may only be run without command line arguments.
How do I use sudo?
For example, you want to give user rokcy access to halt/shutdown command and restart apache web server.
1) Login as root user
2) Use visudo command edit to edit the config file:
# visudo
3) Append the following lines to file:
rokcy localhost=/sbin/halt
rokcy dbserver=/etc/init.d/apache-perl restart
4) Save the file and exit to shell prompt.
5) Now rokcy user can restart apache server by typing the following command:
$ sudo /etc/init.d/apache-perl restart
Output:
Password:
Restarting apache-perl 1.3 web server....The sudo command has logged the attempt to the log file /var/log/secure or /var/log/auth.log file:
# tail -f /var/log/auth.log
Output:
May 13 08:37:43 debian sudo: rokcy : TTY=pts/4 ; PWD=/home/rokcy ; USER=root
If rokcy want to shutdown computer he needs to type command:
$ sudo /sbin/halt
Output:
Password:Before running a command with sudo, users usually supply their password. Once authenticated, and if the /etc/sudoers configuration file permits the user access, then the command is run. sudo logs each command run and in some cases has completely supplanted the superuser login for administrative tasks.
More examples
a) Specify multiple commands for user jadmin:
jadmin ALL=/sbin/halt, /bin/kill, /etc/init.d/httpd
b) Allow user jadmin to run /sbin/halt without any password i.e. as root without authenticating himself:
jadmin ALL= NOPASSWD: /sbin/halt
c) Allow user charvi to run any command from /usr/bin directory on the system devl02:
charvi devl02 = /usr/bin/*
Source:
http://www.cyberciti.biz/tips/allow-a-normal-user-to-run-commands-as-root.html