Neko

Virtual Google Search Appliances? Sweet.

A good friend of mine, Eric Lightbody works for the University of Wisconsin Green Bay as a Web Developer. He was playing with their Google Mini Search Appliance (GSA), and I was getting jealous. I had played with the earlier Google Search Appliances a few years earlier at Time Warner Cable, but now I wanted to play with one again.

I was thinking… I wonder if they have a Virtual Appliance for people to play with it before buying one. A Google search later, I found what I was looking for.

Enter the Google Search Appliance Virtual Edition.  This is distributed by Google as a VMware Virtual Machine.   Within about an hour, I had my Virtual GSA up and running and indexing all of my websites including some sites that had NTLM based protected content.   Pretty neat.  It would have been much quicker, but the decompress process (p7zip) took like 30 minutes on a Core 2 Quad Ubuntu workstation, leaving me with about a 40GB VMDK.

The Virtual Edition is meant for development use.  Legally, you are not allowed to use it for any production use.  Plus, it’s limited to 50,000 indexed files.   I would highly recommend a real Google Mini search appliance.  But if you already have one,  the Virtual Edition would be great for testing website search integration changes before implementing them on your real GSA.  And, if you are thinking about getting one, trying out the Virtual Edition will allow you to get your feet wet, and experiment with it’s features.

I would strongly recommend giving it plenty of Virtual Resources when you fire this up.  I starved mine, at first, from the resources it wanted, and it performed a little sluggishly (to be expected).

Also, after booting it up, make sure you wait a bit before trying to connect to it.  It seemed to take a good 5 to 10 minutes before I could point a browser at it to connect — again, this could be related to me not giving it much memory.  I gave it 1GB, and the requirements said 3GB was the minimum needed.

Enjoy.

StorageTek 5320 Command Line Cheat Sheet

The StorageTek 5320 NAS Appliances are pretty neat devices.  I find it necessary to utilize the command line interface (CLI) quite often. Unfortunately, the documentation for the 5320’s CLI is pretty limited — at least I couldn’t find anything in the manual talking about each command.

So, for your enjoyment, I’ve summarized each command below.  Having everything on one page will save you the trouble of going through each one doing “help command” to find out what each does! (Which is what I seem to have to do every time I hop on one of these and try to remember which command does what).

Command Description Usage
? Alias for the help command
approve manipulate approve file approve [reload | list | test | help]
aratewrite write contents of a file (async), report performance aratewrite FILENAME [+OFFSET] TOTALKB [BLOCKSIZE] [MB_PER_COMMIT]
arp show arp table arp [-d] [ -a | DEST | -i Interf ]
audit Audit Enable Command audit –help for details
auditconfig Audit Configuration Command auditconfig –help for details
cacls Display an access control list (ACL) cacls pathname
calog Compliance Audit Log Utility calog –help
cat show contents of a file cat filename
chattr change extended attributes chattr [-RV] <[-+iumq] | [[+r] [[[[[cc]yy]mm]dd]HH]MM][.ss]>
chgrp change file group chgrp [-R] group file…
chkpnt checkpoint management command chkpnt sub-commands [parameters] …
chkpntabort summarily abort checkpoints on a volume chkpntabort VOLUME
chkpntls list checkpoints chkpntls VOLUME …
chkpntmk make (create) a checkpoint chkpntmk VOLUME CHECKPOINT-NAME
chkpntmv move (rename) a checkpoint chkpntmv VOLUME OLD-CHECKPOINT-NAME NEW-CHECKPOINT-NAME
chkpntoff stop checkpoints on volume(s) chkpntoff VOLUME …
chkpnton show checkpoint status / start checkpoints on volume(s) chkpnton [VOLUME ...]
chkpntrm remove (delete) a checkpoint chkpntrm VOLUME CHECKPOINT-NAME/PATTERN
chmod change file mode chmod [-R] mode file…
chown change file owner chown [-R] owner[:group] file…
chsmb Remove UNIX attribute update restrictions chsmb pathname
cls clear the screen cls
cp copy files cp [-v] [-c] [-r] src dst
crashlocks simulate file lock manager crash crashlocks
datalock show or reset multi-user rpc locks datalock {show | reset }
date display or set date and time date [[[[[cc]yy]mm]dd]HH]MM]
dbg Enable or disable the debugger dbg [ -i [ enable | disable ] | -s ]
del delete files del FILE …
df show (disk) volumes and free space df
diag Save diagnostic file in /dvol/diagnostic diag
disk show disks / edit disk partition table disk -help for details
dtq Manage directory tree quotas dtq help (for usage)
du estimate file space usage du [-s] [-h] [PATH] …
echo echo a message echo WORD …
em Set and show speed & duplex mode for Intel NIC em [set,show] NICname
exit exit command interpreter (close connection) exit
find find files bigger than size (in MB), owned by uid or gid find PATH [-name NAME] [-size SIZE] [-uid UID] [-gid GID]
fsck check file volumes fsck VOLUME …
fsctl Filesystem control command fsctl command [sub-command] volume
fszap Zap Partition fszap disk partition
grouplook lookup group by name or GID grouplook GROUPNAME/UID …
gunzip uncompress file gunzip FILE
gzip compress file gzip FILE
halt halt system halt [-fm]
hbainfo Display HBA Information hbainfo [-a]
help show available commands help [COMMAND ...]
hostlook lookup hosts by name/address hostlook HOSTNAME/ADDR …
hostmatch match hosts by name/address hostmatch HOSTPATTERN HOSTNAME/ADDR …
ioapic Program the IO-APIC ioapic prog/show/list/mask/umask apicid intpin [level polarity vector]
iscsi iSCSI service control iscsi
isp2suspend Suspend file system I/O for maintainance isp2suspend
keymon monitor keys keymon STOPCHAR
ldapcfg Configure NISLDAP client ldapcfg [-i | -d | -e]
ll long list directory contents (ls -lz) ll [-Ra] [dir]
lm License Manager Command lm –help for details
ln create hard/symbolic links ln [-s] target link
load report loaded modules/load a module load [MODULE]
loadvars load variables from values file loadvars
locks file lock manager report locks
logtail show last few log messages logtail [NLINE|all]
ls list directory contents ls [-Ralz] [file]
lsattr display extended attributes lsattr [-Radv]
luninfo Display LUN Information luninfo
maccnvrt Convert resource.frk directories maccnvrt share-name
man show contents of manpage with pagination man [-ah] [section] name …
mbstat print mbuf stats mbstat [-sv]
menu use the menus menu
mkdir make directories mkdir [-m mode] directory …
more show contents of a file with pagination more filename
mount Mount volume(s) mount [-rwfuh] [-o options] [VOLUME [...]]
ndmp NDMP service control ndmp
netserv set network service options netserv {enable|disable|status} {service name} [options]
netstat show routing/interface tables/network addresses and ports netstat [-rin]
nfsmount Inspect, establish, or change an NFS volume nfsmount [LOCALNAME [SERVER:PATH] [OPTION=VAL ...]]
nfsstat nfsstat nfsstat NFS statistics
nfsunmount NFS unmount a remote directory nfsunmount LOCALNAME
nic Set and show speed & duplex mode for Network NIC nic [set,show] NICname
niscfg Configure NIS+ client niscfg [ -i | -k ]
oplocks Configure oplocks oplocks [enable | disable] volume-name
password set password for admin password
ping ping hosts by name/address ping [-c count] HOSTNAME/ADDR …
poolstat show object pool status poolstat poolname …
praudit Audit Print Command praudit –help for details
pwtest test admin password pwtest
quit alias for exit quit (see exit)
quota show a quota / set quota limits quota [-g|-u] /VOLUME|/* NAME|ID|NAME/ID [bh=KB] [bs=KB] [fh=N] [fs=N]
ratecopy copy a file, report performance ratecopy SOURCE_FILENAME DEST_FILENAME [BLOCKSIZE]
ratecpu cpu performance tests ratecpu
ratemem memory performance tests ratemem
ratenet rate network performance ratenet HOSTNAME/ADDR [NSEND [SIZE [MAXFAIL]]]
rateread read contents of a file, report performance rateread FILENAME [+OFFSET] [[BLOCKSIZE] [TOTALKB]]
ratewrite write contents of a file, report performance ratewrite FILENAME [+OFFSET] TOTALKB [BLOCKSIZE]
reboot reboot system reboot [-fm]
recover perform head/controller recovery recover {head|ctlr}
repquota report quotas repquota [-g][-u][-n][-e|-E][-q|-v] VOLUME|QUOTAFILE …
revoke revoke client mounts revoke HOSTNAME/ADDR …
rm remove files rm [-v] [-r] path
rmdir remove directories rmdir directory …
rotatelog rotates partner log file rotatelog
route manipulate routing table route [-n] COMMAND [ [ MODIFIERS ] [ DEST GATEWAY ] ]
rpcinfo report RPC information rpcinfo
savecore save a core dump of the operating system savecore savecore [ -chi | -fpkmv directory ]
savevars save variables to values file savevars
scanvars scan variables for printer, modem, and network changes scanvars
sdate display the secure date sdate [-d]
set set a system variable set NAME VALUE
show show system variables show [NAME/PATTERN ...]
shutdown shutdown the system shutdown [-fm]
snmp SNMP trap destination control snmp
source read commands from a file source FILENAME
sp show stack trace for thread ID sp Thread_ID
stat stat files stat FILE …
status system status report status [hi|si|ai|ti|dm|fd|fs|rm|up|mm|all ...]
strmstat report list of associated named streams strmstat file-path
su login in as privileged user su
sync_sys Synchoronize system files sync_sys Source-Head-ID
syslog set local syslog parameters syslog timestamp=[0|1|2] debug=[on|off]
tar TAR Command tar –help for details
telnet telnet to host by name/address telnet HOSTNAME/ADDR
top Display active threads top [-b] [-i secs] [-t cnt] [-l]
tsp Display threads and their stack tsp [-s] [-t cnt] [-l] thread_name
umount Unmount volume(s) umount VOLUME [...]
uname print system information uname
unload unload a module unload MODULE
unset unset a system variable unset NAME
uptime show how long system has been up and cpu usage uptime
userlook lookup user by name or UID userlook USERNAME/UID …
version show software version version
vscan Virus Scan command vscan {help|check|stats}
xj extended journal functions xj -help for details
zic timezone utility zic [help | zone | dumpzones | list | ]

The Command Line Interface (CLI) is always handy, and usually the first place I check out. It’s especially handy in a clustered environment as you can login to one of your NAS heads, and then telnet from there to the heartbeat interface of the other NAS head. This is useful when your volumes are still failed over to the surviving NAS head along with it’s main IP address.

The 5320’s “menu” command will bring up a nice menu that lets you do most (all?) of the things you can do in the GUI.

One other note about the GUI. It appears that only one person can be logged in at a time. So if you are in it, and keep getting disconnected, it’s probably because someone else is trying to access it too. What you are seeing is a ping-pong effect of two (or more) people getting kicked out, and logging back in. Utilizing the CLI over telnet or SSH does not have this problem.

I may go into details about some of these commands in the future.

Enjoy!

Determining if your kernel and hardware is 32bit or 64bit

I’m often asked by end-users, developers, and fellow system administrators, how to determine if the installed OS is 32bit or 64bit.   I’m also asked if the hardware is capable of 64 bit, 32 bit, or both.   Here is how to tel by OS:

HPUX

# getconf KERNEL_BITS
64

This will tell you if your currently running kernel is 64 bits or 32 bits.  It returns the number of bits used by the kernel for pointer and long data types.  Current values include 32 and 64.

# getconf HW_32_64_CAPABLE
1

Returns which kernel is supported on the hardware.

You can also test this with a short C program:

# cat check3264.c
#include <stdio.h>
#include <unistd.h>

main()
{
long ret = sysconf(_SC_HW_32_64_CAPABLE);

if (_SYSTEM_SUPPORTS_ILP32OS(ret) != 0) {
printf(”system supports 32-bit OS\n”);
}

if (_SYSTEM_SUPPORTS_LP64OS(ret) != 0) {
printf(”system supports 64-bit OS\n”);
}
}
# cc check3264.c
# ./a.out
system supports 64-bit OS
#

In this example, we see the test system, an HP L2000-36 running HPUX 11.11, only supports a 64 bit HPUX Kernel.

# getconf HW_CPU_SUPP_BITS
64

This will show you if the CPU’s are capable of running 32, 64, or 32/64 bit kernels.

Solaris

For solaris, the isainfo -v should, at least for versions past 2.6,  show you want you need:

# isainfo -v
32-bit i386 applications

This may also say “sparc” if you are running on the sparc architecture. If you were on a 64 bit system, you would see something like this:

64-bit sparcv9 applications
32-bit sparc applications

This would be telling you that you are running a 64 bit kernel and can run either 64-bit sparcv9 applications, or 32-bit sparc applications.

To determine the cpu’s bit size capabilities, isainfo -b does the trick here too:

# isainfo -b
64

AIX

For AIX, we will use the bootinfo command.

bootinfo -y
32

This show is the hardware is 32 bit or 64 capable.

bootinfo -K
32

Shows the running kernel’s bit size.

Linux

For linux, we will look at the cpuinfo from /proc. Here, we are mainly interested in the “flags” for the CPU’s:

# cat /proc/cpuinfo | grep -i flags
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm

We are interested in three values in the output, as they indicate the bit size capabilities of the CPU:

16 Bit = rm (Real Mode)
32 Bit = tm (Transparent Mode)
64 Bit = lm (Long Mode)

This doesn’t necessarily mean the Motherboard is capable of 64 bit.

To determine the bit size of your running kernel, you can also use getconf, similar to HPUX, to find this info:

# getconf LONG_BIT
64

This shows that my kernel is running 64 bit.

Nice IBM RedPaper on SSD Drives

While the article is discussing the SSD drives as an option for IBM’s DS8000 Disk Storage system, it has alot of really good information about SSD drives.   It discusses topics such as  SSD Endurance and advantages over traditional spinning disks.

Ubuntu 9.04 Release Candidate now Available.

I’m a big fan of Ubuntu and I’m excited for the new 9.04 RC release.   Keep in mind this is a “for testing only” release.  Currently I run Ubuntu 8.10 for my main desktop in my home/office.   I’m generally late to upgrade and prefer to play with new releases in a Virtual Environment  (or,  in some cases, an alternate physical environment) before I upgrade to the latest and greatest.  This is true with all OSes that I work with.

Right now, I’m downloading 9.04 and will install it in Sun’s (Oracle’s?) VirtualBox and play with some of the new features.

Oracle to buy Sun Microsystems

It’s all over the news… Unfortunately, I wondered away from keep track of this.  The last time I was watching, IBM was considering it.

The press release confirms that Oracle is buying Sun Microsystems.  I was a little surprised by the news, but not surprised by the move.

I think it’s a great move.  I’ve worked with Sun and Oracle for over 14 years and both are great companies with great products.  I’m very curious to see how things progress from here.

I also wonder… what’s going to happen to MySQL?   Ideally it would be great if it continued to mature with Oracle’s help.  But I wonder if Oracle wants to keep it around.   Selling it would just be creating competition.   Keeping it would give customers an option other than going with Oracle’s RDBMS products.

We should find out soon.

COTD: fuser - identify processes using files or sockets

Today’s Command Of The Day is the “fuser” command.

Have you ever wanted to unmounted an NFS share, or local filesystem and couldn’t because it was in use?  The same applies to CD/DVD’s, PenDrives, etc.  Generally, you receive a message similar to the following:

# umount /dev/sdc1
umount: /media/disk: device is busy.

The fuser has many options.  The ones I use the most are “-cu”.  The “u” option appends the username to the PID and the “c” option is the same as the “m” option, is what you use if you are telling it to check a file, filesystem, or mounted device.

# fuser -cu /dev/sdc1
/dev/sdc1:            8500c(gsmeiers)

# ps  u -p 8500
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
gsmeiers  8500  0.0  0.2  21588  4488 pts/3    Ss+  08:25   0:00 bash

Here we see that /dev/sdc1 has one PID that is using it.  A further ps shows what that PID is.  You can specify either a device name or filesystem when using cu and it will work the same.

You can also, in some versions of fuser, specify a “-v” option to have it show the information about the processes it finds.   For example:

$ fuser -cv /dev/sdc1
USER        PID ACCESS COMMAND
/dev/sdc1:           gsmeiers   7367 ..c.. bash
gsmeiers   8500 ..c.. bash
gsmeiers   9534 F.c.. vi

Fuser also has other commands — such as the “-k” command, which can kill all processes that are using a particular filesystem/device.  Be careful with this one!

Windows Services for UNIX

This is not a Windows blog.  However, I wanted to mention a Microsoft product called Windows Services for UNIX.   This is a Windows Add-on that provides some UNIX interoperability.

It provides things like; Password Syncronization between Windows and UNIX, a Telnet Server, Remote Shell execution Services, a NFS Server, a NFS Client, and more.

I’ve only used it for a NFS Client.  It’s a “cheap” way to add NFS mounting capabilities to a Windows workstation.  I run primarily a UNIX (and Linux) environment at home, but I do have a few Windows workstations.   With this, I can keep all of my data on a UNIX NFS server and not have to mess with Samba/CIFS to access the data.

I won’t go into much details on how to configure it, but it’s pretty straightforward.

Xenix

I remember my first modem.  It was a Radio Shack/Tandy 300 Baud Manual Connect modem.  I had purchased it on clearance at a local Radio Shack for $15.   I brought it home and using the pinouts in the back of the manual for my VTECH Laser 128 Apple compatible computer, proceeded to wire-up a cable to connect the modem to my computer.  I don’t remember when, exactly, but this was somewhere around the mid to late 80’s.

I don’t remember where I got the communication software from — most likely from a friend.  But soon I was a frequent visitor to many local Bulletin Board Systems (BBSes) in the area.  For many years I would now use my Laser 128 as a VT100/ANSI Terminal — my gateway to the world.  Keep in mind, this was way before the Internet was easily accessed aside from Universities.

One of the BBSes I discovered was a BBS that went by the name of EDSI — Enterprise Data Systems Incorporated.   This system was different than the other BBSes that I had been connecting to.  All of the other ones shared the same variety of BBS software.  EDSI was different.

Here is an excerpt from the NixPub publication of that Era.  The NixPub was a list of UNIX systems which provided public access.

09/91  414-734-2499       edsi       Appleton        WI 3/12/24     24
IBM PS/2 Model 55SX, SCO Xenix 2.3.2;  Running STARBASE II Software.
Enterprise Data Systems Incorporated (Non-profit).  100+ local rooms,
PLUS USENET, Multi Channel Chat, 9 ports, $15 yr, flat rate for full
access to net news, mail.  The Fox Valley’s only public access Unix
based BBS.  Contact: Chuck Tomasi (chuck@edsi.plexus.COM)

Later I went on to meet Andy White, a good friend even to this day.  Andy showed me how to “sneak” into the UNIX shell from VI.   That little  “$”  prompt hooked me for life.  EDSI was running Xenix 2.3.2 on an IBM PS/2 Model 55SX.   What made EDSI so cool, was that it was “connected” to what we call today the “Internet”… Well, kinda.

EDSI was “connected” to the Internet via UUCP.  Chuck Tomasi, the sysop of EDSI, worked for a large company in the area that had an Internet connection.  Through Chuck’s arrangement, a system at this company would connect up to EDSI, via Modem, periodically to deliver e-mail and new USENET messages, as well as pick up outgoing items.

EDSI wasn’t connected directly to the Internet… So you couldn’t telnet, ftp, etc.   However, I did fully utilize a FTP to Email Gateway service that was run my Digital Equipment Corp. to navigate FTP sites and fetch files… all via e-Mail.

For the next several years, I would spend all of my time learning UNIX.  I would visit the public library, check out every book on UNIX, and then spent all day and night learning by example.

Recently, my cousin offered me an IBM PS/2 Model 55SX machine that he had laying around.  I’m hoping to clean it up, and setup a XENIX system as a tribute to the system that first got me started in the UNIX world.

VirtualBox

I know Sun’s VirtualBox has been out for awhile now.  However, when it came out, I already have VMware running and had plenty of VMs operational.  Recently, I upgraded to a new workstation and switched OSes.  Instead of going with VMware, I tried VirtualBox.

To be frank, I love it.

Other than a few quirks, it works well — and it’s quite fast.   I installed it at first to install the latest build of OpenSolaris.  Everything went flawlessly.   I then used it to build a Windows 2K VM that I use for some testing.  That went flawlessly.

I’ve only scraped the surface of VirtualBox, but some of the commandline tools are very handy.  I’ll post some more tips and tricks as I find them.

VirtualBox runs on Windows, Linux, OpenSolaris, and OSX hosts and supports a nice array of guests.