Omnitech DP server lives

10 days ago, the drive enclosure for the TSM server failed during a storm. The enclosure is an RSV-S5 from 2010. The PSU died, and seems to be a specialty part. The part costs $250. A newer version of the enclosure $180 from Sans Digital. This is a bulk data server, so a 4-pay box was fine. I picked up a Mediasonic Probox 4-bay JBOD with ESATA and USB3 ports. It’s a faster port multiplier, better functionality, and half the volume on the server shelf.

I still plan to migrate everything to Linux on Spectrum Protect 8, with container pools, and maybe use glacier for off-site storage. This is compounded by CrashPlan ditching their non-business plans, and never being able to sync anyway. I really need a better way to store off-site DR data. BOX for a critical chunk is okay. Google and Dropbox for active data is okay. But for an off-site DR pool, it would be too expensive to put into either of those. Plus, SP8 is chunk aware much better. I’d hate for a CDP product to revert a chunk, or be constantly out of sync.

SATA chipset reference

The SIL3132 card (SATA-II, PCIe 1.0) ran at 122MB/sec.

The 88SE9128 card (SATA-III, PCIe 2.0) ran at 75MB/sec, or 35MB/sec with FIS disabled.

The 88SE9235 card runs at 195MB/sec.

My two test enclosures are:

  • SIL3726 based enclosure (RSV-5S)
  • 88SM9715 based enclosure (TR5M6G)
  • Linux, MDADM, RAID6, sequential read, 256k blocks.

Ableton said I should go with a single SSD behind a JMS575 port multiplier to get best performance out of the 88SE9128.

I pointed out that a single drive is not the same as multiple (switching delays),
and that replacing all of my spinning disks with SSD is not a valid solution.

Posted in News, Reference | Comments Off on SATA chipset reference

gallery upgrading

I’m finally updating the Gallery 1.5.10 server from 2004 to Gallery 3.0.9.
This fixes the PHP errors that kept showing up on the old version.
However, for right now, I can only log in with FireFox.

Anyway, 6851 photos, 160 albums, 16 users, 1535 comments getting imported.
When it’s done, we’ll see if everything looks okay before I swap it in place.
I honestly don’t think any on my users still use this.

Bad Subnet Kills DHCPD

One, single bad IP in DHCPD config will kill the entire config file. :(

On an EdgeRouter, and probably anything with Ubiquiti, and maybe anything using the same config style (Brocade and others have the same command set)….

If you add a static reservation outside of the DHCP server’s subnet,
as in, if you typo one octet, or decide to do another subnet just because,
your DHCP server will be offline after reboot. No errors, just silently not serving.

It can be outside of the start/stop range, and that’s fine.

Really, this should give you a warning from the webUI, or it should just say “OKAY, We’ll let you hand out stupid IP addresses.” I mean, what if I wanted this to be my DHCP server, but I had a different router and subnet on the same segment?

From command line, you’ll see the error though:

admin@gw1# commit
[ service dhcp-server ]
Static DHCP lease IP '' under mapping 'CustomerLaptop'
under shared network name 'LAN' is outside of the DHCP lease network ''.
DHCP server configuration commit aborted due to error(s).

Compressed Dovecot Maildir on Debian

I just saved a few gigs with this. Figured I need to document this or I’ll never remember. :)

Add this into /etc/dovecot/conf.d/10*

# Enable zlib plugin globally for reading/writing:
mail_plugins = $mail_plugins zlib
# Enable these only if you want compression while saving:
plugin {
 zlib_save_level = 6 # 1..9; default is 6
 zlib_save = gz # or bz2, xz or lz4

Add this into /etc/dovecot/conf.d/20*

protocol imap {
  mail_plugins = zlib
protocol pop3 {
  mail_plugins = zlib

Remove extra spaces and leftover courier garbage

rename 's/\ /_/g' /home/jdavis/Maildir/.[a-zA-Z]*
rename 's/\__/_/g' /home/jdavis/Maildir/.[a-zA-Z]*
rename 's/\_\./\./g' /home/jdavis/Maildir/.[a-zA-Z]*
rm -r /home/jdavis/Maildir/courier*
rm -r /home/jdavis/Maildir/.[a-zA-Z]*/courier*

Create the script to compress all maildir files

compress_maildir () {
cd $1
DIRS=`find -maxdepth 2 -type d -name cur`
for dir in $DIRS; do
       echo $dir
       cd $dir
       FILES=`find -type f -name "*,S=*" -not -regex ".*:2,.*Z.*"`
       #compress all files
       for FILE in $FILES; do
               #echo bzip $FILE $NEWFILE
               if ! bzip2 -9 $FILE -c > $NEWFILE; then
                       echo compressing failed
                       exit -1;
               #reset mtime
               if ! touch -r $FILE $NEWFILE; then
                       echo setting time failed
                       exit -1
       echo Locking $dir/..
       if PID=`/usr/lib/dovecot/maildirlock .. 120`; then
               #locking successfull, moving compressed files
               for FILE in $FILES; do
                       if [ -s $FILE ] && [ -s $NEWFILE ]; then
                               echo mv $FILE $NEWFILE
                               mv $FILE /tmp
                               mv $NEWFILE ${FILE}Z
                               echo mv failed
                               exit -1
               kill $PID
               echo lock failed
               exit -1
       cd - >/dev/null

Actually RUN the script to compress all maildir files

./compress_maildir /home/jdavis/Maildir/


IMAP Email fixed

Courier-imap-ssl has been flaky for a long time, but now, it turns out it’s been very unhappy with current Thunderbird. Even after manually playing with the TLS settings, it was a beast.

So, I installed dovecot. One line for maildirs, one line for ssl enable, copy over my ssl keys, and set EXIM to use SASL instead of Courier. Poof. Everything *just works*.

While I was at it, I set Thunderbird sort and threading defaults (so I don’t have to set it on every folder individually), and that also is wonderful.

I don’t like having to subscribe to all of the folders manually (I have around 590 folders, one for each project, for each customer, for each partner, plus about 10 tech archives), but if I want it to save everything locally, I cannot just uncheck “show only subscribed folders” and expect it to work.

BUT, really, swapping over and doing all of the manual reconfig was way less time than trying to figure out why Courier was not working. (It was probably something to do with it not being updated any time in the last several updates I’ve tried.)

Now, I’d really like if TB would use an Outbox folder, rather than demanding SMTP, I could switch work over to IMAP vs Exquilla…

Posted in News | Comments Off on IMAP Email fixed

Apache 2.4 on Debian

ns1 got converted to 64-bit, and upgraded to Jessie. It’s been a little painful, but worked for the most part.

1) The biggest thing was installing core packages with :amd64 such that we were never left without dpkg nor apt.

2) PERL broke horribly, and that’s why we moved to Jessie — the only way to get it to REALLY reinstall/rebuild CPAN.

3) A couple of days later, 2 more Seagate drives threw a media chip, and racked up 3k-4k reallocated sectors over a couple of days. Not only have Seagate drives failed extremely rapidly under controlled power and temperatures (some of these were replacements of failed original drives), now, the warranty page on Seagate’s website gives a 404 error.

The drives were replaced with WD RED drives, which have been very stable in this environment. RAID6 ensured that at no point did we lose access to data, nor suffer any losses.

4) Apache 2.4 has changed a whole bunch. In 2.2, there were transitional packages off of the base names, and in 2.4, the transitional packages moved it back. Whatever. *sigh* That’s cleaned up, but was no real factor.

These changes in Apache 2.4 have been resolved:

  • conf.d is no longer used
  • sites-enabled/* must have “.conf” appended
  • mod_auth_pam is no longer available
  • “Require user” is now requires “pwauth” and “libapache2-mod-authnz-external” and new directives
  • Require group” is now replaced with “Require unix-group”, and requires “libapache2-mod-authz-unixgroup”, which is different from user authentication.

I’m still working on one of my aliased directories which is not working.

I’m also trying to sort out why Tine 2.0 is stuck “upgrading”. That’s normal for Tine 2.0 though. It’s really an annoying beast, and I’m glad I don’t rely on it. I really just want CalDAV, IMAP, and maybe something to sync notes and reminders. It seems this is nearly impossible.


Things I learned today:

  • If you are American, then it’s okay to go after you for 33 seconds of some song in the background of a video from a friend’s party.
  • If you are not American, it’s okay to post the whole song online.
  • Italy does not have “Fair Use” in their copyright laws, and everything is licensed, even blank recording media.
  • YouTube does not have a way for individuals to restrict their videos to specific country’s viewers without joining a syndication network and signing distribution agreements.

Failing drive in the array

I collected info from the failing drive in the array, and compared to other drives in the array.
It actually looked good, comparatively, until I found this one line near the end:
Warning: device does not support SCT Error Recovery Control command
GAH. No wonder. I still have to replace it.

This is another reason why Seagate is on my poop list.
They sent me a lower function device as a warranty replacement.

On the flip side, if anyone needs a 2TB 5900RPM drive for a desktop system, I can hoox you up.
It’s still in good condition, just not suitable for an array.
Also, it’s out of warranty (but only a year old).

Posted in News | Tagged , | Comments Off on Failing drive in the array

RAID maintenance

My RAID drive that went offline last week went offline again yesterday. That means real failure.

It’s a warranty replacement of a previously failed drive. Out of the Seagate drives I’ve used in arrays, I’ve had 1 drive not fail in 4 years, and I’ve had more failures overall than actual drives.

This is at three different sites, four arrays, different enclosures, systems, etc. Everything on UPS and surge suppression.

My WD RED drives are happy. At 1 year, no failures out of 8 drives. Though, one array was going offline due to a flaky controller. No problem with the drives though.

This array was populated in 2010, so the warranty is up. I’m replacing this drive with a WD Red 3TB. I’m only replacing one drive at this time, just due to budget, but Linux MD-RAID will happily let me mix in this newer, higher capacity drive.

We’re running RAID6, so there shouldn’t be any interruptions.

  • knock on wood*

New UPS batteries

The storm last night performed a UPS test that was long over-due.

Unfortunately, the server UPS failed. One battery was 0V, and the other was 8.6V.

Both 7AH-12V batteries were replaced with new 9AH-12V batteries.

The old ones were the factory batteries, plut into service 2008-07-03 as per:

The UPS seems happy with the new batteries. They should provide a little longer run-time.

I’ll add a reminder to replace them in 5 years.

Freedom Pop

Freedompop has 2 bars but cannot ping the gateway. Sigh

powerfail at the home office has pointed out some UPS defgiciencies. FiOS only provides voice service on UPS power. Also, our 2008 server hangs on UPS power (probably need sine wave). Lastly, the cordless base is not on a UPS.

Everything else was hibernated or shutdown safely.

Fixed URLs in WP posts

My WPMediawiki plugin was converting anchor tags into nested anchor tags, which was failing horribly.

Also, I installed a redirect plugin which should allow me to use the wiki markup tag with a subset and still get the right plage.

For instance, Sprouts should redirect to Sprouts-6224 about tasty toffee peanuts, and Khai Ranks should link to a page about Khai ranking up in Karate.

Posted in News, xaminmo | Comments Off on Fixed URLs in WP posts

Optiplex 755

NS1 has been upgraded to an Optiplex 755 Core2-Duo 3GHz.
It has 5x SATA ports on the motherboard, but does not support Port Multipliers properly.
It sees the drives, but there are hangs, lags, etc all the time. 6MB/sec aggregate isn’t okay.

I didn’t want to use my PCI-32 SIL-3124, mostly for performance reasons.
I have a 1x SIL3132, and found the drive enclosure works fine on port 0, but LILO gives L 01 01 01 01 01 etc if I use port 1.
This is pulling 102MB/sec sequential from a 5-way mirror, and 124MB/sec from a 5-way RAID6.
Compare to the 3124 which topped out about 50MB/sec regardless of which slice I abused.

In theory I could just cable all of the drives up to individual ports, and the performance would likely be slightly better.
Unfortunately, I’d need to pull the optical drive, and install a 2-3 bay converter.
I didn’t want to spend MORE money, because if I did, I’d probably just start buying 2.5″ drives, or other things I didn’t need.
1 2 3 4 5 6
7 8 (plus cables, brackets, etc)

Anyway, I’ve bumped up to a 64-bit kernel, and may be swinging to a 64-bit OS if I feel particularly frisky.

My goal would be to install SDFS (open dedupe), mhVTL (Linux VTL, now with iSCSI support) and see how they’re doing vs 2007.

Omnitech News Service

According to a U.S. District Court Judge in Oregon, Honorable Marco A. Hernandez, `press`, in context of the First Amendment to the US Constitution, specifically means persons `affiliated with any newspaper, magazine, periodical, book, pamphlet, news service, wire service, news or feature syndicate, broadcast station or network, or cable television system.` Online journalism is now legally excluded unless affiliated with an entity of one or more of those types.

As such, be it known that all of my communication via any online forum is formally a media for communication to the public as an agent of the `OmniTech News Service`. This service will include fact, fiction, assumptions, satire, and any other form of communication deemed appropriate by the staff.


Posted in News | Comments Off on Omnitech News Service

Broken domains

  • has the wrong name servers and the owner never responded to requests to update this.
  • has the wrong name servers and the owner never responded to requests to update this. The prior owner was interested in having it rehosted at his other site should the current owner become reachable.

Expiring and code broken:

  • is expiring in 5 months. Everyone moved to Facebook. The PHPBB was broken. Lack of interest, the domain will be allowed to expire.
  • will expire in 4 months. It is a redirect to now. The domain will be allowed to expire.
  • has broken code and had low interest. It was replaced by a default gallery install. It expires in 7 months. It’s fate is unknown but may be allowed to expire.
  • serves no real purpose and will be allowed to expire in 2 months. Everything on it has been moved to

Posted in News | Comments Off on Broken domains

Deleted domains

  • was expired, and the owner said they didn’t use it anymore.
  • is pointed to a different server, and the owner said they didn’t use it anymore.
  • was expired, the owner didn’t have registrar access, and the domain was picked up by a squatter.
  • was expired, the owner didn’t have registrar access, and the domain was delisted from dotster.
  • was expired and delisted. It is accessible from
  • was expired and delisted. It is accessible as
  • expires in 3 days and has been unused for 3 years. Owner said OK to let drop.
  • now forwards to

Posted in News | Comments Off on Deleted domains

WE’VE MOVED! has finished it’s migration to dreamhost.

If you have an account, and haven’t seen my email, then contact me (omnimax at omnitech dawt knet) to find out your new login credentials.

Some domains are still spotty due to failure on my part to update whois promptly when the ethernet switch at the old office silently died.

Everything that was working should be working in the same manner by Tuesday around lunchtime (DNS propagation delays).

If you have problems with your site, let me know. Some files might have been lost, but I used rsync and waited for successful complete, so it’s more likely that the .htaccess is wrong, or something little like that.



Posted in News | Comments Off on WE’VE MOVED!

More on HSCLA27C

#hmc# lslparmigr -r lpar -m Server-9117-MMA-SN10XXX64
Everyone is `Not Migrating`

#hmc# lslparmigr -r msp -m Server-9117-MMA-SN10XXX64 -t Server-9117-MMA-SN10E8524 --filter lpar_names=aix02

#hmc# lslparmigr -r virtualio -m Server-9117-MMA-SN10XXX64 -t Server-9117-MMA-SN10XXX24 --filter lpar_names=aix02
HSCLA27C The operation to get the physical device location for adapter U9117.MMA.10XXX64-V2-C40 on the virtual I/O server partition viod has failed.
The partition command is:
migmgr -f get_adapter -t vscsi -s U9117.MMA.10XXX64-V2-C40 -w 13857705XXXXYYYY5464 -W 13857705XXXXYYYY5465 -d 1
The partition standard error is:

#vios# lscfg -vp | grep U9117.MMA.10XXX64-V2-C40
vfchost20 U9117.MMA.10XXX64-V2-C40 Virtual FC Server Adapter

#vios# ioscli lsmap -all -npiv
Name Physloc ClntID ClntName ClntOS
------------- ---------------------------------- ------ -------------- -------
vfchost20 U9117.MMA.10XXX64-V2-C40 8 aix02 AIX

FC name:fcs0 FC loc code:U789D.001.DQDXXXX-P1-C1-T1
Ports logged in:1
VFC client name:fcs4 VFC client DRC:U9117.MMA.10XXX64-V8-C9-T1

NOTE: All LPARs on this system get the same failure for their NPIV interface that maps to physical fcs0

Errpt shows FCP_ERR10 on fscsi3 on both during the cfgmgr, but nothing on fcs0.

fcs0 and fscsi0 look fine from lscfg, lsattr… they’re logged in to the switch and everything.

There’s only vscsi mapped LUNs – nothing mapping directly through the NPIV yet. The vscsi mapped LUNs have happy, enabled paths on fscsi0.

cfgmgr on VIOS and VIOC doesn’t seem to have any effect.

ioscli lsnports shows both fcs0 and fcs2 are available.

My initial guess is that cables were unplugged, and some odd state was entered.

Remapping NPIV didn’t work.

Tried rebooting VIO server and digging into SAN switches.

I had to manually create the unique_id fields with odmadd. My new, VIO clone makes them, as do my new NPIV clients, but my older VIO servers do not make them.

To find the unique_id:
odmget -q attribute=unique_id CuAt

I think it’s a problem with the EMC.Symmetrix.aix.rte and fcp.rte, or maybe the ODM entries. The new systems that work had reinstalled during migration and upgrade. I’m hesitant to do that on the older, running systems for fear of existing LUNs being broken.

Since I’m eventually migrating off of these 2 frames, here’s my temporary (4 years now) fix:
Uncomment the portions you need at any given time:

SYMSER=`XXXX`   # The last 4 digits of the symmetrix serial number
SYMID=`YY`      # The 2 digits prepended to each volume from inq
echo `hdiskpower10	lp9_sap_vts23
hdiskpower19	lp9_sap_vts24
hdiskpower20	lp9_sap_vts25` | while read disk vts ; do
### Removal of disks
#   for j in powermt display dev=$disk 2>&1 | grep fscsi ; do
#		if [[ -n echo ${j} | grep hdisk 2>&1  ]] ; then DISKS=`$DISKS $j` ; fi
#   done
#echo $disk $DISKS $vts
#for i in $vts $disk $DISKS ; do rmdev -dl $i ; done
### Removal of just the VTD
#rmdev -dl $vts
### Make unique_id fields where missing (older VIO servers)
#ID=powermt display dev=$disk | grep `Logical device ID` | cut -f 2 -d =
#echo `CuAt:
#       name = `${disk}`
#        attribute = `unique_id`
#        value = `$SYMSER$SYMID${ID}09SYMMETRIX03EMCfcp`
#        type = `R`
#        generic = ``
#        rep = `nl`
#        nls_index = 124
#` >> uniques.out
### Make a new PV out of the disk
#dd if=/dev/zero of=/dev/$disk bs=256k count=1
#chdev -a pv=yes -l $disk
#mkdev -l $disk
#lspv | grep $disk
### Make a new VTD
#ioscli mkvdev -dev $vts -vadapter vhost6 -vdev $disk
### verify unique_id file, then add it into the running config
#odmadd < uniques.out
### Remake any removed devices - Remember to reset no_reserve as above.

Posted in News | Comments Off on More on HSCLA27C