Compressed Dovecot Maildir on Debian

I just saved a few gigs with this. Figured I need to document this or I’ll never remember. :)

Add this into /etc/dovecot/conf.d/10*
# Enable zlib plugin globally for reading/writing:
mail_plugins = $mail_plugins zlib

# Enable these only if you want compression while saving:
plugin {
zlib_save_level = 6 # 1..9; default is 6
zlib_save = gz # or bz2, xz or lz4

Add this into /etc/dovecot/conf.d/20*
protocol imap {
mail_plugins = zlib
protocol pop3 {
mail_plugins = zlib

Remove extra spaces and leftover courier garbage
rename ‘s/\ /_/g’ /home/jdavis/Maildir/.[a-zA-Z]*
rename ‘s/\__/_/g’ /home/jdavis/Maildir/.[a-zA-Z]*
rename ‘s/\_\./\./g’ /home/jdavis/Maildir/.[a-zA-Z]*
rm -r /home/jdavis/Maildir/courier*
rm -r /home/jdavis/Maildir/.[a-zA-Z]*/courier*

Create the script to compress all maildir files
compress_maildir () {
cd $1
DIRS=`find -maxdepth 2 -type d -name cur`
for dir in $DIRS; do
echo $dir
cd $dir
FILES=`find -type f -name “*,S=*” -not -regex “.*:2,.*Z.*”`
#compress all files
for FILE in $FILES; do
#echo bzip $FILE $NEWFILE
if ! bzip2 -9 $FILE -c > $NEWFILE; then
echo compressing failed
exit -1;
#reset mtime
if ! touch -r $FILE $NEWFILE; then
echo setting time failed
exit -1
echo Locking $dir/..
if PID=`/usr/lib/dovecot/maildirlock .. 120`; then
#locking successfull, moving compressed files
for FILE in $FILES; do
if [ -s $FILE ] && [ -s $NEWFILE ]; then
echo mv $FILE $NEWFILE
mv $FILE /tmp
echo mv failed
exit -1
kill $PID
echo lock failed
exit -1
cd – >/dev/null

Actually RUN the script to compress all maildir files
./compress_maildir /home/jdavis/Maildir/


IMAP Email fixed

Courier-imap-ssl has been flaky for a long time, but now, it turns out it’s been very unhappy with current Thunderbird. Even after manually playing with the TLS settings, it was a beast.

So, I installed dovecot. One line for maildirs, one line for ssl enable, copy over my ssl keys, and set EXIM to use SASL instead of Courier. Poof. Everything *just works*.

While I was at it, I set Thunderbird sort and threading defaults (so I don’t have to set it on every folder individually), and that also is wonderful.

I don’t like having to subscribe to all of the folders manually (I have around 590 folders, one for each project, for each customer, for each partner, plus about 10 tech archives), but if I want it to save everything locally, I cannot just uncheck “show only subscribed folders” and expect it to work.

BUT, really, swapping over and doing all of the manual reconfig was way less time than trying to figure out why Courier was not working. (It was probably something to do with it not being updated any time in the last several updates I’ve tried.)

Now, I’d really like if TB would use an Outbox folder, rather than demanding SMTP, I could switch work over to IMAP vs Exquilla…

Posted in News | Comments Off on IMAP Email fixed

Apache 2.4 on Debian

ns1 got converted to 64-bit, and upgraded to Jessie. It’s been a little painful, but worked for the most part.

1) The biggest thing was installing core packages with :amd64 such that we were never left without dpkg nor apt.

2) PERL broke horribly, and that’s why we moved to Jessie — the only way to get it to REALLY reinstall/rebuild CPAN.

3) A couple of days later, 2 more Seagate drives threw a media chip, and racked up 3k-4k reallocated sectors over a couple of days. Not only have Seagate drives failed extremely rapidly under controlled power and temperatures (some of these were replacements of failed original drives), now, the warranty page on Seagate’s website gives a 404 error.

The drives were replaced with WD RED drives, which have been very stable in this environment. RAID6 ensured that at no point did we lose access to data, nor suffer any losses.

4) Apache 2.4 has changed a whole bunch. In 2.2, there were transitional packages off of the base names, and in 2.4, the transitional packages moved it back. Whatever. *sigh* That’s cleaned up, but was no real factor.

These changes in Apache 2.4 have been resolved:
* conf.d is no longer used
* sites-enabled/* must have “.conf” appended
* mod_auth_pam is no longer available
* “Require user” is now requires “pwauth” and “libapache2-mod-authnz-external” and new directives
* Require group” is now replaced with “Require unix-group”, and requires “libapache2-mod-authz-unixgroup”, which is different from user authentication.

I’m still working on one of my aliased directories which is not working.

I’m also trying to sort out why Tine 2.0 is stuck “upgrading”. That’s normal for Tine 2.0 though. It’s really an annoying beast, and I’m glad I don’t rely on it. I really just want CalDAV, IMAP, and maybe something to sync notes and reminders. It seems this is nearly impossible.


Things I learned today:
* If you are American, then it’s okay to go after you for 33 seconds of some song in the background of a video from a friend’s party.
* If you are not American, it’s okay to post the whole song online.
* Italy does not have “Fair Use” in their copyright laws, and everything is licensed, even blank recording media.
* YouTube does not have a way for individuals to restrict their videos to specific country’s viewers without joining a syndication network and signing distribution agreements.
Starchaser, aka Fausto Fanizza, has claimed that the 33 second chunk of this song in the background friend’s house while playing with a laser has caused him severe damages, and that it is not in fact fair use.  He claims he has no choice other than to fight my dispute, and that I better get a lawyer.  I said we always have a choice.  If he had asked for a couple hundred dollars, I would have happily sent that to him for the inconvenience.
Eventually, I was able to get the video unlocked, and I simply deleted it. It’s not worth the hassle of ever accidentally letting people know about his music if he’s going to come after me for more money than he’s ever made off of that song.

Failing drive in the array

I collected info from the failing drive in the array, and compared to other drives in the array.
It actually looked good, comparatively, until I found this one line near the end:
Warning: device does not support SCT Error Recovery Control command
GAH. No wonder. I still have to replace it.

This is another reason why Seagate is on my poop list.
They sent me a lower function device as a warranty replacement.

On the flip side, if anyone needs a 2TB 5900RPM drive for a desktop system, I can hoox you up.
It’s still in good condition, just not suitable for an array.
Also, it’s out of warranty (but only a year old).

Posted in News | Tagged | Comments Off on Failing drive in the array

RAID maintenance

My RAID drive that went offline last week went offline again yesterday. That means real failure.

It’s a warranty replacement of a previously failed drive. Out of the Seagate drives I’ve used in arrays, I’ve had 1 drive not fail in 4 years, and I’ve had more failures overall than actual drives.

This is at three different sites, four arrays, different enclosures, systems, etc. Everything on UPS and surge suppression.

My WD RED drives are happy. At 1 year, no failures out of 8 drives. Though, one array was going offline due to a flaky controller. No problem with the drives though.

This array was populated in 2010, so the warranty is up. I’m replacing this drive with a WD Red 3TB. I’m only replacing one drive at this time, just due to budget, but Linux MD-RAID will happily let me mix in this newer, higher capacity drive.

We’re running RAID6, so there shouldn’t be any interruptions.

*knock on wood*

New UPS batteries

The storm last night performed a UPS test that was long over-due.

Unfortunately, the server UPS failed. One battery was 0V, and the other was 8.6V.

Both 7AH-12V batteries were replaced with new 9AH-12V batteries.

The old ones were the factory batteries, plut into service 2008-07-03 as per:

The UPS seems happy with the new batteries. They should provide a little longer run-time.

I’ll add a reminder to replace them in 5 years.

Freedom Pop

Freedompop has 2 bars but cannot ping the gateway. Sigh

powerfail at the home office has pointed out some UPS defgiciencies. FiOS only provides voice service on UPS power. Also, our 2008 server hangs on UPS power (probably need sine wave). Lastly, the cordless base is not on a UPS.

Everything else was hibernated or shutdown safely.

Fixed URLs in WP posts

My WPMediawiki plugin was converting anchor tags into nested anchor tags, which was failing horribly.

Also, I installed a redirect plugin which should allow me to use the wiki markup tag with a subset and still get the right plage.

For instance, [[sprouts|Sprouts]] should redirect to about tasty toffee peanuts, and [[khai-ranks|Khai Ranks]] should link to , a page about Khai ranking up in Karate.

UPDATE 2018 – this is WAY broken and I have remove the WP MediaWiki plugin. Hopefully WP-Markdown will get me some of what I want.

Posted in News, xaminmo | Comments Off on Fixed URLs in WP posts

Gallery Despammed

Just updated blacklists on and cleaned up some spam.

I haven’t really considered a major update though.

Posted in News | Comments Off on Gallery Despammed