Dovecot recompress

I was getting an error about the file size being too large.

May 4 17:42:57 ns1 dovecot: imap(jdavis)<21859><XXXXXXXXX/YYYYYYYYY>: Error: Corrupted record in index cache file /home/jdavis/Maildir/.Archivedir/dovecot.index.cache: UID 1: Broken physical size in mailbox Archivedir: read(zlib(/home/jdavis/Maildir/.Archivedir/cur/1111111111.M555555P333V000000000000FD05I0001A11F_2212.mailhost,S=9794:2,SZ,Z)) failed: Cached message size larger than expected (9794 > 3254, box=Archivedir, UID=1)

I might have clobbered some things while trying to fix it, so I restored a backup to maildir.tmp, and did the following to try to repair/rebuild.

################################
### Clean up the restored mail repo
################################
cd /storage/uploads/CustomerImages/mailtemp/Maildir.tmp
for i in .[a-zA-Z]*/cur/* ; do rm cur/`basename $i` ; done

IFS=$'\n'
for i in $(find . -type f); do
   if file "$i" |grep gzip >/dev/null; then
      # echo "Extracting GZIP:" "$i" 
      mv "$i" "$i".gz
      gunzip "$i".gz
   fi
done &

for i in $(find . -type f); do
   if file "$i" |grep bzip2 >/dev/null; then
      # echo "Extracting BZIP2:" "$i"
      bunzip2 -q "$i"
      mv "$i".out "$(echo $i |sed 's/.out//')"
   fi
done &



################################
### Copy in the missing or damaged files
################################
cd /home/jdavis/Maildir
for i in .[a-zA-Z]* [a-z]* ; do rsync -avS --partial /storage/uploads/CustomerImages/mailtemp/Maildir.tmp/Maildir/"${i}" ./ ; done
for i in .[a-zA-Z]*/cur/* ; do rm cur/`basename $i` ; done

IFS=$'\n'
for i in $(find . -type f); do
   if file "$i" |grep gzip >/dev/null; then
      # echo "Extracting GZIP:" "$i" 
      mv "$i" "$i".gz
      gunzip "$i".gz
   fi
done &

for i in $(find . -type f); do
   if file "$i" |grep bzip2 >/dev/null; then
      # echo "Extracting BZIP2:" "$i"
      bunzip2 -q "$i"
      mv "$i".out "$(echo $i |sed 's/.out//')"
   fi
done &


################################
### Now, remove duplicates
################################
find /storage/uploads/CustomerImages/mailtemp/Maildir.tmp /home/jdavis/Maildir -type d -exec fdupes -dNI {} \;



################################
### Now, recompress it all
################################
compress_maildir () {
   cd $1
   DIRS=`find -maxdepth 2 -type d -name cur`
   for dir in $DIRS; do
      echo $dir
      cd $dir
      FILES=`find -type f -name “*,S=*” -not -regex “.*:2,.*Z.*”`
      #compress all files
      for FILE in $FILES; do
         NEWFILE=../tmp/${FILE}
         #echo bzip $FILE $NEWFILE
         if ! bzip2 -9 $FILE -c > $NEWFILE; then
            echo compressing failed
            exit -1;
         fi
         #reset mtime
         if ! touch -r $FILE $NEWFILE; then
            echo setting time failed
            exit -1
         fi
      done
      echo Locking $dir/..
      if PID=`/usr/lib/dovecot/maildirlock .. 120`; then
         #locking successfull, moving compressed files
         for FILE in $FILES; do
            NEWFILE=../tmp/${FILE}
            if [ -s $FILE ] && [ -s $NEWFILE ]; then
               echo mv $FILE $NEWFILE
               mv $FILE /tmp
               mv $NEWFILE ${FILE}Z
            else
               echo mv failed
               exit -1
            fi
         done
         kill $PID
      else
         echo lock failed
         exit -1
      fi
      cd – >/dev/null
   done
}


################################
### Actually RUN the script to compress all maildir files
################################
./compress_maildir /home/jdavis/Maildir/ &

Related: http://omnitech.net/news/2015/11/14/compressed-dovecot-maildir/


Compressed Dovecot Maildir on Debian

I just saved a few gigs with this. Figured I need to document this or I’ll never remember. :)

Add this into /etc/dovecot/conf.d/10*
# Enable zlib plugin globally for reading/writing:
mail_plugins = $mail_plugins zlib

# Enable these only if you want compression while saving:
plugin {
zlib_save_level = 6 # 1..9; default is 6
zlib_save = gz # or bz2, xz or lz4
}

Add this into /etc/dovecot/conf.d/20*
protocol imap {
mail_plugins = zlib
}
protocol pop3 {
mail_plugins = zlib
}

Remove extra spaces and leftover courier garbage
rename ‘s/\ /_/g’ /home/jdavis/Maildir/.[a-zA-Z]*
rename ‘s/\__/_/g’ /home/jdavis/Maildir/.[a-zA-Z]*
rename ‘s/\_\./\./g’ /home/jdavis/Maildir/.[a-zA-Z]*
rm -r /home/jdavis/Maildir/courier*
rm -r /home/jdavis/Maildir/.[a-zA-Z]*/courier*

Create the script to compress all maildir files
#!/bin/sh
compress_maildir () {
cd $1
DIRS=`find -maxdepth 2 -type d -name cur`
for dir in $DIRS; do
echo $dir
cd $dir
FILES=`find -type f -name “*,S=*” -not -regex “.*:2,.*Z.*”`
#compress all files
for FILE in $FILES; do
NEWFILE=../tmp/${FILE}
#echo bzip $FILE $NEWFILE
if ! bzip2 -9 $FILE -c > $NEWFILE; then
echo compressing failed
exit -1;
fi
#reset mtime
if ! touch -r $FILE $NEWFILE; then
echo setting time failed
exit -1
fi
done
echo Locking $dir/..
if PID=`/usr/lib/dovecot/maildirlock .. 120`; then
#locking successfull, moving compressed files
for FILE in $FILES; do
NEWFILE=../tmp/${FILE}
if [ -s $FILE ] && [ -s $NEWFILE ]; then
echo mv $FILE $NEWFILE
mv $FILE /tmp
mv $NEWFILE ${FILE}Z
else
echo mv failed
exit -1
fi
done
kill $PID
else
echo lock failed
exit -1
fi
cd – >/dev/null
done
}

Actually RUN the script to compress all maildir files
./compress_maildir /home/jdavis/Maildir/

References
* http://wiki.dovecot.org/Plugins/Zlib
* http://wiki2.dovecot.org/Plugins/Zlib
* http://abma.de/blog/2010/449
* https://bbs.archlinux.org/viewtopic.php?id=36305