Dovecot recompress

I was getting an error about the file size being too large.

May 4 17:42:57 ns1 dovecot: imap(jdavis)<21859><XXXXXXXXX/YYYYYYYYY>: Error: Corrupted record in index cache file /home/jdavis/Maildir/.Archivedir/dovecot.index.cache: UID 1: Broken physical size in mailbox Archivedir: read(zlib(/home/jdavis/Maildir/.Archivedir/cur/1111111111.M555555P333V000000000000FD05I0001A11F_2212.mailhost,S=9794:2,SZ,Z)) failed: Cached message size larger than expected (9794 > 3254, box=Archivedir, UID=1)

I might have clobbered some things while trying to fix it, so I restored a backup to maildir.tmp, and did the following to try to repair/rebuild.

################################
### Clean up the restored mail repo
################################
cd /storage/uploads/CustomerImages/mailtemp/Maildir.tmp
for i in .[a-zA-Z]*/cur/* ; do rm cur/`basename $i` ; done

IFS=$'\n'
for i in $(find . -type f); do
   if file "$i" |grep gzip >/dev/null; then
      # echo "Extracting GZIP:" "$i" 
      mv "$i" "$i".gz
      gunzip "$i".gz
   fi
done &

for i in $(find . -type f); do
   if file "$i" |grep bzip2 >/dev/null; then
      # echo "Extracting BZIP2:" "$i"
      bunzip2 -q "$i"
      mv "$i".out "$(echo $i |sed 's/.out//')"
   fi
done &



################################
### Copy in the missing or damaged files
################################
cd /home/jdavis/Maildir
for i in .[a-zA-Z]* [a-z]* ; do rsync -avS --partial /storage/uploads/CustomerImages/mailtemp/Maildir.tmp/Maildir/"${i}" ./ ; done
for i in .[a-zA-Z]*/cur/* ; do rm cur/`basename $i` ; done

IFS=$'\n'
for i in $(find . -type f); do
   if file "$i" |grep gzip >/dev/null; then
      # echo "Extracting GZIP:" "$i" 
      mv "$i" "$i".gz
      gunzip "$i".gz
   fi
done &

for i in $(find . -type f); do
   if file "$i" |grep bzip2 >/dev/null; then
      # echo "Extracting BZIP2:" "$i"
      bunzip2 -q "$i"
      mv "$i".out "$(echo $i |sed 's/.out//')"
   fi
done &


################################
### Now, remove duplicates
################################
find /storage/uploads/CustomerImages/mailtemp/Maildir.tmp /home/jdavis/Maildir -type d -exec fdupes -dNI {} \;



################################
### Now, recompress it all
################################
compress_maildir () {
   cd $1
   DIRS=`find -maxdepth 2 -type d -name cur`
   for dir in $DIRS; do
      echo $dir
      cd $dir
      FILES=`find -type f -name “*,S=*” -not -regex “.*:2,.*Z.*”`
      #compress all files
      for FILE in $FILES; do
         NEWFILE=../tmp/${FILE}
         #echo bzip $FILE $NEWFILE
         if ! bzip2 -9 $FILE -c > $NEWFILE; then
            echo compressing failed
            exit -1;
         fi
         #reset mtime
         if ! touch -r $FILE $NEWFILE; then
            echo setting time failed
            exit -1
         fi
      done
      echo Locking $dir/..
      if PID=`/usr/lib/dovecot/maildirlock .. 120`; then
         #locking successfull, moving compressed files
         for FILE in $FILES; do
            NEWFILE=../tmp/${FILE}
            if [ -s $FILE ] && [ -s $NEWFILE ]; then
               echo mv $FILE $NEWFILE
               mv $FILE /tmp
               mv $NEWFILE ${FILE}Z
            else
               echo mv failed
               exit -1
            fi
         done
         kill $PID
      else
         echo lock failed
         exit -1
      fi
      cd – >/dev/null
   done
}


################################
### Actually RUN the script to compress all maildir files
################################
./compress_maildir /home/jdavis/Maildir/ &

Related: http://omnitech.net/news/2015/11/14/compressed-dovecot-maildir/


Comments are closed.