Security Defect in Intel, ARM and AMD processors

The defect allows a user process to read any system memory.
A VM can read memory from the host or another guest in some environments.

This does NOT affect POWER/PPC architecture.
Only some of this affects AMD, and only in some modes.
Almost every ARM and Intel processor since 1995 is affected.
That includes desktops, laptops, servers, cellphones, routers, automobiles with Sync/Onstar/autopilot, etc.

This defect was reported in June, 2017, but due to pervasiveness, has been embargoed.
It is only fully described now because patch notes leaked the problem.

The actual fix would be replacement of the affected CPUs with new silicon, which does not exist yet.
There is a partial software workaround which decreases system performance.

The issue is because the processor does not perform access checking prior to loading L1 cache.
Due to this design issue, data can be forced into L1 cache, and read, before access is denied by the TLB.
It’s fairly slow, at around 2k/second, but a long-running process can harvest everything.

Hardware Statuses:
• ARM has provided workarounds to vendors, but it’s up to them to implement
• Intel’s CEO sold off as much of his stock as possible last year after glowing projections.
• Not a peep from AMD.
• POWER/PPC is not affected.

Software Statuses:
• Windows included a partial workaround in the November security rollup.
• MacOS released a partial workaround in December’s 10.13.2
• Linux included a partial workaround in the mainline kernels 4.15, and 4.14.11.
• The workarounds decrease performance between 1% and 45% depending on the workload.
• Cloud providers are scheduling maintenance January 2018.

More Reading:
• Community:
• Google:
• Workaround:
• AMD:
• A better write-up:
• Outlet that broke the embargo:

Convert EXT3 to EXT4

### Change to EXT4 mount mode (OKAY before conversion)
vi /etc/fstab

### Reboot into single user mode
shutdown -r now
LILO: linux S

### Unmount or read-only every filesystem
umount -a
mount -oremount,ro /usr
mount -oremount,ro /

### Convert all ext4 into new metadata formats
grep ext4 /etc/fstab | tr -s [:space:] | cut -f 1 -d \ | tune2fs -O extents,uninit_bg,dir_index

### Build the directory index and verify metadata
grep ext4 /etc/fstab | tr -s [:space:] | cut -f 1 -d \ | fsck.ext4 -yfD

### Reboot back to multiuser mode
shutdown -r now

### Covert all files in EXT4 filesystems to extent mode (was bitmap)
for dir in `mount | grep ext4 | cut -f 3 -d \ ` ; do LC_ALL=C find $dir -xdev -type d -print0 | LC_ALL=C xargs -r0 -P3 chattr +e ; done
for dir in `mount | grep ext4 | cut -f 3 -d \ ` ; do LC_ALL=C find $dir -xdev -type f -print0 | LC_ALL=C xargs -r0 -P3 chattr +e ; done

### References
* _ext4_filesystem

apt sandbox permissions

Every repo was giving signature errors in apt:
Err:6 stretch/updates InRelease
At least one invalid signature was encountered.

This was pretty recent. My updates in May were fine.
This ONLY affected apt* update. Not clean, install, purge, etc.

I could bypass the error by telling the sandbox to become root:
apt -o APT::Sandbox::User=root update

/tmp was still 1777. I did find /var/tmp was linked to /tmp, which killed dovecot install.
No idea why that’s a problem, because my /tmp is persistent across reboots.
A snotty developer somewhere indicated it was the end of the universe.
Now, /var/tmp is just part of /var. Whatever.

So, someone did a hard cleanup of cache, and that fixed it for me:
sudo apt-get clean
sudo mv /var/lib/apt/lists /tmp
sudo mkdir -p /var/lib/apt/lists/partial
sudo apt-get clean
sudo apt-get update

Then I compared /tmp/lists and /var/lib/apt/lists.
Exactly the same for everything, except top level permissions.
The old one was 755 and the new one is 750.

WTF?!?!? Why do we care if “other” can read the package lists?
There is ZERO sensitive data in there?

I decided someone was intoxicated, watching Rick and Morty, making out with their significant other, and coding with their non-dominant hand, just to see if they could maintain focus on a dare.

PPC64 Linux on Intel

QEMU on Windows will run ppc64 and ppc64le emulation.
It emulates the same as what PowerKVM on an S812L would provide.
It’s kind of slow because there is no KVM module, AND Intel vs PPC,
AND emulator mode is single-core/proc/thread.

You can get Windows installer here:

You really want ANSI/VT100 escape codes on you “cmd.exe” also:

To build a blank disk:
qemu-img create -f qcow2 qemu-disk-ppc64.img 32G

You can boot with this:
qemu-system-ppc64 -M type=pseries -m 1G,slots=4,maxmem=8G
-cpu POWER8E -smp 1 -vga none -nographic
-netdev user,id=net0 -device spapr-vlan,netdev=net0
-device spapr-vscsi -device scsi-hd,drive=drive0
-drive id=drive0,if=none,file=qemu-disk-ppc64.img
-cdrom D:\Downloads\debian-testing-ppc64el-DVD-1.iso

The QEMU part is all one line. The cdrom image is up to you. I like Debian.

Other Notes:
Any issues with cursor keys, use ctrl-i for TAB, ctrl-n and ctrl-p for next/previous.

Emulation mode is flaky with more than one core.

There is a QEMU AIX build on PERZL.ORG which would be faster, especially for ppc64 BigEndian.

PowerKVM is just PPC Linux, QEMU, KVM, and LIBVIRT. KVM is just a kernel module for spee-dup. LIMVIRT is just a GUI and CLI tool to build VM definitions. QEMU is the emulator. Works best on POWER8, with hypervisor disabled (OPAL mode).

QEMU still does not have enough RTAS and NVRAM to boot AIX. AIX hangs during “Starting AIX”, and Diags just says it’s an unsupported machine type. There is a little bit of dev for this, but not much.​

Compressed Dovecot Maildir on Debian

I just saved a few gigs with this. Figured I need to document this or I’ll never remember. :)

Add this into /etc/dovecot/conf.d/10*
# Enable zlib plugin globally for reading/writing:
mail_plugins = $mail_plugins zlib

# Enable these only if you want compression while saving:
plugin {
zlib_save_level = 6 # 1..9; default is 6
zlib_save = gz # or bz2, xz or lz4

Add this into /etc/dovecot/conf.d/20*
protocol imap {
mail_plugins = zlib
protocol pop3 {
mail_plugins = zlib

Remove extra spaces and leftover courier garbage
rename ‘s/\ /_/g’ /home/jdavis/Maildir/.[a-zA-Z]*
rename ‘s/\__/_/g’ /home/jdavis/Maildir/.[a-zA-Z]*
rename ‘s/\_\./\./g’ /home/jdavis/Maildir/.[a-zA-Z]*
rm -r /home/jdavis/Maildir/courier*
rm -r /home/jdavis/Maildir/.[a-zA-Z]*/courier*

Create the script to compress all maildir files
compress_maildir () {
cd $1
DIRS=`find -maxdepth 2 -type d -name cur`
for dir in $DIRS; do
echo $dir
cd $dir
FILES=`find -type f -name “*,S=*” -not -regex “.*:2,.*Z.*”`
#compress all files
for FILE in $FILES; do
#echo bzip $FILE $NEWFILE
if ! bzip2 -9 $FILE -c > $NEWFILE; then
echo compressing failed
exit -1;
#reset mtime
if ! touch -r $FILE $NEWFILE; then
echo setting time failed
exit -1
echo Locking $dir/..
if PID=`/usr/lib/dovecot/maildirlock .. 120`; then
#locking successfull, moving compressed files
for FILE in $FILES; do
if [ -s $FILE ] && [ -s $NEWFILE ]; then
echo mv $FILE $NEWFILE
mv $FILE /tmp
echo mv failed
exit -1
kill $PID
echo lock failed
exit -1
cd – >/dev/null

Actually RUN the script to compress all maildir files
./compress_maildir /home/jdavis/Maildir/


Apache 2.4 on Debian

ns1 got converted to 64-bit, and upgraded to Jessie. It’s been a little painful, but worked for the most part.

1) The biggest thing was installing core packages with :amd64 such that we were never left without dpkg nor apt.

2) PERL broke horribly, and that’s why we moved to Jessie — the only way to get it to REALLY reinstall/rebuild CPAN.

3) A couple of days later, 2 more Seagate drives threw a media chip, and racked up 3k-4k reallocated sectors over a couple of days. Not only have Seagate drives failed extremely rapidly under controlled power and temperatures (some of these were replacements of failed original drives), now, the warranty page on Seagate’s website gives a 404 error.

The drives were replaced with WD RED drives, which have been very stable in this environment. RAID6 ensured that at no point did we lose access to data, nor suffer any losses.

4) Apache 2.4 has changed a whole bunch. In 2.2, there were transitional packages off of the base names, and in 2.4, the transitional packages moved it back. Whatever. *sigh* That’s cleaned up, but was no real factor.

These changes in Apache 2.4 have been resolved:
* conf.d is no longer used
* sites-enabled/* must have “.conf” appended
* mod_auth_pam is no longer available
* “Require user” is now requires “pwauth” and “libapache2-mod-authnz-external” and new directives
* Require group” is now replaced with “Require unix-group”, and requires “libapache2-mod-authz-unixgroup”, which is different from user authentication.

I’m still working on one of my aliased directories which is not working.

I’m also trying to sort out why Tine 2.0 is stuck “upgrading”. That’s normal for Tine 2.0 though. It’s really an annoying beast, and I’m glad I don’t rely on it. I really just want CalDAV, IMAP, and maybe something to sync notes and reminders. It seems this is nearly impossible.

Raspberry Pi Quick TTY Setup

This was how I set up my Pi 2 without using the HDMI/USB console. I have a wifi adapter in one USB port, and a Prolific TTY to Serial USB adapter.

### Prepare the installer
Download NOOBS from
If the zip file is corrupt, then pull down with torrent. Mine had one corrupt block from the webserver.
Format your TF/uSD card as FAT32 (not exFAT, ext*, nor NTFS)
Unpack into your TF/uSD card
Remove all but Raspbian from “os” dir
edit recovery.cmdline to have “silentinstall”
edit flavours.json to have only raspbian, not the scratch version.

### Connect the TTY console with power
Connect Prolific TTL to Serial adapter
Red pin (5V) to pin 2 (furthest from USB, closest to edge)
Skip pin 4. Black is pin 6, white is pin 8, green is pin 10
DO NOT USE THE USB POWER PORT. We are powering through the red pin. Both power at same time will kill the board. You CAN unplug the red pin so as to allow more amps through the power micro-USB port.

### Power and autoboot
Plug in the USB port and connect to your comm port (putty, hyperterm, whatever) at 115200,8,n,1
It takes 4 seconds to say “Recovery Console”, then two for unpacking, and about 20 mins for complete install. Green LED (drive light) should blink.
The pi and it will install then reboot

### Basic config
Login as pi / raspberry, then sudo to root
Walk through raspi-config, then “FINISH”
sudo to root and set root password.

### Set up network
Plug in your wifi adapter, OR the ethernet cable, or both.
Edit /etc/wpa_supplicant/wpa_supplicant.conf to include:
ifdown wlan0 ; ifup wlan0
Wait 20 seconds for reconnect
cp -p /etc/network/interfaces /etc/network/interfaces.bak
cat < <"EOF" > /etc/network/interfaces
auto lo
iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf
#wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf

iface default inet dhcp

### Change the primary console to be tty instead of GUI
cp -p /boot/cmdline.txt /boot/cmdline.txt.bak
cat < <"EOF" > /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 console=ttyAMA0,115200 root=/dev/mmcblk0p6 rootfstype=ext4 elevator=deadline rootwait

### Install “resize” for console
apt-get update
apt-get install xterm
cat < <"EOF" >> /etc/profile
alias ll=’ls -laF’

### Set time and certificates (required for firmware update)
apt-get install ntpdate ca-certificates
cat < <"EOF" >> /etc/ntp.conf
/etc/init.d/ntp stop
/etc/init.d/ntp start

### Update firmware
apt-get install rpi-update

### Update base OS
sudo apt-get upgrade

### Other references

### Power test
2015-02-20 03:55am 90% or better 5x-18650 power brick, base install, boot then idle with rtl8188cu wifi-N and SSH connected.
2015-02-20 10:49am 75% battery left
2015-02-20 15:47pm 25% still last LED, getting dimmer
2015-02-20 19:55pm Blinking final LED
2015-02-20 20:59pm last syslog entry before going down.

MPIO on Linux

UPDATE 2012-09-30:
URL changed to top level. This is the home to multipath-tools. No longer in the wiki.
Copying page here for reference.


Homepage of the upstream project used to drive the Device Mapper multipathing driver.
As multipathing rhymes with clustering, you might be interested in opensvc GPL clustering stack.

Contacts :


Contribute :

You can contribute code, editorial work and testing.
If you are in such position, you can also contribute hardware or hardware access.

git repository :

mkdir -p ~/scm/multipath-tools.upstream
cd ~/scm/multipath-tools.upstream

build :

On debian based systems :

apt-get install libreadline-dev libaio-dev libdevmapper-dev

Documentation :

Latest release :


incremental bugfixes :

None for now.

Announce :

2010-05-22 multipath-tools-0.4.9 shortlog :

  • Changes that may require your altering your config file :
    • rename hp_sw hwhandler
    • directio is now the default checker (async)
    • deprecated prio_callout keyword: now use prio/prio_args
    • move bindings file location to /etc
    • add ‘max_fds’ keyword
    • add ‘verbosity’ keyword
    • add ‘queue_without_daemon’ keyword
    • add ‘mode’, ‘uid’ and ‘gid’ keywords
    • add ‘fast_io_fail’ and ‘dev_loss_tmo’ keywords
    • add ‘checker’ keyword as alias to ‘path_checker’
  • Features
    • make kpartx deal with more than 256 minor numbers
    • fail/reinstate CLI commands disable/enable path checker
    • support multipath-capable cciss hardware, add cciss_tur checker
    • ‘show config’ output usable as a config file replacement
    • create read-only multipath if rw try failed
    • care about offlined paths
    • support resizing of multipath maps
    • kpartx removes its loop devices
    • support other load-balancing policies
    • add datacore prioritizer
    • add ‘-r’ multipath arg to force maps reload
    • add ‘show paths format’ cli command
    • add ‘show maps format’ cli command
    • add ‘show status’ cli command
    • add ‘show wildcards’ cli command
    • add ‘quit’ cli command
    • add ‘enablequeueing’ and ‘disablequeueing’ cli commands
  • Fixes
    • leaks plugged (virtualiron, suse, redhat work)
    • honour ALUA preference indicator
  • Developers changes :
    • no more prioritizer callouts. replaced by mem-pinned .so to avoid having to cache the callouts.
    • move checkers to mem-pinned .so too
    • allow arguments passing to prioritizers
  • Contributors :
    • Alex Zeffertt <>
    • Andrey Panin <>
    • Babu Moger <>
    • Benjamin Marzinski <>
    • Brian King <>
    • Bryn M.Reeves <>
    • Chandra Seetharaman <>
    • Charlie Brady <>
    • Chauhan, Vijay <>
    • Christophe Varoqui <>
    • Colin Watson <>
    • Dave Wysochanski <>
    • Guido Guenther <>
    • Guido Günther <>
    • Hannes Reinecke <>
    • Jim Lester <>
    • Kiyoshi Ueda <>
    • Konrad Rzeszutek <>
    • Martin Petermann <>
    • Matthew Kent <>
    • Mike Anderson <>
    • Mike Snitzer <>
    • Moger, Babu <>
    • Nikanth Karthikesan <>
    • Ritesh Raj Sarraf <>
    • Rumko <>
    • Stefan Bader <>
    • Stefan Haberland <>
    • Stefan Weinhuber <>
    • Yanqing Liu <>

Dependencies :

Linux kernel

  • 2.6.10-rc*-udm2 or later
  • 2.6.11-mm* or later
  • 2.6.12-rc1 or later

udev 050+
libdevmapper 1.2.38+

Known issues :

  • Update your config file : selector = ’round-robin 0′ (since 0.3.9)
  • If you relied on the ‘-D’ multipath command flag, please adapt (since 0.4.0).
  • The kernel feature ‘queue_if_no_path’ is known to have bugs (up to 2.6.10rc?-udm2).
  • The EMC checker has a known bug, fixed in the next release (up to 0.4.1).
  • The tools exits if /sys/fc_transport/target$h:$b:$t does not exists : not fair, fixed in next release (0.4.1 only).
  • failback doesn’t work (up to 0.4.5 included)
  • make rpm doesn’t work (0.4.5 only)

Previous releases :


Change logs :

full pre-0.4.5 ChangeLog
0.4.5+ changes are tracked through the git repository : here

2007-08-02 multipath-tools-0.4.8 shortlog :

  • Changes that may require your altering your config file :
    • seed cciss multipathing support for recent MSA hardware : warning, the cciss blacklist is not hardcoded anymore. Proliant owners should add it to your config file if you don’t want surprises.
    • pp_sgi renamed to pp_rdac : no behaviour change otherwise
    • Add ‘^dcssblk[0-9]*’ to default blacklist
    • Directio is now the default checker
    • Move id callouts to standardized /lib/udev
    • Adopt Suse udev rules
    • Add path group timeout handling (pg_timeout keyword)
    • Add blacklist exceptions
    • Almost every vendor updated the internal defaults for their hardware (‘show config’ cli command is your friend)
    • We support matching by SCSI revision now, in addition to Vendor and Product. No rev specified means all rev matches.
    • Blacklist entries and other strings in the config file are true regular expressions, not shell expansion syntax
  • Features
    • prio: add a prioritizer for HP storageworks arrays
    • prio: add a prioritizer for HDS modular arrays
    • checkers: asynchronous checker framework. Directio is the first and only user. Means no more checking loop blocking (while holding lock the cli tries to acquire for example).
    • checkers: EMC checker support for ALUA EMC arrays
    • checkers: add RDAC path checker
    • lib: pick up existing alias (if any) from current dm config
    • cli: add syntax auto-completion
    • cli: new command, show blacklist
    • doc: More man pages
    • kpartx: handling for SUN partition tables
    • kpartx: c-slice handling for BSD partition tables
    • kpartx: handling for extended partitions
    • kpartx: accept /dev/dm-* as arg
  • Fixes
    • checkers: add a timeout to SG_IO ioctls
    • checkers: read buffer size bumped to 4096 bytes (Infiniband)
    • checkers: handling CLARiiON I/O to inactive snapshot logical units
    • lib: support newer kernels sysfs layout (remove libsysfs dependency)
    • lib: we keep stale paths in multipaths in case they get back online, but multipathd had trouble dealing with device name change in this situation : fixed
    • lib: listen events from udev socket instead of netlink : proper serialization allows to remove wait_for_devnode_appearance() hacks
    • lib: fix user_friendly_names race leading to a mpathX being served multiple times
    • lib: fix bug limiting to <8 paths luns
    • lib: don’t discard paths with large uuid
    • lib: don’t segfault when no config file
    • kpartx: large file support
    • kpartx: fix partition calculations of DASD partitions
    • prio: fix ALUA for minor >256
  • Developers changes :
    • checkers: Checkers attached to different paths of a same multipath can share information through a context pointer in ‘struct multipath’
    • lib: use DM uuid ‘mpath-‘ prefix
    • kpartx: use DM uuid ‘part%d-‘ prefix

2006-03-13 multipath-tools-0.4.7 shortlog :

  • multipathd empowering
    • DM driving directly from the daemon, without resorting to multipath(8)
    • udev rules deprecation, as events are acted upon directly through uevent in the daemon. The kpartx rule remains though
  • multipathd CLI commands
    • enh : the CLI parser now accept unambiguous shortcuts, like ‘sho topo’ for ‘show topology’
    • new : ‘show config’ drops a config file as seen by the live daemon
    • new : ‘show maps stats’ drops some statistics about multipaths events
    • new : ‘show topology’, mimics the ‘multipath -ll’ output
    • new : ‘show map $map topology’, mimics the ‘multipath -ll $map’ output
    • enh : ‘reconfigure’ correctness issues fixed, scope extended
    • enh : ‘add/del map’ now apply to the DM live configuration too (!!caution, maps are really removed!!)
    • rem : ‘dump pathvec’, deprecated
  • prioritizers
    • new : netapp prioritizer
    • new : tpc prioritizer (SGI, …)
  • netlink uevent robustness
    • bigger receive buffer
    • uevent queueing and async dequeueing
  • configuration
    • rr_min_io can be put in ‘device’ and ‘multipath’ config blocks
    • internal defaults are now exhaustive
    • tweak defaults for HP, SGI, IBM, EMC, …
  • multipaths renaming through DM renames rather than remove/add cycles
  • logging
    • ‘multipathd -d’ now prints the date
    • overall output beautifying
  • drop the ‘path cache’ framework
  • kpartx
    • fixes to dos partition handler
    • new mac partition handler
  • make the tools work with DASD
  • blacklist
    • add per-device product-string blacklist
    • add a device blacklist
    • add a wwid blacklist
    • deprecate the ‘devnode_blacklist’ keyword ub favor of ‘blacklist’
  • adapt to libsysfs2 api, without breaking compat with libsysfs1
  • checkers
    • API rewrite
    • reduce allocation on the directio checking code path

2005-11-11 multipath-tools-0.4.6 shortlog :

  • fixes
  • add/del cli commands accept map names as param
  • new add ‘reinstate/fail’ cli commands
  • new add ‘suspend/resume’ cli commands
  • extended information display in ‘show map/paths’ cli commands
  • fix thread reaping logic
  • user friendly map names (mpath[0..n]) without alias defines
  • option to disable io queueing if no valid path after a delay
  • vendor/product as regexp in config file
  • set devmap uuid as multipath wwid
  • move hotplug logic to udev’s rules.d/
  • configlets now adds up or replace the hardcoded ones
  • ALUA prioritizer update
  • option to use priorities as weights in the round-robin scheduler
  • add the directio checker

2005-09-02 multipath-tools-0.4.5 shortlog :

  • multipathd keeps Device Mapper path status in sync with path checker status : proactive fail_path and reinstate_path
  • proactive path group re-enable, as soon as one path goes up
  • multipathd is now fully event driven :
    • NetLink uevents for kernels/distro that enables the feature
    • Hotplug
    • Unix socket messages
  • a Command Line Interface to drive and inspect the daemon
  • remove the multipath(8) cache file logic in favour of a multipathd<->multipath unix socket request/reply design
  • split multipath(8) listing option into
    • short-list (-l) : don’t fetch wwid, path checker status, priorities … fast
    • detailed-list (-ll) : refetch all info
  • the daemon asynchronous logger is now solid and avoid sending critical messages
  • daemon CLI command to reload the config file
  • path checking interval gets longer when the path is reliably up, resets to short as soon as a problem occur
  • daemon implements 3 path group failback methods :
    • manual
    • immediate : failback as soon as an (enabled && !active) PG gets the highest priority
    • defered : failback to the highest priority (enabled && !active) PG n seconds after the last PG priority order change
  • remove over-complicated and thread-confusing daemon initial clone() w/ private namespace
  • prioritizers official prefix is now mpath_prio_
  • multipath(8) displays PG priorities when available
  • new hp_sw checker. It’s a tri-state UP/DOWN/GHOST
  • new directio checker. Useful with DASD devices. May suplant readsector0 in the long run ?
  • stricter default blacklist regexps
  • new ‘-f’ multipath(8) flag to selectively suppress a multipath map, and its device-mapped partitions
  • alua prioritizer update, should work on x86
  • support for >2TB multipaths
  • claimed device detection : no more warning/error messages with paths pinned by mounted FS
  • option to use priorities as weights in the round-robin scheduler : for people who can’t wait for a qlen-based scheduler

2005-04-24 multipath-tools-0.4.4 shortlog :

  • 2 new path prioritizers : pp_alua and pp_emc
  • devmap_name can now filter by target_type (-t flag).
  • For example, no output if not ‘multipath’ type.
  • the daemon sets /proc/self/oom_adj to -17, hence is
  • protected from the oom killer. Watch out for leaks.
  • libdevmapper 1.01.01 or newer is now recquired
  • blacklist synthax is now regex based. Update your config files. The packaged templates are up to date.
  • you can now set wwid in the blacklist
  • multipathd gained a ‘-v’ flag to tune verbosity
  • daemon logging is now staged and asynchronous to avoid syslogd possible hangs
  • droped local klibc, libsysfs and libdevmapper
  • multipath caches path info. Cache expire in 5sec.
  • ‘multipath -l’ accepts a devname or dev_t filter