Automatically resolving Gluster GFID Mismatch

Published: 2019-12-13 10:42:47 +0000
Categories: BASH,




GFID mismatch is something that can occur on nodes running GlusterFS, particularly after a split-brain or similar incident.

The symptoms of a GFID mismatch are I/O errors for certain files within the Gluster filesystem, and/or question marks in the output of ls. Ultimately, though within Glusters logs you'll find Gfid mismatch detected.

The technically correct way to resolve GFID mismatch is to compare mtimes for each affected GFID in order to identify which node has the most recent copy.

However, where a lot of file are affected, this may not be possible due to the time/effort required - particularly if there's a reasonable certainty that the files themselves don't differ between the nodes

This snippet provides a script which will pull affected GFIDs from the Gluster logs, resolve those GFIDs back to a path and then move that path. It should be run on all but one of your Gluster nodes

Based On


# Defaults:
# - gluster volume called "shared". Set environment variable GLUSTERVOL to override
# - brick paths /data1/gluster and /data2/gluster. Set env var BRICKS to a space seperated list to override
# Should run on all but one of your gluster nodes. That remaining node will be the "master" copy
# Note - this script pays absolutely *no* attention to which node has the most 
# recent copy of the data
# If you care about that, then see 

BRICKS=${BRICKS:-"/data1/gluster /data2/gluster"}

# Pull the affected GFIDs out of the gluster logs
GFIDS=$(grep "Gfid mismatch detected" /var/log/glusterfs/${GLUSTERVOL}.log | \
        grep -o -P "<gfid:([0-9a-fA-F\-]+)" | sed 's/<gfid://g' | sort | uniq)

# Grab a copy of the GFID resolve script
curl -o ./
chmod +x

# Assumption: the affected GFID will always be a directory, 
# so will exist on all bricks
brick1=$(echo "$BRICKS" | cut -d\  -f1)

for gfid in $GFIDS
    gfid_dir_path=$(./ "$brick1" "$gfid" -q)
    brickless_path=$(echo "$gfid_dir_path" | sed "s~^$brick1~~g")

    echo "Path: $brickless_path will be moved to $brickless_path.old"

    for brick in $BRICKS
        mv "$brick/$brickless_path" "$brick/$brickless_path.old"  

echo "You'll need to run 'gluster volume heal $GLUSTERVOL' on one of your nodes once you've run this script on all but one"

Usage Example

BRICKS="/mnt/brick1 /mnt/brick2"
export BRICKS
gluster volume heal $GLUSTERVOL 




gluster, glusterfs, gfid, mismatch,

Latest Posts

Copyright © 2022 Ben Tasker | Sitemap | Privacy Policy
Available at, http://phecoopwm6x7azx26ctuqcp6673bbqkrqfeoiz2wwk36sady5tqbdpqd.onion and http://snippets.bentasker.i2p
hit counter