Troubleshooting DataCore Bolt
Manual volume file system recovery
When a file system is corrupted, the below steps are followed to attempt to recover the data.
-
Identify volumes with a corrupt file system.
The volumes cannot be mounted by the bolt-csi node plugin. In the pod that tried to make use of the volume we may see something similar to the following:
Copy│ Warning FailedMount 2s (x5 over 12s) kubelet MountVolume.MountDevice failed for v │
│ olume "pvc-897f97c1-c72a-46a5-bce0-677e6cd2843e" : rpc error: code = Internal desc = Failed to stage volume 897f97 │
│ c1-c72a-46a5-bce0-677e6cd2843e: failed to mount device /dev/nvme0n1 onto /var/lib/kubelet/plugins/kubernetes.io/cs │
│ i/pv/pvc-897f97c1-c72a-46a5-bce0-677e6cd2843e/globalmount: Structure needs cleaning (os error 117)The
dmesgmay show something like the following:Copy[Wed Mar 9 18:28:15 2022] nvme nvme0: new ctrl: NQN "nqn.2019-05.io.openebs:897f97c1-c72a-46a5-bce0-677e6cd2843e", addr 192.168.122.248:8420
[Wed Mar 9 18:28:16 2022] XFS (nvme0n1): Mounting V5 Filesystem
[Wed Mar 9 18:28:16 2022] XFS (nvme0n1): Metadata CRC error detected at xfs_agi_read_verify+0xc8/0xf0 [xfs], xfs_agi block 0x2
[Wed Mar 9 18:28:16 2022] XFS (nvme0n1): Unmount and run xfs_repair
[Wed Mar 9 18:28:16 2022] XFS (nvme0n1): First 128 bytes of corrupted metadata buffer:
[Wed Mar 9 18:28:16 2022] 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[Wed Mar 9 18:28:16 2022] XFS (nvme0n1): metadata I/O error in "xfs_trans_read_buf_map" at daddr 0x2 len 1 error 74
[Wed Mar 9 18:28:16 2022] XFS (nvme0n1): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -117, agno 0
[Wed Mar 9 18:28:16 2022] XFS (nvme0n1): Failed to read root inode 0x80, error 117
[Wed Mar 9 18:28:16 2022] nvme nvme0: Removing ctrl: NQN "nqn.2019-05.io.openebs:897f97c1-c72a-46a5-bce0-677e6cd2843eThe kubelet will keep retrying and failing, unless the filesystem is fixed. It is not advisable to fix it at this stage, as the volume will be staged or unstaged by the kubelet during every retry. An exclusive and stable connection to the volume is required in order to attempt to fix the file system.
-
Delete all pods that make use of the PVC.
-
Ensure no nodes have access to the volume.
After ensuring exclusive access to the volume, get the volume uuid from the pv name which has the format (pvc-$uuid)
Copyuser@node-name:~$ kubectl get pvc pvc-name
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-name Bound pvc-897f97c1-c72a-46a5-bce0-677e6cd2843e 500Mi RWO my-storage-class 166mIn this case, the uuid is
897f97c1-c72a-46a5-bce0-677e6cd2843eUsing the
nvme-cliutility, ssh to all bolt-csi nodes and ensure that there are no remaining connections.Copyuser@node-name:~$ sudo nvme disconnect -n nqn.2019-05.io.openebs:897f97c1-c72a-46a5-bce0-677e6cd2843e -
Make the volume available on a node which can be connected using ssh.
Publish the volume to a bolt-csi node (current constraint, which may be lifted in the future). Use the below curl:
Copyuser@node-name:~$ curl -X PUT '$NODE_IP:30011/v0/volumes/897f97c1-c72a-46a5-bce0-677e6cd2843e/target?protocol=nvmf&node=node-name'The json output of the curl command should contain the following field:
"
deviceUri":"nvmf://192.168.122.43:8420/nqn.2019-05.io.openebs:897f97c1-c72a-46a5-bce0-677e6cd2843e"The IP address is of the node that is just published.
When the volume is published, a ssh is established into the node and then by using the nvme-cli utility, the node is instructed to connect to the nvmf volume target.
Copyuser@node-name:~$ sudo nvme connect -t tcp -s 8420 -a 192.168.122.43 -n nqn.2019-05.io.openebs:897f97c1-c72a-46a5-bce0-677e6cd2843e
user@node-name:~$ sudo nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 DCSD1B1275F80CF3596E DataCore NVMe controller 1 519.03 MB / 519.03 MB 512 B + 0 B 22.01 -
Repair the filesystem as appropriate
This will vary depending on your chosen filesystem. For example,
xfsprogscan provide you with several userspace utilities to manage xfs file systems.The below example shows fixing a volume with a corrupt superblock:
Copyuser@node-name:~$ sudo xfs_info /dev/nvme0n1
Metadata CRC error detected at 0x562b1021dbee, xfs_agf block 0x1/0x200
xfs_info: cannot init perag data (74). Continuing anyway.
meta-data=/dev/nvme0n1 isize=512 agcount=4, agsize=31679 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=126715, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1368, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
user@node-name:~$ sudo xfs_repair /dev/nvme0n1
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
Metadata CRC error detected at 0x5580577afeee, xfs_agf block 0x1/0x200
Metadata CRC error detected at 0x55805778c41e, xfs_agi block 0x2/0x200
bad magic # 0x0 for agf 0
bad version # 0 for agf 0
bad length 0 for agf 0, should be 31679
bad uuid 00000000-0000-0000-0000-000000000000 for agf 0
bad magic # 0x0 for agi 0
bad version # 0 for agi 0
bad length # 0 for agi 0, should be 31679
bad uuid 00000000-0000-0000-0000-000000000000 for agi 0
reset bad agf for ag 0
reset bad agi for ag 0
bad agbno 0 for btbno root, agno 0
bad agbno 0 for btbcnt root, agno 0
bad agbno 0 for refcntbt root, agno 0
bad agbno 0 for inobt root, agno 0
bad agbno 0 for finobt root, agno 0
agi unlinked bucket 0 is 0 in ag 0 (inode=0)
....
agi unlinked bucket 63 is 0 in ag 0 (inode=0)
sb_icount 64, counted 0
sb_ifree 60, counted 0
sb_fdblocks 10115, counted 9889
root inode chunk not found
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
correcting imap
- agno = 1
- agno = 2
- agno = 3
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
user@node-name:~$ sudo xfs_info /dev/nvme0n1
meta-data=/dev/nvme0n1 isize=512 agcount=4, agsize=31679 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=126715, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1368, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0 -
Disconnect from the volume from the node.
Copyuser@node-name:~$ sudo nvme disconnect -n nqn.2019-05.io.openebs:897f97c1-c72a-46a5-bce0-677e6cd2843e3 -
Unpublish the volume.
Copyuser@node-name:~$ curl -X DELETE '$NODE_IP:30011/v0/volumes/897f97c1-c72a-46a5-bce0-677e6cd2843e/target'