Erroneous "No space left on device" #421

kurczynski · 2020-05-02T00:41:55Z

kurczynski
May 2, 2020

I've created a Stratis pool using two 6 TB LUKS devices:

$ stratis blockdev
Pool Name        Device Node                            Physical Size   Tier
redundant-pool   /dev/mapper/stratis-backup-storage-0        5.46 TiB   Data
redundant-pool   /dev/mapper/stratis-backup-storage-1        5.46 TiB   Data

$ stratis pool
Name                                 Total Physical
redundant-pool   10.92 TiB / 1020.59 GiB / 9.92 TiB

With a filesystem on top of it:

$ stratis fs
Pool Name        Name             Used          Created             Device                                   UUID
redundant-pool   backup-storage   1020.50 GiB   May 01 2020 12:24   /stratis/redundant-pool/backup-storage   75a9f6ee5f614d88b35c11f43899bcf9

However, I'm not able to use more than about 1 TB of space on the filesystem:

$ fallocate -l 10G test.img
fallocate: fallocate failed: No space left on device

Are there any other steps that need to be taken to use all of the available storage?

I'm using Stratis 2.0.1 on Fedora 32.

drckeefe · 2020-05-04T12:59:44Z

drckeefe
May 4, 2020
Maintainer

kurczynski, would you mind providing the blkid ouput? Also, are there any errors for stratis in the logs?

0 replies

drckeefe · 2020-05-04T20:36:07Z

drckeefe
May 4, 2020
Maintainer

@kurczynski
We discussed this issue in stand-up today and require more information about this issue.
Can you provide information about how this filesystem was filled (which tools or applications)?
I wonder if it is possible if fallocate sparsely filled the filesystem without writing any data to the
underlying storage. In this case, it might be possible to consume the space without Stratis
knowing the filesystem should be extended.

0 replies

kurczynski · 2020-05-05T02:32:35Z

kurczynski
May 5, 2020
Author

I've since formatted the disks to use a btrfs raid0, so I don't have the blkid info. But I did check the logs and there was nothing but INFO and DEBUG level messages. I can post that if it may be helpful, it's over 500 lines though.

Of course. I was using rsync to transfer files from my single 6 TB btrfs disk to the Stratis pool I created and it failed with a No space left on device error during transfer. I restarted it a couple of times but was met with the same error. I ended up trying fallocate just to see if maybe it was something to do with the way rsync prepared files before sending them but as I posted, it came back with the same error.

0 replies

drckeefe · 2020-05-06T15:56:47Z

drckeefe
May 6, 2020
Maintainer

Thank you for the feedback, kurczynski.
From the information provided in this issue I'm adding a link to this bugzilla. https://bugzilla.redhat.com/show_bug.cgi?id=1687974

@mulkieran do we have another github issue related to filesystems filling before a grow can complete?

0 replies

jwtrhs · 2022-02-16T08:13:04Z

jwtrhs
Feb 16, 2022

Sorry to revive an old issue, but I believe I'm experiencing the same behavior as OP. I've got a 4TB HDD with an SSD cache configured as a storage pool with a single filesystem.

$ stratis blockdev
Pool Name   Device Node      Physical Size    Tier
nas         /dev/md0              3.64 TiB    Data
nas         /dev/nvme0n1p3      209.26 GiB   Cache

$ stratis fs
Pool Name   Name   Used          Created             Device                 UUID
nas         nas    1017.06 GiB   Feb 16 2022 02:06   /dev/stratis/nas/nas   f0b957b8-4d9a-41e0-9434-8ee823ca8561

I'd been using rsync as well to copy a couple of TB of data to the stratis storage pool from another HDD, and am getting No space left on device errors.

Also trying with fallocate gives the same error as above.

$ fallocate -l 10G /mnt/nas/test.img
fallocate: fallocate failed: No space left on device

I'm on version 2.4.2.

$ stratis --version
2.4.2

Most of the files being copied with rsync are quite large (10-40GB) if that helps.

Is there any way to manually increase the size of a filesystem? Thanks.

Edit:
Also found this in the logs after creating the filesystem:

Feb 16 02:06:40 home-server stratisd[11147]: [2022-02-16T02:06:40Z INFO  libstratis::dbus_api::pool::pool_2_0::methods] The following filesystems name: nas, UUID: f0b957b8-4d9a-41e0-9434-8ee823ca8561 were successfully created
Feb 16 02:06:40 home-server stratisd[11147]: thread 'stratis-wt-5' panicked at 'assertion failed: `(left == right)`
Feb 16 02:06:40 home-server stratisd[11147]:   left: `Device { major: 253, minor: 7 }`,
Feb 16 02:06:40 home-server stratisd[11147]:  right: `Device { major: 253, minor: 0 }`', src/engine/strat_engine/thinpool/thinpool.rs:516:9
Feb 16 02:06:40 home-server stratisd[11147]: stack backtrace:
Feb 16 02:06:40 home-server stratisd[11147]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

0 replies

drckeefe · 2022-02-16T14:34:47Z

drckeefe
Feb 16, 2022
Maintainer

Could be related to this issue: #391
@jwtrhs do you have the output of stratis pool list?

0 replies

jwtrhs · 2022-02-16T22:38:09Z

jwtrhs
Feb 16, 2022

Thanks for the quick reply.

stratis pool list
Name                      Total Physical   Properties                                   UUID
nas    3.64 TiB / 1018.23 GiB / 2.64 TiB       Ca,~Cr   7d237765-7f70-4d59-8da0-48c80b47f432

I've added the filesystem to my fstab now, and am getting a log from stratisd saying an attempt to mount has failed due to missing metadata, though it does seem to succeed afterwards. Not sure if that's relevant.

Feb 16 22:32:53 home-server systemd[1]: Started Stratis daemon.
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z DEBUG libstratis::engine::strat_engine::liminal::identify] Stratis block device with Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "c34a4ef3-6d03-4fb6-a7dd-3368cab42150", device number: "259:3", devnode: "/dev/nvme0n1p3" identified
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z INFO  libstratis::engine::strat_engine::liminal::device_info] Device information Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "c34a4ef3-6d03-4fb6-a7dd-3368cab42150", device number: "259:3", devnode: "/dev/nvme0n1p3" discovered and inserted into the set for its pool UUID
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z INFO  libstratis::engine::strat_engine::liminal::liminal] Attempt to set up pool failed, but it may be possible to set up the pool later, if the situation changes: No metadata found on devices associated with pool UUID 7d237765-7f70-4d59-8da0-48c80b47f432
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z DEBUG libstratis::engine::strat_engine::liminal::identify] Stratis block device with Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "13dbd992-7838-4775-b312-30f4f9a97718", device number: "9:0", devnode: "/dev/md0" identified
Feb 16 22:32:53 home-server stratisd[304]: [2022-02-16T22:32:53Z INFO  libstratis::engine::strat_engine::liminal::device_info] Device information Stratis device description: Stratis pool UUID: "7d237765-7f70-4d59-8da0-48c80b47f432", Stratis device UUID: "13dbd992-7838-4775-b312-30f4f9a97718", device number: "9:0", devnode: "/dev/md0" discovered and inserted into the set for its pool UUID
Feb 16 22:32:55 home-server stratisd[304]: [2022-02-16T22:32:55Z INFO  libstratis::engine::strat_engine::thinpool::thinpool] Status of thinpool device with "devicemapper name: stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool, device number: 254:6, device node: /dev/dm-6" changed from "none" to "rw"
Feb 16 22:32:55 home-server stratisd[304]: [2022-02-16T22:32:55Z INFO  libstratis::engine::strat_engine::liminal::liminal] Pool with name "nas" and UUID "7d237765-7f70-4d59-8da0-48c80b47f432" set up
Feb 16 22:33:22 home-server stratisd[304]: [2022-02-16T22:33:22Z DEBUG libstratis::stratis::run] 11: thread started
Feb 16 22:33:32 home-server stratisd[304]: [2022-02-16T22:33:32Z DEBUG libstratis::stratis::run] 0: thread finished

0 replies

drckeefe · 2022-02-17T04:39:18Z

drckeefe
Feb 17, 2022
Maintainer

@jwtrhs

OK, there isn't enough information yet, but I have some thoughts.
There might be two issues here.

the filesystem is reporting out of space (can you provide the output of a df?)
the cache device isn't set up correctly (this could be a known bug)

Some Information:
To fix the cache you could do one of a few things, let me know if you have done these already.

Reboot - this might be the easiest way to resolve the issue. (after you reboot check the filesystem file from df. Did it change?)
Manually bring down the Stratis devices. I can provide this process if you can provide the output of a lsblk.
Add a second cache device to your configuration. This is typically not an option for most.

It is possible that the cache issue is preventing the filesystem from growing, so fixing the cache
state should be first. Stratis 3.0 has a few improvement to handle cases like this. Is there any opportunity to upgrade to a 3.0 version?

0 replies

jwtrhs · 2022-02-17T06:21:00Z

jwtrhs
Feb 17, 2022

Here's the df:

$ df
Filesystem                                                                                       1K-blocks       Used Available Use% Mounted on
dev                                                                                                8126424          0   8126424   0% /dev
run                                                                                                8136636       5920   8130716   1% /run
/dev/nvme0n1p2                                                                                   263174212   59300840 190435220  24% /
tmpfs                                                                                              8136636          0   8136636   0% /dev/shm
tmpfs                                                                                              8136640          0   8136640   0% /tmp
tmpfs                                                                                                 1024          0      1024   0% /run/stratisd/keyfiles
/dev/nvme0n1p1                                                                                      523248      56876    466372  11% /boot
/dev/mapper/stratis-1-7d2377657f704d598da048c80b47f432-thin-fs-f0b957b84d9a41e094348ee823ca8561 1073217536 1072126356   1091180 100% /mnt/nas

I've tried rebooting a few times but no luck. I've also tried deleting ~100GB of files on the stratis filesystem and copying them again to see if it would trigger a resize, but that didn't work either.

I'm on Arch and their community package is still version 2.4.2, but it was flagged out of date a few days ago. In my case this isn't critical data, so happy to wait for the repo to hopefully update soon and see if it resolves the issue especially if you think this is likely fixed in the new release.

But I'm happy to debug it further too if it would be helpful. Here's my lsblk:

lsblk
NAME                                                                                            MAJ:MIN RM   SIZE RO TYPE    MOUNTPOINTS
sda                                                                                               8:0    0   3.6T  0 disk
└─md0                                                                                             9:0    0   3.6T  0 raid1
  └─stratis-1-private-7d2377657f704d598da048c80b47f432-physical-originsub                       254:0    0   3.6T  0 stratis
    └─stratis-1-private-7d2377657f704d598da048c80b47f432-physical-cache                         254:3    0   3.6T  0 stratis
      ├─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-thinmeta                        254:4    0   3.7G  0 stratis
      │ └─stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool                      254:6    0   3.6T  0 stratis
      │   └─stratis-1-7d2377657f704d598da048c80b47f432-thin-fs-f0b957b84d9a41e094348ee823ca8561 254:8    0     1T  0 stratis /mnt/nas
      ├─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-thindata                        254:5    0   3.6T  0 stratis
      │ └─stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool                      254:6    0   3.6T  0 stratis
      │   └─stratis-1-7d2377657f704d598da048c80b47f432-thin-fs-f0b957b84d9a41e094348ee823ca8561 254:8    0     1T  0 stratis /mnt/nas
      └─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-mdv                             254:7    0    16M  0 stratis
sdb                                                                                               8:16   0   3.6T  0 disk
└─sdb1                                                                                            8:17   0   3.6T  0 part
  └─md127                                                                                         9:127  0   3.6T  0 raid1
nvme0n1                                                                                         259:0    0 465.8G  0 disk
├─nvme0n1p1                                                                                     259:1    0   512M  0 part    /boot
├─nvme0n1p2                                                                                     259:2    0   256G  0 part    /var/lib/containers/storage/overlay
│                                                                                                                            /
└─nvme0n1p3                                                                                     259:3    0 209.3G  0 part
  ├─stratis-1-private-7d2377657f704d598da048c80b47f432-physical-metasub                         254:1    0   512M  0 stratis
  │ └─stratis-1-private-7d2377657f704d598da048c80b47f432-physical-cache                         254:3    0   3.6T  0 stratis
  │   ├─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-thinmeta                        254:4    0   3.7G  0 stratis
  │   │ └─stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool                      254:6    0   3.6T  0 stratis
  │   │   └─stratis-1-7d2377657f704d598da048c80b47f432-thin-fs-f0b957b84d9a41e094348ee823ca8561 254:8    0     1T  0 stratis /mnt/nas
  │   ├─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-thindata                        254:5    0   3.6T  0 stratis
  │   │ └─stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool                      254:6    0   3.6T  0 stratis
  │   │   └─stratis-1-7d2377657f704d598da048c80b47f432-thin-fs-f0b957b84d9a41e094348ee823ca8561 254:8    0     1T  0 stratis /mnt/nas
  │   └─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-mdv                             254:7    0    16M  0 stratis
  └─stratis-1-private-7d2377657f704d598da048c80b47f432-physical-cachesub                        254:2    0 208.8G  0 stratis
    └─stratis-1-private-7d2377657f704d598da048c80b47f432-physical-cache                         254:3    0   3.6T  0 stratis
      ├─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-thinmeta                        254:4    0   3.7G  0 stratis
      │ └─stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool                      254:6    0   3.6T  0 stratis
      │   └─stratis-1-7d2377657f704d598da048c80b47f432-thin-fs-f0b957b84d9a41e094348ee823ca8561 254:8    0     1T  0 stratis /mnt/nas
      ├─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-thindata                        254:5    0   3.6T  0 stratis
      │ └─stratis-1-private-7d2377657f704d598da048c80b47f432-thinpool-pool                      254:6    0   3.6T  0 stratis
      │   └─stratis-1-7d2377657f704d598da048c80b47f432-thin-fs-f0b957b84d9a41e094348ee823ca8561 254:8    0     1T  0 stratis /mnt/nas
      └─stratis-1-private-7d2377657f704d598da048c80b47f432-flex-mdv                             254:7    0    16M  0 stratis

1 reply

jbaublitz Feb 21, 2022
Maintainer

Hi @jwtrhs! This looks like a similar problem to what we saw previously. @drckeefe's suggestion below about creating a separate filesystem and writing more than 2 GB to it and then calling sync may work as a temporary workaround. My guess is that this behavior is related to rsync's IO pattern so I'll need to do some further research into what could be causing that.

drckeefe · 2022-02-21T16:07:37Z

drckeefe
Feb 21, 2022
Maintainer

@jwtrhs
Thank you for the output. I've tried to reproduce this a few times (not on arch), but haven't had any luck.
Attempts:
OS: RHEL

created pool and fs, mounted fs, wrote zeros to a single file until fs grew from 1TB to 2TB
created pool and fs, mounted fs, wrote zeros to multiple 50GB files until fs grew from 1TB to 2TB
created second fs from the same pool as attempt 2, then rsync'd files from fs1 to fs2 until fs grew from 1TB to 2TB

I'm working on finding a system to install Arch on and running the same process as above. Without recreating this
issue any guidance I provide is just a guess at the moment. As an example: would creating another filesystem in
this pool and filling it to near fill (less than 2GB) of the max, trigger both filesystems to grow?

@jbaublitz - any thoughts here?

1 reply

jbaublitz Feb 21, 2022
Maintainer

Hi @drckeefe! I think creating a separate filesystem and writing to that could cause the low water mark to be passed. If this is a problem with the low water mark, that would cause the filesystem extension on all filesystems for that pool to be retried. I am still unsure of why this is happening in the first place. This seems to be resolved in 3.0.

drckeefe · 2022-02-21T17:08:13Z

drckeefe
Feb 21, 2022
Maintainer

@jwtrhs I would be also curious to see the output of dmsetup status too.

0 replies

drckeefe · 2022-02-23T15:21:00Z

drckeefe
Feb 23, 2022
Maintainer

I've found hardware to set up 2TB SSD with Archlinux and then run a few attempts to recreate this issue, but each time the
filesystem grows to 2TB. In my test I create 40 files using truncate each file is 50GB in size.

for i in $(seq 1 40);do truncate -s 50g truncatedFiles/file.$i;done

Though these are empty files, rsync will read zeros when copying them to the Stratis filesystem.

I wasn't able to reproduce this on two attempts. My last attempt I tried to recreate the issues running multiple rsync jobs
at the same time. Here I was trying to create a high enough load on the system to see this issue. The idea is, if the system is busy
enough, can the attempt to grow be missed. The load on the system during this test was 8.39.

Run multiple rsync jobs:
ls -1 truncatedFiles/ | xargs -I {} -P 5 -n 1 rsync -avh truncatedFiles{} /fs1

I'll pause on testing further until we discuss further here and/or have more information.

0 replies

drckeefe · 2022-03-02T00:15:46Z

drckeefe
Mar 2, 2022
Maintainer

I did some further investigation after talking to jbaublitz. I can reproduce this issue on Arch Linux by doing the following:

create a pool
create a cache
create a filesystem
fill the filesystem
At this point the filesystem will not grow as a result of this bug:
add-data after init-cache causes assertion failure in stratisd stratisd#2789

This bug is fixed in stratisd 3.0+

A description of the cache related issue can be found here:
https://bugzilla.redhat.com/show_bug.cgi?id=2007018#c4

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Erroneous "No space left on device" #421

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 13 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Erroneous "No space left on device" #421

kurczynski May 2, 2020

Replies: 13 comments · 2 replies

drckeefe May 4, 2020 Maintainer

drckeefe May 4, 2020 Maintainer

kurczynski May 5, 2020 Author

drckeefe May 6, 2020 Maintainer

jwtrhs Feb 16, 2022

drckeefe Feb 16, 2022 Maintainer

jwtrhs Feb 16, 2022

drckeefe Feb 17, 2022 Maintainer

jwtrhs Feb 17, 2022

jbaublitz Feb 21, 2022 Maintainer

drckeefe Feb 21, 2022 Maintainer

jbaublitz Feb 21, 2022 Maintainer

drckeefe Feb 21, 2022 Maintainer

drckeefe Feb 23, 2022 Maintainer

drckeefe Mar 2, 2022 Maintainer

kurczynski
May 2, 2020

Replies: 13 comments 2 replies

drckeefe
May 4, 2020
Maintainer

drckeefe
May 4, 2020
Maintainer

kurczynski
May 5, 2020
Author

drckeefe
May 6, 2020
Maintainer

jwtrhs
Feb 16, 2022

drckeefe
Feb 16, 2022
Maintainer

jwtrhs
Feb 16, 2022

drckeefe
Feb 17, 2022
Maintainer

jwtrhs
Feb 17, 2022

jbaublitz Feb 21, 2022
Maintainer

drckeefe
Feb 21, 2022
Maintainer

jbaublitz Feb 21, 2022
Maintainer

drckeefe
Feb 21, 2022
Maintainer

drckeefe
Feb 23, 2022
Maintainer

drckeefe
Mar 2, 2022
Maintainer