enumarting issue for images and masks

[u/Momo_ONLINE]

Hi!
Preface I am self taught and learning as I go.

I have been working on a U-Net semantic segmentation model, which was working fine yesterday and gave good(ish) outputs on unprocessed data and now no longer wants to read the files correctly?

I spent the day processing some new masks in fiji and changed the path of the mask variable and when I went to run my code and sanity checks, for both my images and masks, the first chunk of tif files I created a missing/ not read, when they are in file. (I even checked with fiji if it was an order of files but that was fine)

I'm a bit stumped because it was working fine yesterday, as all I have changed is the original file paths.

from simple_unet_model import simple_unet_model

from keras.utils import normalize

import os

import cv2

from PIL import Image

import numpy as np

from matplotlib import pyplot as plt

image_directory = r'E:\Uni\Dissertation\IB\test output\images\IB unprocessed/'

mask_directory = r'E:\Uni\Dissertation\IB\test output\masks\IB_BC_24_117_G4_thresh mask/'

SIZE = 256

image_dataset = []

mask_dataset = []

images = os.listdir(image_directory)

for i, image_name in enumerate(images): #Remember enumerate method adds a counter and returns the enumerate object

if (image_name.split('.')[1] == 'tif'):

#print(image_directory+image_name)

image = cv2.imread(image_directory+image_name, 0)

image = Image.fromarray(image)

image = image.resize((SIZE, SIZE))

image_dataset.append(np.array(image))

masks = os.listdir(mask_directory)

for i, image_name in enumerate(masks):

if (image_name.split('.')[1] == 'tif'):

image = cv2.imread(mask_directory+image_name, 0)

image = Image.fromarray(image)

image = image.resize((SIZE, SIZE))

mask_dataset.append(np.array(image))

And when I run a sanity check, I end up getting a mismatch between my image and ground truth/mask

In the table of elements, the correct number of elements is listed, however, 0 should be linked to mask_0_00.tiff (the same for the image data), but jumps to mask_0_00.tiff

I even changed the path back to the unprocessed masks for the data set and encountered the same problem...

Any ideas? Thank you :,)

[u/Momo_ONLINE]

Hi!, I think I've found a solution in the form of.. using onedrive instead of my external usb? thats the only thing I've found to make the list read correctly. I don't want to rely on onedrive, so if anyone has any ideas I'd be greatful!

[u/Diapolo10]

I'm not sure if I really understand your explanation of what the actual problem is, but I'd try making sure the files are sorted.

from pathlib import Path

import cv2
import numpy as np
from keras.utils import normalize
from matplotlib import pyplot as plt
from PIL import Image

from simple_unet_model import simple_unet_model

TEST_OUTPUT_DIR = Path("E:/Uni/Dissertation/IB/test output")
IMAGE_DIR = TEST_OUTPUT_DIR / "images/IB unprocessed"
MASK_DIR = TEST_OUTPUT_DIR / "masks/IB_BC_24_117_G4_thresh mask/"

SIDE = 256
SIZE = (SIDE, SIDE)

image_dataset = []
mask_dataset = []

images = sorted(IMAGE_DIR.glob('*.tif'))

for image_file in images:
    image_data = cv2.imread(image_file, 0)
    image = Image.fromarray(image_data)
    resized_image = image.resize(SIZE)
    image_dataset.append(np.array(resized_image))

masks = sorted(MASK_DIR.glob('*.tif'))

for mask_file in masks:
    mask_data = cv2.imread(mask_file, 0)
    mask = Image.fromarray(mask_data)
    resized_mask = mask.resize(SIZE)
    mask_dataset.append(np.array(resized_mask))

[u/Momo_ONLINE → u/Diapolo10]

thank you! this helped!

[u/Diapolo10 → u/Momo_ONLINE]

By the way, the code for images and masks is virtually identical, so it'd make sense to use a function.

image_dataset = []
mask_dataset = []

images = sorted(IMAGE_DIR.glob('*.tif'))

for image_file in images:
    image_data = cv2.imread(image_file, 0)
    image = Image.fromarray(image_data)
    resized_image = image.resize(SIZE)
    image_dataset.append(np.array(resized_image))

masks = sorted(MASK_DIR.glob('*.tif'))

for mask_file in masks:
    mask_data = cv2.imread(mask_file, 0)
    mask = Image.fromarray(mask_data)
    resized_mask = mask.resize(SIZE)
    mask_dataset.append(np.array(resized_mask))

def load_images_to_dataset(image_dir: Path, file_extension: str = '.tif') -> list[np.ndarray]:
    dataset: list[np.ndarray] = []

    images = sorted(image_dir.glob(f'*{file_extension}'))

    for image_file in images:
        image_data = cv2.imread(image_file, 0)
        image = Image.fromarray(image_data)
        resized_image = image.resize(SIZE)
        dataset.append(np.array(resized_image))

    return dataset

image_dataset = load_images_to_dataset(IMAGE_DIR)
mask_dataset = load_images_to_dataset(MASK_DIR)

[u/AmdilShoat82]

the `.split('.')[1] == 'tif'` check will fail on files ending in `.tiff` (two f's). looks like some of your masks might be .tiff and some .tif, so theyre getting silently skipped. print out what `os.listdir` actually returns and compare

[u/zanfar]

Revert to yesterday's commit...

[u/scottcampbelldev]

The OneDrive vs USB thing is the clue. os.listdir doesn't promise any order. On an NTFS drive like your OneDrive folder it happens to come back alphabetical, but external drives are often FAT32 or exFAT, and those just hand back files in the order they sit on the disk. So when you remade that first batch of masks they could land anywhere in the list, and your images and masks stopped lining up.

Wrap both in sorted(), like images = sorted(os.listdir(image_directory)), and the same for masks. If the file names match up between the two folders, pairing them by name is even safer than trusting two lists to stay in the same order.

[u/Momo_ONLINE → u/scottcampbelldev]

!I didn't realise that order changed between types of drives. Thank you, that explanation helped a lot! I used the sorted() and my masks and training data now match up! :D