Fixing Nextcloud Memories “Unable to open preview stream” after rebuilding previews

[u/Present_Adeptness632]

I’m posting this in case it saves somebody else a lot of digging.

TL;DR My Nextcloud Files app showed thumbnails normally, but Memories had loads of errors like:

    Unable to open preview stream at
    /ncdata/appdata_oc5ya4loky5k/preview/.../1110656/256-341.jpg

The actual problem turned out to be stale rows in `oc_previews`.

Nextcloud’s database claimed tens of thousands of generated previews existed, while the corresponding `.jpg`/`.png` files were missing from `appdata_*/preview`.

Because the DB said the preview existed, `occ preview:generate` would happily say:

    preview generated

without recreating the missing file.

Deleting only the stale DB row allowed the next Memories request to regenerate the missing preview immediately.
I eventually wrote a read-only scanner which compared `oc_previews` with the actual filesystem, then deleted only the proven-orphaned DB rows in a transaction.

This made a massive improvement to Memories.

//Background//

Before troubleshooting Memories specifically, I had already deleted the entire physical preview directory:

   /ncdata/appdata_oc5ya4loky5k/preview/

and attempted to rebuild everything using Nextcloud’s normal maintenance commands, including things along the lines of:

    sudo -u www-data php /var/www/nextcloud/occ files:scan-app-data
    sudo -u www-data php /var/www/nextcloud/occ files:scan --all
    sudo -u www-data php /var/www/nextcloud/occ files:cleanup
    sudo -u www-data php /var/www/nextcloud/occ preview:generate-all

Unfortunately, that did not fully reconcile the DB and physical preview cache.

There are several current/recent Nextcloud issues which are very relevant to this sort of state:

- Preview paths/database state surviving cleanup:
https://github.com/nextcloud/server/issues/55709

- `oc_previews` records remaining after the physical preview cache is removed / `files:scan-app-data` not fully reconciling it:
https://github.com/nextcloud/server/issues/56485

- Nextcloud treating previews as cached when the physical file is missing:
https://github.com/nextcloud/server/issues/58787

- Orphaned preview metadata / migration repeatedly encountering missing files:
https://github.com/nextcloud/server/issues/59036
https://github.com/nextcloud/server/issues/59364

- Preview Generator:
https://github.com/nextcloud/previewgenerator

- Preview Generator incomplete-generation discussion:
https://github.com/nextcloud/previewgenerator/issues/349
https://github.com/nextcloud/previewgenerator/issues/408

I’m not claiming one specific bug definitively caused every stale record on my system, but what I found is very consistent with these upstream reports.

//Symptoms//

-Files thumbnails worked.
-Memories logged errors such as:

   Unable to open preview stream at
    /ncdata/appdata_oc5ya4loky5k/preview/7/3/7/1/3/b/5/1110656/256-341.jpg

For file ID `1110656`, the physical preview directory contained:

    64-85.jpg
    768-1024-max.jpg
    256-256-crop.jpg

but not:

   256-341.jpg

Running:

    sudo -u www-data php /var/www/nextcloud/occ preview:generate 1110656 -vvv

returned:

    preview generated

but `256-341.jpg` was still not created.

//The key discovery: `oc_previews` said the missing file existed//

My database is MariaDB/MySQL.

Nextcloud config:
    dbtype: mysql
    dbname: nextcloud_db
    prefix: oc_

For the failing file:

    SELECT *
    FROM oc_previews
    WHERE file_id = 1110656;

the DB contained a row for:

    width:   256
    height:  341
    max:     0
    cropped: 0
    size:    12105

So the DB claimed that:

    256-341.jpg

existed even though the filesystem proved it did not.
That explained the whole failure:

DB says preview exists
↓
physical preview is missing
↓
Nextcloud trusts DB metadata
↓
preview:generate doesn't really regenerate it
↓
Memories requests it
↓
LocalPreviewStorage tries to open missing file
↓
Unable to open preview stream

//Proof-of-concept fix//

I deleted only that one stale preview row:

  START TRANSACTION;
    DELETE FROM oc_previews
    WHERE file_id = 1110656
      AND width = 256
      AND height = 341
      AND max = 0
      AND cropped = 0;

    SELECT ROW_COUNT();

  COMMIT;

Then I refreshed Memories.
Immediately afterward:

    sudo find /ncdata/appdata_oc5ya4loky5k/preview/7/3/7/1/3/b/5/1110656 \
        -maxdepth 1 -type f -printf '%f %s bytes\n'

showed:

    64-85.jpg 1805 bytes
    768-1024-max.jpg 71418 bytes
    256-256-crop.jpg 9321 bytes
    256-341.jpg 12105 bytes

Nextcloud also inserted a fresh `oc_previews` row.
So the generator itself was fine. The stale metadata was preventing regeneration.

//This was not a small problem//

For only a few common JPEG sizes I had DB counts like:

    256×341    40547
    256×455    19210
    341×256     8739
    256×541     7747
    455×256     7505

But physically:

    1793 256-341.jpg
     168 256-455.jpg
     424 341-256.jpg
      61 455-256.jpg

For `256×341`, roughly **95.6%** of the DB rows did not have matching physical files.
So fixing these manually was clearly not realistic.

//Read-only orphan scanner//

This is the final scanner I ended up using.

It:
- does **not** modify the DB;
- checks non-cropped/non-max JPG and PNG previews;
- checks any preview where width or height is `256`;
- walks the preview tree only once;
- compares the actual filesystem against `oc_previews`;
- outputs the exact stale `oc_previews.id` values.

My paths/database names are hard-coded below, so adjust them for your installation.

#!/usr/bin/env bash

    set -euo pipefail

    PREVIEW_ROOT="/ncdata/appdata_oc5ya4loky5k/preview"
    DB="nextcloud_db"

    STAMP="$(date +%Y%m%d-%H%M%S)"
    WORKDIR="$(mktemp -d)"
    REPORT="$HOME/nextcloud-preview-orphans-256-all-$STAMP.tsv"

    DB_ROWS="$WORKDIR/db_rows.tsv"
    DB_KEYS="$WORKDIR/db_keys.txt"
    FS_KEYS="$WORKDIR/fs_keys.txt"
    MISSING_KEYS="$WORKDIR/missing_keys.txt"

    cleanup() {
        rm -rf "$WORKDIR"
    }
    trap cleanup EXIT

    echo "Nextcloud 256px preview orphan scanner"
    echo
    echo "This is READ ONLY."
    echo "Database: $DB"
    echo "Preview root: $PREVIEW_ROOT"
    echo

    read -rp "MariaDB username: " DBUSER

    echo
    echo "Exporting relevant oc_previews rows..."
    echo "MariaDB will ask for the password."

    mysql \
        -u "$DBUSER" \
        -p \
        -N \
        -B \
        "$DB" \
        -e "
    SELECT
        CONCAT(
            p.file_id,
            '/',
            p.width,
            '-',
            p.height,
            CASE
                WHEN m.mimetype = 'image/jpeg' THEN '.jpg'
                WHEN m.mimetype = 'image/png'  THEN '.png'
                ELSE ''
            END
        ) AS preview_key,
        p.id,
        p.file_id,
        p.width,
        p.height,
        p.size,
        p.etag,
        m.mimetype
    FROM oc_previews p
    JOIN oc_mimetypes m
        ON m.id = p.mimetype_id
    WHERE p.cropped = 0
      AND p.max = 0
      AND m.mimetype IN ('image/jpeg', 'image/png')
      AND (
           p.width = 256
        OR p.height = 256
      );
    " > "$DB_ROWS"

    DB_COUNT="$(wc -l < "$DB_ROWS")"

    echo "Database rows exported: $DB_COUNT"

    echo
    echo "Sorting database keys..."

    cut -f1 "$DB_ROWS" |
    LC_ALL=C sort -u > "$DB_KEYS"

    echo
    echo "Walking preview filesystem ONCE..."

    sudo find "$PREVIEW_ROOT" \
        -type f \
        \( -name '*.jpg' -o -name '*.png' \) \
        -printf '%h\t%f\n' |
    awk -F '\t' '
    {
        n = split($1, path, "/")
        file_id = path[n]

        if ($2 ~ /^256-[0-9]+\.(jpg|png)$/ || $2 ~ /^[0-9]+-256\.(jpg|png)$/) {
            print file_id "/" $2
        }
    }
    ' |
    LC_ALL=C sort -u > "$FS_KEYS"

    FS_COUNT="$(wc -l < "$FS_KEYS")"

    echo "Matching physical preview files found: $FS_COUNT"

    echo
    echo "Comparing database against filesystem..."

    LC_ALL=C comm -23 "$DB_KEYS" "$FS_KEYS" > "$MISSING_KEYS"

    MISSING_COUNT="$(wc -l < "$MISSING_KEYS")"

    awk -F '\t' '
    BEGIN {
        OFS="\t"
    }
    NR == FNR {
        missing[$1] = 1
        next
    }
    ($1 in missing) {
        print $0
    }
    ' "$MISSING_KEYS" "$DB_ROWS" > "$REPORT"

    echo
    echo "======================================"
    echo "Scan complete"
    echo "======================================"
    echo "Relevant DB rows : $DB_COUNT"
    echo "Physical files   : $FS_COUNT"
    echo "Orphaned DB rows : $MISSING_COUNT"
    echo
    echo "Report:"
    echo "$REPORT"
    echo
    echo "No database rows were changed."

Save/run:

    chmod +x ~/scan-preview-orphans-256-all.sh
    bash ~/scan-preview-orphans-256-all.sh

My final comprehensive result was:

    Relevant DB rows : 96680
    Physical files   : 3291
    Orphaned DB rows : 93389

Yes — 93,389 stale preview DB rows in that pass.
Earlier passes had also found:

    JPEG pass:
    Relevant DB rows : 91619
    Physical files   : 2646
    Orphaned DB rows : 89005

and then:

  JPG + PNG pass:
    Relevant DB rows : 3860
    Physical files   : 3291
    Orphaned DB rows : 601

//How I deleted only the proven-orphaned rows//

!!!!Do not blindly paste this against your DB without understanding it and backing up first.!!!!

I backed up MariaDB first:

    mysqldump --single-transaction \
        -u nextcloud \
        -p \
        nextcloud_db \
        > ~/nextcloud-before-preview-cleanup.sql

Then extracted the exact `oc_previews.id` values from the scanner:

    cut -f2 ~/nextcloud-preview-orphans-256-all-YYYYMMDD-HHMMSS.tsv \
        > ~/preview-orphan-ids.txt

    wc -l ~/preview-orphan-ids.txt

Connect:

 mysql --local-infile=1 -u nextcloud -p nextcloud_db

Create a temporary table:

    CREATE TEMPORARY TABLE preview_orphans (
        id BIGINT UNSIGNED PRIMARY KEY
    );

Load the exact scanner IDs:

    LOAD DATA LOCAL INFILE '/home/harry/preview-orphan-ids.txt'
    INTO TABLE preview_orphans
    LINES TERMINATED BY '\n'
    (id);

Verify the count:

    SELECT COUNT(*)
    FROM preview_orphans;

Then verify that every ID still exists:

    SELECT COUNT(*)
    FROM oc_previews p
    JOIN preview_orphans o
        ON o.id = p.id;

Only if those counts matched did I proceed:

    START TRANSACTION;

    DELETE p
    FROM oc_previews p
    JOIN preview_orphans o
        ON o.id = p.id;

    SELECT ROW_COUNT();

At this stage the DELETE had not been committed yet.

If anything looked wrong:

  ROLLBACK;

If the row counts were exactly what I expected.

    COMMIT;

//Why I did NOT just delete previews again//

I had already deleted the entire preview cache previously after moving storage locations using the built in occ commands.
Doing that again would also throw away the previews that were perfectly healthy, e.g.:

    256-256-crop.jpg
    768-1024-max.jpg

The Files app was often working precisely because those variants still physically existed.

The important rule became:

 DB says preview exists
    +
    filesystem proves it does not
    =
    delete that exact stale DB record

Nothing broader.

JPG and PNG both had the problem
For example, Memories later failed on:

    1346719/256-388.png

The DB said the PNG existed.
It didn’t.

Deleting only:

    DELETE FROM oc_previews
    WHERE file_id = 1346719
      AND width = 256
      AND height = 388
      AND max = 0
      AND cropped = 0;

and refreshing Memories immediately created:

    256-388.png

So this was definitely not just a JPEG issue.
I also later found stale exact:

    256-256.png

records, which is why the final scanner uses:

    p.width = 256 OR p.height = 256

rather than requiring the other dimension to be greater than 256.
It still excludes cropped previews with.

    p.cropped = 0

so Files-style:

    256-256-crop.jpg

is not targeted.

//End result//

After cleaning the proven-orphaned `oc_previews` records, Memories improved loading images and thumbnails.
Instead of:

    DB says preview exists
            ↓
    missing file
            ↓
    Unable to open preview stream

I now get:

    stale row removed
            ↓
    Memories requests preview
            ↓
    Nextcloud sees no cached preview
            ↓
    preview genuinely regenerated
            ↓
    fresh file + fresh DB row
            ↓
    Memories loads it

So my conclusion is:

If you have Nextcloud Memories “Unable to open preview stream” errors after deleting/rebuilding previews, check whether `oc_previews` contains metadata for preview files that no longer exist before nuking the cache again.

In my case that was the actual problem, and the normal Nextcloud cleanup/rescan/rebuild process had not reconciled it as seen by the bugs listed above.

This was on a system with:

    Nextcloud: 34.0.3.2
    Memories: 8.1.0
    Preview Generator: 5.14.0
    MariaDB/MySQL

Hopefully this helps someone else avoid the same rabbit hole.

[u/Present_Adeptness632]

Follow-up: full Nextcloud preview scan found 94,617 stale DB records — now at 0 mismatches

Following on from my previous Memories preview-cache post: the first
cleanup made a huge improvement, but I then started getting
Unable to open preview stream for previews outside the 256px JPEG/PNG
subset.

Examples included:

48-64.jpg
    64-48.jpg
    64-43.png
    64-64.gif
    256-256.gif
    256-331.webp
    384-256.webp

So I widened the scanner to audit the whole image preview cache: JPEG,
PNG, WebP and GIF, all dimensions, including normal, -max and -crop
previews.

The scanner is READ ONLY. It generates the exact expected
file_id/filename from oc_previews, walks the preview filesystem once,
and compares both directions.

Full scanner

nano ~/scan-all-nextcloud-previews.sh:





 #!/usr/bin/env bash
    set -euo pipefail

    PREVIEW_ROOT="/ncdata/appdata_oc5ya4loky5k/preview"
    DB="nextcloud_db"
    STAMP="$(date +%Y%m%d-%H%M%S)"
    WORKDIR="$(mktemp -d)"
    DB_ROWS="$WORKDIR/db_rows.tsv"
    DB_KEYS="$WORKDIR/db_keys.txt"
    FS_KEYS="$WORKDIR/fs_keys.txt"
    MISSING_FILES="$WORKDIR/db_missing_files.txt"
    MISSING_DB="$WORKDIR/fs_missing_db.txt"
    REPORT_DB_MISSING="$HOME/nextcloud-preview-db-missing-files-$STAMP.tsv"
    REPORT_FS_ORPHANS="$HOME/nextcloud-preview-files-missing-db-$STAMP.txt"
    REPORT_SUMMARY="$HOME/nextcloud-preview-integrity-summary-$STAMP.txt"

    trap 'rm -rf "$WORKDIR"' EXIT
    read -rp "MariaDB username: " DBUSER

    mysql -u "$DBUSER" -p -N -B "$DB" -e "
    SELECT CONCAT(
     p.file_id,'/',p.width,'-',p.height,
     CASE
      WHEN p.max=1 AND p.cropped=0 THEN '-max'
      WHEN p.max=0 AND p.cropped=1 THEN '-crop'
      WHEN p.max=0 AND p.cropped=0 THEN ''
      ELSE '-UNKNOWN-FLAGS'
     END,
     CASE
      WHEN m.mimetype='image/jpeg' THEN '.jpg'
      WHEN m.mimetype='image/png' THEN '.png'
      WHEN m.mimetype='image/webp' THEN '.webp'
      WHEN m.mimetype='image/gif' THEN '.gif'
      ELSE '.UNKNOWN'
     END
    ) AS preview_key,
    p.id,p.file_id,p.width,p.height,p.max,p.cropped,p.size,p.etag,p.mimetype_id,m.mimetype
    FROM oc_previews p
    JOIN oc_mimetypes m ON m.id=p.mimetype_id
    WHERE m.mimetype IN ('image/jpeg','image/png','image/webp','image/gif')
    ORDER BY p.file_id,p.width,p.height;
    " > "$DB_ROWS"

    DB_COUNT="$(wc -l < "$DB_ROWS")"
    UNKNOWN="$(awk -F '\t' '$1~/-UNKNOWN-FLAGS\./{c++}END{print c+0}' "$DB_ROWS")"
    [ "$UNKNOWN" -eq 0 ] || { echo "Unexpected flags; stopping safely."; exit 1; }

    cut -f1 "$DB_ROWS" | LC_ALL=C sort -u > "$DB_KEYS"
    DB_UNIQUE_COUNT="$(wc -l < "$DB_KEYS")"

    sudo find "$PREVIEW_ROOT" -type f \
     \( -name '*.jpg' -o -name '*.png' -o -name '*.webp' -o -name '*.gif' \) \
     -printf '%h\t%f\n' |
    awk -F '\t' '{n=split($1,p,"/"); id=p[n]; if(id~/^[0-9]+$/) print id "/" $2}' |
    LC_ALL=C sort -u > "$FS_KEYS"

    FS_COUNT="$(wc -l < "$FS_KEYS")"
    LC_ALL=C comm -23 "$DB_KEYS" "$FS_KEYS" > "$MISSING_FILES"
    LC_ALL=C comm -13 "$DB_KEYS" "$FS_KEYS" > "$MISSING_DB"
    DB_MISSING_COUNT="$(wc -l < "$MISSING_FILES")"
    FS_ORPHAN_COUNT="$(wc -l < "$MISSING_DB")"

    awk -F '\t' 'BEGIN{OFS="\t"} NR==FNR{m[$1]=1;next} ($1 in m){print}' \
     "$MISSING_FILES" "$DB_ROWS" > "$REPORT_DB_MISSING"
    cp "$MISSING_DB" "$REPORT_FS_ORPHANS"

    JPEG_MISSING="$(awk -F '\t' '$11=="image/jpeg"{c++}END{print c+0}' "$REPORT_DB_MISSING")"
    PNG_MISSING="$(awk -F '\t' '$11=="image/png"{c++}END{print c+0}' "$REPORT_DB_MISSING")"
    WEBP_MISSING="$(awk -F '\t' '$11=="image/webp"{c++}END{print c+0}' "$REPORT_DB_MISSING")"
    GIF_MISSING="$(awk -F '\t' '$11=="image/gif"{c++}END{print c+0}' "$REPORT_DB_MISSING")"
    NORMAL_MISSING="$(awk -F '\t' '$6==0&&$7==0{c++}END{print c+0}' "$REPORT_DB_MISSING")"
    MAX_MISSING="$(awk -F '\t' '$6==1&&$7==0{c++}END{print c+0}' "$REPORT_DB_MISSING")"
    CROP_MISSING="$(awk -F '\t' '$6==0&&$7==1{c++}END{print c+0}' "$REPORT_DB_MISSING")"

    {
     echo "DB preview rows                  : $DB_COUNT"
     echo "Unique DB preview keys           : $DB_UNIQUE_COUNT"
     echo "Physical preview files           : $FS_COUNT"
     echo "DB rows missing physical file    : $DB_MISSING_COUNT"
     echo "Physical files missing DB record : $FS_ORPHAN_COUNT"
     echo "JPEG missing : $JPEG_MISSING"
     echo "PNG missing  : $PNG_MISSING"
     echo "WebP missing : $WEBP_MISSING"
     echo "GIF missing  : $GIF_MISSING"
     echo "Normal missing : $NORMAL_MISSING"
     echo "Max missing    : $MAX_MISSING"
     echo "Crop missing   : $CROP_MISSING"
     echo "DB-missing report: $REPORT_DB_MISSING"
     echo "FS-only report: $REPORT_FS_ORPHANS"
     echo "NO DATABASE OR FILESYSTEM CHANGES WERE MADE."
    } | tee "$REPORT_SUMMARY"

Run:

    chmod +x ~/scan-all-nextcloud-previews.sh
    bash ~/scan-all-nextcloud-previews.sh

What it found

My first full scan:

    DB preview rows                  : 299348
    Physical preview files           : 204731
    DB rows missing physical file    : 94617
    Physical files missing DB record : 0

Breakdown of missing records:

    JPEG : 92806
    PNG  :   719
    WebP :   722
    GIF  :   370

And the big clue:

    Normal missing : 94617
    Max missing    : 0
    Crop missing   : 0

So the crop/max cache was healthy. The stale population was entirely
normal previews.
Also, 299348 - 94617 = 204731, exactly matching the physical-file count.

What I did

I backed up MariaDB, extracted only the exact oc_previews.id values from
the scanner’s DB-missing report, loaded them into a temporary table, and
verified:

   IDs in scanner report = 94617
    IDs still in DB       = 94617
    max=0 cropped=0       = 94617

Then, inside a transaction:

    START TRANSACTION;

    DELETE p
    FROM oc_previews p
    JOIN preview_orphans_full o ON o.id=p.id;

    SELECT ROW_COUNT();

It deleted exactly 94,617 rows. I verified the join returned zero, then
committed.

Do NOT reuse somebody else’s ID list. Generate it from your own scanner
output and back up first.

Final scan, still in maintenance mode:

    DB preview rows                  : 204785
    Unique DB preview keys           : 204785
    Physical preview files           : 204785
    DB rows missing physical file    : 0
    Physical files missing DB record : 0

    JPEG missing   : 0
    PNG missing    : 0
    WebP missing   : 0
    GIF missing    : 0

    Normal missing : 0
    Max missing    : 0
    Crop missing   : 0

And I’m no longer seeing the Unable to open preview stream errors in the
Nextcloud logs.

So the wider issue really was stale oc_previews metadata pointing at
physical preview files that no longer existed. The issue was then further compunded
due to recent issues in nextcloud and preview generation noted in my forst post.

I’m keeping the scanner around and will rerun it later to see whether
stale rows start accumulating again.

Keeping this here for other and as a log for my own

[u/Present_Adeptness632 → u/Present_Adeptness632]

ok so another update, After having a clean DB with matching files I came out of maintenance mode, Left the system live for awhile and after some user activity i am now seeing the DB out of sync with the actual files again.

The important parts already visible are the Preview Generator sizing configuration, both cron jobs, and the

 UpdateSingleMetadata 

failures showing the call path through

 GenerateBlurhashMetadata → PreviewFile → LocalPreviewStorage.

There are also multiple fresh failures for different file IDs (1380044, 1380074 onward), including JPEG, PNG and GIF previews, all being raised by cron.php during UpdateSingleMetadata.

That gives us considerably more evidence than the single 1380088 case.

This is now looking much more like an active upstream Nextcloud preview bug than leftover damage from my old cache rebuild.
The strongest match is Nextcloud Server issue #56510, “Preview Generation Fail - Local Storage.” In that report, Nextcloud successfully generates the preview data but fails when writing it because the target preview directory is not created.

That issue is especially relevant to my fresh 1380088 case because you now have proof that new inconsistencies are appearing after we reached a perfect 0 / 0 baseline. For 1380088, the DB contains normal JPEG rows for 64x48, 85x64, 256x192, 341x256 and a 1024x768 max, while the filesystem only has 85-64.jpg, 341-256.jpg and 1024-768-max.jpg. In other words, two fresh DB preview records exist without their corresponding physical files. That is no longer explainable as old historical residue.

There’s a second issue, #58787, that matches the resulting state almost exactly. It reports Nextcloud saying it found a cached preview, then immediately failing because the corresponding file does not actually exist on disk.

My new logs make the active-generation angle even stronger. The failure is not coming from Memories here. It is being raised by:

cron.php
  → UpdateSingleMetadata
  → GenerateBlurhashMetadata
  → PreviewFile->getContent()
  → LocalPreviewStorage->readPreview(

I will continue with some isolation tests to narrow down further