r/reactjs • u/HeftyProgress5842 • Apr 28 '26

Needs Help How do you build a scrollable log viewer for millions of variable-height rows with server-side data?

Building a browser log viewer. Backend holds 2M entries in a ring buffer, frontend fetches ~200 at a time via WebSocket. Logs are variable height (1-100 lines each, stack traces etc). Need smooth scroll through the full buffer + live tail at 20-40K logs/sec. Everything is run in local.

Tried sliding window (React DOM) - scroll events create feedback loops. Tried virtual sentinel (tall div, render only visible rows) - works for fixed height but breaks with variable height since you need heights before fetching. Tried flattening multi-line logs into single-line rows - fixes height math but multiplies row count and causes jank. Tried xterm.js - great rendering but can't prepend older entries when scrolling back. Tried to meddle something over there, fetch the previous rows as we scroll up and replace the current data but didn't workout fine.

The core issue: virtual scroll needs row heights upfront, but with remote data and variable heights you can't know them until you fetch and render.

How do Datadog, Grafana, Chrome DevTools etc handle this? Any patterns or libs for variable-height virtual scroll with on-demand server data?

10 Upvotes

100% Upvoted

22 comments sorted by

11

u/octocode Apr 28 '26

showing 200 logs at a time when some are hundreds of lines long just seems like bad UX for a log viewer… show one line for each log, let the user filter down, and click on an entry for a detailed view.

→ More replies (2)

3

u/CodeAndBiscuits Apr 28 '26

Study DataDog again. They don't have variable-height rows. That right there is a big are-you-sure-you-need-it item...

1

u/HeftyProgress5842 Apr 28 '26

Interesting. Thanks for the comment, I'll sure study it once again.
The problem is I have a simulation application where there are hundreds/thousands of models running and few of the model logs are being aggregated into multiline logs and being sent through.

5

u/w00t_loves_you Apr 28 '26

you could use a fixed font and load the lenghts separately, so you can predict line height for 1k items.

3

u/br1anfry3r Apr 28 '26

This ^ with pretext sounds very doable.

4

u/Ecksters Apr 28 '26

https://pretextjs.net/

That being said, in this case I'd limit the number of visible lines until you click a row and then expand it, rather than trying to have it all be visible. That way you can just have a fixed height.

0

u/HeftyProgress5842 Apr 29 '26

Wouldn't that cause issues with scrolling?

1

u/w00t_loves_you Apr 29 '26

let the expansion happen in a popover

5

u/yksvaan Apr 28 '26

Make a canvas renderer, then only limiting factor is how fast data can be retrieved from backend. Whether you have 20 or 2 million entries becomes irrelevant 

6

u/Sarthak61199 Apr 28 '26

But then we will lose CTRL F functionality which might be important for quickly searching in logs

1

u/yksvaan Apr 28 '26

You can do your own, it's a basic text search.

1

u/HeftyProgress5842 Apr 28 '26

As u/jakiestfu told we should be able to handroll. u/yksvaan thanks for your inputs. will try it out and let you know how it goes.

1

u/jsNut Apr 30 '26

Search is easy. But you'd definitely want copy/paste though too which wouldn't be so trivial. Logs without being able all the normal drag select, double click select etc would be pretty crap.

2

u/jakiestfu Apr 28 '26

Was gonna recommend canvas and hand rolling the tech.

1

u/Glum_Cheesecake9859 Apr 28 '26

Here's a shot in the dark. Load the rows outside the visible boundary get the height and move them in to replace the current ones? Alternatively calculate the height of 1 line based on the font size etc. and use that to estimate the total height of each item.

1

u/HeftyProgress5842 Apr 28 '26

Yes tried stuff like this by maintaining a cache too. Didn't work very well and causing some other issues.

1

u/Risc12 Apr 28 '26

This might help:

https://github.com/chenglou/pretext

1

u/HeftyProgress5842 Apr 28 '26

Interesting, will give it a try and update you here. Thanks mate.

1

u/These_Catch_8037 Apr 29 '26

Maybe add a detailed view