kohiiIntroduction
I am currently developing a CSV editor called SmoothCSV 3. It uses the framework Tauri, and for the renderer, I employ web technologies (React + TypeScript).
CSV editors need to display a large number of rows and cells. In web technologies, virtual scrolling is the standard approach for such scenarios. SmoothCSV also utilized virtual scrolling, but I encountered a problem where extremely large datasets, such as those with millions of rows, could not be fully displayed due to the limitations of virtual scrolling.
In this article, I will introduce the basics of virtual scrolling and explain how I overcame its limitations.
Why Do We Need Virtual Scrolling?
Ordinary Scrolling
When displaying lists or tables with a very large number of items in a browser, a naive implementation will suffer performance degradation as the number of items increases, eventually causing the browser to freeze. This is because the browser constructs and renders the DOM tree for all items, including those outside the viewport.
Virtual Scrolling as a Solution
To solve this problem, virtual scrolling is often used. Virtual scrolling is a technique where only the items visible based on the scroll position are dynamically generated using JavaScript, thereby reducing the load on the browser.
The visible area is called the viewport, and the content outside the viewport is called the buffer. To enable proper scrolling in the browser, dummy elements (spacers) are placed in the buffer areas to maintain the height of the content.
(Alternatively, you can specify the height of the content container and position the displayed items using position: absolute.)
How to Implement Virtual Scrolling
There are many libraries available to easily implement virtual scrolling, and usually, you would use these. In React, the following are well-known:
I personally prefer using TanStack Virtual, which also supports Vue, Svelte, Solid, Lit, and Angular besides React.
The Limits of Virtual Scrolling
It's not widely known, but there is an upper limit to the width and height of elements that can be displayed in a browser. The values I measured are as follows:
- Safari: 33,554,428px
- Chromium: 16,777,216px
Elements with height or width larger than this will be truncated to the maximum value. (Properties like top, left, margin, and padding also have such upper limits.)
<!-- Example in Safari -->
<!-- Truncated to 33,554,428px -->
<div style="height: 9999999999999px;"></div>
<!-- Same even without directly specifying the size -->
<!-- Should total 40,000,000px, but truncated to 33,554,428px -->
<div>
<div style="height: 20000000px"></div>
<div style="height: 20000000px"></div>
</div>
Virtual scrolling is also affected by this limitation. If the height of the scrollable content (i.e., the total height of all items) exceeds this upper limit, you cannot scroll to the end even with virtual scrolling.
In my CSV editor, since the height per row is 22px, the limit is reached at around 1,525,200 rows on macOS (Safari), resulting in the inability to display rows beyond that point.
While this number of rows might not be a common consideration, CSV files can have a very large number of rows. Accepting this limitation would diminish the value of a CSV editor.
Strategies to Overcome the Limits
Virtual scrolling relies on the browser for the scrolling mechanism itself, so the height of the scrollable content, including the buffer, must be the same as the actual total height of all rows. In other words, as long as we depend on the browser's scrolling functionality, we are bound by this limitation.
So, let's create our own scrolling mechanism!
(it might even be better to render using canvas.)
Implementing Beyond the Limits
Let's try implementing this in React.
1. Creating a Scrollbar
First, we'll create a scrollbar by combining div elements. Generally, the rail of the scrollbar is called the Track, and the thumb is called the Thumb.
The ScrollBar component receives properties such as the viewport height (viewportSize), the height of the content to be scrolled (contentSize), and the current scroll position (scrollPosition), and renders a custom scrollbar. It notifies the result of user interactions via a callback (onScroll).
The implementation is roughly as follows. It is a stateless, controlled component.
export type ScrollBarProps = {
viewportSize: number; // Size of the viewport
contentSize: number; // Total size of the content
scrollPosition: number; // Scroll position
onScroll?: (scrollPosition: number) => void; // Callback when scroll position changes
};
export function ScrollBar({
contentSize,
viewportSize,
scrollPosition,
onScroll,
}: ScrollBarProps) {
// Calculate the size and position of the scrollbar from contentSize, viewportSize, and scrollPosition
const scrollRatio = viewportSize / contentSize;
const thumbSize = Math.max(
16, // Minimum size of the thumb (to prevent it from becoming too small)
scrollRatio * viewportSize
);
const maxScrollPosition = contentSize - viewportSize;
const thumbPosition =
(scrollPosition / maxScrollPosition) * (viewportSize - thumbSize);
// Do not display the scrollbar if the content fits within the viewport
const scrollBarVisible = contentSize > viewportSize;
// Convert the thumb position back to the actual scroll position
const translateToScrollPosition = (thumbPosition: number) => {
const newPosition =
(thumbPosition / (viewportSize - thumbSize)) * maxScrollPosition;
return Math.min(maxScrollPosition, Math.max(0, newPosition));
};
// Handler for when the thumb is grabbed and dragged
const handleMouseDownOnThumb = (event: React.MouseEvent) => {
if (!scrollBarVisible) return;
event.preventDefault();
event.stopPropagation();
const startMousePosition = event.clientY;
const startThumbPosition = thumbPosition;
// Handler for mouse movement while holding down the button
// (Register the event on the document to capture movement anywhere)
const handleMouseMove = (event: MouseEvent) => {
const delta = event.clientY - startMousePosition;
onScroll?.(translateToScrollPosition(startThumbPosition + delta));
};
const handleMouseUp = () => {
document.removeEventListener('mousemove', handleMouseMove);
document.removeEventListener('mouseup', handleMouseUp);
};
document.addEventListener('mousemove', handleMouseMove);
document.addEventListener('mouseup', handleMouseUp);
};
const handleMouseDownOnTrack = (event: React.MouseEvent) => {
// Handler for when the track outside the thumb is clicked
// Option 1: Move the thumb to the clicked position
// Option 2: Move one page in the clicked direction
};
return (
<div
style={{
position: 'relative',
height: viewportSize,
width: 14,
}}
onMouseDown={handleMouseDownOnTrack}
>
{scrollBarVisible && (
<div
onMouseDown={handleMouseDownOnThumb}
style={{
position: 'absolute',
top: thumbPosition,
height: thumbSize,
width: 8,
borderRadius: 8,
margin: '0 3px',
background: 'rgba(0, 0, 0, 0.5)',
}}
/>
)}
</div>
);
}
Note that you need to handle more aspects to consider accessibility and behavior on mobile devices.
2. Creating a Scroll Pane
Next, we'll create a component that provides state management for the scrollbar and a viewport that displays content synchronized with it.
(This component provides only the scrolling mechanism, and the content to be rendered based on the scroll position is received via children.)
import type React from 'react';
import { useState } from 'react';
import { ScrollBar } from './ScrollBar';
export type ScrollPaneProps = {
contentSize: number;
viewportSize: number;
// Receives a function as a child that returns content to display based on the scroll position
// (Function as a Child (FaCC) pattern)
children: (scrollPosition: number) => React.ReactNode;
};
export function ScrollPane({
children,
contentSize,
viewportSize,
}: ScrollPaneProps) {
const [scrollPosition, setScrollPosition] = useState(0);
const handleWheel = (event: React.WheelEvent) => {
// Handler for scrolling with mouse wheel
setScrollPosition((prev) => {
const newScrollPosition = prev + event.deltaY;
return Math.max(
0,
Math.min(contentSize - viewportSize, newScrollPosition)
);
});
};
return (
<div style={{ display: 'flex', height: viewportSize }} onWheel={handleWheel}>
{/* Viewport */}
<div style={{ flex: 1, position: 'relative', overflow: 'hidden' }}>
{children(scrollPosition)}
</div>
<ScrollBar
contentSize={contentSize}
viewportSize={viewportSize}
scrollPosition={scrollPosition}
onScroll={setScrollPosition}
/>
</div>
);
}
Using this component looks like this; you pass the content you want to display in the ScrollPane as a function.
// Here, we're simply displaying the scroll position
export default function App() {
return (
<ScrollPane
contentSize={1000}
viewportSize={300}
style={{ border: '1px solid #ddd' }}
>
{(scrollPosition) => <div>scrollPosition: {scrollPosition}</div>}
</ScrollPane>
);
}
3. Rendering the Content
Finally, we render the content we want to display based on the scroll position. Here, we'll show an example of displaying a list of items with uniform height.
const ITEM_HEIGHT = 30;
// Generate a list of 3 million items
const items = Array.from({ length: 3000000 }, (_, i) => `Item ${i}`);
// 3 million * 30px = 90,000,000px (> browser limit)
const totalHeight = ITEM_HEIGHT * items.length;
const viewportSize = 300;
export default function App() {
return (
<ScrollPane contentSize={totalHeight} viewportSize={viewportSize}>
{(scrollPosition) => {
// Calculate the items to display within the viewport
const startIndex = Math.floor(scrollPosition / ITEM_HEIGHT);
const endIndex = Math.min(
Math.ceil((scrollPosition + viewportSize) / ITEM_HEIGHT) + 1,
items.length
);
const visibleItems = items.slice(startIndex, endIndex);
// Logical position of the first visible item
const startPosition = startIndex * ITEM_HEIGHT;
return (
<div
style={{
position: 'absolute',
top: startPosition - scrollPosition, // Distance from the top of the viewport
}}
>
{visibleItems.map((item) => (
<div key={item} style={{ height: ITEM_HEIGHT }}>
{item}
</div>
))}
</div>
);
}}
</ScrollPane>
);
}
And with that, we're done! 🎉
We adjust the display position of the items by specifying the distance from the top of the viewport using top. For items towards the end, startPosition becomes very large, but since scrollPosition is also large, the value passed to top remains within a practical range.
Here's a cleaner version of the completed code (using TailwindCSS):
https://github.com/kohii/react-custom-scroll-example
Conclusion
Thank you for reading to the end 🙇♂️
Please give SmoothCSV a try if you're interested.
