Inkwell Data / Big file tools
Big file tools
Files too big for Excel or a normal browser tool, like 5 million rows or 3 GB? DuckDB reads them straight from your disk in the background, so you can filter, de-duplicate, trim down and total them up without anything freezing or uploading.
How it works
Drop a CSV or TSV file. The engine (about 8 MB, downloaded once) counts the rows and shows the columns and first rows. Then make a new file with only the columns and rows you need, duplicates removed, as a new CSV; or total it up by any column.
To use the other tools on a big file, take a slice of up to 200,000 rows and open it in Clean, Transform, Check and the rest.
Everything runs in a background thread in your browser. Very large files need a computer with enough memory for the result, but the source file is read in pieces, never all at once.
Questions
How big can the file be?
Files of several gigabytes and tens of millions of rows work on a typical laptop, since the file is read in pieces. The new file you download has to fit in memory, so keep only the columns and rows you need.
Why can't I make an Excel file here?
Excel stops at 1,048,576 rows. Make a smaller file first (filter, pick columns or summarize), then open it in Convert for Excel or Parquet.
Is my file uploaded?
No. DuckDB runs inside your browser and reads the file from your disk. Nothing is sent anywhere.