Inkwell Data / Pipelines
Pipelines
Several files in, one or more files out. Add your files, then build the steps: join a customer list onto orders, stack this month under last month, clean, sort and total. Use the simple step list or switch to a drag-and-drop diagram. Save the pipeline and re-run it next month in one click.
- Add your files
- Add steps
- Run and download
How a pipeline works
Several files go in, your steps run top to bottom, and one or more files come out. Save it and next month it runs again in one click.
How it works
- Add your files. Up to five. The first is where the pipeline starts; give each one a short name like Orders or Customers.
- Add steps. Every Transform step is here, plus Join with another file (match rows on a column and bring columns over), Stack another file underneath (add its rows, matching columns by name) and Save a copy here (an extra file of the data at that point). The preview updates as you go.
- Run the pipeline. Every step runs on every row, and you download the final file and any copies.
List or diagram. The list is the default and does everything. Prefer to see it? Click Diagram: the steps become boxes, other files plug in from the left, and copies branch off to the right. Drag a step's ⠿ handle to move it, drag a file's ● onto the steps to join or stack it there, or click + to add a step at that point.
Big files. A big CSV shows its first 5,000 rows while you build. Running the pipeline works through the whole file a piece at a time, so a file of millions of rows won't freeze the page.
Monthly reports. Save the pipeline, and next month choose it and add the new files. A Transform recipe can be the starting point too: pick it under Or start from a recipe.
Questions
Are my files uploaded anywhere?
No. Every file is read and processed by your own browser. Big files are worked through a piece at a time inside this tab, and the built-in DuckDB engine that sorts and de-duplicates them runs here too. Nothing is sent to a server.
How big can the files be?
CSV and TSV files of hundreds of megabytes or millions of rows work: while you build, you see the first 5,000 rows, and running the pipeline processes every row in the background. On a typical laptop 1.5 million rows take well under a minute. Excel, JSON and Parquet files are read whole, so keep those under about 80 MB.
What's the difference between the list and the diagram?
Nothing but the view. The list is the simple way: steps run top to bottom. The diagram draws the same pipeline as boxes and lines, which you can drag around. Switch any time; the pipeline doesn't change.
Can I re-run it next month?
Yes. Click Save pipeline. Next month, choose it from Saved pipelines and add the new files: each file box asks for its file by name, and every step runs again. Pipelines remember file names, never their contents.
Can I share a pipeline?
Yes. Copy share link puts the steps and file names in a link; Export pipeline file saves them as a small file. Neither contains your data.
Can I get more than one file out?
Yes. Add a Save a copy here step anywhere in the list, and you get a file of the data at that point as well as the final result, for example the cleaned rows and a summary.