VECTORIO
EN
EnglishDeutsch

“Your instrument outgrew the workstation, and the queue outgrew your patience”

Instrument data. Answers in minutes.

Interactive reduction of large instrument files without queueing a cluster job

Start a pilotHow it runs
The problem
  • The raw data no longer fits on a workstation, and the HPC queue takes days.
  • My pipeline is a notebook only I can run, and I cannot reproduce last year's result.
  • Nobody can say which code and which inputs produced this figure.
With PROTOR
  • Interactive reduction of large instrument files without queueing a cluster job
  • Re-runnable, citable analysis: same inputs give the same outputs, with a hash per step
  • Develop at the bench, scale up without a rewrite, no data egress

From today to the first result.

What you do today

a workstation, a notebook and an HPC queue

The raw data no longer fits on a workstation, the HPC queue takes days, and nobody can say which code and inputs produced a figure.

With PROTOR

100 million rows on two machines summed in 14.7 s, 1 billion rows in 32.1 ms on one notebook — and a signed workflow gives the same result from the same inputs.

Your first step

Run one reduction step on a notebook beside the HPC job and compare the results.

one reduction
How it runs

One query over two machines. Step by step.

100 million rows are split across two machines by key and answered by one query.

PROTOR · One query over two machinesrunning
RandomDataProviderGenerators.RandomCluster NodeCluster.NodeCommand Control NodeWorkflow.Streaming.CommaCLI ProcessorETL.CLIProcessor
  1. RandomDataProviderGenerators.Random
  2. Cluster NodeCluster.Node
  3. Command Control NodeWorkflow.Streaming.CommandControlNode
  4. CLI ProcessorETL.CLIProcessor
nodes
src → Generators.Random
fab → Cluster.Node
q → Workflow.Streaming.CommandControlNode
out → ETL.CLIProcessor
100 M rows on two machines · SUM + COUNT in 14.7 s · measured
  1. RandomDataProviderGenerates 100,000,000 seeded rows.
  2. Cluster NodeSplits the rows across master and peer by key hash.
  3. Command Control NodeRuns SELECT SUM(v), COUNT(*) on the cluster.
  4. CLI ProcessorPrints the result.

A real workflow from our end-to-end test suite; the designer view is illustrated.

What you get. From the first pilot.

Interactive reduction of large instrument files without queueing a cluster job

Re-runnable, citable analysis: same inputs give the same outputs, with a hash per step

Develop at the bench, scale up without a rewrite, no data egress

Why your tools fall short.

×

Spark/Dask need a cluster and ops skills most labs lack

×

Snakemake/Nextflow orchestrate jobs but do not make the compute itself fast or deterministic

×

Cloud analytics are blocked by data-protection and egress cost

What drives it
  • Data volumes outgrowing workstations
  • Reproducibility expectations from funders and journals
  • Data-protection limits on moving data
  • Idle expensive instruments waiting on analysis

Your pilot. Three steps.

1

Your data shape

Tell us what data you have, where it lives and what the result must be.

2

A scoped pilot

On your data and your hardware, with the success measure written down first.

3

A clear decision

You see the result against that measure — and keep every workflow and record.

Related roles in Research & geodata.

Climate, weather, geodata & earth observation

Satellite scenes to a signed city map.

Digital pathology & microscopy imaging

Slides classified. Results signed. On-prem.

Automotive & mobility test data

Test-drive logs, filtered at the garage.

Media & streaming analytics

Real-time media, no dropped frames.

Beta · design partners wanted

Bring us your hardest data problem. Let's solve it together.

A short pilot on your data and your hardware. We agree on what success looks like before we start.

Start a pilotSee the numbers