Research computing software development occupies a distinct niche: the tools, pipelines, and analysis systems that scientists, engineers, and research institutions build to do their work. These systems have different requirements than standard business software — they handle large, irregular data; require reproducibility; and are often built by people whose primary expertise is their domain, not software engineering.
This post covers where research computing software typically breaks down and what productionizing it looks like.
Where Research Computing Software Fails
Reproducibility. A pipeline that produced a result on a researcher's workstation 18 months ago can't be reproduced because the Python environment isn't specified, data preprocessing had undocumented manual steps, and the intermediate outputs were overwritten. Reproducibility is a software engineering problem, not a scientific one.
Scale. Workflows developed on sample data fail or run for days on full datasets. Serial loops that work on 1,000 records don't work on 10 million. Without parallelization and distributed compute, research pipelines don't scale to production data volumes.
Operational fragility. Cron jobs that run analysis scripts, scripts that assume specific directory structures, pipelines that fail silently without alerting anyone. When the job fails at 3 AM on the day before a grant deadline, nobody knows.
No API or interface. The researcher runs the script. Other researchers, systems, or stakeholders can't access the output without also running the script. Useful research results need to be accessible.
Data management. Input data downloaded manually, intermediate results scattered across directories, outputs stored without version control. When the analysis needs to be re-run with corrected input data, nobody knows which output corresponds to which input.
What We Build for Research Computing Teams
Containerized Pipeline Infrastructure
The foundation of reproducible research computing: containerize the pipeline with all dependencies, and store inputs and outputs in versioned, addressable storage.
FROM python:3.11-slim
# Pin all system dependencies
RUN apt-get update && apt-get install -y \
libgdal-dev \
libhdf5-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Pin Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
# Pipeline runs with explicit input/output paths
ENTRYPOINT ["python", "-m", "src.pipeline"]
CMD ["--help"]
Input data paths and output paths are explicit parameters, never hardcoded. Every pipeline run produces a manifest: what input data, what code version, what parameters, what output.
Parallel Processing with AWS Batch
For computationally intensive pipelines that need to scale beyond a single machine, AWS Batch provides managed compute with job queuing:
import boto3
batch = boto3.client("batch")
def submit_analysis_jobs(input_files: list[str], job_definition: str):
job_ids = []
for file_key in input_files:
response = batch.submit_job(
jobName=f"analysis-{file_key.replace('/', '-')}",
jobQueue="research-compute-queue",
jobDefinition=job_definition,
containerOverrides={
"command": [
"--input", f"s3://research-data/{file_key}",
"--output", f"s3://research-outputs/{file_key}",
"--params", "s3://research-config/params.json"
]
},
retryStrategy={"attempts": 2}
)
job_ids.append(response["jobId"])
return job_ids
Batch handles scheduling, retry logic, and cluster scaling. The research team submits jobs and retrieves results — they don't manage EC2 instances.
For workflows with complex dependencies (process A before B, aggregate C and D into E), Step Functions orchestrates the job DAG.
Data Versioning and Lineage
Using DVC (Data Version Control) for research data management:
# dvc.yaml — defines the pipeline stages
stages:
preprocess:
cmd: python src/preprocess.py -i data/raw -o data/processed
deps:
- src/preprocess.py
- data/raw
outs:
- data/processed
analyze:
cmd: python src/analyze.py -i data/processed -o results/
deps:
- src/analyze.py
- data/processed
outs:
- results/
metrics:
- results/metrics.json
With DVC: every pipeline run is reproducible from a commit. Datasets are versioned in S3 without bloating the git repository. Results are traceable to their inputs.
Results API
Research outputs that other systems need to consume — whether an internal dashboard, a collaborator's system, or a stakeholder portal — need an API.
from fastapi import FastAPI, Query
import boto3
import pandas as pd
app = FastAPI()
s3 = boto3.client("s3")
@app.get("/results/{analysis_id}")
async def get_results(
analysis_id: str,
format: str = Query("json", enum=["json", "csv"])
):
# Retrieve results from S3
key = f"results/{analysis_id}/output.parquet"
try:
obj = s3.get_object(Bucket=RESULTS_BUCKET, Key=key)
df = pd.read_parquet(obj["Body"])
if format == "csv":
return Response(df.to_csv(index=False), media_type="text/csv")
return df.to_dict(orient="records")
except s3.exceptions.NoSuchKey:
raise HTTPException(status_code=404, detail="Analysis not found")
Operational Monitoring
Research pipelines need the same operational monitoring as any other production system:
- CloudWatch Alarms on Batch job failure rate
- SNS notification to the team when a job fails with context (job ID, log link)
- Dashboard showing pipeline throughput and success rate
- Daily automated check that expected pipeline runs completed
# SNS alert on job failure
def handle_batch_state_change(event):
if event["detail"]["status"] == "FAILED":
job_id = event["detail"]["jobId"]
job_name = event["detail"]["jobName"]
log_url = f"https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups/..."
sns.publish(
TopicArn=ALERT_TOPIC,
Subject=f"Batch job failed: {job_name}",
Message=f"Job {job_id} failed. Logs: {log_url}"
)
For the infrastructure patterns behind these deployments, see our data analytics capability and our work on real-time data dashboard development.
Frequently Asked Questions
Can you work with domain-specific research software (MATLAB, R, specialized simulation tools)?
Yes, with caveats. MATLAB: we can containerize MATLAB runtime environments for deployment, though licensing has implications. R: full support, commonly containerized and deployed on AWS Lambda or ECS. Specialized simulation tools: depends on licensing model and whether they support headless execution. We assess case by case.
How do you handle sensitive research data (PHI, CUI, proprietary datasets)?
Data classification determines the infrastructure requirements. PHI requires HIPAA-eligible services and BAA with AWS. CUI (Controlled Unclassified Information) for federally funded research has specific handling requirements under NIST 800-171. We scope the appropriate controls to the data classification.
Our pipeline takes 72 hours to run. Can it be parallelized?
Almost always yes, partially if not fully. We start by profiling where time is spent — usually a small fraction of the pipeline takes most of the time. That fraction is the parallelization target. Using AWS Batch with 50 parallel jobs on a 72-hour serial pipeline can reduce wall-clock time dramatically, often to under 4 hours for highly parallelizable workloads.
What documentation do we get with a delivered pipeline?
Architecture documentation (what runs where, what inputs/outputs), operational runbooks (how to trigger runs, how to investigate failures, how to add new data inputs), and developer documentation (how to modify the pipeline code, how to run locally for testing). We write documentation as part of delivery, not as an afterthought.
How do you handle the transition when the research team's domain expert built the original pipeline?
We work collaboratively with the domain expert during the rebuild — they own the scientific correctness, we own the engineering. We don't need to understand the science deeply; we need to understand the inputs, outputs, and transformations well enough to implement them reliably. The domain expert validates results throughout.
Discuss your automation project → rutagon.com/contact | 907-841-8407 | contact@rutagon.com