Real-time data dashboard development sounds simpler than it is. "Just pull the data and show it" works at small scale with a database that isn't under load. It breaks when the data volume grows, when multiple users are refreshing simultaneously, and when "real-time" means sub-30-second updates rather than hourly batch jobs.
This post covers the architecture decisions that make or break real-time dashboards — data source design, query patterns, update mechanisms, and the caching layers that keep dashboards fast under real traffic.
What "Real-Time" Actually Means
Before designing anything, define what real-time means for the use case:
- Sub-second: Streaming architecture required. Think trading platforms, live monitoring, process control.
- 1–30 seconds: WebSocket push or aggressive polling. Operational dashboards, live order tracking.
- 1–5 minutes: Standard polling interval. Sufficient for most business intelligence dashboards, KPI tracking.
- Hourly or daily: Scheduled batch refresh. Traditional BI tools (Tableau, QuickSight, Looker) handle this well.
Architecture scales with freshness requirement. Don't build a streaming pipeline for a dashboard that refreshes every hour. Don't build a nightly ETL for a dashboard that needs to show current inventory.
The Architecture Stack for Real-Time Dashboards
Data Layer: Choose the Right Store for the Query Pattern
Time-series data (metrics, sensors, events over time): - AWS Timestream: managed, purpose-built, automatic data tiering, SQL interface. Right choice for IoT, application metrics, financial time series. - InfluxDB: self-managed option with InfluxQL/Flux query language. More control, more operational overhead. - DynamoDB with time-sorted keys: viable for moderate volume, familiar if you're already AWS-native.
Aggregated operational data (KPIs, rollups, status): - DynamoDB: fast key-value lookups, sub-millisecond for pre-aggregated values. - ElastiCache (Redis): in-memory cache for hot aggregates. Works for values computed on write and cached for reads. - RDS/Aurora: for dashboards backed by relational data with complex joins. Requires careful query optimization.
Large-scale analytics (ad hoc queries, historical analysis): - Athena over S3 Parquet: serverless, cheap for intermittent queries, scales to petabytes. - Redshift: when query volume is high enough to justify a dedicated cluster.
The mistake: using a transactional database as the dashboard query target. OLTP databases optimize for row-level reads and writes. Dashboard queries aggregate across millions of rows. Separate your transactional store from your analytics store.
Update Mechanism: Polling vs. WebSocket vs. Server-Sent Events
Polling (setInterval): Simple. Client requests new data every N seconds. Right for dashboards where freshness > 30 seconds and user count is low.
// Simple polling implementation
function useDashboardData(refreshInterval: number = 30000) {
const [data, setData] = useState(null);
useEffect(() => {
const fetchData = async () => {
const result = await fetch('/api/dashboard-metrics');
setData(await result.json());
};
fetchData(); // Initial load
const interval = setInterval(fetchData, refreshInterval);
return () => clearInterval(interval);
}, [refreshInterval]);
return data;
}
Polling at scale: if 500 users all poll every 30 seconds, that's 16 requests/second to your backend. Fine for most databases; painful if each query is expensive.
Server-Sent Events (SSE): Server pushes updates to client over a persistent HTTP connection. Simpler than WebSockets (unidirectional), works through proxies and load balancers without special configuration. Right for dashboards where you want push updates without the complexity of WebSockets.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import asyncio
import json
app = FastAPI()
@app.get("/events/dashboard-updates")
async def dashboard_stream(site_id: str):
async def event_generator():
while True:
metrics = await get_current_metrics(site_id)
yield f"data: {json.dumps(metrics)}\n\n"
await asyncio.sleep(10) # Push every 10 seconds
return StreamingResponse(
event_generator(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive"
}
)
WebSockets: Bidirectional, lowest latency. Right for sub-10-second freshness requirements where you need to send client state to the server (filters, subscriptions). AWS API Gateway WebSocket APIs manage connection state.
Caching Strategy
Dashboard queries that aggregate millions of rows can't run on every user's page load. The caching strategy:
Write-time aggregation: Compute aggregates when data arrives, not when the dashboard loads. An EventBridge rule triggers a Lambda every minute that recomputes KPIs and writes to a DynamoDB cache table. Dashboard reads from DynamoDB (sub-millisecond) rather than running aggregate queries.
def recompute_dashboard_cache(site_id: str):
"""Runs every minute via EventBridge. Writes results to DynamoDB cache."""
# Expensive aggregate query against Timestream
metrics = timestream_client.query(Query=f"""
SELECT
avg(measure_value::double) as avg_pressure,
max(measure_value::double) as max_pressure,
min(measure_value::double) as min_pressure
FROM "field-operations"."sensor-readings"
WHERE site_id = '{site_id}'
AND time > ago(1h)
AND measure_name = 'pressure'
""")
# Write aggregated result to DynamoDB cache
dynamodb_table.put_item(Item={
"pk": f"DASHBOARD#{site_id}",
"sk": "METRICS#CURRENT",
"metrics": parse_timestream_results(metrics),
"computed_at": datetime.utcnow().isoformat(),
"ttl": int(time.time()) + 120 # 2-minute TTL
})
CloudFront caching: For dashboard API endpoints returning non-user-specific data, cache at the CDN layer with short TTLs (30s–5min). This handles traffic spikes without the backend seeing every request.
Query result caching: Expensive queries cached in ElastiCache with TTL matching the refresh interval. Cache invalidated on write events.
Frontend Architecture for Dashboard Performance
// React Query for dashboard data with smart caching
function OperationalDashboard({ siteId }: Props) {
// Stale-while-revalidate: show cached data instantly, refresh in background
const { data: metrics, isLoading } = useQuery({
queryKey: ['dashboard-metrics', siteId],
queryFn: () => fetchDashboardMetrics(siteId),
staleTime: 25000, // Data stays fresh for 25s
refetchInterval: 30000 // Refetch every 30s
});
const { data: alerts } = useQuery({
queryKey: ['active-alerts', siteId],
queryFn: () => fetchActiveAlerts(siteId),
staleTime: 5000, // Alerts refresh more aggressively
refetchInterval: 10000
});
if (isLoading) return <DashboardSkeleton />;
return (
<DashboardGrid>
<KPIPanel metrics={metrics} />
<AlertPanel alerts={alerts} />
<TrendCharts siteId={siteId} />
</DashboardGrid>
);
}
Virtual rendering for dashboards with many time-series charts: only render chart components in the viewport, virtualize the rest. react-window or TanStack Virtual handle this.
Incremental data loading: Historical chart data loads lazily — show the last hour immediately, load 24-hour history in the background.
A Production Dashboard We Built
One production SaaS platform we support needed an operational dashboard for their business — real-time transaction metrics, geographic distribution, anomaly alerts. The previous approach: a manually refreshed spreadsheet.
Architecture delivered: - DynamoDB as the pre-aggregated metrics store (updated via EventBridge + Lambda every 60 seconds) - API Gateway + Lambda for the dashboard API (sub-100ms response times from DynamoDB) - CloudFront in front of the API for geographic performance and traffic burst handling - React frontend with React Query for smart data caching - WebSocket subscription for alert push
Result: a dashboard that updates automatically, handles hundreds of concurrent users, and delivers < 100ms page loads worldwide.
For more on the data infrastructure backing these dashboards, see our data analytics capability and the full-stack patterns in our full-stack development capability.
Frequently Asked Questions
When should we use a BI tool (Looker, Tableau, QuickSight) vs. custom development?
BI tools excel at ad hoc exploration, self-service analytics, and dashboards backed by database queries that run in seconds. Custom development is right when: sub-minute data freshness is required, the UX needs to exceed what BI tools support (custom visualization, mobile-first, embedded dashboards), or the data sources require custom integration logic that BI tools can't handle.
How do we handle dashboards with hundreds of concurrent users?
The key is not running expensive queries per-user, per-request. Pre-aggregate at write time, cache at the API layer, and serve static data from CloudFront. With this architecture, 1,000 concurrent users generate the same database load as 10.
What's the right database for a dashboard that needs to track millions of events?
Depends on the query pattern. For time-series queries ("show me pressure over the last 24 hours"): Timestream or TimescaleDB. For pre-aggregated KPI lookups ("show me current total"): DynamoDB. For complex ad hoc queries across historical data: Athena on Parquet in S3. For moderate-scale relational data with known query patterns: Aurora PostgreSQL with read replicas.
How do we add mobile support to our dashboard?
Design API responses to be mobile-appropriate from the start — smaller payloads, pre-aggregated to the display level needed. React Native shares logic with the web dashboard (queries, state management, business logic). Chart libraries like Victory Native work across web and mobile. Progressive web app (PWA) approach for simpler mobile requirements.
How do we handle timezone complexity for global dashboards?
Store all data in UTC at the database level — never store in local time. Convert to display timezone at the presentation layer. Dashboard filter controls that let users set their timezone. Time-series APIs that accept timezone parameter and return data with appropriate labels.
Discuss your automation project → rutagon.com/contact | 907-841-8407 | contact@rutagon.com