E2E Smoke Testing with Screenshots: No Selenium, No WebDriver
After every deploy, there's a window of uncertainty. Did everything come up? Is the homepage rendering? Is the login page blank? Is there a broken CSS bundle serving a completely unstyled experience?
Most teams handle this with one of: - Manual spot-checking (slow, inconsistent, doesn't scale) - Uptime monitors (tells you the server is up, not that the page looks right) - Full E2E test suites (Selenium, Playwright — expensive to maintain, often flaky)
There's a middle ground: a screenshot-based smoke test that takes 2 minutes to set up, runs in 30 seconds after every deploy, and catches the class of failures that matter most — pages that render wrong, not just pages that return 404.
What Smoke Tests Actually Need to Catch
The failures that happen most often after a deploy:
- Broken asset bundles — CSS or JS fails to load. Page renders unstyled or broken.
- Missing environment variables — Feature flags, API keys, config missing in production. Page shows error state.
- Database migration issues — Data-dependent pages show empty state or error.
- CDN cache issues — Old version of a bundle served for hours after deploy.
- Wrong environment deployed — Staging config in production, wrong domain, test data visible.
Screenshot smoke tests catch all of these. They don't catch logic errors in data processing or incorrect calculations — that's what unit tests are for. But they catch the category of failure that makes users immediately say "the site is broken."
The Smoke Test Runner
import os
import io
import sys
import json
import time
import requests
import hashlib
from pathlib import Path
from PIL import Image
import numpy as np
API_KEY = os.environ['SCREENSHOT_API_KEY']
SCREENSHOT_URL = 'https://hermesforge.dev/api/screenshot'
def capture(url, width=1280, height=800, delay=1500, auth_cookie=None):
params = {
'url': url,
'width': width,
'height': height,
'format': 'png',
'delay': delay,
'full_page': 'false',
}
headers = {'X-API-Key': API_KEY}
if auth_cookie:
params['cookies'] = auth_cookie
for attempt in range(3):
try:
resp = requests.get(SCREENSHOT_URL, params=params, headers=headers, timeout=60)
resp.raise_for_status()
return Image.open(io.BytesIO(resp.content)).convert('RGB')
except Exception as e:
if attempt == 2:
raise
time.sleep(2 ** attempt)
def is_error_page(img):
"""
Heuristic: detect blank/error pages.
- Nearly all-white: blank page or loading spinner stall
- Very high uniformity in top section: likely error page with plain background
"""
arr = np.array(img)
# Check if top 300px is >95% white (blank page heuristic)
top = arr[:300, :, :]
white_ratio = (top > 240).all(axis=2).mean()
if white_ratio > 0.95:
return True, 'page appears blank (>95% white in top 300px)'
# Check for all-same-color (server error page with solid background)
std = arr.std(axis=(0, 1)).mean()
if std < 8:
return True, f'page has very low visual variance (std={std:.1f}) — possible error page'
return False, None
def check_for_text_content(img, min_dark_pixel_pct=0.5):
"""
Verify the page has some dark pixels (text/content).
A page with no dark pixels has no visible text.
"""
arr = np.array(img)
dark = (arr < 100).any(axis=2)
pct = dark.mean() * 100
if pct < min_dark_pixel_pct:
return False, f'only {pct:.2f}% dark pixels — page may have no text content'
return True, None
class SmokeTest:
def __init__(self, name, url, check_for_text=True, auth_cookie=None,
width=1280, height=800, delay=1500):
self.name = name
self.url = url
self.check_for_text = check_for_text
self.auth_cookie = auth_cookie
self.width = width
self.height = height
self.delay = delay
def run(self, output_dir=None):
result = {'name': self.name, 'url': self.url, 'passed': False, 'failures': []}
try:
img = capture(
self.url,
width=self.width,
height=self.height,
delay=self.delay,
auth_cookie=self.auth_cookie,
)
except Exception as e:
result['failures'].append(f'Capture failed: {e}')
return result
# Save screenshot for artifact upload
if output_dir:
Path(output_dir).mkdir(exist_ok=True)
img.save(f'{output_dir}/{self.name}.png')
# Check 1: not a blank/error page
is_err, reason = is_error_page(img)
if is_err:
result['failures'].append(f'Error page detected: {reason}')
# Check 2: has text content
if self.check_for_text:
has_text, reason = check_for_text_content(img)
if not has_text:
result['failures'].append(f'No text content: {reason}')
result['passed'] = len(result['failures']) == 0
result['screenshot'] = f'{output_dir}/{self.name}.png' if output_dir else None
return result
def run_smoke_tests(tests, output_dir='smoke-screenshots'):
print(f'Running {len(tests)} smoke tests...\n')
results = []
start = time.time()
for test in tests:
print(f' [{test.name}] {test.url}')
result = test.run(output_dir=output_dir)
if result['passed']:
print(f' PASS')
else:
for failure in result['failures']:
print(f' FAIL: {failure}')
results.append(result)
elapsed = time.time() - start
failed = [r for r in results if not r['passed']]
print(f'\n{"="*50}')
print(f'Results: {len(results) - len(failed)}/{len(results)} passed in {elapsed:.1f}s')
if failed:
print(f'\nFailed tests:')
for r in failed:
print(f' - {r["name"]} ({r["url"]}):')
for f in r['failures']:
print(f' {f}')
# Write JSON report
report = {
'timestamp': time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime()),
'total': len(results),
'passed': len(results) - len(failed),
'failed': len(failed),
'results': results,
}
with open(f'{output_dir}/report.json', 'w') as f:
json.dump(report, f, indent=2)
return len(failed) == 0
Defining Your Test Suite
PROD_COOKIE = os.environ.get('SMOKE_SESSION_COOKIE', '')
SMOKE_TESTS = [
# Public pages — no auth needed
SmokeTest('homepage', 'https://yoursite.com/', delay=1000),
SmokeTest('pricing', 'https://yoursite.com/pricing', delay=1000),
SmokeTest('docs-home', 'https://yoursite.com/docs', delay=1000),
SmokeTest('blog-home', 'https://yoursite.com/blog', delay=1000),
SmokeTest('status-page', 'https://yoursite.com/status', delay=1000),
# Auth pages — check they render (don't check for text, login form may be minimal)
SmokeTest('login', 'https://yoursite.com/login', delay=1000),
SmokeTest('signup', 'https://yoursite.com/signup', delay=1000),
# Authenticated pages — require session cookie
SmokeTest('dashboard', 'https://yoursite.com/app/dashboard',
auth_cookie=PROD_COOKIE, delay=2000),
SmokeTest('settings', 'https://yoursite.com/app/settings',
auth_cookie=PROD_COOKIE, delay=1500),
# Mobile viewports for critical pages
SmokeTest('homepage-mobile', 'https://yoursite.com/',
width=390, height=844, delay=1000),
]
CI Integration
# .github/workflows/smoke-tests.yml
name: Post-Deploy Smoke Tests
on:
deployment_status:
jobs:
smoke:
runs-on: ubuntu-latest
if: github.event.deployment_status.state == 'success' &&
github.event.deployment_status.environment == 'production'
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dependencies
run: pip install requests Pillow numpy
- name: Run smoke tests
env:
SCREENSHOT_API_KEY: ${{ secrets.SCREENSHOT_API_KEY }}
SMOKE_SESSION_COOKIE: ${{ secrets.SMOKE_SESSION_COOKIE }}
run: python smoke_tests.py
- name: Upload screenshots on failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: smoke-test-screenshots-${{ github.run_id }}
path: smoke-screenshots/
retention-days: 14
- name: Notify on failure
if: failure()
uses: slackapi/slack-github-action@v1
with:
payload: |
{
"text": ":red_circle: Smoke tests failed after production deploy ${{ github.sha }}. <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View screenshots>"
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
The smoke tests run automatically after every production deploy. If any test fails, the pipeline uploads the screenshots as artifacts and sends a Slack alert with a link to view them.
Tuning the Heuristics
The default heuristics (>95% white = blank, <0.5% dark pixels = no content) work for most sites. Adjust for your specific case:
Dark-themed sites: Lower the white threshold. A dark site's "blank" page will be all-black, not all-white.
def is_error_page(img, dark_theme=False):
arr = np.array(img)
top = arr[:300, :, :]
if dark_theme:
# Check for all-dark (blank dark-theme page)
dark_ratio = (top < 30).all(axis=2).mean()
if dark_ratio > 0.95:
return True, 'page appears blank (>95% black in top 300px, dark theme)'
# ... rest of checks
Pages with minimal content (e.g., a simple login form): The check_for_text=False flag skips the dark-pixel check. Use for pages where sparse content is expected.
SPAs with long load times: Increase delay. 1500ms works for most SSR sites; React SPAs with API calls may need 2500–3000ms.
What This Doesn't Replace
Screenshot smoke tests catch visual failures. They don't catch: - API returning wrong data (page looks right, data is wrong) - Performance regressions (page renders correctly but takes 15 seconds) - Accessibility violations - Broken form submissions or interactions
For those, keep your unit tests, integration tests, and proper E2E tests. Smoke tests are the fast, cheap layer that catches the 80% of post-deploy failures that show up visually.
Running Locally
SCREENSHOT_API_KEY=your-key python smoke_tests.py
open smoke-screenshots/homepage.png
After a bad deploy, open the artifact screenshots directly. You'll see within 5 seconds exactly what the failure looks like.
Get Your API Key
Free API key at hermesforge.dev/screenshot. A 10-test smoke suite costs 10 API calls per deploy — negligible at any usage level.