# HTTP Request Flow **Document Version:** 1.0 **Last Updated:** 2026-04-04 **Status:** Production LIVE (OVH VPS) **Flow Type:** HTTP Request Handling & Response Cycle --- ## Overview This document describes the **complete HTTP request flow** for the Norda Biznes Partner application, from external user through nginx reverse proxy to Flask application and back. It covers: - **Complete request path** (Internet → Nginx → Flask → Response) - **SSL/TLS termination** and security boundaries - **Request/response transformation** at each layer - **Common failure scenarios** and troubleshooting **Key Infrastructure:** - **Public Entry:** 57.128.200.27:443 (OVH VPS, direct) - **Reverse Proxy:** Nginx on 57.128.200.27:443 (SSL termination) - **Backend Application:** Flask/Gunicorn on 127.0.0.1:5000 - **Protocol Flow:** HTTPS → Nginx → HTTP (localhost) → Flask → HTTP → Nginx → HTTPS > **Note:** The old on-prem setup used FortiGate NAT (85.237.177.83) and NPM (10.22.68.250) as intermediate layers. Production now runs directly on OVH VPS without FortiGate or NPM. **Related Documentation:** - Incident Report: `docs/INCIDENT_REPORT_20260102.md` (ERR_TOO_MANY_REDIRECTS) - Container Diagram: `docs/architecture/02-container-diagram.md` - Deployment Architecture: `docs/architecture/03-deployment-architecture.md` --- ## 1. Complete HTTP Request Flow ### 1.1 Successful Request Sequence Diagram ```mermaid sequenceDiagram actor User participant Browser participant Nginx as 🔒 Nginx Reverse Proxy
57.128.200.27:443 participant Flask as 🌐 Flask/Gunicorn
127.0.0.1:5000 participant DB as 💾 PostgreSQL
localhost:5432 Note over User,DB: SUCCESSFUL REQUEST FLOW User->>Browser: Navigate to https://nordabiznes.pl/ Browser->>Nginx: HTTPS GET / (Port 443) Note over Nginx: SSL/TLS Termination
Let's Encrypt Certificate (certbot) Note over Nginx: Request Processing
• Validate certificate
• Decrypt HTTPS
• Extract headers
• proxy_pass to localhost:5000 Nginx->>Flask: HTTP GET / (127.0.0.1:5000) Note over Flask: Flask Request Handling
• WSGI via Gunicorn
• Route matching (app.py)
• Session validation
• CSRF check (if POST) Flask->>DB: SELECT * FROM companies WHERE status='active' DB->>Flask: Company data (80 rows) Note over Flask: Template Rendering
• Jinja2 template: index.html
• Inject company data
• Apply filters & sorting Flask->>Nginx: HTTP 200 OK
Content-Type: text/html
Set-Cookie: session=...
HTML content Note over Nginx: Response Processing
• Encrypt response (HTTPS)
• Add security headers
• HSTS, CSP, X-Frame-Options Nginx->>Browser: HTTPS 200 OK Browser->>User: Display page ``` ### 1.2 Historical: Failed Request (Old NPM Setup - Port Misconfiguration) > **Note:** This failure scenario applied to the old on-prem setup with NPM. It is no longer applicable to the current OVH VPS production setup. ```mermaid sequenceDiagram actor User participant Browser participant NPM as 🔒 NPM Reverse Proxy
10.22.68.250:443 participant NginxSys as ⚠️ Nginx System
10.22.68.249:80 participant Flask as 🌐 Flask/Gunicorn
10.22.68.249:5000 Note over User,Flask: FAILED REQUEST FLOW (REDIRECT LOOP) — HISTORICAL User->>Browser: Navigate to https://nordabiznes.pl/ Browser->>NPM: HTTPS GET / (Port 443) Note over NPM: SSL/TLS Termination
Decrypt HTTPS NPM->>NginxSys: ❌ HTTP GET / (Port 80)
WRONG PORT! Note over NginxSys: Nginx System Config
Redirects ALL HTTP to HTTPS NginxSys->>NPM: HTTP 301 Moved Permanently
Location: https://nordabiznes.pl/ Note over NPM: Follow redirect NPM->>NginxSys: HTTP GET / (Port 80) NginxSys->>NPM: HTTP 301 Moved Permanently
Location: https://nordabiznes.pl/ Note over NPM: Redirect loop detected
After 20 redirects... NPM->>Browser: ERR_TOO_MANY_REDIRECTS Browser->>User: ❌ Error: Too many redirects Note over User: Portal UNAVAILABLE
30 minutes downtime
See: INCIDENT_REPORT_20260102.md ``` --- ## 2. Layer-by-Layer Request Processing ### 2.1 Layer 1: DNS + Direct Connection (OVH VPS) **Server:** OVH VPS **Public IP:** 57.128.200.27 **Function:** Direct internet-facing server (no NAT, no FortiGate for production) **Processing Steps:** 1. **Receive external request:** ``` Source: User IP (e.g., 93.104.x.x) Destination: 85.237.177.83:443 Protocol: HTTPS ``` 2. **NAT Translation:** ``` External: 85.237.177.83:443 → Internal: 10.22.68.250:443 ``` 3. **Firewall Rules:** - Allow TCP port 443 (HTTPS) - Allow TCP port 80 (HTTP, redirects to HTTPS) - Block all other inbound ports - Stateful connection tracking 4. **Forward to NPM:** ``` Destination: 10.22.68.250:443 (NPM reverse proxy) Protocol: HTTPS (encrypted tunnel) ``` **Configuration:** ``` NAT Rule: DNAT 85.237.177.83:443 → 10.22.68.250:443 Firewall: ALLOW from any to 85.237.177.83:443 (state: NEW,ESTABLISHED) ``` --- ### 2.2 Layer 2: Nginx Reverse Proxy (SSL Termination) **Server:** OVH VPS (inpi-vps-waw01) **IP:** 57.128.200.27 **Port:** 443 (HTTPS) **Technology:** Nginx with Let's Encrypt (certbot) **Processing Steps:** 1. **Receive HTTPS request:** ``` Source: Fortigate (10.22.68.250:443) Method: GET Host: nordabiznes.pl Protocol: HTTPS/1.1 or HTTP/2 ``` 2. **SSL/TLS Termination:** - Load SSL certificate (Let's Encrypt, Certificate ID: 27) - Validate certificate (nordabiznes.pl, www.nordabiznes.pl) - Decrypt HTTPS traffic - Establish secure connection with client 3. **Request Header Processing:** ```http GET / HTTP/1.1 Host: nordabiznes.pl User-Agent: Mozilla/5.0 ... Accept: text/html,application/xhtml+xml Accept-Language: pl-PL,pl;q=0.9 Accept-Encoding: gzip, deflate, br Connection: keep-alive ``` 4. **NPM Proxy Configuration Lookup:** ```sql -- NPM internal database query SELECT * FROM proxy_host WHERE id = 27; -- Result: domain_names: ["nordabiznes.pl", "www.nordabiznes.pl"] forward_scheme: "http" forward_host: "57.128.200.27" forward_port: 5000 ← CRITICAL! ssl_forced: true certificate_id: 27 ``` 5. **⚠️ CRITICAL ROUTING DECISION:** ``` ✓ CORRECT: Forward to http://57.128.200.27:5000 ❌ WRONG: Forward to http://57.128.200.27:80 (causes redirect loop!) ``` 6. **Forward to Backend (HTTP, unencrypted):** ```http GET / HTTP/1.1 Host: nordabiznes.pl X-Real-IP: 93.104.x.x (original client IP) X-Forwarded-For: 93.104.x.x X-Forwarded-Proto: https X-Forwarded-Host: nordabiznes.pl Connection: close ``` **NPM Configuration (Proxy Host ID: 27):** | Parameter | Value | Notes | |-----------|-------|-------| | Domain Names | nordabiznes.pl, www.nordabiznes.pl | Primary + www alias | | Forward Scheme | http | NPM→Backend uses HTTP (secure internal network) | | Forward Host | 127.0.0.1 | Localhost (same OVH VPS) | | **Forward Port** | **5000** | **Flask/Gunicorn port (CRITICAL!)** | | SSL Certificate | 27 (Let's Encrypt) | Auto-renewal enabled | | SSL Forced | Yes | Redirect HTTP→HTTPS | | HTTP/2 Support | Yes | Modern protocol support | | HSTS Enabled | Yes | max-age=31536000; includeSubDomains | | Block Exploits | Yes | Nginx security module | | Websocket Support | Yes | For future features | **Verification Command:** ```bash # Check NPM configuration ssh maciejpi@10.22.68.250 "docker exec nginx-proxy-manager_app_1 \ sqlite3 /data/database.sqlite \ \"SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE id = 27;\"" # Expected output: # 27|["nordabiznes.pl","www.nordabiznes.pl"]|57.128.200.27|5000 ``` --- ### 2.3 Layer 3: Flask/Gunicorn Application (Request Processing) **Server:** OVH VPS (inpi-vps-waw01) **IP:** 127.0.0.1 (localhost, via nginx proxy_pass) **Port:** 5000 **Technology:** Gunicorn 20.1.0 + Flask 3.0 **Processing Steps:** 1. **Gunicorn Receives HTTP Request:** ``` Binding: 127.0.0.1:5000 (all interfaces) Workers: 4 (Gunicorn worker processes) Worker Class: sync (synchronous workers) Timeout: 120 seconds ``` 2. **Worker Selection:** - Gunicorn master process receives connection - Distributes request to available worker (round-robin) - Worker loads WSGI app (Flask application) 3. **WSGI Interface:** ```python # Gunicorn calls Flask's WSGI application from app import app # WSGI environ dict contains: environ = { 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/', 'QUERY_STRING': '', 'SERVER_NAME': 'nordabiznes.pl', 'SERVER_PORT': '5000', 'HTTP_HOST': 'nordabiznes.pl', 'HTTP_X_REAL_IP': '93.104.x.x', 'HTTP_X_FORWARDED_PROTO': 'https', 'wsgi.url_scheme': 'http', # ... more headers } ``` 4. **Flask Request Handling (app.py):** **a) Request Context Setup:** ```python # Flask creates request context from flask import request, session, g # Parse request request.method = 'GET' request.path = '/' request.url = 'https://nordabiznes.pl/' request.args = {} # Query parameters request.headers = {...} # HTTP headers ``` **b) Before Request Hooks:** ```python @app.before_request def load_logged_in_user(): # Check session for user_id user_id = session.get('user_id') if user_id: g.user = db.session.query(User).get(user_id) else: g.user = None ``` **c) Route Matching:** ```python # Flask router matches route @app.route('/') def index(): # Main catalog page pass ``` **d) View Function Execution:** ```python @app.route('/') def index(): # Query companies from database companies = db.session.query(Company)\ .filter_by(status='active')\ .order_by(Company.name)\ .all() # 80 companies # Render template return render_template('index.html', companies=companies) ``` 5. **Database Query (PostgreSQL):** ```python # SQLAlchemy ORM generates SQL SELECT * FROM companies WHERE status = 'active' ORDER BY name; ``` 6. **Template Rendering (Jinja2):** ```python # templates/index.html from jinja2 import Environment, FileSystemLoader # Render template with context html = render_template('index.html', companies=companies, user=g.user, config=app.config) ``` 7. **Response Construction:** ```python # Flask creates HTTP response response = Response( response=html, # Rendered HTML status=200, # HTTP status code headers={ 'Content-Type': 'text/html; charset=utf-8', 'Content-Length': len(html), 'Set-Cookie': 'session=...; HttpOnly; Secure; SameSite=Lax' } ) ``` **Gunicorn Configuration:** ```ini # /etc/systemd/system/nordabiznes.service [Service] User=maciejpi Group=maciejpi WorkingDirectory=/var/www/nordabiznes ExecStart=/var/www/nordabiznes/venv/bin/gunicorn \ --bind 127.0.0.1:5000 \ --workers 4 \ --timeout 120 \ --access-logfile /var/log/nordabiznes/access.log \ --error-logfile /var/log/nordabiznes/error.log \ --log-level info \ app:app ``` **Verification Command:** ```bash # Test Flask directly (from server) curl -I http://57.128.200.27:5000/health # Expected: HTTP/1.1 200 OK # Check Gunicorn workers ssh maciejpi@57.128.200.27 "ps aux | grep gunicorn" # Expected: 1 master + 4 worker processes ``` --- ### 2.4 Layer 4: PostgreSQL Database (Data Retrieval) **Server:** OVH VPS (same server as Flask) **IP:** 127.0.0.1 (localhost only) **Port:** 5432 **Technology:** PostgreSQL 14 **Processing Steps:** 1. **Receive SQL Query:** ```sql -- SQLAlchemy ORM generates query SELECT companies.id, companies.name, companies.slug, companies.short_description, companies.category, companies.nip, companies.city, companies.website FROM companies WHERE companies.status = 'active' ORDER BY companies.name; ``` 2. **Query Execution:** - Parse SQL syntax - Create execution plan (Query planner) - Use indexes if available (idx_companies_status, idx_companies_name) - Fetch rows from disk/cache 3. **Return Results:** ```python # SQLAlchemy ORM returns Company objects [ Company(id=1, name='ALMARES', slug='almares', ...), Company(id=2, name='AMA', slug='ama-spolka-z-o-o', ...), # ... 78 more companies ] ``` **Connection Configuration:** ```python # app.py database connection DATABASE_URL = 'postgresql://nordabiz_app:PASSWORD@localhost:5432/nordabiz' # SQLAlchemy engine config engine = create_engine(DATABASE_URL, pool_size=10, # Connection pool max_overflow=20, # Additional connections pool_pre_ping=True, # Validate connections pool_recycle=3600 # Recycle after 1 hour ) ``` **Security:** - PostgreSQL listens on **127.0.0.1 ONLY** (no external access) - Application connects via localhost socket - User: `nordabiz_app` (limited privileges, no DROP/CREATE) - SSL: Not required (localhost connection) **Verification:** ```bash # Check PostgreSQL status ssh maciejpi@57.128.200.27 "sudo systemctl status postgresql" # Test connection (from server) ssh maciejpi@57.128.200.27 "psql -U nordabiz_app -h 127.0.0.1 -d nordabiz -c 'SELECT COUNT(*) FROM companies;'" # Expected: 80 ``` --- ## 3. Response Flow (Flask → User) ### 3.1 Response Path Sequence ```mermaid sequenceDiagram participant Flask as 🌐 Flask/Gunicorn
57.128.200.27:5000 participant NPM as 🔒 NPM Reverse Proxy
10.22.68.250:443 participant Fortigate as 🛡️ Fortigate Firewall
85.237.177.83 participant Browser Note over Flask: Response Construction Flask->>Flask: Render Jinja2 template
HTML content (50 KB) Flask->>Nginx: HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 51234
Set-Cookie: session=...

[HTML content] Note over NPM: Response Processing NPM->>NPM: Add security headers:
• Strict-Transport-Security
• X-Frame-Options: SAMEORIGIN
• X-Content-Type-Options: nosniff
• Referrer-Policy: strict-origin NPM->>NPM: Compress response (gzip)
Size: 50 KB → 12 KB NPM->>NPM: Encrypt response (TLS 1.3)
Certificate: Let's Encrypt Nginx->>Browser: HTTPS 200 OK
Encrypted response
Size: 12 KB (gzip) Note over Fortigate: NAT reverse translation Fortigate->>Browser: HTTPS 200 OK
Source: 85.237.177.83 Note over Browser: Response Handling Browser->>Browser: Decrypt HTTPS
Decompress gzip
Parse HTML
Render page ``` ### 3.2 Response Headers (NPM → User) **Headers Added by NPM:** ```http HTTP/2 200 OK server: nginx/1.24.0 (Ubuntu) date: Fri, 10 Jan 2026 10:30:00 GMT content-type: text/html; charset=utf-8 content-length: 12345 strict-transport-security: max-age=31536000; includeSubDomains x-frame-options: SAMEORIGIN x-content-type-options: nosniff referrer-policy: strict-origin-when-cross-origin content-security-policy: default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline' ``` **Headers from Flask (preserved):** ```http set-cookie: session=eyJ...; HttpOnly; Path=/; SameSite=Lax; Secure vary: Cookie x-request-id: abc123def456 ``` **Response Body:** ```html Norda Biznes Partner - Katalog firm ``` --- ## 4. Port Mapping Reference ### 4.1 Complete Port Flow ``` ┌─────────────────────────────────────────────────────────────────┐ │ EXTERNAL USER │ │ Browser: https://nordabiznes.pl/ │ └────────────────────────────┬────────────────────────────────────┘ │ HTTPS (Port 443) ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FORTIGATE FIREWALL │ │ Public IP: 85.237.177.83:443 │ │ NAT: 85.237.177.83:443 → 10.22.68.250:443 │ └────────────────────────────┬────────────────────────────────────┘ │ HTTPS (Port 443) ▼ ┌─────────────────────────────────────────────────────────────────┐ │ NPM REVERSE PROXY (R11-REVPROXY-01) │ │ IP: 10.22.68.250:443 │ │ Function: SSL Termination │ │ Certificate: Let's Encrypt (nordabiznes.pl) │ │ │ │ ⚠️ CRITICAL ROUTING DECISION: │ │ ✓ Forward to: http://57.128.200.27:5000 (CORRECT) │ │ ❌ DO NOT use: http://57.128.200.27:80 (WRONG!) │ └────────────────────────────┬────────────────────────────────────┘ │ HTTP (Port 5000) ✓ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FLASK/GUNICORN (OVH VPS) │ │ IP: 57.128.200.27:5000 │ │ Binding: 127.0.0.1:5000 │ │ Workers: 4 (Gunicorn) │ │ Function: Application logic, template rendering │ └────────────────────────────┬────────────────────────────────────┘ │ SQL (localhost:5432) ▼ ┌─────────────────────────────────────────────────────────────────┐ │ POSTGRESQL (OVH VPS) │ │ IP: 127.0.0.1:5432 (localhost only) │ │ Database: nordabiz │ │ Function: Data storage │ └─────────────────────────────────────────────────────────────────┘ ``` ### 4.2 Port Table (OVH VPS) | Port | Service | Binding | User | Purpose | NPM Should Use? | |------|---------|---------|------|---------|-----------------| | **5000** | **Gunicorn/Flask** | **0.0.0.0** | **maciejpi** | **Main Application** | **✓ YES** | | 80 | Nginx (system) | 0.0.0.0 | root | HTTP→HTTPS redirect | ❌ NO (causes loop!) | | 443 | Nginx (system) | 0.0.0.0 | root | HTTPS redirect | ❌ NO (NPM handles SSL) | | 5432 | PostgreSQL | 127.0.0.1 | postgres | Database | ❌ NO (localhost only) | | 22 | SSH | 0.0.0.0 | - | Administration | ❌ NO (not HTTP) | **⚠️ CRITICAL WARNING:** ``` Port 80 and 443 on OVH VPS run a system nginx that: 1. Redirects ALL HTTP requests to HTTPS 2. Redirects ALL HTTPS requests to https://nordabiznes.pl If NPM forwards to port 80: NPM (HTTPS) → Nginx (port 80) → 301 redirect to HTTPS → NPM receives redirect → forwards to port 80 again → INFINITE LOOP → ERR_TOO_MANY_REDIRECTS SOLUTION: NPM must ALWAYS forward to port 5000! ``` --- ## 5. Request Types and Routing ### 5.1 Static Assets **Request:** `GET https://nordabiznes.pl/static/css/style.css` **Flow:** ``` User → Nginx → Flask → Static file handler → Return CSS ``` **Flask Handling:** ```python # Flask serves static files from /static directory @app.route('/static/') def static_files(filename): return send_from_directory('static', filename) ``` **Optimization:** NPM could cache static assets (future enhancement) --- ### 5.2 API Endpoints **Request:** `GET https://nordabiznes.pl/api/companies` **Flow:** ``` User → Nginx → Flask → API route → Database → JSON response ``` **Response:** ```json { "companies": [ { "id": 1, "name": "PIXLAB", "slug": "pixlab-sp-z-o-o", "category": "IT" } ], "total": 80 } ``` **Headers:** ```http Content-Type: application/json; charset=utf-8 Access-Control-Allow-Origin: * (if CORS enabled) ``` --- ### 5.3 Form Submissions (POST) **Request:** `POST https://nordabiznes.pl/login` **Flow:** ``` User → Nginx → Flask → CSRF validation → Auth check → Database → Redirect ``` **Additional Processing:** - **CSRF Token Validation:** Flask-WTF checks token in form - **Session Creation:** Flask-Login creates session cookie - **Database Update:** Update `last_login` timestamp **Response:** ```http HTTP/2 302 Found Location: https://nordabiznes.pl/dashboard Set-Cookie: session=...; HttpOnly; Secure; SameSite=Lax ``` --- ### 5.4 Health Check Endpoint **Request:** `GET https://nordabiznes.pl/health` **Purpose:** Monitoring and verification **Flow:** ``` Monitor → Nginx → Flask → Simple response (no DB query) ``` **Response:** ```http HTTP/2 200 OK Content-Type: application/json { "status": "healthy", "timestamp": "2026-01-10T10:30:00Z" } ``` **Use Cases:** - Verify NPM configuration after changes - External monitoring (Zabbix) - Load balancer health checks (future) --- ## 6. Security Layers ### 6.1 Security Boundaries ```mermaid graph TB Internet["🌐 Internet
(Untrusted)"] subgraph "Security Zone 1: DMZ" Fortigate["🛡️ Fortigate Firewall
• NAT
• DPI
• IPS"] NPM["🔒 NPM Reverse Proxy
• SSL/TLS Termination
• HSTS
• Rate Limiting"] end subgraph "Security Zone 2: Application Layer" Flask["🌐 Flask Application
• CSRF Protection
• Input Validation
• XSS Prevention
• Session Management"] end subgraph "Security Zone 3: Data Layer" DB["💾 PostgreSQL
• No external access
• User privileges
• SQL injection protection"] end Internet --> Fortigate Fortigate --> NPM NPM --> Flask Flask --> DB classDef zoneStyle fill:#e74c3c,stroke:#c0392b,color:#fff classDef appStyle fill:#3498db,stroke:#2980b9,color:#fff classDef dataStyle fill:#2ecc71,stroke:#27ae60,color:#fff class Fortigate,NPM zoneStyle class Flask appStyle class DB dataStyle ``` ### 6.2 Security Features by Layer **Layer 1: Fortigate Firewall** - Deep Packet Inspection (DPI) - Intrusion Prevention System (IPS) - Rate limiting (connection-level) - Geo-blocking (if configured) **Layer 2: NPM Reverse Proxy** - SSL/TLS 1.3 encryption - HSTS (HTTP Strict Transport Security) - Certificate validation - Security headers injection - Request size limits **Layer 3: Flask Application** - Flask-Login session management - CSRF token validation (Flask-WTF) - Input sanitization (XSS prevention) - SQL injection protection (SQLAlchemy ORM) - Rate limiting (Flask-Limiter) - Authentication & Authorization **Layer 4: PostgreSQL Database** - Localhost-only binding (127.0.0.1) - User privilege separation (nordabiz_app) - Parameterized queries (SQLAlchemy) - Connection pooling with limits --- ## 7. Performance Metrics ### 7.1 Typical Request Latency | Layer | Processing Time | Notes | |-------|-----------------|-------| | Fortigate NAT | < 1 ms | Hardware NAT, negligible latency | | NPM SSL Termination | 10-20 ms | TLS handshake + decryption | | Nginx → Flask Network | < 1 ms | Internal 10 Gbps network | | Flask Request Handling | 50-150 ms | Route matching, template rendering | | Database Query | 10-30 ms | Indexed queries, connection pool | | Template Rendering | 20-50 ms | Jinja2 template compilation | | NPM SSL Encryption | 5-10 ms | Response encryption | | **Total (typical)** | **96-262 ms** | **Median: ~180 ms** | **Factors Affecting Latency:** - Database query complexity (JOIN operations, FTS) - Template size (company catalog vs. simple page) - Gunicorn worker availability (max 4 concurrent) - Network congestion (internal or external) ### 7.2 Throughput **Gunicorn Workers:** 4 synchronous workers **Concurrent Requests:** Max 4 simultaneous requests **Requests per Second (RPS):** - Simple pages (health check): ~20-30 RPS - Database queries (catalog): ~5-10 RPS - Complex AI chat: ~2-3 RPS (limited by Gemini API) **Bottlenecks:** 1. Gunicorn worker count (4 workers) 2. PostgreSQL connection pool (max 30) 3. AI API rate limits (1,500 req/day Gemini) --- ## 8. Troubleshooting Guide ### 8.1 Common Issues and Diagnostics #### Issue 1: ERR_TOO_MANY_REDIRECTS **Symptoms:** - Browser error: "ERR_TOO_MANY_REDIRECTS" - Portal inaccessible from external network - Works from internal network (10.22.68.0/24) **Root Cause:** - NPM forwarding to port 80 instead of 5000 **Diagnosis:** ```bash # 1. Check NPM configuration ssh maciejpi@10.22.68.250 "docker exec nginx-proxy-manager_app_1 \ sqlite3 /data/database.sqlite \ \"SELECT forward_port FROM proxy_host WHERE id = 27;\"" # Expected: 5000 # If shows: 80 ← PROBLEM! # 2. Test direct backend access curl -I http://57.128.200.27:80/ # If returns: HTTP 301 → Problem confirmed curl -I http://57.128.200.27:5000/ # Should return: HTTP 200 OK ``` **Solution:** ```bash # Update NPM configuration via API or UI # Set forward_port to 5000 # Verify fix curl -I https://nordabiznes.pl/health # Expected: HTTP 200 OK ``` **Reference:** `docs/INCIDENT_REPORT_20260102.md` --- #### Issue 2: 502 Bad Gateway **Symptoms:** - NPM returns "502 Bad Gateway" - Error in NPM logs: "connect() failed (111: Connection refused)" **Root Cause:** - Flask/Gunicorn not running - Flask listening on wrong port - Firewall blocking port 5000 **Diagnosis:** ```bash # 1. Check if Gunicorn is running ssh maciejpi@57.128.200.27 "sudo systemctl status nordabiznes" # Expected: Active (running) # 2. Check if port 5000 is listening ssh maciejpi@57.128.200.27 "sudo netstat -tlnp | grep 5000" # Expected: 127.0.0.1:5000 ... gunicorn # 3. Test direct connection curl -I http://57.128.200.27:5000/health # Expected: HTTP 200 OK ``` **Solution:** ```bash # Restart Gunicorn ssh maciejpi@57.128.200.27 "sudo systemctl restart nordabiznes" # Check logs ssh maciejpi@57.128.200.27 "sudo journalctl -u nordabiznes -n 50" ``` --- #### Issue 3: 504 Gateway Timeout **Symptoms:** - Request takes > 60 seconds - NPM returns "504 Gateway Timeout" **Root Cause:** - Flask processing takes too long - Database query timeout - External API timeout (Gemini, PageSpeed) **Diagnosis:** ```bash # 1. Check Gunicorn worker status ssh maciejpi@57.128.200.27 "ps aux | grep gunicorn" # Look for workers in state 'R' (running) vs 'S' (sleeping) # 2. Check application logs ssh maciejpi@57.128.200.27 "tail -f /var/log/nordabiznes/error.log" # 3. Check database connections ssh maciejpi@57.128.200.27 "sudo -u postgres psql -c \ \"SELECT count(*) FROM pg_stat_activity WHERE datname='nordabiz';\"" ``` **Solution:** - Increase Gunicorn timeout (default: 120s) - Optimize slow database queries - Add timeout to external API calls --- #### Issue 4: SSL Certificate Error **Symptoms:** - Browser warning: "Your connection is not private" - Certificate expired or invalid **Root Cause:** - Let's Encrypt certificate expired - Certificate renewal failed - Wrong certificate for domain **Diagnosis:** ```bash # Check certificate expiry echo | openssl s_client -servername nordabiznes.pl \ -connect 85.237.177.83:443 2>/dev/null | openssl x509 -noout -dates # Expected: notAfter date in the future ``` **Solution:** ```bash # Renew certificate via NPM UI or API # NPM auto-renews 30 days before expiry ``` --- ### 8.2 Diagnostic Commands Reference **Quick Health Check:** ```bash # External access test curl -I https://nordabiznes.pl/health # Expected: HTTP/2 200 OK # Internal access test (from INPI network) curl -I http://57.128.200.27:5000/health # Expected: HTTP/1.1 200 OK ``` **NPM Configuration Check:** ```bash # Show proxy host configuration ssh maciejpi@10.22.68.250 "docker exec nginx-proxy-manager_app_1 \ sqlite3 /data/database.sqlite \ \"SELECT id, domain_names, forward_host, forward_port, ssl_forced \ FROM proxy_host WHERE id = 27;\"" ``` **Application Status:** ```bash # Service status ssh maciejpi@57.128.200.27 "sudo systemctl status nordabiznes" # Worker processes ssh maciejpi@57.128.200.27 "ps aux | grep gunicorn | grep -v grep" # Recent logs ssh maciejpi@57.128.200.27 "sudo journalctl -u nordabiznes -n 20 --no-pager" ``` **Database Status:** ```bash # PostgreSQL status ssh maciejpi@57.128.200.27 "sudo systemctl status postgresql" # Connection count ssh maciejpi@57.128.200.27 "sudo -u postgres psql -c \ \"SELECT count(*) FROM pg_stat_activity WHERE datname='nordabiz';\"" # Database size ssh maciejpi@57.128.200.27 "sudo -u postgres psql -c \ \"SELECT pg_size_pretty(pg_database_size('nordabiz'));\"" ``` **Network Connectivity:** ```bash # Test Nginx → Flask connectivity ssh maciejpi@10.22.68.250 "curl -I http://57.128.200.27:5000/health" # Test Flask → Database connectivity ssh maciejpi@57.128.200.27 "psql -U nordabiz_app -h 127.0.0.1 \ -d nordabiz -c 'SELECT 1;'" ``` --- ## 9. Configuration Reference ### 9.1 NPM Proxy Host Configuration **File Location (Docker):** `/data/database.sqlite` (inside NPM container) **Proxy Host ID:** 27 **Complete Configuration:** ```json { "id": 27, "domain_names": ["nordabiznes.pl", "www.nordabiznes.pl"], "forward_scheme": "http", "forward_host": "57.128.200.27", "forward_port": 5000, "access_list_id": 0, "certificate_id": 27, "ssl_forced": true, "caching_enabled": false, "block_exploits": true, "advanced_config": "", "allow_websocket_upgrade": true, "http2_support": true, "hsts_enabled": true, "hsts_subdomains": true } ``` **How to Update (NPM API):** ```python import requests NPM_URL = "http://10.22.68.250:81/api" # Login to get token first, then: data = { "domain_names": ["nordabiznes.pl", "www.nordabiznes.pl"], "forward_scheme": "http", "forward_host": "57.128.200.27", "forward_port": 5000, # CRITICAL! "certificate_id": 27, "ssl_forced": True, "http2_support": True, "hsts_enabled": True } response = requests.put( f"{NPM_URL}/nginx/proxy-hosts/27", headers={"Authorization": f"Bearer {token}"}, json=data ) ``` --- ### 9.2 Gunicorn Configuration **Systemd Unit File:** `/etc/systemd/system/nordabiznes.service` ```ini [Unit] Description=Norda Biznes Partner Flask Application After=network.target postgresql.service [Service] Type=notify User=maciejpi Group=maciejpi WorkingDirectory=/var/www/nordabiznes Environment="PATH=/var/www/nordabiznes/venv/bin" ExecStart=/var/www/nordabiznes/venv/bin/gunicorn \ --bind 127.0.0.1:5000 \ --workers 4 \ --worker-class sync \ --timeout 120 \ --keep-alive 5 \ --access-logfile /var/log/nordabiznes/access.log \ --error-logfile /var/log/nordabiznes/error.log \ --log-level info \ --capture-output \ app:app Restart=always RestartSec=10 [Install] WantedBy=multi-user.target ``` **Parameters Explained:** - `--bind 127.0.0.1:5000` - Listen on all interfaces, port 5000 - `--workers 4` - 4 worker processes (matches CPU cores) - `--worker-class sync` - Synchronous workers (default) - `--timeout 120` - 120 second request timeout - `--keep-alive 5` - 5 second keep-alive for connections --- ### 9.3 Flask Application Configuration **Environment Variables (.env):** ```bash # Database DATABASE_URL=postgresql://nordabiz_app:PASSWORD@localhost:5432/nordabiz # Flask SECRET_KEY=random_secret_key_here FLASK_ENV=production # Security SESSION_COOKIE_SECURE=True SESSION_COOKIE_HTTPONLY=True SESSION_COOKIE_SAMESITE=Lax # External APIs GOOGLE_API_KEY=AIza... BRAVE_API_KEY=BSA... MS_GRAPH_CLIENT_ID=abc... MS_GRAPH_CLIENT_SECRET=def... ``` **Flask Config (app.py):** ```python app.config['SQLALCHEMY_DATABASE_URI'] = os.getenv('DATABASE_URL') app.config['SECRET_KEY'] = os.getenv('SECRET_KEY') app.config['SESSION_COOKIE_SECURE'] = True app.config['SESSION_COOKIE_HTTPONLY'] = True app.config['SESSION_COOKIE_SAMESITE'] = 'Lax' app.config['PERMANENT_SESSION_LIFETIME'] = timedelta(days=7) ``` --- ## 10. Monitoring and Logging ### 10.1 Log Locations **NPM Logs (Docker):** ```bash # Access logs ssh maciejpi@10.22.68.250 "docker logs nginx-proxy-manager_app_1 --tail 100" # Follow logs in real-time ssh maciejpi@10.22.68.250 "docker logs -f nginx-proxy-manager_app_1" ``` **Gunicorn Logs:** ```bash # Application logs (systemd journal) ssh maciejpi@57.128.200.27 "sudo journalctl -u nordabiznes -f" # Access logs (file-based) ssh maciejpi@57.128.200.27 "tail -f /var/log/nordabiznes/access.log" # Error logs ssh maciejpi@57.128.200.27 "tail -f /var/log/nordabiznes/error.log" ``` **PostgreSQL Logs:** ```bash # Query logs (if enabled) ssh maciejpi@57.128.200.27 "sudo tail -f /var/log/postgresql/postgresql-14-main.log" ``` ### 10.2 Key Metrics to Monitor **HTTP Metrics (NPM):** - Requests per second (RPS) - HTTP status codes (200, 301, 404, 500, 502, 504) - Response time (p50, p95, p99) - SSL certificate expiry **Application Metrics (Gunicorn):** - Worker utilization (all 4 workers busy?) - Request timeout rate - Error rate (5xx responses) - Memory usage per worker **Database Metrics (PostgreSQL):** - Connection count (should be < 30) - Query execution time - Database size growth - Table bloat **System Metrics (OVH VPS):** - CPU usage (should be < 80%) - Memory usage (should be < 6 GB / 8 GB) - Disk I/O (should be low) - Network throughput --- ## 11. Future Enhancements ### 11.1 Planned Improvements **1. CDN Integration** - Cloudflare or custom CDN for static assets - Reduces load on NPM and Flask - Improves global latency **2. Load Balancing** - Multiple Flask backend instances - NPM upstream configuration - Session affinity handling **3. Caching Layer** - Redis cache for frequent queries - Page cache for anonymous users - API response caching **4. Monitoring System** - Zabbix integration for health checks - Alert on 5xx errors, high latency - Dashboard for key metrics **5. Rate Limiting** - NPM-level rate limiting by IP - Protection against DDoS - API endpoint throttling --- ## 12. Related Documentation - **Incident Report:** `docs/INCIDENT_REPORT_20260102.md` - NPM port configuration incident - **System Context:** `docs/architecture/01-system-context.md` - High-level system view - **Container Diagram:** `docs/architecture/02-container-diagram.md` - Container architecture - **Deployment Architecture:** `docs/architecture/03-deployment-architecture.md` - Infrastructure details - **Flask Components:** `docs/architecture/04-flask-components.md` - Application structure - **Authentication Flow:** `docs/architecture/flows/01-authentication-flow.md` - User authentication - **Search Flow:** `docs/architecture/flows/02-search-flow.md` - Company search process - **CLAUDE.md:** Main project documentation with NPM configuration reference --- ## Glossary - **NPM**: Nginx Proxy Manager - Docker-based reverse proxy with web UI - **NAT**: Network Address Translation - Fortigate translates external IP to internal - **SSL Termination**: Decrypting HTTPS at NPM, forwarding HTTP to backend - **HSTS**: HTTP Strict Transport Security - Forces browsers to use HTTPS - **WSGI**: Web Server Gateway Interface - Python web application interface - **Gunicorn**: Python WSGI HTTP server (Green Unicorn) - **Round Robin**: Load balancing method distributing requests evenly - **ERR_TOO_MANY_REDIRECTS**: Browser error when redirect loop detected (> 20 redirects) --- **Document Status:** Production-ready, verified against live system **Last Incident:** 2026-01-02 (ERR_TOO_MANY_REDIRECTS due to port 80 configuration) **Next Review:** After any NPM configuration changes --- *This document was created as part of the architecture documentation initiative to prevent configuration incidents and improve system understanding.*