Maciej Pienczyn cebe52f303 refactor: Rebranding i aktualizacja modelu AI

- Zmiana nazwy: "Norda Biznes Hub" → "Norda Biznes Partner"
- Aktualizacja modelu AI: Gemini 2.0 Flash → Gemini 3 Flash
- Zachowano historyczne odniesienia w timeline i dokumentacji

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

2026-01-29 14:08:39 +01:00

59 KiB

Raw Blame History

11. Troubleshooting Guide

Document Type: Operations Guide Last Updated: 2026-01-10 Maintainer: DevOps Team

Quick Reference
Infrastructure & Network Issues
Application & Service Issues
Database Issues
API Integration Issues
Authentication & Security Issues
Performance Issues
Monitoring & Diagnostics
Emergency Procedures
Diagnostic Commands Reference

1. Quick Reference

1.1 Emergency Contacts

Role	Contact	Availability
System Administrator	maciejpi@inpi.local	Business hours
Database Administrator	maciejpi@inpi.local	Business hours
On-Call Support	See CLAUDE.md	24/7

1.2 Critical Services Status Check

# Quick health check - run this first!
curl -I https://nordabiznes.pl/health

# Expected: HTTP/2 200
# If failed, proceed to relevant section below

1.3 Issue Decision Tree

graph TD
    A[Issue Detected] --> B{Can access site?}
    B -->|No| C{From where?}
    B -->|Yes, but slow| D[Check Performance Issues]
    C -->|Nowhere| E[Section 2.1: ERR_TOO_MANY_REDIRECTS]
    C -->|Only internal| E
    C -->|500 Error| F[Section 3.1: Application Crash]
    B -->|Yes, specific feature broken| G{Which feature?}
    G -->|Login/Auth| H[Section 6: Authentication Issues]
    G -->|Search| I[Section 3.3: Search Issues]
    G -->|AI Chat| J[Section 5.2: Gemini API Issues]
    G -->|Database| K[Section 4: Database Issues]

1.4 Severity Levels

Level	Description	Response Time	Example
CRITICAL	Complete service outage	Immediate	ERR_TOO_MANY_REDIRECTS
HIGH	Major feature broken	< 1 hour	Database connection lost
MEDIUM	Minor feature degraded	< 4 hours	Search slow
LOW	Cosmetic or minor bug	Next business day	UI glitch

2. Infrastructure & Network Issues

2.1 ERR_TOO_MANY_REDIRECTS

Severity: CRITICAL Incident History: 2026-01-02 (30 min outage)

Symptoms

Browser error: ERR_TOO_MANY_REDIRECTS
Portal completely inaccessible via https://nordabiznes.pl
Internal access works fine (http://10.22.68.249:5000)
Affects 100% of external users

Root Cause

Nginx Proxy Manager (NPM) configured to forward to port 80 instead of port 5000.

Why this causes redirect loop:

NPM forwards HTTPS → HTTP to backend port 80
Nginx on port 80 sees HTTP and redirects to HTTPS
Request goes back to NPM, creating infinite loop
Browser aborts after ~20 redirects

Diagnosis

# 1. Check NPM proxy configuration
ssh maciejpi@10.22.68.250
docker exec nginx-proxy-manager_app_1 \
  sqlite3 /data/database.sqlite \
  "SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE id = 27;"

# Expected output:
# 27|["nordabiznes.pl","www.nordabiznes.pl"]|10.22.68.249|5000

# If forward_port shows 80 → PROBLEM FOUND!

# 2. Test backend directly
curl -I http://10.22.68.249:80/
# If this returns 301 redirect → confirms issue

curl -I http://10.22.68.249:5000/health
# Should return 200 OK if Flask is running

Solution

Option A: Fix via NPM Web UI (Recommended)

# 1. Access NPM admin panel
open http://10.22.68.250:81

# 2. Navigate to: Proxy Hosts → nordabiznes.pl (ID 27)
# 3. Edit configuration:
#    - Forward Hostname/IP: 10.22.68.249
#    - Forward Port: 5000 (CRITICAL!)
#    - Scheme: http
# 4. Save and test

Option B: Fix via NPM API

import requests

NPM_URL = "http://10.22.68.250:81/api"
# Login to get token first (see NPM API docs)

data = {
    "domain_names": ["nordabiznes.pl", "www.nordabiznes.pl"],
    "forward_scheme": "http",
    "forward_host": "10.22.68.249",
    "forward_port": 5000,  # CRITICAL: Must be 5000!
    "certificate_id": 27,
    "ssl_forced": True,
    "http2_support": True
}

response = requests.put(
    f"{NPM_URL}/nginx/proxy-hosts/27",
    headers={"Authorization": f"Bearer {token}"},
    json=data
)

Verification

# 1. External test (from outside INPI network)
curl -I https://nordabiznes.pl/health
# Expected: HTTP/2 200

# 2. Check NPM logs
ssh maciejpi@10.22.68.250
docker logs nginx-proxy-manager_app_1 --tail 20
# Should show 200 responses, not 301

Prevention

ALWAYS verify port 5000 after ANY NPM configuration change
Add monitoring alert for non-200 responses on /health
Document NPM configuration in change requests
Test from external network before marking changes complete

2.2 502 Bad Gateway

Severity: HIGH

Symptoms

Browser shows "502 Bad Gateway" error
NPM logs show "upstream connection failed"
Site completely inaccessible

Root Causes

Flask/Gunicorn service stopped
Backend server (10.22.68.249) unreachable
Firewall blocking port 5000

Diagnosis

# 1. Check Flask service status
ssh maciejpi@10.22.68.249
sudo systemctl status nordabiznes

# 2. Check if port 5000 is listening
sudo netstat -tlnp | grep :5000
# Expected: gunicorn process listening

# 3. Check Flask logs
sudo journalctl -u nordabiznes -n 50 --no-pager

# 4. Test backend directly
curl http://localhost:5000/health
# Should return JSON with status

Solution

If service is stopped:

# Restart Flask application
sudo systemctl restart nordabiznes

# Check status
sudo systemctl status nordabiznes

# Verify it's working
curl http://localhost:5000/health

If service won't start:

# Check for syntax errors
cd /var/www/nordabiznes
sudo -u www-data /var/www/nordabiznes/venv/bin/python3 -m py_compile app.py

# Check for missing dependencies
sudo -u www-data /var/www/nordabiznes/venv/bin/python3 -c "import flask; import sqlalchemy"

# Check environment variables
sudo -u www-data cat /var/www/nordabiznes/.env | grep -v "PASSWORD\|SECRET\|KEY"

# Try running manually (for debugging)
sudo -u www-data /var/www/nordabiznes/venv/bin/python3 app.py

If network issue:

# Test connectivity from NPM to backend
ssh maciejpi@10.22.68.250
curl -I http://10.22.68.249:5000/health

# Check firewall rules
ssh maciejpi@10.22.68.249
sudo iptables -L -n | grep 5000

Verification

curl -I https://nordabiznes.pl/health
# Expected: HTTP/2 200

2.3 504 Gateway Timeout

Severity: MEDIUM

Symptoms

Browser shows "504 Gateway Timeout"
Requests take >60 seconds
Some requests succeed, others timeout

Root Causes

Database query hanging
External API timeout (Gemini, PageSpeed, etc.)
Insufficient Gunicorn workers
Resource exhaustion (CPU, memory)

Diagnosis

# 1. Check Gunicorn worker status
ssh maciejpi@10.22.68.249
ps aux | grep gunicorn
# Look for zombie workers or high CPU usage

# 2. Check database connections
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -c \
  "SELECT count(*) FROM pg_stat_activity WHERE datname = 'nordabiz';"

# 3. Check for long-running queries
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -c \
  "SELECT pid, now() - query_start AS duration, query
   FROM pg_stat_activity
   WHERE state = 'active' AND now() - query_start > interval '5 seconds';"

# 4. Check system resources
top -n 1
free -h
df -h

# 5. Check Flask logs for slow requests
sudo journalctl -u nordabiznes -n 100 --no-pager | grep -E "slow|timeout|took"

Solution

If database query hanging:

# Identify and kill long-running query
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

# Find problematic query
SELECT pid, query FROM pg_stat_activity
WHERE state = 'active' AND now() - query_start > interval '30 seconds';

# Kill it (replace PID)
SELECT pg_terminate_backend(12345);

If resource exhaustion:

# Restart Flask to clear memory
sudo systemctl restart nordabiznes

# Consider increasing Gunicorn workers (edit systemd service)
sudo nano /etc/systemd/system/nordabiznes.service
# Change: --workers=4 (adjust based on CPU cores)

sudo systemctl daemon-reload
sudo systemctl restart nordabiznes

If external API timeout:

# Check if Gemini API is responsive
curl -I https://generativelanguage.googleapis.com/v1beta/models

# Check PageSpeed API
curl -I https://www.googleapis.com/pagespeedonline/v5/runPagespeed

# Check Brave Search API
curl -I https://api.search.brave.com/res/v1/web/search

Verification

# Test response time
time curl -I https://nordabiznes.pl/health
# Should complete in < 2 seconds

2.4 SSL Certificate Issues

Severity: HIGH

Symptoms

Browser shows "Your connection is not private"
SSL certificate expired or invalid
Mixed content warnings

Diagnosis

# 1. Check certificate expiry
echo | openssl s_client -servername nordabiznes.pl -connect nordabiznes.pl:443 2>/dev/null | \
  openssl x509 -noout -dates

# 2. Check certificate details
curl -vI https://nordabiznes.pl 2>&1 | grep -E "SSL|certificate"

# 3. Check NPM certificate status
ssh maciejpi@10.22.68.250
docker exec nginx-proxy-manager_app_1 \
  sqlite3 /data/database.sqlite \
  "SELECT id, nice_name, expires_on FROM certificate WHERE id = 27;"

Solution

If certificate expired:

# NPM auto-renews Let's Encrypt certificates
# Force renewal via NPM UI or API

# Via UI:
# 1. Access http://10.22.68.250:81
# 2. SSL Certificates → nordabiznes.pl
# 3. Click "Renew" button

# Via CLI (if auto-renewal failed):
ssh maciejpi@10.22.68.250
docker exec nginx-proxy-manager_app_1 \
  node /app/index.js certificate renew 27

If mixed content warnings:

# Check Flask is generating HTTPS URLs
# Verify in templates: url_for(..., _external=True, _scheme='https')

# Check CSP headers in app.py
grep "Content-Security-Policy" /var/www/nordabiznes/app.py

2.5 DNS Resolution Issues

Severity: MEDIUM

Symptoms

nslookup nordabiznes.pl fails
Site accessible by IP but not domain
Inconsistent access from different networks

Diagnosis

# 1. Check external DNS (OVH)
nslookup nordabiznes.pl 8.8.8.8
# Should return: 85.237.177.83

# 2. Check internal DNS (inpi.local)
nslookup nordabiznes.inpi.local 10.22.68.1
# Should return: 10.22.68.249

# 3. Test from different locations
curl -I -H "Host: nordabiznes.pl" http://85.237.177.83/health

# 4. Check Fortigate NAT rules
# Access Fortigate admin panel and verify NAT entry:
# External: 85.237.177.83:443 → Internal: 10.22.68.250:443

Solution

If external DNS issue:

# Check OVH DNS settings
# Login to OVH control panel
# Verify A record: nordabiznes.pl → 85.237.177.83
# Verify A record: www.nordabiznes.pl → 85.237.177.83

If internal DNS issue:

# Update internal DNS server
# This requires access to INPI DNS management (see dns-manager skill)

3. Application & Service Issues

3.1 Application Crash / Won't Start

Severity: CRITICAL

Symptoms

Flask service status shows "failed" or "inactive"
Systemd shows error in logs
Manual start fails with traceback

Diagnosis

# 1. Check service status
ssh maciejpi@10.22.68.249
sudo systemctl status nordabiznes

# 2. Check recent logs
sudo journalctl -u nordabiznes -n 100 --no-pager

# 3. Try manual start for detailed error
cd /var/www/nordabiznes
sudo -u www-data /var/www/nordabiznes/venv/bin/python3 app.py
# Read the traceback carefully

Common Root Causes & Solutions

A. Python Syntax Error

# Symptom: SyntaxError in logs
# Cause: Recent code change introduced syntax error

# Fix: Check syntax
sudo -u www-data /var/www/nordabiznes/venv/bin/python3 -m py_compile app.py

# Rollback if necessary
cd /var/www/nordabiznes
sudo -u www-data git log --oneline -5
sudo -u www-data git revert HEAD  # or specific commit
sudo systemctl restart nordabiznes

B. Missing Environment Variables

# Symptom: KeyError or "SECRET_KEY not found"
# Cause: .env file missing or incomplete

# Fix: Check .env exists and has required variables
sudo -u www-data ls -la /var/www/nordabiznes/.env
sudo -u www-data cat /var/www/nordabiznes/.env | grep -E "^[A-Z_]+=" | wc -l
# Should have ~20 environment variables

# Required variables (add if missing):
# - SECRET_KEY
# - DATABASE_URL
# - GEMINI_API_KEY
# - BRAVE_SEARCH_API_KEY
# - GOOGLE_PAGESPEED_API_KEY
# - ADMIN_EMAIL
# - ADMIN_PASSWORD

C. Database Connection Failed

# Symptom: "could not connect to server" or "FATAL: password authentication failed"
# Cause: PostgreSQL not running or wrong credentials

# Fix: Check PostgreSQL
sudo systemctl status postgresql

# Test connection
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -c "SELECT 1;"

# If password wrong, update .env and restart

D. Missing Python Dependencies

# Symptom: ImportError or ModuleNotFoundError
# Cause: Dependency not installed in venv

# Fix: Reinstall dependencies
cd /var/www/nordabiznes
sudo -u www-data /var/www/nordabiznes/venv/bin/pip install -r requirements.txt

# Verify specific package
sudo -u www-data /var/www/nordabiznes/venv/bin/pip show flask

E. Port 5000 Already in Use

# Symptom: "Address already in use"
# Cause: Another process using port 5000

# Fix: Find and kill process
sudo lsof -i :5000
sudo kill <PID>

# Or restart server if unclear
sudo reboot

Verification

sudo systemctl status nordabiznes
# Should show "active (running)"

curl http://localhost:5000/health
# Should return JSON

3.2 White Screen / Blank Page

Severity: HIGH

Symptoms

Page loads but shows blank white screen
No error message in browser
HTML source is empty or minimal

Diagnosis

# 1. Check browser console (F12)
# Look for JavaScript errors

# 2. Check Flask logs
ssh maciejpi@10.22.68.249
sudo journalctl -u nordabiznes -n 50 --no-pager | grep ERROR

# 3. Check template rendering
curl https://nordabiznes.pl/ -o /tmp/page.html
less /tmp/page.html
# Check if HTML is complete

# 4. Check static assets loading
curl -I https://nordabiznes.pl/static/css/styles.css
# Should return 200

Root Causes & Solutions

A. Template Rendering Error

# Symptom: Jinja2 error in logs
# Cause: Syntax error in template file

# Fix: Check Flask logs for template name
sudo journalctl -u nordabiznes -n 100 | grep -i jinja

# Test template syntax
cd /var/www/nordabiznes
sudo -u www-data /var/www/nordabiznes/venv/bin/python3 -c "
from jinja2 import Template
with open('templates/index.html') as f:
    Template(f.read())
"

B. JavaScript Error

# Symptom: Console shows JS error
# Cause: Syntax error in JavaScript code

# Fix: Check browser console
# Common issues:
# - extra_js block has <script> tags (shouldn't!)
# - undefined variable reference
# - missing semicolon

# Template fix for extra_js:
# WRONG: {% block extra_js %}<script>code</script>{% endblock %}
# RIGHT: {% block extra_js %}code{% endblock %}

C. Database Query Failed

# Symptom: 500 error in network tab
# Cause: Database query error preventing page render

# Fix: Check Flask logs
sudo journalctl -u nordabiznes -n 50 | grep -i "sqlalchemy\|database"

# Check database connectivity
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -c "SELECT 1;"

3.3 Search Not Working

Severity: MEDIUM

Symptoms

Search returns no results for valid queries
Search is very slow (>5 seconds)
Search returns "Database error"

Diagnosis

# 1. Test search endpoint
curl "https://nordabiznes.pl/search?q=test" -v

# 2. Check search_service.py logs
ssh maciejpi@10.22.68.249
sudo journalctl -u nordabiznes -n 100 | grep -i search

# 3. Test database FTS
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

# Test FTS query
SELECT name, ts_rank(search_vector, to_tsquery('polish', 'web')) AS score
FROM companies
WHERE search_vector @@ to_tsquery('polish', 'web')
ORDER BY score DESC LIMIT 5;

# Check pg_trgm extension
SELECT * FROM pg_extension WHERE extname = 'pg_trgm';

Root Causes & Solutions

A. Full-Text Search Index Outdated

# Symptom: Recent companies don't appear in search
# Cause: search_vector not updated

# Fix: Rebuild FTS index
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

UPDATE companies SET search_vector =
  to_tsvector('polish',
    COALESCE(name, '') || ' ' ||
    COALESCE(description, '') || ' ' ||
    COALESCE(array_to_string(services, ' '), '') || ' ' ||
    COALESCE(array_to_string(competencies, ' '), '')
  );

VACUUM ANALYZE companies;

B. Synonym Expansion Not Working

# Symptom: Search for "www" doesn't find "strony internetowe"
# Cause: SYNONYM_EXPANSION dict in search_service.py incomplete

# Fix: Check synonyms
cd /var/www/nordabiznes
grep -A 20 "SYNONYM_EXPANSION" search_service.py

# Add missing synonyms if needed
# Restart service after editing

C. Search Timeout

# Symptom: Search takes >30 seconds
# Cause: Missing database indexes

# Fix: Add indexes
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

CREATE INDEX IF NOT EXISTS idx_companies_search_vector ON companies USING gin(search_vector);
CREATE INDEX IF NOT EXISTS idx_companies_name_trgm ON companies USING gin(name gin_trgm_ops);

VACUUM ANALYZE companies;

Verification

# Test search
curl "https://nordabiznes.pl/search?q=web" | grep -c "company-card"
# Should return number of results found

3.4 AI Chat Not Responding

Severity: MEDIUM

Symptoms

Chat shows "thinking..." forever
Chat returns error message
Empty responses from AI

Root Causes & Solutions

See Section 5.2: Gemini API Issues for detailed troubleshooting.

Quick check:

# 1. Verify Gemini API key
ssh maciejpi@10.22.68.249
sudo -u www-data cat /var/www/nordabiznes/.env | grep GEMINI_API_KEY
# Should not be empty

# 2. Test Gemini API directly
curl -H "x-goog-api-key: YOUR_API_KEY" \
  "https://generativelanguage.googleapis.com/v1beta/models/gemini-3-flash-preview:generateContent" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"parts":[{"text":"Hello"}]}]}'

# 3. Check quota
# Visit: https://console.cloud.google.com/apis/api/generativelanguage.googleapis.com/quotas

4. Database Issues

4.1 Database Connection Failed

Severity: CRITICAL

Symptoms

Flask logs show "could not connect to server"
All database queries fail
500 error on all pages

Diagnosis

# 1. Check PostgreSQL service
ssh maciejpi@10.22.68.249
sudo systemctl status postgresql

# 2. Check PostgreSQL is listening
sudo netstat -tlnp | grep 5432
# Should show: LISTEN on 127.0.0.1:5432

# 3. Check logs
sudo journalctl -u postgresql -n 50

# 4. Test connection
psql -h localhost -U nordabiz_app -d nordabiz -c "SELECT 1;"

Solution

If PostgreSQL is stopped:

sudo systemctl start postgresql
sudo systemctl status postgresql

# If fails to start, check logs
sudo journalctl -u postgresql -n 100 --no-pager

If connection refused:

# Check pg_hba.conf allows local connections
sudo cat /etc/postgresql/*/main/pg_hba.conf | grep "127.0.0.1"
# Should have: host all all 127.0.0.1/32 md5

# Reload if changed
sudo systemctl reload postgresql

If authentication failed:

# Verify user exists
sudo -u postgres psql -c "\du nordabiz_app"

# Reset password if needed
sudo -u postgres psql
ALTER USER nordabiz_app WITH PASSWORD 'NEW_PASSWORD';
\q

# Update .env with new password
sudo nano /var/www/nordabiznes/.env
# Update DATABASE_URL line

# Restart Flask
sudo systemctl restart nordabiznes

Verification

sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -c "SELECT count(*) FROM companies;"
# Should return count

4.2 Database Query Slow

Severity: MEDIUM

Symptoms

Pages load slowly (>5 seconds)
Database queries take long time
High CPU usage on database server

Diagnosis

# 1. Check for slow queries
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active' AND now() - query_start > interval '1 second'
ORDER BY duration DESC;

# 2. Check for missing indexes
SELECT schemaname, tablename, indexname, idx_scan
FROM pg_stat_user_indexes
WHERE idx_scan = 0 AND indexname NOT LIKE '%pkey';

# 3. Check table statistics
SELECT schemaname, tablename, n_live_tup, n_dead_tup,
       last_autovacuum, last_autoanalyze
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY n_live_tup DESC;

# 4. Enable query logging temporarily
ALTER DATABASE nordabiz SET log_min_duration_statement = 1000;
-- Log queries taking > 1 second

Solution

If missing indexes:

# Add appropriate indexes based on queries
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

-- Example: Index on foreign key
CREATE INDEX idx_company_news_company_id ON company_news(company_id);

-- Example: Composite index for common query
CREATE INDEX idx_users_email_active ON users(email, is_active);

-- Rebuild search index
REINDEX INDEX idx_companies_search_vector;

VACUUM ANALYZE;

If high dead tuple ratio:

# Run vacuum
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz
VACUUM ANALYZE;

# For severe cases
VACUUM FULL companies;  -- Locks table!

If table statistics outdated:

sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz
ANALYZE companies;
ANALYZE users;
ANALYZE ai_chat_messages;

Verification

# Check query performance improved
\timing
SELECT * FROM companies WHERE name ILIKE '%test%' LIMIT 10;
# Should complete in < 100ms

4.3 Database Disk Full

Severity: HIGH

Symptoms

PostgreSQL logs show "No space left on device"
INSERT/UPDATE queries fail
Database becomes read-only

Diagnosis

# 1. Check disk usage
ssh maciejpi@10.22.68.249
df -h
# Check /var/lib/postgresql usage

# 2. Check database size
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT pg_size_pretty(pg_database_size('nordabiz'));

SELECT tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename))
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

# 3. Check WAL files
sudo du -sh /var/lib/postgresql/*/main/pg_wal/

Solution

If WAL files accumulating:

# Check WAL settings
sudo -u postgres psql -c "SHOW max_wal_size;"
sudo -u postgres psql -c "SHOW wal_keep_size;"

# Trigger checkpoint
sudo -u postgres psql -c "CHECKPOINT;"

If old backups not cleaned:

# Remove old backups (keep last 7 days)
find /backup/nordabiz/ -name "*.sql" -mtime +7 -delete

If logs too large:

# Truncate old logs
sudo journalctl --vacuum-time=7d

# Rotate PostgreSQL logs
sudo -u postgres pg_archivecleanup /var/lib/postgresql/*/main/pg_wal/ 000000010000000000000001

Emergency: Archive and purge old data:

# Archive old data before deletion
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

-- Example: Archive old AI chat messages (>6 months)
CREATE TABLE ai_chat_messages_archive AS
SELECT * FROM ai_chat_messages
WHERE created_at < NOW() - INTERVAL '6 months';

DELETE FROM ai_chat_messages
WHERE created_at < NOW() - INTERVAL '6 months';

VACUUM FULL ai_chat_messages;

4.4 Database Migration Failed

Severity: HIGH

Symptoms

Migration script returns error
Database schema out of sync with code
Missing tables or columns

Diagnosis

# 1. Check current schema version
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz
\dt
# List all tables

\d companies
# Describe companies table

# 2. Check migration logs
ls -la /var/www/nordabiznes/database/migrations/

# 3. Check Flask-Migrate status (if using Alembic)
cd /var/www/nordabiznes
sudo -u www-data /var/www/nordabiznes/venv/bin/flask db current

Solution

If table missing:

# Re-run migration script
cd /var/www/nordabiznes
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz < database/schema.sql

If column added but missing:

# Add column manually
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

ALTER TABLE companies ADD COLUMN IF NOT EXISTS new_column VARCHAR(255);

-- Grant permissions
GRANT ALL ON TABLE companies TO nordabiz_app;

If migration stuck:

# Rollback last migration
sudo -u www-data /var/www/nordabiznes/venv/bin/flask db downgrade

# Re-apply
sudo -u www-data /var/www/nordabiznes/venv/bin/flask db upgrade

Verification

sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz
\d companies
# Verify schema matches expected structure

5. API Integration Issues

5.1 API Rate Limit Exceeded

Severity: MEDIUM

Symptoms

429 "Too Many Requests" errors
API calls fail with quota exceeded message
Features stop working after heavy usage

Diagnosis

# 1. Check API usage in database
ssh maciejpi@10.22.68.249
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

-- Gemini API usage today
SELECT COUNT(*), SUM(input_tokens), SUM(output_tokens)
FROM ai_api_costs
WHERE DATE(created_at) = CURRENT_DATE;

-- PageSpeed API usage today
SELECT COUNT(*) FROM company_website_analysis
WHERE DATE(created_at) = CURRENT_DATE;

-- Brave Search API usage this month
SELECT COUNT(*) FROM company_news
WHERE DATE(created_at) >= DATE_TRUNC('month', CURRENT_DATE);

# 2. Check rate limiting logs
sudo journalctl -u nordabiznes -n 100 | grep -i "rate limit\|quota\|429"

API Quotas Reference

API	Free Tier Limit	Current Usage Query
Gemini AI	1,500 req/day	`SELECT COUNT(*) FROM ai_api_costs WHERE DATE(created_at) = CURRENT_DATE;`
PageSpeed	25,000 req/day	`SELECT COUNT(*) FROM company_website_analysis WHERE DATE(created_at) = CURRENT_DATE;`
Brave Search	2,000 req/month	`SELECT COUNT(*) FROM company_news WHERE created_at >= DATE_TRUNC('month', CURRENT_DATE);`
Google Places	Limited	Check Google Cloud Console
MS Graph	Per tenant	Check Azure AD logs

Solution

If Gemini quota exceeded:

# Wait until next day (quota resets at midnight UTC)
# OR upgrade to paid tier

# Temporary workaround: Disable AI chat
sudo nano /var/www/nordabiznes/app.py
# Comment out @app.route('/chat') temporarily

sudo systemctl restart nordabiznes

If PageSpeed quota exceeded:

# Stop SEO audit script
pkill -f seo_audit.py

# Wait until next day
# Consider batching audits to stay under quota

If Brave Search quota exceeded:

# Disable news monitoring temporarily
# Wait until next month

# Consider upgrading to paid tier ($5/month for 20k requests)

Prevention

# Add quota monitoring alerts
# Create script: /var/www/nordabiznes/scripts/check_api_quotas.sh

#!/bin/bash
GEMINI_COUNT=$(sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -t -c \
  "SELECT COUNT(*) FROM ai_api_costs WHERE DATE(created_at) = CURRENT_DATE;")

if [ "$GEMINI_COUNT" -gt 1400 ]; then
  echo "WARNING: Gemini API usage at $GEMINI_COUNT / 1500"
  # Send alert email
fi

# Add to crontab: run hourly
# 0 * * * * /var/www/nordabiznes/scripts/check_api_quotas.sh

5.2 Gemini API Issues

Severity: MEDIUM

Symptoms

AI chat returns empty responses
"Safety filter blocked response" error
Gemini API timeout
"Conversation not found" error

Diagnosis

# 1. Test Gemini API directly
GEMINI_KEY=$(sudo -u www-data grep GEMINI_API_KEY /var/www/nordabiznes/.env | cut -d= -f2)

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-3-flash-preview:generateContent" \
  -H "x-goog-api-key: $GEMINI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"parts":[{"text":"Hello, test"}]}]}'

# 2. Check Flask logs for Gemini errors
ssh maciejpi@10.22.68.249
sudo journalctl -u nordabiznes -n 100 | grep -i gemini

# 3. Check conversation ownership
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT id, user_id, created_at
FROM ai_chat_conversations
WHERE id = 123;  -- Replace with conversation ID

Common Issues & Solutions

A. Empty AI Responses

# Cause: Safety filters blocking response
# OR context too long

# Check last message for safety filter
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT message, ai_response, error_message
FROM ai_chat_messages
ORDER BY created_at DESC
LIMIT 5;

# If error_message contains "safety" or "blocked":
# - Rephrase user query to be less controversial
# - No technical fix needed, it's Gemini's safety system

B. Conversation Not Found

# Cause: User trying to access someone else's conversation

# Verify conversation ownership
SELECT c.id, c.user_id, u.email
FROM ai_chat_conversations c
JOIN users u ON c.user_id = u.id
WHERE c.id = 123;  -- Replace ID

# Fix: Ensure frontend passes correct conversation_id
# OR create new conversation for user

C. Token Limit Exceeded

# Cause: Conversation history too long (>200k tokens)

# Check token usage
SELECT id, input_tokens, output_tokens,
       input_tokens + output_tokens AS total_tokens
FROM ai_chat_messages
WHERE conversation_id = 123
ORDER BY created_at DESC;

# Fix: Trim old messages
DELETE FROM ai_chat_messages
WHERE conversation_id = 123
AND created_at < (
  SELECT created_at
  FROM ai_chat_messages
  WHERE conversation_id = 123
  ORDER BY created_at DESC
  LIMIT 1 OFFSET 10
);

D. API Key Invalid

# Symptom: 401 Unauthorized or 403 Forbidden

# Verify API key
sudo -u www-data grep GEMINI_API_KEY /var/www/nordabiznes/.env

# Test key directly
curl -H "x-goog-api-key: YOUR_KEY" \
  "https://generativelanguage.googleapis.com/v1beta/models"

# If invalid, regenerate key in Google Cloud Console
# https://console.cloud.google.com/apis/credentials

Verification

# Test AI chat endpoint
curl -X POST https://nordabiznes.pl/api/chat \
  -H "Content-Type: application/json" \
  -d '{"conversation_id":123,"message":"test"}' \
  -b "session=YOUR_SESSION_COOKIE"

# Should return JSON with AI response

5.3 PageSpeed API Issues

Severity: LOW

Symptoms

SEO audit fails with API error
PageSpeed scores show as null/0
Timeout errors in audit script

Diagnosis

# 1. Test PageSpeed API directly
PAGESPEED_KEY=$(sudo -u www-data grep GOOGLE_PAGESPEED_API_KEY /var/www/nordabiznes/.env | cut -d= -f2)

curl "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://nordabiznes.pl&key=$PAGESPEED_KEY"

# 2. Check audit logs
ssh maciejpi@10.22.68.249
sudo journalctl -u nordabiznes -n 100 | grep -i pagespeed

# 3. Check recent audits
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT company_id, url, seo_score, performance_score,
       audited_at, error_message
FROM company_website_analysis
ORDER BY audited_at DESC
LIMIT 10;

Solution

If API key invalid:

# Regenerate key in Google Cloud Console
# https://console.cloud.google.com/apis/credentials?project=gen-lang-client-0540794446

# Update .env
sudo nano /var/www/nordabiznes/.env
# GOOGLE_PAGESPEED_API_KEY=NEW_KEY

sudo systemctl restart nordabiznes

If quota exceeded:

# Wait until next day (25k/day limit)

# Check usage
# https://console.cloud.google.com/apis/api/pagespeedonline.googleapis.com/quotas

If timeout:

# Increase timeout in seo_audit.py
sudo nano /var/www/nordabiznes/scripts/seo_audit.py

# Find: timeout=30
# Change to: timeout=60

# Or run audits in smaller batches
python seo_audit.py --batch 1-10
# Wait 5 minutes between batches

5.4 Brave Search API Issues

Severity: LOW

Symptoms

News monitoring returns no results
Brave API 429 error
Invalid search results

Diagnosis

# 1. Test Brave API directly
BRAVE_KEY=$(sudo -u www-data grep BRAVE_SEARCH_API_KEY /var/www/nordabiznes/.env | cut -d= -f2)

curl -H "X-Subscription-Token: $BRAVE_KEY" \
  "https://api.search.brave.com/res/v1/news/search?q=test&count=5"

# 2. Check usage this month
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT COUNT(*) AS searches_this_month
FROM company_news
WHERE created_at >= DATE_TRUNC('month', CURRENT_DATE);
-- Free tier: 2,000/month

# 3. Check for error logs
sudo journalctl -u nordabiznes -n 100 | grep -i brave

Solution

If quota exceeded (2000/month):

# Wait until next month
# OR upgrade to paid tier

# Temporary: Disable news monitoring
# Comment out news fetch cron job

If API key invalid:

# Get new key from https://brave.com/search/api/

# Update .env
sudo nano /var/www/nordabiznes/.env
# BRAVE_SEARCH_API_KEY=NEW_KEY

sudo systemctl restart nordabiznes

6. Authentication & Security Issues

Severity: MEDIUM

Symptoms

"Invalid credentials" despite correct password
Redirected to login immediately after logging in
Session expires too quickly
"CSRF token missing" error

Diagnosis

# 1. Check user exists and is active
ssh maciejpi@10.22.68.249
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT id, email, is_active, email_verified, failed_login_attempts
FROM users
WHERE email = 'user@example.com';

# 2. Check session configuration
grep -E "SECRET_KEY|PERMANENT_SESSION_LIFETIME" /var/www/nordabiznes/app.py

# 3. Check Flask logs for auth errors
sudo journalctl -u nordabiznes -n 100 | grep -i "login\|session\|auth"

# 4. Test from server (bypass network)
curl -c /tmp/cookies.txt -X POST http://localhost:5000/login \
  -d "email=test@nordabiznes.pl&password=TEST_PASSWORD"

Common Issues & Solutions

A. Account Locked (Failed Login Attempts)

# Check failed attempts
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT email, failed_login_attempts, last_failed_login
FROM users
WHERE email = 'user@example.com';

# If >= 5 attempts, reset:
UPDATE users SET failed_login_attempts = 0
WHERE email = 'user@example.com';

B. Email Not Verified

# Check verification status
SELECT email, email_verified, verification_token, verification_token_expiry
FROM users
WHERE email = 'user@example.com';

# Force verify (for testing)
UPDATE users SET email_verified = TRUE
WHERE email = 'user@example.com';

C. Session Cookie Not Persisting

# Check cookie settings in app.py
grep -A 5 "SESSION_COOKIE" /var/www/nordabiznes/app.py

# Should have:
# SESSION_COOKIE_SECURE = True  # HTTPS only
# SESSION_COOKIE_HTTPONLY = True  # No JS access
# SESSION_COOKIE_SAMESITE = 'Lax'  # CSRF protection

# If accessing via HTTP (not HTTPS), session won't work
# Ensure using https://nordabiznes.pl not http://

D. CSRF Token Mismatch

# Symptom: "400 Bad Request - CSRF token missing"

# Cause: Form submitted without CSRF token

# Fix: Ensure all forms have:
# {{ form.hidden_tag() }}  # WTForms
# OR
# <input type="hidden" name="csrf_token" value="{{ csrf_token() }}">

# Check template
grep -r "csrf_token" /var/www/nordabiznes/templates/login.html

E. Password Hash Algorithm Changed

# Symptom: Old users can't login after upgrade

# Check hash format
SELECT id, email, SUBSTRING(password_hash, 1, 20)
FROM users
WHERE email = 'user@example.com';

# Should start with: pbkdf2:sha256:

# If different, user needs password reset
# Send reset email via /forgot-password

Verification

# Test login flow
curl -c /tmp/cookies.txt -X POST http://localhost:5000/login \
  -d "email=test@nordabiznes.pl&password=TEST_PASSWORD" \
  -L -v

# Should see: Set-Cookie: session=...
# Should redirect to /dashboard

6.2 Unauthorized Access / Permission Denied

Severity: HIGH

Symptoms

"403 Forbidden" error
User can access pages they shouldn't
Admin panel not accessible

Diagnosis

# 1. Check user role
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT id, email, is_admin, is_norda_member
FROM users
WHERE email = 'user@example.com';

# 2. Check route decorators
grep -B 2 "@app.route('/admin" /var/www/nordabiznes/app.py

# Should have: @login_required and @admin_required

# 3. Check Flask logs
sudo journalctl -u nordabiznes -n 50 | grep -i "forbidden\|unauthorized"

Solution

If user should be admin:

# Grant admin role
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

UPDATE users SET is_admin = TRUE
WHERE email = 'admin@nordabiznes.pl';

If authorization check broken:

# Check app.py decorators

# Should have:
@app.route('/admin/users')
@login_required
@admin_required
def admin_users():
    ...

# Verify @admin_required is defined:
grep -A 5 "def admin_required" /var/www/nordabiznes/app.py

If company ownership check failed:

# Verify company-user association
SELECT c.id, c.name, u.email
FROM companies c
LEFT JOIN users u ON c.id = u.company_id
WHERE c.slug = 'company-slug';

# Update user's company
UPDATE users SET company_id = 123
WHERE email = 'user@example.com';

6.3 Password Reset Not Working

Severity: MEDIUM

Symptoms

Password reset email not received
Reset token expired or invalid
"Invalid token" error

Diagnosis

# 1. Check user reset token
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT email, reset_token, reset_token_expiry
FROM users
WHERE email = 'user@example.com';

# 2. Check email service logs
sudo journalctl -u nordabiznes -n 100 | grep -i "email\|smtp"

# 3. Test MS Graph API (email service)
# Check if AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID set
sudo -u www-data grep AZURE /var/www/nordabiznes/.env

Solution

If token expired:

# Tokens expire after 1 hour

# Generate new token via /forgot-password
# OR manually extend expiry:
UPDATE users SET reset_token_expiry = NOW() + INTERVAL '1 hour'
WHERE email = 'user@example.com';

If email not sent:

# Check MS Graph credentials
sudo -u www-data python3 << 'EOF'
import os
from email_service import EmailService

service = EmailService()
result = service.send_email(
    to_email="test@example.com",
    subject="Test",
    body="Test email"
)
print(result)
EOF

# If fails, check Azure AD app registration
# Ensure "Mail.Send" permission granted

Manual password reset (emergency):

# Generate new password hash
sudo -u www-data python3 << 'EOF'
from werkzeug.security import generate_password_hash
password = "NewPassword123"
print(generate_password_hash(password))
EOF

# Update database
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

UPDATE users SET password_hash = 'HASH_FROM_ABOVE'
WHERE email = 'user@example.com';

7. Performance Issues

7.1 Slow Page Load Times

Severity: MEDIUM

Symptoms

Pages take >5 seconds to load
TTFB (Time to First Byte) is high
Browser shows "waiting for nordabiznes.pl..."

Diagnosis

# 1. Measure response time
time curl -I https://nordabiznes.pl/

# 2. Check Gunicorn worker status
ssh maciejpi@10.22.68.249
ps aux | grep gunicorn
# Look for: worker processes (should be 4-8)

# 3. Check server load
top -n 1
# Look at: CPU usage, memory usage, load average

# 4. Check database query times
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT calls, mean_exec_time, query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
-- If pg_stat_statements not enabled, see solution below

# 5. Profile Flask app
sudo journalctl -u nordabiznes -n 100 | grep -E "took|slow|timeout"

Root Causes & Solutions

A. Too Few Gunicorn Workers

# Current workers
ps aux | grep gunicorn | grep -v grep | wc -l

# Recommended: (2 x CPU cores) + 1
# For 4 core VM: 9 workers

# Update systemd service
sudo nano /etc/systemd/system/nordabiznes.service

# Change:
ExecStart=/var/www/nordabiznes/venv/bin/gunicorn --workers=9 \
  --bind 0.0.0.0:5000 --timeout 120 app:app

sudo systemctl daemon-reload
sudo systemctl restart nordabiznes

B. Slow Database Queries

# Enable query stats (if not enabled)
sudo -u postgres psql
ALTER SYSTEM SET shared_preload_libraries = 'pg_stat_statements';
# Restart PostgreSQL
sudo systemctl restart postgresql

# Check slow queries
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT calls, mean_exec_time, query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;

# Add indexes for slow queries (see Section 4.2)

C. External API Timeouts

# Check for API timeout logs
sudo journalctl -u nordabiznes -n 200 | grep -i timeout

# Common culprits:
# - Gemini API (text generation)
# - PageSpeed API (site audit)
# - Brave Search API

# Solution: Add caching

# Example: Cache PageSpeed results for 24 hours
# app.py modification (pseudocode):
# if last_audit < 24h ago:
#     return cached_result
# else:
#     fetch new audit

D. Missing Static Asset Caching

# Check cache headers
curl -I https://nordabiznes.pl/static/css/styles.css | grep -i cache

# Should have: Cache-Control: max-age=31536000

# If missing, add to NPM proxy or app.py:
@app.after_request
def add_cache_header(response):
    if request.path.startswith('/static/'):
        response.cache_control.max_age = 31536000
    return response

E. Large Database Result Sets

# Check for N+1 queries or loading too much data

# Example bad query:
# for company in Company.query.all():  # Loads ALL companies!
#     print(company.name)

# Fix: Add pagination
# companies = Company.query.paginate(page=1, per_page=20)

Verification

# Test response time
for i in {1..5}; do
  time curl -s -o /dev/null https://nordabiznes.pl/
done

# Should average < 500ms

7.2 High Memory Usage

Severity: MEDIUM

Symptoms

Server OOM (Out of Memory) errors
Swapping active (slow performance)
Gunicorn workers killed by OOM killer

Diagnosis

# 1. Check memory usage
ssh maciejpi@10.22.68.249
free -h

# 2. Check which process using memory
ps aux --sort=-%mem | head -10

# 3. Check for memory leaks
# Monitor over time:
watch -n 5 'ps aux | grep gunicorn | awk "{sum+=\$6} END {print sum/1024 \" MB\"}"'

# 4. Check OOM killer logs
sudo dmesg | grep -i "out of memory\|oom"

Solution

If Gunicorn workers too many:

# Reduce workers
sudo nano /etc/systemd/system/nordabiznes.service

# Change: --workers=9 to --workers=4

sudo systemctl daemon-reload
sudo systemctl restart nordabiznes

If memory leak in application:

# Restart workers periodically
sudo nano /etc/systemd/system/nordabiznes.service

# Add: --max-requests=1000 --max-requests-jitter=100
# This restarts workers after 1000 requests

sudo systemctl daemon-reload
sudo systemctl restart nordabiznes

If PostgreSQL using too much memory:

# Check PostgreSQL memory settings
sudo -u postgres psql -c "SHOW shared_buffers;"
sudo -u postgres psql -c "SHOW work_mem;"

# Reduce if necessary
sudo nano /etc/postgresql/*/main/postgresql.conf

# shared_buffers = 256MB  # Was 512MB
# work_mem = 4MB          # Was 16MB

sudo systemctl restart postgresql

If server needs more RAM:

# Increase VM RAM in Proxmox
# OR add swap space

# Add 2GB swap
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Make permanent
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

7.3 High CPU Usage

Severity: MEDIUM

Symptoms

CPU at 100% constantly
Server load average > number of cores
Slow response times

Diagnosis

# 1. Check CPU usage
ssh maciejpi@10.22.68.249
top -n 1

# Look for processes using >80% CPU

# 2. Check load average
uptime
# Load should be < number of CPU cores

# 3. Identify CPU-heavy queries
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT pid, now() - query_start AS duration, state, query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY now() - query_start DESC;

Solution

If database query CPU-intensive:

# Kill long-running query
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz
SELECT pg_terminate_backend(PID);

# Add index to optimize query
# See Section 4.2

If AI chat overwhelming CPU:

# Add rate limiting to chat endpoint
# app.py modification:
from flask_limiter import Limiter

limiter = Limiter(app, key_func=lambda: current_user.id)

@app.route('/api/chat', methods=['POST'])
@limiter.limit("10 per minute")  # Add this
def chat_api():
    ...

If search causing high CPU:

# Optimize search query
# Use indexes instead of ILIKE
# Cache search results

# Add to app.py:
from functools import lru_cache

@lru_cache(maxsize=100)
def search_companies_cached(query):
    return search_companies(db, query)

8. Monitoring & Diagnostics

8.1 Health Check Endpoints

# Application health
curl https://nordabiznes.pl/health

# Expected response:
{
  "status": "healthy",
  "database": "connected",
  "timestamp": "2026-01-10T12:00:00Z"
}

# Database health
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -c "SELECT 1;"

# NPM health (from proxy server)
ssh maciejpi@10.22.68.250
docker ps | grep nginx-proxy-manager
# Should show: Up X hours

# Flask service health
ssh maciejpi@10.22.68.249
sudo systemctl status nordabiznes
# Should show: active (running)

8.2 Log Locations

# Flask application logs
sudo journalctl -u nordabiznes -n 100 --no-pager

# Follow live logs
sudo journalctl -u nordabiznes -f

# PostgreSQL logs
sudo journalctl -u postgresql -n 50

# NPM logs
ssh maciejpi@10.22.68.250
docker logs nginx-proxy-manager_app_1 --tail 50 -f

# System logs
sudo journalctl -n 100

# Nginx access logs (on backend)
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

8.3 Performance Metrics

# Response time monitoring
# Create script: /usr/local/bin/check_nordabiz_performance.sh

#!/bin/bash
RESPONSE_TIME=$(curl -w '%{time_total}\n' -o /dev/null -s https://nordabiznes.pl/health)
echo "Response time: ${RESPONSE_TIME}s"

if (( $(echo "$RESPONSE_TIME > 2" | bc -l) )); then
  echo "WARNING: Slow response time!"
fi

# Add to cron: */5 * * * * /usr/local/bin/check_nordabiz_performance.sh

# Database performance
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

-- Connection count
SELECT count(*) FROM pg_stat_activity;

-- Active queries
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

-- Cache hit ratio (should be > 99%)
SELECT
  sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) * 100 AS cache_hit_ratio
FROM pg_statio_user_tables;

-- Table sizes
SELECT schemaname, tablename,
       pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

8.4 Database Backup Verification

# Check last backup
ssh maciejpi@10.22.68.249
ls -lah /backup/nordabiz/ | head -10

# Expected: Daily backups (.sql files)

# Test restore (to test database)
sudo -u postgres createdb nordabiz_test
sudo -u postgres psql nordabiz_test < /backup/nordabiz/nordabiz_YYYY-MM-DD.sql

# Verify restore
sudo -u postgres psql nordabiz_test -c "SELECT count(*) FROM companies;"

# Cleanup
sudo -u postgres dropdb nordabiz_test

9. Emergency Procedures

9.1 Complete Service Outage

Severity: CRITICAL

Immediate Actions (First 5 Minutes)

# 1. Verify outage scope
curl -I https://nordabiznes.pl/health
# If fails, proceed

# 2. Check from internal network
ssh maciejpi@10.22.68.249
curl -I http://localhost:5000/health
# If this works → Network/NPM issue
# If this fails → Application issue

# 3. Notify stakeholders
# Send email/message: "nordabiznes.pl experiencing outage, investigating"

# 4. Check service status
sudo systemctl status nordabiznes
sudo systemctl status postgresql

If Network/NPM Issue

# 1. Verify NPM is running
ssh maciejpi@10.22.68.250
docker ps | grep nginx-proxy-manager

# If not running:
docker start nginx-proxy-manager_app_1

# 2. Check NPM configuration
docker exec nginx-proxy-manager_app_1 \
  sqlite3 /data/database.sqlite \
  "SELECT id, forward_host, forward_port FROM proxy_host WHERE id = 27;"

# Must show: 27|10.22.68.249|5000

# 3. Check Fortigate NAT
# Access Fortigate admin panel
# Verify: 85.237.177.83:443 → 10.22.68.250:443

If Application Issue

# 1. Check Flask service
ssh maciejpi@10.22.68.249
sudo systemctl status nordabiznes

# If failed, check logs
sudo journalctl -u nordabiznes -n 50

# 2. Try restart
sudo systemctl restart nordabiznes

# If restart fails, check manually
cd /var/www/nordabiznes
sudo -u www-data /var/www/nordabiznes/venv/bin/python3 app.py
# Read error message

# 3. Common quick fixes:
# - Syntax error: git revert last commit
# - Database down: sudo systemctl start postgresql
# - Port conflict: sudo lsof -i :5000 && kill PID

If Database Issue

# 1. Check PostgreSQL
sudo systemctl status postgresql

# If stopped:
sudo systemctl start postgresql

# If start fails:
sudo journalctl -u postgresql -n 50

# 2. Check disk space
df -h
# If full, clean old backups/logs (see Section 4.3)

# 3. Emergency: Restore from backup
sudo systemctl stop nordabiznes
sudo -u postgres dropdb nordabiz
sudo -u postgres createdb nordabiz
sudo -u postgres psql nordabiz < /backup/nordabiz/latest.sql
sudo systemctl start nordabiznes

9.2 Data Loss / Corruption

Severity: CRITICAL

Immediate Actions

# 1. STOP the application immediately
sudo systemctl stop nordabiznes

# 2. Create emergency backup of current state
sudo -u postgres pg_dump nordabiz > /tmp/nordabiz_emergency_$(date +%Y%m%d_%H%M%S).sql

# 3. Assess damage
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

-- Check table counts
SELECT 'companies' AS table, count(*) FROM companies
UNION ALL
SELECT 'users', count(*) FROM users
UNION ALL
SELECT 'ai_chat_conversations', count(*) FROM ai_chat_conversations;

-- Compare with expected counts (should have ~80 companies, etc.)

Recovery Procedures

If recent corruption (< 24 hours ago):

# Restore from last night's backup
sudo systemctl stop nordabiznes

sudo -u postgres dropdb nordabiz
sudo -u postgres createdb nordabiz
sudo -u postgres psql nordabiz < /backup/nordabiz/nordabiz_$(date -d yesterday +%Y-%m-%d).sql

# Re-grant permissions
sudo -u postgres psql nordabiz << 'EOF'
GRANT ALL PRIVILEGES ON DATABASE nordabiz TO nordabiz_app;
GRANT ALL ON ALL TABLES IN SCHEMA public TO nordabiz_app;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO nordabiz_app;
EOF

sudo systemctl start nordabiznes

If partial data loss:

# Identify missing/corrupted records
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

-- Example: Find companies with NULL required fields
SELECT id, slug, name FROM companies WHERE name IS NULL;

-- Restore specific table from backup
# Extract table from backup
pg_restore -t companies -d nordabiz /backup/nordabiz/latest.sql

9.3 Security Breach

Severity: CRITICAL

Immediate Actions (First 10 Minutes)

# 1. ISOLATE the server
ssh maciejpi@10.22.68.249

# Block all incoming traffic except your IP
sudo iptables -A INPUT -s YOUR_IP -j ACCEPT
sudo iptables -A INPUT -j DROP

# 2. Create forensic copy
sudo -u postgres pg_dump nordabiz > /tmp/forensic_$(date +%Y%m%d_%H%M%S).sql
sudo tar czf /tmp/www_forensic.tar.gz /var/www/nordabiznes/

# 3. Check for unauthorized access
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

-- Check for new admin users
SELECT id, email, created_at, is_admin
FROM users
WHERE is_admin = TRUE
ORDER BY created_at DESC;

-- Check for recent logins
SELECT user_id, ip_address, created_at
FROM user_login_history
WHERE created_at > NOW() - INTERVAL '24 hours'
ORDER BY created_at DESC;

# 4. Check logs for suspicious activity
sudo journalctl -u nordabiznes --since "24 hours ago" | grep -iE "admin|delete|drop|unauthorized"

# 5. Notify stakeholders
# Email: "Security incident detected on nordabiznes.pl, investigating"

Investigation

# Check for SQL injection attempts
sudo journalctl -u nordabiznes --since "7 days ago" | grep -i "UNION\|DROP\|;--"

# Check for unauthorized file changes
sudo find /var/www/nordabiznes/ -type f -mtime -1 -ls

# Check for backdoors
sudo grep -r "eval\|exec\|system\|subprocess" /var/www/nordabiznes/*.py

# Check database for malicious data
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz

SELECT * FROM users WHERE email LIKE '%<script%';
SELECT * FROM companies WHERE description LIKE '%<script%';

Remediation

# 1. Change all passwords
# - Database passwords
# - Admin user passwords
# - API keys

# 2. Revoke compromised sessions
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz
DELETE FROM flask_sessions;  -- Force all users to re-login

# 3. Update all API keys
sudo nano /var/www/nordabiznes/.env
# Regenerate all keys in respective consoles

# 4. Patch vulnerability
# Based on investigation findings

# 5. Restore normal operation
sudo iptables -F  # Clear firewall rules
sudo systemctl restart nordabiznes

10. Diagnostic Commands Reference

10.1 Quick Health Checks

# Complete health check (run all at once)
echo "=== Application Health ===" && \
curl -I https://nordabiznes.pl/health && \
echo -e "\n=== Service Status ===" && \
ssh maciejpi@10.22.68.249 "sudo systemctl status nordabiznes --no-pager | head -5" && \
echo -e "\n=== Database Connection ===" && \
ssh maciejpi@10.22.68.249 "sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz -c 'SELECT count(*) FROM companies;'" && \
echo -e "\n=== Server Load ===" && \
ssh maciejpi@10.22.68.249 "uptime"

10.2 NPM Proxy Diagnostics

# NPM configuration check
ssh maciejpi@10.22.68.250 "docker exec nginx-proxy-manager_app_1 \
  sqlite3 /data/database.sqlite \
  \"SELECT id, domain_names, forward_host, forward_port, enabled FROM proxy_host WHERE id = 27;\""

# NPM logs (live)
ssh maciejpi@10.22.68.250 "docker logs nginx-proxy-manager_app_1 --tail 20 -f"

# NPM container status
ssh maciejpi@10.22.68.250 "docker ps | grep nginx-proxy-manager"

# Test backend from NPM server
ssh maciejpi@10.22.68.250 "curl -I http://10.22.68.249:5000/health"

10.3 Database Diagnostics

# Database quick stats
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz << 'EOF'
SELECT 'Companies' AS metric, count(*) AS value FROM companies
UNION ALL SELECT 'Users', count(*) FROM users
UNION ALL SELECT 'Active sessions', count(*) FROM pg_stat_activity
UNION ALL SELECT 'DB size (MB)', pg_database_size('nordabiz')/1024/1024;
EOF

# Find slow queries
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz << 'EOF'
SELECT pid, now() - query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active' AND now() - query_start > interval '2 seconds'
ORDER BY duration DESC;
EOF

# Check locks
sudo -u www-data psql -h localhost -U nordabiz_app -d nordabiz << 'EOF'
SELECT relation::regclass, mode, granted
FROM pg_locks
WHERE NOT granted;
EOF

10.4 Performance Diagnostics

# Response time test (10 requests)
for i in {1..10}; do
  curl -w "Request $i: %{time_total}s\n" -o /dev/null -s https://nordabiznes.pl/
done

# Server resource usage
ssh maciejpi@10.22.68.249 "top -b -n 1 | head -20"

# Disk usage
ssh maciejpi@10.22.68.249 "df -h && echo -e '\n=== Top 10 Directories ===\n' && du -sh /* 2>/dev/null | sort -rh | head -10"

# Network connectivity
ping -c 5 nordabiznes.pl
traceroute nordabiznes.pl

# SSL certificate check
echo | openssl s_client -servername nordabiznes.pl -connect nordabiznes.pl:443 2>/dev/null | openssl x509 -noout -dates -subject

10.5 API Integration Diagnostics

# Test all external APIs
ssh maciejpi@10.22.68.249

# Gemini API
GEMINI_KEY=$(sudo -u www-data grep GEMINI_API_KEY .env | cut -d= -f2)
curl -s -H "x-goog-api-key: $GEMINI_KEY" \
  "https://generativelanguage.googleapis.com/v1beta/models" | jq '.models[0].name'

# PageSpeed API
PAGESPEED_KEY=$(sudo -u www-data grep GOOGLE_PAGESPEED_API_KEY .env | cut -d= -f2)
curl -s "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://nordabiznes.pl&key=$PAGESPEED_KEY" | jq '.lighthouseResult.categories.performance.score'

# Brave Search API
BRAVE_KEY=$(sudo -u www-data grep BRAVE_SEARCH_API_KEY .env | cut -d= -f2)
curl -s -H "X-Subscription-Token: $BRAVE_KEY" \
  "https://api.search.brave.com/res/v1/web/search?q=test&count=1" | jq '.web.results[0].title'

# KRS API
curl -s "https://api-krs.ms.gov.pl/api/krs/OdpisAktualny/0000878913" | jq '.odpis.dane.dzial1'

10.6 Git & Deployment Diagnostics

# Check current deployment version
ssh maciejpi@10.22.68.249 "cd /var/www/nordabiznes && git log --oneline -5"

# Check for uncommitted changes
ssh maciejpi@10.22.68.249 "cd /var/www/nordabiznes && git status"

# Check remote sync
ssh maciejpi@10.22.68.249 "cd /var/www/nordabiznes && git remote -v && git fetch && git status"

# Verify file permissions
ssh maciejpi@10.22.68.249 "ls -la /var/www/nordabiznes/ | head -10"

System Architecture: 01-system-context.md
Container Diagram: 02-container-diagram.md
Deployment Architecture: 03-deployment-architecture.md
Network Topology: 07-network-topology.md
Critical Configurations: 08-critical-configurations.md
Security Architecture: 09-security-architecture.md
API Endpoints: 10-api-endpoints.md
HTTP Request Flow: flows/06-http-request-flow.md
Authentication Flow: flows/01-authentication-flow.md
Incident Report: ../../INCIDENT_REPORT_20260102.md

Document Status: ✅ Complete Version: 1.0 Last Review: 2026-01-10

Maintenance Notes

When to Update This Guide:

After any production incident → Add to relevant section
When new features added → Add new troubleshooting scenarios
When infrastructure changes → Update diagnostic commands
Monthly review → Verify commands still work
After major version upgrades → Test all procedures

Contribution Guidelines:

Keep solutions actionable (copy-paste commands when possible)
Include expected output for diagnostic commands
Reference related architecture docs
Test all commands before adding
Use consistent formatting (bash code blocks)

End of Troubleshooting Guide

59 KiB Raw Blame History

11. Troubleshooting Guide

Table of Contents

1. Quick Reference

1.1 Emergency Contacts

1.2 Critical Services Status Check

1.3 Issue Decision Tree

1.4 Severity Levels

2. Infrastructure & Network Issues

2.1 ERR_TOO_MANY_REDIRECTS

Symptoms

Root Cause

Diagnosis

Solution

Verification

Prevention

2.2 502 Bad Gateway

Symptoms

Root Causes

Diagnosis

Solution

Verification

2.3 504 Gateway Timeout

Symptoms

Root Causes

Diagnosis

Solution

Verification

2.4 SSL Certificate Issues

Symptoms

Diagnosis

Solution

2.5 DNS Resolution Issues

Symptoms

Diagnosis

Solution

3. Application & Service Issues

3.1 Application Crash / Won't Start

Symptoms

Diagnosis

Common Root Causes & Solutions

Verification

3.2 White Screen / Blank Page

Symptoms

Diagnosis

Root Causes & Solutions

3.3 Search Not Working

Symptoms

Diagnosis

Root Causes & Solutions

Verification

3.4 AI Chat Not Responding

Symptoms

Root Causes & Solutions

4. Database Issues

4.1 Database Connection Failed

Symptoms

Diagnosis

Solution

Verification

4.2 Database Query Slow

Symptoms

Diagnosis

Solution

Verification

4.3 Database Disk Full

Symptoms

Diagnosis

Solution

4.4 Database Migration Failed

Symptoms

Diagnosis

Solution

Verification

5. API Integration Issues

5.1 API Rate Limit Exceeded

Symptoms

Diagnosis

API Quotas Reference

Solution

59 KiB

Raw Blame History