Add header testing scripts for debugging API and security headers

README_headers_testing.md

@@ -0,0 +1,97 @@
+# API Headers Testing Scripts
+
+This directory contains scripts for testing the security headers and API compatibility of the Ploughshares application.
+
+## Scripts Overview
+
+1. **test_headers.sh** - Tests general HTTP headers for both UI and API endpoints
+2. **test_api_curl.sh** - Tests API endpoints with various curl commands and parameters
+3. **test_api_headers.sh** - Specifically analyzes security headers for API compatibility
+
+## Usage
+
+All scripts accept an optional URL parameter. If not provided, they default to `http://localhost:5005`.
+
+```bash
+# Test against local development environment
+./test_headers.sh
+
+# Test against production
+./test_headers.sh https://ploughshares.nixc.us
+```
+
+## Security Headers Analysis
+
+The scripts check for the following important headers:
+
+### Content Security Policy (CSP)
+- Ensures `unsafe-inline` and `unsafe-eval` are present for API compatibility
+- Verifies `connect-src` includes wildcard (*) for API access
+- Checks critical protections like `object-src: 'none'` and `frame-ancestors: 'none'`
+
+### CORS Headers
+- `Access-Control-Allow-Origin`
+- `Access-Control-Allow-Methods`
+- `Access-Control-Allow-Headers`
+
+### Resource Policies
+- `Cross-Origin-Resource-Policy` should be `cross-origin` for API endpoints
+- `Cross-Origin-Resource-Policy` should be `same-origin` for UI endpoints
+
+## Troubleshooting
+
+If API requests are failing, check the following:
+
+1. **CSP Issues**: The Content-Security-Policy header must include:
+   - `'unsafe-inline'` and `'unsafe-eval'` in script-src
+   - `*` in connect-src
+   - `data:` and `blob:` in appropriate directives
+
+2. **CORS Issues**: API endpoints must have:
+   - `Access-Control-Allow-Origin: *` (or specific origins)
+   - `Access-Control-Allow-Methods` with appropriate HTTP methods
+   - `Access-Control-Allow-Headers` with at least `Content-Type` and `Authorization`
+
+3. **Resource Policy Issues**: API endpoints should have:
+   - `Cross-Origin-Resource-Policy: cross-origin`
+
+## Current Configuration
+
+The current configuration uses a permissive CSP that works for both UI and API routes:
+
+```python
+csp = {
+    'default-src': ["'self'", "'unsafe-inline'", "'unsafe-eval'", "data:", "blob:"],
+    'script-src': ["'self'", "'unsafe-inline'", "'unsafe-eval'"] + ([f"'sha256-{CSP_JS_HASH}'"] if CSP_JS_HASH else []),
+    'style-src': ["'self'", "'unsafe-inline'"] + 
+                ([f"'sha256-{CSP_CSS_HASH}'"] if CSP_CSS_HASH else []) + 
+                ([f"'sha256-{CSP_CUSTOM_CSS_HASH}'"] if CSP_CUSTOM_CSS_HASH else []),
+    'img-src': ["'self'", "data:", "blob:"],
+    'font-src': ["'self'", "data:"],
+    'connect-src': ["'self'", "*"],
+    'manifest-src': "'self'",
+    'object-src': "'none'",  # Still explicitly disallow objects
+    'frame-ancestors': "'none'",  # Still prevent framing
+    'base-uri': "'self'",
+    'form-action': "'self'"
+}
+```
+
+CORS headers are added for API routes in an after_request handler:
+
+```python
+@app.after_request
+def add_api_headers(response):
+    if request.path.startswith('/api/'):
+        # Add CORS headers for API routes
+        response.headers['Access-Control-Allow-Origin'] = '*'
+        response.headers['Access-Control-Allow-Methods'] = 'GET, POST, PUT, DELETE, OPTIONS'
+        response.headers['Access-Control-Allow-Headers'] = 'Content-Type, Authorization'
+        response.headers['Cross-Origin-Resource-Policy'] = 'cross-origin'
+    else:
+        # For UI routes, add additional security headers
+        for header, value in additional_headers.items():
+            response.headers[header] = value
+    
+    return response
+``` 

test_api_curl.sh

@@ -0,0 +1,110 @@
+#!/bin/bash
+
+# Colors for better readability
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+YELLOW='\033[0;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Base URLs to test
+DEV_URL="http://localhost:5005"
+PROD_URL="https://ploughshares.nixc.us"
+
+# Default to development URL unless specified
+URL=${1:-$DEV_URL}
+
+echo -e "${BLUE}Testing API endpoints with curl for ${URL}${NC}"
+echo "=================================================="
+
+# Function to test API endpoint with different curl options
+test_api() {
+    local endpoint=$1
+    local method=$2
+    local description=$3
+    local headers=$4
+    local data=$5
+    local full_url="${URL}${endpoint}"
+    
+    echo -e "\n${YELLOW}Test: ${description}${NC}"
+    echo "------------------------------------------------"
+    
+    # Build curl command based on parameters
+    cmd="curl -s"
+    
+    # Add method if specified
+    if [ -n "$method" ]; then
+        cmd="$cmd -X $method"
+    fi
+    
+    # Add headers if specified
+    if [ -n "$headers" ]; then
+        cmd="$cmd $headers"
+    fi
+    
+    # Add data if specified
+    if [ -n "$data" ]; then
+        cmd="$cmd -d '$data'"
+    fi
+    
+    # Add URL
+    cmd="$cmd \"$full_url\""
+    
+    # Show the command being executed
+    echo -e "${BLUE}Command:${NC} $cmd"
+    
+    # Execute the command
+    echo -e "${BLUE}Response:${NC}"
+    eval $cmd | jq 2>/dev/null || echo "Response is not valid JSON"
+    
+    echo "------------------------------------------------"
+}
+
+# Test 1: Basic GET request to API test endpoint
+test_api "/api/test" "GET" "Basic GET request to API test endpoint" "" ""
+
+# Test 2: GET request with Accept header
+test_api "/api/test" "GET" "GET request with Accept: application/json header" "-H \"Accept: application/json\"" ""
+
+# Test 3: OPTIONS request (CORS preflight)
+test_api "/api/test" "OPTIONS" "OPTIONS request (CORS preflight)" "-H \"Origin: http://example.com\"" ""
+
+# Test 4: POST request with JSON data
+test_api "/api/transactions" "POST" "POST request with JSON data" "-H \"Content-Type: application/json\"" "{\"test\": \"data\"}"
+
+# Test 5: GET request with custom User-Agent
+test_api "/api/test" "GET" "GET request with custom User-Agent" "-H \"User-Agent: CustomClient/1.0\"" ""
+
+# Test 6: GET request with Authorization header
+test_api "/api/test" "GET" "GET request with Authorization header" "-H \"Authorization: Bearer test-token\"" ""
+
+# Test 7: POST request with form data
+test_api "/api/transactions" "POST" "POST request with form data" "-H \"Content-Type: application/x-www-form-urlencoded\"" "test=data"
+
+# Test 8: GET request with multiple custom headers
+test_api "/api/test" "GET" "GET request with multiple custom headers" "-H \"Accept: application/json\" -H \"X-Custom-Header: test\"" ""
+
+echo -e "\n${BLUE}API curl tests completed for ${URL}${NC}"
+
+# Now test the API with a more complex curl command that simulates a real client
+echo -e "\n${YELLOW}Simulating a real API client with complex curl command${NC}"
+echo "------------------------------------------------"
+
+echo -e "${BLUE}Command:${NC}"
+echo "curl -v -X POST \"${URL}/api/transactions\" \\"
+echo "  -H \"Content-Type: application/json\" \\"
+echo "  -H \"Accept: application/json\" \\"
+echo "  -H \"User-Agent: APIClient/1.0\" \\"
+echo "  -H \"Authorization: Bearer test-token\" \\"
+echo "  -d '{\"transaction_type\":\"test\",\"amount\":100,\"recipient\":\"Test Corp\"}'"
+
+echo -e "\n${BLUE}Response:${NC}"
+curl -v -X POST "${URL}/api/transactions" \
+  -H "Content-Type: application/json" \
+  -H "Accept: application/json" \
+  -H "User-Agent: APIClient/1.0" \
+  -H "Authorization: Bearer test-token" \
+  -d '{"transaction_type":"test","amount":100,"recipient":"Test Corp"}' 2>&1
+
+echo -e "\n------------------------------------------------"
+echo -e "${GREEN}All tests completed.${NC}" 

test_api_headers.sh

@@ -0,0 +1,138 @@
+#!/bin/bash
+
+# Colors for better readability
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+YELLOW='\033[0;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Base URLs to test
+DEV_URL="http://localhost:5005"
+PROD_URL="https://ploughshares.nixc.us"
+
+# Default to development URL unless specified
+URL=${1:-$DEV_URL}
+
+echo -e "${BLUE}Testing API Security Headers for ${URL}${NC}"
+echo "=================================================="
+
+# Function to test an endpoint's headers
+test_headers() {
+    local endpoint=$1
+    local description=$2
+    local full_url="${URL}${endpoint}"
+    
+    echo -e "\n${YELLOW}Testing headers for ${description} (${endpoint})${NC}"
+    echo "------------------------------------------------"
+    
+    # Get headers only with curl
+    headers=$(curl -s -I "${full_url}")
+    
+    # Display all headers for reference
+    echo -e "${BLUE}All Headers:${NC}"
+    echo "$headers" | grep -v "Date:" | grep -v "Server:"
+    
+    echo -e "\n${BLUE}Security Header Analysis:${NC}"
+    
+    # Check for Content-Security-Policy
+    if echo "$headers" | grep -q "Content-Security-Policy"; then
+        csp=$(echo "$headers" | grep "Content-Security-Policy" | sed "s/Content-Security-Policy: //")
+        echo -e "${GREEN}✓ Content-Security-Policy:${NC}"
+        
+        # Check for unsafe-inline and unsafe-eval in script-src
+        if echo "$csp" | grep -q "script-src.*'unsafe-inline'"; then
+            echo -e "  ${YELLOW}⚠ script-src contains 'unsafe-inline' - needed for API clients${NC}"
+        else
+            echo -e "  ${RED}✗ script-src missing 'unsafe-inline' - may cause issues with some clients${NC}"
+        fi
+        
+        if echo "$csp" | grep -q "script-src.*'unsafe-eval'"; then
+            echo -e "  ${YELLOW}⚠ script-src contains 'unsafe-eval' - needed for API clients${NC}"
+        else
+            echo -e "  ${RED}✗ script-src missing 'unsafe-eval' - may cause issues with some clients${NC}"
+        fi
+        
+        # Check for connect-src
+        if echo "$csp" | grep -q "connect-src.*\\*"; then
+            echo -e "  ${GREEN}✓ connect-src includes * - good for API access${NC}"
+        else
+            echo -e "  ${RED}✗ connect-src restrictive - may block API access${NC}"
+        fi
+    else
+        echo -e "${RED}✗ Content-Security-Policy header not found${NC}"
+    fi
+    
+    # Check for CORS headers for API endpoints
+    if [[ "$endpoint" == "/api/"* ]]; then
+        echo -e "\n${BLUE}CORS Headers (critical for API):${NC}"
+        
+        if echo "$headers" | grep -q "Access-Control-Allow-Origin"; then
+            origin=$(echo "$headers" | grep "Access-Control-Allow-Origin" | sed "s/Access-Control-Allow-Origin: //")
+            echo -e "${GREEN}✓ Access-Control-Allow-Origin: $origin${NC}"
+        else
+            echo -e "${RED}✗ Access-Control-Allow-Origin header not found - CORS will fail${NC}"
+        fi
+        
+        if echo "$headers" | grep -q "Access-Control-Allow-Methods"; then
+            methods=$(echo "$headers" | grep "Access-Control-Allow-Methods" | sed "s/Access-Control-Allow-Methods: //")
+            echo -e "${GREEN}✓ Access-Control-Allow-Methods: $methods${NC}"
+        else
+            echo -e "${RED}✗ Access-Control-Allow-Methods header not found - CORS will fail${NC}"
+        fi
+        
+        if echo "$headers" | grep -q "Access-Control-Allow-Headers"; then
+            allowed_headers=$(echo "$headers" | grep "Access-Control-Allow-Headers" | sed "s/Access-Control-Allow-Headers: //")
+            echo -e "${GREEN}✓ Access-Control-Allow-Headers: $allowed_headers${NC}"
+        else
+            echo -e "${RED}✗ Access-Control-Allow-Headers header not found - CORS will fail${NC}"
+        fi
+    fi
+    
+    # Check for Cross-Origin-Resource-Policy
+    if echo "$headers" | grep -q "Cross-Origin-Resource-Policy"; then
+        corp=$(echo "$headers" | grep "Cross-Origin-Resource-Policy" | sed "s/Cross-Origin-Resource-Policy: //")
+        echo -e "\n${BLUE}Resource Policy:${NC}"
+        
+        if [[ "$endpoint" == "/api/"* ]]; then
+            if [[ "$corp" == "cross-origin" ]]; then
+                echo -e "${GREEN}✓ Cross-Origin-Resource-Policy: $corp - correct for API${NC}"
+            else
+                echo -e "${RED}✗ Cross-Origin-Resource-Policy: $corp - should be 'cross-origin' for API endpoints${NC}"
+            fi
+        else
+            if [[ "$corp" == "same-origin" ]]; then
+                echo -e "${GREEN}✓ Cross-Origin-Resource-Policy: $corp - correct for UI${NC}"
+            else
+                echo -e "${YELLOW}⚠ Cross-Origin-Resource-Policy: $corp - should be 'same-origin' for UI endpoints${NC}"
+            fi
+        fi
+    else
+        echo -e "${YELLOW}⚠ Cross-Origin-Resource-Policy header not found${NC}"
+    fi
+    
+    echo "------------------------------------------------"
+}
+
+# Test UI endpoint
+test_headers "/" "Main UI page"
+
+# Test API endpoints
+test_headers "/api/test" "API test endpoint"
+
+# Test OPTIONS request for CORS preflight
+echo -e "\n${YELLOW}Testing OPTIONS request for CORS preflight${NC}"
+echo "------------------------------------------------"
+curl -s -X OPTIONS -H "Origin: http://example.com" -I "${URL}/api/test" | grep -v "Date:" | grep -v "Server:"
+echo "------------------------------------------------"
+
+# Test with a real-world API client simulation
+echo -e "\n${YELLOW}Testing with a simulated API client${NC}"
+echo "------------------------------------------------"
+echo -e "${BLUE}Command:${NC} curl -s -X GET -H \"Origin: http://example.com\" -H \"Accept: application/json\" \"${URL}/api/test\""
+response=$(curl -s -X GET -H "Origin: http://example.com" -H "Accept: application/json" "${URL}/api/test")
+echo -e "${BLUE}Response:${NC}"
+echo "$response" | jq 2>/dev/null || echo "$response"
+echo "------------------------------------------------"
+
+echo -e "\n${GREEN}All header tests completed.${NC}" 

test_headers.sh

@@ -0,0 +1,123 @@
+#!/bin/bash
+
+# Colors for better readability
+GREEN='\033[0;32m'
+RED='\033[0;31m'
+YELLOW='\033[0;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Base URLs to test
+DEV_URL="http://localhost:5005"
+PROD_URL="https://ploughshares.nixc.us"
+
+# Default to development URL unless specified
+URL=${1:-$DEV_URL}
+
+echo -e "${BLUE}Testing headers for ${URL}${NC}"
+echo "=================================================="
+
+# Function to test an endpoint
+test_endpoint() {
+    local endpoint=$1
+    local method=$2
+    local description=$3
+    local full_url="${URL}${endpoint}"
+    
+    echo -e "\n${YELLOW}Testing ${method} ${endpoint}: ${description}${NC}"
+    echo "------------------------------------------------"
+    
+    if [ "$method" == "HEAD" ]; then
+        # Use -I for HEAD requests
+        curl -s -I "${full_url}" | grep -v "Date:" | grep -v "Server:"
+    elif [ "$method" == "OPTIONS" ]; then
+        # Use -X OPTIONS for OPTIONS requests
+        curl -s -X OPTIONS "${full_url}" -I | grep -v "Date:" | grep -v "Server:"
+    elif [ "$method" == "GET" ]; then
+        # First check headers
+        echo -e "${BLUE}Headers:${NC}"
+        curl -s -I "${full_url}" | grep -v "Date:" | grep -v "Server:"
+        
+        # Then check response body
+        echo -e "\n${BLUE}Response:${NC}"
+        curl -s "${full_url}"
+        echo -e "\n"
+    else
+        echo "Unsupported method: ${method}"
+    fi
+    
+    echo "------------------------------------------------"
+}
+
+# Function to check for specific security headers
+check_security_headers() {
+    local endpoint=$1
+    local full_url="${URL}${endpoint}"
+    
+    echo -e "\n${YELLOW}Checking security headers for ${endpoint}${NC}"
+    echo "------------------------------------------------"
+    
+    # Store headers in a variable
+    headers=$(curl -s -I "${full_url}")
+    
+    # Check for important security headers
+    check_header "Content-Security-Policy" "$headers"
+    check_header "Cross-Origin-Resource-Policy" "$headers"
+    check_header "Cross-Origin-Embedder-Policy" "$headers"
+    check_header "Cross-Origin-Opener-Policy" "$headers"
+    check_header "X-Frame-Options" "$headers"
+    check_header "X-Content-Type-Options" "$headers"
+    check_header "X-XSS-Protection" "$headers"
+    check_header "Referrer-Policy" "$headers"
+    
+    # For API endpoints, check CORS headers
+    if [[ "$endpoint" == "/api/"* ]]; then
+        check_header "Access-Control-Allow-Origin" "$headers"
+        check_header "Access-Control-Allow-Methods" "$headers"
+        check_header "Access-Control-Allow-Headers" "$headers"
+    fi
+    
+    echo "------------------------------------------------"
+}
+
+# Function to check for a specific header
+check_header() {
+    local header=$1
+    local headers=$2
+    
+    if echo "$headers" | grep -q "$header"; then
+        header_value=$(echo "$headers" | grep "$header" | sed "s/$header: //")
+        echo -e "${GREEN}✓ $header: $header_value${NC}"
+    else
+        echo -e "${RED}✗ $header not found${NC}"
+    fi
+}
+
+# Test UI endpoints
+test_endpoint "/" "HEAD" "Main page headers"
+check_security_headers "/"
+
+# Test API endpoints
+test_endpoint "/api/test" "HEAD" "API test endpoint headers"
+check_security_headers "/api/test"
+
+test_endpoint "/api/test" "GET" "API test endpoint response"
+
+# Test OPTIONS request for CORS preflight
+test_endpoint "/api/test" "OPTIONS" "API CORS preflight request"
+
+# Test a POST request to API
+echo -e "\n${YELLOW}Testing POST to /api/transactions${NC}"
+echo "------------------------------------------------"
+curl -s -X POST "${URL}/api/transactions" \
+  -H "Content-Type: application/json" \
+  -d '{"test": "data"}' | jq 2>/dev/null || echo "Response is not valid JSON"
+echo -e "\n------------------------------------------------"
+
+# Test with different Accept headers
+echo -e "\n${YELLOW}Testing with Accept: application/json header${NC}"
+echo "------------------------------------------------"
+curl -s -H "Accept: application/json" "${URL}/api/test" | jq 2>/dev/null || echo "Response is not valid JSON"
+echo -e "\n------------------------------------------------"
+
+echo -e "\n${BLUE}Tests completed for ${URL}${NC}"