Baffle-WAF/baffle-hub
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
3f274c842c4e59f665128b6f1be0d9aa4c35d8f9
baffle-hub/docs/maxmind.md

11 KiB
Raw Blame History

MaxMind GeoIP Integration

This document describes the MaxMind GeoIP integration implemented in the Baffle Hub WAF analytics system.

Overview

The Baffle Hub application uses MaxMind's free GeoLite2-Country database to provide geographic location information for IP addresses. The system automatically enriches WAF events with country codes and provides manual lookup capabilities for both IPv4 and IPv6 addresses.

Features

  • On-demand lookup - Country code lookup by IP address
  • Automatic enrichment - Events are enriched with geo-location data during processing
  • Manual lookup capability - Rake tasks and model methods for manual lookups
  • GeoLite2-Country database - Uses MaxMind's free country-level database
  • Automatic updates - Weekly background job updates the database
  • IPv4/IPv6 support - Full protocol support for both IP versions
  • Performance optimized - Database caching and efficient lookups
  • Graceful degradation - Fallback handling when database is unavailable

Architecture

Core Components

1. GeoIpService

  • Central service for all IP geolocation operations
  • Handles database loading from file system
  • Provides batch lookup capabilities
  • Manages database updates from MaxMind CDN
  • Uses MaxMind's built-in metadata for version information

2. UpdateGeoIpDatabaseJob

  • Background job for automatic database updates
  • Runs weekly to keep the database current
  • Simple file-based validation and updates

3. Enhanced Models

  • Event Model - Automatic geo-location enrichment for WAF events
  • IPv4Range/IPv6Range Models - Manual lookup methods for IP ranges

4. File-System Management

  • Database stored as single file: db/geoip/GeoLite2-Country.mmdb
  • Version information queried directly from MaxMind database metadata
  • No database tables needed - simplified approach

Installation & Setup

Dependencies

The integration uses the following gems:

  • maxmind-db - Official MaxMind database reader (with built-in caching)
  • httparty - HTTP client for database downloads

Database Storage

  • Location: db/geoip/GeoLite2-Country.mmdb
  • Automatic creation of storage directory
  • File validation and integrity checking
  • Version information queried directly from database metadata
  • No additional caching needed - MaxMind DB has its own internal caching

Initial Setup

# Install dependencies
bundle install

# Download the GeoIP database
rails geoip:update

# Verify installation
rails geoip:status

Configuration

The system is configurable via environment variables or application configuration:

Variable Default Description
MAXMIND_DATABASE_URL MaxMind CDN URL Database download URL
MAXMIND_AUTO_UPDATE true Enable automatic weekly updates
MAXMIND_UPDATE_INTERVAL_DAYS 7 Days between update checks
MAXMIND_MAX_AGE_DAYS 30 Maximum database age before forced update
Note: MaxMind DB has built-in caching, no additional caching needed
MAXMIND_FALLBACK_COUNTRY nil Fallback country when lookup fails
MAXMIND_ENABLE_FALLBACK false Enable fallback country usage

Example Configuration

# config/application.rb or .env file
MAXMIND_AUTO_UPDATE=true
MAXMIND_UPDATE_INTERVAL_DAYS=7
MAXMIND_MAX_AGE_DAYS=30
MAXMIND_FALLBACK_COUNTRY=US
MAXMIND_ENABLE_FALLBACK=true
# Note: No caching configuration needed - MaxMind has built-in caching

Usage

Rake Tasks

Database Management

# Download/update the GeoIP database
rails geoip:update

# Check database status and configuration
rails geoip:status

# Test the implementation with sample IPs
rails geoip:test

# Manual lookup for a specific IP
rails geoip:lookup[8.8.8.8]
rails geoip:lookup[2001:4860:4860::8888]

Data Management

# Enrich existing events missing country codes
rails geoip:enrich_missing

# Clean up old inactive database records
rails geoip:cleanup

Ruby API

Service-Level Lookups

# Direct country lookup
country = GeoIpService.lookup_country('8.8.8.8')
# => "US"

# Batch lookup
countries = GeoIpService.new.lookup_countries(['8.8.8.8', '1.1.1.1'])
# => { "8.8.8.8" => "US", "1.1.1.1" => nil }

# Check database availability
service = GeoIpService.new
service.database_available?  # => true/false
service.database_info         # => Database metadata

Event Model Integration

# Automatic enrichment during event processing
event = Event.find(123)
event.enrich_geo_location!           # Updates event with country code
event.lookup_country                 # => "US" (with fallback to service)
event.has_geo_data?                  # => true/false
event.geo_location                   # => { country_code: "US", city: nil, ... }

# Batch enrichment of existing events
updated_count = Event.enrich_geo_location_batch
puts "Enriched #{updated_count} events with geo data"

IP Range Model Integration

# IPv4 Range lookups
range = Ipv4Range.find(123)
range.geo_lookup_country!            # Updates range with country code
range.geo_lookup_country             # => "US" (without updating)
range.has_country_info?              # => true/false
range.primary_country                # => "US" (best available country)

# Class methods
country = Ipv4Range.lookup_country_by_ip('8.8.8.8')
updated_count = Ipv4Range.enrich_missing_geo_data(limit: 1000)

# IPv6 Range lookups (same interface)
country = Ipv6Range.lookup_country_by_ip('2001:4860:4860::8888')
updated_count = Ipv6Range.enrich_missing_geo_data(limit: 1000)

Background Processing

Automatic Updates

The system automatically schedules database updates:

# Manually trigger an update (usually scheduled automatically)
UpdateGeoIpDatabaseJob.perform_later

# Force update regardless of age
UpdateGeoIpDatabaseJob.perform_later(force_update: true)

Event Processing Integration

Geo-location enrichment is automatically included in WAF event processing:

# This is called automatically in ProcessWafEventJob
event = Event.create_from_waf_payload!(event_id, payload, project)
event.enrich_geo_location! if event.ip_address.present? && event.country_code.blank?

Database Information

GeoLite2-Country Database

  • Source: MaxMind GeoLite2-Country (free version)
  • Update Frequency: Weekly (Tuesdays)
  • Size: ~9.5 MB
  • Coverage: Global IP-to-country mapping
  • Format: MaxMind DB (.mmdb)

Database Fields

  • country.iso_code - Two-letter ISO country code
  • Supports both IPv4 and IPv6 addresses
  • Includes anonymous/proxy detection metadata

Performance Considerations

Performance

  • MaxMind DB has built-in internal caching optimized for lookups
  • Typical lookup time: <1ms
  • Database size optimized for fast lookups
  • No additional caching layer needed

Lookup Performance

  • Typical lookup time: <1ms
  • Database size optimized for fast lookups
  • Efficient range queries for IP networks

Memory Usage

  • Database loaded into memory for fast access
  • Approximate memory usage: 15-20 MB for the country database
  • Automatic cleanup of old database files

Error Handling

Graceful Degradation

  • Service returns nil when database unavailable
  • Logging at appropriate levels for different error types
  • Event processing continues even if geo-location fails

Common Error Scenarios

  1. Database Missing - Automatic download triggered
  2. Database Corrupted - Automatic re-download attempted
  3. Network Issues - Graceful fallback with error logging
  4. Invalid IP Address - Returns nil with warning log

Troubleshooting

Check System Status

# Verify database status
rails geoip:status

# Test with known IPs
rails geoip:test

# Check logs for errors
tail -f log/production.log | grep GeoIP

Common Issues

Database Not Available

# Force database update
rails geoip:update

# Check file permissions
ls -la db/geoip/

Lookup Failures

# Test specific IPs
rails geoip:lookup[8.8.8.8]

# Check database validity
rails runner "puts GeoIpService.new.database_available?"

Performance Issues

  • Increase cache size in configuration
  • Check memory usage on deployment server
  • Monitor lookup times with application metrics

Monitoring & Maintenance

Health Checks

# Rails console health check
service = GeoIpService.new
puts "Database available: #{service.database_available?}"
puts "Database age: #{service.database_record&.age_in_days} days"

Scheduled Maintenance

  • Database automatically updated weekly
  • Old database files cleaned up after 7 days
  • No manual maintenance required

Monitoring Metrics

Consider monitoring:

  • Database update success/failure rates
  • Lookup performance (response times)
  • Database age and freshness
  • Cache hit/miss ratios

Security & Privacy

Data Privacy

  • No personal data stored in the GeoIP database
  • Only country-level information provided
  • No tracking or logging of IP lookups by default

Network Security

  • Database downloaded from official MaxMind CDN
  • File integrity validated with MD5 checksums
  • Secure temporary file handling during updates

API Reference

GeoIpService

Class Methods

  • lookup_country(ip_address) - Direct lookup
  • update_database! - Force database update

Instance Methods

  • lookup_country(ip_address) - Country lookup
  • lookup_countries(ip_addresses) - Batch lookup
  • database_available? - Check database status
  • database_info - Get database metadata
  • update_from_remote! - Download new database

Model Methods

Event Model

  • enrich_geo_location! - Update with country code
  • lookup_country - Get country code (with fallback)
  • has_geo_data? - Check if geo data exists
  • geo_location - Get full geo location hash

IPv4Range/IPv6Range Models

  • geo_lookup_country! - Update range with country code
  • geo_lookup_country - Get country code (without update)
  • has_country_info? - Check for existing country data
  • primary_country - Get best available country code
  • lookup_country_by_ip(ip) - Class method for IP lookup
  • enrich_missing_geo_data(limit:) - Class method for batch enrichment

Support & Resources

MaxMind Documentation

Ruby Libraries

Troubleshooting Resources

  • Application logs: log/production.log
  • Rails console for manual testing
  • Database status via rails geoip:status
Reference in New Issue View Git Blame Copy Permalink