Admin/ocean-cleanup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement hotspot calculation system with scheduled jobs

Browse Source 
- Add hotspot calculation methods to analytics service
- Implement MySQL spatial queries for clustering (5+ reports within 500m)
- Calculate severity scores based on report count and frequency
- Create GET /api/analytics/hotspots endpoint
- Create GET /api/analytics/hotspots/:id endpoint for details
- Create POST /api/analytics/hotspots/calculate for manual trigger
- Implement scheduled job with node-cron (runs every 5 minutes)
- Add comprehensive test scripts for hotspot functionality
- Create detailed implementation documentation

Requirements addressed:
- 3.2: Identify areas with 5+ reports within 500m radius
- 3.3: Rank hotspots by severity score
- 3.4: Update calculations within 5 minutes
- 3.5: Provide data for map visualization with severity indicators
This commit is contained in:
Paul Bailey
2025-11-17 00:19:35 +00:00
parent d91a13da60
commit 052ba46bdc
9 changed files with 793 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

316
server/HOTSPOT_CALCULATION_IMPLEMENTATION.md Normal file
View File

@@ -0,0 +1,316 @@
# Hotspot Calculation System Implementation
## Overview
The hotspot calculation system identifies and tracks areas with high concentrations of litter reports. It automatically calculates hotspots every 5 minutes using MySQL spatial queries and stores the results for efficient retrieval.
## Requirements Addressed
- **Requirement 3.2**: Identify areas with at least 5 reports within a 500-meter radius over the past 30 days
- **Requirement 3.3**: Rank Litter Hotspots by severity score calculated from report quantity and frequency
- **Requirement 3.4**: Update Litter Hotspot calculations within 5 minutes of new report submissions
- **Requirement 3.5**: Display Litter Hotspot boundaries on the Map Interface with color-coded severity indicators
## Architecture
### Components
1. **Spatial Utilities** (`server/src/utils/spatial.ts`)
- `calculateHotspots()`: Performs MySQL spatial clustering using ST_GeoHash
- `updateHotspotsTable()`: Clears old hotspots and inserts new calculations
2. **Analytics Service** (`server/src/services/analyticsService.ts`)
- `getHotspots()`: Retrieves current hotspots from database
- `calculateAndUpdateHotspots()`: Triggers hotspot calculation
- `getHotspotDetails(id)`: Gets detailed information about a specific hotspot
3. **Analytics Controller** (`server/src/controllers/analyticsController.ts`)
- `getHotspots()`: GET endpoint handler
- `getHotspotDetails()`: GET endpoint handler for specific hotspot
- `calculateHotspots()`: POST endpoint handler for manual calculation
4. **Scheduled Jobs** (`server/src/services/scheduledJobs.ts`)
- Runs hotspot calculation every 5 minutes using node-cron
- Initialized on server startup
## API Endpoints
### GET /api/analytics/hotspots
Retrieves all current hotspots (calculated within the last hour).
**Response:**
```json
{
"success": true,
"data": [
{
"id": "uuid",
"latitude": 53.3498,
"longitude": -6.2603,
"radius": 500,
"reportCount": 12,
"severityScore": 28.5,
"lastReportDate": "2025-11-17T00:00:00.000Z",
"calculatedAt": "2025-11-17T00:15:00.000Z"
}
]
}
```
### GET /api/analytics/hotspots/:id
Retrieves detailed information about a specific hotspot, including all reports within its radius.
**Response:**
```json
{
"success": true,
"data": {
"hotspot": {
"id": "uuid",
"latitude": 53.3498,
"longitude": -6.2603,
"radius": 500,
"reportCount": 12,
"severityScore": 28.5,
"lastReportDate": "2025-11-17T00:00:00.000Z",
"calculatedAt": "2025-11-17T00:15:00.000Z"
},
"reports": [
{
"id": "uuid",
"latitude": 53.3500,
"longitude": -6.2600,
"litterType": "plastic",
"quantity": "significant",
"createdAt": "2025-11-16T12:00:00.000Z",
"distance": 45.2
}
]
}
}
```
### POST /api/analytics/hotspots/calculate
Manually triggers hotspot calculation (useful for testing or immediate updates).
**Response:**
```json
{
"success": true,
"data": {
"success": true,
"hotspotsCalculated": 3,
"timestamp": "2025-11-17T00:15:00.000Z"
}
}
```
## Calculation Algorithm
### Step 1: Spatial Clustering
The system uses MySQL's `ST_GeoHash()` function to group nearby reports:
```sql
SELECT
AVG(latitude) as center_lat,
AVG(longitude) as center_lng,
COUNT(*) as report_count,
AVG(severity_value) as avg_severity,
MAX(created_at) as last_report_date
FROM reports
WHERE created_at >= DATE_SUB(NOW(), INTERVAL 30 DAY)
GROUP BY ST_GeoHash(location, 7)
HAVING COUNT(*) >= 5
```
### Step 2: Severity Score Calculation
Severity score is calculated as:
```
severity_score = report_count × avg_severity
```
Where severity values are:
- minimal: 1
- moderate: 2
- significant: 3
- severe: 4
### Step 3: Storage
Hotspots are stored in the `hotspots` table with:
- Geographic center point (POINT spatial type)
- 500m radius
- Report count
- Severity score
- Last report date
- Calculation timestamp
### Step 4: Cleanup
Old hotspots (calculated more than 1 hour ago) are automatically removed before new calculations.
## Scheduled Job
The hotspot calculation runs automatically every 5 minutes:
```typescript
cron.schedule('*/5 * * * *', async () => {
await analyticsService.calculateAndUpdateHotspots();
});
```
This ensures hotspots are updated within 5 minutes of new report submissions (Requirement 3.4).
## Database Schema
```sql
CREATE TABLE hotspots (
id VARCHAR(36) PRIMARY KEY DEFAULT (UUID()),
location POINT NOT NULL SRID 4326,
latitude DECIMAL(10, 8) NOT NULL,
longitude DECIMAL(11, 8) NOT NULL,
radius DECIMAL(10, 2) NOT NULL COMMENT 'Radius in meters',
report_count INT NOT NULL,
severity_score DECIMAL(5, 2) NOT NULL,
last_report_date TIMESTAMP NOT NULL,
calculated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
SPATIAL INDEX idx_location (location),
INDEX idx_severity_score (severity_score)
) ENGINE=InnoDB;
```
## Testing
### Unit Tests
Run the hotspot calculation test:
```bash
cd server
npx tsx src/scripts/testHotspots.ts
```
### API Endpoint Tests
1. Start the server:
```bash
cd server
npm run dev
```
2. Run endpoint tests:
```bash
npx tsx src/scripts/testHotspotEndpoints.ts
```
### Manual Testing
```bash
# Get all hotspots
curl http://localhost:3005/api/analytics/hotspots
# Trigger manual calculation
curl -X POST http://localhost:3005/api/analytics/hotspots/calculate
# Get specific hotspot details
curl http://localhost:3005/api/analytics/hotspots/{hotspot-id}
```
## Frontend Integration
To display hotspots on the map:
```typescript
import { useQuery } from '@tanstack/react-query';
function MapView() {
const { data: hotspots } = useQuery({
queryKey: ['hotspots'],
queryFn: async () => {
const response = await fetch('/api/analytics/hotspots');
return response.json();
},
refetchInterval: 5 * 60 * 1000, // Refetch every 5 minutes
});
// Render hotspots as circles on the map
hotspots?.data.forEach(hotspot => {
L.circle([hotspot.latitude, hotspot.longitude], {
radius: hotspot.radius,
color: getSeverityColor(hotspot.severityScore),
fillOpacity: 0.3,
}).addTo(map);
});
}
function getSeverityColor(score: number): string {
if (score >= 50) return '#dc2626'; // red - critical
if (score >= 30) return '#ea580c'; // orange - high
if (score >= 15) return '#f59e0b'; // yellow - medium
return '#84cc16'; // green - low
}
```
## Performance Considerations
1. **Spatial Indexes**: The `hotspots` table uses spatial indexes for efficient location queries
2. **Caching**: Hotspots are cached for 1 hour to reduce calculation frequency
3. **Scheduled Updates**: 5-minute intervals balance freshness with server load
4. **Query Optimization**: Uses MySQL spatial functions for efficient clustering
## Monitoring
The scheduled job logs its activity:
```
[Scheduled Jobs] Initializing scheduled jobs...
[Scheduled Jobs] ✓ Hotspot calculation job scheduled (every 5 minutes)
[Scheduled Jobs] Running hotspot calculation...
[Hotspot Calculation] Starting hotspot calculation...
[Hotspot Calculation] Successfully calculated 3 hotspots
[Scheduled Jobs] Hotspot calculation completed: 3 hotspots calculated
```
## Future Enhancements
1. **Historical Tracking**: Store hotspot history for trend analysis
2. **Notification System**: Alert users when new hotspots appear in their areas of interest
3. **Predictive Analysis**: Use ML to predict future hotspot locations
4. **Custom Thresholds**: Allow admins to configure minimum report count and radius
5. **Hotspot Decay**: Reduce severity scores for hotspots with no recent reports
## Troubleshooting
### No Hotspots Calculated
- Verify there are at least 5 reports within 500m in the last 30 days
- Check that reports have valid spatial data
- Review logs for calculation errors
### Scheduled Job Not Running
- Verify node-cron is installed: `npm list node-cron`
- Check server logs for initialization messages
- Ensure server is running continuously
### Spatial Query Errors
- Verify MySQL version is 8.0+ with spatial support
- Check that POINT columns use SRID 4326
- Ensure spatial indexes are created
## Related Files
- `server/src/services/analyticsService.ts` - Hotspot business logic
- `server/src/controllers/analyticsController.ts` - API endpoints
- `server/src/routes/analyticsRoutes.ts` - Route definitions
- `server/src/services/scheduledJobs.ts` - Cron job configuration
- `server/src/utils/spatial.ts` - Spatial utility functions
- `server/prisma/schema.prisma` - Database schema