Restructure logging architecture with file-based storage and automated rotation

[anshikasharmaa1517]

API Changes:

• Refactored logger.js to use file-based logging instead of Redis
• Added new logging directories and files
• Improved error handling and logging format
• Renamed theatIntelligence.js to threatIntelligence.js

Docker and Configuration Changes:

• Updated docker-compose.yml with new volume mappings for logs
• Added new logstash.conf file for centralized logging

Imp Note on Logstash:
Currently using volume-mapped config:
volumes: ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
Future improvement: Create custom Logstash image

[anshikasharmaa1517]

Storage Paths:

• /var/log/dicomhawk: Centralized logging (API, DICOM, exceptions)
• /opt/dicomhawk/db: Database storage
• /opt/dicomhawk/storage: DICOM files and blackhole list
• /opt/dicomhawk/tcia: TCIA data and staging area

Logging Infrastructure: (This is following the T-pot logstash structure)

• Updated logstash.conf for structured log handling
• Configured log types: simplified, pynetdicom, API, reputation, exceptions
• Standardized log paths across all services

Volume Management:

• Replaced shared_data volume with specific mounts
• Separated concerns: logs, database, DICOM storage
• Maintained data persistence between container restarts

Now the logs have one parent folder which have a sub-folder and files for all logs.

Fix required :

1. Caching of osm is not working. (fixed)

#52

[anshikasharmaa1517]

• Added dicomhawkinit container for automated log management.
• Implemented logrotate with pigz compression and daily rotation.
• Added cron job scheduling for log cleanup and rotation.
• Updated docker-compose.yml with merged services and proper networking.

[anshikasharmaa1517]

DICOMHawk was experiencing critical failures due to database path mismatches and file staging errors. The TCIA integration process was failing with shutil.Error: Destination path already exists because the file staging operations didn't handle cases where destination files already existed, causing the entire data refresh process to fail.
Additionally, both the DICOM server and API were configured to use an empty database file at /opt/dicomhawk/db/db.db while the actual data was stored in /opt/dicomhawk/storage/db.db, resulting in database initialization errors and the API displaying empty database.
Fixed these issues by adding existence checks before file operations in the TCIA management code, updating the database paths in both the DICOM server configuration and API database connector to point to the correct location, fixing Docker volume mounts, and improving error handling in the API routes to prevent crashes when database queries returned undefined results.
After rebuilding the containers to apply these changes, the system now successfully downloads new DICOM files from TCIA, updates the database with fresh patient names, and provides stable API access without crashes.