Building a Real-Time Weather Text Ingestion and AI Forecast Summarization System with GOES‑16 EMWIN Data

• Extract the forecast text for that zone from the file.
• Parse and split the forecast into day segments (e.g., “TODAY”, “TONIGHT”, “SATURDAY”, etc.).
• Convert UTC dates to local time (CST) based on the forecast’s issuance timestamp so that North American users see the correct day.
• Insert the processed forecast segments into a MySQL database using an upsert strategy.
• Generate concise summaries using OpenAI’s GPT models.
• Schedule regular updates via cron running in a Docker container.

• List Date Folders: I used the S3 API’s listObjectsV2 with a delimiter (/) to list folders corresponding to dates.
• Select a Date: The system checks a GET parameter or defaults to today’s date (or the most recent available date).
• Identify the Target File: I then look for the “Zone Forecast Product” file (e.g., a file named like A_FPUS54KMOB071445_C_KWIN_20250207144511_831106-3-ZFPMOBAL.TXT) in that folder. The above file is specific to the Mobile, AL region and includes several Gulf Coast areas including Mobile County, Baldwin County, Washington County, Escambia County etc. The data I extract is further specific to Baldwin Central which includes the cities of Bay Minette, Daphne, Fairhope, Foley, and Spanish Fort.

// Instantiate S3 client.
$s3 = new S3Client([
    'version' => 'latest',
    'region'  => 'us-east-1',
]);

$bucket     = 'your_s3_bucket';
$basePrefix = 'kmob-texts/';
$today        = date('Y-m-d');
$selectedDate = $_GET['date'] ?? $today;

// List available date folders.
$result = $s3->listObjectsV2([
    'Bucket'    => $bucket,
    'Delimiter' => '/',
    'Prefix'    => $basePrefix,
]);
$availableDates = [];
if (isset($result['CommonPrefixes'])) {
    foreach ($result['CommonPrefixes'] as $prefixData) {
        $folder = str_replace([$basePrefix, '/'], '', $prefixData['Prefix']);
        if (!empty($folder)) {
            $availableDates[] = $folder;
        }
    }
}
rsort($availableDates);
if (!in_array($selectedDate, $availableDates, true)) {
    $selectedDate = !empty($availableDates) ? $availableDates[0] : $today;
}
$datePrefix = $basePrefix . $selectedDate . '/';
$result = $s3->listObjectsV2([
    'Bucket' => $bucket,
    'Prefix' => $datePrefix,
]);

// Find the Zone Forecast Product file.
$targetKey = null;
foreach ($result['Contents'] as $object) {
    $basename = basename($object['Key']);
    if (preg_match('/^A_FPUS54KMOB\d{6}_C_KWIN_\d{14}_\d+-\d+-ZFPMOBAL\.TXT$/i', $basename)) {
        $targetKey = $object['Key'];
        break;
    }
}

941 PM CST Fri Feb 7 2025

• I used a regular expression to locate this line and PHP’s DateTime::createFromFormat() to parse it.
Convert & Override Date:
• Since the issuance string already includes “CST,” I extract the local date and override the selected date with this value.

if (preg_match('/(\d{1,2}\d{2}\s?(AM|PM)\s+[A-Z]{3}\s+[A-Za-z]{3}\s+\d{1,2}\s+\d{4})/i', $fileContent, $timeMatches)) {
    $issuanceStr = trim($timeMatches[1]);
    $issuedAt = DateTime::createFromFormat('gia T D M j Y', $issuanceStr);
    if ($issuedAt) {
        $localForecastDate = $issuedAt->format('Y-m-d');
        $selectedDate = $localForecastDate;
    }
}

• Extract the Forecast Section: Using a regular expression to extract the block of text for ALZ264.
• Split into Segments: Each segment starts with a line like .REST OF TODAY... or .TONIGHT.... I then split the text into an array of segments.

if (preg_match('/^(ALZ264[^\n]*\n(?:.*\n)*?)(?=^[A-Z]{3}\d{3}-|\$\$)/m', $fileContent, $matches)) {
    $extractedZoneForecast = trim($matches[1]);
} else {
    $extractedZoneForecast = 'Zone forecast for ALZ264 not found in the file.';
}

$dayForecasts = [];
if (preg_match_all('/\.(?[A-Z ]+)\.\.\.(?.*?)(?=\.[A-Z ]+\.\.\.|$)/s', $extractedZoneForecast, $matches)) {
    foreach ($matches['day'] as $i => $day) {
        $dayForecasts[] = [
            'day'      => trim($day),
            'forecast' => trim($matches['forecast'][$i])
        ];
    }
}

CREATE TABLE zone_forecasts (
    id INT AUTO_INCREMENT PRIMARY KEY,
    zone VARCHAR(10) NOT NULL,
    forecast_date DATE NOT NULL,
    forecast_day VARCHAR(50) NOT NULL,
    forecast_text TEXT NOT NULL,
    issued_at DATETIME NULL,
    UNIQUE KEY uniq_forecast (zone, forecast_date, forecast_day)
);

$zoneCode = 'ALZ264';
$insertMessages = [];

$dsn = "mysql:host=" . DB_HOST . ";dbname=" . DB_NAME . ";charset=utf8mb4";
$pdo = new PDO($dsn, DB_USER, DB_PASS);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

$stmt = $pdo->prepare("
    INSERT INTO zone_forecasts (zone, forecast_date, forecast_day, forecast_text)
    VALUES (:zone, :forecast_date, :forecast_day, :forecast_text)
    ON DUPLICATE KEY UPDATE forecast_text = VALUES(forecast_text)
");

foreach ($dayForecasts as $dayForecast) {
    $forecast_day  = trim($dayForecast['day']);
    $forecast_day = str_replace(
        ['REST OF TODAY', 'REST OF TONIGHT'],
        ['TODAY', 'TONIGHT'],
        $forecast_day
    );
    $forecast_text = trim($dayForecast['forecast']);

    $stmt->bindParam(':zone', $zoneCode);
    $stmt->bindParam(':forecast_date', $selectedDate);
    $stmt->bindParam(':forecast_day', $forecast_day);
    $stmt->bindParam(':forecast_text', $forecast_text);
    $stmt->execute();
    $insertMessages[] = "Inserted/Updated forecast for {$forecast_day}";
}

$finalInsertMessage = "Operation complete: " . implode(", ", $insertMessages);
error_log($finalInsertMessage);
echo $finalInsertMessage;

Prompt Structure Example:

$promptToday = <<<PROMPT
You are a meteorologist specializing in weather forecast analysis for zone ALZ264.
Below is the detailed forecast for TODAY:
{$todayText}

Provide a structured summary in standard paragraphs with the following sections:

Observations:
Summarize the key weather conditions, including temperature, wind, and precipitation details for today.

Trends:
Highlight any noticeable trends or changes expected during the day.

Recommendations:
Offer any advisories or recommendations for today (for example, warnings about dense fog or high winds).

IMPORTANT:
- Do not use asterisks, bullet points, or dashes unless they are part of numeric data or proper names.
- Write in standard paragraphs.
- Keep your response succinct.
PROMPT;

• Fetches GOES‑16 weather texts from an S3 bucket,
• Correctly groups and processes forecast data based on the local issuance timestamp,
• Inserts/upserts the forecast data into a MySQL database,
• Uses OpenAI to generate clear, structured summaries,
• And schedules the entire process using cron inside a Docker container.