6G Multivariate Time-Series Imputation Pipeline

The project in this repository is part of my thesis for the Research Project 2026 Q4 edition at TU Delft.

The paper written to describe the background of this research and to document the methodology and analysis of results can be found here.

---

This repository contains the end-to-end evaluation framework for benchmarking multivariate time-series imputation architectures under simulated 6G edge telemetry outages and connection drops.

This pipeline was developed to evaluate the Pareto-optimal trade-off between reconstruction accuracy (RMSE/MAPE) and algorithmic latency across traditional statistical baselines and modern deep learning paradigms.

Evaluated Architectures

• Statistical Baselines: Nearest Neighbor, Kalman Filter (Darts)
• Recurrent Neural Network: BRITS (PyPOTS)
• Generative Diffusion Model: CSDI (PyPOTS)
• Convolutional Neural Network: TimesNet (PyPOTS)

Note on Hyperparameter Tuning: Model optimization (specifically for TimesNet) was conducted offline using the Optuna framework with GPU acceleration (device="cuda"). However, to ensure a strictly fair and hardware-agnostic comparison for the final Pareto latency frontier, all model inferences within the Airflow evaluation pipeline are forced to execute on the CPU.

Prerequisites

To ensure absolute reproducibility, regardless of OS, the Airflow evaluation framework is containerized. To run the pipeline and the local dashboard, you will need:

• Docker Desktop installed and running.
• Astronomer CLI (astro) installed.
• Python 3.12+ (Required locally for the interactive Streamlit dashboard).
• Recommended Hardware: 16GB RAM and a multi-core CPU. (Note: Running the full pipeline on 8GB of RAM is possible, but you may need to increase your operating system's Swap/Pagefile memory to prevent Docker Out-Of-Memory container crashes during the heavy deep learning training).

Setup & Execution Instructions

1. Clone the Repository

Clone this repository to your local machine:

git clone https://github.com/neriAle/6G-Time-Series-Imputation.git
cd 6G-Time-Series-Imputation

2. Launch the Pipeline

Start the Airflow environment using the Astronomer CLI. This will spin up the necessary Postgres, Scheduler, and Webserver containers:

astro dev start

3. Access the Airflow UI

Because this project utilizes dynamic port mapping to prevent local collisions, the Airflow Webserver port changes on each initialization. To find the correct UI link:

1. Open Docker Desktop.
2. Locate the running container named api-server-1.
3. Under the "Ports" column, click the dynamically generated localhost link (e.g., http://127.0.0.1:11371/).

4. Run the Experiments

The pipeline is modularized into four distinct DAGs with a dynamic staging area. To execute the full evaluation:

1. Start the Pipeline: In the Airflow dashboard, click the "Trigger" (Play) button on the data_preparation DAG. Here you can specify the dataset_folder (e.g., amf or python), indicate if the data is_pre_split, and adjust the 24-scenario simulation grid. Click "Trigger" at the bottom of the page.
2. Wait for Imputation: The data_preparation DAG will automatically trigger the data_imputation DAG upon completion. The pipeline utilizes a throttled concurrency limit (max_active_tasks=2) to protect local RAM while maintaining parallel branching. Wait for data_imputation to fully succeed for all models. (Note: Training deep learning architectures from scratch on new datasets requires significant compute time).
3. Evaluate Metrics: Once imputation is complete, manually click the "Trigger" button on model_evaluation to calculate the RMSE and MAPE scores.
4. Stage Results for Visualization: The pipeline outputs results to a volatile staging area. To visualize the data and allow the Streamlit dashboard to dynamically discover your dataset, you must move the outputs into a dedicated folder. Create a new folder named after your dataset (e.g., include/data/results/amf/) and move the following items into it:
   • include/data/results/streamlit_dataset.csv
   • include/data/intermediate/test_gt.csv
   • The entire include/data/intermediate/imputed/ folder
5. Interactive Dashboard: With the evaluation complete and files safely staged, you can interactively explore the results across any generated datasets. Open a terminal in the root directory of the project and launch the web dashboard:

   pip install streamlit pandas
   streamlit run streamlit_app.py

6. Generate Static Visuals: Alternatively, you can manually trigger the generate_plots DAG to output the aggregate tables, line charts, and Pareto frontiers as static files.

5. Shut Down

To safely spin down the containers run:

astro dev stop

Dataset Information

This repository utilizes 5G/6G Core Edge Telemetry (specifically 5G AMF request configurations and Python Web Server metrics) sourced from a public Zenodo repository designed for microservice benchmarking. The datasets can be placed into the /include/data/datasets directory for dynamic pipeline ingestion.

📄 License

This project is licensed under the Apache-2.0 License.