Chapter 7: Trial Results and Historic Comparison

How SimulationInputs, TrialResults and HistoricResults turn the event log and the historical data files into metrics and plots, and where that code is fragile.

Written for: Intermediate Coder Confident Coder Decision-makers Domain experts General public

Resource Allocation ended with the event log recording who was sent to each call, or that nobody was. This chapter covers what reads that log afterwards.

What it is

Three classes in air_ambulance_des/ turn files on disk into numbers and Plotly figures. SimulationInputs loads the parameters, rota, servicing and callsign files that the run used. TrialResults takes the combined event log (data/run_results.csv, see Simulation Runs and Replications) and does all its derivation in the constructor. HistoricResults loads the aggregated files in historical_data/ plus a pre-computed simulated baseline that uses historic rotas. All three are wrapped by graceful_methods, which swallows exceptions.

Callers can rely on this much, and no more: construct TrialResults(...) once with the run’s run_results frame and every derived DataFrame attribute exists afterwards (call_df, utilisation_df_overall, missed_jobs_per_run_care_cat_summary, and so on), possibly as None if its step failed (see graceful_methods below). Plot methods return a Plotly figure or None. The PLOT_*, get_* and display_* methods read from those attributes. The main caller is the results page in app/model.py, which uses it to show missed jobs, utilisation and comparisons with history (Figure 1). Nothing here feeds back into the simulation.

flowchart LR
  RR[data/run_results.csv] --> TR[TrialResults]
  RP[data/run_params_used.csv<br>actual_data rota files] --> SI[SimulationInputs]
  SI --> TR
  DA[data/daily_availability.csv] --> TR
  HD[historical_data/ and<br>tests/rotas_historic] --> HR[HistoricResults]
  HR -->|passed into plot methods| TR
  TR --> APP[app/model.py tabs and report]
  HR --> APP

Figure 1: Files in, objects in the middle, figures out

The use case I will follow throughout is the “Missed Jobs” and utilisation panels: what fraction of calls found no resource (for example, get_perc_unattended_string_normalised returns a string such as "2 of 480 (< 0.1%)" when the fraction is under 1%), and how busy each callsign was compared with history.

How it works

The error-swallowing decorator

Everything below sits under graceful_methods, so read this first.

air_ambulance_des/_processing_functions.py:40-74

def graceful_methods(cls):
    """Wrap all methods in a class with context-aware error handling:
    - If running in Streamlit, shows st.error("Failed to generate graph.")
    - Otherwise, logs the error to console.
    """
    for name, method in cls.__dict__.items():
        if callable(method) and not name.startswith("__"):

            def make_wrapper(fn):
                def wrapper(self, *args, **kwargs):
                    try:
                        return fn(self, *args, **kwargs)
                    except Exception as e:
                        # Detect Streamlit
                        try:
                            import streamlit.runtime

                            in_streamlit = streamlit.runtime.exists()
                        except Exception:
                            in_streamlit = False

                        if in_streamlit:
                            import streamlit as st

                            st.error(
                                f"⚠️ Failed to generate output from `{fn.__name__}`: {e}."
                            )
                        else:
                            print(f"⚠️ Failed to run {fn.__name__}: {e}")
                        return None

                return wrapper

            setattr(cls, name, make_wrapper(method))
    return cls

It rebinds every callable in cls.__dict__ whose name does not start with __. Any exception becomes either st.error(...) (if streamlit.runtime.exists()) or a print, and the method returns None. Three consequences that matter when you debug:

__init__ is not wrapped (it starts with __), but every method it calls is. A failing step in a constructor therefore leaves the attribute at its None placeholder and carries on.
The wrapper is not decorated with functools.wraps, so wrapped methods lose their docstrings and __name__ (the name in the message comes from the closure variable fn).
There is no traceback in this variant. The sibling graceful function in the same file does print one, but it is not applied to these classes.

SimulationInputs

air_ambulance_des/class_simulation_inputs.py:5-22

@graceful_methods
class SimulationInputs:
    def __init__(
        self, data_folder_path="../data", actual_data_folder_path="../actual_data"
    ):
        self.params_df_path = f"{data_folder_path}/run_params_used.csv"
        self.rota_path = f"{actual_data_folder_path}/HEMS_ROTA.csv"
        self.service_path = f"{data_folder_path}/service_dates.csv"
        self.callsign_path = (
            f"{actual_data_folder_path}/callsign_registration_lookup.csv"
        )
        self.rota_times = f"{actual_data_folder_path}/rota_start_end_months.csv"

        self.params_df = pd.read_csv(self.params_df_path)
        self.rota_df = pd.read_csv(self.rota_path)
        self.service_dates_df = pd.read_csv(self.service_path)
        self.callsign_registration_lookup_df = pd.read_csv(self.callsign_path)
        self.rota_start_end_months_df = pd.read_csv(self.rota_times)

It is a bag of five DataFrames. Note the split: the parameters and service_dates.csv come from data_folder_path (outputs of the run), whereas the rota, callsign lookup and rota month bounds come from actual_data_folder_path (inputs, see Streamlit App and Scenario Parameters). Because __init__ is unwrapped, a missing file raises FileNotFoundError straight out.

TrialResults: the constructor is the pipeline

air_ambulance_des/class_simulation_trial_results.py:197-231

        # Add extra useful columns to run results
        self.enhance_run_results()
        # Add extra useful columns to the daily availability data
        self.update_daily_availability_df()
        # Create self.call_df
        self.make_job_count_df()
        # Create self.hourly_calls_per_run
        self.get_hourly_calls_per_run()
        # Create self.daily_calls_per_run
        self.get_daily_calls_per_run()
        # Create simulation event duration breakdown and summary
        self.create_simulation_event_duration_df()
        self.summarise_event_times()
        # Create event_counts_df and event_counts_long
        self.get_event_counts()
        # Create theoretical available resource time dataframe
        self.calculate_available_hours(
            summer_start=int(
                self.simulation_inputs.rota_start_end_months_df[
                    self.simulation_inputs.rota_start_end_months_df["what"]
                    == "summer_start_month"
                ]["month"].values[0]
            ),
            summer_end=int(
                self.simulation_inputs.rota_start_end_months_df[
                    self.simulation_inputs.rota_start_end_months_df["what"]
                    == "summer_end_month"
                ]["month"].values[0]
            ),
        )

        self.make_utilisation_model_dataframe()

        self.get_missed_call_df()
        self.prep_util_df_from_call_df()

The order matters because each step reads attributes set by an earlier one. make_job_count_df needs enhance_run_results; make_utilisation_model_dataframe needs total_avail_minutes from calculate_available_hours. Nothing enforces this.

make_job_count_df collapses the event log to one row per patient per run. Since hems_result, outcome, vehicle_type, callsign and registration are only filled on later events (see Calls and Patient Journeys), it back-fills them within each (P_ID, run_number) group and then keeps only the arrival row:

air_ambulance_des/class_simulation_trial_results.py:621-631

        # TODO - see what we can do about any instances where these columns remain NA
        # Think this is likely to relate to instances where there was no resource available?
        # Would be good to populate these columns with a relevant indicator if that's the case

        # Reduce down to just the 'arrival' row for each patient, giving us one row per patient
        # per run
        self.call_df = run_results_bfilled[
            run_results_bfilled["time_type"] == "arrival"
        ].drop(columns=["time_type", "event_type"])

        self.call_df.to_csv("data/call_df.csv", index=False)

A call with no resource has no callsign anywhere in its rows, so the back-fill leaves NaN. get_UNATTENDED_calls_per_run relies on that, and the TODO above flags the open question of what to do with these NaNs. Note the side effect: call_df is written to data/call_df.csv, a path relative to the current working directory.

Theoretical available hours: the utilisation denominator

Utilisation is time in use divided by time the resource was rostered. calculate_available_hours builds the denominator by walking every day from warm_up_end_date to sim_end_date and adding each rota line’s minutes, using the summer or winter columns depending on the month bounds from rota_start_end_months.csv.

air_ambulance_des/class_simulation_trial_results.py:345-368

        # Scenario 1: Shift is fully within one day (e.g., 7:00 to 19:00)
        if start_hour < end_hour:
            # Check if this shift is active on current_date (it always is in this logic,
            # as we are calculating for the current_date based on its rota)
            minutes_for_callsign_on_date = (end_hour - start_hour) * 60
        # Scenario 2: Shift spans midnight (e.g., 19:00 to 02:00)
        elif start_hour > end_hour:
            # Part 1: Minutes from start_hour to midnight on current_date
            minutes_today = (24 - start_hour) * 60
            minutes_for_callsign_on_date += minutes_today

            # Part 2: Minutes from midnight to end_hour on the *next* day
            # These minutes need to be added to the *next day's* total for this callsign
            if date_idx + 1 < len(
                daily_df
            ):  # Ensure there is a next day in our df
                next_date = daily_df.index[date_idx + 1]
                minutes_on_next_day = end_hour * 60

                daily_df.loc[next_date, callsign] = (
                    daily_df.loc[next_date, callsign] + minutes_on_next_day
                )

        daily_df.loc[current_date, callsign] += minutes_for_callsign_on_date

An overnight shift (start after end) credits 24 - start hours to today and end hours to the next day, which is why the loop indexes date_idx + 1. The final day’s overnight spill is dropped. Equal start and end hours credit zero minutes.

The docstring says servicing is taken into account, and service_df is loaded and merged with the callsign lookup, but a search of the method finds no later use of it: the minutes are computed from the rota alone. Treat the denominator as “rostered minutes”, not “rostered minus servicing”.

The method then writes three files as side effects (data/daily_availability.csv, data/monthly_availability.csv, data/daily_availability_summary.csv), again relative to the working directory.

Utilisation

make_utilisation_model_dataframe pivots the resource_use and resource_use_end events to one row per use, fills missing ends with the sim end date and missing starts with the warm-up end, then sums minutes per callsign.

air_ambulance_des/class_simulation_trial_results.py:563-580

self.utilisation_df_overall = self.utilisation_df_per_run.groupby(
    ["callsign", "vehicle_type"]
)[["resource_use_duration"]].sum()

self.utilisation_df_overall["resource_use_duration"] = (
    self.utilisation_df_overall["resource_use_duration"] / self.n_runs
)

self.utilisation_df_overall = self.utilisation_df_overall.reset_index(
    drop=False
).merge(self.total_avail_minutes, on="callsign", how="left")

self.utilisation_df_overall["perc_time_in_use"] = (
    self.utilisation_df_overall["resource_use_duration"].astype(float)
    /
    # float(_processing_functions.get_param("sim_duration", params_df))
    self.utilisation_df_overall["total_available_minutes_in_sim"].astype(float)
)

Per-run durations are summed over runs, divided by n_runs, then divided by the single-run availability. The code comment elsewhere says availability is assumed identical across runs, which holds because the rota is fixed for a trial.

Missed jobs

Missed jobs are defined by event_type == "resource_request_outcome" rows whose time_type is "No Resource Available". The normalised string annualises both counts using sim_duration_days:

air_ambulance_des/class_simulation_trial_results.py:2304-2322

total_calls = len(
    self.run_results[
        (self.run_results["event_type"] == "resource_request_outcome")
    ]
)

# print(f"==get_perc_unattended_string - total calls: {total_calls}==")

try:
    perc_unattendable = num_unattendable / total_calls

    sim_duration_days = float(get_param("sim_duration_days", self.params_df))

    total_average_calls_missed_per_year = (
        num_unattendable / self.n_runs / sim_duration_days
    ) * 365
    total_average_calls_received_per_year = (
        total_calls / self.n_runs / sim_duration_days
    ) * 365

get_missed_call_df does the same grouped by care_cat and time_type, recoding REG with heli_benefit == "y" to "REG - Helicopter Benefit", and produces the per-run breakdown and the min/mean/max summary the plots use. There is also a second, older route, get_UNATTENDED_calls_per_run, that counts NaN callsigns instead. The two should agree but are computed from different columns.

Comparing with history

HistoricResults.__init__ is a long run of “set None, then call the loader” pairs, each loader a pd.read_csv on historical_data/<name>.csv or historical_data/calculated/<name>.csv. The calculated/ files are the simulated baseline produced by the offline pipeline in Distribution Fitting. The constructor comment explains the purpose: some quantities (such as patients who would have benefited from enhanced care but did not get it) cannot be observed, so they are simulated with historic rotas.

air_ambulance_des/class_historic_results.py:73-94

        ########################################
        # 'Calculated' historical data         #
        ########################################
        # There is various data that we don't have - for example, we don't
        # know how many patients who would have benefitted from enhanced care
        # failed to receive it, by the very nature of those jobs
        # Therefore we simulate it to the best of our ability during the fitting
        # process using historical rotas and the model, effectively running a
        # 'base case'/'current situation' that we can use for comparisons against
        # different scenarios

        self.run_params_used = None
        self.get_run_params_used()

        # Historic rotas and scheduling details
        self.historic_rota_df = pd.read_csv(historic_rota_df_path)
        self.historic_callsign_df = pd.read_csv(historic_callsign_df_path)
        self.historic_servicing_df = pd.read_csv(historic_servicing_df_path)

        # Calculated/simulated historical comparisons
        self.SIM_hist_params_missed_jobs_care_cat_summary = None
        self.get_SIM_hist_params_missed_jobs_care_cat_summary_df()

TrialResults never imports HistoricResults. The historical object is passed into the plot method that needs it, which then reaches into its attributes:

air_ambulance_des/class_simulation_trial_results.py:2649-2660

for num_idx, callsign_str in enumerate(x_axis_categories):
    if (
        callsign_str
        in historical_results_obj.historical_utilisation_df_summary.index
    ):
        row = historical_results_obj.historical_utilisation_df_summary.loc[
            callsign_str
        ]

        min_val = row["min"] / 100.0  # Convert percentage to 0-1 scale
        max_val = row["max"] / 100.0
        mean_val = row["mean"] / 100.0

PLOT_SIMULATION_utilisation_summary draws the simulated bars, then, for each callsign that also appears in historical_utilisation_df_summary, overlays a min-to-max rectangle and a dotted mean line. That summary is built in make_RWC_utilisation_dataframe (RWC is not expanded anywhere in the code; it presumably means “real-world comparison”, i.e. built from historic data), which recomputes theoretical availability per month from the historic rota files in the same way as above.

The app wires all of this together like so:

app/model.py:245-275

try:
    simulation_inputs = SimulationInputs(
        data_folder_path="data", actual_data_folder_path="actual_data"
    )
    print("simulation input object created successfully")

except FileNotFoundError:
    print("data folder or actual_data folder not found")

try:
    historical_data = HistoricResults(
        historical_data_folder_path="historical_data",
        historic_rota_df_path="tests/rotas_historic/HISTORIC_HEMS_ROTA.csv",
        historic_callsign_df_path="tests/rotas_historic/HISTORIC_callsign_registration_lookup.csv",
        historic_servicing_df_path="tests/rotas_historic/HISTORIC_service_dates.csv",
    )
    print("historical data object created successfully")

except FileNotFoundError:
    print("historical data folder or historic rotas not found")

try:
    trial_results = TrialResults(
        simulation_inputs=simulation_inputs,
        historical_data=historical_data,
        run_results=pd.read_csv("data/run_results.csv"),
        daily_availability_data=pd.read_csv("data/daily_availability.csv"),
    )
    print("trial results object created successfully")
except FileNotFoundError:
    print("run results not found")

Extending this

Add a metric or plot. Add a method to TrialResults (it is wrapped automatically, since the decorator walks cls.__dict__). If it needs a derived table, build it in a make_*/get_* method and call that from the constructor sequence at lines 197-231, after anything it reads. Name plot methods PLOT_* and text/Streamlit helpers display_* or get_* to match the convention. Then call it from the relevant tab in app/model.py.

Add a historical comparison. Add a get_historical_<name> loader to HistoricResults, initialise the attribute to None in __init__ just before calling it, and make sure the CSV is written by DistributionFitUtils (the historical_* methods, see Distribution Fitting). If you add a calculated/ file it must also be produced by run_sim_on_historical_params. Pass the HistoricResults instance into your plot method as the existing plots do.

Add an input file to SimulationInputs. Add the path and pd.read_csv in __init__, and make sure whichever stage writes it (the run in Simulation Runs and Replications or the setup page) does so before the results page runs.

Change what counts as a missed job. The definition is repeated: get_perc_unattended_string, get_perc_unattended_string_normalised, get_missed_call_df, get_UNATTENDED_calls_per_run and the historical equivalents all filter independently. Change them together, and check the event strings against those emitted in Resource Allocation.

Tests. This abstraction has no tests. None of tests/ imports TrialResults, HistoricResults or SimulationInputs. The repo documents pytest from the project root (and pytest -m <marker> for subsets; see tests/README.md), which will at least confirm the simulation still produces an event log these classes can read, but it will not exercise this code. For a change here, build a small run_results DataFrame by hand, construct TrialResults in a script run from the repo root, and check the derived attributes. The working directory matters because of the relative data/ paths.

Sharp edges

Silent None. Any exception in a wrapped method yields None plus a printed message (or a Streamlit error box). get_AVERAGE_calls_per_run is a live example: it calls self.get_calls_per_run(self.call_df), but get_calls_per_run takes no argument, so the call raises TypeError and the method always returns None.
Constructor continues after failures. Because __init__ calls wrapped methods, a failure in, say, calculate_available_hours leaves total_avail_minutes as None and the utilisation step then fails in turn. The first printed message is the real cause.
Working-directory coupling. TrialResults writes data/call_df.csv, data/daily_availability.csv, data/monthly_availability.csv and data/daily_availability_summary.csv. HistoricResults writes historical_data/calculated/theoretical_availability_historical.csv. All are relative paths, so the code only works when started from the repo root (as the app is).
Stale availability input. app/model.py reads data/daily_availability.csv and passes it in as daily_availability_data, but calculate_available_hours later in the constructor overwrites that same file. The daily_availability_df used by PLOT_daily_availability therefore comes from whatever was on disk before this run, which may be a previous scenario’s. This follows from the call order and is not documented.
Hard-coded summer in the historic path. In HistoricResults.make_RWC_utilisation_dataframe, the nested is_summer ignores its argument, reads current_date from the enclosing loop, and hard-codes months 4 to 9 (class_historic_results.py:324-325). TrialResults instead takes the bounds from rota_start_end_months.csv. If you change the summer months, the simulated side follows and the historic side does not.
Mismatched message. In get_perc_unattended_string (class_simulation_trial_results.py:2278-2279) and get_perc_unattended_string_normalised (:2324-2327), the test < 0.01 (1%) prints “(< 0.1%)”. The string should read “(< 1%)”.
Vehicle-grouping assumption. make_SIMULATION_utilisation_headline_figure classes a callsign group with one member as “solo car” and with two as “helicopter backup car”, and says in a comment that this may not hold forever.
Cost. calculate_available_hours and its historic twin loop over days times rota rows with .loc writes. Long simulations with many rota lines are slow, and this runs on every results-page build.
Dependence on the Streamlit runtime. _processing_functions.py imports streamlit at module level, so even headless use of these classes needs it installed.

Design notes

Documented.

Refactor to class-based results: commit 8b393b5 (2025-10-07) is described as a major interim refactor to TrialResults and HistoricResults. Commit 0704f3d restored the first page of output and five follow-ups restored the remaining plots one tab at a time. Commit 7ae9c47 improved the layout and clarity of the historic class.
Daily availability data flow: commit 3039d72 changed how daily availability data enters the app and TrialResults, and 81982cc made sure it is assigned to a class attribute.
Streamlit Cloud indexing bug: commits cfa31e0 and a7684b9.

Inferred. The state of the code suggests the refactor is unfinished. Markers say “NOT YET CONVERTED TO OO” (line 2407) and “convert to use historical class” (line 2741), and a few methods take historical_results_obj or historical_data_class as an argument instead of using self.historical_data. The decorator presumably exists so that one broken plot cannot take down a results page with a dozen tabs; the cost is the silent None behaviour above. Computing everything in __init__ presumably trades construction time for cheap plot calls, since Streamlit reruns the script on every interaction. No commit or doc states either reason.

Limitations and open questions

Why the model makes simplifying assumptions

Every simulation model rests on assumptions and simplifications, and it has to. No model can be a perfect copy of reality. Adding ever more detail does not fix that, and can make things worse: each extra detail is one more thing to get wrong, so an over-complicated model can drift further from reality, not closer. Judge the items below, and any change you are tempted to make, against whether the assumption is reasonable for the questions asked of the model. Closing every gap with reality is not the aim, and extra fidelity can add more error than it removes.

TODO markers in class_simulation_trial_results.py:

Line 471 (repeated near the end of the file): add a check that missing resource-use start/end times are only back-filled for calls at the edges of the model, not for errors elsewhere.
Line 621: decide what to do with columns that remain NA after back-filling.
Lines 931 and 1180: pull historical data from the historical class rather than elsewhere.
Line 2226: confirm the difference between run_results and event_log_df.
Line 2407: a block of plot code not yet converted to the object-oriented layout.
Line 2741: convert to use the historical class.

Issues open as of 2026-10-19: #217 (utilisation breakdown plot breaks when one helicopter/car pairing is removed), #215 (car utilisation metrics break when one car is removed), #176 (no formal tests for the utilisation denominator), #165 (discrepancies between metrics and historical data) and #244 (a copyable summary table of key metrics).

See the issue tracker for the current state.

Conclusion

Next, in Distribution Fitting, we look at the offline pipeline that writes the historical_data/ files these classes read.