Chapter 2: Simulation Runs and Replications

How runSim and parallelProcessJoblib derive per-run seeds, run seeded DES_HEMS replications across cores, and how the per-run CSVs are collated.

Written for: Intermediate Coder Confident Coder Decision-makers Domain experts General public

Chapter 1 ended with the Run button handing session-state values to the run functions. This chapter follows that call into the package.

What it is

air_ambulance_des/des_parallel_process.py is the package’s run API. runSim builds one DES_HEMS model for a run number and seed, calls run() and returns that run’s results DataFrame. parallelProcessJoblib turns a single master seed into one per-run seed per replication, then fans runSim out over CPU cores with joblib. collateRunResults merges the per-run CSVs into data/run_results.csv, which is the file the results classes read (see Chapter 7).

The contract callers rely on:

Inputs: durations are in simulated minutes (the model’s unit of time), warm_up_time is in the same unit, and the other parameters come from the scenario the app wrote to actual_data/.
Outputs are files, not return values. Every run writes data/output_run_N.csv (N starts at 1); the first run also writes data/run_params_used.csv via write_run_params. The return value of parallelProcessJoblib is a list of DataFrames that no in-repo caller uses.
Determinism: the same master_seed and total_runs give the same per-run seeds, whatever n_cores is.

Warm-up period: simulated minutes at the start of a run whose events are run but not recorded. It lets the model reach a realistic state (resources mid-job, servicing under way) before results begin.

Replication: one independently seeded run of the whole model. Results are only meaningful as a set of replications.

The use case for this chapter is the one every scenario comparison starts with: twelve replications of a two-year scenario from the app’s Run button or from run_parallel_process.py, producing one combined event log.

flowchart LR
    A["app/model.py<br/>or run_parallel_process.py"] --> B[parallelProcessJoblib]
    B -->|"SeedSequence(master_seed).spawn(n)"| C[runSim x n]
    C --> D[DES_HEMS.run]
    D --> E["data/output_run_N.csv"]
    C -->|"run == 0"| F["data/run_params_used.csv"]
    E --> G[collateRunResults]
    G --> H["data/run_results.csv"]
    D --> I["data/all_results.csv"]

Figure 1: Files and functions involved in a parallel run

The call generator and sampling the run relies on are covered in Chapter 4 and Chapter 3.

How it works

Deriving one seed per replication

air_ambulance_des/des_parallel_process.py:294-319

    # Generate a number of uncorrelated seeds that will always be the same given the same
    # master seed (which is determined as a parameter)
    # We start with a SeedSequence from the master seed, and then generate a number of
    # child SeedSequences equal to the total number of runs
    seed_sequence = SeedSequence(master_seed).spawn(total_runs)
    # We then turn these seeds into integer random numbers, and we will pass a different seed
    # into each run of the simulation.
    seeds = [i.generate_state(1)[0] for i in seed_sequence]

    # Run the simulation in parallel, using all available cores
    return Parallel(n_jobs=n_cores)(
        delayed(runSim)(
            run=run,
            total_runs=total_runs,
            sim_duration=sim_duration,
            warm_up_time=warm_up_time,
            sim_start_date=sim_start_date,
            amb_data=amb_data,
            random_seed=seeds[run],
            save_params_csv=save_params_csv,
            demand_increase_percent=demand_increase_percent,
            activity_duration_multiplier=activity_duration_multiplier,
            print_debug_messages=print_debug_messages,
        )
        for run in range(total_runs)
    )

SeedSequence(master_seed).spawn(total_runs) yields total_runs child sequences. Each is collapsed to a single integer with generate_state(1)[0] and passed to runSim as random_seed. The seed for replication i is therefore a function of (master_seed, i) only. Nothing depends on which worker picks up the job, so results do not change with n_cores. Because SeedSequence.spawn keys children by index (numpy behaviour, not repo code), asking for more runs should not change the seeds of the earlier ones. tests/test_reproducibility.py checks same-seed and different-seed behaviour end to end.

Parallel(n_jobs=n_cores) with the default n_cores=-1 uses every core. The generator expression passes every setting explicitly, so runSim never reads session state or globals for its configuration.

One replication

air_ambulance_des/des_parallel_process.py:167-197

    start = time.process_time()

    print(f"{Utils.current_time()}: Run {run + 1} of {total_runs}")
    logging.debug(f"{Utils.current_time()}: Run {run + 1} of {total_runs}")

    # print(f"Sim start date is {sim_start_date}")
    daa_model = DES_HEMS(
        run_number=run,
        sim_duration=sim_duration,
        warm_up_duration=warm_up_time,
        sim_start_date=sim_start_date,
        amb_data=amb_data,
        random_seed=random_seed,
        demand_increase_percent=demand_increase_percent,
        activity_duration_multiplier=activity_duration_multiplier,
        print_debug_messages=print_debug_messages,
    )
    daa_model.run()

    print(
        f"{Utils.current_time()}: Run {run + 1} took {round((time.process_time() - start) / 60, 1)} minutes to run"
    )
    logging.debug(
        f"{Utils.current_time()}: Run {run + 1} took {round((time.process_time() - start) / 60, 1)} minutes to run"
    )

    # SR NOTE: This could cause issues if we decide to use 1 as the starting number of runs
    if (run == 0) and (save_params_csv):
        write_run_params(daa_model)

    return daa_model.results_df

runSim is deliberately thin. It constructs DES_HEMS, calls run(), and returns daa_model.results_df. The timing uses time.process_time(), which is CPU time of the worker process, not wall-clock time, so the “took N minutes” message is not a stopwatch reading.

Parameter recording is gated on run == 0 and save_params_csv. The SR NOTE code comment above it (a maintainer’s note-to-self convention in this repo) is a warning to keep in mind: the check assumes runs are numbered from zero.

What `DES_HEMS` does with the seed and the run number

air_ambulance_des/des_hems.py:56-71

        self.run_number = run_number + 1  # Add 1 so we don't have a run_number 0
        self.sim_duration = sim_duration
        self.warm_up_duration = warm_up_duration
        self.sim_start_date = sim_start_date

        self.random_seed = random_seed
        self.random_seed_sequence = SeedSequence(self.random_seed)

        self.print_debug_messages = print_debug_messages

        self.utils = Utils(
            master_seed=self.random_seed_sequence.spawn(1)[0],
            print_debug_messages=self.print_debug_messages,
        )

        self.utils.setup_seeds()

Two things matter here. First, self.run_number = run_number + 1: the model’s run number is one-based, so the file for the first replication is output_run_1.csv while runSim still sees run == 0. Second, the integer seed is wrapped in a second SeedSequence, and a spawned child of that goes to Utils, whose setup_seeds() makes the named random streams. How those streams are derived is the subject of Chapter 3. HEMSAvailability is also handed the parent SeedSequence as master_seed, but it only stores it and does not use it. Its random draws (ad hoc unavailability, know_heli_benefit) come from the shared Utils instance and so from the streams described in Chapter 3.

The warm-up period and the end of the run

air_ambulance_des/des_hems.py:764-772

        if self.env.now >= self.warm_up_duration:
            self.store_patient_results(results)

    def store_patient_results(self, results: dict) -> None:
        """
        Adds a row of data to the Class' `result_df` dataframe
        """

        self.results_list.append(results)

The warm-up is not a flag and not a separate phase. Events during the warm-up are simulated but simply not appended to results_list, because add_patient_result_row checks self.env.now >= self.warm_up_duration (Chapter 4 covers the rows themselves). The commit that removed the old warm_up_period boolean (“rows are simply not added during the warm-up period”, 3bf122e, 2025-04-02) documents this.

air_ambulance_des/des_hems.py:812-832

    def run(self) -> None:
        """
        Function to start the simulation.

        """
        self.debug(
            f"HEMS class initialised with the following: run {self.run_number}, duration {self.sim_duration}, warm-up {self.warm_up_duration}, start date {self.sim_start_date}, demand increase multiplier {self.demand_increase_percent}, activity duration multiplier {self.activity_duration_multiplier}"
        )

        # Start entity generators
        self.env.process(self.generate_calls())

        # Run simulation
        self.env.run(until=(self.sim_duration + self.warm_up_duration))

        # Convert results to a dataframe
        self.convert_results_to_df(results=self.results_list)

        # Write run results to file
        self.write_all_results()
        self.write_run_results()

run() starts the generate_calls process, runs the SimPy environment until sim_duration + warm_up_duration, converts the row list to a DataFrame and then writes two files. The recorded window is therefore exactly sim_duration long and starts at warm_up_duration.

Writing and collating

air_ambulance_des/des_hems.py:795-810

    def write_run_results(self) -> None:
        """
        Writes the content of `result_dfs` to csv files that contains only the results from the
        single run

        Note that this cannot be done in a similar manner to write_all_results due to the impacts of
        the parallisation approach that is taken with joblib - depending on process timings it can lead to
        not all
        """

        self.results_df.to_csv(
            f"{Utils.RESULTS_FOLDER}/output_run_{self.run_number}.csv",
            header="column_names",
            index=True,
            encoding="utf-8-sig",
        )

The docstring is the documented reason for per-run files: appending to one shared file does not work reliably under joblib, because it depends on process timing. write_all_results (just above, at lines 782-793) still appends to data/all_results.csv from every worker; nothing in the collation path reads that file.

air_ambulance_des/des_parallel_process.py:200-211

def collateRunResults() -> None:
    """
    Collates results from a series of runs into a single csv
    """
    matching_files = glob.glob(os.path.join(Utils.RESULTS_FOLDER, "output_run_*.csv"))

    combined_df = pd.concat([pd.read_csv(f) for f in matching_files], ignore_index=True)

    combined_df.to_csv(Utils.RUN_RESULTS_CSV, index=True, encoding="utf-8-sig")

    for file in matching_files:
        os.remove(file)

Collation globs output_run_*.csv in Utils.RESULTS_FOLDER, concatenates them with ignore_index=True, writes run_results.csv (index column included), and deletes the per-run files. The run identity survives in the data because each row already carries a run_number column from the model, not because of anything collation does.

air_ambulance_des/des_parallel_process.py:214-230

def removeExistingResults(remove_run_results_csv=False) -> None:
    """
    Removes results from previous simulation runs
    """
    matching_files = glob.glob(os.path.join(Utils.RESULTS_FOLDER, "output_run_*.csv"))

    for file in matching_files:
        os.remove(file)

    all_results_file_path = os.path.join(Utils.RESULTS_FOLDER, "all_results.csv")
    if os.path.isfile(all_results_file_path):
        os.unlink(all_results_file_path)

    if remove_run_results_csv:
        run_results_file_path = os.path.join(Utils.RESULTS_FOLDER, "run_results.csv")
        if os.path.isfile(run_results_file_path):
            os.unlink(run_results_file_path)

removeExistingResults is the counterpart you must call before a fresh batch, because collation takes whatever matches the glob. It always removes output_run_*.csv and all_results.csv, and removes run_results.csv only on request.

The three callers

parallelProcessJoblib is called from the app, from the CLI script and from tests. The app also has a second path, which matters for seeds.

app/model.py:156-178

            for run in range(st.session_state.number_of_runs_input):
                run_results = runSim(
                    run=run,
                    total_runs=st.session_state.number_of_runs_input,
                    sim_duration=float(st.session_state.sim_duration_input * 24 * 60),
                    warm_up_time=float(st.session_state.warm_up_duration * 60),
                    sim_start_date=datetime.combine(
                        datetime.strptime(
                            st.session_state.sim_start_date_input, "%Y-%m-%d"
                        ).date(),
                        datetime.strptime(
                            st.session_state.sim_start_time_input, "%H:%M"
                        ).time(),
                    ),
                    amb_data=st.session_state.amb_data,
                    demand_increase_percent=float(st.session_state.overall_demand_mult)
                    / 100.0,
                    activity_duration_multiplier=float(
                        st.session_state.activity_duration_multiplier
                    ),
                    print_debug_messages=debug_messages,
                    random_seed=st.session_state.master_seed + run,
                )

When platform.processor() == "", which the code comments describe as a guess that the app is on Streamlit Community Cloud, the app skips joblib and loops over runSim, using master_seed + run as the seed (see “Two seeding schemes” under Sharp edges). It then concatenates the returned frames and writes Utils.RUN_RESULTS_CSV directly (lines 187-190), so it does not use collateRunResults on this path. Commit f77f7b8 (“Changed how run results are saved when executing sequentially rather than in parallel”) is the documented change to that behaviour.

run_parallel_process.py is the command-line equivalent:

run_parallel_process.py:7-19

if __name__ == "__main__":
    removeExistingResults()
    # parallelProcessJoblib(1, (1*365*24*60), (0*60), datetime.strptime("2023-01-01 05:00:00", "%Y-%m-%d %H:%M:%S"), False, False, 1.0, 1.0, True)
    parallelProcessJoblib(
        12,
        (2 * 365 * 24 * 60),
        (0 * 60),
        datetime.strptime("2023-01-01 05:00:00", "%Y-%m-%d %H:%M:%S"),
        False,
        False,
        1.0,
        1.0,
    )

It calls removeExistingResults(), runs 12 replications of two years with no warm-up, amb_data=False, save_params_csv=False, and leaves master_seed at its default of 42. It does not call collateRunResults(), so it leaves output_run_N.csv files in data/ and writes neither run_results.csv nor run_params_used.csv.

Extending this

Add a run-level parameter (for example another multiplier). Touch these in order:

DES_HEMS.__init__ (des_hems.py:44-55) to accept and store it.
runSim and parallelProcessJoblib signatures, and the delayed(runSim)(...) call, which lists every argument explicitly.
write_run_params, if the setting should be recorded in run_params_used.csv. The DataFrame there is a hand-built dict, so a new key is not recorded automatically.
The app’s two call sites in app/model.py (sequential and parallel) and, if wanted, DEFAULT_INPUTS and the setup page (Chapter 1).
run_parallel_process.py, which passes arguments positionally. Inserting a parameter before existing positional ones shifts the meaning of the later arguments.

Change the parallel backend or worker count. Everything is in the Parallel(n_jobs=n_cores)(...) expression. Keep the seeds computed before dispatch, in the parent, so reproducibility stays independent of scheduling.

Change what is collated. Edit collateRunResults and keep Utils.RUN_RESULTS_CSV and the output_run_ filename pattern in step with DES_HEMS.write_run_results. They are linked only by a string pattern.

Tests. From the repo root (per tests/README.md):

Example usage (see tests/README.md)

pytest tests/test_reproducibility.py   # same seed, same results; different seed, different results
pytest tests/test_unittest_model.py    # includes the warm-up period behaviour
pytest -m backtest                     # compares a fixed scenario with the stored baseline

tests/conftest.py builds shared simulation fixtures using exactly the removeExistingResults(), parallelProcessJoblib(...), collateRunResults() sequence. test_backtest.py restores the data folder afterwards, but the README warns that other tests delete and rewrite data/run_results.csv and data/all_results.csv, so don’t run them while you care about a results file in data/. If a seed change is intentional, the README documents pytest -m backtest --update-backtest-baseline.

Sharp edges

Stale files poison collation. collateRunResults merges whatever output_run_*.csv exists. Skipping removeExistingResults() after an aborted batch mixes old and new replications. If nothing matches, pd.concat on an empty list raises ValueError.
Glob order is not sorted, so row order in run_results.csv is not guaranteed to follow run number. Anything that depends on order should sort.
Concurrent appends to all_results.csv. Every worker calls write_all_results, which checks os.path.isfile and then appends. The check and write are not atomic, so in principle the header could be written twice or rows interleaved (inferred from the code, not observed). This is why the per-run files exist; treat all_results.csv as unreliable in parallel runs.
Parameters and results are not linked. run_params_used.csv is written once, by run 0, from that model instance only. The SR NOTE in write_run_params records that no identifier ties it to the results and that a failed write could leave the two out of step. Note also that its sim_end_date is sim_start_date + sim_duration, which does not include the warm-up, whereas the model actually runs until sim_duration + warm_up_duration. Its summer and winter start dates are hardcoded strings (April and October), with a comment saying class_hems must be updated in step.
Two seeding schemes. Parallel: spawned per-run seeds. Sequential (hosted app): master_seed + run. Don’t compare outputs across the two.
The one-based file number versus the zero-based run argument (see above). Someone starting run from 1 would silently lose run_params_used.csv.
Unit conversions live in the caller. The app multiplies days by 24 x 60 and warm-up hours by 60 and passes floats. runSim’s docstring says “minutes or other time unit”, but the model assumes minutes throughout.
A hung run blocks the pool. Issue #233 (open as of 2026-10-19) reports the model hanging indefinitely if a servicing schedule cannot be resolved. Joblib has no timeout here, so one such replication stalls the batch.
Empty runs. convert_results_to_df swallows a KeyError on set_index("P_ID") (the TODO at des_hems.py:778). A run with no recorded rows therefore yields an empty frame without an error.

Design notes

Seed sequences instead of consecutive integers. Documented: commit 9a05b28 (2025-04-17) “Adjust random seeds to use seed sequences more robustly”, following 2af661f (2025-04-16) which introduced random number streams for activity times. The commented-out Utils.get_distribution_seeds call above the spawn code shows an earlier attempt at a different scheme. The later reason for keeping master_seed + run in the hosted path is not documented; presumably it predates the change.
Per-run files plus collation. Documented in the write_run_results docstring: shared-file appends are unreliable under joblib.
Warm-up by omission. Documented: 3bf122e.
Time in minutes. Documented: 1f646db (2024-10-25) adjusted the simulation unit of time to minutes.
Everything passed explicitly into delayed(runSim). Inferred: it keeps workers free of shared state, which suits joblib’s process-based workers.
Wrapping the integer seed in a new SeedSequence inside DES_HEMS. Inferred: it lets runSim stay a simple integer interface while the model still gets independent child streams.

Limitations and open questions

Why the model makes simplifying assumptions

Every simulation model rests on assumptions and simplifications, and it has to. No model can be a perfect copy of reality. Adding ever more detail does not fix that, and can make things worse: each extra detail is one more thing to get wrong, so an over-complicated model can drift further from reality, not closer. Judge the items below, and any change you are tempted to make, against whether the assumption is reasonable for the questions asked of the model. Closing every gap with reality is not the aim, and extra fidelity can add more error than it removes.

des_hems.py:778: TODO - Improve error handling here (the swallowed KeyError).
write_run_params carries the SR NOTE about linking parameter and results files with an identifier.
Open as of 2026-10-19: #183 (quantify the variation due to random seeds), #233 (hang on unresolvable servicing schedule), #165 (discrepancies against historical metrics), #245 (faster alternatives to pandas for processing run output).

Check the project’s issue tracker for the current state. For command-line usage from a user’s perspective, see the README.

Conclusion

Next, Chapter 3 opens up the Utils object each run builds from its seed and shows how the named random streams are made.