Crossing-risk study — Christopher Zheyu Li

The question

Crossing-safety programs have to decide where to look first. A ranking can use what a crossing is — its warning devices, road traffic and train traffic — and what has happened there before. I wanted to know how much the second part adds.

To isolate that, I compared the same model with and without each crossing's incident history, keeping everything else fixed.

Building the data

FRA publishes its crossing inventory as a history of revisions, not as yearly snapshots. So for each year I rebuilt what the inventory said on January 1: the latest revision before that date for every crossing recorded as open, public and at grade. That produced 1,526,612 crossing-years.

I then joined FRA incident reports to each crossing and year. A crossing-year counts as positive when at least one incident was reported there.

The history features — reports in the previous one, three and five years, time since the last report, and past casualties — only look backward, so no model ever sees the year it is predicting.

How it was tested

Models learned from 2014–2021, were tuned on 2022–2023, and were scored once on 2024–2025, data they had never seen: 251,028 crossing-years, 1.4% of them with a reported incident.

Two measures matter most: how well each model ranks crossings (average precision), and what share of reported incidents falls within each year's top-ranked 10% of crossings.

What it found

History helps. With it, the top 10% of crossings held 48.8% of the crossing-years with a reported incident; without it, 45.1%. Average precision rose from 0.077 to 0.101, an improvement of about a third.

The gain held up when the evaluation was resampled: the 95% interval for the difference in top-10% capture ran from 2.3 to 4.9 percentage points.

Model	Average precision	Brier score	Top-10% capture
Training prevalence	0.0139	0.01374	10.0%
Exposure logistic	0.0487	0.01350	37.8%
History-only logistic	0.0729	0.01330	38.2%
Infrastructure + exposure logistic	0.0541	0.01346	37.9%
Infrastructure + exposure HGB	0.0766	0.01329	45.1%
Infrastructure + exposure + history HGB	0.1012	0.01312	48.8%

Scroll the table sideways for Brier score and capture.

Average precision for the six recorded models, with crossing-cluster uncertainty intervals. — Ranking performance on the 2024–2025 test years, with uncertainty intervals.

Recorded capacity-capture comparison for annual crossing-risk rankings. — Share of reported incidents that fall within each year's top-ranked crossings.

Recorded probability calibration comparison for the retrospective evaluation. — How closely each model's predicted probabilities match the rates actually observed.

Limits

This is a look back, not a live forecast. Because the yearly data was rebuilt from revision dates, I can't confirm that every record was publicly available on the date the model assumes.

The study ranks where incidents were reported. It doesn't show which safety improvements work or how many incidents a ranking would prevent, and it doesn't compare against FRA's own accident-prediction model.

Reproducing it

The research package keeps the exact FRA queries and file hashes, the rebuilt yearly dataset, every model's predictions and the figures, so each number can be regenerated.

Tools: Python (pandas, scikit-learn), PyArrow and matplotlib.

AI tools helped with the code and the drafting; I directed the study and reviewed the results. The downloadable summary has the full technical detail.