01
The problem
Imagine you're modelling something that was measured many times on the same subjects — a patient's lab results across several visits, or a fish stock sampled year after year. Rows that come from the same subject tend to be more alike than rows picked at random: if one visit for patient 7 came in high, the next one probably will too. An ordinary neural network has no way to represent that. It treats every row as if it came from a brand‑new, unrelated subject, so it either shrugs off that similarity as noise or, worse, quietly memorizes "patient 7" as if it were a meaningful pattern rather than a coincidence of the sample.
Statisticians have a standard fix for this called a mixed‑effects model: split the prediction into a fixed effect (a shared, population‑level pattern that applies to everyone) plus a random effect (a small, subject‑specific adjustment layered on top). The architecture in the next section does exactly that — it uses a neural network for the flexible, population‑level part, and adds a random‑effects term just before the final prediction to account for whichever subject or group an observation belongs to.
Fig 1
Figure · Model architecture
The architecture of a multi‑layer GNMM
Layer indices run 1 → … → L from input to output | click any node to trace its signal path
Click a node on the canvas to trace its connections · click an empty area or "All" to reset
FIGURE The architecture of a multi-layer GNMM. The output μib depends on inputs xi, weights w(l), and biases d(l) through a series of nested nonlinear activation functions g(·) for layers l = 1, …, L. The linear predictor for the random effects ziTbi is included in the final layer of the network.
Fig 2
What does the random effect actually do?
Drag τ, the cohort‑spread, to fan the cohorts apart
Each thin line is one cohort's curve · the thick line is the shared population curve
Schematic, drawn from the model's functional form rather than fitted values — illustrative, not a reproduction of any figure in the paper.
02
How it's fit
Having the right architecture is only half the problem — it still has to be fit to data. Normally that means two headaches: hand‑deriving gradients for every weight and bias (worse as the network gets deeper), and mathematically integrating over the random effects, which rarely has a clean closed‑form solution. The paper's real contribution is a workflow, built on Template Model Builder (TMB), that automates both. Click through the four stages below.
You write one equation
TMB differentiates it
Random effects integrated out
An optimizer finds the fit
You're looking at:
Fig 3
Where this sits among the alternatives
Four ways to model the same clustered, curved relationship — only the last one has both properties
GLM
One curve for everyone, fixed shape. Clustering is ignored, and curvature is limited to the link function.
GLMM
Clustering is handled by a random intercept, but every cohort's curve is forced into the same symmetric shape.
Neural network
Any shape it likes, including shapes that fit noise — and still just one curve for every cohort at once.
NMM — this paper
Flexible where the data asks for it, clustered by cohort, and constrainable — here, monotonic in age (§ Figure 4).
Illustrative sketches, not digitized data.
03
Does it work? (Simulation)
To test the fitting workflow, the paper simulates 100 subjects with a handful of repeat observations each, built from six binary predictors combined into three hidden, XOR‑style nonlinear patterns — the kind of structure a plain linear model can't recover but a neural network can. Using leave‑one‑out cross‑validation, they compared the TMB implementation against an earlier approach that required deriving its objective function and gradients by hand (Mandel et al. 2023, referred to below as the "manual" approach).
The simulation's theoretical noise floor is ≈0.0025 — TMB's error sits right next to it. Illustrative summary of the paper's reported ranges, not a reproduction of Figures 2–3, which plot these values as curves against the random-intercept variance τ.
Fig 4
Real‑world case: female American plaice
The paper also applies the framework to a real fisheries problem: predicting the probability that a female American plaice has reached sexual maturity, as a function of its age, using 58 cohorts (1958–2015) split across three regions — 31,427 training observations, 23,392 for validation, and 24,104 held out for testing. Cohort is used as the random effect — but not as 58 independent, unrelated adjustments: the paper links the cohort effects with a first‑order autoregressive (AR(1)) process across birth years, so a cohort's fitted deviation is pulled toward its neighbors' rather than estimated in isolation, reflecting that nearby birth years tend to share similar environmental conditions.
Biologically, maturity probability should never decrease with age. But nothing stops a plain neural network from fitting a wobbly, non‑monotonic curve if that happens to fit the training noise slightly better. So the paper adds a monotonicity constraint: a mathematical penalty that grows whenever the fitted curve's slope goes negative anywhere, tightened step by step (following the Karush–Kuhn–Tucker conditions) until the curve never decreases. Because TMB differentiates through this constrained objective automatically too, adding the constraint required no extra hand‑derived math.
Without a constraint
Predicted maturity could wobble down as fish get older — physically nonsensical.
Monotonic NMM
Constrained so the curve never decreases — matches how maturity actually works.
Illustrative sketch, not a reproduction of Figure 4 or the paper's actual data.
On held-out test data, the monotonic network and a standard GLMM (with a linear age effect) landed almost exactly together on accuracy:
Accuracy alone doesn't obviously favor the more complex model. But the network's curve showed mild asymmetry in how quickly different cohorts matured — a shape a GLMM's symmetric logistic curve can't represent by construction. The extra flexibility captured a real biological detail, even though it barely moved the headline number.
See this exact model running: query any cohort/age, with real confidence intervals and a chat panel (grounded in this walkthrough via retrieval) that explains the results.
Open the live app →04
Takeaways
The architecture itself isn't the novelty — mixing a neural network with random effects has been tried before. What this paper removes is the hand‑derivation bottleneck: describe the negative log‑likelihood, and let TMB's automatic differentiation and Laplace approximation handle the gradients and the random‑effects integration exactly, constraints like monotonicity included. That should make richer mixed‑effects architectures practical for a lot more applied problems than they currently are.