LLM sampling stunts
Inference-time-scaling considered unreasonably accurate
2026-07-30 — 2026-10-13
Wherein Sequential Monte Carlo Methods Are Applied to LLM Inference, \(N\) Candidate Trajectories Are Reweighted by Constraints, and the Resulting Particle-Like Filtering Is Contrasted Against the Manual, Lineage-Collapsing Technique Known as Looming.
I’ve noticed a number of interesting sampling tricks for LLMs that seem related. The category in my mind is something like sampling from LLMs in ways that resemble classic particle methods, i.e., using the kinematics of multiple hypothetical trajectories to inform the next step(s). So! A new notebook!
1 Sequential Monte Carlo Steering for LLMs
The thing that first drew my attention was the Sequential Monte Carlo (SMC) steering trick for LLMs (Lew et al. 2023): instead of decoding one continuation—or keeping a fixed beam—we keep a small population of \(N\) candidate continuations, score them against a desired constraint or reward, and then repeatedly resample promising candidates as generation proceeds. This can enforce constraints or steer semantic properties at inference time without retraining the base model, and it should look reminiscent of particle filtering except the likelihood is a dog’s breakfast and generally intractable in an annoying way.
Relatedly, LLaMPPL (Lew et al. 2023) claims to be a Probabilistic Programming Language for LLMs based on this idea. It uses Sequential Monte Carlo (SMC) methods to perform probabilistic inference over the outputs of LLMs, which seems logical for prefix sampling; I’m a bit confused about how it works for infill sampling, although they claim both.
Basic idea: We treat a completion \(y = (y_1,\ldots,y_T)\) as a trajectory sampled from the stochastic base model \(p_\theta(y\mid x)\), assuming that it comes from something similar to, but not the same as, the target distribution that we want to hit,
\[ \pi(y\mid x) \propto p_\theta(y\mid x)\,R(y) \]
where \(R(y)\) is a score—e.g., “is valid JSON,” “satisfies this regex/grammar,” “answers both prompts,” “is non-toxic,” or “meets a programmatic verifier.”
We…
- Maintain \(N\) partial completions.
- Sample the next token (or tokens) for each from the “base” LLM (which in SD-SMC might be a different model entirely).
- Give each continuation an incremental weight based on how well it performs under the constraint.
- When weights become concentrated, resample: copy high-weight trajectories and discard weak ones.
- Continue until completion; return a high-weight sample or sample from the final weighted population.
The distinction from simple rejection sampling is that SMC doesn’t wait until the entire response is finished to discover failure. We can prune bad paths early and concentrate compute on promising ones.
The “trick” is worthwhile because sometimes we can write a prefix-aware potential—a function that scores a partial answer, not merely a final answer.
Examples:
- Strict JSON: incremental parses can bail early from impossible prefixes
- Prompt intersection: Likelihood or task score under each of several prompts
- Other stuff that doesn’t seem very practical to me.
For hard constraints, a particle that makes the remaining task impossible gets weight zero. For soft preferences, the weight rises smoothly to interpolate between “worse” and “better” paths.
The original SMC-steering formulation interprets constrained language generation as posterior inference in a discrete sequence model. It reports capabilities including infilling, syntactic constraints, and “prompt intersection,” at a computational cost comparable to beam search.
LLM vibe-coded pseudo-code example:
particles = [("", 0.0, cache_i) for i in range(N)] # text, log_weight, KV cache
for t in range(max_new_tokens):
proposed = []
for text, logw, cache in particles:
token, logp, cache2 = sample_next_token(model, prompt + text, cache)
text2 = text + decode(token)
# potential should assess the prefix / remaining feasibility
delta = log_potential(text2, t + 1) - log_potential(text, t)
proposed.append((text2, logw + delta, cache2))
particles = proposed
weights = softmax([logw for _, logw, _ in particles])
if effective_sample_size(weights) < N * 0.5:
particles = systematic_resample(particles, weights)
particles = [(text, 0.0, cache) for text, _, cache in particles]
answer = select_or_sample_final(particles)A common formulation uses a sequence of potentials \(\phi_t(y_{1:t})\), with the incremental correction:
\[ \log w_t = \log w_{t-1} + \log \phi_t(y_{1:t}) - \log \phi_{t-1}(y_{1:t-1}) \]
That incremental difference prevents us from repeatedly counting the same “goodness” evidence at every token. In practice, this is not the true target potential, because that is almost always super weird in token space.
There are a few adjacent “SMC for LLMs” ideas that have a similar shape.
- SMC steering
- use particle filtering to steer toward constraints or a reward at inference time, as discussed above. SMC speculative decoding (SMC-SD)
- drafts from a cheaper model, score blocks under an expensive target model, then reweights/resamples rather than rejecting drafts token-by-token. The aim is faster inference while retaining close target-model behaviour (Emara et al. 2026) Self-consistency
- sample many complete reasoning trajectories and majority-vote the answer. This is not sequential resampling, but people sometimes loosely describe it as a particle-like trick. This one has the shape of, and relevance to, Reasoning LLMs where it is called maj@\(k\). Power-SMC
- sample from a sharpened sequence distribution \(p_\theta(y\mid x)^\alpha\), using parallel particles and token-level importance weights/resampling. It is positioned as a low-latency, training-free alternative to serial Metropolis–Hastings-style sampling (Azizi et al. 2026) of (in the model’s opinion) higher likelihood completions. NGL I cannot even work out what this one is supposed to do.
2 Looming
I have just been schooled on the prior art: the cyborgists have been doing population-based steering of language models since 2020 under the name looming. HT Theia Vogel.
Loom is Janus’s tree-based writing interface to base models, which is to say models that continue a text as opposed to answer a chat turn. The original is pyloom; there are descendants such as Loomsidian and Conjecture’s Bonsai, more on the cyborgism wiki. The procedure, in Janus’s description, is “manual iterative rejection sampling”. The operator (loomer, let us say) generates \(N\) continuations from the current node, a paragraph or less each, reads them, picks the one we like, and generates \(N\) more from there. Janus recommends \(N\) from 5 to over 100, depending on how fussy the passage is and the loomer’s patience. The interface keeps every rejected branch, so we can go back into history and replay to get different outcomes. (Similarities to fan-out in mathematical proving left as an exercise). Figure 2 shows a small tree from Janus’s own examples.
In the vocabulary of this notebook, this is a particle method with a human defining the selection process instead of a potential. The proposal is generated by a base model \(p_\theta\), as in the bootstrap version of SMC steering. The weight function is “black box”; i.e. it is whatever goes on in the loomer’s head. In terms of the pseudo-code above, we replace systematic_resample with “\(N\) copies of the one I clicked on”.
3 Looming vs SMC
Damn but that resembles SMC. There are a few differences however.
3.1 Looming focuses on one lineagge
SMC in the default forward mode resamples \(N\) survivors from \(N\) candidates and ideally keeps several distinct ancestries alive, because collapsing onto a single ancestor is indicative of failure for most purposes. Looming collapses onto one ancestor at every step (Figure 3) — although the loomer can rewind and replay if desired. Anyway, per default, we are sampling from a different statistical object.
Split the text into blocks \(b_1,\dots,b_K\) between branch points and suppose that the loomer picks candidate \(i\) with some Boltzmann-rational preference \(g_k(b_{1:k}^{(i)})\) (which is not what a loomer would typically claim to be doing AFAICT) but it is fun to imagine. For large \(N\), looming draws from the locally normalized distribution
\[ q(y) = \prod_{k=1}^K \frac{p_\theta(b_k\mid b_{<k})\,g_k(b_{1:k})}{Z_k(b_{<k})}, \qquad Z_k(b_{<k}) = \mathbb{E}_{p_\theta}\left[g_k(b_{1:k})\mid b_{<k}\right], \]
whereas SMC with the same incremental weights targets the globally normalized
\[ \pi(y) \propto \prod_{k=1}^K p_\theta(b_k\mid b_{<k})\,g_k(b_{1:k}). \]
Their ratio we can call the distortion,
\[ D(y) = \frac{q(y)}{\pi(y)} = \frac{Z}{\prod_{k=1}^K Z_k(b_{<k})}, \qquad Z = \mathbb{E}_{p_\theta}\left[\prod_{k=1}^K g_k(b_{1:k})\right], \]
where \(Z\) is the normalizing constant of \(\pi\), so the only part that depends on the path is \(1/\prod_k Z_k(b_{<k})\). \(Z_k(b_{<k})\) measures how good the options were at step \(k\). SMC ‘remembers’ that information, in that a particle whose children are mostly duds ends up with few descendants. Looming throws that information away. All \(N\) candidates at a branch point share one parent, so \(Z_k(b_{<k})\) is the same for all of them and cancels out of the choice; the current prefix never competes against a rival parent prefix. So \(D\) is large on completions that pass through prefixes where the options were poor, and looming over-weights them. Suppose only 1 continuation in 100 from here is any good. Looming attributes that one continuation the whole probability mass of the step, as if all 100 had been good; SMC would weight the prefix down by a factor of 100 against a prefix whose continuations were all good. Drawing more candidates per branch point does not help: \(N\) only controls how closely each pick matches \(p_\theta g_k / Z_k\), so as \(N\to\infty\) looming converges to \(q\), not \(\pi\). Looming still finds good text, in that every block it keeps scores well under \(g_k\); what it cannot do, by selection alone, is favour a prefix because that prefix leads somewhere good.
The most common case of this distortion in practice is constrained decoding, where we need the output to be, say, valid JSON or a match for a regex. The standard method, token masking, works one token at a time: it sets to zero the probability of every next token after which no valid completion exists, renormalises the probabilities of the tokens that remain, and samples from those. Outlines and llama.cpp’s grammars work this way. Token masking is looming with one-token blocks and \(g_k\) a 0/1 validity check, except that it needs no candidates: the model already gives a probability for every token in its tokenizer vocabulary, so masking the invalid ones and renormalising gives each step’s distribution exactly. In the \(N\to\infty\) case, an exact sampler of \(q\). Here \(Z_k\) is the probability the model put on the allowed tokens, so the over-weighted completions are the ones that pass through prefixes where the model mostly wanted to say something invalid. A two-token example shows the size of the effect (Figure 4). The first token is a or b, equally likely, and both are allowed. After a, only 10% of the model’s next-token probability lands on valid tokens; after b, 80% does. Conditioned on validity, \(\pi\) starts with a about 11% of the time (\(0.05/0.45\)). Masking cannot see at the first step that a leads to few valid continuations, so it picks a half the time and then forces a valid token, putting a at 50%, a distortion of about 4.5.
a x gets 0.11. Masking rescales within each step, so a x inherits all of a’s 0.5.
The output is valid but drawn from \(q\) rather than from \(p_\theta\) conditioned on validity, and correcting that is the stated motivation for SMC steering (Lew et al. 2023; Loula et al. 2025).
There is an interesting special case where the distortion vanishes (HT Claude for pointing this out). If \(g_k = \psi_k/\psi_{k-1}\) with \(\psi_k(b_{1:k}) = \mathbb{E}_{p_\theta}[R(y)\mid b_{1:k}]\), the expected final score given the prefix, then every \(Z_k = 1\), so \(D \equiv 1\) and \(q = \pi\). That \(\psi_k\) is the optimal twist function of twisted SMC. Zhao et al. (2024) try to learn that from data. This gives us a fun interpretation of the role of a loomer, as a twist function: Their job is to judge a paragraph by where it is likely to lead, not by how it reads now. I do not know how well people do this, but I suspect we humans are not terrible at it, insofar as we solve problems like this every time we flap our gobs. The invective that streams from my cake hole may not be globally optimal, but it certainly seems to go to controllable places.
3.2 Looming is not supposed to have a state target
SMC steering starts from explicitly given \(R(y)\), and the challenge is finding prefix potentials that approximate it. Specifically, the loomer assumes no particular Boltzmann-rational preference is guiding the search \(g_k(b_{1:k}^{(i)})\). A loomer has only the prefix potential, and it is not required to be consistent from one step to the next. Janus says they expect whatever purpose they started with to itself mutate/branch along the way. In that sense there is no posterior for the procedure to be “wrong” about, and the guarantees that make SMC attractive to statisticians (consistency, unbiased estimates of the normalizing constant) are not especially relevant to cyborgists. As such, maybe we should think of looming as closer to search than to sampling. There turns out to be some mileage in this perspective, which we come back to in a moment.
Janus attempts to quantify such curation: Choosing 1 of \(N\) equiprobable completions applies \(\log_2 N\) bits of optimization, and bits add across choices. AFAICT the particle-filter equivalent is \(\log_2(N/\mathrm{ESS})\) (Effective Sample Size) bits per resampling, since \(N/\mathrm{ESS}\) is the factor by which the weights concentrate the population. The ESS < N/2 rule in the pseudo-code says “resample once we have accumulated about 1 bit”. Loaming with \(N=8\) takes ESS to 1 and injects 3 bits of information every paragraph.
The tree is kept. SMC in the default forward mode is forward-only; a discarded particle is gone, and the history of the survivors is of interest only as a diagnostic. A loom stores the whole genealogy and extensions over the default allow us to “rewind” and “replay”, walking back up the tree and back down to alternative limbs, branching again from an earlier node when the current line goes nowhere fun. Backtracking is, as such, an interactive, manual fix for the greediness of looming. It makes looming look more like tree search than like filtering.
3.3 Loom branch points follow the model’s uncertainty
SMC resamples when the weights degenerate (concentrating on a small number of options), which is to say when the constraint has said something informative. Janus’s adaptive branching puts branch points in according to their more idiosyncratic logic, e.g. where the model’s own next-token distribution is spread out, or where a particular sampled token had low probability. Figure 2 is an example: one candidate per step through a challenge to the traditional, then five at once. One argument for this is that forcing a branch (do loomers force them to be distinct?) where the model is 99% sure of the next token only produces incoherent alternatives. I assume that some hip flavour of adaptive SMC does such things, but I have not seen it.
There is some work to unify looming and SMC. JD Pressman’s MiniHF is one galaxy-brained example. Pressman argues that because picking branches by hand is tiring, a loomer will want to distill their judgement into a reward model to do it for them. MiniHF’s Weave algorithm does that, running Monte Carlo tree search against an evaluator trained to approximate the loomer’s preferences, and by the author’s account it ends up close to Tree of Thoughts (Yao et al. 2023). Once the human has been replaced by a learned evaluator we seem to be doing something like SMC steering with a learned potential, give or take the choice of search algorithm. c.f. InFerActive (Hwangbo et al. 2025) which uses SMC with adaptive resampling to pre-populate the tree that a human evaluator then explores.
If we wanted to take the parallel totally seriously, we could combine them completely: devise a loom that keeps several lineages alive and asks the operator for rough weights on a handful of candidates each step. That would be a particle filter with a human likelihood. It would cost more reading per step but OTOH would buy back the global normalization, which is to say, estimate the partition function. Or is that taking the analogy too far? If, as Janus says, the loop is more exploratory than inferential, does it even make sense to worry about global normalization? Anyway, that is a side quest for someone with more side quest time than I.