The interpretation of RV densities as point process intensities and vice versa

1 Basis function method for density

Let’s consider the case where we try to estimate this function by constructing it from some given basis of $p$ functions ${\varphi }_j:{\mathbb{R}}^d\to [0,\mathrm{\infty }),$ so that

$$f(x)=\sum _{j\le p}{w}_j{\varphi }_j(x)$$

and $\theta \equiv \{w_j{\}}_{j\le p}.$ We keep this simple by requiring $\int {\varphi }_j(x)dx=1,$ so that they are all valid densities. Then the requirement that $\int f(x)dx=1$ will imply that $\sum _j{w}_j=1,$ i.e. we are taking a convex combination of these basis densities.

Then the maximum likelihood estimator can be written $$\begin{array}{rl}\hat{\theta }& ={\arg \max }_{\{w_i\}}f(\{x_i\};\{w_i\})\\ & ={\arg \max }_{\{w_i\}}\sum _i\log \sum _{j\le p}{w}_j{\varphi }_j(x_i)\end{array}$$

A moment’s thought reveals that this equation has no finite optimum, since it is strictly increasing in each $w_j$. However, we are missing a constraint, which is that to be a well-defined probability density, it must integrate to unity, i.e. $$\int f(\{x\};\{w_i\})dx=1$$ and therefore $$\begin{array}{rl}\int \sum _{j\le p}{w}_j{\varphi }_j(x)dx& =1\\ \sum _{j\le p}{w}_j\int {\varphi }_j(x)dx& =1\\ \sum _{j\le p}{w}_j& =1.\end{array}$$

By messing around with Lagrange multipliers to enforce that constraint we eventually find $$\widehat{\{w_k\}}=\frac{\sum _i{\varphi }_k(x_i)}{\sum _i\sum _j{\varphi }_j(x_i)}.$$

2 Intensities

Consider the problem of estimating the intensity $\lambda$ of a simple, non-interacting inhomogeneous point process $N(B)$ on some compact $W\subset {\mathbb{R}}^d$ from a realization $\{x_i{\}}_{i\le n}$, and this counting function $N(B)$ counts the number of points that fall on a set $B\subset {\mathbb{R}}^d$.

The intensity is (in the simple non-interacting case — see Daley and Vere-Jones (2003) for other cases) a function $\lambda :\mathbb{R}\to [0,\mathrm{\infty })$ such that, for any box $B\subset W$, $$N(B)\sim \mathrm{Poisson}(\mathrm{\Lambda }(B))$$ where $$\mathrm{\Lambda }(B):={\int }_{B}d\lambda (x)dx$$ and for any disjoint boxes, $N(A)\perp N(B).$

After some argumentation about intensities we can find a likelihood for the observed distribution: $$f(\{x_i\};\tau )=\prod _i\lambda (x_i;\tau )\exp (-\int \lambda (x;\tau )dx).$$

Say that we wish to find the inhomogeneous intensity function by the method of maximum likelihood. We allow the intensity function to be described by a parameter vector $\tau ,$ which we write $\lambda (x;\tau )$, and we once again construct an estimate: $$\begin{array}{rl}\hat{\tau }& :={\arg \max }_{\tau }\sum _i\log f(x;\tau )\\ & ={\arg \max }_{\tau }\sum _i\log (\lambda (x_i;\tau )\exp (-{\int }_W\lambda (x;\tau )dx))\\ & ={\arg \max }_{\tau }\sum _i\log \lambda (x_i;\tau )-{\int }_W\lambda (x;\tau )dx\\ & ={\arg \max }_{\tau }\sum _i\log \lambda (x_i;\tau )-\mathrm{\Lambda }(W).\end{array}$$

3 Basis function method for intensity

Now consider the case where we assume that the intensity can be written in a ${\varphi }_k$ basis as above, so that $$\lambda (x)=\sum _{j\le p}{\omega }_j{\varphi }_j(x)$$ with $\tau \equiv \{{\omega }_j\}.$ Then our estimate may be written $$\begin{array}{rl}\hat{\tau }& :={\arg \max }_{\{{\omega }_j\}}f(\{x_i\};\{{\omega }_j\})\\ & :={\arg \max }_{\{{\omega }_j\}}\sum _i(\log \lambda (x_i;\tau )-\mathrm{\Lambda }(W))\\ & ={\arg \max }_{\{{\omega }_j\}}\sum _i(\log \sum _{j\le p}{\omega }_j{\varphi }_j(x_i)-{\int }_W\sum _{j\le p}{\omega }_j{\varphi }_j(x)dx)\\ & ={\arg \max }_{\{{\omega }_j\}}\sum _i(\log \sum _{j\le p}{\omega }_j{\varphi }_j(x_i)-\sum _{j\le p}{\omega }_j{\int }_W{\varphi }_j(x)dx)\\ & ={\arg \max }_{\{{\omega }_j\}}\sum _i(\log \sum _{j\le p}{\omega }_j{\varphi }_j(x_i))-\sum _{j\le p}{\omega }_j\end{array}$$ We have a similar log-likelihood to the density estimation case.

Under the constraint that $E(\hat{N}=N)$ we have $\sum _j{\omega }_j=n$ and therefore $$\hat{\tau }={\arg \max }_{\{{\omega }_i\}}\sum _i(\log \sum _{j\le p}{\omega }_j{\varphi }_j(x_i))-n.$$

Note that if we consider the points as a shifted density we find the result is the same as the maximum obtained by considering the points as an inhomogeneous spatial point pattern, up to an offset of $n$, i.e. ${\omega }_j\equiv n{w}_j.$

4 Count regression

From the other direction, we can formulate density estimation as a count regression; For “nice” distributions this will be the same as estimating the correct Poisson intensity for every given small region of the state space (e.g. (Gu 1993; Eilers and Marx 1996)). 🏗

5 Probability over boxes

Consider a box in $B\subset {\mathbb{R}}^d$. The probability of any one $X_i$ falling within that box, $$P(X_i\subset B)=E(\mathbb{I}\{X_i\subset B\})={\int }_{B}dF(x).$$

We know that the expected number of $X_i$ to fall within that box is $N$ times the probability of any one falling in that box, i.e. $$E(\sum _{i\le N}\mathbb{I}\{X_i\subset B\})=N{\int }_{B}dF(x)$$ and thus $$P(N(B)=k)=\frac{\exp (-\mathrm{\Lambda }(B))\mathrm{\Lambda }(B{)}^k}{k!}.$$ …Where was I going with this? Something to do with linear point process estimation perhaps? 🏗

6 Score function versus hazard function

The score function and log-hazard rates are similar beasts. We can exploit that in other ways perhaps, e.g. in a Langevin dynamics algorithm? But will we gain something useful from that? Does Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation leverage something like that?

7 Interacting point processes

Interacting point processes have intensities too which may also be re-interpreted as densities. What kind of relations are implied between the RVs which would have this “dynamically evolving” density? Clearly not i.i.d. But useful somewhere?

8 References