May (1972) demonstrated that as the complexity of a system increases—measured by the number of species and the density of their interactions—the likelihood of the system being stable decreases when interactions are assigned randomly.
May’s Paradox: Real ecosystems are both complex and seemingly stable. May’s work challenged the prevailing notion that complexity begets stability. How does nature do it?
Naively, we might say that we only observe stable ecosystems because the unstable ones have already collapsed. We cannot quite make that as an evolutionary argument, though, because selection operates on the individuals within the ecosystem, not the ecosystem as a whole. There are more careful arguments to be made.
I know about this concept mostly because of how often it is referenced in certain financial stability papers. Oh, and I spent a summer in 2001 running ecological simulations for the mathematics program at ANU.
Look! Cosma Shalizi has recently updated his Stability and Complexity of Ecosystems notebook, also.