That is why I won't blog about this research quite yet (except for the shameless self-promotion above) and instead focus on the talks I heard, rather than the one I gave. Don't get me wrong, many of them were very interesting to me on a technical level; some of them even pierced my 90s habitus of acerbic cynicism and got me a bit excited. Quite generally, a fun time was had by all. But the talks made me aware of a gapping hole in my understanding of the field, a hole that one of you (I believe we have some readers with serious modelling chops) may be able to plug for me: Just what is the point of cognitive modelling?
Advantages of ModellingDon't get me wrong, I understand why modelling can be useful, and human cognition is one of the most interesting things to study. But from where I'm standing, the two simply do not fit together. Or rather, I think what people are trying to do cannot be done by modelling and instead requires an approach grounded in mathematics --- theorems and proofs.
Let's first outline some clear advantages of developing computational models:
- Proof of concept
Your idea might sound batshit crazy, but if it can be turned into a working model that works on a wide range of problem instances, that demonstrates a certain level of sophistication. So maybe we shouldn't dismiss it right away.
- Getting results
Models are great from a utalitarian point of view: you have a problem, and your model solves it for you. You want to know if tomorrow's picnic will be a pleasant sunshine siesta or a rainy rancor trigger? Let's feed the data into our weather model and see what it has to say.
- Testing for problems
A model can test a much bigger set of data than any group of humans can, so they're a great way of hunting for holes in your theory.
Why do we Study Cognition?Just because modelling has advantages doesn't mean that its advantages are of much use for a given area: a wine cooler is a nifty thing to keep in your kitchen, but it's pretty worthless at a rehab clinic. In the case of cognitive modelling, it really depends on what you are trying to achieve. For computational linguists, cognition might be just another pesky quirk of humans that makes language needlessly complicated for computers. They just need an efficient method for constructing discourse representations, making semantic associations, and whatever else you need to simulate human-like understanding and usage of language. Given such a tool, they also need to verify that it works for a wide variety of industrial applications. Both issues are covered by advantages 2 and 3 above, so modelling does indeed fit the bill.
But I, for one, do not care that much about cognition as a problem for non-sentient machines. I am interested in how human cognition works and, most importantly, why it doesn't work in different ways. More boldly: what makes human cognition human?
If that's your main interest, it's not enough to show that some model works for a given problem. The important questions are:
- Is it guaranteed to succeed on for every problem for a given problem space? In formal terms: is it sound and complete?
- Why does its solution to the problem actually work --- what does that tell us about the problem?
- How is the workload distributed across the assumptions and techniques your model incorporates?
- Are there different ways of doing it? Can we translate between different models in an automatic fashion?
But even if you're not ready to take that extreme stance, a model by itself is still a very boring thing and provides little insight. What matters is its relation to other models --- those that succeed as well as those that fail. Now since that is an infinite class, the standard strategy of designing and testing models via simulations won't be able to answer any of these issues. If you need to understand the structure of an infinite object, you are firmly within the realm of theorems and proofs. And I don't see any of that in the cognitive modelling community.
The Argument Against Theorems and ProofsI suppose one reply to this little rant of mine would be that in an ideal world a proof-based approach would indeed be preferable, but the problem is simply too complex to be studied in this fashion. Just like you can't prove many things about the behavior of leaves blowing in the wind, the system is too complex to be studied in this fashion. So rather than fruitlessly toiling away for hundreds of years, we accept the limitations of the approach and sacrifice a little bit of rigor for a huge increase in data coverage.
To this I have two replies, one personal and one more objective. On a personal level, one of my guiding credos is that a question that cannot be answered in a satisfying manner is not worth asking. So if the problems the cognitive modellers are trying to solve are indeed too complicated to be studied in an insightful manner (according to my high standards of what counts as insightful), then they simply aren't worth studying scientifically (the engineering angle is still perfectly viable, though). Pretty black and white, but a simple(minded) view of things is comforting once in a while.
More generally, though, my hunch is that the reply itself relies on a false equivocation. The problems themselves may indeed be complicated, but your models are mathematical objects. So simplify them, figure out the math for the simple cases, and keep expanding your results until you reach the level of the original models again. In many cases we are dealing with the construction of special cases of hypergraphs that are evaluated over a probabilistic semiring. That's not exactly the epitome of mathematical inscrutability. Why, then, don't we see any work along those lines? Or is this actually a big research area, and the only thing to blame is my ignorance of the field?