Comments

Sunday, December 11, 2016

Two more little provocative pieces

I ran across two more things of interest.

The first is this post on Gelman's blog that reviews a recent paper in PNAS suggesting that the standard stats used to interpret fMRI findings are very very unreliable ("the general message is clear: don't trust FMRI p-values"). Gelman provides what seems to me a reasonable set of comments on all of this, including another discussion of the perverse incentives favoring statistical abuse. However, there is another issue that gets shorter shrift. It appears that even seasoned practitioners have a very hard time applying the techniques correctly (unless we make the silly assumption that most everyone using FMRI over the last 30 years is a fraud). This suggests that we ought to be very skeptical about any stats based report about anything. What the recent replication problems indicate is that even the best labs have a weak grasp of their stats tools.

Coming from a field which is often lectured on the  primitive nature of its data collection techniques,
I admit to experiencing quite a bit of very pleasant schadenfreude reading that the biggest problem in science today seems to be coming from just the techniques that my own field of interest has done without.  IMO, linguistics has done very well despite eschewing statistical sophistication, or indeed statistical crudeness. Of course I know the response: the right use of these stats techniques is what linguistics needs. My reply: first show me that the techniques can be reliably applied correctly! Right now it seems that this is far from obvious.

Indeed, it suggests a counter: maybe the right position is not to to apply the hard to apply technique correctly but to figure out how to get results that don't rely on these techniques at all. Call this Rutherford's dictum: "If your experiment needs statistics, you ought to have done a better experiment." One of the happy facts about most of linguistics is that our experiments, informal as they are, are generally speaking so good as to not require stats to interpret the results. Lucky us!

The second post is an interview with Freeman Dyson. It is short and fun. He says threes things that I found provocative and I'd be interested to hear opinions on them.

The first is his observation that great teachers are ones that can "find the right problem for each student, just difficult enough but not too difficult." I think that this is indeed one important mark of a great graduate mentor, and it is not something that I myself have been very good at. It also focuses on something that we often tend to take for granted. The capacity to generate good solvable problems is as important, maybe more important, that being able to provide solutions to said problems. Getting the problem "right" is more than half the battle, IMO, but I suspect that we tend to identify and value those that do this less than we should.

Second, Dyson rails against the PhD as a useful academic hurdle. He never received one and considers himself lucky never to have been required to. He thinks it antiquated, too arduous, and too intellectually disruptive.

Up to a point I agree. Certainly the classical thesis which develops a single topic over 300 pages with extensive critical review of the literature is more aimed at fields where the book is the primary research vehicle. Many places have long since replaced the book with the "stapled dissertation" where several research papers on possibly diverse topics are a thesis. This does not mean that a long form single topic thesis is a bad idea, only that a paper-oriented dissertation is a legit option. What the long form provides that the stapled essays don't is an opportunity to take a broader view of the discipline in which one is becoming a professional. Once one is a professional then until one attains senior status, thinking big is often (always?) frowned upon. This might be the only chance many get to see forests and not just trees. That said, I'd be curious to know what my younger colleagues think. And what if anything could replace the PhD that would be fair and useful.

Last point Dyson raises is where originality comes from. His vote, ignorance.
First of all, it helps to be ignorant. The time when I did my best work was when I was most ignorant. Knowing too much is a great handicap.
One of the complaints that older people always have about their younger colleagues concerns how little they know. Don't they know that we've done this before? Don't they know about so and sos research? Don't they know anything about the field before [put in very recent date here]? At any rate, what Dyson notes is that knowing too much may well be a problem. In fact, growing knowledge, rather than loss of energy coming with age, maybe what slows down senior scholars.

Syntacticians have had an antidote for this until recently. Chomsky used to change the landscape every decade or so, unnerving past students and emboldening young'uns. When junior you loved this. If senior you grumped. If Dyson is right, what Chomsky did was a great service for the field for he made it possible to be legitimately ignorant: things have changed so being erudite was not that important. Dyson's view is that ignorance is not so much bliss as liberating, allowing one to think about issues in new ways. Is he right? I'm not sure, but then look how old I am.
 

10 comments:

  1. With respect to the statistics and fMRI, I agree that the methods are definitely being used wrong. However, I think that the critics are wholly off-base in making statements like "These results question the validity of some 40,000 fMRI studies ". This is because we have massive priors concerning the function of different brain areas, as there are usually dozens of studies with very similar manipulations finding similar results in the same brain areas, and neuropsychological data that adds even more information. This varies depending on the experiment and the brain areas, but I think the far greater problem in neuroimaging is the lack of theory rather than the stats.

    For instance, take sentence processing. If you do a contrast of sentences > word lists, you will undoubtedly activate the anterior temporal lobe. I can easily think of ten studies off the top of my head that have replicated this effect, with very different kinds of materials, languages, modalities (speech vs. sign language), tasks, scanners, etc., and there are probably dozens of such studies. Many (maybe most) of these studies have not applied the correct stats. But who cares - I am not about to doubt that there is something about sentences that activates the anterior temporal lobe relative to word lists. Likewise for sentences with noncanonical sentence structure - a contrast of something like object relative > subject relative will undoubtedly active the left and right inferior frontal gyrus and probably the posterior temporal lobe. I am not about to doubt that there is something about noncanonicity that makes the inferior frontal gyrus activate, regardless of whether the stats were applied perfectly.

    The big problem is theory - nobody knows what is going on with respect to language and the brain, and I think that the hypotheses are only starting to get worse with time. The obsession with stats only drives people further away from theory, which means they are only going to miss the real problem even more. Maybe I am wrong about this with respect to other cognitive neurosciences, but I think that the general point that every fMRI study that is published adds information to our priors about brain areas, and this helps massively with inference to function.

    ReplyDelete
    Replies
    1. " the general point that every fMRI study that is published adds information to our priors about brain areas, and this helps massively with inference to function."
      These aren't priors then but posteriors, and that is exactly the problem.

      Delete
    2. The results of previous studies form the priors for future studies, do they not?

      My point is that there is convergence of results - that greatly strengthens our confidence in these results, does it not?

      Delete
    3. Well no. Not if the statistics in each are weak: that and the file-drawer effect may mean that an entire theory is based on noise.

      In social psychology, people are coming to realize that just having 10 studies that confirm some phenomena is not enough. You do a funnel plot and you can see that there is no real effect.

      It is fine to update your beliefs with each study, but that is done on the assumption that the statistical claims are correct. If they don't do the Bonferroni corrections (or whatever the problem is), then each study provides no reason to update your beliefs at all. So all your "priors" will be wrong.

      Delete
    4. I think that people that have not dealt with neuroimaging data come away with this impression. For instance, Pallier et al. (2011) showed that a set of 6 peri-sylvian brain regions showed enhanced activity for degree of linguistic structure, and that 3 of these regions showed this effect for both jabberwocky and natural stimuli, and the other regions only showed the effect for natural stimuli. We performed a modified replication of this study, showing nearly the exact effects: enhanced activity for more structure in the same network, and the same 3 regions showed the parallel jabberwocky/natural effect while the others showed the effect only for natural. I find it completely ridiculous that this happened by chance.

      This type of thing happens constantly in neuroimaging data for all sorts of paradigms. Maybe this is different for social psychology, but I am not about to believe these findings are all spurious. I AM about to believe that people have very little idea about what it means.

      Delete
    5. That observation does not conflict at all with the general methodological point I was making.

      Delete
  2. Second post containing interview details seems good because I am always fond of knowing new and successful people and knowing all about their entire life. dissertation writing services

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. But I LIKE theses because so many of them function very well as textbooks for professors ... the bibliographies tend to be more extensive and the presentations of the basic ideas slower and more careful than in journal articles.

    ReplyDelete
  5. Re Dyson's third point, consider the conclusion of an interview with molecular biologist Seymour Benzer:

    Aspaturian: I have one more question. The Indian physicist [Subrahmanyan] Chandrasekhar—he and Willy [William A.] Fowler [Institute Professor of Physics, emeritus] shared the Nobel Prize in 1983—is famous for completely changing the direction of his field every ten years or so. He’s said he does it partly for the reasons you enunciated and partly because he feels people tend to
    stagnate in their own fields. Did you consciously think that as well when you moved from one field into another? Was it sort of to rev yourself back up?

    Benzer: I’ll say it again. In every case I switched, it’s because of interest in something different. But I can’t divorce the excitement of that interest from the fact that it also meant getting away from the trappings of another subject that was getting too big. So subconsciously, that’s surely part of the motivation, partly an escape as well as an attraction. When a subject develops very thoroughly, there’s too much you have to know. It gets sort of overwhelming. So that’s why I was saying the big attraction is starting something new and being very stupid about it. Ask stupid questions, and you often get amazing answers.

    (http://resolver.caltech.edu/CaltechOH:OH_Benzer_S)

    ReplyDelete