journalists and editors really ought to take more science and math courses. at least they should take more statistics. at the very least they ought to study Tufte (e.g., Envisioning Information) and read things like Huff's How to Lie with Statistics and Barnett's "How Numbers Can Trick You: The Six Deadly Sins of Statistical Misrepresentation" (Technolology Review, pp 38-45, October 1994).

so, today's New York Times gave an example of where they have. there is a timely article about Harvey R Miller, bankruptcy lawyer workin' the Lehman Brothers fiasco. accompanying this article is a graphic, drawn from data at BankruptcyData.com. they don't have this graphic, so i presume it's a New York Times production.

(image not necessarily to scale as appeared in New York Sunday Times)

anyway, this caught my eye. it's nice, but there are lots of ways to compare circles. the most honest and obvious way is to compare the circular area of one to another. it wouldn't be the first time a major news source messed up a graphic by failing to pay proper attention to area, nor are financial institutions immune. maps are the usual victim. for instance, someone might have scaled diameters proportional to losses, thinking that would suffice. they'd be wrong.

here, just looking, it didn't appear like the diameters of these circles were all scaled proportional to the sizes of bankruptcies. so, curious, i took four, and measured them:

Bankruptcy	diameter (cm)	amount (billion $)	area (cm2	ratio of areas (Lehman=1)	ratio of amounts (Lehman=1)
Lehman	3.0	691.1	7.07	1	1
WaMu	2.0	328.0	3.14	0.44	0.47
IndyMac	0.5	32.7	0.20	0.03	0.05
WorldCom	1.2	125.1	1.13	0.16	0.18

note i worked from the printed version of the Sunday Times as we are subscribers.

not too shabby. it may seem like a small thing, but when conveying information, these kinds of things are important. it's nice to see attention to detail respected.

this figure from Election Maps 2008 is evidence in favor of a hunch i have, that sharp edges in data graphics convey more significance to the brain than they should. blending is better, i suspect.

the first is an election map of counties with pro-McCain as red, pro-Obama as blue, but smoothed so close ties are purple.

the cartogram beneath it rescales areas to be in proportion to population. the traditional map misrepresents large sparsely populated areas of territory. this misrepresentation goes back at least as far as How to Lie with Statistics, in its comments on "The Darkening Cloud", a (Newsweek? Time?) graphic which purported to show how much of the annual national income was being consumed by taxes.

Pharyngula talks more about this.

i don't know math well, even if my life and work are all about it. surely, i am a perrenial student of mathematics, and even a big fan of it. my work is primarily about quantitative applications in software and in statistics. i'm trained as an engineer, with a strong bent towards numerical problems and those of embedded controls and software. but i don't grok deep math and proofs well, like those in real analysis (see also an interesting interactive). i very much respect mathematics. i think my relationship to the universe is mathematical, if anything. i think we don't do enough of it. i think it's badly misunderstood and misappreciated. my kids, Dave and Jeff, each know a good deal more about it than i do, and i am very proud of them.

one of the things about maths is that, like the PERL programmers say, there's more than one way to do it. so, while formal proofs might be out of my ken, there are other rigorous approaches to solving problems which aren't. you can, for instance, do numerical integration using Monte Carlo methods, even if i wouldn't recommend it, given the other techniques available out there. statistical approaches to problems can provide new insights, seeing things in probabilistic perspectives. there is such a thing as a statistical approach to problems, be they mathematical or engineering or computer sourced. it's a combination of technique and skeptical sense, an appreciation for good, repeatable data, and of data that, while it may have a lot of noise, doesn't show much bias.

i think, too, that knowledge of many technical fields, like statistics, is all about knowing their particular stories. in physical electromagnetism, it's knowing the standard story of two (infinitely thin) conducting wires 1 meter apart each carrying 1 ampere of current, its directions, and what the resulting force is between them. in fluid dynamics, it's understanding how, if you have two identically sized and porous tubes in contact for some finite range, carrying flowing stuff, if the flow is in the same direction and speed in both, the rate of percolation from one to the other is higher than you would think, and surely much higher than if the flows were in opposite directions. in statistics, it's things like the Central Limit Theorem or the Laws of Large Numbers or the Monty Hall problem or detecting diseases or intruders that are captivating and motivating stories. these are each paradigms for how certain problems should be approached, analyzed, and solved.

finally, four references, and an opinion.

• one reference is IMO a tremendous book, The Mathematical Experience, by P.J.Davis, R.Hersh, E.Marchisotto, G.-C.Rota
  
• the second reference is F.L.Ramsey, D.W.Schafer, The Statistical Sleuth. wanna know what it's like to do statistics? love that book!
  
• the third reference concerns the programming language R, but indirectly: Vincent Zoonekynd's wonderful Statistics With R.
  
• the last reference is J.A.Rice's Mathematical Statistics and Data Analysis. Rice tells more of these stories than any other text i know. it's therefore invaluable. and don't pay attention to its rating on amazon.com. trust me: it is a great book.
  

the United States has achieved much of what it has become during the last hundred years because of its appreciation for education, for scholarship, and especially for mathematics. i sense and fear in our current clime that "book learnin'" has been relegated to the realm of the impractical, uncorrelated with getting ahead in a job, in life, in what's perceived to be the real. mathematics isn't the only skill that is seeing less public interest: Americans cannot parse or understand speeches or rhetoric as well as ancient Athenians could, and that can only serve them ill. if the United States pursues its own form of burning of the library of Alexandria, American exceptionalism will be proved the lie, barbarians at the gate will triumph, and the American experiment declared a failure by history. knowledge and scholarship are not only the future, they are our only future. if we fail to pursue it, i have every confidence that they who follow us here will do so, changing the Constitution to suit, and lead us out from darkness. that shadow, in my opinion and as before, owes its being to the excessively powerful influence of Christianity. i'll have more to say about that someday. i have recently come to the opinion that Christianity, as practiced, is a horrible mistake, an instantiated misunderstanding of Judaism.

this entry isn't conceptually well-organized. accordingly, it's more like blog entries were supposed to be, at least in the heady days before professional bloggers existed. (ah, the unfettered dreams of youth!) so, it may be more interesting than the technical and political things i usually throw up here.

but it has some of that.

( bulk of post below the line )

here's another little but excellent lesson from the readership of that excellent weekly newsmag, The Economist.

Crime scene investigation

SIR – Peter Plotts compares the high murder rate in abolitionist Washington, DC—“an eye-popping 35.4” per 100,000 people—with the lower rate in his native Texas to belittle opponents of the death penalty (Letters, September 22nd). Such a selective use of crime statistics is easily countered. Boston is a city in a state without the death penalty, yet its murder rate in 2005 was lower than that of Dallas and Houston. And the murder rate for Richmond, a city in a state second only to Texas in its enthusiasm for executions, was an eye-popping 43. Ill-considered comparisons yield no useful conclusions about the effectiveness of the death penalty.

Andrew Marshall
Gaithersburg, Maryland

and there's more:

Never, never land

SIR – While it is true that abstinence from sex is the only sure-fire way of avoiding sexually transmitted diseases, the same holds true for any activity (“Time to grow up”, September 22nd). If I never drink, I don't run the risk of liver damage or alcoholism; if I never smoke, I do not risk getting lung cancer; if I never travel in a car, I will not risk being in a car accident; and if I never use stairs, I won't risk falling down a flight or two.

Actually, if I never did any activity and stayed on the ground floor of my house I would never be at risk of anything. But how dull life would be. With life comes risks, and it is up to all who educate children and teenagers to help them navigate those risks with intelligence and foresight—not command them simply to avoid all dangers.

Diana Camosy
Chicago

one of my favorite Web sites is Insultingly Stupid Movie Physics. Intuitor does an entertaining job tearing up plotlines and scenes from the most popular movies made. their dissection of the movie, Argmageddon is a classic. their slash of Pearl Harbor comes close, but Armageddon's takedown is better because people can learn so much more from the critique. i love the nose-down bombs business. Intuitor's review of Armageddon ends with the line, "Sometimes there's just no way to have a happy ending". i wish supporters of continued military intervention would learn that.

it might be useful for someone to take a similarly critical eye for science and mathematics reported in the general media. surely, Scott Berry's column "A Statistician Reads the Sports Pages" (PDF of table of contents) which runs in Chance shows how useful and educational the pricking of popular notions can be.

or perhaps the critical eye oughtn't limit its scorn to mathematics and science. the rationale offered by Russell Defreitas, suspect in a plot to bomb JFK Airport in New York, is singularly stupid. if this is the new face of al-Qaida, their "franchises", then they're trying to fasten with stripped screws. but noone in the media i have read or seen says so. by not commenting, they are enhancing and embellishing the gravity of the threat, let alone it's probability. and the FBI acts like they're stupid, too, because they apparently buy the entire package, at least as related by U.S. Attorney Mauskopf's "chilling" description.

but maybe the problem is terrorists, attornies, FBI, and others have seen too many movies with bad physics. or maybe they didn't pay attention to science class. or just maybe the learned, inquisitive minds at the FBI and elsewhere just don't make it to the policy ranks where, instead, ideology and loyalty are prized, irrespective of fact.

update 200706060024: well, at least drivelblog is talking about it.

revised 200706061238: "You're Pitiful" (5 MB MP3), by "Weird Al" Yankovic

update 200706170100: Portrait of the Modern Terrorist as an Idiot, by Bruce Schneier, one of the smartest people i know when it comes to matters of security.

some consequences of the recent Nor'easter which battered New England:



there's a good deal more information available on these processes.

global climate is a system with plenty of lags. and, so, Al Gore has it right.

not only is the recently released UN IPCC report alarming, it was, according to reports, toned down from what scientists wanted.

At one point, a U.S. representative, NASA's Cynthia Rosenzweig, filed a formal protest and left the building in which the talks were being held, only to return, make peace and talk in positive tones. Others talked about abandoning the process.

"There was no split in the science; they were all mad," said John Coequyt, who observed the closed-door negotiations for the environmental group Greenpeace.

But Yvo de Boer, the top climate official for the United Nations, countered that it was a "difficult choice." If it stayed the way scientists originally wrote it, some countries would not accept nor be bound by the science in the document," de Boer said.

(emphasis added)

update RealClimate talks about how media channels can "frame" discussion to make it appear a point is contested when it is not, especially to an audience which really doesn't understand.

the American Association for the Advancement of Science ("AAAS") has issued a new statement on climate change.

disclaimer: i am a member of the AAAS, as well as a member of the American Statistical Association.

(updated 2006-11-09, 1305 EST)


one thing people with a limited appreciation for mathematics suffer is the inability to see how incredibly complicated behavior or structure can be gotten from the repeated application of simple rules. indeed, that serves as one of my personal definitions.

take

KNOTS

for instance (4.1 mb Quicktime movie on the link).

this great column shows how they can result from dynamical systems related to the strange attractors of Lorenz. the possibility that the Earth climate system has such attractors is why a lot of knowledgeable folks worry about the possibly repercussions of climate change. (more here.) this doesn't, of course, only affect climate predictions, but most features of atmospheric and planetary phenomena. the point is, chaotic systems are not predictable, whether climate models are used or not.

but the knots are pretty in and by themselves.

update 2006-11-08, 1616 EST: not only are strange attractors possible, but there is evidence in the climate record such abrupt changes have happened in the (relatively) recent geologic past.

update 2006-11-09, 1305 EST: RealClimate debunks the climate is as it's always been implications of an article in The New York Times by William Broad, where Broad repeats the Helplessness Mantra, "More and more data, point to the Sun and stars as the dominant driver.”

RealClimate dabbles deeply in the intricacies of radiative forcings for climate, what scientists mean when they attribute warming to various causes, and how they analyze it.

NY's Republican Representative Sherwood Boehlert, who, despite his recent unfortunate support of the Military Commissions Act ("MCA"), has been a staunch supporter for science and scientific issues in the House, reports "rumors" that the Bush junta may be announcing a major initiative on climate change some time in the next two weeks. perhaps this may be part of a Rovian "October surprise".

Boehlert, by the way, has announced his retirement from Congress.

update 20061006 1350 EDT: y'know, of course, that if they do, it will all pomp, image, and no substance, with plenty of carveouts for their corporate friends and plans that will take years to realize. just look at the way W makes a big deal out of signing legislation while, at the same time, cutting its guts using Signing Statements minutes later.

... well, then so are the Republicans in Congress. both political parties have apparently suddenly decided it's real.

what a f#@kin' circus!

and to all the doubter dummies at Wunderground : welcome to the new normal!

(IIIb) The robot always takes into account all of the evidence it has relevant to a question. It does not arbitrarily ignore some of the information, basing its conclusions only on what remains. In other words, the robot is completely nonideological.

-- E T Jaynes, Probability Theory, The Logic of Science, section 1.7, page 19

my posting frequency is diminishing because i'm tooling a software project for my company, and because i've decided to invest a significant amount of time catching up on reading, including a careful study of E T Jaynes 2003 posthumous book, Probability Theory, The Logic of Science. with the latter i'm essentially relearning statistics, having been taught the classic Neymann-Pearson mysticism in college and subsequently, practicing it for many years. i did start a course in Bayesian statistics at Cornell but never finished (*), based upon Berger's rigorous tome, which was about as much a course in philosophy of statistics as anything else. i'm also working through Gelman, Carlin, Stern, and Rubin, Bayesian Data Analysis.

i want to mention two other things.

first, Bruce Schneier has a kickass issue of his Crypto-Gram Newsletter available. dated 15th September 2006, it has several must-read articles including "What Terrorists Want", "Details on the British Terrorist Arrest", "More Than 10 Ways to Avoid the Next 9/11", "Fifth Anniversary of September 11, 2001", and "USBDumper".

second, Joel Spolsky of Fog Creek Software has piqued many responses with a couple of recent entries in his newsletter.

in particular, some folks were questioning Fog Creek's wisdom in developing FogBugz the way Fog Creek chose to do, using a methodology which is out of present fashion. in particular, many seemed astounded that Fog Creek would commit some kind of mortal sin by developing their own language for the project, rather than indulge in lust for the LAMP stack or worshiping mindlessly at the altar of standards. some may also have been unhappy about his comments regarding the programming language, Ruby.

i offered this view, crossposted from "Has Joel Spolsky Jumped the Shark?" in a comment at September 14, 2006 05:48 PM:

i don't have time to look at Wasabi carefully and i raise my eyebrows at a suggestion that you can't do X in an interpreted language or, in this case, Ruby, except maybe for hard real-time things. but i understand why Joel Spolsky and company did Wasabi. to adapt an aphorism popular in a different community, "There's more than one way to do it."

in particular, setting aside OO and standardization fever and a lust for the LAMP stack, there is an old and strong school of development which looks at building applications in terms of devising just the right languages for its various parts and then implementing them. this is the school most specifically taught by Waite (ISBN 0134518985) but also practiced by many others, including Burge (ISBN 0201144506). this is the world of macro processing, techniques like the full and half bootstrap, the source of ideas like streams. it a world that list processing languages grew up in, like LISP. it's viable.

indeed, the only problem with it is that its practitioners need to know a lot, both in terms of data structures and analytical techniques. it fell out of popularity partly because it was beyond the skill or training of most people to master. how many can read and understand Burge's book? or do the problems in Knuth's second volume?

but if you have an in-house team with superb programming skills, why the heck not? just because the rest of the world doesn't do things that way? why should they care what other people think?

as i've written in print, software development is less engineering and more a social movement, one which big companies make lots of money from. actual progress in developing software is slow so, in addition to the winnings gleaned by the rotation of platforms due to progress in hardware (true engineering), marketing wisdom jumps on this or that language or bandwagon or methodology, simply so more software, more upgrades, more books, more courses devoted to their way of doing things can be hawked, evangelized, and sold. meanwhile, programmers tread water -- actually, they probably slowly sink -- as they try to cope with a never-ending stream of stuff all intended to do the same things, but with a different style. advocates for this or that methdology embrace it like a religion or, worse, a local football club where every meetup with a contender is fought to the death.

this is not a new phenomenon. when IBM was still king of everything having to do with computers there were those who saw that technological progress was being driven by marketing forces rather than doing better for people. it's worse now that computers have become primarily entertainment platforms, whether that's to feed an iPod or write a blog (like this one) or sync your handheld or cell, to play a hot new game or to watch a downloaded movie. sure, i know, there are many computers used to do serious work, including the engineering i so implicitly admire. but where's the money made?

there are better, smarter ways of doing software. in the rush for the Big New Thing we've lost a lot of what's already been discovered. maybe it was too poorly taught, or just too hard for people to grasp. perhaps the exigencies of getting a product out the door were too overwhelming for people to appreciate these matters. perhaps rate of progress itself killed off this knowledge, or maybe it was just capitalism which, despite its ideals and pronouncements, really does not do a good job serving the Market at times.

the prime political statement that sums up this frustration comes from that unrepentant gadfly, critic, commentator, and student of software, the late Edsger W Dijkstra, sometimes vilified, but always worth listening to, and most of the time correct:

Sad remark. Since then we have witnessed the proliferation of baroque, ill-defined and, therefore, unstable software systems. Instead of working with a formal tool, which their task requires, many programmers now live in a limbo of folklore, in a vague and slippery world, in which they are never quite sure of what the system will do to their programs. Under such regretful circumstances the whole notion of a correct program -- let alone a program that has been proved to be correct -- becomes void. What the proliferation of such systems has done to the morale of the computing community is more than I can describe. (End of sad remark.)

[page 202, A Discipline of Programming]

see also Dijkstra's "The end of computing science?", designated EWD1304-0.

===========================================

(*) that detour happened during a time of difficulty which perhaps i'll chronicle here some day.

thus spake The Economist , a periodical some corporate executives, economists, and policy wonks have described as "arguably, the best news magazine in the world". it is also a magazine with a viewpoint, a limited government, paleocon champion of the free market. yet, in a leader dubbed 'The Heat Is On', the editors of that esteemed publication throw their weight behind doing something and "America should lead the way". in an accompanying and well done survey, Economist journalist Emma Duncan explains how "the underlying calculation is fairly straightforward" despite the complexities of climate and its integrated systems.

then, of course, there is the reality, the damage BushCo has done to government and its agencies which would normally respond to this threat, not to mention the pansies in Congress. and the reality that Americans put them there. as my online friend luna_the_cat recently commented:

Pardon my deep, deep cynicism, but the American public will do absolutely nothing which adversely impacts its perceived "quality of life".

There will be no push for cultural change until the economic/resource distribution hardships are so obvious and undeniable, and so obviously and undeniably the result of unsustainable environmental practices, that a public which seems to have an increasingly poor science education, little to no training in logic or philosophy (both disciplines which actually enable people to mentally follow chains of causality more than one or two links long), and which is happy to get the bulk of its news from Fox, gets smacked in the face with it.

Humans have an intelligence geared toward the short-term and immediate. Unfortunately, because of America's infrastructure, that tends to insulate a huge mass of people from repercussions which are happening now, but elsewhere.

Sadly, this is also true for the UK.

Plus, of course, you have a couple of generations of people now who have grown up valuing "feeling" over thinking, which doesn't help. My own resident neo-con, delphshadow, is perfectly willing to admit that humans change the environment and climate on a local level wherever they are; but she "doesn't believe" in climate change because she simply doesn't feel that humans are "big" enough to change an entire planet. This is used as a supposedly valid counter-argument -- and what the heck, it feels valid to them! -- in a lot of places. How do you counter that?

You are looking for a culture shift. That won't come until it's forced, I tell you.

to some extent i agree and sympathize. it's basically that a local village won't accept the expense of putting up a traffic light at an intersection until enough mangled bodies have been hauled away.

alas, it's not only time to Think Global, it's time to Act Global. we don't know how, not here.

we hate the idea. we don't want to sign a treaty which forbids nuclear testing. we've refused to put U.S. soldiers, sailors, and airmen under international laws. we are distancing ourselves from the Geneva Conventions. and, still, we expect everyone else to follow them and do more. we are dumb individualists to the bitter end. and we'll pay the consequences, ignoring the reality of Simpson's paradox: What's good for the group can be bad for every subgroup . (see also this extensive article and this technical treatment.) refusal to recognize the reality of that paradox has galvanized refusals of some American parents to submit their children to standard vaccinations.

yet, there is a contrarian spirit to Americans, even if they eschew education, and a wisdom which is capable of startling reversals, not always to the good, mind you, but reversals nonetheless.

as cynical as it might suggest Americans are, the assessment that Americans lack confidence in their government may be positive. that goal might be what the neocon Republicans intended to achieve, but Americans don't like situations where there are no answers to things and noone seems to be doing anything about it.

just as in some countries there are underground economies which systematically evade taxes, if the federals and big states can't deal with matters which concern Americans, i retain enough trust in their rugged, rogue, creative, and non-militaristic disrespect (*) for government and authority to believe there are networks of solutions growing out there to respond to perceived challenges. now, they may well not be enough, particularly if international cooperation is really required and if Americans avoid peer-to-peer contacts with peoples of, say, Asia. but if they have established such contacts at the grass roots and come to believe that climate change poses a threat, the government might be told to be damned while solutions are pursued. after all, a vote in a booth is an incredibly crude means for establishing policy. people elect people, not positions. and when positions in referenda are posed, they are as internally conflicted as politicians can make them.

whatever, i think luna_the_cat is correct in that if Americans don't perceive a self-interest to change rather than an ideological one, they won't. at some point, maybe after enough mangled bodies are carted away, they'll come to new appreciation of Ben Franklin's

We must all hang together, or most assuredly we will all hang separately.

update 2006-09-09 0238 EDT: journalist Duncan is interviewed here (6 mb).

====================================================

(*) people should recall that the current fad idolizing of people in the military, itself an echo complement of the unwarranted disrespect shown to Vietnam vets returning home, is not typical or consistent with the American experience. in the 1920s, for example, soldiers and sailors were considered little better than street bums, not to be dated or associated with. veterans of the Civil War, even if drafted, had a difficult time finding work and jobs.

11^th September 2001 was bad: 3000 killed, a shocked nation successfully terrorized. at the time we did not know it was 3000. it might have been 15000. later, we had the smallpox anthrax attacks, still unsolved and, as far as anyone can tell, completely off the radar screens of the Bush administration and its Republican apologists in Congress.

one reason why so few were killed because of the 2001-09-11 attacks and the smallpox anthrax scare was a vibrant and deeply skilled public health system, both in New York City and headquartered at the Centers for Disease Control in Atlanta. they mobilized. they knew what to do. they are, or were, really sharp. the best health statistics in the world come from their National Center for Health Statistics. not even the FBI's Bureau of Justice Statistics does as good a job.

then CDC and the public health system began suffering its own attacks, this time upon its budget and mandate by the Bush White House and its Republican minions. as reported in part at truthout.org , their mission has changed, people have left, some encouraged to do so, our national program to combat HIV and AIDS has changed complexion, and anything having to do with contraception or abortion is now controversial. scientists there, as in most federal agencies across the country, need to have their public press releases, reports, and announcements screened for ideological compliance by a new cadre of party apparatachiks, worthy of the Soviet Union in its heyday.

Chris Mooney and Effect Measure report more.

update 20060905 2109 EDT: an anonymous commenter pointed out i said "smallpox" when i should have said "anthrax". thank you, to whoever they are. it is corrected.

many aerospace engineers, in my experience, believe that statistical independence is a point of view and frame of mind. they also believe that about Gaussian distributions. (given that Gaussians can be a limiting form of binomial events, the two are connected.) they invoke laws of large numbers, believing they say that independence is a good approximation to most real phenomena. engineers may do this because of certain great successes in physics and engineering which succeed in doing that, notably Boltzmann's theory of ideal gasses and independent component analysis. (there are other examples, like the naive Bayes classifier.)

it's interesting that Boltzmann encountered a lot of grief when he first presented his analysis of energies and gasses, because it seemed too simple. Boltzmann turned out to be right, but i'd say the skepticism of the scientists who heard his work was proper. in most systems of interest, the coupling or covariance is the thing, and understanding it and the subtleties of correlation is what's meant by understanding those systems.

when risks, decision, and policy are involved, the systems under study, often groups of people, are highly coupled. so, for instance, if one examines incidence of HIV and AIDS in human populations using assumptions of independence, the examination yields ridiculous conclusions. for example, given a recent reading of the Duesberg misrepresentation, it takes 80000 unprotected encounters to have a 25% chance(*) of transmitting HIV once.



that's wildly incorrect, so if the numbers are based upon anything real (i don't really know), there must be something wrong with the model.

in this case, the model is almost certainly wrong, as it assumes any person in the U.S. population is as likely to have a sexual encounter with a particular person as any other. even granted the matter of gender preference, this is vastly incorrect for two reasons.

first, most people bed people they know, and they get to know them through networks of friends or from places where they meet people with similar interests and goals.

second, even if people had sex with random people they knew, their "knowing nets" would be constrained by the long-tailed cluster-like distributions characterizing groups of people which have been the subjects of network analysis, statisticians, economists, sociologists(**), marketing specialists, and others.

if either or both of these is true, then the fraction of people with HIV depends upon its distribution across these clusters and which clusters an individual happens to be connected with. indeed, organisms like HIV depend upon this kind of connecting organization in its host population to propagate.

the second feature i personally doubt is that 0.001 chance of propagation of HIV per unprotected encounter. it's not that i believe people can't do averages, it's that those moments of this distribution are, i bet, uninformative. that distribution is at least multimodal and it is contingent upon the variety of HIV as well as the immune response of the non-infected partner. it is true that oral sex tends to transmit HIV poorly. in that case i would buy a 0.001 unprotected risk.

it might be that aerospace engineers and others have these views of how statistical practice is done because the examples they are shown are dumbed down. in order to make them understandable to classes without the requisite backgrounds, only the cases involving statistical independence and normality are taught and tested. that's how it was done in most of my classes. we rarely saw anything of the larger world, like the complexities of reliability engineering. (see Reliability Engineering by Elsayed for more.) even that field still has little about Bayesian approaches in its curriculum.

update 20060905 0121 EDT: there's an important comment related to all this by Ethan Romero at Good Math, Bad Math. he's "doing a doctoral thesis at the University of Michigan in modeling HIV epidemiology" so he's the Bee's Knees.

(cross-posted in a different form to Good Math, Bad Math )

===============================================

(*) for an individual 25% may be high, it may be low. i adopted it because it represents the chance of a healthy couple in their 20s conceiving while having sex during the course of a single month. that's reasonably representative of what people consider certain.
(**) some of the more provincial physicalists among us, myself included, would say that's all there is to sociology, or should be.

RealClimate today discusses statistical predictability for weather and climate and how it depends upon spatial and temporal scales. but the matter isn't at all simple. see especially the commentary and discussion following the one by Professor Pielke.

it is true we're not sure if curtailing CO₂ emissions is all we need to do. and we're not sure if large scale melts at the poles is all that's going to be seen. the sense i get from one of the papers the paper Professor Pielke refers to is that its a dangerous business messing with any portion of the climate system. that's not reassuring.