PC-Bjørn Mathisen

fun fact in #disinformation research: filtering out disinformation is a solved problem — has been for 1000s of years however, disinformation’s solution creates major problems for liberalism and democracy that’s the *real* #disinformation problem 1/🧵 IMO, @karpathy’s post is terribly fascinating, because it presents probably the best *technologist-translated* description of the core belief behind #why communities already filter out disinformation. The communities who filter out (what they consider to be) disinfo very effectively are "fundamentalist" communities. They remain "fundamentalist" because they filter out information which contradicts with their “fundamental” beliefs, because that information might be brain-poison. Or to use @karpathy’s terms, fundamentalist communities filter out information that might be “an (attacker controlled) executable that you instantly run on your brain”…. Fundamentalist communities believe brain-poison exists, and that they must shield most people’s brains from brain-poison. You can take my word for it, because in my youth I spent time in and around fundamentalist communities (I was homeschooled in the American South™, but not for the same reasons most of my then-peers were homeschooled, which is a pretty good vantage point to observe fundamentalism). Now, fundamentalist communities have obvious failure-modes, and that’s kindof my point. Disinformation is a problem, but once you believe that brain-poison exists, it’s a direct path towards fundamentalism, and that’s the real reason why disinformation hasn’t been solved by the liberal/tech research movement. “Disinformation filtered out by liberals” is a contradiction in terms, because in the liberal worldview, true brain-poison can’t exist, because free individuals should use logic and reasoning to filter it out — AFTER consuming it. The logical link from “brain-poison exists” -> “fundamentalism” is fascinating: - Starting belief: brain-poison exists. (@karpathy's "direct, untrusted information”). Brain-poison is too dangerous to deserialise into your brain — and must be avoided. - before consuming brain-poison, all you have is metadata from its senders who consumed it already - however, brain-poison doesn’t come with a sign saying “do not open… this is brain poison”. - brain-poison usually only comes in an envelope with a sender and receiver. The question is… do you open it? - sometimes brain-poison comes with additional metadata from the sender —  endorsements like “this is good brain-food!!!”. However, if the sender ate brain-poison, this can’t be trusted. - thus, brain-poison only really comes with one possibly-reliable piece of metadata to use… the identity of the sender. - so, brain-poison can only be avoided by not opening letters from senders who send brain-poison. - thus, to defend from brain-poison, you need to create a set of denylists, and if a sender is on a deny list, you don’t open their brain-food… it’s probably poison. - the problem is sometimes your friends don't have the same denylists as you, and so they become tainted. This means that someday… you open a letter from a friend and… BOOM: brain poison!!! - the only way to protect yourself is to setup whole communities with the same denylists (a quarantineed group of friends… or to use security language… a VPN) - however, in order for your whole community to have the same deny list, someone (or a group) in the community needs to curate the denylists… someone has to be poison-checker - to evaluate a candidate for a denylist, this poison-checker has to eat some sketchy brain-food to determine whether it is harmful, and then add/remove people from the community denylist based on whether the food they send is poisonous. - because being brain-food-poison checker requires deserializing the information into their brain, it might poison the poison-checker. - thus, not everyone can be brain-poison-checker, because it sortof defeats the point of having a poison-checker at all. if everyone is processing all information to see whether it should be processed, everyone's getting poisoned all the time. - thus, only a small number of culturally-hardened individuals can drink the informational poison on behalf of the group, and inform the group on whether its safe for everyone to drink — updating the denylists - this creates a small set of people who can bottleneck control over the whole community, by limiting the sources from which they take information - ....voila: a censored community IMO, this reveals the real challenge of fighting #disinformation, and one of the reasons the conversation progresses so slowly: filtering out disinformation is a *deeply* *conservative* *idea*, directly at odds with liberal politics of free-flowing information and individual freedom of choice. Going all the way back to ideas from Plato and Socrates, liberalism has a foundational belief that consuming more information is better, and that internal consistency (or inconsistency!) in that information will filter out truth from fiction. Everyone is a free individual and philosopher king! Everyone can run their own truth-checker. However, if you start to believe that internal consistency of information isn't enough — that it's possible that the widest set of facts available might create a distorted/false view of reality because certain internal facts in the widest possible view poison the whole system — then conservative ideals start creeping in. Because if we want to live in a world which is something other than "everyone consuming all information all the time", we would have to re-instate individuals who professionally steward our culture by studying and describing what information should be kept or thrown out (and being immune to that information themselves). And we need a system for appointing those individuals that's rooted in the values of the people. And if we have people whose job it is to protect our existing beliefs from brain-poison, then we’ll change our beliefs much more slowly. If they do a really good job at protecting us from anything other than the brain-food we like, we won’t change our beliefs hardly at all… … in other words… we’d be fundamentalist conservatives I could write a more on this — but instead of digging deeper, I’ll leave just a few mental chew toys to play with. Hope you enjoy! - the belief that adtech can’t poison people’s brains is conveniently aligned with liberalism, and the biggest adtech firms tend to be liberal. true conservatives can't work in adtech and think they're making the world better because adtech makes it possible for anyone to force-feed brain-poison into people's minds. - one way to filter out brain-poison is to flood people’s diet with “good brain-food” so that brain-poison is watered down to nothing. This also avoids the appearance of running a censorship operation. It just looks like abundance, open-ness, and universal access to more info than you ever could want. But the effect is the same. - liberal researchers advocating to protect liberalism by fighting conservative disinformation is a strange loop - LLMs are getting ~pretty good at checking whether two statements are logically in the same political camp — possible poison-checker? - if brain-poison exists, discovering new ideas always runs the risk of finding brain-poison that kills you. This is why discovery and adventure needs to be highly rewarded in society, and we should allow scientists to believe weird things. For the good of the community, they eat brain-poison. Otherwise, local incentives to avoid brain-poison incentivise no discovery of new ideas. - as problematic as it is to run experiments on humans, the only ground-truth way to check for brain-poison is for someone to try it, and they’re going to expose their sub-community. However, the only thing worse than exposing one sub-community, is every sub-community trying at the same time. The existence of mass-media, especially the “everyone is a mass-media broadcaster” trait of social media is the real disinformation accelerant. - a really good mitigator of the problems with fundamentalism is pluralism… if you acknowledge that nobody has a perfect truth… but we need many whole communities experimenting with truths to see which ones work… and double down on communities who find them… you get the long-term advancement of liberal views in a system which can protect itself from disinformation. However, this still requires communities to judge whether each other are thriving without exposing themselves to each other’s ideals. Traditionally, this was done through tribal warfare killing weaker tribes and taking their stuff. What is the modern equivalent? Economic competition? Political competition? Immigration? - anecdotal: fundamentalists can be much less happy than liberals when they are caught in a sub-community with terrible beliefs. if they think brain-poison is good food, and they eat it all the time... their lives can suck in a way they can't escape. however, fundamentalists can also achieve higher levels of happiness than liberals because they can live in information bubbles that make them happy, where there is no brain-poison. however, from the liberal perspective, this will always look like these happy fundamentalists are living in a delusional dream-world. thus, grumpy liberals throwing shade on deluded, dream-world conservatives will continue to be a meme. However, the unhappy truth for liberals is that they can never live in a world devoid of brain-poison. They'll always be happier than the worst fundamentalists but less happy than the best. This is Matrix blue/red pill kind of stuff — although the core tennant of the matrix is the big question here: is there a *real* outer truth that can just be known... or is it all one long truth-journey in which each step is not a single logical conclusion... but each step is each sub-community that tries on a belief for size. That's the real tension beneath liberalism and conservatism: is truth something you deduce or something you observe by measuring how well a community thrives. It's probably a mix of both. - structured transparency can radically help with all of this in practice: arxiv.org/abs/2012.08347