Recommended reading!

The most convincing model I’ve heard for an AI that can actually surpass its programming comes from Rudy Rucker’s Ware Tetralogy. I suspect that it’d be way easier to develop something organically than to try to skip to the end stage, which is what we seem to dream of doing. Shorter lifecycles might help? From the wiki article:

“The central technological speculation of the series are the “boppers”, a kind of robot with artificial intelligence developed through natural selection rather than through design. Crediting mathematician Kurt Gödel with the germ of the idea, Software declares: “We cannot build an intelligent robot…. But we can cause one to evolve.” By creating self-replicating robots whose programming is randomly altered periodically (and who can exchange programming information with each other in a form of sexual reproduction), and then forcing these robots to pass “fitness tests” in order to survive, Rucker suggests, true artificial intelligence that equals or surpasses the human brain could be developed. (Rucker discusses this same idea in his nonfiction work Infinity and the Mind.)”

Off the cuff, I’m struck by the thought that when we can clearly and concisely describe the obstacles and “to do”s for a goal in a simple and general way that a lay person can understand, we are much further from that goal than we can imagine.