Symbolic AI (also: Good Old Fashioned AI, GOFAI) holds that intelligence consists of the manipulation of symbols discrete, human readable tokens according to rules. Knowledge is represented explicitly: as logical propositions, ontologies, production rules, or semantic networks. Reasoning proceeds by search, inference, or theorem proving over these structures.

Core assumption: intelligence = formal symbol manipulation.

Connectionism / Neural Networks hold that intelligence emerges from the interaction of many simple processing units (neurons) connected by weighted edges. Knowledge is represented implicitly distributed across millions or billions of learned parameters. Reasoning is pattern matching: the network maps inputs to outputs through a cascade of nonlinear transformations.

Core assumption: intelligence = learned statistical patterns.

• Logic Theorist (1956) Newell and Simon's programme proved 38 of the 52 theorems in Chapter 2 of Russell and Whitehead's Principia Mathematica. It was the first programme to automate deductive reasoning.
• General Problem Solver (1957) Newell and Simon's GPS used means ends analysis to solve puzzles like the Tower of Hanoi and missionaries and cannibals. It formalised the idea of problem solving as search through a state space.
• LISP (1958) McCarthy invented LISP, a programming language based on lambda calculus, designed specifically for symbolic AI. It remained the lingua franca of AI research for three decades.
• ELIZA (1966) Weizenbaum's chatbot used pattern matching and substitution rules to simulate a Rogerian therapist. It demonstrated how little intelligence was needed to produce convincing dialogue and how willing humans were to anthropomorphise machines.
• SHRDLU (1970) Winograd's natural language system could understand and execute commands in a microworld of coloured blocks. It was a triumph of symbolic NLP and a preview of the brittleness problem.

The canonical architecture:

Knowledge Base (KB):
  IF patient has fever AND rash AND stiff_neck
  THEN suspect meningitis (confidence 0.85)

Inference Engine:
  Forward chaining: given facts → derive conclusions
  Backward chaining: given goal → find supporting facts

Working Memory:
  Current set of asserted facts about the case

Notable expert systems included MYCIN (1976, Stanford diagnosed bacterial infections with performance rivalling infectious disease specialists),DENDRAL (1969 identified molecular structures from mass spectrometry data), and R1/XCON (1980, DEC configured VAX computer orders, saving DEC an estimated $40M/year).

• Frames (Minsky, 1974): Structured records with slots for properties and default values. A "restaurant" frame would have slots for cuisine, price range, seating with defaults that could be overridden by specific instances.
• Scripts (Schank & Abelson, 1977): Stereotyped sequences of events. The "restaurant script" encodes: enter → be seated → read menu → order → eat → pay → leave. Understanding a story about a restaurant means recognising which script is active and which slots are filled.
• Semantic networks: Graph structures where nodes are concepts and edges are relations (is a, has a, part of). Later formalised into description logics and ontologies like Cyc (Lenat, 1984 to present), which attempted to hand-code all of common-sense knowledge millions of assertions like "water is wet" and "dead people do not buy groceries."

Key criticisms:

• Combinatorial explosion: Search based methods hit exponential walls on any non trivial problem. The number of possible states in chess ($\sim 10^{47}$) or Go ($\sim 10^{170}$) made brute-force approaches hopeless.
• Brittleness: Symbolic systems failed catastrophically on inputs slightly outside their design specification. SHRDLU worked perfectly in its blocks world and not at all outside it.
• Knowledge bottleneck: Expert systems required months of painstaking "knowledge engineering" to encode rules that any human expert could apply effortlessly.

The perceptron computes a weighted sum of inputs and applies a threshold:

The perceptron convergence theorem guarantees that if the data is linearly separable, the learning rule will find a separating hyperplane in finite steps. This was a genuine theoretical achievement.

The XOR function: $f(x_1, x_2) = x_1 \oplus x_2$. The four points $\{(0,0), (0,1), (1,0), (1,1)\}$ with labels$\{0, 1, 1, 0\}$ are not linearly separable no single hyperplane can separate the classes.

Multi layer networks can compute XOR, but Minsky and Papert argued (and were widely interpreted as proving) that there was no known learning algorithm for multi layer networks. This perception partly fair, partly exaggerated effectively killed neural network funding for over a decade.

For a network with layers $l = 1, \ldots, L$, the forward pass computes:

where $\sigma$ is a nonlinear activation function. Given a loss$\mathcal{L}$, backpropagation computes:

The gradient with respect to each layer's preactivations is computed recursively from the output layer backward hence "backpropagation." Stochastic gradient descent then updates weights: $\mathbf{W} \leftarrow \mathbf{W} - \eta \nabla_{\mathbf{W}} \mathcal{L}$.

• Expert systems collapsed commercially. Maintaining enormous rule bases proved impossibly expensive. Lisp machine companies went bankrupt. The Japanese Fifth Generation Project ended in 1992 without achieving its goals.
• Neural networks couldn't scale. Without GPU hardware, training networks with more than a few hidden layers was computationally infeasible. The vanishing gradient problem gradients shrinking exponentially through deep layers made deep networks appear untrainable.
• SVMs and kernel methods (Vapnik, 1995) offered superior classification performance with rigorous theoretical guarantees, further marginalising neural networks in the 1990s machine learning community.

1. Data: The internet produced datasets of previously unimaginable scale ImageNet (14M labelled images), Common Crawl (petabytes of text), YouTube (500 hours of video uploaded per minute).
2. Compute: GPUs, originally designed for graphics, turned out to be ideal for the matrix multiplications at the heart of neural network training. NVIDIA's CUDA (2007) made GPU programming accessible.
3. Algorithms: Key innovations ReLU activations (which mitigate vanishing gradients), dropout regularisation, batch normalisation, residual connections made very deep networks trainable.

The error rate progression on ImageNet tells the story of the revolution:

• 2011: 25.8% (hand engineered features + SVM)
• 2012: 15.3% (AlexNet deep CNN)
• 2014: 6.7% (GoogLeNet / VGGNet)
• 2015: 3.6% (ResNet 152 surpassed human performance at ~5%)

Within three years of AlexNet, neural networks had gone from curiosity to state of the art (SOTA) across vision, speech, and NLP.

Where $Q$ (queries), $K$ (keys), and$V$ (values) are linear projections of the input. The scaling factor $\sqrt{d_k}$ prevents the dot products from growing too large in high dimensions.

Multi head attention runs $h$ parallel attention operations with different learned projections:

The Transformer processes all positions simultaneously (not sequentially), enabling massive parallelism on GPUs. This architectural innovation is the foundation of GPT, BERT, PaLM, Claude, and virtually every frontier language model.

Where $N$ is the number of parameters,$D$ is the dataset size, $C$ is the compute budget, and $\alpha_N \approx 0.076$,$\alpha_D \approx 0.095$. The loss decreases smoothly and predictably as any of these resources increases with no sign of plateauing.

This has driven the "scaling hypothesis": that simply making models larger, training them on more data, and using more compute will continue to yield capability improvements. The progression from GPT 2 (1.5B parameters) to GPT 3 (175B) to GPT 4 (rumoured ~1.7T) has largely validated this so far.

• Formal verification & theorem proving: Systems like Lean, Coq, and Isabelle/HOL use symbolic methods to produce machine checked proofs. No neural network can guarantee that a mathematical proof is correct.
• Interpretability & auditability: A decision tree or a set of production rules can be read and understood by a human. A neural network's "reasoning" is distributed across billions of parameters fundamentally opaque without additional interpretability tools.
• Compositional generalisation: Symbolic systems generalise perfectly to novel combinations of known primitives. Given "John loves Mary" and the rule "if X loves Y, X will help Y," a symbolic system immediately infers "John will help Mary" regardless of whether that specific combination appeared in training. Neural networks notoriously struggle with this kind of systematic compositionality (Lake & Baroni, 2018).
• Causal reasoning: Pearl's do calculus and structural causal models provide a formal framework for distinguishing causation from correlation. Neural networks learn correlations; they cannot, in general, distinguish cause from effect without additional structure.
• Small data reasoning: An expert system can encode knowledge from a single textbook. A neural network typically needs thousands or millions of examples to learn equivalent competence.

• Perception: Vision, speech, audio, video any high dimensional sensory signal. Neural networks dominate every benchmark in object detection, speech recognition, medical imaging, and more.
• Natural language: Large language models generate fluent text, translate between languages, summarise documents, and answer questions with remarkable (though imperfect) competence.
• Learning from raw data: Neural networks require no feature engineering. Given enough data, they learn to extract relevant features automatically often discovering representations that humans never considered.
• Fuzzy pattern matching: The real world is noisy, ambiguous, and continuous. Neural networks handle this gracefully; symbolic systems require every edge case to be anticipated and encoded.
• Transfer learning: A model pretrained on a large corpus can be fine tuned for downstream tasks with very little additional data enabling practical AI deployment in datascarce domains.
• Scalability: Neural networks benefit predictably from more data, parameters, and compute. Symbolic systems hit diminishing returns as rule bases grow the knowledge acquisition bottleneck.

• Neural networks with symbolic outputs: A neural network perceives the world and produces symbolic representations that are then fed into a logic engine. Example: Neural Theorem Proving (Rocktäschel & Riedel, 2017) a neural network learns to select which inference rules to apply.
• Differentiable programming over symbolic structures: Neural Logic Machines (Dong et al., 2019) implement differentiable versions of logical operators, allowing end to end gradient based training on symbolic tasks. Similarly, ∂ILP (Evans & Grefenstette, 2018) makes inductive logic programming differentiable.
• Knowledge graph augmented neural networks: Systems like KGAT (Knowledge Graph Attention Networks) use graph neural networks to reason over structured knowledge bases, combining the relational structure of symbolic knowledge with neural attention mechanisms.
• LLMs as reasoning engines with tool use: Modern LLMs (GPT 4, Claude) can call external symbolic tools calculators, code interpreters, databases, theorem provers blending neural language understanding with symbolic computation. This is arguably the most commercially deployed form of neuro symbolic AI today.
• Program synthesis: DreamCoder (Ellis et al., 2021) learns to write programs in a domain specific language, building a reusable library of symbolic abstractions from neural search. It combines neural perception (to guide search) with symbolic structure (the programs themselves are interpretable and compositional).

• AlphaGeometry (DeepMind, 2024): Solved International Mathematical Olympiad geometry problems at a silver medal level by combining a neural language model (to propose auxiliary constructions) with a symbolic deduction engine (to verify and chain proofs). Neither component alone could solve the problems.
• AlphaFold 2 (DeepMind, 2020): While primarily neural, AlphaFold incorporates physical constraints (bond angles, steric clashes) as structured inductive biases effectively embedding symbolic domain knowledge into the architecture.
• Wolfram|Alpha + ChatGPT: The integration of Wolfram's symbolic computation engine with OpenAI's LLM allows the system to understand natural language queries (neural) and compute exact answers (symbolic) a practical neuro symbolic product used by millions.

• Physical Symbol System Hypothesis (Newell & Simon, 1976): "A physical symbol system has the necessary and sufficient means for general intelligent action." This is the foundational claim of GOFAI intelligence is symbol manipulation, and nothing more is needed.
• Chinese Room Argument (Searle, 1980): A person in a room following Chinese symbol manipulation rules can produce correct Chinese responses without understanding Chinese. Searle argued this refutes strong AI: syntax is not sufficient for semantics. Connectionists note that the argument targets symbol manipulation specifically it's less clear how it applies to distributed neural representations.
• Embodied cognition: Intelligence may require a body sensorimotor interaction with a physical environment. This challenges both disembodied symbolic reasoning and text only neural networks. Roboticists like Rodney Brooks argued in the 1990s that intelligence emerges from situated behaviour, not internal representation of any kind.
• The binding problem: How does a neural network bind features into coherent objects? How does "the red square" remain distinct from "the blue circle"? Symbolic systems solve this trivially with variable binding; neural networks continue to struggle with systematic feature binding.

• Hallucination: LLMs generate plausible but factually incorrect outputs. They have no grounded model of truth only statistical patterns over training text. Symbolic verification or retrieval augmented generation (RAG) is the standard mitigation.
• Reasoning failures: Despite impressive performance on benchmarks, LLMs still fail on multi step logical reasoning, especially when problems require systematic search or backtracking. Chain of thought prompting helps but doesn't solve the underlying issue.
• Lack of world models: LLMs do not build causal models of the world. They cannot reliably answer counterfactual questions ("what would happen if X had not occurred?") because they have no mechanism for interventional reasoning.