The small set expansion hypothesis or small set expansion conjecture in computational complexity theory is an unproven computational hardness assumption. Under the small set expansion hypothesis it is assumed to be computationally infeasible to distinguish between a certain class of expander graphs called "small set expanders" and other graphs that are very far from being small set expanders. This assumption implies the hardness of several other computational problems, and the optimality of certain known approximation algorithms.

The small set expansion hypothesis is related to the unique games conjecture, another unproven computational hardness assumption according to which accurately approximating the value of certain games is computationally infeasible. If the small set expansion hypothesis is true, then so is the unique games conjecture.

The edge expansion of a set ${\displaystyle X}$ of vertices in a graph ${\displaystyle G}$ is defined as $${\displaystyle {\frac {|\partial X|}{|X|}},}$$ where the vertical bars denote the number of elements of a set, and ${\displaystyle \partial X}$ denotes the set of edges that have one endpoint in ${\displaystyle X}$ and the other endpoint in its complement. This number can be as low as zero, when ${\displaystyle X}$ is a connected component of the graph, because in this case there are no edges connecting ${\displaystyle X}$ to other parts of the graph. A graph is called regular or ${\displaystyle d}$-regular when every vertex is incident to the same number of edges, ${\displaystyle d}$, the degree of the graph. For a ${\displaystyle d}$-regular graph, the maximum possible edge expansion is ${\displaystyle d}$. This expansion is achieved by any subset ${\displaystyle X}$ that induces an independent set, as in this case all of the edges that touch vertices in ${\displaystyle X}$ belong to ${\displaystyle \partial X}$.

The edge expansion of a graph with ${\displaystyle n}$ vertices is defined to be the minimum edge expansion among its subsets of at most ${\displaystyle n/2}$ vertices. Instead, the small set expansion is defined as the same minimum, but only over smaller subsets, of at most ${\displaystyle n/\log _{2}n}$ vertices. Informally, a small set expander is a graph whose small set expansion is large.

The small set expansion hypothesis uses a real number ${\displaystyle \varepsilon }$ as a parameter to formalize what it means for the small set expansion of a graph to be large or small. It asserts that, for every ${\displaystyle \varepsilon >0}$, it is NP-hard to distinguish between ${\displaystyle d}$-regular graphs with small set expansion at least ${\displaystyle (1-\varepsilon )d}$ (good small set expanders), and ${\displaystyle d}$-regular graphs with small set expansion at most ${\displaystyle \varepsilon d}$ (very far from being a small set expander). Here, the degree ${\displaystyle d}$ is a variable that might depend on the choice of ${\displaystyle \varepsilon }$, unlike in many applications of expander graphs where the degree is assumed to be a fixed constant.

The small set expansion hypothesis implies the NP-hardness of several other computational problems. Because it is only a hypothesis, this does not prove that these problems actually are NP-hard. Nevertheless, it suggests that it would be difficult to find an efficient solution for these problems, because solving any one of them would also solve other problems whose solution has so far been elusive (including the small set expansion problem itself). In the other direction, this implication opens the door to disproving the small set expansion hypothesis, by providing other problems through which it could be attacked.

In particular, there exists a polynomial-time reduction from the recognition of small set expanders to the problem of determining the approximate value of unique games, showing that the small set expansion hypothesis implies the unique games conjecture. Boaz Barak has suggested more strongly that these two hypotheses are equivalent. In fact, the small set expansion hypothesis is equivalent to a restricted form of the unique games conjecture, asserting the hardness of unique games instances whose underlying graphs are small set expanders. On the other hand, it is possible to quickly solve unique games instances whose graph is "certifiably" a small set expander, in the sense that their expansion can be verified by sum-of-squares optimization.

Another application of the small set expansion hypothesis concerns the computational problem of approximating the treewidth of graphs, a structural parameter closely related to expansion. For graphs of treewidth ${\displaystyle w}$, the best approximation ratio known for a polynomial time approximation algorithm is ${\displaystyle O({\sqrt {\log w}})}$. The small set expansion hypothesis, if true, implies that there does not exist an approximation algorithm for this problem with constant approximation ratio. It also can be used to imply the inapproximability of finding a complete bipartite graph with the maximum number of edges (possibly restricted to having equal numbers of vertices on each side of its bipartition) in a larger graph.

The small set expansion hypothesis implies the optimality of known approximation ratios for certain variants of the edge cover problem, in which one must choose as few vertices as possible to cover a given number of edges in a graph.

The small set expansion hypothesis was formulated, and connected to the unique games conjecture, by Prasad Raghavendra and David Steurer in 2010, as part of a body of work for which they were given the 2018 Michael and Sheila Held Prize of the National Academy of Sciences.

One approach to resolving the small set expansion hypothesis is to seek approximation algorithms for the edge expansion of small vertex sets that would be good enough to distinguish the two classes of graphs in the hypothesis. In this light, the best approximation known, for the edge expansion of subsets of at most ${\displaystyle n/\log n}$ vertices in a ${\displaystyle d}$-regular graph, has an approximation ratio of ${\displaystyle O({\sqrt {\log n\log \log n}})}$. This is not strong enough to refute the hypothesis; doing so would require finding an algorithm with a bounded approximation ratio.