Definition:Complement of Relation

Let $$\mathcal{R} \subseteq S \times T$$ be a relation.

The complement of $$\mathcal{R}$$ is the relative complement of $$\mathcal{R}$$ with respect to $$S \times T$$:

$$\mathcal{C}_{S \times T} \left({\mathcal{R}}\right) = \left\{{\left({s, t}\right) \in S \times T: \left({s, t}\right) \notin \mathcal{R}}\right\}$$

An alternative to $$\mathcal{C}_{S \times T} \left({\mathcal{R}}\right)$$ is $$\overline{\mathcal{R}}$$ which is more compact and convenient, but the context needs to be established so that it does not get confused with other usages of the overline notation. Specific conventional symbols used to denote certain frequently-encountered relations often consist of lines in various configurations, for example $$=$$, $$\le$$, $$\equiv$$, and adding an overline to these can only make for confusion.

Some authors use $$\mathcal{R}'$$ to denote the complement of $$\mathcal{R}$$, but $$'$$ is already heavily overused.

However, it is also apparent from the definition as given here that:

$$\left ({s, t}\right) \in \mathcal{C}_{S \times T} \left({\mathcal{R}}\right) \iff s \not \mathcal{R} t$$

Thus the complement of a relation is generally most conveniently indicated by drawing a line through the relation symbol.