Jacobi iterationsconsist in simply inverting the diagonal of A, letting the under-diagonal and the over-diagonal part of A work on the previous estimation of x.
In the Gauss Seidel method, one uses the updated part of x immediately, either by letting the under-diagonal part of A work on the new estimation of x, renewing x with index i=0,...,n (Forward Gauss Seidel,FGS) or letting the over-diagonal part of A work on the new estimation of x, renewing x with index i=n,...,0 (Backward Gauss Seidel, BGS).
The convergence rate of Gauss Seidel iterations can sometimes be improved by relaxation. Relaxation is done by overcorrecting the new value from the Gauss seidel iteration with the previous value, so that x(k+1)=x(k)+w*r(k), where r(k) is the residual of the GS iteration, and w is the relaxation parameter.
Successive Over Relaxation, SOR, is over-relaxed when w is between 1 and 2, and under-relaxated when w is between 0 and 1. The method does not converge for other values of w.
Symmetric Successive Over Relaxation SSOR, consists of one half step relaxated FGS, followed by one half step relaxated BGS.

Both the Jacobi and the Gauss Seidel methods are of the form Mx(k+1)=Nx(k)+b, where A=M-N. As long as M is non-singular and the spectral radius (largest eigenvector) of inv(M)N is less than one, both these methods will converge to Ax=b, for any initial guess x(0). Since the spectral radius of a matrix is always smaller than the matrix norm, a sufficient condition for convergence is that the matrix norm is strictly less than one. If A is strictly diagonally dominant, this will always be the case. The more diagonally dominant A is, the more rapid the convergence.
If the spectral radius of inv(M)N is close to unity, Gauss Seidel will be very slow, since the error tend to zero as the spectral radius to the power of k. When this is the case, SOR can significantly reduce the number of iterations needed. The SOR method converges for all x(0) if A is positive definite and symmetric, as long as 0 < w < 2. The problem with SOR methods is choosing the optimal w. The benefits of relaxation only occurs for a narrow window around the optimal w.One can proove that the optimal w is w=2/(1+sqrt(1-p(J)*p(J))), where p(J) is the spectral radius of the Jacobi iteration matrix corresponding to A. p(J) can be approximated by setting all coefficients constant for the problem , there are also automated schemes for getting the optimal w. another problem with SOR, is that the convergence rate is not improved until after ~N iterations. In the first iterations, the error might become a lot larger than for GS. This can be helped by Chebyshev acceleration, where w is updated for each step, so that the error decreases with each iteration. w(0)=1, w(1)= 2/(1+sqrt(1-p(J)*p(J))), w(2)=2/(1+sqrt(1-p(J)*p(J)*w(1))), etc

Iterative methods are used for large sparse systems, where Gaussian elimination would introduce more non-zero elements which would have to be stored and calculated whith. Jacobi and Gauss Seidel require only as many flops as there are entries in the A matrix. Gauss Seidel generally, but not always, converge faster than Jacobi, and GS requires less storage than Jacobi, since x is overwritten with new values as soon as they are calculated. Jacobi is useful for parallell computing. In the one-dimensional case, there is no benefit from using iterative methods, only in the 2D and 3D case, the number of operations and the storage requirements are less than for the direct solver.