Let $Cr_n(F_p)$ denote the algebra of $n times n$ circulant matrices over $F_p$, the finite field of order $p$ a prime. The order of the unit groups $mathcal{U}(Cr_3(F_p))$, $mathcal{U}(Cr_4(F_p))$ and $mathcal{U}(Cr_5(F_p))$ of algebras of circulant matrices over $F_p$ are computed.