n x n matrix multiplication with rounding