Click at the Department to be removed > Users > Remove Department
NOTE: Users can also delete department by right clicking at the department at the left panel > Remove Department