Staff at the Driver and Vehicle Licensing Agency in Swansea are "scared to go to work" over Covid concerns, a union has said.
Members of the PCS union started a four-day strike at the agency's headquarters earlier.
DVLA sites in the city have recorded more than 500 Covid cases since September.
A spokesman for the agency said it had followed Welsh government Covid guidance "at every single point".
Public Health Wales declared a Covid-19 outbreak at the DVLA's contact centre in Swansea Vale in Llansamlet in December, following more than 350 cases at the site.
Along with the DVLA headquarters in the Clase area of Swansea, the agency employs more than 6,000 people in the city.
PCS said it was asking its 3,300 members at the Swansea HQ to strike "to protect workers' safety", following an industrial action ballot in early March.
The union said those involved would include operational staff who have not been working from home.
PCS general secretary Mark Serwotka said strike action was "unfortunately" necessary.
"The DVLA has had more Covid cases than any other workplace in the whole of the United Kingdom, over 600 cases," he said
"Sadly, there has been a fatality and, even in recent weeks, we have seen people contacted at work and we have seen cases break out at work.
"None of this is necessary because every other UK civil service department has a majority of people working from home, yet the DVLA insists on 2,500 people going to work every single day, and we believe that is not safe."
He said the lowest paid staff were instructed to attend the office while senior managers stayed home, adding: "It's really a scandal."
The union's DVLA branch secretary, Sarah Evans, told BBC Wales Breakfast: "They are being forced to take this action for the sake of not just their own health and safety, but the safety of their family and their work colleagues.
"They don't feel the DVLA are taking their health and safety seriously."
PCS said progress had been made in talks, but the lack of immediate moves to reduce numbers on site meant industrial action would go ahead.
Agreement had been reached on removing more than 300 desks, revising risk assessments, which has led to a further 300 staff being sent home, and a commitment on how to proceed in talks over the coming months.
However, the union said there were still "well in excess of 2,000" staff a day going into its Swansea offices.
The agency said the strike was likely to delay paper applications it received, as well as calls to its contact centre, but online services would still be available.
It said it had had no new confirmed Covid cases since the 24 March.
A DVLA spokesman said: "The safety of our staff is paramount and we have continuously adapted our measures over the past year and will continue to do so.
"There is currently not a single member of staff in the 10 day isolation period, out of a workforce of more than 6,000."
He added the agency had "consistently worked with Public Health Wales, environmental health and Swansea Bay Health Board to introduce a wide range of safety measures".