American workers are becoming more and more pessimistic about their retirement prospects, according to a new survey by Sun Life Financial, but those who have invested in annuities or hold long-term care insurance are decidedly more confident.

Still, the survey of 1,500 employees last month recorded an 18.2% decline in overall retirement confidence compared to September of 2010. More telling, there was a 31.7% drop in workers’ confidence that they will have both the defined benefit assets and employee health benefits they need to support a comfortable retirement when they’re done working.

“This represents the most significant drop in retirement confidence we’ve seen in the four years we’ve compiled the Sun Life Unretirement Index,” Wes Thompson, president of Sun Life Financial U.S., said in a statement. “Although the recession officially ended in 2009, average Americans feel that the downturn has not ended for them, which is substantially eroding their trust in their retirement future.”

It’s so bad for some workers that one in five polled said they plan to never retire.

Only 23% of working Americans said they feel “very confident” that they will meet basic living expenses in retirement -- way down from 42% who felt that way last year.

Confidence in the future of Social Security has plunged over the last four years, to 9% in 2011 from over double that (22%) in 2008 while confidence about Medicare benefits has also plummeted, to 8% in 2011 from 20% in 2008.

On the positive side, at all wealth levels, respondents who expect to receive guaranteed lifetime income from annuities by age 67, or who own long-term care insurance products, feel significantly more confident about retirement.

The Unretirement Index is scored on a scale of 0 to 100, with 100 reflecting highest confidence about retirement. The Index score dropped from near the midrange (44) in September 2010 toward the bottom third of the scale (36) in September 2011, down 18.2%.

The overall Index is a composite score based on five subject areas, which all revealed declines in confidence compared to last year. On a year-to-year basis, confidence dropped as follows in the following subject areas: employee benefits (-31.7%), the economy (-25%), government benefits (-21.6%), personal finances (-13.9%), and personal health (-13.2%).