The government says up to 300,000 families in England could benefit from plans to scrap council tax for so-called granny flats.
Some exemptions already exist for over 65s living in annexes but Communities Secretary Eric Pickles wants to broaden those out to cover any family member.
He said the move could ease the pressure on the supply of homes.
But Labour said the move appeared to be a government attempt to "deflect attention from their housing crisis".
Currently, annexes are usually treated as distinct dwellings - and charged council tax - if they have a separate kitchen, bathroom and place to sleep. They do not need to have their own front door.
Local authorities typically charge more than £1,000 a year for full rates of council tax.
But under changes introduced in 1997, an annexe is exempt if the occupier is over 65, "mentally impaired", or "substantially and permanently disabled".
This move would actually make things worse for most Council Tax payers. Councils are already facing a shortfall* in the amount of Council Tax support they receive from central government and the loss of revenue from granny flats would have to be made up by the rest of us.
*From next April Council Tax Benefit for all but the eldest claimants will be reduced by an average fof 20%. The exact amount will vary from one Council Tax area to another. Some residents of working age could lose all their Council Tax Benefit and others keep all of their's. Coventry City council has yet to state which groups in this area will be affected most.
