Falls also tracked weakness in European and other Asian markets following lacklustre global corporate earnings and a credit ratings downgrade of five Spanish regional governments.
Caution also prevailed ahead of the expiry of October derivative contracts on Thursday, and earnings of blue chip companies such as ICICI Bank
Markets are closed on Wednesday for a public holiday.
"In the medium term, markets may show positive momentum because quarterly results by and large are better than expected," said R.K. Gupta, managing director at Taurus Mutual Fund.
0.44 percent, or 83.42 points, to 18,710.02 points.
The 50-share Nifty fell 0.45 percent, or 25.75 points, to 5,691.4 points.
ITC fell 1.4 percent, falling for the second day after touching an all-time high of 299 rupees on Friday.
The cigarette maker posted a 22 percent rise in July-September quarterly profit as higher prices protected margins, but investors worry about the continued impact from higher taxes and tighter anti-smoking regulations in several Indian states.
Hero MotoCorp fell 1.75 percent. The world's largest two-wheeler maker reported after the market close July-September net profit had fallen 27 percent, missing estimates.
Shares in blue chip drug maker Lupin fell 1.2 percent despite reporting stronger-than-expected earnings as investors worried about the higher taxes paid during the previous quarter.
Maruti Suzuki shares fell 0.2 percent after CLSA downgraded India's largest auto maker to 'sell' from 'underperform', saying a recovery in demand for passenger vehicles might not be as large as expected.
However, among gainers Larsen & Toubro
Shares in Yes Bank rose 3.18 percent after the private lender reported a slight decrease in non-performing assets, while quarterly net profit rose 30.2 percent.
Shares in IDEA Cellular