Health care system is one of the vital sectors in each country for its national interest. This paper presents a model of e-health system which covers all medical entities like patients, doctors, nurses, medical technologist, pharmacist, healthcare organizations, pathology labs, pharmacies etc. The model also introduces e-Health Card (Smart Card) for each patient for authentication and better e-health service. The work incorporates two important aspects of the proposed e-Health system: (i) the development and deployment of a medical information management system where user’s authentication is performed by NID (National Identification Number) through smart card and (ii) the privacy and security concern of the proposed system which is implemented by Role Based Access Control (RBAC) to access patient’s medical record with audit logs and by Encryption Algorithm to Exchange of medical documents. "NID" is a 17 digit unique identification number provided by LGED (Local Government & Engineering Department), a government organization of Bangladesh. This proposed model will provide an easy to carry and access facilities everywhere in the country.