MongoDB Documentation
On Ubuntu installation follow below link
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
it would make all the appropriate installtion at path
and config file it as /etc/mongodb.conf
Download & Extract
http://mylinuxnotebook.blogspot.in/2008/11/create-extract-targz-files.html
https://www.mongodb.org/downloads
Download it from
https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1204-3.0.3.tgz
tar -zxvf mongodb-linux-x86_64-ubuntu1204-3.0.3.tgz
This is already compiled binary for ubuntu 12.02 LTS
There are 2 storage engines supported by mongodb in major. mmapv1 and WiredTiger
mmapv1 storage engine
- mmapv1 sotrage engine good for our requirement heavy volumes of insert, reads and in-place updates.
- journaling enabled by default.
- preallocation files enabled by default with journalling.
- clear shutdown removes all the files in the journal directory. dirty shutdown doesnt , used to recover the database to a consistent state when mongo restarted.
- To speed the frequent sequential writes that occur to the current journal file, you can ensure that the journal directory is on a different filesystem from the database data files.
- it is more efficient to preallocate journal files than create new journal files as needed.
- The goal here is to have "the next" file already allocated when the last file runs out of space. Pre-allocation is often much faster than on-demand allocation.
- prealloc files do not contain data, but are rather simply preallocated files that are ready to use that are truly preallocated by the file system (i.e. they are not "sparse"). It is thus safe to remove them, but if you restart mongod with journaling, it will create them again if they are missing.*
- Add "smallfiles=true" in the MongoDB configuration file.This will cause MongoDB to use 128M prealloc files instead of 1GB, but does not seem to reduce total journal size
- disable http interface by default.
- MongoDB will use as much free memory as it can, swapping to disk as needed. Deployments with enough memory to fit the application’s working data set
in RAM will achieve the best performance.
- mmapv1 have concurrency model. so adding more CPU may help but doesn't improve much.Adding extra RAM it would reduce page faults.
- read concern = "majority"
- connection pool size to suit use case , beginning at 110-115% of typical no of concurrent db requests.
connPoolStats command returns information regarding the number of open connections to the current database
- Due to its concurrency model, the MMAPv1 storage engine does not require many CPU cores .
As such, increasing the number of cores can help but does not provide significant return.
- Increasing the amount of RAM accessible to MongoDB may help reduce the frequency of page faults
- The output from mongostat provides statistics on the number of active reads/writes in the (ar|aw) column.
- MongoDB has good results and a good price-performance ratio with SATA SSD (Solid State Disk).
- Using SSDs or increasing RAM may be more effective in increasing I/O throughput.
- If the NUMA (Non-Uniform Access Memory)configuration may degrade performance, MongoDB prints a warning.
There is a big difference between mmapv1 and WiredTiger in the way they use memory.
------------------------------------------------------------------------------------------------------------------------
mmapv1 Storage Engine -
While mmap actually uses memory mapping, so that the database cache is actually accounted as cache in the OS.
MongoDB automatically uses all free memory on the machine as its cache. System resource monitors show that MongoDB uses a lot of memory, but its usage is dynamic.
If another process suddenly needs half the server’s RAM, MongoDB will yield cached memory to the other process
This means that MongoDB will use as much free memory as it can, swapping to disk as needed. Deployments with enough memory to fit the application’s working data set in RAM will achieve the best performance.
mmapv1 have concurrency model. so adding more cpu may help but doesn't improve much.Adding extra RAM it would reduce page faults.
mongodb while writing a document, it blocks all read request till that write is complete. once write is complete, lock is released to process blocked requests.
------------------------------------------------------------------------------------------------------------------------
Wired Tiger Storage Engine -
While WiredTiger uses memory pool defined at the application start.
Depending on the amount of memory in your system, by default it should use either 1GB or half the system RAM for WiredTiger pool.
(that would explain jump from 8 % to 58 % or so) wiredtiger is multithread storage engine . so add cpu can increase throughput.
With WiredTiger, MongoDB utilizes both the WiredTiger cache and the filesystem cache.
Via the filesystem cache, MongoDB automatically uses all free memory that is not used by the WiredTiger cache or by other processes.
Data in the filesystem cache is compressed.
WiredTiger cache can be configurable.The WiredTiger cache is only one component of the RAM used by MongoDB.
In addition, the operating system will use any free RAM to buffer filesystem blocks.
To accommodate the additional consumers of RAM, you may have to decrease WiredTiger cache size.
To view statistics on the cache and eviction rate, see the wiredTiger.cache field returned from the serverStatus command
------------------------------------------------------------------------------------------------------------------------
http://docs.mongodb.org/manual/tutorial/add-user-administrator/
$ mongo
> db.createUser(
... {
... user:"theadmin",
... pwd:"theadminpassword",
... roles:[{role:"userAdminAnyDatabase",db:"admin"}]
... }
... );
> db.getUsers();
[
{
"_id" : "admin.theadmin",
"user" : "theadmin",
"db" : "admin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
}
]
}
]
> use test_keyspace;
> db.createUser(
... ... ... {
... ... ... user:"amit",
... ... ... pwd:"pwdforamit",
... ... ... roles:[
... ... {role:"readWrite",db:"test_keyspace"}
... ... ]
... ... }
... ... );
> db.getUsers();
> db.getUser("amit");
show collections;
use test_keyspace
db.contentDump.find().pretty()
db.tablename.drop()
http://docs.mongodb.org/manual/reference/method/js-user-management/
following commands are success
mongo --port 27017 -u theadmin -p theadminpassword --authenticationDatabase admin --verbose
mongo --port 27017 -u amit -p pwdforamit --authenticationDatabase test_keyspace --verbose
-------------------------------------------------------------------------------------------------------------------------------------------
While Creating New User in MongoDB, it has to be created under admin?
If no users created you can create new user without any authentification, but if you have created admin user for specific database you should authentifcate, and then perform any operation.
Documentation:
If no users are configured in admin.system.users, one may access the database from the localhost interface without authenticating. Thus, from the server running the database (and thus on localhost), run the database shell and configure an administrative user:
$ ./mongo
> use admin
> db.addUser("theadmin", "anadminpassword")
We now have a user created for database admin. Note that if we have not previously authenticated, we now must if we wish to perform further operations, as there is a user in admin.system.users.
> db.auth("theadmin", "anadminpassword")
We can view existing users for the database with the command:
> db.system.users.find()
Now, let's configure a "regular" user for another database.
> use test_keyspace
> db.addUser("amit", "pwdforamit")
-------------------------------------------------------------------------------------------------------------------------------------------
How to use authentication in Mongodb 3.0?
1 Start MongoDB 3.0 without --auth enabled (in /etc/mongod.conf), so you can change how it authenticates.
2 create admin user in mongodb explained above
3. mongo --port 27017 -u theadmin -p theadminpassword --authenticationDatabase admin --verbose
4. on mongod shell do the following.
Modify the collection admin.system.version such that the authSchema's currentVersion is 3 instead of 5 (3 is using SCRAM-SHA-1).
As stated here (http://docs.mongodb.org/manual/core/authentication/)MongoDB 3 authentication mechanism has been changed from : MongoDB Challenge and Response (MONGODB-CR) to challenge and response mechanism (SCRAM-SHA-1). You have to delete your created user then change admin.system.version.authSchema to 3 instead of 5. Then recreating your user should solve the problem.
>
> var schema = db.system.version.findOne({"_id" : "authSchema"})
> schema.currentVersion = 3
> db.system.version.save(schema)
> exit
currentVersion = 3 will make the default MONGODB-CR (challenge and response), the default for MongoDB version 2x. Version 3.0 uses SCRAM-SHA-1 by default instead.
From Java code, you need to call sh1 mongodcredential API instead of cr credentials , as its changed in 3.x version.
5. as schema.currentVersion is made 3 (existing value being 5), no need to mention authenticationMechanisms like below
restart MongoDB with --auth enabled. from /webapps/mongo/bin I would run ./mongod --dbpath $HOME/webapps/mongo/data --setParameter authenticationMechanisms=MONGODB-CR --auth --port 1400
The setParameter authenticationMechanisms=MONGODB-CR may be redundant, or meaningless.
you also need to comment the bind_ip line in /etc/mongod.conf sothat anybody can connect to mongodb.
-------------------------------------------------------------------------------------------------------------------------------------------
Robmongo --
The File etc/mongod.conf has a line 'bind_ip'. In this line, you originally have to add the IP address which you want to access your database.
But, it don't work! You should better comment this line.
But, you don't have any authentication now, so you have to add authentication. Here you have an tutorial about this: http://ghosttx.com/2012/03/how-to-connect-to-a-remote-mongodb-server-with-mongohub-for-mac/
When you have done that, you have to enable authentication. You can do this by editing etc/mongod.conf again, and uncomment the line 'Auth = true'.
Now you can connect with you Mongo Database ;)
http://www.mongovue.com/2011/08/04/mongovue-connection-to-remote-server-over-ssh/
Im on OSX and connecting to Ubuntu 14 / Mongo 2.6.7 on VPS and when Ive added my ssh details to the Robomongo all seem to work ok (Ive also changed the mongo config to remove the ip_bing and enabled port 27017)
-------------------------------------------------------------------------------------------------------------------------------------------
it was using SCRAM-SHA-1 as it's authentication mechanism instead of MongoDB-CR.
Run-time database configuration : (http://docs.mongodb.org/manual/administration/configuration/)
copy it to your any user path , make appropriate changes like making it as daemon, log, dbpath etc..
start mongodb as daemon as follows:
mongod --config //mongodb.conf OR mongod --f //mongodb.conf
mongod --dbpath /usr/local/mongodb-data can also be done.. but better to do it in file as it has many options
default mongodb would be listening on port 27017 for db connections
default admin web console listening on port 28017 for connections
if using the mongo shell as a client, you can specify the database for the user with the –authenticationDatabase option.
Together, the user’s name and database serve as a unique identifier for that user.
MongoDB stores all user information, including name, password, and the user's database, in the system.users collection in the admin database.
User administrators
create the user administrator as the first user. Then let this user create all other users.
1) Security Introduction
Maintaining a secure MongoDB deployment requires administrators to implement controls to ensure that users and applications have access to only the data that they require. MongoDB provides features that allow administrators to implement these controls and restrictions for any MongoDB deployment.
2) Authentication
Before gaining access to a system all clients should identify themselves to MongoDB. This ensures that no client can access the data stored in MongoDB without being explicitly allowed.
MongoDB supports a number of authentication mechanisms that clients can use to verify their identity. MongoDB supports two mechanisms: a password-based challenge and response protocol and x.509
Authentication is the process of verifying the identity of a client. When access control, i.e. authorization, is enabled, MongoDB requires all clients to authenticate themselves first in order to determine the access for the client.
Although authentication and authorization are closely connected, authentication is distinct from authorization. Authentication verifies the identity of a user; authorization determines the verified user’s access to resources and operations.
3) Client Users
To authenticate a client in MongoDB, you must add a corresponding user to MongoDB. When adding a user, you create the user in a specific database. Together, the user’s name and database serve as a unique identifier for that user. That is, if two users have the same name but are created in different databases, they are two separate users. To authenticate, the client must authenticate the user against the user’s database
For instance, if using the mongo shell as a client, you can specify the database for the user with the –authenticationDatabase option.
To add and manage user information, MongoDB provides the db.createUser() method as well as other user management methods. For an example of adding a user to MongoDB, see Add a User to a Database.
MongoDB stores all user information, including name, password, and the user's database, in the system.users collection in the admin database.
4) Add a User to a Database & Create a User Administrator:
Overview
User administrators create users and create and assigns roles. A user administrator can grant any privilege in the database and can create new ones. In a MongoDB deployment, create the user administrator as the first user. Then let this user create all other users.
a)userAdmin
Provides the ability to create and modify roles and users on the current database. This role also indirectly provides superuser access to either the database or, if scoped to the admin database, the cluster. The userAdmin role allows users to grant any user any privilege, including themselves.
The userAdmin role explicitly provides the following actions:
changeCustomData
changePassword
createRole
createUser
dropRole
dropUser
grantRole
revokeRole
viewRole
viewUser
To provide user administrators, MongoDB has userAdmin and userAdminAnyDatabase roles, which grant access to actions that support user and role management. Following the policy of least privilege userAdmin and userAdminAnyDatabase confer no additional privileges.
Carefully control access to these roles. A user with either of these roles can grant itself unlimited additional privileges. Specifically, a user with the userAdmin role can grant itself any privilege in the database. A user assigned either the userAdmin role on the admin database or the userAdminAnyDatabase can grant itself any privilege in the system.
b)userAdminAnyDatabase
Provides the same access to user administration operations as userAdmin, except it applies to all databases in the cluster. The role also provides the following actions on the cluster as a whole:
authSchemaUpgrade
invalidateUserCache
listDatabases
The role also provides the following actions on the admin.system.users and admin.system.roles collections on the admin database, and on legacy system.users collections from versions of MongoDB prior to 2.6:
collStats
dbHash
dbStats
find
killCursors
planCacheRead
Prerequisites:
a) Required Access
You must have the createUser action on a database to create a new user on that database.
You must have the grantRole action on a role’s database to grant the role to another user.
If you have the userAdmin or userAdminAnyDatabase role, you have those actions.
b) First User Restrictions
If your MongoDB deployment has no users, you must connect to mongod using the localhost exception or use the --noauth option when starting mongod to gain full access the system. Once you have access, you can skip to Creating the system user administrator in this procedure.
If users exist in the MongoDB database, but none of them has the appropriate prerequisites to create a new user or you do not have access to them, you must restart mongod with the --noauth option.
-------------------------------------------------------------------------------------------------------------------------------------------
http interface http://107.108.205.184:28017
run mongodb with --rest flaf
sudo /usr/bin/mongod --config /etc/mongod.conf --rest
public class ConnectInfo{
private String host;
private int port;
public ConnectInfo(final String newHost, final int newPort){
this.host = newHost;
this.port = newPort;
}
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public int getPort() {
return port;
}
public void setPort(int port) {
this.port = port;
}
}
import java.util.ArrayList;
import java.util.List;
import org.apache.log4j.Logger;
import org.jongo.Jongo;
import org.jongo.MongoCollection;
import com.mongodb.DB;
import com.mongodb.MongoClient;
import com.mongodb.MongoCredential;
import com.mongodb.ServerAddress;
public class MongoConnector {
static Logger logger = Logger.getLogger(MongoConnector.class);
MongoClient mongoClient;
private static MongoConnector instance = null;
private Jongo jongo = null;
public MongoConnector(ConnectInfo cnInfo){
DB database;
this.connect(cnInfo.getHost(), cnInfo.getPort(),DataStore.dbname);
database = mongoClient.getDB(DataStore.dbname);
jongo = new Jongo(database);
}
public static MongoConnector getInstance(ConnectInfo cnInfo){
if( null == instance){
instance = new MongoConnector(cnInfo);
}
return instance;
}
public MongoCollection getCollection(String tableName){
return jongo.getCollection(tableName); // mapper
}
/**
* Connect to Mongo Cluster specified by provided node IP
* address and port number.
*
* @param node Cluster node IP address.
* @param port Port of cluster host.
*/
public void connect(final String node, final int port, String database)
{
/*
List seeds = new ArrayList();
seeds.add(new ServerAddress(node,port));
List credentials = new ArrayList();
credentials.add(
MongoCredential.createScramSha1Credential(
"amit",
database,
"pwdforamit".toCharArray()
)
);
mongoClient = new MongoClient( new ServerAddress(node,port), credentials );
*/
this.mongoClient = new MongoClient( node , port );
}
public MongoClient getMongoClient() {
return mongoClient;
}
public void setMongoClient(MongoClient mongoClient) {
this.mongoClient = mongoClient;
}
}
package com.test.mongodb;
import java.util.Date;
import java.util.List;
import java.util.UUID;
import org.apache.log4j.Logger;
import org.jongo.MongoCollection;
import org.jongo.MongoCursor;
import com.google.common.collect.Lists;
import com.mongodb.MongoClient;
import com.test.mongodb.ConnectInfo;
import com.test.mongodb.MongoConnector;
import com.test.mongodb.QueryObject;
import com.test.mongodb.ContentDump;
public class MongoContentDumpInsertImpl{
static Logger logger = Logger.getLogger(MongoContentDumpInsertImpl.class);
private static MongoContentDumpInsertImpl instance =null;
private MongoClient mongoClient = null;
private MongoConnector dbConnect = null;
//private Jongo jongo = null;
private MongoCollection collection = null;
private String pKey = null;
MongoContentDumpInsertImpl(){
pKey = new String("dump_id");
}
public static MongoContentDumpInsertImpl getInstance(){
if( null == instance){
instance = new MongoContentDumpInsertImpl();
}
return instance;
}
@Override
public void setup() throws DataStoreException {
// TODO Auto-generated method stub
}
@Override
public void connect(ConnectInfo cnInfo) throws DataStoreException {
/*DB database;
dbConnect = new MongoConnector();
dbConnect.connect(cnInfo.getHost(), cnInfo.getPort(),"test_keyspace");
this.mongoClient = dbConnect.getMongoClient();
database = mongoClient.getDB("test_keyspace");
jongo = new Jongo(database);
collection = jongo.getCollection("contentDump"); // mapper
*/
dbConnect = MongoConnector.getInstance(cnInfo);
collection = dbConnect.getCollection(DataStore.contentDump);
}
@Override
public ContentDump query(QueryObject qObject) throws DataStoreException {
if (qObject.getField() == null || qObject.getValue() == null){
throw new DataStoreException("Contentdump input query param is null for query");
}
String query = "{"+ qObject.getField()+":#}";
ContentDump record = null;
try{
record = collection.findOne(query,qObject.getValue()).as(ContentDump.class);
}catch(Exception e){
throw new DataStoreException("ContentDump findOne failed");
}
logger.info("[Contentdump] : SELECT Query Successful");
return record;
}
@Override
public void insert(ContentDump record) throws DataStoreException {
logger.info(record.toString());
try{
collection.ensureIndex("{expired:1}","{expireAfterSeconds:0}");
// by setting this attribute , it purges records when timestamp in the 'expired' field passes.
collection.save(record);
}catch(Exception e){
throw new DataStoreException("ContentDump save failed");
}
logger.info("[Contentdump] : INSERT Query Successful");
}
@Override
public void update(ContentDump record) throws DataStoreException {
String query = "{"+ pKey +":#}";
try{
collection.update(query,record.getDump_id()).with(record);
}catch(Exception e){
throw new DataStoreException("ContentDump update failed");
}
logger.info("[Contentdump] : UPDATE Query Successful");
}
@Override
public void delete(QueryObject qObject) throws DataStoreException {
if (qObject.getField() == null || qObject.getValue() == null){
logger.info("[Contentdump] : DELETE Query Failed");
throw new DataStoreException("Contentdump input query param is null for delete query");
}
String query = "{"+ qObject.getField()+":#}";
try{
collection.remove(query,qObject.getValue());
}catch(Exception e){
throw new DataStoreException("ContentDump update failed");
}
logger.info("[Contentdump] : DELETE Query Successful");
}
@Override
public List getAllDumps(UUID txn_id) throws DataStoreException{
List dumpList = null;
try{
//Date currentTime = new Date();
String query = "{docid:#}";
MongoCursor deals = collection.find(query,txn_id).as(ContentDump.class);
dumpList = Lists.newArrayList(deals.iterator());
}catch(Exception e){
throw new DataStoreException("[ContentDump]: Mongo DB access error for getting all dumps - "+e.getMessage());
}
return dumpList;
}
@Override
public void close() {
// TODO Auto-generated method stub
this.mongoClient.close();
}
}
package com.test.mongodb
import java.util.UUID;
import org.bson.types.Binary;
public class MongoUUIDConverter {
/**
* Convert a UUID object to a Binary with a subtype 0x04
*/
public static Binary toStandardBinaryUUID(java.util.UUID uuid) {
long msb = uuid.getMostSignificantBits();
long lsb = uuid.getLeastSignificantBits();
byte[] uuidBytes = new byte[16];
for (int i = 15; i >= 8; i--) {
uuidBytes[i] = (byte) (lsb & 0xFFL);
lsb >>= 8;
}
for (int i = 7; i >= 0; i--) {
uuidBytes[i] = (byte) (msb & 0xFFL);
msb >>= 8;
}
return new Binary((byte) 0x04, uuidBytes);
}
/**
* Convert a Binary with a subtype 0x04 to a UUID object
* Please note: the subtype is not being checked.
*/
public static UUID fromStandardBinaryUUID(Binary binary) {
long msb = 0;
long lsb = 0;
byte[] uuidBytes = binary.getData();
for (int i = 8; i < 16; i++) {
lsb <<= 8;
lsb |= uuidBytes[i];
}
for (int i = 0; i < 8; i++) {
msb <<= 8;
msb |= uuidBytes[i];
}
return new UUID(msb, lsb);
}
}
On Ubuntu installation follow below link
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/
it would make all the appropriate installtion at path
and config file it as /etc/mongodb.conf
Download & Extract
http://mylinuxnotebook.blogspot.in/2008/11/create-extract-targz-files.html
https://www.mongodb.org/downloads
Download it from
https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-ubuntu1204-3.0.3.tgz
tar -zxvf mongodb-linux-x86_64-ubuntu1204-3.0.3.tgz
This is already compiled binary for ubuntu 12.02 LTS
Theory
mmapv1 storage engine
- mmapv1 sotrage engine good for our requirement heavy volumes of insert, reads and in-place updates.
- journaling enabled by default.
- preallocation files enabled by default with journalling.
- clear shutdown removes all the files in the journal directory. dirty shutdown doesnt , used to recover the database to a consistent state when mongo restarted.
- To speed the frequent sequential writes that occur to the current journal file, you can ensure that the journal directory is on a different filesystem from the database data files.
- it is more efficient to preallocate journal files than create new journal files as needed.
- The goal here is to have "the next" file already allocated when the last file runs out of space. Pre-allocation is often much faster than on-demand allocation.
- prealloc files do not contain data, but are rather simply preallocated files that are ready to use that are truly preallocated by the file system (i.e. they are not "sparse"). It is thus safe to remove them, but if you restart mongod with journaling, it will create them again if they are missing.*
- Add "smallfiles=true" in the MongoDB configuration file.This will cause MongoDB to use 128M prealloc files instead of 1GB, but does not seem to reduce total journal size
- disable http interface by default.
- MongoDB will use as much free memory as it can, swapping to disk as needed. Deployments with enough memory to fit the application’s working data set
in RAM will achieve the best performance.
- mmapv1 have concurrency model. so adding more CPU may help but doesn't improve much.Adding extra RAM it would reduce page faults.
- read concern = "majority"
- connection pool size to suit use case , beginning at 110-115% of typical no of concurrent db requests.
connPoolStats command returns information regarding the number of open connections to the current database
- Due to its concurrency model, the MMAPv1 storage engine does not require many CPU cores .
As such, increasing the number of cores can help but does not provide significant return.
- Increasing the amount of RAM accessible to MongoDB may help reduce the frequency of page faults
- The output from mongostat provides statistics on the number of active reads/writes in the (ar|aw) column.
- MongoDB has good results and a good price-performance ratio with SATA SSD (Solid State Disk).
- Using SSDs or increasing RAM may be more effective in increasing I/O throughput.
- If the NUMA (Non-Uniform Access Memory)configuration may degrade performance, MongoDB prints a warning.
There is a big difference between mmapv1 and WiredTiger in the way they use memory.
------------------------------------------------------------------------------------------------------------------------
mmapv1 Storage Engine -
While mmap actually uses memory mapping, so that the database cache is actually accounted as cache in the OS.
MongoDB automatically uses all free memory on the machine as its cache. System resource monitors show that MongoDB uses a lot of memory, but its usage is dynamic.
If another process suddenly needs half the server’s RAM, MongoDB will yield cached memory to the other process
This means that MongoDB will use as much free memory as it can, swapping to disk as needed. Deployments with enough memory to fit the application’s working data set in RAM will achieve the best performance.
mmapv1 have concurrency model. so adding more cpu may help but doesn't improve much.Adding extra RAM it would reduce page faults.
mongodb while writing a document, it blocks all read request till that write is complete. once write is complete, lock is released to process blocked requests.
------------------------------------------------------------------------------------------------------------------------
Wired Tiger Storage Engine -
While WiredTiger uses memory pool defined at the application start.
Depending on the amount of memory in your system, by default it should use either 1GB or half the system RAM for WiredTiger pool.
(that would explain jump from 8 % to 58 % or so) wiredtiger is multithread storage engine . so add cpu can increase throughput.
With WiredTiger, MongoDB utilizes both the WiredTiger cache and the filesystem cache.
Via the filesystem cache, MongoDB automatically uses all free memory that is not used by the WiredTiger cache or by other processes.
Data in the filesystem cache is compressed.
WiredTiger cache can be configurable.The WiredTiger cache is only one component of the RAM used by MongoDB.
In addition, the operating system will use any free RAM to buffer filesystem blocks.
To accommodate the additional consumers of RAM, you may have to decrease WiredTiger cache size.
To view statistics on the cache and eviction rate, see the wiredTiger.cache field returned from the serverStatus command
------------------------------------------------------------------------------------------------------------------------
http://docs.mongodb.org/manual/tutorial/add-user-administrator/
$ mongo
> db.createUser(
... {
... user:"theadmin",
... pwd:"theadminpassword",
... roles:[{role:"userAdminAnyDatabase",db:"admin"}]
... }
... );
> db.getUsers();
[
{
"_id" : "admin.theadmin",
"user" : "theadmin",
"db" : "admin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
}
]
}
]
> use test_keyspace;
> db.createUser(
... ... ... {
... ... ... user:"amit",
... ... ... pwd:"pwdforamit",
... ... ... roles:[
... ... {role:"readWrite",db:"test_keyspace"}
... ... ]
... ... }
... ... );
> db.getUsers();
> db.getUser("amit");
show collections;
use test_keyspace
db.contentDump.find().pretty()
db.tablename.drop()
http://docs.mongodb.org/manual/reference/method/js-user-management/
following commands are success
mongo --port 27017 -u theadmin -p theadminpassword --authenticationDatabase admin --verbose
mongo --port 27017 -u amit -p pwdforamit --authenticationDatabase test_keyspace --verbose
-------------------------------------------------------------------------------------------------------------------------------------------
While Creating New User in MongoDB, it has to be created under admin?
If no users created you can create new user without any authentification, but if you have created admin user for specific database you should authentifcate, and then perform any operation.
Documentation:
If no users are configured in admin.system.users, one may access the database from the localhost interface without authenticating. Thus, from the server running the database (and thus on localhost), run the database shell and configure an administrative user:
$ ./mongo
> use admin
> db.addUser("theadmin", "anadminpassword")
We now have a user created for database admin. Note that if we have not previously authenticated, we now must if we wish to perform further operations, as there is a user in admin.system.users.
> db.auth("theadmin", "anadminpassword")
We can view existing users for the database with the command:
> db.system.users.find()
Now, let's configure a "regular" user for another database.
> use test_keyspace
> db.addUser("amit", "pwdforamit")
-------------------------------------------------------------------------------------------------------------------------------------------
How to use authentication in Mongodb 3.0?
1 Start MongoDB 3.0 without --auth enabled (in /etc/mongod.conf), so you can change how it authenticates.
2 create admin user in mongodb explained above
3. mongo --port 27017 -u theadmin -p theadminpassword --authenticationDatabase admin --verbose
4. on mongod shell do the following.
Modify the collection admin.system.version such that the authSchema's currentVersion is 3 instead of 5 (3 is using SCRAM-SHA-1).
As stated here (http://docs.mongodb.org/manual/core/authentication/)MongoDB 3 authentication mechanism has been changed from : MongoDB Challenge and Response (MONGODB-CR) to challenge and response mechanism (SCRAM-SHA-1). You have to delete your created user then change admin.system.version.authSchema to 3 instead of 5. Then recreating your user should solve the problem.
>
> var schema = db.system.version.findOne({"_id" : "authSchema"})
> schema.currentVersion = 3
> db.system.version.save(schema)
> exit
currentVersion = 3 will make the default MONGODB-CR (challenge and response), the default for MongoDB version 2x. Version 3.0 uses SCRAM-SHA-1 by default instead.
From Java code, you need to call sh1 mongodcredential API instead of cr credentials , as its changed in 3.x version.
5. as schema.currentVersion is made 3 (existing value being 5), no need to mention authenticationMechanisms like below
restart MongoDB with --auth enabled. from /webapps/mongo/bin I would run ./mongod --dbpath $HOME/webapps/mongo/data --setParameter authenticationMechanisms=MONGODB-CR --auth --port 1400
The setParameter authenticationMechanisms=MONGODB-CR may be redundant, or meaningless.
you also need to comment the bind_ip line in /etc/mongod.conf sothat anybody can connect to mongodb.
-------------------------------------------------------------------------------------------------------------------------------------------
Robmongo --
The File etc/mongod.conf has a line 'bind_ip'. In this line, you originally have to add the IP address which you want to access your database.
But, it don't work! You should better comment this line.
But, you don't have any authentication now, so you have to add authentication. Here you have an tutorial about this: http://ghosttx.com/2012/03/how-to-connect-to-a-remote-mongodb-server-with-mongohub-for-mac/
When you have done that, you have to enable authentication. You can do this by editing etc/mongod.conf again, and uncomment the line 'Auth = true'.
Now you can connect with you Mongo Database ;)
http://www.mongovue.com/2011/08/04/mongovue-connection-to-remote-server-over-ssh/
Im on OSX and connecting to Ubuntu 14 / Mongo 2.6.7 on VPS and when Ive added my ssh details to the Robomongo all seem to work ok (Ive also changed the mongo config to remove the ip_bing and enabled port 27017)
-------------------------------------------------------------------------------------------------------------------------------------------
it was using SCRAM-SHA-1 as it's authentication mechanism instead of MongoDB-CR.
Run-time database configuration : (http://docs.mongodb.org/manual/administration/configuration/)
copy it to your any user path , make appropriate changes like making it as daemon, log, dbpath etc..
start mongodb as daemon as follows:
mongod --config /
mongod --dbpath /usr/local/mongodb-data can also be done.. but better to do it in file as it has many options
default mongodb would be listening on port 27017 for db connections
default admin web console listening on port 28017 for connections
if using the mongo shell as a client, you can specify the database for the user with the –authenticationDatabase option.
Together, the user’s name and database serve as a unique identifier for that user.
MongoDB stores all user information, including name, password, and the user's database, in the system.users collection in the admin database.
User administrators
create the user administrator as the first user. Then let this user create all other users.
1) Security Introduction
Maintaining a secure MongoDB deployment requires administrators to implement controls to ensure that users and applications have access to only the data that they require. MongoDB provides features that allow administrators to implement these controls and restrictions for any MongoDB deployment.
2) Authentication
Before gaining access to a system all clients should identify themselves to MongoDB. This ensures that no client can access the data stored in MongoDB without being explicitly allowed.
MongoDB supports a number of authentication mechanisms that clients can use to verify their identity. MongoDB supports two mechanisms: a password-based challenge and response protocol and x.509
Authentication is the process of verifying the identity of a client. When access control, i.e. authorization, is enabled, MongoDB requires all clients to authenticate themselves first in order to determine the access for the client.
Although authentication and authorization are closely connected, authentication is distinct from authorization. Authentication verifies the identity of a user; authorization determines the verified user’s access to resources and operations.
3) Client Users
To authenticate a client in MongoDB, you must add a corresponding user to MongoDB. When adding a user, you create the user in a specific database. Together, the user’s name and database serve as a unique identifier for that user. That is, if two users have the same name but are created in different databases, they are two separate users. To authenticate, the client must authenticate the user against the user’s database
For instance, if using the mongo shell as a client, you can specify the database for the user with the –authenticationDatabase option.
To add and manage user information, MongoDB provides the db.createUser() method as well as other user management methods. For an example of adding a user to MongoDB, see Add a User to a Database.
MongoDB stores all user information, including name, password, and the user's database, in the system.users collection in the admin database.
4) Add a User to a Database & Create a User Administrator:
Overview
User administrators create users and create and assigns roles. A user administrator can grant any privilege in the database and can create new ones. In a MongoDB deployment, create the user administrator as the first user. Then let this user create all other users.
a)userAdmin
Provides the ability to create and modify roles and users on the current database. This role also indirectly provides superuser access to either the database or, if scoped to the admin database, the cluster. The userAdmin role allows users to grant any user any privilege, including themselves.
The userAdmin role explicitly provides the following actions:
changeCustomData
changePassword
createRole
createUser
dropRole
dropUser
grantRole
revokeRole
viewRole
viewUser
To provide user administrators, MongoDB has userAdmin and userAdminAnyDatabase roles, which grant access to actions that support user and role management. Following the policy of least privilege userAdmin and userAdminAnyDatabase confer no additional privileges.
Carefully control access to these roles. A user with either of these roles can grant itself unlimited additional privileges. Specifically, a user with the userAdmin role can grant itself any privilege in the database. A user assigned either the userAdmin role on the admin database or the userAdminAnyDatabase can grant itself any privilege in the system.
b)userAdminAnyDatabase
Provides the same access to user administration operations as userAdmin, except it applies to all databases in the cluster. The role also provides the following actions on the cluster as a whole:
authSchemaUpgrade
invalidateUserCache
listDatabases
The role also provides the following actions on the admin.system.users and admin.system.roles collections on the admin database, and on legacy system.users collections from versions of MongoDB prior to 2.6:
collStats
dbHash
dbStats
find
killCursors
planCacheRead
Prerequisites:
a) Required Access
You must have the createUser action on a database to create a new user on that database.
You must have the grantRole action on a role’s database to grant the role to another user.
If you have the userAdmin or userAdminAnyDatabase role, you have those actions.
b) First User Restrictions
If your MongoDB deployment has no users, you must connect to mongod using the localhost exception or use the --noauth option when starting mongod to gain full access the system. Once you have access, you can skip to Creating the system user administrator in this procedure.
If users exist in the MongoDB database, but none of them has the appropriate prerequisites to create a new user or you do not have access to them, you must restart mongod with the --noauth option.
-------------------------------------------------------------------------------------------------------------------------------------------
http interface http://107.108.205.184:28017
run mongodb with --rest flaf
sudo /usr/bin/mongod --config /etc/mongod.conf --rest
Example classes:
Features (used in the code)
- create db connection.
- CRUD queries.
- table contains timestamp when expired, mongo purge rows automatically.
- ensuring index while inserting record : it creates index for provided column/field and restrict adding duplicate entries.
- can create Geospatial indexes : easy for searching location based range records(contain long,lat {order to be maintained}).
Good performance compared to pro-grammatically comparing each record (long,lat) with input location.
ConnectInfo.java
package com.test.mongodb;public class ConnectInfo{
private String host;
private int port;
public ConnectInfo(final String newHost, final int newPort){
this.host = newHost;
this.port = newPort;
}
public String getHost() {
return host;
}
public void setHost(String host) {
this.host = host;
}
public int getPort() {
return port;
}
public void setPort(int port) {
this.port = port;
}
}
MongoConnector.java
package com.test.mongodb;import java.util.ArrayList;
import java.util.List;
import org.apache.log4j.Logger;
import org.jongo.Jongo;
import org.jongo.MongoCollection;
import com.mongodb.DB;
import com.mongodb.MongoClient;
import com.mongodb.MongoCredential;
import com.mongodb.ServerAddress;
public class MongoConnector {
static Logger logger = Logger.getLogger(MongoConnector.class);
MongoClient mongoClient;
private static MongoConnector instance = null;
private Jongo jongo = null;
public MongoConnector(ConnectInfo cnInfo){
DB database;
this.connect(cnInfo.getHost(), cnInfo.getPort(),DataStore.dbname);
database = mongoClient.getDB(DataStore.dbname);
jongo = new Jongo(database);
}
public static MongoConnector getInstance(ConnectInfo cnInfo){
if( null == instance){
instance = new MongoConnector(cnInfo);
}
return instance;
}
public MongoCollection getCollection(String tableName){
return jongo.getCollection(tableName); // mapper
}
/**
* Connect to Mongo Cluster specified by provided node IP
* address and port number.
*
* @param node Cluster node IP address.
* @param port Port of cluster host.
*/
public void connect(final String node, final int port, String database)
{
/*
List
seeds.add(new ServerAddress(node,port));
List
credentials.add(
MongoCredential.createScramSha1Credential(
"amit",
database,
"pwdforamit".toCharArray()
)
);
mongoClient = new MongoClient( new ServerAddress(node,port), credentials );
*/
this.mongoClient = new MongoClient( node , port );
}
public MongoClient getMongoClient() {
return mongoClient;
}
public void setMongoClient(MongoClient mongoClient) {
this.mongoClient = mongoClient;
}
}
MongoContentDumpInsertImpl.java
package com.test.mongodb;
import java.util.Date;
import java.util.List;
import java.util.UUID;
import org.apache.log4j.Logger;
import org.jongo.MongoCollection;
import org.jongo.MongoCursor;
import com.google.common.collect.Lists;
import com.mongodb.MongoClient;
import com.test.mongodb.ConnectInfo;
import com.test.mongodb.MongoConnector;
import com.test.mongodb.QueryObject;
import com.test.mongodb.ContentDump;
public class MongoContentDumpInsertImpl{
static Logger logger = Logger.getLogger(MongoContentDumpInsertImpl.class);
private static MongoContentDumpInsertImpl instance =null;
private MongoClient mongoClient = null;
private MongoConnector dbConnect = null;
//private Jongo jongo = null;
private MongoCollection collection = null;
private String pKey = null;
MongoContentDumpInsertImpl(){
pKey = new String("dump_id");
}
public static MongoContentDumpInsertImpl getInstance(){
if( null == instance){
instance = new MongoContentDumpInsertImpl();
}
return instance;
}
@Override
public void setup() throws DataStoreException {
// TODO Auto-generated method stub
}
@Override
public void connect(ConnectInfo cnInfo) throws DataStoreException {
/*DB database;
dbConnect = new MongoConnector();
dbConnect.connect(cnInfo.getHost(), cnInfo.getPort(),"test_keyspace");
this.mongoClient = dbConnect.getMongoClient();
database = mongoClient.getDB("test_keyspace");
jongo = new Jongo(database);
collection = jongo.getCollection("contentDump"); // mapper
*/
dbConnect = MongoConnector.getInstance(cnInfo);
collection = dbConnect.getCollection(DataStore.contentDump);
}
@Override
public ContentDump query(QueryObject qObject) throws DataStoreException {
if (qObject.getField() == null || qObject.getValue() == null){
throw new DataStoreException("Contentdump input query param is null for query");
}
String query = "{"+ qObject.getField()+":#}";
ContentDump record = null;
try{
record = collection.findOne(query,qObject.getValue()).as(ContentDump.class);
}catch(Exception e){
throw new DataStoreException("ContentDump findOne failed");
}
logger.info("[Contentdump] : SELECT Query Successful");
return record;
}
@Override
public void insert(ContentDump record) throws DataStoreException {
logger.info(record.toString());
try{
collection.ensureIndex("{expired:1}","{expireAfterSeconds:0}");
// by setting this attribute , it purges records when timestamp in the 'expired' field passes.
collection.save(record);
}catch(Exception e){
throw new DataStoreException("ContentDump save failed");
}
logger.info("[Contentdump] : INSERT Query Successful");
}
@Override
public void update(ContentDump record) throws DataStoreException {
String query = "{"+ pKey +":#}";
try{
collection.update(query,record.getDump_id()).with(record);
}catch(Exception e){
throw new DataStoreException("ContentDump update failed");
}
logger.info("[Contentdump] : UPDATE Query Successful");
}
@Override
public void delete(QueryObject qObject) throws DataStoreException {
if (qObject.getField() == null || qObject.getValue() == null){
logger.info("[Contentdump] : DELETE Query Failed");
throw new DataStoreException("Contentdump input query param is null for delete query");
}
String query = "{"+ qObject.getField()+":#}";
try{
collection.remove(query,qObject.getValue());
}catch(Exception e){
throw new DataStoreException("ContentDump update failed");
}
logger.info("[Contentdump] : DELETE Query Successful");
}
@Override
public List
List
try{
//Date currentTime = new Date();
String query = "{docid:#}";
MongoCursor
dumpList = Lists.newArrayList(deals.iterator());
}catch(Exception e){
throw new DataStoreException("[ContentDump]: Mongo DB access error for getting all dumps - "+e.getMessage());
}
return dumpList;
}
@Override
public void close() {
// TODO Auto-generated method stub
this.mongoClient.close();
}
}
MongoUUIDConverter.java
package com.test.mongodb
import java.util.UUID;
import org.bson.types.Binary;
public class MongoUUIDConverter {
/**
* Convert a UUID object to a Binary with a subtype 0x04
*/
public static Binary toStandardBinaryUUID(java.util.UUID uuid) {
long msb = uuid.getMostSignificantBits();
long lsb = uuid.getLeastSignificantBits();
byte[] uuidBytes = new byte[16];
for (int i = 15; i >= 8; i--) {
uuidBytes[i] = (byte) (lsb & 0xFFL);
lsb >>= 8;
}
for (int i = 7; i >= 0; i--) {
uuidBytes[i] = (byte) (msb & 0xFFL);
msb >>= 8;
}
return new Binary((byte) 0x04, uuidBytes);
}
/**
* Convert a Binary with a subtype 0x04 to a UUID object
* Please note: the subtype is not being checked.
*/
public static UUID fromStandardBinaryUUID(Binary binary) {
long msb = 0;
long lsb = 0;
byte[] uuidBytes = binary.getData();
for (int i = 8; i < 16; i++) {
lsb <<= 8;
lsb |= uuidBytes[i];
}
for (int i = 0; i < 8; i++) {
msb <<= 8;
msb |= uuidBytes[i];
}
return new UUID(msb, lsb);
}
}
ContentDump.java
package com.test.mongodb
import java.nio.ByteBuffer;
import java.util.Date;
import java.util.UUID;
import com.test.mongodb.MongoUUIDConverter;
public class ContentDump{
private UUID dump_id; // primary key generated using uuid() method to store it as uuid
private UUID docid; // transaction id
private String source; //source of the document
private String access_url; //url with which the content is accessed
private String typeOfContent;
private String dumpMimeType; // mimetype of dump
private String source_dump; //Text dump of the source document
private Date created; //type 'timestamp' auto updated
private Date expired; //type 'timestamp' Expire Time for the Data if required
public ContentDump(){
this.dump_id = UUID.randomUUID();
}
public UUID getDump_id() {
return this.dump_id;
}
public void setDump_id(UUID dump_id) {
this.dump_id = dump_id;
}
public UUID getDocid() {
return this.docid;
}
public void setDocid(UUID docid) {
this.docid = docid;
}
public String getSource() {
return source;
}
public void setSource(String source) {
this.source = source;
}
public String getAccess_url() {
return access_url;
}
public void setAccess_url(String access_url) {
this.access_url = access_url;
}
public String getTypeOfContent() {
return typeOfContent;
}
public void setTypeOfContent(String typeOfContent) {
this.typeOfContent = typeOfContent;
}
public String getSource_dump() {
return source_dump;
}
public void setSource_dump(String source_dump) {
this.source_dump = source_dump;
}
public Date getCreated() {
return created;
}
public void setCreated(Date created) {
this.created = created;
}
public Date getExpired() {
return expired;
}
public void setExpired(Date expired) {
this.expired = expired;
}
public String getDumpMimeType() {
return dumpMimeType;
}
public void setDumpMimeType(String dumpMimeType) {
this.dumpMimeType = dumpMimeType;
}
}
QueryObject.java
package com.test.mongodb
import java.util.UUID;
public class QueryObject{
/*As of now only one fieldValue pair has been considered*/
// column name
private String field;
// column value
private UUID value;
public QueryObject(){
field = null;
value = null;
}
public String getField() {
return field;
}
public void setField(String field) {
this.field = field;
}
public UUID getValue() {
return value;
}
public void setValue(UUID value) {
this.value = value;
}
}
OnlineItems .java
package com.test.mongodb
import java.nio.ByteBuffer;
import java.util.Date;
import java.util.UUID;
import org.jongo.marshall.jackson.oid.MongoId;
import com.fasterxml.jackson.annotation.JsonIgnore;
public class OnlineItems {
@MongoId
UUID docid; //Primary key
UUID txn_id;
String title; //Title Of the Deal
String description; //description can be text or HTML
@JsonIgnore
String source; //source of the document
String url; //public url from where this document can be accessed from source server
String prime_image; //associated image URL; only relative url
String item_category; //Category
@JsonIgnore
String item_type; //Item Type
@JsonIgnore
String deal_type;
//@JsonIgnore commented this as TTL index is dependent on this field
Date validity_end; // type="timeanddate"
public OnlineItems(){
this.docid = UUID.randomUUID();
}
public UUID getDocid() {
return docid;
}
public void setDocid(UUID docid) {
this.docid = docid;
}
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title;
}
public String getDescription() {
return description;
}
public void setDescription(String description) {
this.description = description;
}
public String getSource() {
return source;
}
public void setSource(String source) {
this.source = source;
}
public String getUrl() {
return url;
}
public void setUrl(String url) {
this.url = url;
}
public String getPrime_image() {
return prime_image;
}
public void setPrime_image(String prime_image) {
this.prime_image = prime_image;
}
public String getItem_category() {
return item_category;
}
public void setItem_category(String item_category) {
this.item_category = item_category;
}
public String getItem_type() {
return item_type;
}
public void setItem_type(String item_type) {
this.item_type = item_type;
}
public String getDeal_type() {
return deal_type;
}
public void setDeal_type(String deal_type) {
this.deal_type = deal_type;
}
public Date getValidity_end() {
return validity_end;
}
public void setValidity_end(Date validity_end) {
this.validity_end = validity_end;
}
public UUID getTxn_id() {
return txn_id;
}
public void setTxn_id(UUID txn_id) {
this.txn_id = txn_id;
}
}
OnlineItemsImpl.java
package com.test.mongodb
import java.util.List;
import org.apache.log4j.Logger;
import org.jongo.Jongo;
import org.jongo.MongoCollection;
import org.jongo.MongoCursor;
import com.google.common.collect.Lists;
import com.mongodb.DB;
import com.mongodb.MongoClient;
import com.test.mongodb.ConnectInfo;
import com.test.mongodb.DataStoreException;
import com.test.mongodb.MongoConnector;
import com.test.mongodb.QueryObject;
import com.test.mongodb.OnlineItems;
public class OnlineItemsImpl{
static Logger logger = Logger.getLogger(OnlineItemsImpl.class);
private static OnlineItemsImpl instance =null;
private MongoClient mongoClient = null;
private MongoConnector dbConnect = null;
//private Jongo jongo = null;
private MongoCollection collection = null;
private String pKey = null;
OnlineItemsImpl(){
pKey = new String("docid");
}
public static OnlineItemsImpl getInstance(){
if( null == instance){
instance = new OnlineItemsImpl();
}
return instance;
}
@Override
public void setup() throws DataStoreException {
// TODO Auto-generated method stub
}
@Override
public void connect(ConnectInfo cnInfo) throws DataStoreException {
dbConnect = MongoConnector.getInstance(cnInfo);
collection = dbConnect.getCollection("OnlineItems");
}
@Override
public OnlineItems query(QueryObject qObject) throws DataStoreException {
if (qObject.getField() == null || qObject.getValue() == null){
throw new DataStoreException("OnlineItems input query param is null for query");
}
String query = "{"+ qObject.getField()+":#}";
OnlineItems record = null;
try{
record = collection.findOne(query,qObject.getValue()).as(OnlineItems.class);
}catch(Exception e){
throw new DataStoreException("OnlineItems findOne failed");
}
logger.info("Online[OnlineItems] : SELECT Query Successful");
return record;
}
@Override
public void insert(OnlineItems record) throws DataStoreException {
logger.info(record.toString());
try{
collection.save(record);
}catch(Exception e){
throw new DataStoreException("OnlineItems save failed");
}
logger.info("Online[OnlineItems] : INSERT Query Successful");
}
@Override
public void update(OnlineItems record) throws DataStoreException {
String query = "{"+ pKey +":#}";
try{
collection.update(query,record.getDocid()).with(record);
}catch(Exception e){
throw new DataStoreException("OnlineItems save failed");
}
logger.info("Online[OnlineItems] : UPDATE Query Successful");
}
@Override
public void delete(QueryObject qObject) throws DataStoreException {
if (qObject.getField() == null || qObject.getValue() == null){
logger.info("Online[OnlineItems] : DELETE Query Failed");
throw new DataStoreException("OnlineItems input query param is null for delete query");
}
String query = "{"+ qObject.getField()+":#}";
try{
collection.remove(query,qObject.getValue());
}catch(Exception e){
throw new DataStoreException("OnlineItems update failed");
}
logger.info("Online[OnlineItems] : DELETE Query Successful");
}
@Override
public void close() {
// TODO Auto-generated method stub
this.mongoClient.close();
}
@Override
public List getAllOnlineItems(int offset, int limit,double longi, double latti)
throws DataStoreException {
List restuarantList = null;
String Query = "{location:"
+"{$geoWithin:"
+"{"
+"$centerSphere:"
+"[[#,#],#]"
+"}"
+"}"
+ "}";
try{
// find("{ inappropriate: false }")
collection.ensureIndex("{location:\"2d\"}");
//MongoCursor restaurants = collection.find(Query,longi,latti).skip(offset).limit(limit).as(OnlineItems.class);
MongoCursor restaurants = collection.find(Query,longi,latti,5/3963.2).as(OnlineItems.class);
restuarantList = Lists.newArrayList(restaurants.iterator());
}catch(Exception e){
throw new DataStoreException("[OnlineItems] : Mongo DB access error for getAllOnlineItems - "+e.getMessage());
}
return restuarantList;
}
}
DataStoreException.java
package com.test.mongodb;
public class DataStoreException extends Exception {
String errString;
public DataStoreException(){
super();
}
public DataStoreException(String err){
super(err);
errString = err;
}
public String getError(){
return errString;
}
}
MongoDB Deployment
Tips :
1 Three member replica sets provide enough redundancy to survive most network partitions and other system failures
2 ensure that each member of a replica set is accessible by way of resolvable DNS or hostnames.
You should either configure your DNS names appropriately or set up your systems’ /etc/hosts file to reflect this configuration.
3 The rs.reconfig() shell method can force the current primary to step down, causing an election.
When the primary steps down, all clients will disconnect. This is the intended behavior.
While most elections complete within a minute, always make sure any replica configuration changes occur during scheduled maintenance periods.
4 To be completely sure that the replica set configuration is gone from the instance, make sure that the local.system.replset collection is empty.
Once that is done, and you are happy with your standalone instance, you can then restart with a different --replSet argument (OR start mongo and
mention new replica set name in config in mongo shell and configure) and go through the replica set configuration process again.
-define config.
-initiate config rs.initiate(config).
-add other members rs.add().
5 Always start rolling replica set maintenance with the secondaries, and finish with the maintenance on primary member.
6 Always use rs.stepDown() to force the primary to become a secondary, before stopping the server.
This facilitates a more efficient election process.
1.
to check mongodb installation
$yum list installed | grep mongo
2.
mongod --replSet "rs0" OR mongod --config $HOME/.mongodb/config (repl set is defined in config file)
rs.initiate()
MongoDB initiates a set that consists of the current member and that uses the default replica set configuration.
rs.conf()
rs.status()
3. For testing or development, you can start 3 instances of mongo on single machine.
a.mkdir -p /home/user/mongodb/rs0-0 /home/user/mongodb/rs0-1 /home/user/mongodb/rs0-2
folder to contain 3 databases files for 3 instances
b.Start mongod instance in different shells with different ports
mongod --port 27017 --dbpath /srv/mongodb/rs0-0 --replSet rs0 --smallfiles --oplogSize 128
mongod --port 27018 --dbpath /srv/mongodb/rs0-1 --replSet rs0 --smallfiles --oplogSize 128
mongod --port 27019 --dbpath /srv/mongodb/rs0-2 --replSet rs0 --smallfiles --oplogSize 128
The --smallfiles and --oplogSize settings reduce the disk space that each mongod instance uses. This is ideal for testing and development deployments as it prevents overloading your machine
c. Connect to any of mongo instance , .e.g. mongo --port 27017
d. In mongo shell,
>rsconf = {
_id: "rs0",
members: [
{
_id: 0,
host: "localhost:27017"
}
]
}
>rs.initiate( rsconf )
>rs.conf() // to check whats current conf
e. current shell become primary , add second and third mongod instances to rs.
rs.add("localhost:27018")
rs.add("localhost:27019")
new rs wll elect primary
4.
A) This for fresh instance
-set replica set name in /etc/mongod.conf for all machines/instance who all participating in mongo cluster.
- start mongo service in all instances
-mongo shell
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3}]}
>rs.initiate(cfg)
>rs.add("mongo-secondary:27017"); // by default priority is 1
>rs.addArb("restOnline:27017"); // arbiter can vote but cannt become primary.
B) This in AMI created instances (replica set name is already initiated)
You could do this similar to the follow from the {{mongo}} shell:
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3}]}
rs.add("mongo-secondary:27017"); // by default priority is 1
rs.addArb("restOnline:27017"); // arbiter can vote but cannt become primary.
OR at one shot you can reconfigure/configure replica set
cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3},{"_id" : 1,"host" : "mongo-secondary:27017","priority":2},{"_id" : 2,"host" : "restOnline:27017","arbiterOnly":true}]}
>rs.reconfig(cfg, {force:true})
e.g. to remove rs.remove("mongo-secondary:27017");
Here we don't need to initiate (rs.initiate()) replica set because its already initiated as it does have existing configuration like our current case.
If its fresh replica set and it doesn't have any configuration then need to initiate with config mentioned.
I have kept same existing replica set name in all members of replica set and tried changing/modifying configuration.
Once launched instances (2 count , one for primary & other for secondary) from AMI. mongo shell was not showing neither primary/secondary
for both.rs.status() mentioning that mongo instance has been removed from replicaset.
The reason being dns name have been changed to mongo-primary only (its not having dev_mongo-primary , same for replica machine/instance).
And Mongo replica set configuration mentions primary and secondary hostnames as dev_mongo-primary and dev_mongo-secondary respectively.
(you can check config using command rs.conf())
To solve this , we need to reconfigure replica set configuration with new dns names/ips. This can be only done through primary mongo. sothat
need to bring either of the machine to primary status.
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3}]}
> rs.reconfig(cfg, {force:true})
This would bring mongo instance to primary status. can be checked in mongo shell , it would turn to mongo primary.
Now start mongod in other machine (which is supposed to be secondary/replica machine).
{note : It may show earlier configuration as soon as you start and check conf using rs.conf(). After some time,
it would reflect with latest/changed config in other machine but still other machine is not yet secondary/replica machine.)
Go back to primary machine and add secondary machine host in rs.Using in mongo shell of primary execute following command
>rs.add("mongo-secondary:27017");
It would reflect replica set new config it in other mongo instance as its sharing same replica set name.
and other mongo instance turn into secondary. can be checked in mongo shell and verify rs.conf().
Same execution for adding arbiter member in replicaset
>rs.addArb("restOnline:27017"); // arbiter is running in online machine
5.Adding members to replica set
You can use these procedures
a. to add new members to an existing set.
- Adding new members freshly having empty data directory. make sure added new member become secondary in rs.
mongodb can copy all data to this instance as part of replication process.this process takes times doenst need admin intevention
- Manually copy data directory from an existing member.This new secondary member will catch up to the current state of replica set.
It would shorten the amount of time for this new member to become current.
(Always use filesystem snapshots to create a copy of a member of the existing replica set. Do not use mongodump and mongorestore to seed a new replica set member.)
Use rs.printReplicationInfo() to check the current state of replica set members with regards to the oplog.
Ensure that you can copy the data directory to the new member and begin replication within the window allowed by the oplog i.e. oplog size
shouldn't be so small that by that time you copy to new member , oplog size (in machine from which you copied) shouldnt spill. otherwise
new instance will perform initial sync which resynchronises the data.
oplog maintain idempotency (The quality of an operation to produce the same result given the same input, whether run once or run multiple times)
oplog translate multi-updates into individual operations in order to maintain idempotency.
b. to “re-add” a removed member
(note- If the removed member’s data is still relatively recent, it can recover and catch up easily due to idempotency of oplog)
By this time , you might have come to know to re-add the member . refer a section.
Data Files
Always use filesystem snapshots to create a copy of a member of the existing replica set. Do not use mongodump and mongorestore to seed a new replica set member
6.
when ip of one of the member of replica set is to be changed , then rs.reconfig(cfg) need to command on mongo shell
with new ip and port for member . As part of bringing the new member on board and to sync data set to new member.
it need to initialize by command , rs.initiate().
it wouldnt work if there oplog is not empty. so clear oplog.
>use local
>db.oplog.rs.drop()
Before:
config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.100:15000'}]
}
After:
config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.200:15000'}]
}
This above thing is as good as moving replica set to new location and configure
OR just changing one of the member ip.
In case still its not working , delete all replication and oplog.
>use local
>db.dropDatabase()
>config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.100:15000'}]
}
>rs.initiate(config)
still not, try force option explained in point 2.
7. While removing members from replica set , you can access mongod in secondary/arbiter instances from primary machine or vice versa or any other machine.
you don't really need to visit mongo shell of other instances.
you can access mongo shell of other instances using following command
$mongo --host mongoreplica@core
to shut down mongo server
>db.shutdownServer()
now connect to mongo shell of primary instance.
$mongo --host mongo-primary
>rs.remove("mongo-secondary:27017")
>rs.remove("rsOnline:27017")
OR you can manually visit instance and shutdown mongod service and remove from config in primary shell.
8. Removing a member from replica set
a.Remove a Member Using rs.remove()
b.Remove a Member Using rs.reconfig()
9.
Replace a Replica Set Member
To change the hostname for a replica set member modify the members[n].host field
cfg = rs.conf()
cfg.members[0].host = "mongo2.example.net"
rs.reconfig(cfg)
Any replica set configuration change can trigger the current primary to step down, which forces an election.
During the election, the current shell session and clients connected to this replica set disconnect,
which produces an error even when the operation succeeds.
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "52.26.239.117:27017","priority":3}]}
rs.reconfig(cfg, {force:true})
rs.add("52.33.147.186:27017");
rs.add("54.152.103.42:27017");
10.
Change the Size of the Oplog on the Primary And secondary
Secondary Member
1) Recreate the Oplog with a New Size and a Seed Entry
> rs.shutdownServer() And restarting member in standalone mode i.e. restart it on different port without replica set.
> mongod --port 37017 --dbpath /srv/mongodb
> use local
> db = db.getSiblingDB('local')
> db.temp.drop()
> db.temp.save( db.oplog.rs.find( { }, { ts: 1, h: 1 } ).sort( {$natural : -1} ).limit(1).next() )
> db.temp.find()
Remove existing oplog existing collection
> db = db.getSiblingDB('local')
> db.oplog.rs.drop()
Now restart mongod as a member of the replica set on its usual port :
> db.shutdownServer()
> mongod --replSet rs0 --dbpath /srv/mongodb
The replica set member will recover and “catch up” before it is eligible for election to primary.
2) Create backup of existing oplog and create new oplog
> mongodump --db local --collection 'oplog.rs' --port 37017
> db.shutdownServer()
set new oplog size using command or in config file.
and now start mongod as a member of the replica set on its usual port.
(mongod --replSet rs0 --dbpath /srv/mongodb)
Primary Member
To finish the rolling maintenance operation,
- step down the primary with the rs.stepDown() method.and then rs.shutdownServer() And restarting member in standalone mode i.e. restart it on different port without replica set.
- Recreate the oplog with the new size and with an old oplog entry as a seed. (For this follow procedure 1 mentioned for secondary member above)
- Restart the mongod instance as a member of the replica set.
NOTE : Always start rolling replica set maintenance with the secondary members, and finish with the maintenance on primary member.
10.YCSB benchmark tool (https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload)
a. Running this tool requires python, java. (for bulding requires maven)
python required modules mentioned in /bin/ycsb script file.
you may need to install 'yum install python-argparse' ( this module is not present in python 2.6.6)
or install using pip(python package index - package manager).
b. Source code for this tool is at (https://github.com/brianfrankcooper/YCSB)
you can use this tool by compiling the source code for mongodb specific code
and run the tool.
OR
You can run this tool directly, the runnable jars are present at
(https://github.com/brianfrankcooper/YCSB/releases/download/0.5.0/ycsb-0.5.0.tar.gz)
c. To benchmark , this tool provides command to load workload i.e. db,tables & documents into mongo db.
Commands for mongodb:
./bin/ycsb load mongodb-async -s -P workloads/workloada > outputLoad.txt
./bin/ycsb run mongodb-async -s -P workloads/workloada > outputRun.txt
./bin/ycsb load mongodb -s -P workloads/workloada > outputLoad.txt
./bin/ycsb run mongodb -s -P workloads/workloada > outputRun.txt
d. To test mongodb using this tool, there are Parameters which are configurable to benchmark
mongodb.url
mongodb.batchsize
mongodb.upsert
mongodb.writeConcern
mongodb.readPreference
mongodb.maxconnections
mongodb.threadsAllowedToBlockForConnectionMultiplier
command example with configured parameter:
./bin/ycsb load mongodb-async -s -P workloads/workloada -p mongodb.url=mongodb://localhost:27017/ycsb?w=0
./bin/ycsb load mongodb -s -P workloads/workloada -p mongodb.url=mongodb://localhost:27017/ycsb?w=0
Hidden member configuration for backup and reporting
>rs.add({_id:1,host: "testone-mongo-arbiter:27017", priority: 0, hidden: true})
Optimal Queue depth to achieve lower latency and highest IOPS is between 4-8;1 QD per 500 IOPS
EBS-Optimized offers consistent latency experience
To increase write performance/throughput
- increase in RAM
- SSD
To reduce write latency
- increase network bandwidth
- Use EBS-Optimized or 10 Gigabit Network Instances
IOPS provisioned IOPS SSD volume attached to an EBS-optimised EC2 instance (OR)
General purpose SSD volume attached to an EBS-optimised EC2 instance.
To consistently achieve IOPS , EC2 instance should be launched as EBS-optimised
General Purpose SSD or Provisioned IOPS SSD volumes that are attached to an EBS-optimized instance or an instance with 10 Gigabit network connectivity
The only way to ensure sustained reliable network bandwidth between your EC2 instance and your EBS volumes is to launch the EC2 instance as EBS-optimized or choose an instance type with 10 Gigabit network connectivity.
Launching an instance that is EBS-optimized provides you with a dedicated connection between your EC2 instance and your EBS volume.
Be sure to choose an EBS-optimized instance that provides more dedicated EBS throughput than your application needs; otherwise, the Amazon EBS to Amazon EC2 connection will become a performance bottleneck.
The dedicated throughput to Amazon EBS, the maximum amount of IOPS the instance can support if you are using a 16 KB I/O size,
Amazon EBS measures each I/O operation per second that is 256 KiB or smaller as one IOPS.
I/O operations that are larger than 256 KiB are counted in 256 KiB capacity units. For example, a single 1,024 KiB I/O operation would count as 4 IOPS; however, 1,024 I/O operations at 1 KiB each would count as 1,024 IOPS
When you create a 3,000 IOPS volume, either a 3,000 IOPS Provisioned IOPS SSD volume or a 1,000 GiB General Purpose SSD volume, and attach it to an EBS-optimized instance that can provide the necessary bandwidth, you can transfer up to 3,000 chunks of data per second (provided that the I/O does not exceed the per volume throughput limit of the volume).
Ephemeral storage is lost when instances are stopped/terminated, so it is generally not recommended unless you understand the data loss implications.
-Due to its concurrency model, the MMAPv1 storage engine does not require many CPU cores .
As such, increasing the number of cores can help but does not provide significant return.
-Increasing the amount of RAM accessible to MongoDB may help reduce the frequency of page faults
-The output from mongostat provides statistics on the number of active reads/writes in the (ar|aw) column.
- MongoDB has good results and a good price-performance ratio with SATA SSD (Solid State Disk).
- Using SSDs or increasing RAM may be more effective in increasing I/O throughput.
- If the NUMA (Non-Uniform Access Memory)configuration may degrade performance, MongoDB prints a warning.
For almost all deployments EBS will be the better choice. For production systems we recommend using
EBS-optimized EC2 instances
Provisioned IOPS (PIOPS) EBS volumes
For best performance we recommend separate volumes for data files, the journal, and the log.
Each has different write behavior, and placing them on separate volumes reduces I/O contention.
Using different storage devices will affect your ability to create snapshot-style backups of your data, since the files will be on different devices and volumes.
MongoDB installed via yum
Individual PIOPS EBS volumes for data (1000 IOPS), journal (250 IOPS), and log (100 IOPS)
Ensure that readahead settings for the block devices that store the database files are appropriate. For random access use patterns, set low readahead values. A readahead of 32 (16 kB) often works well.
For a standard block device, you can run sudo blockdev --report to get the readahead settings and sudo blockdev --setra to change the readahead settings. Refer to your specific operating system manual for more information.
Assign swap space for your systems. Allocating swap space can avoid issues with memory contention and can prevent the OOM Killer on Linux systems from killing mongod.
For the MMAPv1 storage engine, the method mongod uses to map files to memory ensures that the operating system will never store MongoDB data in swap space
Most MongoDB deployments should use disks backed by RAID-10
With the MMAPv1 storage engine, the Network File System protocol (NFS) is not recommended as you may see performance problems when both the data files and the journal files are hosted on NFS. You may experience better performance if you place the journal on local or iscsi volumes.
If you decide to use NFS, add the following NFS options to your /etc/fstab file: bg, nolock, and noatime.
With MMAPv1, MongoDB automatically uses all free memory on the machine as its cache. System resource monitors show that MongoDB uses a lot of memory, but its usage is dynamic. If another process suddenly needs half the server’s RAM, MongoDB will yield cached memory to the other process.
Technically, the operating system’s virtual memory subsystem manages MongoDB’s memory. This means that MongoDB will use as much free memory as it can, swapping to disk as needed. Deployments with enough memory to fit the application’s working data set in RAM will achieve the best performance.
When running MongoDB in production on Linux, it is recommended that you use Linux kernel version 2.6.36 or later.
With the MMAPv1 storage engine, MongoDB preallocates its database files before using them and often creates large files. As such, you should use the XFS and EXT4 file systems. If possible, use XFS as it generally performs better with MongoDB.
To explain, there are other considerations here - snapshots for one. Thanks to the journal you no longer have to fsync and lock the database to get a consistent snapshot (be it EBS or LVM or other). However the journal has to be included in the snapshot for that to be the case. Hence whatever node
you use for backing up, if you intend to snapshot without taking downtime for that node, then you need to make sure the journal is included.
MongoDB will only pre-allocate the journal files if it believes that it'll be faster to pre-allocate files of a given size (three files 128 MB each if running with --smallfiles and three files 1 GB each if not running with --smallfiles) than to allocate them on-demand.
General Purpose SSD volumes have a throughput limit between 128 MB/s and 160 MB/s per volume (depending on volume size), which pairs well with a 1,000 Mbps EBS-optimized connection. Instance types that offer more than 1,000 Mbps of throughput to Amazon EBS can use more than one General Purpose SSD volume to take advantage of the available throughput
Provisioned IOPS SSD volumes have a throughput limit range of 256 KiB for each IOPS provisioned, up to a maximum of 320 MiB/s (at 1,280 IOPS)
Launching an instance that is EBS-optimized provides you with a dedicated connection between your EC2 instance and your EBS volume. However, it is still possible to provision EBS volumes that exceed the available bandwidth for certain instance types, especially when multiple volumes are striped in a RAID configuration. The following table shows which instance types are available to be launched as EBS-optimized, the dedicated throughput to Amazon EBS, the maximum amount of IOPS the instance can support if you are using a 16 KB I/O size, and the approximate I/O bandwidth available on that connection in MB/s. Be sure to choose an EBS-optimized instance that provides more dedicated EBS throughput than your application needs; otherwise, the Amazon EBS to Amazon EC2 connection will become a performance bottleneck.
IOPS are input/output operations per second. Amazon EBS measures each I/O operation per second (that is 256 KiB or smaller) as one IOPS. I/O operations that are larger than 256 KiB are counted in 256 KiB capacity units. For example, a single 1,024 KiB I/O operation would count as 4 IOPS; however, 1,024 I/O operations at 1 KiB each would count as 1,024 IOPS.
MongoDB processes operations on data in memory. That is, for a document to be accessed that document must be present in the in-memory cache. Any request for a document that is not currently in memory is first read from disk and loaded into memory. The document is retained in the cache once it has been loaded, and subsequent operations on the document will be much faster as access to disk won't be necessary.
The MMAPv1 storage engine uses memory-mapped files, whereas WiredTiger manages objects through its in-memory cache. When you perform a read or write operation on an object that is not currently in memory, it leads to a page fault (MMAPv1) or cache miss (WiredTiger) so that the object can be read from disk and loaded into memory.
If your application’s working set is much larger than the available memory, access requests to some objects will cause reads from disk before the operation can complete. Such requests are often the largest driver of random I/O, especially for databases that are larger in size than available memory. If your working set exceeds available memory on a single server, you should consider sharding your system across multiple servers.
You should pay attention to the read-ahead settings on your block device to see how much data is read in such situations. Having a large setting for read-ahead is discouraged, because it will cause the system to read more data into memory than is necessary, and it might possibly evict other data that may be used by your application. This is particularly true for services that limit block size, such as Amazon Elastic Block Store (EBS) volumes,
Several factors can affect the performance of Amazon EBS volumes, such as instance configuration, I/O characteristics, workload demand, and storage configuration. After you learn the basics of working with EBS volumes, it's a good idea to look at the I/O performance you require and at your options for increasing EBS performance to meet those requirements.
The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk.”. [Note: The difference between fsync() and fdatasync() is that the later does not necessarily update the meta-data associated with a file – such as the “last modified” date – but only the file data.]
1 Three member replica sets provide enough redundancy to survive most network partitions and other system failures
2 ensure that each member of a replica set is accessible by way of resolvable DNS or hostnames.
You should either configure your DNS names appropriately or set up your systems’ /etc/hosts file to reflect this configuration.
3 The rs.reconfig() shell method can force the current primary to step down, causing an election.
When the primary steps down, all clients will disconnect. This is the intended behavior.
While most elections complete within a minute, always make sure any replica configuration changes occur during scheduled maintenance periods.
4 To be completely sure that the replica set configuration is gone from the instance, make sure that the local.system.replset collection is empty.
Once that is done, and you are happy with your standalone instance, you can then restart with a different --replSet argument (OR start mongo and
mention new replica set name in config in mongo shell and configure) and go through the replica set configuration process again.
-define config.
-initiate config rs.initiate(config).
-add other members rs.add().
5 Always start rolling replica set maintenance with the secondaries, and finish with the maintenance on primary member.
6 Always use rs.stepDown() to force the primary to become a secondary, before stopping the server.
This facilitates a more efficient election process.
1.
to check mongodb installation
$yum list installed | grep mongo
2.
mongod --replSet "rs0" OR mongod --config $HOME/.mongodb/config (repl set is defined in config file)
rs.initiate()
MongoDB initiates a set that consists of the current member and that uses the default replica set configuration.
rs.conf()
rs.status()
3. For testing or development, you can start 3 instances of mongo on single machine.
a.mkdir -p /home/user/mongodb/rs0-0 /home/user/mongodb/rs0-1 /home/user/mongodb/rs0-2
folder to contain 3 databases files for 3 instances
b.Start mongod instance in different shells with different ports
mongod --port 27017 --dbpath /srv/mongodb/rs0-0 --replSet rs0 --smallfiles --oplogSize 128
mongod --port 27018 --dbpath /srv/mongodb/rs0-1 --replSet rs0 --smallfiles --oplogSize 128
mongod --port 27019 --dbpath /srv/mongodb/rs0-2 --replSet rs0 --smallfiles --oplogSize 128
The --smallfiles and --oplogSize settings reduce the disk space that each mongod instance uses. This is ideal for testing and development deployments as it prevents overloading your machine
c. Connect to any of mongo instance , .e.g. mongo --port 27017
d. In mongo shell,
>rsconf = {
_id: "rs0",
members: [
{
_id: 0,
host: "localhost:27017"
}
]
}
>rs.initiate( rsconf )
>rs.conf() // to check whats current conf
e. current shell become primary , add second and third mongod instances to rs.
rs.add("localhost:27018")
rs.add("localhost:27019")
new rs wll elect primary
4.
A) This for fresh instance
-set replica set name in /etc/mongod.conf for all machines/instance who all participating in mongo cluster.
- start mongo service in all instances
-mongo shell
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3}]}
>rs.initiate(cfg)
>rs.add("mongo-secondary:27017"); // by default priority is 1
>rs.addArb("restOnline:27017"); // arbiter can vote but cannt become primary.
B) This in AMI created instances (replica set name is already initiated)
You could do this similar to the follow from the {{mongo}} shell:
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3}]}
rs.add("mongo-secondary:27017"); // by default priority is 1
rs.addArb("restOnline:27017"); // arbiter can vote but cannt become primary.
OR at one shot you can reconfigure/configure replica set
cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3},{"_id" : 1,"host" : "mongo-secondary:27017","priority":2},{"_id" : 2,"host" : "restOnline:27017","arbiterOnly":true}]}
>rs.reconfig(cfg, {force:true})
e.g. to remove rs.remove("mongo-secondary:27017");
Here we don't need to initiate (rs.initiate()) replica set because its already initiated as it does have existing configuration like our current case.
If its fresh replica set and it doesn't have any configuration then need to initiate with config mentioned.
I have kept same existing replica set name in all members of replica set and tried changing/modifying configuration.
Once launched instances (2 count , one for primary & other for secondary) from AMI. mongo shell was not showing neither primary/secondary
for both.rs.status() mentioning that mongo instance has been removed from replicaset.
The reason being dns name have been changed to mongo-primary only (its not having dev_mongo-primary , same for replica machine/instance).
And Mongo replica set configuration mentions primary and secondary hostnames as dev_mongo-primary and dev_mongo-secondary respectively.
(you can check config using command rs.conf())
To solve this , we need to reconfigure replica set configuration with new dns names/ips. This can be only done through primary mongo. sothat
need to bring either of the machine to primary status.
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3}]}
> rs.reconfig(cfg, {force:true})
This would bring mongo instance to primary status. can be checked in mongo shell , it would turn to mongo primary.
Now start mongod in other machine (which is supposed to be secondary/replica machine).
{note : It may show earlier configuration as soon as you start and check conf using rs.conf(). After some time,
it would reflect with latest/changed config in other machine but still other machine is not yet secondary/replica machine.)
Go back to primary machine and add secondary machine host in rs.Using in mongo shell of primary execute following command
>rs.add("mongo-secondary:27017");
It would reflect replica set new config it in other mongo instance as its sharing same replica set name.
and other mongo instance turn into secondary. can be checked in mongo shell and verify rs.conf().
Same execution for adding arbiter member in replicaset
>rs.addArb("restOnline:27017"); // arbiter is running in online machine
5.Adding members to replica set
You can use these procedures
a. to add new members to an existing set.
- Adding new members freshly having empty data directory. make sure added new member become secondary in rs.
mongodb can copy all data to this instance as part of replication process.this process takes times doenst need admin intevention
- Manually copy data directory from an existing member.This new secondary member will catch up to the current state of replica set.
It would shorten the amount of time for this new member to become current.
(Always use filesystem snapshots to create a copy of a member of the existing replica set. Do not use mongodump and mongorestore to seed a new replica set member.)
Use rs.printReplicationInfo() to check the current state of replica set members with regards to the oplog.
Ensure that you can copy the data directory to the new member and begin replication within the window allowed by the oplog i.e. oplog size
shouldn't be so small that by that time you copy to new member , oplog size (in machine from which you copied) shouldnt spill. otherwise
new instance will perform initial sync which resynchronises the data.
oplog maintain idempotency (The quality of an operation to produce the same result given the same input, whether run once or run multiple times)
oplog translate multi-updates into individual operations in order to maintain idempotency.
b. to “re-add” a removed member
(note- If the removed member’s data is still relatively recent, it can recover and catch up easily due to idempotency of oplog)
By this time , you might have come to know to re-add the member . refer a section.
Data Files
Always use filesystem snapshots to create a copy of a member of the existing replica set. Do not use mongodump and mongorestore to seed a new replica set member
6.
when ip of one of the member of replica set is to be changed , then rs.reconfig(cfg) need to command on mongo shell
with new ip and port for member . As part of bringing the new member on board and to sync data set to new member.
it need to initialize by command , rs.initiate().
it wouldnt work if there oplog is not empty. so clear oplog.
>use local
>db.oplog.rs.drop()
Before:
config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.100:15000'}]
}
After:
config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.200:15000'}]
}
This above thing is as good as moving replica set to new location and configure
OR just changing one of the member ip.
In case still its not working , delete all replication and oplog.
>use local
>db.dropDatabase()
>config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.100:15000'}]
}
>rs.initiate(config)
still not, try force option explained in point 2.
7. While removing members from replica set , you can access mongod in secondary/arbiter instances from primary machine or vice versa or any other machine.
you don't really need to visit mongo shell of other instances.
you can access mongo shell of other instances using following command
$mongo --host mongoreplica@core
to shut down mongo server
>db.shutdownServer()
now connect to mongo shell of primary instance.
$mongo --host mongo-primary
>rs.remove("mongo-secondary:27017")
>rs.remove("rsOnline:27017")
OR you can manually visit instance and shutdown mongod service and remove from config in primary shell.
8. Removing a member from replica set
a.Remove a Member Using rs.remove()
b.Remove a Member Using rs.reconfig()
9.
Replace a Replica Set Member
To change the hostname for a replica set member modify the members[n].host field
cfg = rs.conf()
cfg.members[0].host = "mongo2.example.net"
rs.reconfig(cfg)
Any replica set configuration change can trigger the current primary to step down, which forces an election.
During the election, the current shell session and clients connected to this replica set disconnect,
which produces an error even when the operation succeeds.
> cfg = { "_id" : "mongo-cluster", "version":1, "members":[{"_id" : 0,"host" : "52.26.239.117:27017","priority":3}]}
rs.reconfig(cfg, {force:true})
rs.add("52.33.147.186:27017");
rs.add("54.152.103.42:27017");
10.
Change the Size of the Oplog on the Primary And secondary
Secondary Member
1) Recreate the Oplog with a New Size and a Seed Entry
> rs.shutdownServer() And restarting member in standalone mode i.e. restart it on different port without replica set.
> mongod --port 37017 --dbpath /srv/mongodb
> use local
> db = db.getSiblingDB('local')
> db.temp.drop()
> db.temp.save( db.oplog.rs.find( { }, { ts: 1, h: 1 } ).sort( {$natural : -1} ).limit(1).next() )
> db.temp.find()
Remove existing oplog existing collection
> db = db.getSiblingDB('local')
> db.oplog.rs.drop()
Now restart mongod as a member of the replica set on its usual port :
> db.shutdownServer()
> mongod --replSet rs0 --dbpath /srv/mongodb
The replica set member will recover and “catch up” before it is eligible for election to primary.
2) Create backup of existing oplog and create new oplog
> mongodump --db local --collection 'oplog.rs' --port 37017
> db.shutdownServer()
set new oplog size using command or in config file.
and now start mongod as a member of the replica set on its usual port.
(mongod --replSet rs0 --dbpath /srv/mongodb)
Primary Member
To finish the rolling maintenance operation,
- step down the primary with the rs.stepDown() method.and then rs.shutdownServer() And restarting member in standalone mode i.e. restart it on different port without replica set.
- Recreate the oplog with the new size and with an old oplog entry as a seed. (For this follow procedure 1 mentioned for secondary member above)
- Restart the mongod instance as a member of the replica set.
NOTE : Always start rolling replica set maintenance with the secondary members, and finish with the maintenance on primary member.
10.YCSB benchmark tool (https://github.com/brianfrankcooper/YCSB/wiki/Running-a-Workload)
a. Running this tool requires python, java. (for bulding requires maven)
python required modules mentioned in /bin/ycsb script file.
you may need to install 'yum install python-argparse' ( this module is not present in python 2.6.6)
or install using pip(python package index - package manager).
b. Source code for this tool is at (https://github.com/brianfrankcooper/YCSB)
you can use this tool by compiling the source code for mongodb specific code
and run the tool.
OR
You can run this tool directly, the runnable jars are present at
(https://github.com/brianfrankcooper/YCSB/releases/download/0.5.0/ycsb-0.5.0.tar.gz)
c. To benchmark , this tool provides command to load workload i.e. db,tables & documents into mongo db.
Commands for mongodb:
./bin/ycsb load mongodb-async -s -P workloads/workloada > outputLoad.txt
./bin/ycsb run mongodb-async -s -P workloads/workloada > outputRun.txt
./bin/ycsb load mongodb -s -P workloads/workloada > outputLoad.txt
./bin/ycsb run mongodb -s -P workloads/workloada > outputRun.txt
d. To test mongodb using this tool, there are Parameters which are configurable to benchmark
mongodb.url
mongodb.batchsize
mongodb.upsert
mongodb.writeConcern
mongodb.readPreference
mongodb.maxconnections
mongodb.threadsAllowedToBlockForConnectionMultiplier
command example with configured parameter:
./bin/ycsb load mongodb-async -s -P workloads/workloada -p mongodb.url=mongodb://localhost:27017/ycsb?w=0
./bin/ycsb load mongodb -s -P workloads/workloada -p mongodb.url=mongodb://localhost:27017/ycsb?w=0
Hidden member configuration for backup and reporting
>rs.add({_id:1,host: "testone-mongo-arbiter:27017", priority: 0, hidden: true})
MongoDB Tuning Hints
5 Areas influencing throughput
- EC2 instance : network transfer rate (Mbps)
- EBS Optimized : EC2 instance option (On/Off)
- Workload : Block Size, read/write ration, serialization
- Queue Depth : The number of outstanding I/Os
- RAID : Stripe volumes to maximize performance
- EBS Optimized : EC2 instance option (On/Off)
- Workload : Block Size, read/write ration, serialization
- Queue Depth : The number of outstanding I/Os
- RAID : Stripe volumes to maximize performance
Optimal Queue depth to achieve lower latency and highest IOPS is between 4-8;1 QD per 500 IOPS
EBS-Optimized offers consistent latency experience
To increase write performance/throughput
- increase in RAM
- SSD
To reduce write latency
- increase network bandwidth
- Use EBS-Optimized or 10 Gigabit Network Instances
IOPS provisioned IOPS SSD volume attached to an EBS-optimised EC2 instance (OR)
General purpose SSD volume attached to an EBS-optimised EC2 instance.
To consistently achieve IOPS , EC2 instance should be launched as EBS-optimised
General Purpose SSD or Provisioned IOPS SSD volumes that are attached to an EBS-optimized instance or an instance with 10 Gigabit network connectivity
The only way to ensure sustained reliable network bandwidth between your EC2 instance and your EBS volumes is to launch the EC2 instance as EBS-optimized or choose an instance type with 10 Gigabit network connectivity.
Launching an instance that is EBS-optimized provides you with a dedicated connection between your EC2 instance and your EBS volume.
Be sure to choose an EBS-optimized instance that provides more dedicated EBS throughput than your application needs; otherwise, the Amazon EBS to Amazon EC2 connection will become a performance bottleneck.
The dedicated throughput to Amazon EBS, the maximum amount of IOPS the instance can support if you are using a 16 KB I/O size,
Amazon EBS measures each I/O operation per second that is 256 KiB or smaller as one IOPS.
I/O operations that are larger than 256 KiB are counted in 256 KiB capacity units. For example, a single 1,024 KiB I/O operation would count as 4 IOPS; however, 1,024 I/O operations at 1 KiB each would count as 1,024 IOPS
When you create a 3,000 IOPS volume, either a 3,000 IOPS Provisioned IOPS SSD volume or a 1,000 GiB General Purpose SSD volume, and attach it to an EBS-optimized instance that can provide the necessary bandwidth, you can transfer up to 3,000 chunks of data per second (provided that the I/O does not exceed the per volume throughput limit of the volume).
Ephemeral storage is lost when instances are stopped/terminated, so it is generally not recommended unless you understand the data loss implications.
-Due to its concurrency model, the MMAPv1 storage engine does not require many CPU cores .
As such, increasing the number of cores can help but does not provide significant return.
-Increasing the amount of RAM accessible to MongoDB may help reduce the frequency of page faults
-The output from mongostat provides statistics on the number of active reads/writes in the (ar|aw) column.
- MongoDB has good results and a good price-performance ratio with SATA SSD (Solid State Disk).
- Using SSDs or increasing RAM may be more effective in increasing I/O throughput.
- If the NUMA (Non-Uniform Access Memory)configuration may degrade performance, MongoDB prints a warning.
For almost all deployments EBS will be the better choice. For production systems we recommend using
EBS-optimized EC2 instances
Provisioned IOPS (PIOPS) EBS volumes
For best performance we recommend separate volumes for data files, the journal, and the log.
Each has different write behavior, and placing them on separate volumes reduces I/O contention.
Using different storage devices will affect your ability to create snapshot-style backups of your data, since the files will be on different devices and volumes.
MongoDB installed via yum
Individual PIOPS EBS volumes for data (1000 IOPS), journal (250 IOPS), and log (100 IOPS)
Ensure that readahead settings for the block devices that store the database files are appropriate. For random access use patterns, set low readahead values. A readahead of 32 (16 kB) often works well.
For a standard block device, you can run sudo blockdev --report to get the readahead settings and sudo blockdev --setra
Assign swap space for your systems. Allocating swap space can avoid issues with memory contention and can prevent the OOM Killer on Linux systems from killing mongod.
For the MMAPv1 storage engine, the method mongod uses to map files to memory ensures that the operating system will never store MongoDB data in swap space
Most MongoDB deployments should use disks backed by RAID-10
With the MMAPv1 storage engine, the Network File System protocol (NFS) is not recommended as you may see performance problems when both the data files and the journal files are hosted on NFS. You may experience better performance if you place the journal on local or iscsi volumes.
If you decide to use NFS, add the following NFS options to your /etc/fstab file: bg, nolock, and noatime.
With MMAPv1, MongoDB automatically uses all free memory on the machine as its cache. System resource monitors show that MongoDB uses a lot of memory, but its usage is dynamic. If another process suddenly needs half the server’s RAM, MongoDB will yield cached memory to the other process.
Technically, the operating system’s virtual memory subsystem manages MongoDB’s memory. This means that MongoDB will use as much free memory as it can, swapping to disk as needed. Deployments with enough memory to fit the application’s working data set in RAM will achieve the best performance.
When running MongoDB in production on Linux, it is recommended that you use Linux kernel version 2.6.36 or later.
With the MMAPv1 storage engine, MongoDB preallocates its database files before using them and often creates large files. As such, you should use the XFS and EXT4 file systems. If possible, use XFS as it generally performs better with MongoDB.
To explain, there are other considerations here - snapshots for one. Thanks to the journal you no longer have to fsync and lock the database to get a consistent snapshot (be it EBS or LVM or other). However the journal has to be included in the snapshot for that to be the case. Hence whatever node
you use for backing up, if you intend to snapshot without taking downtime for that node, then you need to make sure the journal is included.
MongoDB will only pre-allocate the journal files if it believes that it'll be faster to pre-allocate files of a given size (three files 128 MB each if running with --smallfiles and three files 1 GB each if not running with --smallfiles) than to allocate them on-demand.
General Purpose SSD volumes have a throughput limit between 128 MB/s and 160 MB/s per volume (depending on volume size), which pairs well with a 1,000 Mbps EBS-optimized connection. Instance types that offer more than 1,000 Mbps of throughput to Amazon EBS can use more than one General Purpose SSD volume to take advantage of the available throughput
Provisioned IOPS SSD volumes have a throughput limit range of 256 KiB for each IOPS provisioned, up to a maximum of 320 MiB/s (at 1,280 IOPS)
Launching an instance that is EBS-optimized provides you with a dedicated connection between your EC2 instance and your EBS volume. However, it is still possible to provision EBS volumes that exceed the available bandwidth for certain instance types, especially when multiple volumes are striped in a RAID configuration. The following table shows which instance types are available to be launched as EBS-optimized, the dedicated throughput to Amazon EBS, the maximum amount of IOPS the instance can support if you are using a 16 KB I/O size, and the approximate I/O bandwidth available on that connection in MB/s. Be sure to choose an EBS-optimized instance that provides more dedicated EBS throughput than your application needs; otherwise, the Amazon EBS to Amazon EC2 connection will become a performance bottleneck.
IOPS are input/output operations per second. Amazon EBS measures each I/O operation per second (that is 256 KiB or smaller) as one IOPS. I/O operations that are larger than 256 KiB are counted in 256 KiB capacity units. For example, a single 1,024 KiB I/O operation would count as 4 IOPS; however, 1,024 I/O operations at 1 KiB each would count as 1,024 IOPS.
MongoDB processes operations on data in memory. That is, for a document to be accessed that document must be present in the in-memory cache. Any request for a document that is not currently in memory is first read from disk and loaded into memory. The document is retained in the cache once it has been loaded, and subsequent operations on the document will be much faster as access to disk won't be necessary.
The MMAPv1 storage engine uses memory-mapped files, whereas WiredTiger manages objects through its in-memory cache. When you perform a read or write operation on an object that is not currently in memory, it leads to a page fault (MMAPv1) or cache miss (WiredTiger) so that the object can be read from disk and loaded into memory.
If your application’s working set is much larger than the available memory, access requests to some objects will cause reads from disk before the operation can complete. Such requests are often the largest driver of random I/O, especially for databases that are larger in size than available memory. If your working set exceeds available memory on a single server, you should consider sharding your system across multiple servers.
You should pay attention to the read-ahead settings on your block device to see how much data is read in such situations. Having a large setting for read-ahead is discouraged, because it will cause the system to read more data into memory than is necessary, and it might possibly evict other data that may be used by your application. This is particularly true for services that limit block size, such as Amazon Elastic Block Store (EBS) volumes,
Always use 64-bit builds for production. 32-bit builds support systems with only 2 GB of virtual memory.
Write Operations
At a high level, both storage engines write data to memory, and then periodically synchronize the data to disk.
MMAPv1 implements collection-level concurrency control with atomic in-place updates of document values. To ensure that all modifications to a MongoDB dataset are durably written to disk, MongoDB records all modifications in a journal that it writes to disk more frequently than it writes the data files. By default, data files are flushed to disk every 60 seconds. You can change this interval by using the mongod syncDelay option.
WiredTiger implements document-level concurrency control with support for multiple concurrent writers and native compression. WiredTiger rewrites the document instead of implementing in-place updates. WiredTiger uses a write-ahead transaction log in combination with checkpoints to ensure data persistence. By default, data is flushed to disk every 60 seconds after the last checkpoint, or after 2 GB of data has been written. You can change this interval by using the mongod wiredTigerCheckpointDelaySecs option.
At a high level, both storage engines write data to memory, and then periodically synchronize the data to disk.
MMAPv1 implements collection-level concurrency control with atomic in-place updates of document values. To ensure that all modifications to a MongoDB dataset are durably written to disk, MongoDB records all modifications in a journal that it writes to disk more frequently than it writes the data files. By default, data files are flushed to disk every 60 seconds. You can change this interval by using the mongod syncDelay option.
WiredTiger implements document-level concurrency control with support for multiple concurrent writers and native compression. WiredTiger rewrites the document instead of implementing in-place updates. WiredTiger uses a write-ahead transaction log in combination with checkpoints to ensure data persistence. By default, data is flushed to disk every 60 seconds after the last checkpoint, or after 2 GB of data has been written. You can change this interval by using the mongod wiredTigerCheckpointDelaySecs option.
Several factors can affect the performance of Amazon EBS volumes, such as instance configuration, I/O characteristics, workload demand, and storage configuration. After you learn the basics of working with EBS volumes, it's a good idea to look at the I/O performance you require and at your options for increasing EBS performance to meet those requirements.
The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk.”. [Note: The difference between fsync() and fdatasync() is that the later does not necessarily update the meta-data associated with a file – such as the “last modified” date – but only the file data.]
This is normal and nothing to worry about. MongoDB grabs as much memory as it can to cache as much of your data as possible. However, MongoDB only grabs memory which is not required by other processes. So when another process needs more memory, the memory held by MongoDB will be released.
MongoDB Deployment on AWS
Following steps are
1. Launch instance with EBS volume and security group settings
2. Set outbound security rules
3. configure repository and install mongodb (refer mongodb website documentation)
4. configure mongodb /etc/mongod.conf (refer mongod.conf available)
5. System Tunings at /etc/sysctl.conf and pam security settings in /etc/security/limits.conf and /etc/security/limits.d/90-nproc.conf
6. sudo chkconfig mongod on
7. disable-transparent-hugepages
8. set appropriate hostnames.
9. update /etc/sysctl.conf and pam security settings in /etc/security/limits.conf and /etc/security/limits.d/90-nproc.conf
10. Format EBS and attach to EC2 instance.
11. Readahead settings for EBS volumes/devices
12. Update mongodb config file in /etc/mongod.conf
13. configure replica set in primary.
MongoDB EC2 Instance Mongod Replicaset should be created as ebs optimised.
MongoDB has 1 primary and 2 secondary instances in its replica set.
-----------------------------------------------------------------
System Tuning
-----------------------------------------------------------------
0) Set the hostnames in /etc/hosts and /etc/sysconfig/network
And change hostname using following command
e.g.
$hostname mongomaster2Core ( for primary)
Set hostnames to all 3 instances of mongodb accordingly.
1) ulimit settings
$vi /etc/sysctl.conf
#################################################################################
# The maximum number of concurrently open files
fs.file-max = 999999
# Make more local ports available
net.ipv4.ip_local_port_range = 1024 65535
# Connections tracking
net.nf_conntrack_max = 1248576
# Allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint.generally a safer alternative to tcp_tw_recycle
#net.ipv4.tcp_tw_reuse = 1
# enables fast recycling of TIME_WAIT sockets
#net.ipv4.tcp_tw_recycle = 1
# Increase number of incoming connections backlog queue
# Sets the maximum number of packets, queued on the INPUT
# side, when the interface receives packets faster than
# kernel can process them.
net.core.netdev_max_backlog = 32768
#Maximum number of remembered connection requests, which are still did not receive an acknowledgement from connecting client
net.ipv4.tcp_max_syn_backlog = 10240
# The time that must elapse before TCP/IP can release a closed connection and reuse its resources.
#net.ipv4.tcp_fin_timeout = 10
# Increase number of incoming connections
# somaxconn defines the number of request_sock structures
# allocated per each listen call. The
# queue is persistent through the life of the listen socket.
net.core.somaxconn = 65535
# This will ensure that immediately subsequent connections use the new values
#net.ipv4.route.flush=1
net.ipv4.tcp_window_scaling = 1 #TCP Window Scaling
#This will increase the amount of memory available for socket input/output queues
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem=4096 258000 16777216 # Memory reserved for TCP receive window (reserved memory per connection default)
# Arrays of three parameter values: the 3 numbers represent minimum, initial and maximum memory values for one TCP connection
net.ipv4.tcp_wmem=4096 129000 16777216 # Memory reserved for TCP send window (reserved memory per connection default)
#net.core.rmem_max=8388608 # Maximum receive window size
#net.core.wmem_max=8388608 # Maximum send window size
#net.core.rmem_default=65536 # Default TCP read window size in bytes
#net.core.wmem_default=65536 # Default TCP write window size in bytes
# Don't set tcp_mem itself! Let the kernel scale it based on RAM.
#net.ipv4.tcp_mem=8388608 8388608 8388608
#################################################################################
$vi /etc/security/limits.conf
* soft nofile 999999
* hard nofile 999999
* soft nproc 500000
* hard nproc 500000
$vi /etc/security/limits.d/90-nproc.conf
mongod soft nproc 500000
mongod hard nproc 500000
3) Disable Transparent Huge Pages (THP)
$sudo vi /etc/init.d/disable-transparent-hugepages ( refer disable-transparent-hugepages file)
$sudo chmod 755 /etc/init.d/disable-transparent-hugepages
$sudo chkconfig --add disable-transparent-hugepages
check status of THP support by issuing following commands:
$cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
$cat /sys/kernel/mm/transparent_hugepage/defrag
always madvise [never]
4) Format EBS volumes/devices with ext4 FileSystem.And mount EBS volumes.
create 3 EBS volumes sizing 1TB(3000 IOPS),100GB (300IOPS) and 50GB (150IOPS) other than EBS volume attached to EC2 instance.
sudo mkdir /mongo/data /mongo/log /mongo/journal in EC2 instace to which these are being attached.
echo '/dev/xvdf /mongo/data ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdb /mongo/journal ext4 defaults,auto,noatime,noexec 0 0
/dev/xvdc /mongo/log ext4 defaults,auto,noatime,noexec 0 0' | sudo tee -a /etc/fstab
sudo mkfs.ext4 /dev/xvdf
sudo mkfs.ext4 /dev/xvdb
sudo mkfs.ext4 /dev/xvdc
vi /etc/fstab
mount /mongo/data
mount /mongo/journal
mount /mongo/log
cd /mongo/data
sudo ln -s /mongo/journal /mongo/data/journal
sudo chown -h mongod:mongod journal
lsblk
Note: Update respective path in mongod.conf for data and log folders.
5)
For random access use patterns, set low readahead values. A readahead of 32 (16 kB) often works well.
sudo blockdev --report
sudo blockdev --setra 32 /dev/xvda
echo 'ACTION=="add", KERNEL=="xvda", ATTR{bdi/read_ahead_kb}="16"' | sudo tee -a /etc/udev/rules.d/85-ebs.rules
-----------------------------------------------------------------
Installation
-----------------------------------------------------------------
1) Installation of MongoDB
Create a /etc/yum.repos.d/mongodb-org-3.0.repo file so that you can install MongoDB directly, using yum.
[mongodb-org-3.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.0/x86_64/
gpgcheck=0
enabled=1
sudo yum install -y mongodb-org-3.0.5 mongodb-org-server-3.0.5 mongodb-org-shell-3.0.5 mongodb-org-mongos-3.0.5 mongodb-org-tools-3.0.5
To exclude unintended upgrades , exclude directive in /etc/yum.conf
exclude=mongodb-org,mongodb-org-server,mongodb-org-shell,mongodb-org-mongos,mongodb-org-tools
-----------------------------------------------------------------
Configuration
-----------------------------------------------------------------
1) mongod configuration
/etc/mongod.conf
refer mongod.conf
2) a) To start mongod service
sudo service mongod start
b) To stop mongod service
sudo service mongod stop
c) To restart mongod service
sudo service mongod restart
3) You can optionally ensure that MongoDB will start following a system reboot by issuing the following command:
sudo chkconfig mongod on
4) Creating MongoDB ReplicaSet
mongo shell : mongo
$mongo
>cfg = { "_id" : "mongo-ycsb-test", "version":1, "members":[{"_id" : 0,"host" : "mongo-primary:27017","priority":3}]}
>rs.initiate(cfg)
>rs.add("mongo-replica:27017");
>rs.addArb("mongo-arbiter:27017");
OR
>cfg = { "_id" : "mongo-ycsb-test", "version":1, "members":[{"_id" : 0,"host" : "mongodb-primary:27017","priority":3}]}
>rs.initiate(cfg)
>rs.add("mongodb-replica:27017");
>rs.add({host: "mongodb-backup:27017", priority: 0, hidden: true})
-----------------------------------------------------------------
FAQ
-----------------------------------------------------------------
1) How to verify MongoDB replicaSet is up and running.?
Highest priority instance becomes Primary and other instances becomes secondary instance in rs(replicaset).
mongo shell command
$mongo
>db.isMaster();
This above command would outputs if current machine is if primary or not.
2)How to check replicaset configuration and its status?
mongo shell command
$mongo
>rs.conf();
>rs.status();
3) To Uninstall MongoDB packages
$sudo service mongod stop
$sudo yum erase $(rpm -qa | grep mongodb-org)
$sudo rm -r /var/log/mongodb
$sudo rm -r /var/lib/mongo
4) To Remove softlink
$rm -d symlink
5) lsblk
unmounting EBS volumes commands
$sudo umount /dev/xvdf
$sudo umount /dev/xvdb
$sudo umount /dev/xvdc
6) How to change ownership of any folder/file to mongod - user and group?
$sudo chown mongod:mongod mongo
Backup
1 Get the EBS ID of the disk you want to back up (on which mongo data resides) from the instance description (block storage).
2 Go to the tab volumes under elastic block storage (from the list on left corner)
3 Right click on the volume you want to back up and choose create snap shot.
4 A snapshot of the current disk state should appear under snapshots
Recovery
1 Launch an EBS volume from the snapshot (create volume option appears on right click on the corresponding snapshot)
2 Attach the new EBS volume to the instance you want (again right click on the volume under volume list)
3 SSH to the instance
4 Run the command lsblk
5 All your ebs volumes will be listed. The latest mongo back up volume you attached wont be mounted yet (MOUNTPOINT field is empty). The volume has to be mounted
6 The volume already has a file system, since its from a backup (so no need to create a file system)
7 Create a mount point if you do not have one already (create a folder using mkdir where you want to mount the disk) Eg: mkdir /datamongo
8 To mount the device run the command
mount device_name mount_point
where device name can be obtained from step 4 (under recovery)
eg : mount /dev/sda1 /datamongo
9 Location inside the disk can be accessed from the mount point (Eg /datamongo )
10 Open /etc/mongod.conf and edit db path to new the db path on the newly mounted disk
Eg : if db files are at /var/lib/mongo on the new disk
give path /datamongo/var/lib/mongo (where /datamongo is the mount point)
Tailable Cursor
Refer mongo docs for tailable cursor and capped collection
In the Commit to solr code,
1 First we check if the tailable cursor is null.
2 If it is null, we create a new capped collection , we insert a dummy event and obtain its tailable cursor.
3 Now the tailable cursor waits for entries to be made .
4 Say for committing a list to solr, we create an entry in the capped collection with the list id.
5 The waiting cursor will pick it up , obtain the correspondng calendar from DB and then commit it to solr (CommitToSolr method creates the solr document)
No comments:
Post a Comment