Extracting data from MongoDB using Python and pyMongo

Use the Python Package Installer (pip3) to install the pyMongo package:

% pip3 install pyMongo
Collecting pyMongo
  Downloading https://files.pythonhosted.org/packages/5e/25/372cc21c6ad318356fa819ca9b5bf021b747cd5a00b868c1db1e3e8fe228/pymongo-3.10.1-cp38-cp38-macosx_10_9_x86_64.whl (351kB)
     |████████████████████████████████| 358kB 1.1MB/s 
Installing collected packages: pyMongo
Successfully installed pyMongo-3.10.1

This simple example show how to connect to MongoDB from Python 3 using pyMongo. It reads all documents from the development db in the transactions collection and writes them to a CSV file named transactions.csv.

from pymongo import MongoClient
import csv
from decimal import Decimal

TWO_PLACES = Decimal(10) ** -2

client = MongoClient()

db = client['development']

collection = db['transactions']

cursor = collection.find()

with open('transactions.csv', 'w', newline='') as file:
    writer = csv.writer(file, quoting=csv.QUOTE_NONNUMERIC)
    for doc in cursor:
        account_id = doc['accountId']
        desc = doc['description']
        amount = Decimal(doc['amount']).quantize(TWO_PLACES)
        currency = (doc['currency'])
        date_created = doc['dateCreated']
        writer.writerow([account_id, desc, amount, currency, date_created])

The output in transactions.csv will be as follows: –

"A00000001","Bowers & Wilkins P5 Headphones",299.00,"GBP","2020-02-09 14:30:00"
"A00000001","Apple Homepod",329.00,"GBP","2020-02-09 15:30:00"
"A00000002","AAA Batteries",12.00,"GBP","2020-02-09 15:35:00"

Creating and populating a new MongoDB

Open a new terminal session and start the MongoDB CLI.

mongo

At the command prompt > enter the following command to create a new database named development.

> show databases
admin   0.000GB
config  0.000GB
local   0.000GB
> use development
switched to db transactions
> show databases
admin        0.000GB
config       0.000GB
development  0.000GB
local        0.000GB

If the database does not already exist it will be created for you.

Now that we have a database created we need to create a collection and insert some documents into it so we can use for later testing/development.

> db.createCollection("transactions")
> show collections
transactions
> db.transactions.insert({
    accountId: 'A00000001',
    description: 'Bowers & Wilkins P5 Headphones',
    amount: 299.00,
    currency: 'GBP',
    dateCreated: new Date(2020,1,9,14,30)
})

We can now query the document we have inserted into our new collection:

> db.transactions.find().pretty()
{
	"_id" : ObjectId("5e6693f2313ea58d23458752"),
	"accountId" : "A00000001",
	"description" : "Bowers & Wilkins P5 Headphones",
	"amount" : 299,
	"currency" : "GBP",
	"dateCreated" : ISODate("2020-02-09T14:30:00Z")
}

Changing the Log and Storage Directories for MongoDB on macOS

Open a terminal session and open /usr/local/etc/mongod.conf

systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/mongo.log
  logAppend: true
storage:
  dbPath: /usr/local/var/mongodb
net:
  bindIp: 127.0.0.1

Make the required changes to path and dbPath and start/restart your service.

systemLog:
  destination: file
  path: /Users/mark/devops/mongo-logs/mongo.log
  logAppend: true
storage:
  dbPath: /Users/mark/devops/mongo-data
net:
  bindIp: 127.0.0.1

Installing MongoDB on macOS

To install MongoDB on macOS (if you have Homebrew installed (https://brew.sh/):

brew tap mongodb/brew

This will clone the github repo. Once the command finishes you can issue the command to install mongodb-community. The following will install the latest version of the community edition.

brew install mongodb-community

In order to start the mongo service and restart on login run the following: –

brew services start mongodb/brew/mongodb-community

If you don’t want to run as a background service then run:

mongod --config /usr/local/etc/mongod.conf