Week 6
Object Data Modeling with Mongoose
Quiz 5: Databases 15 mins
There will be a quiz today. It will be worth 2% of your final grade.
# Agenda
- AMA (10 min)
- Quiz (10 min)
- Mongoose Models (20 mins)
- Mongoose Query Methods (10 mins)
- Break (10 min)
- EX6-1 Mongo Cars (50 min)
- Break (10 min)
- EX6-2 Car Owner (50 mins)
# Mongoose Models
Mongoose is a Node.js based Object Data Modelling (ODM) library for MongoDB. It is used to abstract away the lower level MongoDB access APIs and manage the shape of application resource data structures. This is the M in the MVC software architecture pattern.
Mongoose provides two core features to simplify the implementation of a MongoDB backed application.
The
mongoose.model()
method is a class factory -- meaning it is a function that returns a new class object which we can then use to instantiate individual document objects. This factory function takes two arguments: the name of the resource that this model represents, and a schema object describing the properties of that resource.The
mongoose.Schema
class is used to define the expected shape of resource properties -- their names, types, and validation limits.
# Schema
A model schema could be a simple object specifying the property names and types.
const catSchema = new mongoose.Schema({
name: String,
age: Number
})
Or it can be expressed with more detailed options.
const catSchema = new mongoose.Schema({
name: {type: String, minlength: 3, maxlength: 50, required: true},
age: {type: Number, min: 0, max: 25}
})
# Schema Types
The following Mongoose Schema Types (opens new window) are permitted:
- Array
- Boolean
- Buffer
- Date
- Mixed (A generic / flexible data type)
- Number
- ObjectId
- String
Mixed and ObjectId are defined under mongoose.Schema.Types
. We will look at this a little later when we create an owner relationship for the cars API.
# Making Models
Start by using the schema object with the model()
factory method to generate a new class.
const Cat = mongoose.model('Cat', catSchema)
... and then we can use that Cat model to instantiate a new Mongoose document instance.
const kitty = new Cat({name: 'Spot', age: 2})
The mongoose.model
factory function generates new class objects, customized with the given schema, that inherit all of the methods required to interact with the MongoDB service. This includes a list of static class methods (opens new window) as well as instance methods (opens new window). e.g.
// example class method
Cat.find()
// example instance method
kitty.save()
# Relationships
Most applications have more complex data structures than our simple Cat model. Often this involves a relationship to another model. For example we might want to extend the Cat model to include an owner. In traditional SQL database systems, this means creating a new table to store the details about the Person that is the owner of the Cat, and then adding a new property like owner_id
to the Cat schema to store the foreign key pointer to people.id
.
Document databases like MongoDB give us two options (opens new window): embedded documents, and document references.
# Document References
Document references are similar to an SQL foreign key. It stores the _id
value of a document in another collection. Unlike SQL databases this relationship is not enforced at the database level -- there is no validation that the reference id exists. We would add it to the schema like this, where the ref
option is the name of another model class in our application.
const catSchema = new mongoose.Schema({
name: {type: String, minlength: 3, maxlength: 50, required: true},
age: {type: Number, min: 0, max: 25},
owner: {type: mongoose.Schema.Types.ObjectId, ref: 'Person'}
})
Another difference from SQL is that the reference property could be an array.
const catSchema = new mongoose.Schema({
name: {type: String, minlength: 3, maxlength: 50, required: true},
age: {type: Number, min: 0, max: 25},
owners: [{type: mongoose.Schema.Types.ObjectId, ref: 'Person'}]
})
# Example usage
The owners
property stores an array of one or more ObjectId (opens new window) references to the unique _id
of a document in the Person
collection.
const Cat = mongoose.model('Cat', catSchema)
const kitty = new Cat({
name: 'Fluffy',
age: 4,
owners: ['6021b7be9d383359e30a1327', '6021b94e6a3a0b5a03fcd7ba']
})
# Embedded Documents
The other option is to embed the related documents as properties of the main document.
const personSchema = new mongoose.Schema({
firstName: String,
lastName: String,
phoneNumber: Number
})
const catSchema = new mongoose.Schema({
name: {type: String, minlength: 3, maxlength: 50, required: true},
age: {type: Number, min: 0, max: 25},
owners: [personSchema]
})
const Cat = mongoose.model('Cat', catSchema)
const kitty = new Cat({
name: 'Fluffy',
age: 4,
owners: [
{firstName: 'Mickey', lastName: 'Mouse', phoneNumber: 14155551212},
{firstName: 'Minnie', lastName: 'Mouse', phoneNumber: 14155551212}
]
})
Trade-offs
The relationship modelling method you choose is ultimately a trade-off between read query speed and data consistency.
The referenced document method ensures that data remains consistent because it is stored in only one place for each schema model, but it requires an additional database query for each related document when retrieving the primary document. For queries returning a single document (most CRUD operations) this may be fine. However for complex search queries returning thousands of results, there could be a significant performance hit.
The embedded document method avoids these additional database queries when retrieving data, but exponentially increases the complexity of maintaining data consistency because you are potentially storing the same data in many places and need to update them all manually.
If we look at the Cat example above, and consider what happens when we want to change Mickey Mouse's phone number. In the reference method, we simply find the Person document for Mickey Mouse and update the phoneNumber property. Then whenever we need to contact Fluffy's owner, the data will be correct.
However if we use the embedded method, we have to lookup all of the Cats that have an owner with the name Mickey Mouse and then update the embedded phone number. But what if there is more than one Person named Mickey Mouse? How do we know if we are updating the correct information? It can get messy if you don't plan it out well.
# Mongoose Query Methods
Mongoose models provide several static helper functions for CRUD operations. Each of these functions returns a mongoose Query object (opens new window) ... A query also has a
.then()
function, and thus can be used as a promise.
Review the mongoose documentation (opens new window) for the full list, but these are the most common ones that we will be using.
Model.find() (opens new window) is used to get an array of documents matching the given filter criteria.
const catsNamedSpot = Cat.find({name: 'Spot'})
These methods act on a single document matching the given id parameter.
- Model.findById()
- Model.findByIdAndRemove()
- Model.findByIdAndUpdate()
These methods act on the first document found matching the given filter criteria.
- Model.findOne()
- Model.findOneAndRemove()
- Model.findOneAndUpdate()
# Mongo Query Operators
In the filter criteria objects we can use any of the standard MongoDB query operators (opens new window). The syntax is to provide an object as the filter condition value for the given property. That object will have one of the Mongo query operators as it's key and the corresponding relative value. Let's look at some examples.
const catsAgedFourOrMore = Cat.find({age: {$gte: 4}})
const catsAgedLessThanFour = Cat.find({age: {$lt: 4}})
const catsAgedTwoFourOrSix = Cat.find({age: {$in: [2, 4, 6]}})
const catsNotAgedTwoFourOrSix = Cat.find({age: {$nin: [2, 4, 6]}})
# Mongoose Query Helpers
Mongoose provides some helper functions (opens new window) to make writing your queries more fluent. The syntax is sometimes more readable, but the end result is exactly the same. So, use whichever method is easier for you.
const catsAgedFourOrMore = Cat.find().where('age').gte(4)
const catsAgedLessThanFour = Cat.find().where('age').lt(4)
const catsAgedTwoFourOrSix = Cat.find().where('age').in([2, 4, 6])
const catsNotAgedTwoFourOrSix = Cat.find().where('age').nin([2, 4, 6])
# EX6-1 Mongo Cars
We are going to recreate our Cars API with Mongoose and MongoDB. Let start by creating a new project folder. I called mine week6
.
mkdir week6
cd week6
echo "'use strict'" > app.js
echo ".DS_Store\nnode_modules/" > .gitignore
git init
npm init --yes
npm install express morgan
git add .
git commit -m "Initialize project folder"
# Connect to MongoDB
Before we can use Mongoose to connect to our MongoDB, we need to add it as an NPM dependency for our project.
npm install mongoose
We only need to create the connection to MongoDB once in our application. You should do it in app.js
, right at the top, before you initialize Express. If it cannot connect to the database, then we cannot proceed. So, use the process.exit([code]) (opens new window) function to terminate the node.js runtime.
Just like last week, we require mongoose and then give it the connection string. This time the database path will be /mad9124
.
'use strict'
const mongoose = require('mongoose')
mongoose
.connect('mongodb://localhost:27017/mad9124', {
useNewUrlParser: true,
useCreateIndex: true,
useFindAndModify: false,
useUnifiedTopology: true
})
.then(() => console.log('Connected to MongoDB ...'))
.catch(err => {
console.error('Problem connecting to MongoDB ...', err.message)
process.exit(1)
})
Let's add the rest of our app.js
boilerplate code.
const morgan = require('morgan')
const express = require('express')
const app = express()
app.use(morgan('tiny'))
app.use(express.json())
const port = process.env.PORT || 3030
app.listen(port, () => console.log(`HTTP server listening on port ${port} ...`))
TIP
Notice that we are using the morgan (opens new window) HTTP request logging middleware that we discussed in week 4. This will help with debugging, by showing the requests received by the server and their response codes.
Make sure that MongoDB database service is running. Go back to the folder where you created the docker-compose.yml
file last week and run docker-compose ps
. If it does not show the service is running, start it up with docker-compose up -d
.
# nodemon
Up until now, each time we have made changes to the node/express application, we have to kill the running instance with CTLC and restart it with node app.js
. Wouldn't it be nice if that happened automatically every time you save changes to the source code?
nodemon is a tool that helps develop node.js based applications by automatically restarting the node application when file changes in the directory are detected.
nodemon does not require any additional changes to your code or method of development. nodemon is a replacement wrapper for node. To use nodemon, replace the word node on the command line when executing your script.
# Install nodemon
Add nodemon as a development dependency for your project.
npm install nodemon --save-dev
Then you can start your app using nodemon instead of the node command like this.
npx nodemon app.js
What is npx?
npx (opens new window) is a CLI tool installed along with npm
and allows you to run packages that are installed as local project dependencies (rather than being installed globally). It will also let you run other CLI tools directly from the NPM registry - without explicitly installing them.
Check out this short article: npm vs npx — What’s the Difference? (opens new window)
If you started the app with nodemon
, you should now see a successful connection message in the terminal.

# Add a script to package.json
Since this is something you will type over and over again, you might want to make a short-cut script in package.json
. Add a start
key in the scripts
object like this ...
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "nodemon app.js"
},
Now you can just type npm start
to fire up you server with automatic reloading.
# Make a git commit
git add -A
git commit -m "Add Mongoose db connection to Express app"
# Car Model
Before you can do any CRUD operations you need to define a Mongoose.Schema and create a Car
Model object. Create a new models
folder at the top level of your project and in it create a new file called Car.js
-- note the capital 'C' as a reminder that this module will be exporting a class.
The basic template for a Mongoose Model module is ...
const mongoose = require('mongoose')
const schema = new mongoose.Schema({})
const Model = mongoose.model('Car', schema)
module.exports = Model
Let's expand the schema definition object on line three. We want to define three properties for the Car documents: make, model and colour. All will be of type String
.
Unique Identifier
An _id property with a unique value will be automatically assigned to all new documents by Mongoose. You should not include the _id
property in your scheme definition.
Review the full list of Mongoose Schema Types (opens new window) in the official documentation.
const schema = new mongoose.Schema({
make: String,
model: String,
colour: String
})
# Make a git commit
git add -A
git commit -m "Add the Mongoose model for Car"
# CRUD with Mongoose
OK, you have a working connection from the Express app to the MongoDB server in your local Docker container. Now let's set-up the API routes to talk to the database instead of an in memory array.
Create a routes
folder and in that folder create a new file called cars.js
and stub out the six route handlers.
const express = require('express')
const router = express.Router()
router.get('/', async (req, res) => {})
router.post('/', async (req, res) => {})
router.get('/:id', async (req, res) => {})
router.patch('/:id', async (req, res) => {})
router.put('/:id', async (req, res) => {})
router.delete('/:id', async (req, res) => {})
module.exports = router
TIP
Since the only time we are using express
in this module is to create a router instance. You can directly request the Router()
method and immediately invoke it. So this code ...
const express = require('express')
const router = express.Router()
Can be simplified to this code ...
const router = require('express').Router()
But you can certainly write it out the long way if that makes it easier to read for you.
Now let's register the cars router in the main app.js
module after the express.json()
middleware.
app.use(express.json())
app.use('/api/cars', require('./routes/cars'))
TIP
One neat trick to simplify registering router modules with your Express app is to write the require()
statement in-line with the app.use()
method call. I find that this makes it easier to trace the application logic.
# Make a git commit
git add -A
git commit -m "Add cars router module"
# GET /api/cars
Use the find()
method to query the Car model for a list all of the cars in the collection. Before you can do that, you need to import the Car
model class.
Put this right at the top of /routes/cars.js
const Car = require('../models/Car')
And then update our route handler. The Mongoose find()
method returns a Promise so you can use the async/await syntax ...
router.get('/', async (req, res) => {
const cars = await Car.find()
res.send({data: cars})
})
TIP
You could also use traditional Promise syntax.
router.get('/', (req, res) => {
Car.find().then(cars => {
res.send({data: cars})
})
})
# Test it with Postman
OK. Let's test that in Postman. Send a GET request to the localhost:3030/api/cars
URL. You should see a status code 200 response with an empty array for the data payload – you haven't added any Car objects to the database yet 😉
{
"data": []
}
# Make a git commit
If everything is working as expected, make a commit.
git add -A
git commit -m "Add the GET all cars route logic"
That was easy! Let's do the POST route now so that you can add some cars to the database.
# POST /api/cars
Use the req.body
as the input data to create a new instance of the Car model class – this will be a new Mongoose document. Then call the .save()
method on that instance to persist it to the database.
# Never trust data from the client!
Before creating the new document, you need to ensure that the client did not try to set the _id
value. You can use the JavaScript delete (opens new window) operator to remove the _id
property from the req.body
, if it exists.
TIP
Mongoose models will automatically discard any properties that are not defined in the Model.schema, so our route handler does not need to worry about extracting only permitted properties from the req.body
as we had done in week 4.
router.post('/', async (req, res) => {
let attributes = req.body
delete attributes._id
let newCar = new Car(attributes)
await newCar.save()
res.status(201).send({data: newCar})
})
Notice the `await` keyword
Any action that communicates with the database is an asynchronous action and your code needs to handle those Promises.
# Test it with Postman
Use the sample data from last week's cars.json
file to use with Postman. Remember that you need to add the Car object as the JSON body for the POST request. Add each of the four cars, one at a time.
# Make a git commit
If everything is working as expected, make a commit.
git add -A
git commit -m "Add the POST route logic"
# GET /api/cars/:id
First let's do the happy path ... no error handling. You can use the .findById()
static method on the Car model class. The id
parameter from the request URI is accessible in the route handler on the req.params
object.
router.get('/:id', async (req, res) => {
const car = await Car.findById(req.params.id)
res.send({data: car})
})
OK, now what if the id
does not exist as a unique identifier in our database? You should wrap the code in a try/catch
block and then send an appropriate error response back to the client.
router.get('/:id', async (req, res) => {
try {
const car = await Car.findById(req.params.id)
if (!car) {
throw new Error('Resource not found')
}
res.send({data: car})
} catch (err) {
res.status(404).send({
errors: [
{
status: '404',
title: 'Resource does not exist',
description: `We could not find a car with id: ${req.params.id}`
}
]
})
}
})
As we have seen before, you will need this same error handler in several other routes. So, extract this to a helper function. At the bottom of the /routes/cars.js
module, but just above the module.exports
line, add this private function.
function sendResourceNotFound(req, res) {
res.status(404).send({
errors: [
{
status: '404',
title: 'Resource does not exist',
description: `We could not find a car with id: ${req.params.id}`
}
]
})
}
module.exports = router
... and then update your route handler's catch block to call this function.
router.get('/:id', async (req, res) => {
try {
const car = await Car.findById(req.params.id)
if (!car) throw new Error('Resource not found')
res.send({data: car})
} catch (err) {
sendResourceNotFound(req, res)
}
})
Better error handling
This is better but still incomplete. We will look at more robust error handling solutions in a couple of weeks.
# Test it with Postman
Do a GET request to the localhost:3030/api/cars
route. Then copy the _id
value for one of the cars. Now do another GET request appending that value to the end of the URL.
e.g. localhost:3030/api/cars/5ff776067b6bf504b6db320c
# Make a git commit
If everything is working as expected, make a commit.
git add -A
git commit -m "Add the GET by id route logic"
# PATCH /api/cars/:id
For the update actions, we can use the findByIdAndUpdate()
static method on the Car model class. Similar to the findById()
method, it takes a document _id
value as the first argument. The second argument is an object with the properties to be stored. The third argument is an options object.
# Second argument - update payload
The _id
property for the update operation is an immutable (read-only) field. To ensure that it has not been modified (accidentally or maliciously), use object destructuring to separate and discard any _id
property that may be included in the req.body
. Then reconstruct the update payload object, explicitly setting _id: req.params.id
and then use the rest operator to extract the remaining attributes.
# Third argument - options object
The documentation lists many options that can be set to modify the behaviour of the update method. We are interested in two.
Setting
new
totrue
will return the updated record from the database. The default behaviour returns the document in it's pre-update state.Setting
runValidators
totrue
ensures that your Model.schema rules are checked before the updates are applied.
router.patch('/:id', async (req, res) => {
try {
const {_id, ...otherAttributes} = req.body
const car = await Car.findByIdAndUpdate(
req.params.id,
{_id: req.params.id, ...otherAttributes},
{
new: true,
runValidators: true
}
)
if (!car) throw new Error('Resource not found')
res.send({data: car})
} catch (err) {
sendResourceNotFound(req, res)
}
})
# Test it with Postman
Do a PATCH request using the same URI as the previous test. This time include a JSON object in the request body. It should change one or more of the Car object properties. e.g. set a new colour.
{
"colour": "Silver Mist"
}
Try adding an invalid _id
property to the update payload (request body).
{
"_id": "5ff776067b6bf504b6db320f",
"colour": "Silver Mist"
}
# Make a git commit
If everything is working as expected, make a commit.
git add -A
git commit -m "Add the PATCH route logic"
# PUT /api/cars/:id
By calling the PUT method, the client is asking entirely to replace the current document with the one supplied in the req.body
. The route handler is identical to the PATH method with one additional property in the options object.
Set the overwrite
option to true
.
This will update any supplied document properties the same as with the PATCH method, but any document properties that are omitted from the req.body
will be removed from the database.
router.put('/:id', async (req, res) => {
try {
const {_id, ...otherAttributes} = req.body
const car = await Car.findByIdAndUpdate(
req.params.id,
{_id: req.params.id, ...otherAttributes},
{
new: true,
overwrite: true,
runValidators: true
}
)
if (!car) throw new Error('Resource not found')
res.send({data: car})
} catch (err) {
sendResourceNotFound(req, res)
}
})
# Test it with Postman
# Make a git commit
If everything is working as expected, make a commit.
git add -A
git commit -m "Add the PUT route logic"
# DELETE /api/cars/:id
This time we want the findByIdAndRemove()
static method on the Car model class.
router.delete('/:id', async (req, res) => {
try {
const car = await Car.findByIdAndRemove(req.params.id)
if (!car) throw new Error('Resource not found')
res.send({data: car})
} catch (err) {
sendResourceNotFound(req, res)
}
})
# Test it with Postman
# Make a git commit
If everything is working as expected, make a commit.
git add -A
git commit -m "Add the DELETE route logic"
That's it! We have recreated the cars API from week 4 using MongoDB and Mongoose.
# Take a short break!
# EX6-2 Car Owner
In real life, cars have owners. Let's extend the API to include a /api/people
resource path, and link the Person and Car models to each other by using the reference method.
# Person Model
Create a new file called Person.js
in the /models
folder.
const mongoose = require('mongoose')
const schema = new mongoose.Schema({
name: {
first: String,
last: String
},
email: String,
birthDate: Date,
phone: Number,
address: {
streetNumber: String,
streetName: String,
city: String,
region: String,
country: String,
postalCode: String
}
})
const Model = mongoose.model('Person', schema)
module.exports = Model
# Update Car Model.schema
Add the owner
property with the type
being a mongoose.Schema.Types.ObjectId (opens new window) reference to the Person
model.
const schema = new mongoose.Schema({
make: String,
model: String,
colour: String,
owner: {type: mongoose.Schema.Types.ObjectId, ref: 'Person'}
})
Also, right at the top we need to require the Person model so that Mongoose knows about it. This model would normally be loaded into our application in the people router module, but we haven't built that yet.
const Person = require('./Person')
# Populate owner
In the Car route handler for, you may include the related owner document as an embedded object with the the Car object. This saves the client from needing to make a second request.
To expand the referenced _id
value for the owner
property, we need to use the Mongoose .populate() (opens new window) method.
Populated paths are no longer set to their original
_id
, their value is replaced with the mongoose document returned from the database by performing a separate query before returning the results.
In the /routes/cars.js
file, update the GET /:id route handler to chain the .populate('owner')
method onto the findById()
method call.
router.get('/:id', async (req, res) => {
try {
const car = await Car.findById(req.params.id).populate('owner')
if (!car) throw new Error('Resource not found')
res.send({data: car})
} catch (err) {
sendResourceNotFound(req, res)
}
})
Let's test it!
Before we can test this out, there needs to be people in the database, and we need to either add a new car with an owner property or update one.
# Manually add people collection
Using MongoDB Compass you can manually create the people
collection and add a document.
- Select the
mad9124
database in the left sidebar. - Click on the
Create Collection
button. - Type the collection name as
people
-- all lowercase. - Click the green
Create Collection
button.
# Manually add a Person document
- Click on the newly created
people
collection in the collection list. - Click on the green
Add Data
drop-down button, and selectInsert Document
. - Add the properties and values for your Person document.
Make sure it matched the schema in the Person model. - Click the green
Insert
button.
# Update a Car document
Now that you have a Person in the database you can link that document as an owner reference in one of the Car documents. Use Postman to do a PATCH update on one of your Car documents with the _id
value from your new Person document as the value for the owner
property of the car.
# Test the GET /api/cars/:id route
Now that there is a Car document with an owner
property linked to a Person document, you can finally test the .populate()
instruction that you added to the GET /api/cars/:id route handler.
You should see the owner
property has been expanded to include the entire related Person document – not just the Person._id
string.
# People Router
Now that you have seen how to implement the cars router module. Use that as a template to implement the people router module on your own.
Create a new /routes/people.js
file to implement a router module for the /api/people
resource path. Be sure to require the Person model in your router module. Then register the router in the main app.js
file.
Test all of the routes with Postman.
Make a git commit after completing each route handler.
Bonus
Add a cars
property to the Person model that holds an array of related Car references. Update the GET /api/people/:id
route to populate the person's cars.
# Submit GitHub Repo
Create a new private repo on GitHub named mad9124-w21-ex6-mongo-cars.
Make sure that you have initialized your local project folder with git init
and created a .gitignore
file to exclude the node_modules
folder and the .DS_Store
files from your git archive.
Create a final commit to include all of your work from today's exercises.
Link the local repo to the GitHub repo and sync them up.
Remember to add rlmckenney
as a collaborator on your GitHub repo so that I can see your code, and submit the GitHub repo's URL on Brightspace.
# For next class
Before next class, please read these additional online resources.
- Introduction to Mongoose for MongoDB (opens new window)
- Mongoose Guide: Queries (opens new window)
- MongoDB query operators (opens new window)
- Mongoose query helper functions (opens new window)
- Mongoose Schema Types (opens new window)
- process.exit([code]) (opens new window)
Start Assignment 2 - Mongo CRUD
You have enough information now to get started on Assignment 2. We will cover the remaining 15% of what you need to know next week.
This assignment is due before the start of our Week 8 class after the break. Don't wait until to the last minute!