Week 6
Object Data Modeling with Mongoose

Quiz 5: Databases 15 mins

There will be a quiz today. It will be worth 2% of your final grade.

# Agenda

  • AMA (10 min)
  • Quiz (10 min)
  • Mongoose Models (20 mins)
  • Mongoose Query Methods (10 mins)
  • Break (10 min)
  • EX6-1 Mongo Cars (50 min)
  • Break (10 min)
  • EX6-2 Car Owner (50 mins)

# Mongoose Models

Mongoose is a Node.js based Object Data Modelling (ODM) library for MongoDB. It is used to abstract away the lower level MongoDB access APIs and manage the shape of application resource data structures. This is the M in the MVC software architecture pattern.

Mongoose provides two core features to simplify the implementation of a MongoDB backed application.

  1. The mongoose.model() method is a class factory -- meaning it is a function that returns a new class object which we can then use to instantiate individual document objects. This factory function takes two arguments: the name of the resource that this model represents, and a schema object describing the properties of that resource.

  2. The mongoose.Schema class is used to define the expected shape of resource properties -- their names, types, and validation limits.

# Schema

A model schema could be a simple object specifying the property names and types.

const catSchema = new mongoose.Schema({
  name: String,
  age: Number
})

Or it can be expressed with more detailed options.

const catSchema = new mongoose.Schema({
  name: {type: String, minlength: 3, maxlength: 50, required: true},
  age: {type: Number, min: 0, max: 25}
})

# Schema Types

The following Mongoose Schema Types (opens new window) are permitted:

  • Array
  • Boolean
  • Buffer
  • Date
  • Mixed (A generic / flexible data type)
  • Number
  • ObjectId
  • String

Mixed and ObjectId are defined under mongoose.Schema.Types. We will look at this a little later when we create an owner relationship for the cars API.

# Making Models

Start by using the schema object with the model() factory method to generate a new class.

const Cat = mongoose.model('Cat', catSchema)

... and then we can use that Cat model to instantiate a new Mongoose document instance.

const kitty = new Cat({name: 'Spot', age: 2})

The mongoose.model factory function generates new class objects, customized with the given schema, that inherit all of the methods required to interact with the MongoDB service. This includes a list of static class methods (opens new window) as well as instance methods (opens new window). e.g.

// example class method
Cat.find()

// example instance method
kitty.save()

# Relationships

Most applications have more complex data structures than our simple Cat model. Often this involves a relationship to another model. For example we might want to extend the Cat model to include an owner. In traditional SQL database systems, this means creating a new table to store the details about the Person that is the owner of the Cat, and then adding a new property like owner_id to the Cat schema to store the foreign key pointer to people.id.

Document databases like MongoDB give us two options (opens new window): embedded documents, and document references.

# Document References

Document references are similar to an SQL foreign key. It stores the _id value of a document in another collection. Unlike SQL databases this relationship is not enforced at the database level -- there is no validation that the reference id exists. We would add it to the schema like this, where the ref option is the name of another model class in our application.




 


const catSchema = new mongoose.Schema({
  name: {type: String, minlength: 3, maxlength: 50, required: true},
  age: {type: Number, min: 0, max: 25},
  owner: {type: mongoose.Schema.Types.ObjectId, ref: 'Person'}
})

Another difference from SQL is that the reference property could be an array.




 


const catSchema = new mongoose.Schema({
  name: {type: String, minlength: 3, maxlength: 50, required: true},
  age: {type: Number, min: 0, max: 25},
  owners: [{type: mongoose.Schema.Types.ObjectId, ref: 'Person'}]
})
# Example usage

The owners property stores an array of one or more ObjectId (opens new window) references to the unique _id of a document in the Person collection.






 


const Cat = mongoose.model('Cat', catSchema)

const kitty = new Cat({
  name: 'Fluffy',
  age: 4,
  owners: ['6021b7be9d383359e30a1327', '6021b94e6a3a0b5a03fcd7ba']
})

# Embedded Documents

The other option is to embed the related documents as properties of the main document.

const personSchema = new mongoose.Schema({
  firstName: String,
  lastName: String,
  phoneNumber: Number
})

const catSchema = new mongoose.Schema({
  name: {type: String, minlength: 3, maxlength: 50, required: true},
  age: {type: Number, min: 0, max: 25},
  owners: [personSchema]
})

const Cat = mongoose.model('Cat', catSchema)

const kitty = new Cat({
  name: 'Fluffy',
  age: 4,
  owners: [
    {firstName: 'Mickey', lastName: 'Mouse', phoneNumber: 14155551212},
    {firstName: 'Minnie', lastName: 'Mouse', phoneNumber: 14155551212}
  ]
})

Trade-offs

The relationship modelling method you choose is ultimately a trade-off between read query speed and data consistency.

The referenced document method ensures that data remains consistent because it is stored in only one place for each schema model, but it requires an additional database query for each related document when retrieving the primary document. For queries returning a single document (most CRUD operations) this may be fine. However for complex search queries returning thousands of results, there could be a significant performance hit.

The embedded document method avoids these additional database queries when retrieving data, but exponentially increases the complexity of maintaining data consistency because you are potentially storing the same data in many places and need to update them all manually.

If we look at the Cat example above, and consider what happens when we want to change Mickey Mouse's phone number. In the reference method, we simply find the Person document for Mickey Mouse and update the phoneNumber property. Then whenever we need to contact Fluffy's owner, the data will be correct.

However if we use the embedded method, we have to lookup all of the Cats that have an owner with the name Mickey Mouse and then update the embedded phone number. But what if there is more than one Person named Mickey Mouse? How do we know if we are updating the correct information? It can get messy if you don't plan it out well.

# Mongoose Query Methods

Mongoose models provide several static helper functions for CRUD operations. Each of these functions returns a mongoose Query object (opens new window) ... A query also has a .then() function, and thus can be used as a promise.

Review the mongoose documentation (opens new window) for the full list, but these are the most common ones that we will be using.

Model.find() (opens new window) is used to get an array of documents matching the given filter criteria.

const catsNamedSpot = Cat.find({name: 'Spot'})

These methods act on a single document matching the given id parameter.

  • Model.findById()
  • Model.findByIdAndRemove()
  • Model.findByIdAndUpdate()

These methods act on the first document found matching the given filter criteria.

  • Model.findOne()
  • Model.findOneAndRemove()
  • Model.findOneAndUpdate()

# Mongo Query Operators

In the filter criteria objects we can use any of the standard MongoDB query operators (opens new window). The syntax is to provide an object as the filter condition value for the given property. That object will have one of the Mongo query operators as it's key and the corresponding relative value. Let's look at some examples.

const catsAgedFourOrMore = Cat.find({age: {$gte: 4}})
const catsAgedLessThanFour = Cat.find({age: {$lt: 4}})
const catsAgedTwoFourOrSix = Cat.find({age: {$in: [2, 4, 6]}})
const catsNotAgedTwoFourOrSix = Cat.find({age: {$nin: [2, 4, 6]}})

# Mongoose Query Helpers

Mongoose provides some helper functions (opens new window) to make writing your queries more fluent. The syntax is sometimes more readable, but the end result is exactly the same. So, use whichever method is easier for you.

const catsAgedFourOrMore = Cat.find().where('age').gte(4)
const catsAgedLessThanFour = Cat.find().where('age').lt(4)
const catsAgedTwoFourOrSix = Cat.find().where('age').in([2, 4, 6])
const catsNotAgedTwoFourOrSix = Cat.find().where('age').nin([2, 4, 6])

# EX6-1 Mongo Cars

We are going to recreate our Cars API with Mongoose and MongoDB. Let start by creating a new project folder. I called mine week6.

mkdir week6
cd week6
echo "'use strict'" > app.js
echo ".DS_Store\nnode_modules/" > .gitignore
git init
npm init --yes
npm install express morgan
git add .
git commit -m "Initialize project folder"

# Connect to MongoDB

Before we can use Mongoose to connect to our MongoDB, we need to add it as an NPM dependency for our project.

npm install mongoose

We only need to create the connection to MongoDB once in our application. You should do it in app.js, right at the top, before you initialize Express. If it cannot connect to the database, then we cannot proceed. So, use the process.exit([code]) (opens new window) function to terminate the node.js runtime.

Just like last week, we require mongoose and then give it the connection string. This time the database path will be /mad9124.

'use strict'

const mongoose = require('mongoose')
mongoose
  .connect('mongodb://localhost:27017/mad9124', {
    useNewUrlParser: true,
    useCreateIndex: true,
    useFindAndModify: false,
    useUnifiedTopology: true
  })
  .then(() => console.log('Connected to MongoDB ...'))
  .catch(err => {
    console.error('Problem connecting to MongoDB ...', err.message)
    process.exit(1)
  })

Let's add the rest of our app.js boilerplate code.

const morgan = require('morgan')
const express = require('express')
const app = express()

app.use(morgan('tiny'))
app.use(express.json())

const port = process.env.PORT || 3030
app.listen(port, () => console.log(`HTTP server listening on port ${port} ...`))

TIP

Notice that we are using the morgan (opens new window) HTTP request logging middleware that we discussed in week 4. This will help with debugging, by showing the requests received by the server and their response codes.

Make sure that MongoDB database service is running. Go back to the folder where you created the docker-compose.yml file last week and run docker-compose ps. If it does not show the service is running, start it up with docker-compose up -d.

# nodemon

Up until now, each time we have made changes to the node/express application, we have to kill the running instance with CTLC and restart it with node app.js. Wouldn't it be nice if that happened automatically every time you save changes to the source code?

nodemon is a tool that helps develop node.js based applications by automatically restarting the node application when file changes in the directory are detected.

nodemon does not require any additional changes to your code or method of development. nodemon is a replacement wrapper for node. To use nodemon, replace the word node on the command line when executing your script.

# Install nodemon

Add nodemon as a development dependency for your project.

npm install nodemon --save-dev 

Then you can start your app using nodemon instead of the node command like this.

npx nodemon app.js

What is npx?

npx (opens new window) is a CLI tool installed along with npm and allows you to run packages that are installed as local project dependencies (rather than being installed globally). It will also let you run other CLI tools directly from the NPM registry - without explicitly installing them.

Check out this short article: npm vs npx — What’s the Difference? (opens new window)

If you started the app with nodemon, you should now see a successful connection message in the terminal.

# Add a script to package.json

Since this is something you will type over and over again, you might want to make a short-cut script in package.json. Add a start key in the scripts object like this ...



 


"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "nodemon app.js"
  },

Now you can just type npm start to fire up you server with automatic reloading.

# Make a git commit

git add -A
git commit -m "Add Mongoose db connection to Express app"

# Car Model

Before you can do any CRUD operations you need to define a Mongoose.Schema and create a Car Model object. Create a new models folder at the top level of your project and in it create a new file called Car.js -- note the capital 'C' as a reminder that this module will be exporting a class.

The basic template for a Mongoose Model module is ...

const mongoose = require('mongoose')

const schema = new mongoose.Schema({})
const Model = mongoose.model('Car', schema)

module.exports = Model

Let's expand the schema definition object on line three. We want to define three properties for the Car documents: make, model and colour. All will be of type String.

Unique Identifier

An _id property with a unique value will be automatically assigned to all new documents by Mongoose. You should not include the _id property in your scheme definition.

Review the full list of Mongoose Schema Types (opens new window) in the official documentation.

const schema = new mongoose.Schema({
  make: String,
  model: String,
  colour: String
})

# Make a git commit

git add -A
git commit -m "Add the Mongoose model for Car"

# CRUD with Mongoose

OK, you have a working connection from the Express app to the MongoDB server in your local Docker container. Now let's set-up the API routes to talk to the database instead of an in memory array.

Create a routes folder and in that folder create a new file called cars.js and stub out the six route handlers.

const express = require('express')
const router = express.Router()

router.get('/', async (req, res) => {})

router.post('/', async (req, res) => {})

router.get('/:id', async (req, res) => {})

router.patch('/:id', async (req, res) => {})

router.put('/:id', async (req, res) => {})

router.delete('/:id', async (req, res) => {})

module.exports = router

TIP

Since the only time we are using express in this module is to create a router instance. You can directly request the Router() method and immediately invoke it. So this code ...

const express = require('express')
const router = express.Router()

Can be simplified to this code ...

const router = require('express').Router()

But you can certainly write it out the long way if that makes it easier to read for you.

Now let's register the cars router in the main app.js module after the express.json() middleware.


 

app.use(express.json())
app.use('/api/cars', require('./routes/cars'))

TIP

One neat trick to simplify registering router modules with your Express app is to write the require() statement in-line with the app.use() method call. I find that this makes it easier to trace the application logic.

# Make a git commit

git add -A
git commit -m "Add cars router module"

# GET /api/cars

Use the find() method to query the Car model for a list all of the cars in the collection. Before you can do that, you need to import the Car model class.

Put this right at the top of /routes/cars.js

const Car = require('../models/Car')

And then update our route handler. The Mongoose find() method returns a Promise so you can use the async/await syntax ...

router.get('/', async (req, res) => {
  const cars = await Car.find()
  res.send({data: cars})
})

TIP

You could also use traditional Promise syntax.

router.get('/', (req, res) => {
  Car.find().then(cars => {
    res.send({data: cars})
  })
})

# Test it with Postman

OK. Let's test that in Postman. Send a GET request to the localhost:3030/api/cars URL. You should see a status code 200 response with an empty array for the data payload – you haven't added any Car objects to the database yet 😉

{
  "data": []
}

# Make a git commit

If everything is working as expected, make a commit.

git add -A
git commit -m "Add the GET all cars route logic"

That was easy! Let's do the POST route now so that you can add some cars to the database.

# POST /api/cars

Use the req.body as the input data to create a new instance of the Car model class – this will be a new Mongoose document. Then call the .save() method on that instance to persist it to the database.

# Never trust data from the client!

Before creating the new document, you need to ensure that the client did not try to set the _id value. You can use the JavaScript delete (opens new window) operator to remove the _id property from the req.body, if it exists.

TIP

Mongoose models will automatically discard any properties that are not defined in the Model.schema, so our route handler does not need to worry about extracting only permitted properties from the req.body as we had done in week 4.

router.post('/', async (req, res) => {
  let attributes = req.body
  delete attributes._id

  let newCar = new Car(attributes)
  await newCar.save()

  res.status(201).send({data: newCar})
})

Notice the `await` keyword

Any action that communicates with the database is an asynchronous action and your code needs to handle those Promises.

# Test it with Postman

Use the sample data from last week's cars.json file to use with Postman. Remember that you need to add the Car object as the JSON body for the POST request. Add each of the four cars, one at a time.

# Make a git commit

If everything is working as expected, make a commit.

git add -A
git commit -m "Add the POST route logic"

# GET /api/cars/:id

First let's do the happy path ... no error handling. You can use the .findById() static method on the Car model class. The id parameter from the request URI is accessible in the route handler on the req.params object.

router.get('/:id', async (req, res) => {
  const car = await Car.findById(req.params.id)
  res.send({data: car})
})

OK, now what if the id does not exist as a unique identifier in our database? You should wrap the code in a try/catch block and then send an appropriate error response back to the client.

router.get('/:id', async (req, res) => {
  try {
    const car = await Car.findById(req.params.id)
    if (!car) {
      throw new Error('Resource not found')
    }
    res.send({data: car})
  } catch (err) {
    res.status(404).send({
      errors: [
        {
          status: '404',
          title: 'Resource does not exist',
          description: `We could not find a car with id: ${req.params.id}`
        }
      ]
    })
  }
})

As we have seen before, you will need this same error handler in several other routes. So, extract this to a helper function. At the bottom of the /routes/cars.js module, but just above the module.exports line, add this private function.

function sendResourceNotFound(req, res) {
  res.status(404).send({
    errors: [
      {
        status: '404',
        title: 'Resource does not exist',
        description: `We could not find a car with id: ${req.params.id}`
      }
    ]
  })
}

module.exports = router

... and then update your route handler's catch block to call this function.







 



router.get('/:id', async (req, res) => {
  try {
    const car = await Car.findById(req.params.id)
    if (!car) throw new Error('Resource not found')
    res.send({data: car})
  } catch (err) {
    sendResourceNotFound(req, res)
  }
})

Better error handling

This is better but still incomplete. We will look at more robust error handling solutions in a couple of weeks.

# Test it with Postman

Do a GET request to the localhost:3030/api/cars route. Then copy the _id value for one of the cars. Now do another GET request appending that value to the end of the URL.

e.g. localhost:3030/api/cars/5ff776067b6bf504b6db320c

# Make a git commit

If everything is working as expected, make a commit.

git add -A
git commit -m "Add the GET by id route logic"

# PATCH /api/cars/:id

For the update actions, we can use the findByIdAndUpdate() static method on the Car model class. Similar to the findById() method, it takes a document _id value as the first argument. The second argument is an object with the properties to be stored. The third argument is an options object.

# Second argument - update payload

The _id property for the update operation is an immutable (read-only) field. To ensure that it has not been modified (accidentally or maliciously), use object destructuring to separate and discard any _id property that may be included in the req.body. Then reconstruct the update payload object, explicitly setting _id: req.params.id and then use the rest operator to extract the remaining attributes.

# Third argument - options object

The documentation lists many options that can be set to modify the behaviour of the update method. We are interested in two.

  1. Setting new to true will return the updated record from the database. The default behaviour returns the document in it's pre-update state.

  2. Setting runValidators to true ensures that your Model.schema rules are checked before the updates are applied.

router.patch('/:id', async (req, res) => {
  try {
    const {_id, ...otherAttributes} = req.body
    const car = await Car.findByIdAndUpdate(
      req.params.id,
      {_id: req.params.id, ...otherAttributes},
      {
        new: true,
        runValidators: true
      }
    )
    if (!car) throw new Error('Resource not found')
    res.send({data: car})
  } catch (err) {
    sendResourceNotFound(req, res)
  }
})

# Test it with Postman

Do a PATCH request using the same URI as the previous test. This time include a JSON object in the request body. It should change one or more of the Car object properties. e.g. set a new colour.

{
  "colour": "Silver Mist"
}

Try adding an invalid _id property to the update payload (request body).

{
  "_id": "5ff776067b6bf504b6db320f",
  "colour": "Silver Mist"
}

# Make a git commit

If everything is working as expected, make a commit.

git add -A
git commit -m "Add the PATCH route logic"

# PUT /api/cars/:id

By calling the PUT method, the client is asking entirely to replace the current document with the one supplied in the req.body. The route handler is identical to the PATH method with one additional property in the options object.

Set the overwrite option to true.

This will update any supplied document properties the same as with the PATCH method, but any document properties that are omitted from the req.body will be removed from the database.









 










router.put('/:id', async (req, res) => {
  try {
    const {_id, ...otherAttributes} = req.body
    const car = await Car.findByIdAndUpdate(
      req.params.id,
      {_id: req.params.id, ...otherAttributes},
      {
        new: true,
        overwrite: true,
        runValidators: true
      }
    )
    if (!car) throw new Error('Resource not found')
    res.send({data: car})
  } catch (err) {
    sendResourceNotFound(req, res)
  }
})

# Test it with Postman

# Make a git commit

If everything is working as expected, make a commit.

git add -A
git commit -m "Add the PUT route logic"

# DELETE /api/cars/:id

This time we want the findByIdAndRemove() static method on the Car model class.



 







router.delete('/:id', async (req, res) => {
  try {
    const car = await Car.findByIdAndRemove(req.params.id)
    if (!car) throw new Error('Resource not found')
    res.send({data: car})
  } catch (err) {
    sendResourceNotFound(req, res)
  }
})

# Test it with Postman

# Make a git commit

If everything is working as expected, make a commit.

git add -A
git commit -m "Add the DELETE route logic"

That's it! We have recreated the cars API from week 4 using MongoDB and Mongoose.

# Take a short break!

# EX6-2 Car Owner

In real life, cars have owners. Let's extend the API to include a /api/people resource path, and link the Person and Car models to each other by using the reference method.

# Person Model

Create a new file called Person.js in the /models folder.

const mongoose = require('mongoose')

const schema = new mongoose.Schema({
  name: {
    first: String,
    last: String
  },
  email: String,
  birthDate: Date,
  phone: Number,
  address: {
    streetNumber: String,
    streetName: String,
    city: String,
    region: String,
    country: String,
    postalCode: String
  }
})
const Model = mongoose.model('Person', schema)

module.exports = Model

# Update Car Model.schema

Add the owner property with the type being a mongoose.Schema.Types.ObjectId (opens new window) reference to the Person model.





 


const schema = new mongoose.Schema({
  make: String,
  model: String,
  colour: String,
  owner: {type: mongoose.Schema.Types.ObjectId, ref: 'Person'}
})

Also, right at the top we need to require the Person model so that Mongoose knows about it. This model would normally be loaded into our application in the people router module, but we haven't built that yet.

const Person = require('./Person')

# Populate owner

In the Car route handler for, you may include the related owner document as an embedded object with the the Car object. This saves the client from needing to make a second request.

To expand the referenced _id value for the owner property, we need to use the Mongoose .populate() (opens new window) method.

Populated paths are no longer set to their original _id , their value is replaced with the mongoose document returned from the database by performing a separate query before returning the results.

In the /routes/cars.js file, update the GET /:id route handler to chain the .populate('owner') method onto the findById() method call.



 







router.get('/:id', async (req, res) => {
  try {
    const car = await Car.findById(req.params.id).populate('owner')
    if (!car) throw new Error('Resource not found')
    res.send({data: car})
  } catch (err) {
    sendResourceNotFound(req, res)
  }
})

Let's test it!

Before we can test this out, there needs to be people in the database, and we need to either add a new car with an owner property or update one.

# Manually add people collection

Using MongoDB Compass you can manually create the people collection and add a document.

  1. Select the mad9124 database in the left sidebar.
  2. Click on the Create Collection button.
  3. Type the collection name as people -- all lowercase.
  4. Click the green Create Collection button.

screenshot of MongoDB compass screenshot of MongoDB compass screenshot of MongoDB compass

# Manually add a Person document

  1. Click on the newly created people collection in the collection list.
  2. Click on the green Add Data drop-down button, and select Insert Document.
  3. Add the properties and values for your Person document.
    Make sure it matched the schema in the Person model.
  4. Click the green Insert button.

screenshot of MongoDB compass

# Update a Car document

Now that you have a Person in the database you can link that document as an owner reference in one of the Car documents. Use Postman to do a PATCH update on one of your Car documents with the _id value from your new Person document as the value for the owner property of the car.

screenshot of MongoDB compass screenshot of Postman PATCH request

# Test the GET /api/cars/:id route

Now that there is a Car document with an owner property linked to a Person document, you can finally test the .populate() instruction that you added to the GET /api/cars/:id route handler.

You should see the owner property has been expanded to include the entire related Person document – not just the Person._id string.

screenshot of Postman GET request

# People Router

Now that you have seen how to implement the cars router module. Use that as a template to implement the people router module on your own.

Create a new /routes/people.js file to implement a router module for the /api/people resource path. Be sure to require the Person model in your router module. Then register the router in the main app.js file.

Test all of the routes with Postman.

Make a git commit after completing each route handler.

Bonus

Add a cars property to the Person model that holds an array of related Car references. Update the GET /api/people/:id route to populate the person's cars.

# Submit GitHub Repo

Create a new private repo on GitHub named mad9124-w21-ex6-mongo-cars.

Make sure that you have initialized your local project folder with git init and created a .gitignore file to exclude the node_modules folder and the .DS_Store files from your git archive.

Create a final commit to include all of your work from today's exercises.

Link the local repo to the GitHub repo and sync them up.

Remember to add rlmckenney as a collaborator on your GitHub repo so that I can see your code, and submit the GitHub repo's URL on Brightspace.

# For next class

Before next class, please read these additional online resources.

Start Assignment 2 - Mongo CRUD

You have enough information now to get started on Assignment 2. We will cover the remaining 15% of what you need to know next week.

This assignment is due before the start of our Week 8 class after the break. Don't wait until to the last minute!

Last Updated: 2/15/2021, 1:15:13 PM