You've profiled your app. Squeezed out of it every tiny bit of performance you possibly could. Yet, when you turn to the app's network requests, you still feel you could make it faster.

Well, if you've cleanly separated your firestore collections, based on what they are, and need to do more than a single query to get your view's data, you are right. You can make it faster.

Or perhaps you are looking at DynamoDB, Firestore & other NoSQL databases, and wondering how to approach running big, complex, queries.

In both case, today's our lucky day. I currently work as an iOS developer (mostly freelance), but not so long ago I was employed as a serverless backend developer (at Spotr), dabbling in front-end. And today is the day where this diverse experience comes in handy, in the form of CQRS and projections.

Today's example

Let's pretend we're making a dating app for lonesome cowboys' horses. I mean, Jolly Jumper too deserves a chance to find love, right?

We have horses (an id, a name, an owner, a species, a reference to the different "characteristics" the horses has, a reference to it's latest known city (after all, these horsies travel quite a lot, all over the Far West), and finally a flag indicating wether or not it should appear in the app),...

interface Horse {
    id: string,
    name: string,
    species: string,
    city: string,
    characteristics: string[],
    hidden: boolean
}

... pictures (the URLs to the various image sizes and a reference to their horse and / or tag via their respective ids, a flag indicating wether or not it should be displayed).

interface Picture {
    id: string,
    urls: {
        small: string,
        medium: string,
        large: string,
        original: string
    },
    horse_id: string | null,
    characteristic_id: string | null,
    city_id: string | null,
    hidden: boolean
}

Since a horse can potentially have many usable or unusable pictures, and a picture can be used for both a horse and a characteristic, we can't really put them in the horse's document directly (yes, I'm stretching a bit here; our horses probably won't have more than 5 or 10 pictures each. But for Spotr's spots it was a totally valid concern, and making up perfect examples is hard).

... And finaly, cities! With a geolocation, a name, and a country (I know a city's name and a country aren't enough to get a unique combination, but let's keep it simple).

interface City {
    id: string,
    name: string,
    geolocation: {
        latitude: number,
        longitude: number
    },
    country: string
}

The problem(s)

What this means is that when you want to display some of the horse's data in a list, along with it's most recent picture, you need to get the horses, and then each and every one's most recent picture, using each horse's id.

And if you want to get a list of all the horses nearby, you need to get a list of all the nearby cities then, for each and every one of them, the list of horses in these cities, then each horse's most recent picture.

And finally, if you then want to display all of a horse informations in your apps, you need to first fetch the horse's own data and all it's displayable pictures using the horse's id, then fetch the horse's city, and each of the horse's characteristics, individually (this is Firestore, no batch get). A minimum of four queries, to show a single view. That's a lot (keep in mind I set aside some of the other models we have).

Each request takes some time to run, and some need to wait for data from a previous request to be started. It's error-prone client-side, slow, and terribly inefficient. You could make it go fast, but it'll never be an optimal experience for your users. How could we reduce that as much as possible?

Today's solution

The answer is something I learned from Command Query Responsibility Segregation (CQRS in short): Projections (or views). Never heard of CQRS? Here's a very nice article from the fantastic Martin Fowler).

We will therefore create additional collections, projections, of our original models, custom tailored to each of our app's views. In our case, the horses lists views and the horses details view. Just like, in Spotr, we have spots' list views and spots detail views.

The ’sources of truth‘ (the original models' collections)

These are the original objects. The ones from which the projections will be built and should be able to be rebuilt from, from the ground up, at any time. Because here's a key thing: the projection's state doesn't matter. You shouldn't care about it. You shouldn't care about preserving it. At all.

The projections

How to define them: working together with the key stakeholders. In this case, your team's front-end and / or mobile developers. They are the ones with their noses in the clients' views all day; the ones who know what data they use and where.

Be careful, and properly ask them questions about what they request. They don't necessarily have your constraints in mind. For example, I was asked if I could put some user specific informations (wether or not the spot is in the user's favorites) in my projections. That would mean having a projection of each spot,  for each user. A big no-no (it would take ages to update them all at each picture or spot update, take up lots of storage, they'd all be updated all the time...). You also don't want some potentially massive properties in your projections (such as all the item's likes or comments).

After discussing and implementing it with them, here's what we would come up for our hypothetical horses dating app.

interface HorsesListHorseProjection
    id: string,
    name: string,
    city: {
    	id: string,
    	name: string,
    	geolocation: {
        	latitude: number,
        	longitude: number
    	},
    	country: string
    },
    picturesUrls: string[]
}

Keeping the projections up to date

Firebase Cloud functions come with triggers. There are Authentication triggers, Remote Config triggers, and what interests us today... Firestore triggers! Quite a few, actually. We get document creation, deletion, and update events (as well as a more generic `write` event). We'll subscribe to the horses' collection's creation, deletion and update triggers, and to the pictures' collection's write trigger. You'll see why. No need to subscribe to the cities collection triggers, because those shouldn't move around.

On horse creation

What do we want to do at the creation of a  horse document? If it's visible, create it's projection. To do that, we need the horse's pictures (if there are any), create the projection, and store it. Otherwise, if the horse is hidden, we do nothing.

Here's how we do it:

Get it's pictures

const onCreation = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onCreate(async (snap, context) => {
	const { data } = snap;
	const horse = data();
	if (horse === undefined || horse.hidden === true) {
		return
	}
	const [{ docs: picturesDocs }, { data: cityData }]  = await Promise.all([
		picturesCollection.where("horse_id", "==", horse.id).get(),
		citiesCollection.doc(horse.city).get()
	])
	const picturesUrls: string[] = picturesDocs.reduce( (array, doc) => {
		const pictureData = doc.data();
		if (pictureData) {
			return array + pictureData.urls.map( (url: any) => url.large as string);
		}
		return array;
	}, []);
});

Create a projection

const onCreation = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onCreate(async (snap, context) => {
	const { data } = snap;
	const horse = data();
	if (horse === undefined || horse.hidden === true) {
		return
	}
	const [{ docs: picturesDocs }, { data: cityData }]  = await Promise.all([
		picturesCollection.where("horse_id", "==", horse.id).get(),
		citiesCollection.doc(horse.city).get()
	])
	const picturesUrls: string[] = picturesDocs.reduce( (array, doc) => {
		const pictureData = doc.data();
		if (pictureData) {
			return array + pictureData.urls.map( (url: any) => url.large as string);
		}
		return array;
	}, []);
	const city = cityData() ?? null;
	const projection = {...horse, city, picturesUrls};
	return projectionsCollection.doc(horse.id).set(projection);
});

On horse update

What about an update on horse document? If it's visible, get it's picture and create it's projection! Otherwise, delete the projection (if it exists). Wait. Isn't that almost the same thing as what we just did? Yes it is! So let's not repeat ourselves, shall we? Here's how it's done:

If horse is not hidden, same as horse creation

const onUpdate = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onUpdate(async (snap, context) => {
	const { id, data } = snap.after;
	const horse = data();
	if (horse === undefined || horse.hidden === true) {
		return return;
	}
	const [{ docs: picturesDocs }, { data: cityData }]  = await Promise.all([
		picturesCollection.where("horse_id", "==", horse.id).get(),
		citiesCollection.doc(horse.city).get()
	])
	const picturesUrls: string[] = picturesDocs.reduce( (array, doc) => {
		const pictureData = doc.data();
		if (pictureData) {
			return array + pictureData.urls.map( (url: any) => url.large as string);
		}
		return array;
	}, []);
	const city = cityData() ?? null;
	const projection = {...horse, city, picturesUrls}
	return projectionsCollection.doc(horse.id).set(projection);
});

Else, if hidden, delete projection

const onUpdate = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onUpdate(async (snap, context) => {
	const { id, data } = snap.after;
	const horse = data();
	if (horse === undefined || horse.hidden === true) {
		return projectionsCollection.doc(id).delete();
	}
	const [{ docs: picturesDocs }, { data: cityData }]  = await Promise.all([
		picturesCollection.where("horse_id", "==", horse.id).get(),
		citiesCollection.doc(horse.city).get()
	])
	const picturesUrls: string[] = picturesDocs.reduce( (array, doc) => {
		const pictureData = doc.data();
		if (pictureData) {
			return array + pictureData.urls.map( (url: any) => url.large as string);
		}
		return array;
	}, []);
	const city = cityData() ?? null;
	const projection = {...horse, city, picturesUrls}
	return projectionsCollection.doc(horse.id).set(projection);
});

On horse deletion

That's a simple one; we just delete the projection and the pictures, if they exist

Delete it's projection

const onDeletion = (picturesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onCreate(async (snap, context) => {
	const { id } = snap;
	return projectionsCollection.doc(id).delete()
});

Delete it's pictures

const onDeletion = (picturesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onCreate(async (snap, context) => {
	const { id } = snap;
	return Promise.all([
        projectionsCollection.doc(id).delete(),
        picturesCollection.where("horse_id", "==", id).get().then( snapshot => snapshot.docs.map( doc => doc.id).map( id => picturesCollection.doc(id).delete()))
    ]);
});

On picture write.

When a picture is created, we want to get its horse, and update the horse's projection (if it has one) with the newly created picture.

What about when a picture is updated? Well, a picture's url shouldn't change, so there isn't much to update. But in any case, should it somehow happen, what would we do? Yup. Get it's horse, and update the horse's projection! So... the same as before.

And deletion? We would.... wait for it... get the horse.... and update the horse's projection! Again!

Now, something I like to avoid is playing around with mutable data, and doing pretty much the same thing in more than one place (the DRY principle). What's the simplest way of doing of updating a horse's projection without breaking immutability, all while writing as little code as possible and maintaining readability? Create a new projection from scratch!

Since we're now doing the same thing for each and every trigger, we don't care wether it was a creation, an update or a deletion. We only write one function, and subscribe it to the write trigger. That's how we avoid duplicating the same code all over the place and deploying three copies of (pretty much) the same function.

Find the horse

const onWrite = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (horsesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/pictures/{pictureId}`).onWrite(async (change, context) => {
	const { data } = change.after.exists ? change.after : change.before;
	const picture = data();
	if (picture === undefined || picture.horse_id === null) {
		return
	}
	const { data: horseData } = await horsesCollection.doc(picture.horse_id).get();
});

Create the horse's projection (same as on horse creation).

Or delete it, depending on wether or not the horse is hidden.

const onWrite = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (horsesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/pictures/{pictureId}`).onWrite(async (change, context) => {
	const { data } = change.after.exists ? change.after : change.before;
	const picture = data();
	if (picture === undefined || picture.horse_id === null) {
		return
	}
	const { data: horseData } = await horsesCollection.doc(picture.horse_id).get();
	if (horse === undefined || horse.hidden === true) {
        return Promise.all([
            projectionsCollection.doc(horseId).delete(),
            picturesCollection.where("horse_id", "==", horseId).get().then( snapshot => snapshot.docs.map( doc => doc.id).map( id => picturesCollection.doc(id).delete()))
        ]);
	};
	const [{ docs: picturesDocs }, { data: cityData }]  = await Promise.all([
		picturesCollection.where("horse_id", "==", horse.id).get(),
		citiesCollection.doc(horse.city).get()
	])
	const picturesUrls: string[] = picturesDocs.reduce( (array, doc) => {
		const pictureData = doc.data();
		if (pictureData) {
			return array + pictureData.urls.map( (url: any) => url.large as string);
		}
		return array;
	}, []);
	const city = cityData() ?? null;
	const projection = {...horse, city, picturesUrls}
	return projectionsCollection.doc(horse.id).set(projection);
});

Now, why do I insist on immutability, even at the cost of regenerating the projections from scratch each and every time? Why the choice to make them immutable?

First, it's "cheap". Second, by systematically discarding projections' current state, we can ensure our projections' won't get corrupted in the long run by piling up mutations. Third, it makes it easier to change their model or fix them if / when they somehow get corrupted by a bug.

Speaking about making things easier, you've probably noticed we repeat ourselves quite a lot, across the different cloud functions. So let's refactor all this.

All together

Here's what it would look like. Of course, if this where a real project, we'd split this across multiple files. But it isn't, and I'm getting tired 🙂.

/**
 * MARK - Interfaces
 */

// Our Horse interface, because we're in Typescript
interface Horse {
    id: string,
    name: string,
    species: string,
    city: string,
    characteristics: string[],
    hidden: boolean
}

/**
 * MARK - Shared / refactored functions
 */

// An extracted, common, function to create a horse's projection from a horse's data
const createProjectionFromHorse = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => (horse: Horse) => {
	const [{ docs: picturesDocs }, { data: cityData }]  = await Promise.all([
		picturesCollection.where("horse_id", "==", horse.id).get(),
		citiesCollection.doc(horse.city).get()
	])
	const picturesUrls: string[] = picturesDocs.reduce( (array, doc) => {
		const pictureData = doc.data();
		if (pictureData) {
			return array + pictureData.urls.map( (url: any) => url.large as string);
		}
		return array;
	}, []);
	const city = cityData() ?? null;
	const projection = {...horse, city, picturesUrls}
	return projectionsCollection.doc(horse.id).set(projection);
}

// A common, shared, function, to delete a horse's projection
const deleteHorseProjection = (picturesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => (horseId: string) {
	return Promise.all([
		projectionsCollection.doc(horseId).delete(),
		picturesCollection.where("horse_id", "==", horseId).get().then( snapshot => snapshot.docs.map( doc => doc.id).map( id => picturesCollection.doc(id).delete()))
	]);
};

// An extracted, common, function to handle what to do when a horse is directly (via it's document) or indirectly (via one of it's pictures) updated
const onHorseUpdate = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => (horseId: string, horse: Horse | undefined) => {
	if (horse === undefined || horse.hidden === true) {
		return deleteHorseProjection(picturesCollection)(projectionsCollection)(horseId);
	}
	return createProjectionFromHorse(picturesCollection)(citiesCollection)(projectionsCollection)(horse);
};

/**
 * MARK - Cloud Functions
 */

// Horse - Creation - The Cloud Function triggered on a Horse's document creation
const onCreation = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onCreate(async (snap, context) => {
	const { data } = snap;
	const horse = data();
	if (horse === undefined || horse.hidden === true) {
		return
	}
	return createProjectionFromHorse(picturesCollection)(citiesCollection)(projectionsCollection)(horse as Horse)
});

// Horse - Update - The Cloud Function triggered on a Horse's document update
const onUpdate = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onUpdate(async (snap, context) => {
	const { id, data } = snap.after;
	const horse = data();
	return onHorseUpdate(picturesCollection)(citiesCollection)(projectionsCollection)(id, horse as Horse);
});

// Horse - Deletion - The Cloud Function triggered on a Horse's document deletion
const onDeletion = (picturesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/horses/{horseId}`).onCreate(async (snap, context) => {
	const { id } = snap;
	return deleteHorseProjection(picturesCollection)(projectionsCollection)(id);
});

// Pictures - Write - The Cloud Function triggered whenever a Picture's document is written (creation / update / deletion)
const onWrite = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (horsesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => functions.region('europe-west2').firestore.document(`/pictures/{pictureId}`).onWrite(async (change, context) => {
	const { data } = change.after.exists ? change.after : change.before;
	const picture = data();
	if (picture === undefined || picture.horse_id === null) {
		return
	}
	const { data: horseData } = await horsesCollection.doc(picture.horse_id).get();
	return onHorseUpdate(picturesCollection)(citiesCollection)(projectionsCollection)(picture.horse_id, horseData() as Horse | undefined);
});

module.exports = (picturesCollection: admin.firestore.CollectionReference) => (citiesCollection: admin.firestore.CollectionReference) => (horsesCollection: admin.firestore.CollectionReference) => (projectionsCollection: admin.firestore.CollectionReference) => ({
	horse: {
    	onCreation: onCreation(picturesCollection)(citiesCollection)(projectionsCollection),
        onUpdate: onUpdate(picturesCollection)(citiesCollection)(projectionsCollection),
        onDeletion: onDeletion(picturesCollection)(projectionsCollection),
    },
    pictures: {
    	onWrite: onWrite(picturesCollection)(citiesCollection)(horsesCollection)(projectionsCollection)
    }
});

Parting words

Now, your front and mobile developers should be quite happy.

Fewer network requests, less filtering, no useless properties showing up, and less data transfers overall, making for a snappier and faster app!

Well done!

P.S. I haven't had the courage to test this exact code so far. I might, in the future, in which case I'll gladly remove this paragraph 🙂. However, the general idea is the same as what I've done in my job, so I know from experience that both the concept and approach work.

As always, if you have any suggestion on how to improve this, or any question, let me know on Twitter! I'll gladly and happily take your feedback, and do my best to answer any and all questions!

Thanks you for reading this far, and I whish you all a great, beautiful day!