Prevent TypeORM from eating up all your memory

Recently, a client told me that the code I had written to sync their systems and a new datasource blew up their AWS EC2 instances. Apparently, the synchronisation engine kept getting killed for using up too much memory. More than a few gigabytes, actually. To sync a few tens of thousands of SQL rows. That didn't seem quite right, but they were adamant, so I reluctantly and doubtfully looked into it. Only to realise they were absolutely right.

Each time I ran a sync locally, node's memory usage blew up. 1GB, 2GB, 3GB.... sometimes even 4, before going suddenly back to less than a hundred MB. Now that was weird. Because there was no way a bit of text (ok, a lot of text, for a human, but certainly not a full gigabyte worth of text, let alone 4) could do that. Did I make a mistake? Somehow copy some dictionaries and objects all over the place.

It turns out, we were using TypeORM (a client requirement). And, unbeknownst to all of us, TypeORM is a greedy bastard, sometimes taking up more than 2.5Gb to fetch 4 rows (and taking its sweet time while doing so). It especially seems to have an issue with resolving one to many and many to many relationships through joins.

From what I've found, the only solution, aside from switching to another ORM, or getting rid of ORMs altogether, is to fetch the relationships ourselves and then repopulate our models ourselves. Meaning more work and less reliance on our ORM's features for us.

Here's an example.