diff options
Diffstat (limited to 'memos/WM-040.txt')
| -rw-r--r-- | memos/WM-040.txt | 293 |
1 files changed, 293 insertions, 0 deletions
diff --git a/memos/WM-040.txt b/memos/WM-040.txt new file mode 100644 index 0000000..29f9679 --- /dev/null +++ b/memos/WM-040.txt @@ -0,0 +1,293 @@ + + + + + + + +Document: WM-040 P. Webb +Category: Project 2019.09.08 + + Socii Dispatch 07 + +Abstract + + It's been a LONG road to 2.0. Old Socii was busted and didn't feel + good to use. Even *I* didn't want to use it so changes had to be + made. New Socii is super pleasant to use and I'm happy again. What + follows is a deep-dive into the history of the architecture and why I + made the changes I made. + +Body + + The previous version of Socii consisted of seven "microservices" and + two web apps that were eventually condensed to two APIs and + (hopefully) a sensible stack. This post will detail the process and + why the changes were made. + + 1. 1.0 + + 1.1. Microservices + + Horus: the notifier + Hu: the authenticator (user management) + Nemty: the emailer + Ra: the one who knows all (GraphQL) + Seshat: the keeper of records (post management) + Thoth: the media converter + + 1.2. Web Apps + + Marketing: the pretty informational site + Network: the actual web app + + 1.3. The Rest + + DigitalOcean: the server + Mailgun: backend for email delivery + MongoDB: the database + + 2. How it worked + + User visited `socii.network` and liked what they saw so they + decided to sign up. Being a new user, they were logged in + immediately. On the backend, Hu created the user and pinged Nemty + with the user's email address. Nemty created and sent a welcome + email to the user. + + If the user decided to make a post, Seshat would get those details + and save it to the database. Thoth took care of uploading the + user's profile picture. Horus facilitated details of post actions + (favorite, reply, &c) to Nemty. Ra was the glue that held + everything together. + + 3. What didn't work + + 3.1. Microservices + + Adding a feature to one microservice often meant updating nearly + all of them and that grew hella tedious. When Socii 1.0 was in + development the web industry was super buzzed about microservices, + everywhere! They would solve all your problems! In my case, they + *added* more than removed problems. It is *fully possible* that I + did not write them in the way microservices are *supposed* to be + written but when most of the tutorials on the subject came from + the Medium[1] hive mind, you can understand how good information + and examples (!!) are hard to come by. + + 3.2. Web Apps + + Like my microservices, I did not enjoy doing a lot of work on the + Network and then realizing Marketing wasn't up to date with things + ranging from a shade of blue to the modules powering the app. I + did a *lot* of copy/pasting and it felt like I was doing something + wrong. Tedium! + + 3.3. The Rest + + DigitalOcean and Mailgun have been doing well for me. In fact, I + believe I'm grandfathered into Mailgun's free tier due to the low + volume of Socii's email output. I've experienced zero issues with + email deliverability so I'd gladly pay for it (make Socii + popular y'all). + + MongoDB? *Sigh* + + Like many front-end developers, I gravitated to it for my projects + because it was the "least scary" database available. Plenty of + ORMs[2] are available for it and I used mongoose[3] for a while + before transitioning to mongoist[4] during one of the phases of + the refactor (in total, there have been at least four phases). + + The issue that stumped me for at *least* a month before urging me + to investigate other databases involved updating the schema of + posts and users. I forget the details exactly but I had a defined + schema and exported the data to perform a database migration. The + migration kept failing due to permission checks of some sort and + no amount of search engine kung fu rescued me from my woes. Mongo + is super unhelpful when things go wrong. + + I ended up cursing myself for being so reliant on a database with + piss-poor error messages. + + 4. The Search + + 4.1. Microservices + + I had been using ZEIT's[5] micro[6]/micro-dev[7] modules as the + basis for my microservices but the lack of updates/interest from + the maintainers[8] urged me to look elsewhere. + + I also read many articles about how companies like Airbnb and + Netflix managed their software, to better inform my own decisions + about the future of my microservices. In the realm of that + research I learned about "monorepos" with the most popular NodeJS + module being Lerna[9]. I decided not to use it but to implement + one of the core features, which was to put all of my microservices + into one API and go from there. + + 4.2. Web Apps + + The original Socii made extensive use of websockets in order to + provide a responsive UI. That…worked…for a while. I soon realized + I was merely replicating what single-page applications do. + + The web industry's favorite framework for SPAs is (was?) React and + I *despise* React. It has always seemed too bulky and I'm allergic + to packages with a boatload of dependencies. + + Far too many times in my web development career I have been forced + to stop using a useful package because of a dependency being out + of date or suffering from a vulnerability and the + author/maintainers of that dependency being unresponsive or plain + not caring. Bulky packages also become targets for misuse. + Companies like Snyk[10] exist to help mitigate this issue but the + first step, IMHO, is to limit the attack area yourself by choosing + smaller (often with full feature-parity) packages. + + Anyhoo, I looked at Preact[11], Composi[12], and a lot of + interesting HyperScript mini-frameworks before settling on + Mithril[13]. I really thought this was *IT*. I found the perfect + front-end framework! Right?! Heh, NO. + + The original codebase was server-side rendered which meant shared + links in Apple Messages and elsewhere online showed rich metadata; + title of the page, description, and Socii's icon image. With + Mithril, I lost all that. Frustrated once more, it was back to the + drawing board. + + 4.3. The Rest + + Your service is only as good as your database. + + Your database of choice is the jewel, the Big Kahuna, the very + *foundation* of your service. So, I tested Apple's FoundationDB. + Long story short, I found it relatively easy to setup but was + confused about how I'd put data into it's Document Layer (document + data model on FoundationDB, implementing MongoDB's wire protocol; + I'd be able to plug 'n play, basically). + + However, I did find an awesome NodeJS module written by Steve + Korshakov, CEO of Openland[14] <@openland/foundationdb[15]. + Unfortunately, it didn't have Document Layer support so my + search continued. + + No other database looked pleasant to use (to me). One of my + developer friends espoused the beauty of MariaDB but I was spoiled + by ORMs. I didn't want to learn how to write weird syntax just to + interface with my database (I also didn't realize at the time that + TypeORM[16] was a thing, which renders my previous sentence null). + + 5. 2.0 + + 5.1. Microservices + + Five of the original six microservices I created were condensed + into a single GraphQL API. The outlier was Thoth (media + converter/file management) because as great as GraphQL is, it + cannot handle file uploads as easy as a standard REST API could. + + Instead, this is the process: when I need to upload something + (like a profile picture for instance), the upload is happening in + the background as soon as possible. The response from Thoth is the + resulting URL from DigitalOcean's Spaces (basically AWS) that + becomes a hidden input's value that will finally get sent to the + API to be inserted into the database, after a user clicks + "save"/"upload". + + Kind of convoluted but it works for now. + + So, I now have two APIs: a GraphQL-based one (primary) and a + REST-based one (file uploads). Both are built upon the restify[17] + framework (it's like Express but focused on API creation). + + 5.2. Web Apps + + Back at the drawing board but *fully understanding* what my + requirements were for a front-end framework, it didn't take me + long to find another one of ZEIT's open-source projects, Next[18]. + Alas, it was built upon React but if I could have an SSR-powered + SPA I'd swallow that. And so I did until approximately a week and + a half later when I came across a blog post[19] comparing Next and + something called "Sapper". This section of the post blew my + mind (emphasis mine): + + > *The same ‘hello world' app that took 204kb with React and Next + > weighs just 7kb with Sapper.* That number is likely to fall + > further in the future as we explore the space of optimisation + > possibilities, such as not shipping any JavaScript *at all* for + > pages that aren't interactive, beyond the tiny Sapper runtime + > that handles client-side routing. + + Sapper has Express-compatible middleware, making it easy to + understand and customize for people who are used to using Express + (and Fastify, Restify, &c). It is built upon a framework that's + not *really* a framework: Svelte. Like the name implies, Svelte + compiles code to tiny, framework-less vanilla JavaScript. Quite + honestly, this is everything I've been looking for; a lightweight + SPA with SSR support! + + With the web app framework FINALLY figured out, I had to then + figure out what I wanted to do about having two front-ends to + manage. And…I didn't want to manage two of them. I decided to + merge them instead and deprecate the use of a sub-domain for the + Network. So, if you were logged-in you would see the global feed + but if you were logged-out you would see the marketing homepage. + The navigation would also change accordingly. + + 5.3. The Rest + + The database I settled upon is RethinkDB[20]. An interesting + choice considering that one of the most commented issues in one of + its repos on Github is titled, "is this project dead?[21]". Well, + the original company behind it is but there's still a small, + vibrant community around it that's been using Rethink in + production for years. + + One of the great things about Rethink is the web interface[22] it + comes with (you can disable this if you want). This interface has + a feature called "Data Explorer" and like the name implies, it + allows you to test database commands (or, quickly query something + from your database). Coming from a database that effectively told + you nothing via opaque errors to a database that gave you the + tools to succeed was a breath of fresh air and made me question + why the company behind Rethink failed in the first place. The + founder of the company, Slava Akhmechet, wrote about their + failure[23] in depth and the TL;DR is: great product, wrong time. + + 6. The Future + + I believe I have created a stable foundation for the future of + Socii's development. So, what's next? A laundry list of features: + images in posts, more profile customization, groups, iOS app, + re-enabled JSON Feed for profiles, end-to-end encryption, &c. I + don't have experience with CI/CD (Continuous + Integration/Development) workflows so maybe that'll be the next + thing major thing I tackle. Of course, I'll share my experiences + along the way. 🕸 + +References + + [1] <https://medium.com> + [2] <https://stackoverflow.com/a/1279678> + [3] <https://github.com/Automattic/mongoose> + [4] <https://github.com/saintedlama/mongoist> + [5] <https://zeit.co> + [6] <https://github.com/zeit/micro> + [7] <https://github.com/zeit/micro-dev> + [8] <https://github.com/zeit/micro-dev/issues/41> + [9] <https://github.com/lerna/lerna> + [10] <https://snyk.io> + [11] <https://github.com/preactjs/preact> + [12] <https://github.com/composi> + [13] <https://github.com/MithrilJS/mithril.js> + [14] <https://openland.com> + [15] <https://fdb.openland.com> + [16] <https://github.com/typeorm/typeorm> + [17] <https://github.com/restify/node-restify> + [18] <https://github.com/zeit/next.js> + [19] <https://svelte.dev/blog/sapper-towards-the-ideal-web-app-framework> + [20] <https://github.com/rethinkdb> + [21] <https://github.com/rethinkdb/rethinkdb/issues/6747> + [22] <https://rethinkdb.com/docs/administration-tools> + [23] <https://www.defmacro.org/2017/01/18/why-rethinkdb-failed.html> |