This place is where we track our engineering tasks and notes. We are dogfooding πŸŒ­πŸ• at the moment and working towards a point where this tool is friendly enough for others to use. Cleaning up the UI and providing some basic edit capability are some steps towards that goal.

TL;DR: Effectively this page is just a scrappy plaintext reader (of Org file to be precise, for the non-Org-mode readers on our team). You should be able to edit notes and tasks in a Confluence/Notion-like manner soon while simultaneously allowing teammembers on your team to interface with this knowledge-base and project-management setup through git and thus keep a full history of your project-management and knowledge-base on their machine.

I, for example, use this file inside of Emacs but am well aware that it would be torture to ask my fellow devs, let alone non-engineering colleagues, to fire up Emacs to get any work done. πŸ˜…

Using git underneat will allow us to facilitate nifty diffing features, time-machine like features or proper roll-backs when shit hits the fan. *Git is a helluva tool* and its time to expose more folks to the magic (magit users will appreciate the pun here) of git without exposing them to git mechanics.

Scroll down to section Background for more information and use the top-right perspective-switch to switch between Prose and Kanban view on this document.

πŸ™Š It's very scrappy but it's a little easier than sending ox-publish HTML or LateX exports over the wire (Slack or mail) or by asking folks to learn Emacs πŸ™ŠπŸ˜….

🎯 Goal

Offer something that is way more fun/pleasant to use than Confluence and which is more tolerable than Notion. We want to bring the joy back in collaboration and are approaching this from a developer-first angle.

😭 Problem

Knowledge bases and project management tools are currently a source of frustration 😑 and devs are largely keeping notes in separate silos: in different note-taking tools or just on folders on their local machines for easy access simply because the current tools are clunky.

There is a much intel lost which hurts organizations! πŸ“‰

Removing the barriers to share knowledge helps teams keep relevant intel together 🧠 and should minimize knowledge loss when people are out-of-office or move on to other challenges. People should ideally ideate in their knowledge management/project-management tools and not be forced to document ideas there as a rote task after conception. We're out to break the barriers to make documenting and ideating together much easier -- challenge accepted. πŸ˜…

🌟 Vision

  • Bring play back into collaboration. Collaboration should be like dancing πŸ’ƒ or playing a game of ball πŸ€.

    There should be a sense of flow that one can enjoy while at it. We are figuring out how to bring that back into this category of tooling.

    Sometimes this means less rigor in the way we work together. There are no fixed lines within which to color! You can start with a basic text file and then gradually start jotting down ideas which grow to become tasks and then can get split up as we need in a very organic manner. Like claymolding -- but at a deskjob and without the sticky fingers.

  • Facilitate people in their optimal modality as we don't all consume information very well in the same way.

    Some of us need long texts to absorb ideas, some need to hear a voice (audio) and many we need visual/spatial representations to process information. Let's empower teams by providing them the modalities that make sense for whatever they're trying to do as the same data can be presented in different ways so we're doing that.

    You should be able to consume a knowledge base while strolling through the park listening to it podcast-style. You should be able to look at a document with tasks in a Kanban-like perspective or as a task dependency graph. We want knowledge-bases and project-management tooling to be more like a Rubik's cubes that you can turn around at study from different perspectives to improve your understanding. 🧩

    πŸ‘¨πŸΏβ€πŸ’» If you're a dev, we'll make it dead-easy to collaborate through the tools you already know very well like plaintext and git, so those notes of yours have no excuse to stay just on your box. Adding them to the knowledge base should be easier than stubbing a prototype and should happen with the toolbox within 🧰 which you play to your strengths.

  • Simplify how we work together by introducing some text assist capabilities for conveniences we haven't had access to before.

    When engineers draft complex technical texts, because this comes natural to them, we can utilize language models to attempt to paraphrase them for more junior, or perhaps, non-technical team members. Think Reddit's ELI5 r/ExplainlikeImFive or Wired's One Concept in 5 Levels of Difficulty.

    Relieving folks from having to write for multiple audiences should lower the barrier for team members to enter their thoughts as quickly as they have them πŸ’‘. Less curation, more ping-ponging πŸ“, as in, bouncing ideas back and forth but maybe also actual ping-pong if much time is being spared by bringing that flow back.

[31%] Prototype 01

This is the scrappy app (scr-app 🀣) that we are using to validate some early ideas.

βš’οΈ Mason's code: Mark tasks as =DONE= only when they are merged into branch =hack=.
TODO
DONE @vidbina
- State "DONE"       from "TODO"       [2022-09-14 Wed 11:40] \\
  Already live!

Based on HN input by moeffju, we do readers a service by not displaying non-functional anchors and nudging them into trying to use them and getting lead astray. This is totally on-point so, let's remove all anchors until we sort out the link following.

DONE

Fix TOC Storybook

@tijan
- State "DONE"       from "TODO"       [2022-09-25 Sun 21:01] \\
  Done by @tijan and already merged. Had something to do with me resolving a merge to use a element instead of divs. 🀦🏿

The TOC is broken in Storybook and this Stack Overflow thread may provide some explanation of the issue at hand. Let's either fix the Storybook or figure out of this case is not testable in Storybook and then just make a note of this issues such that future developers know not to attempt testing click-through inside of Storybook.

DONE

[100%] Design headline linking strategy

@vidbina
- State "DONE"       from "TODO"       [2022-09-25 Sun 21:03] \\
  Mark headline linking as done for now.

Background

Requirements
Headline ids should be shareable between different people

Headlines ids can not be randomly generated on a per-session basis (to map between ToC entries and headlines) as these id are not guaranteed to be the same during another session and therefore not shareable.

Headline ids should be shareable between different timestates

Bookmarked links need to survive non-destructive diffs (when just new content is added).

If the headline is removed then notify the user that the link has been removed at a given version, timepoint $t$, and provide an option to navigate to state $t$ to view the headline anyways

Headline ids should be valid element identifier as per W3C standard
Approach

Rough outline how we will deal with headline identifiers.

Instantiation

This is the scenario where a headline is completely created within our tool.

⚠️ Any headline created in another tool is not guaranteed to satisfy the characteristics of a headline populated by us so, we have to be graceful in how we handle headlines. πŸ˜…
  • If headlines are created within the tool:
    • populate a unique CUSTOM_ID that can will always resolve to the headline as long as the headline or the CUSTOM_ID is not removed (see Internal Links)
  • If headlines are renamed through the tool
    • maintain CUSTOM_ID if already present
    • populate CUSTOM_ID matching what may have been considered the previous headline by either of the following methods (first one wins):
Referencing

This operation does not mutate the document in any form.

  • If CUSTOM_ID is defined, use this value
  • If radio target (see Org Manual) is set for the heading, use this value
  • Iterate over all headings and use naive heading-to-id conversion logic to identify headline
    • if found, index this link for this particular document version
Indexing

With indexing we keep track of headline identifiers throughout the lifetime of a document. This indexing should enable us to provide guidance to the user when a link is followed to a headline that may no longer be there or may have been modified (for example when a CUSTOMID is defined for a heading that was previously referenced through the id obtained by the heading-to-id conversion logic).

TODO
Heading-to-id conversion logic
Background
:ID:       904ea339-a66d-4c68-8278-7b80594f9e70

Org implements its own <<<heading-to-id conversion logic>>> that roughly works as follows:

DONE @vidbina
- State "DONE"       from "TODO"       [2022-09-25 Sun 21:02] \\
  Mark link following on headline links as done.

Following links is a core feature of any document. Let's provide some basic implementation. In the long-tail, we will need to keep track of heading references over the lifetime of a document just to ensure that we can provide useful guidance on how to resolve links (for example, inform a user to go back to a previous version of a document).

DONE
- State "DONE"       from "TODO"       [2022-09-23 Fri 17:37] \\
  Index can be populated but we are populating it at the end of parsing instead of populating it on an ongoing basis. The reason for this is that I simply couldn't figure it out fast enough. Recursing over a document leaves the problem of sharing some document-global state to verify that human-friendly slugs do not collide. Trying to do this with map/reduce like mechanisms is problematic because the index state is copied along and there is a change that competing branches settle on the same slugs, allowing for collisions.
  
  Sleep deprivation and tunnel-vision are almost certainly in the mix here, so after a chat with @stefano we opted for the dead-simple approach of generating unique ids for now and then indexing them post-parse. This sucks because we now have to figure out how to lookup the human-friendly slugs in the rendering step. This will require us to pass some document-local state along to all the render functions. Just ideas. Maybe this is all BS and we'll find a better design after we've enjoyed quality sleep. πŸ€·πŸΏβ€β™‚οΈ

Links such as [[*Headline]] should basically work and resolve to the first match. So if there are multiple headlines with the same text, the first one is always the one reached by such a link.

We can link to a headline by providing the headline text sans the keyword, statistics cookie and tags. Observe list item 1 in the snippet below that demonstrates a valid link to the headline defined at the bottom of the snippet.

  • remove all statistics cookies
  • whitespace
    • trim
    • collapse multiple whitespace characters to single character
DONE
Populate shareable heading ids
- State "DONE"       from "WIP"        [2022-09-25 Sun 20:37] \\
  Done with latest merge to hack (see https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/88)
- State "WIP"        from "TODO"       [2022-09-23 Fri 17:43] \\
  We are currently generating these ids through a helper function which runs through the flat list of headings and produces all slugs that we need. In an ideal world, I imagine populating an index while the document is being transformed (to our internal representation). Fundamentally, I'm torn on whether this information belongs in the internal representation of the headings. Part of me says that slugs, ids and linkTexts belong in the heading while another part says that we should just keep these in indices and provide a fast reference to the data (the heading).

Multiple headings which resolve to the same plaintext, should be numbered such that the id uniqueness constraints remains honored (a given id may only exist once).

We cannot use PRNG-generated heading ids because such ids will be different between sessions.

DONE

Generate unique id for every parsed headline

@vidbina
- State "DONE"       from "TODO"       [2022-09-23 Fri 17:46] \\
  Consider the scrappy part done!

The availability of unique ids for every parsed headline may simplify the implementation of linking. Within a session we can just use the session-scoped unique ids.

These ids will not be stored as data as they are only meant to be used within the intermediate representation to provide us quick references that are "guaranteed" to not collide within a session scope.

With such unique ids, we simplify the implementation of linking as we can just implement a lookup table that points headline links, CUSTOM_ID links, ID links or other linking methods to a single reference.

We'll stub a scrappy implementation and will optimize later. πŸ˜…

TODO

Simplify file source access flow

UX
TODO

Add a clear Follow URL CTA

@siarhei

Remove emphasis from Visit Source button and add a clear Follow URL CTA (Call-to-Action).

Problem

As a user, I press the Visit Source button expecting to be lead to the URL entered inside of the URL input box. This is a usability problem. We need a strong CTA near the URL input box that does exactly what a typical end-user would expect: lead one to the page that matches the entered URL.

πŸ’‘ Try to enter https://gitlab.com/formation.tools/eng/proto-01/-/raw/hack/README.org into the URL input control to see what we mean. When pressing the blue button it only navigates to the source of the current page. The lack of a connection between that prominent button and the input text is a bit confusing. Following the entered text is only possibly by pressing Enter.

Definitions
  • leads a user out of the app to the authoritative source on the internet
  • leads the user to a in-app rendering of the URL that was provided (core use-case of the app)
Background
πŸ’‘ The URL input textbox will very likely disappear once we allow people to login with their own GitHub and GitLab accounts. At that point, users will just navigate to the files they want through the sidebar menu. Since we're not there yet, the URL input box is an interim solution for now so it doesn't have to be perfect but should be at least "usable enough" -- as in, not confusing even the people who made it. We improvised input bar without the guidance from design so this explains the poor UX. Now we're trying to improve since we're hitting a point where the previous solution is no longer good enough.

Users can currently press enter to continue upon entering some text into the URL text input. This isn't always very intuitive since there is a big blue button begging to be clicked right next to it. πŸ˜… This button doesn't quite do what one is expecting, so we want there to be a button for the folks who wouldn't intuitively go for the enter key that does exactly what 80%+ would expect from such a a button without reading its label.

Suggestions

I'm thinking that a good starting point would be to move the Visit Source button to an icon-only source button. Perhaps we can add it as an option to the perspective-selection button groun (where we can switch between Prose and Kanban views). But we're also open to a different design since I would understand if you argue it doesn't belong there since Kanban and Prose are both in-app perspectives and this Visit Source action would lead one out of the app and perhaps deserved a different level of emphasis for that reason.

For reference, you can see how GitHub approached a similar problem on page https://github.com/rasendubi/uniorg/blob/master/README.md?plain=1 in the bar with the Raw and Blame buttons.

This bar has Display the source blob and Display the rendered blob buttons as well. The Raw button does what we previously did with the Visit Source button: it leads one to a different place on the internet that serves as the source of content. The Display the source blob button just displays the source of the file in-app (never leaving the main app).

I'm not sure on whether we eventually 1) lead people out of our app to the source or if we 2) offer them a source-only representation in-app, but since we already handled option 1 with the Visit Source button, we can just model the behavior of GitHub's Raw button which does more or less the same thing. This approach keeps the implementation step easier as we already have the Visit Source bit implemented and would simply need to restyle it. Rendering the source in-app, however; may require us to design another perspective for that behavior that we should schedule as a follow-up task and complete at a later point in time.

TODO

Design source perspective

@siarhei

In Add a clear Follow URL CTA, we touched upon eventually providing an in-app source-only view which would allow folks to see and copy the source code conveniently.

As a user, I want to be able to quickly copy the source code for a given page.

Background

We currently enable this by allowing users to follow a link to the Raw representation (the exact version of the file that we read for parsing but this is technical detail so just ignore this). Going to the raw page, leads a user out of the app and the user has to copy the content by selecting it all (which is easy enough with keybindings) and copying it (also easy enough with keybindings).

Goal

We want there to be a perspective where the end-user can:

  • view the raw representation of the file without leaving the app
  • press a button to copy the raw representation (for future pasting)
  • press a button to leave the app and go-to the raw file at the authoritative source
  • optionally, press a button to leave the app and go-to the git repo that contains the file

    We just want an idea of what this could look like. We are likely not close to implementing anything like this yet.

TODO

Align home-page and reader logic

@vidbina

Right now, the home-page has been the thing we focused on, but we basically want the same behavior that we have on the home page on reader pages.

Handle SSR on the reader pages on par with the home-page and handle indexing on the reader pages on par with the home-page.

TODO

[66%] Linking aids

Linking is a little bit easier now, but we are still lacking some visual aids/cues that can make is considerably easier for readers.

DONE @tijan
- State "DONE"       from "TODO"       [2022-09-27 Tue 13:00] \\
  @tijan came to the rescue πŸ† with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/91

As a user, I want to be able to obtain the link to a heading by simply clicking it or by clicking some paragraph-icon (like ΒΆ) or link icon alongside it (let's start with towards the end of the heading text).

Clicking a headline link shoulf link to itself and update the URL to that specific heading.

Clicking the link icon (whether it is ΒΆ or something else like the link icon to the left of headings on GitHub README pages) should link to the heading by id.

Currently, I can only obtain this link by through the TOC. This is not very convenient when I am already scrolled to a certain part of the text and want to obtain some context by simply refering back to the ToC.

Link for a heading can be obtained by passing looking up the heading id in the headingIdToSlugIndex index on the document (type FDocument). See the TOC, where the lookup is being performed.

DONE

Highlight highlighted ToC entry and Prose entry when navigated to from URL id

@tijan
- State "DONE"       from "TODO"       [2022-09-28 Wed 21:45] \\
  Merging with 4d5dcbc3542b254fcff861ff5f14e1631202b13c

When we navigate to a given heading through clicking an entry in the TOC or by following a link such as DOMAIN.TLD/PAGE/#heading-id, where heading-id is the heading id, we want the active id to be highlighted in the view such that it is easier for the user to spot it.

TODO

Highlight all parent ToC entries

@tijan

Sometimes entries highlighted in the previous task are hidden because the parent may be collapsed. As a user, I want all parents highlighted until the topmost that isn't folded away. This should allow us to follow the trace when we are unfolding the ToC tree to find our intended entry.

πŸ’‘ It would be an idea to automatically unfold the tree to show the highlighted entry but this may be too disruptive if the user has folded the tree into a particular view for better overview. The idea of tracing the parents by higlighting them would be less disruptive to a reader's workflow.
TODO

Implement image rendering

@vidbina

Images in the document need to be rendered correctly. We have no images in this document yet to demonstrate this but this task is here to expand on this point.

TODO @vidbina

Upon sorting out a headline linking strategy, we can implement functioning anchors or buttons wherever we have links. The links listed in the document and in the TOC would need to be updated to facilitate linking.

DONE @vidbina
- State "DONE"       from "TODO"       [2022-09-23 Fri 17:47] \\
  The TOC now links through to the unique ids that we generate with the help of =nanoid= at the moment.

As an example for the later implementations.

TODO @vidbina
:COOKIE_DATA: checkbox recursive

Internal document links in Org can refer to a headline by

  • the headline text
  • the headline ID or by
  • the CUSTOM_ID

Implement the indexing logic to facilitate heading lookup by the tokens mentioned above.

TODO
:COOKIE_DATA: checkbox recursive

The headline links in Org texts are actually called fuzzy links and they fuzzy link values are derivatives of actual heading text.

The following points are copied from the subpoints below:

  • all starting asterisks are collapsed to a single asterisk
  • all brackets are escaped with \
  • all statistics cookies are removed
  • emojis remain in-tact

Observe that headline fuzzy links are formatted as follows: Verify how [this] heading with an asterisk* links*, where:

  • all starting asterisks are collapsed to a single asterisk
  • all brackets are escaped with \

Which means that we need to keep these changes in mind when indexing fuzzy links for headings.

Observe that headline fuzzy links are formatted as follows: Verify how heading with cookie links πŸͺ

  • all statistics cookies are removed
  • emojis remain in-tact
TODO @tijan

Link following in Prose view requires an index to be populated which is a blocking task.

TODO @tijan
TODO

Implement breadcrumbs

Upon navigating to a headline, we want a breadcrumb to indicate the level at which the document is being observed. This breadcrumb view can be used as a method of narrowing the Kanban view as well.

TODO

[0%] Handle basic invalid target cases more gracefully

UX

In the spirit of keeping things simple and not trusting any input, the user should expect some basic checks to prevent them from doing things that must not work.

TODO

Fix timeout error when loading large files

bugfix

Loading of files such as https://raw.githubusercontent.com/sachac/emacs-news/master/index.org currently times out with status 413 and the following logging output. 😰

This problem may be solveable by incrementally parsing larger pages and gradually streaming chunks of parsed content back to the client but I'm not sure how easy it would be to make this happen.

TODO

Check URL before routing

UX@tijan

Avoid people from shooting themselves in the foot by implementing basic heuristics to check that GitHub and GitLab links are raw links:

  • For GitLab that means:
    • matching https://gitlab.com/GROUP/REPO_PATH/-/raw/BRANCH/FILE_PATH
  • For GitHub that means:
    • matching https://raw.githubusercontent.com/USER/REPO/BRANCH/PATH
    • matching https://github.com/USER/REPO/raw/BRANCH/PATH
    • matching https://github.com/USER/REPO/blob/BRANCH/PATH?raw=true

In case the links are not raw, present an alert for now with the error. We'll show something nicer in the future when we have time.

TODO

Check that retrieved data is valid Org data

UX@vidbina

Right now we run HTML input through the Org parser if the URL was invalid and pointing towards a regular webpage. We should at the very least check if we can discern plaintext input from HTML input by checking the mimetypes in the response.

TODO

Setup themes

@tijan

[25%] Automated Testing

devtoolsqa
TODO

Add test stage to Vercel pipeline

@jamba

As a developer, I sometimes catch snapshot breakage when I'm trying to work on a new feature and this should be caught much earlier by the build pipelines.

Failures in test should not be discovered by developers but should be discovered by the build pipelines.

⚠️ The current =hack= branch now contains a gitlab CI pipeline that runs in parallel to the external Vercel job. Let's figure out if we:

1. delay the Vercel job until the test stage in the GitLab CI pipeline passes. That way we don't waste compute on Vercel if the test pipeline fails and we block deploying code that has failing tests. I don't know if this is possible since this job is an external one and perhaps it is hard to influence it.

2. If idea 1 is too tricky, we can setup the Vercel token inside of Gitlab CI and build and deploy the app directly from GitLab CI (as opposed to from Vercel through an external job).

Open to ideas here though especially compelling cons for either of the approaches.
TODO

Setup Playwright to dev tooling

@jamba

As a team, we need to have some tooling in place for E2E testing. Better to set this up sooner rather than later such that we can already start stubbing the first E2E cases to minimize the cost of manual click-through during development.

The first test case, should complete the following steps:

DONE

Setup coverage tracking

@vidbina

Setup Codecov for project because it is a FLOSS-friendly option so we'd happily give them our money.

See https://docs.codecov.com/docs/gitlab-tutorial for implementation details

TODO

Fix Codecov reporting

The https://app.codecov.io/gitlab/formation.tools:eng/proto-01 page is not listing our coverage (it is only listed by going to commit-specific pages). My hunch is that codecov is having some trouble cataloging our coverage data because of our merging workflow which heavily uses fast-forward merges.

See @vidbina Tweet to @codecov for a different phrasing of the issue as I observed it then.

Merging Workflow

We have a main branch which is deploys to production and a hack branch which deploys to staging.

We merge feature branches to hack (staging) and when this is done in some forge like GitLab (in our case) or GitHub, it a non-fast-forward merge into hack is made which will trigger a deployment to staging.

We maintain a single line between hack (staging) and main (production) so main is trailing hack until we decided that we can fast-forward its position. This means that we don't have unique revs for every event but often find our hack and main branches sharing the exact same point on the graph when everything is caught up.

Here is a visual breakdown:

Such a merge results into the following situation (note how the feature branch was merged into hack).

On a day-to-day basis, however; I don't always use GitLab's merge button to trigger merges but I may, start a WIP branch on my machine (actually called vidbina/merge.hack for me as I prefix all my merge-related branches with vidbina/merge. but WIP is shorter for the purpose of this description) starting from the upstream hack and then perform a non-fast-forward merge (like the forges do) locally. This workflow allows me to resolve conflicts more conveniently and run some tests/experiments thereafter.

I may throwaway these local WIP branches if I mess things up or want to approach the merge slightly differently.

Anyways, after confirming that the results are okay, I fast-foward merge (using git merge --ff) WIP into hack such that they share the same rev and then I push hack upstream.

After the staging build has been out there without too much trouble, we catch up main through a fast-forward merge.

TODO

Refile Merging Workflow out of the task

Once the task is done and considering that the merging workflow write-up is not too shabby, we should file it were devs can reference it more easily (like in the proto repo itself).

[0%] Auth

Currently evaluating Auth0, NextAuth, Clerk.dev and ory.sh and optimizing for best ease of use and minimal ops/management load.

TODO

Facilitate login-with-GitLab

TODO

Facilitate login-with-GitHub

TODO

Facilitate login-with-Google

We first need to made a decision on how we deal with non-git data.

[0%] Persistence

TODO

Implement commit-to-git

TODO

Implement commit-to-GitHub repo

TODO

Implement commit-to-GitLab repo

TODO

Implement non-git storage

TODO

Implement multiplayer session store

For collaborative editing (multiplayer mode) we will need to offer a storage overlay that facilitates ephemeral data (like collab session interactions) until these are solidified enough to convert into a change request/MR/PR by way a git commit.

Some of the options in consideration are replicache, liveblocks and Yjs (which is P2P but may still require some information overlay to coordinate session discovery).

TODO

Facilitate text authoring/editing

Just grabbed coffee with Marijn Haverbeke from ProseMirror and CodeMirror to explore how to approach this problem and whether to focus on collaborative editing right away.

ProseMirror offers OT-based collaborative editing. See https://marijnhaverbeke.nl/blog/collaborative-editing.html and https://prosemirror.net/docs/guide/#collab for relevant background information.

πŸ’‘ We can implement authoring/editing without auth or persistence by just allowing folks to copy/paste the source. Really bad UX but I'm just saying that we can implement the UX side of this experience without blocking it with back-end-ish concerns (auth and persistence/storage).
TODO

Adapt parsing options based on document input

@vidbina

The defaultOptions for the uniorg parser sets todoKeywords to TODO and DONE (see src) which is in line with Org's defaults. Custom TODO keywords that are defined through the TODO keyword or its variants TYP_TODO are not being honored and thus these keywords are parsed as "just text".

Examine how to make the parser options region dependent such that we can parse differently within limited scopes (for example parsing with certain keywords known for the scope of a section where these keywords are defined).

πŸ’‘ Perhaps we can study the streaming API of uniorg to figure out to which extend we can modify the options data structure and whether this mechanism implements some stack such that we can pop out of a given context in order to continue parsing of a document.

🦬 Yak Shaving/Background

Note that defaultOptions in parse-options.ts defines TODO and DONE as the default values for todoKeywords. According to a code-base grep, the only place to change the defaultOptions seems to be in the constructor of the Parser in parser.ts. The implementation in parser.ts doesn't seem to allow for updating of options during parsing.

TODO

Setup stories to point towards a few real raw Org files

@tijan

In order to simplify development and to avoid us having to switch between storybook tabs and dev page tabs, it would be nice to have a story for Prose and Board where the content is the raw README file that we are displaying on the home page right now.

DONE

Style fallback components

@tijan@assutech
- State "DONE"       from "TODO"       [2022-08-29 Mon 13:18] \\
  Merged with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/50
  • FallbackBlock, becomes <pre>
    • for example for tables
  • FallbackInline, becomes <code>
TODO

Build storyboards

  • for block
    • show example of the table (above)
    • show just some random code
  • for inline
    • show example of a link (in list above)
    • date
      • e.g.:
TODO

Render all block using the Fallback component

Currently many blocks remain unrendered. This should be resolved by rendering all block-like components in a Fallback component as a interim quick-fix.

DONE

Implements Date components

@tijan@assutech
- State "DONE"       from "TODO"       [2022-09-04 Sun 16:21] \\
  Just merged to mainline and added some notes in-source tagged to @tijan about restructuring the architecture to optimize DRY-ness of the setup.
  • also maybe button
  • inputs
    • time
    • date
DONE @tijan@assutech
- State "DONE"       from "TODO"       [2022-08-31 Wed 15:51] \\
  Merged into mainline a few days ago
  • maybe button
  • inputs
    • URL [+ label] [+ description]
DONE

Implement Tag component

@tijan@assutech
- State "DONE"       from "TODO"       [2022-08-31 Wed 15:51] \\
  Merged into mainline along with the Link component
DONE

Retrieve all top-level tasks

@vidbina
- State "DONE"       from "TODO"       [2022-09-04 Sun 16:22] \\
  Implemented and merged to mainline

In order to populate the Kanban view (where the first TODO may not be part of a first-level heading), we want to be able to extract all top-level tasks from a document.

For the following snippet snippet we can list 3 top-level tasks, being:

  • Task 1: Think about what to do
  • Something in the future, not sure yet
    • Task 2: Procrastinate
    • Task 3: Figure out what is in the future

Any of the subtasks should only be visible when navigating into that level by clicking on the parent card.

TODO

[75%] Add fonts

@tijan
:COOKIE_DATA: todo recursive

Not sure if the following links are relevant here:

TODO

Add emoji font

@tijan

Emojis are currently not consistently rendered. On my Linux box, I see a painful mess of just monochrome Wingdings-style emojis and the more colorful icons. Setting up an emoji font to ensure that everyone sees the same thing would improve consistency here and should be a small enough win for us to lock in.

DONE

[100%] Add text fonts

- State "DONE"       from "TODO"       [2022-09-28 Wed 13:57] \\
  Completed with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/93

On different platforms the selected serif font will look different. In order to control the quality of the reading experience, let's pick a decent open font to use here.

It is important the the fonts have varying weights such that we can still distinguish from bold text when it is presented inside of a Headline which is already presented in bold.

DONE

Add serif font

- State "DONE"       from "TODO"       [2022-09-28 Wed 13:56] \\
  Merged with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/93

Here are the options approved by @siarhei:

DONE

Add sans-serif font

- State "DONE"       from "TODO"       [2022-09-28 Wed 13:56] \\
  Merged with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/93

Here are the options approved by @siarhei:

TODO

[0%] Implement Section folding

TODO

Implement a Section component

@tijan

Implement a section component that uses something like headless's Disclosure to make the entire section foldable.

TODO

Render Section component in renderer

@tijan

Currently the renderer doesn't render anything for a section (we just return an empty list). We may need to think through what rendering sections will "mean" since there are a lot of nested sections in a doc, so once we start rendering sections we should expect some fireworks. πŸŽ†

Let's not worry about the explosions for now and just get this implemented! Once it's there it'll be my chore to figure out how to clean things up when merging the changes.

TODO

Fold completed sections Prose

@tijan

Minimize the noise in the docyment by folding all sections that are marked as DONE, same as Fold completed headings in ToC, above.

DONE

Fold completed headings in ToC

@tijan
- State "DONE"       from "TODO"       [2022-09-27 Tue 13:04] \\
  Implemented by @tijan with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/92

Minimize the noise in the ToC by folding all the headings that are marked as DONE (todoKeyword from type FHeading).

TODO

[66%] Implement Comments

DONE

Parse block comments

@vidbina
- State "DONE"       from "TODO"       [2022-09-25 Sun 20:57] \\
  Block comments are already being rendered albeit not in an optimal presentation.

I know what this means but essentially I want to be able to parse comments that are formatted as follows:

and appears as:

This is a comment
DONE

Parse inline comments

@vidbina
- State "DONE"       from "TODO"       [2022-09-25 Sun 21:10] \\
  Just verified that this works!

Readers need to be able to read inline comments such as the comment below:

This is an inline comment and should be visible on the page!

TODO

Stub Comment component

@tijanassutech

A comment component should open up as a drawer. Recorded a video (on YouTube) to demonstrate the capability.

See Figma frames comment open and comment closed for a design candidate from @siarhei (our designer).

My thinking is that a third pane (to the right), like an aside could function like a side-notes/aside/footnotes section where we can present the comments (however long they are). This would keep the flow of the main text clean. In this case, we would just show some line in the main Prose view to indicate that there is some content hidden away and then the user can click on some icon on that line (like the plus or expand icon) to open the full comment block in the third and right-most pane -- the aside.

DONE

Implement Source Blocks

@vidbina
- State "DONE"       from "TODO"       [2022-09-29 Thu 16:42] \\
  Implemented by rendering source in a fallback component since https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/96. Future work will be to render this in a slicker code component. Battle for another day tho.

As a reader, I want to see the contents of source blocks presented as code. We don't expect any highlighting or any nifty features but we simply want to be able to read the contents of the code blocks.

We should be able to read the source block in section Parse block comments which is currently hidden.

The following code block should be parsed and displayed correctly (open as plaintext just to see what is in the source block).

TODO

[50%] Implement Org tables

DONE

Stub table component

@tijan
- State "DONE"       from "TODO"       [2022-09-15 Thu 12:53] \\
  Looking good and merged into hack [2022-09-15 Thu 12:53].
DONE

Render table in fallback while we await a proper implementation

@vidbina
- State "DONE"       from "TODO"       [2022-09-29 Thu 16:53] \\
  Wrote this down and realized the table renders in a fallback-like style. Good enough for now.

We can't wait forever on this one, so meanwhile we need a way to render the following table in a fallback component and then touch it up into a slicker representation at a later point:

| key           | value                                                                                |
|---------------+--------------------------------------------------------------------------------------|
| name          | formation.tools                                                                      |
| type          | devtool                                                                              |
| stage         | prototype                                                                            |
| current niche | that Emacs crowd                                                                     |
| eventual goal | |
TODO

Parse tables

@vidbina

In order to display tables, we will need to parse them first. πŸ˜… Right now the underlying datastructure is completely unclear (although I do have some ideas). Implementing the parsing of tables will inform which attributes and properties we need to keep in consideration.

TODO

Integrate table component

@vidbina
TODO

[100%] Create tab button group to switch modes

@vidbina

For user to switch between the different modes/perspectives/views, let's just call it perspective, of a document we need something better than the toggle so let's define a button group that looks a bit like the Preview/Code tablist on pages such like https://tailwindui.com/components/application-ui/navigation/tabs to make it much clearer which view is currently active.

I guess perspective is clear enough as it provides a perspective on a document. /Modes/ may introduce confusion if we were to support vim-style modal navigating/editing at some point and /views/ already has a meaning with regards to interface development that could confuse us, as a team, or readers. /Perspective/ seems like a word that isn't used too often within the realm of text-editing and interface development to allow for easy misunderstanding -- I hope πŸ˜….
DONE

Spar with design-team on placement of such a control

- State "DONE"       from "TODO"       [2022-09-25 Sun 22:12] \\
  Top bar is the place for now, but we will revisit this at a later point to localize the perspective-switch in the perspective view itself.

Reach out to @siarhei and team to figure out where to best place the perspective control.

DONE

Update toggle-references in zero section

- State "DONE"       from "TODO"       [2022-09-25 Sun 22:13] \\
  Not very clear. We need to implement images to clarify this even more.

The opening section of this document points people towards the Toggle to switch views. Once's we've been able to rip that out, we can update the text to point viewers towards the new controls.

DONE

Introduce some padding in the ToC (left and right) to render it less crowded

@tijan
- State "DONE"       from "TODO"       [2022-09-14 Wed 11:15] \\
  Already implemented and live.

The current ToC is looking very cluttered. Let's show the TODO tag, the headline text (for majority of the width of the ToC pane) and then the tags. I'm also open to dropping the tags from the TOC just to keep things cleaner because I believe that this is very likely just too much information.

TODO

Fix undefined classes

@vidbinadebt

I did something stupid in the rush of dropping the build. When you study the classes you'll find a few classes named undefined littering the codebase. πŸ’© Clean this up!

TODO

Make Kanban columns scrollable

@purcy

Currently the entire Kanban view is y-scrolled as a whole but we want end-user to scroll individual columns for better control. Especially alter when we will want to facilitate DnD.

DONE

Clean up Kanban cards component

@vidbina
- State "DONE"       from "TODO"       [2022-09-07 Wed 21:53] \\
  More or less done, albeit scrappy πŸ˜…
  • The Kanban cards currently display an edit button and a deadline component. We want to hide the edit button and the deadline tag-like component and instead display the headline tags.
  • Kanban cards may need to provide some context by indicating which heading they belong to.
TODO

Collect all tags for a document and color-code them

@vidbina

For readability's sake, we want to color code all the tags such that the same value tags always appear in the same color. This should allow readers to visually associate related items provided that the readers/observers are capable of discerning the colors.

As a follow-up point, we should define a task to break inventorize which colors are fair to use from an a11y perspective in order to maximize the serviced audience by this feature.

TODO

Implement general wrapper for inline components

@tijandebt

Grep codebase for @tijan and you'll find some notes on our Tailwind usage and DRY-ness (DRY as in "don't repeat yourself"). Let's stub an InlineContainer and a BlockContainer which we can reuse without having to redefine borders, shadows, background colors and basic typography.

Spoke with @tijan about this and we can look at extracting classes the Tailwind way https://tailwindcss.com/docs/reusing-styles#extracting-classes-with-apply to minimize duplication.

For now we have a Block element that also exposes blockClasses in case we need block-like styling anywhere but this point is still outstanding for refactoring.

DONE

Fix ToC Disclosure animation

@tijan
- State "DONE"       from "TODO"       [2022-09-07 Wed 22:09] \\
  All handled by @tijan πŸ†πŸ₯³

When expanding/collapsing the ToC sections, the disclosures are not smoothly animated. The setup seems to be unaware of the changing height of the children and only effects a height-related change when the child elements are removed altogether which creates a bit of a visual shock as the space for the children instantly disappears from one moment to another.

I very likely lack the lingual skills to explains this very clearly, so just have a look in the storybook and then hop into my Google Meet room for us to talk it through for clarification.

πŸ’‘ I'm an idiot! Whatever animation I currently implemented there is horrible. Especially on the longer list displayed in the index, the ugliness is very pronounced. It all looked decent in the shorter lists in Storybook but now it's clear that we can remove whatever I did there.

It's so ugly it's technically criminal! My "engineering" license should be revoked for this one. πŸ˜… Good thing I don't do this for a living.
DONE

Implement list parsing

@vidbina
- State "DONE"       from "TODO"       [2022-09-05 Mon 22:33] \\
  Implemented with vidbina/parse-lists, ready for @tijan to work on the component.

List can be parsed as follows:

where you can use the following raw text samples:

  • unordered list
  • ordered list
  • description list
  • mixed list
πŸ’‘ All of the previously listed samples are verified to parse correctly in the parser spec (core/parser.test.ts).
DONE

Implement list component

@tijan
- State "DONE"       from "TODO"       [2022-09-14 Wed 11:42] \\
  Merged into mainline but the question of folding/unfolding lists is yet to be solved.

In order to render lists correctly, we need to stub out a List component that can handle the inputs demonstated in Implement list parsing.

TODO

Implement comment component

@tijan

INDEV Stub NarrowSidebar layout for reader view

@vidbina
- Note taken on [2022-09-05 Mon 22:30] \\
  Just wired up the home page to display the Engineering doc with branch vidbina/wire-up-layouts. Consider this a babystep to the end-goal.

Page reader[url] will be using the NarrowSidebar which is already demonstrated in a story for reference's sake. The following inputs are needed for the new layout:

  • a ReactNode that is wired up to handle typography and dark mode state
  • empty list for starters
  • empty list for starters
  • empty list for starters
  • static, hard-coded vals for starters

WIP [0%] Restyle reader entrypoint

@vidbina
- Note taken on [2022-09-05 Mon 22:32] \\
  Instead of creating a UI flow, we are improving and simply rendering the Engineering document at index. This is a quick-fix to buy us some time. πŸ˜…

Page /reader/index needs to be restyled to align stylistically with the main reader, otherwise users may be confused into thinking that these are different apps.

What basically has to happen:

  • browse from the Tailwind catalog which UI layouts are best suited for this view
  • copy Tailwind templates over into the codebase
  • delegate follow-up implementation
TODO

[33%] Dev Tooling

devtools
TODO

Setup Storybook to be Next Router aware

devtools
TODO

Upgrade Storybook

devtools

See the following snippet:

DONE

Remove base64encode from codebase or mock

fixdevtools
- State "DONE"       from "TODO"       [2022-09-29 Thu 16:40] \\
  Just merged into Hack as part of wrapping up https://staging.formation.tools/#implement-source-blocks (https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/96)

The use of base64encode Causes Buffer-related errors within storybooks and other more limited environments (iirc unit tests may be struggling with this as well but I could be wrong here) than the browser or Node.js:

We can either

  • refactor code to rip out the Buffer-depending logic from the views or
  • mock the base64encode calls to provide some behavior that Storybook can work with
TODO

Rename FObjectType to FObject

debt

For naming consistency, we should drop the Type prefix. I borrowed this from the reference implementation in uniorg but it is a bit more verbose than needed and actually confusing since I don't follow the pattern for ElementType in a consistent manner.

TODO

Fix unpackElementType to return correct fallbacks

debt

The function unpackElementType returns an incorrect fallback. It returns a e-typed fallback, which is reserved for Element types, for a GreaterElement element which should actually unpack into an E-typed fallback.

Furthermore, there are a bunch of elements for which this function just returns nothing (denoted by the empty list) instead of a fallback entry so a bunch of information goes missing.

TODO

Design DocumentPart

Note that every section can contain its own definition of title

TODO

Merge unpackElementType and convert into single function

debt

Perhaps we even remove unpackElementType in order to fully rely on convert which we may rename to unpack in its new role.

The reasons being:

  • there is quite of repetition between the two functions
  • convert effectively calls unpackElementType and this logic could sensibly be considered part of the same concern
  • converting into a Element alone may not be sufficient as elements tend to exist in the context of a document so we may have to convert an Element into a DocumentPart instead which can be a richer container for some helpful additional context for an Element to be understood correctly
    • a heading with a particular TODO state exists in the context of a section which outlines what that state means in the context of that section (the section may define the workflow)
    • footnote references don't exist without

Background

The components of this syntax can be divided into two classes: β€œobjects” and β€œelements”. To better understand these classes, consider the paragraph as a unit of measurement. Elements are syntactic components that exist at the same or greater scope than a paragraph, i.e. which could not be contained by a paragraph. Conversely, objects are syntactic components that exist with a smaller scope than a paragraph, and so can be contained within a paragraph.

Elements can be stratified into β€œheadings”, β€œsections”, β€œgreater elements”, and β€œlesser elements”, from broadest scope to narrowest. Along with objects, these sub-classes define categories of syntactic environments. Only headings, sections, property drawers, and planning lines are context-free1, 2, every other syntactic component only exists within specific environments. This is a core concept of the syntax.

Expanding on the stratification of elements, lesser elements are elements that cannot contain any other elements. As such, a paragraph is considered a lesser element. Greater elements can themselves contain greater elements or lesser elements. Sections contain both greater and lesser elements, and headings can contain a section and other headings.
DONE

Resolve GPL licensing issue

- State "DONE"       from "TODO"       [2022-09-07 Wed 21:53] \\
  It's GPL3 now. It's a prototype build so we'll have to get over it. The idea of keeping this GPL3 is a fun challenge as well.

At the moment, the front-end will have to be released under GPL3 as far as we understand. This represents a risk for the commercial interests of the project and thus it is important to consider how we design the system going forward. Some approaches would be to:

  • eliminate all GPL3 code and opt for more permisssive parsers if we can find any or write one ourselves
  • design the architecture of the overal system such that we have a clear strategy about what is GPL3 and what we can continue to develop as IP
TODO

Look into non-GPL3 parser availability

The uniorg is licensed under GPL3 and possible must be because it represents a translation (see GPL3 text for the definition of a translated program) of the org-element.el implementation in Emacs also under a GPL license. The alternative org-parser is also licensed as GPL3 so, apart from the effort to integrate, this would not be a valid option either.

TODO

License front-end under GPL3

If we cannot find a permissive parser, we will have to license the front-end project under GPL3.

TODO

Research how to deliver back-end logic as separate modules/services in Vercel

In order to avoid having to bundle everything into a GPL3 app, we can provide all base front-end logic in the app and then take the marketplace approach and offer integrations with forces, communication tools and other third-party services (or our own services) through server-side facilities.

TODO

Replace uuidv4 with uuid

WIP [14%] Replace Grommet for TailwindCSS

debt
TODO

Setup TailwindCSS

Switching from Grommet to Tailwind because:

  • the Grommet UI is a bit more playful than we need our app to look
  • we will be bringing in some dev help that is pretty fluent with Tailwind and we want to be efficient
  • TailwindUI provides paid component options that we can use to fast-track development of our UI
DONE

Setup Tailwind for Storybook

@assutech
- State "DONE"       from "TODO"       [2022-08-23 Tue 17:36] \\
  Sorted by @purcy on [2022-08-23 Tue] so we no longer have to worry about this. Future point would be to look into honoring Tailwind styles in the Storybook views as the current setup renders the components without any styling.

To simplify development of UI parts, we would like to be able to review components in Storybook. For this, we need to configure the project in order for npm run storybook.

Fix the configuration and provide links to the relevant documentation that explains the design choicese in that configuration.

Error

The following error is produced when running npm run storybook:

DONE

Explore if we need HeadlessUI

@vidbina
- State "DONE"       from "WIP"        [2022-08-23 Tue 17:37] \\
  We need HeadlessUI for the React components and some of the dynamic behavior associated with them (think Dialogs for Modals, Disclosures for accordion-like components, etc.). Some of the TailwindUI components seem to depend on this library and other third-party component libraries such as Flowbite 😠 (yeah, I'm not amused about this product ATM) seem to do the same.

In branch vidbina/setup-tailwind-headlessui, we are trying out incorporating HeadlessUI components in our setup. If this works, it may simplify building components in React that are based on top of TailwindCSS.

The HeadlessUI repository is pretty small, as in, only few components.

TODO

Explore what we need from Flowbite

@purcymarte

Flowbite has a UI blocks (based on TailwindCSS) that we can build on such as:

We have to explore to which extend we need this and can bring this in.

If we start a branch e.g.: setup-tailwind-flowbite, start it from vidbina/setup-tailwind.

TODO

Investigate how to set html/body-level classes at run-time

The Next.js docs outline in https://nextjs.org/docs/advanced-features/custom-document how we can set html and body level classes from within the pages/_document.tsx file with the caveat that these are server-side rendered and therefore cannot support dynamic behavior (for example the use of event handlers such as onClick).

The blog post https://smnh.me/add-class-to-body-tag-in-nextjs outlines some ways to deal with html/body-level dark-mode settings which are dynamic in the sense that they are change during run-time and will need to be updated. It isn't quite clear if the instructions here are relevant at this moment so we will need to investigate what the issue here is.

The reason why this information is relevant is because some of the TailwindUI components such as https://tailwindui.com/components/application-ui/application-shells/multi-column will require the following updates to our template:

The use of class bg-white is for non-dark-mode templates. Toggling dark mode will therefore require changes to the html and body classList at runtime in order to work adequately.

TODO

Decompose selected layout from components/Layout into its constituent parts

@vidbina has copied over a few Application UI/Application Shells from TailwindUI and has made the necessary translations from HTML to JSX just to have something that looks okay-ish as a starting point.

We now have to take these compositions, basically UI dumps, and then decompose them into the larger subparts such as (following are just an example):

  • SideBar which may contain
    • Menu which contains
      • MenuItem
    • UserProfileWidget
  • ContentPane
  • NavigationBar
    • Searchbar
    • Menu
    • UserProfileAvatar

Note that the example in components/Layout.tsx is not complete in that some of the animations are broken and state is lacking which is why menus or other state-dependent elements will not behave correctly. All of these capabilities will need to be implemented.

πŸ’‘ As a reference, look at component ToggleDarkMode which represents a basic TailwindCSS component which was translated from the TailwindUI code listing and then retrofitted to contain the correct images and to behave as expected by implementing event handling and by binding state into the component.
DONE

Implement ToC

- State "DONE"       from "TODO"       [2022-09-25 Sun 14:37] \\
  Basic TOC is implemented. Some improvements like linking are still pending but that is to be considered future work.
DONE

Stub ToC

@purcy
- State "DONE"       from "TODO"       [2022-08-23 Tue 17:39] \\
  @purcy stubbed this component today [2022-08-23 Tue] and I just integrated it so we should be good to go. We still want to incorporate HeadlessUI's Disclosure components in order to facilitate folding but this is a nice-to-have that we can address later. First we have to address some higher-level layouting concerns because I just plopped the ToC on a page without much though or any grace. πŸ˜…
DONE

Make ToC dark-mode aware

@purcy
- State "DONE"       from "TODO"       [2022-09-25 Sun 14:36] \\
  This issue is actually unresolved pending because we've refactored the TOC to show in dark-mode for better contrast during the light-mode. In the current form, we are not exposing the dark-mode functionality to end-users and will be recommending the use of a plugin like DarkReader to have a computed browser-wide dark-mode instead of one explicitly designed by us.

Switching between light and dark mode with the ToC rendered doesn't really change the text color. The ToC text should follow the classes for the main text which is altered during light/dark mode transitions. Not sure which TailwindCSS classes need to be set.

DONE

Incorporate Disclosure components to enable folding of levels

- State "DONE"       from              [2022-09-25 Sun 14:34] \\
  Already merged into mainline (work done by @tijan)
DONE
- State "DONE"       from              [2022-09-25 Sun 14:34] \\
  Implemented a naive approach (unique link ids per session)

See Design headline linking strategy for a characterization of the headline labelling problem. Once we've resolved this, we should roll an implementation for the ToC (and also the Prose but that is another concern).

TODO

Build Tailwind components for doc

  • components
    • app, for app ui
      • layout
        • TwoPaneLayout
      • DarkModeToggle
    • doc, for document ui
      • Datetime
      • Link
      • List
TODO

Remove Grommet-specific containers

We still use Col, Row, Main, MainContent and AppContainer from components/View.tsx

I'm upgraded this task (instead of a child of the [[*Setup TailwindCSS][Setup TailwindCSS]] task it is now a sibling thereof) since it doesn't block any of the other tasks. We can therefore move along on the Storybook or TailwindCSS concerns without this affecting us at all. Having it here may allow @purcy to prototype things quickly and then we do the translation afterwards so I'm deprioritizing this as well by moving it to the bottom of the task list.
TODO

Remove all Grommet artifacts

With Tailwind coming in, we should remove gradually remove Grommet. It isn't clear where all Grommet dependencies are, in terms of files that are Grommet-related, but we would need to identify them and then rid them from the dependency (remove Grommet dependency from core components) and also clean up the dependency manifest (packages.json).

TODO

Note that _app.tsx imported GlobalStyle from styles/global.js. We have already removed the import but the styles/global.js file is still in place and will need to be removed.

Research

TODO

[%] Study task dependencies

TODO

Read UOMF: Advanced Usage of Dependencies Between Tasks Using Org Mode by Karl Voit

Background

πŸ‘‹πŸΏ This is an Org file, that markup format associated to Org-mode the so-called killer feature of Emacs πŸƒπŸ™Š but that's not the point. I've hated Confluence, Notion and consorts 😠 with a passion for a while and was keen to have a way to collaborate through plaintext with folks. It's obviously unfair to ask folks to learn Emacs, so I figured building a tool that allows folks to read Org files is at least something I can do to stop my whining about Confluence, Notion, et. al in the "shut up, fix it" spirit. This is the scrappiest prototype that I could whip up to start flirting with the idea. 🌱

The cool thing is that such Org files can be presented differently to provide a better overview to you, the reader. We can look at them as prose, as a kanban board (which displays the entries in this document that are tasks), as an calendar (for a view of everything that has dates/times associated to them) and more.

The toggle top-right can be flipped to switch between prose and Kanban view on this doc. Yeah, it's a horrible UX but it was the quick-and-dirty thing that we could built in there. πŸ™ˆ

πŸ₯± TL;DR

The "big idea" πŸ’‘ here is that most of this is kept as plaintext and that a git repo is the conduit to keep track of your content and changes. For most users, this is not interesting so we can provide them a web tool that abstrats all this git and Org stuff away from them but will allow the other weirdo hackers like myself to just live in their editor of choice and not worry about context-switching between their editor and slow web-tools.

Many of us probably agree that git is a great way to collaborate and track versions of ever-changing text. We don't need our data stored somewhere in someone else's database and request permissions through clunky APIs to access it when you can just have it on a repo on your machine. We want to offer a an easy API and good utils to just do whetever the heck you want with the files on your box. Honestly, with the development in tooling recently, decent search or other capabilities should be easily serviceable by your own machine through different utils (some perhaps built in Rust). Our computers are definitely packing enough heat to handle all the compute needed to for a local knowledge base.

Depending on how poorly (or warmly) this is received, I will decide how to move forward, so that feedback is needed for me to know if this is a moderately sensible idea or totally ludicrous. πŸ˜… We're thinking about supporting collaborative editing in the future. I'm already setting up a coffee meet with the author of ProseMirror to explore this. Imagine being able to look at a prose, kanban or calendar diffs on the state of an Org-file when you come back from vacay πŸ–οΈ to get a quick sense of all the things that have changed. This is also on the wishlist.

So in the spirit of scrappy prototypes, the repo is online and GPL3 (because I'm using @rasendubi's uniorg)... Roast away! πŸ”₯

WIP Stack

  • TypeScript over JavaScript
  • React because of large eco-system (think: Storybook, Next.JS, etc.)
    • good xs to docs and examples
    • ability to deploy easily to Vercel

Infrastructure

file:gen/infra.overview.png

🧭️ Everything that is dotted and in red is on the way out! Things that are gray and dashed are ideas to likely be introduced Everything green and filled is live at the moment and everything else is just there but not highlighed to keep the diagram from being too cluttered.
- red and dotted :: deprecating
- green and filled :: live
- gray and dashed :: idea

file:gen/infra.overview.png

Vercel

We use Vercel as our compute provider because it is easy.

Prototype Application

The prototype app is our current drop-in point -- the one place where you end up for everything. Since we're in an early testing/validation stage, we are keeping this scrappy for now until we know what to really build.

APIs

We anticipate needing different APIs at a later stage for different functionality, e.g.:

  • integrations with forges data hosting platforms (like GitLab or GitHub)
  • integrations with auth services
  • integrations with NLP-services to solve QA tasks or offer text paragraphing features (like an automated variant of Reddit's ELI5)
  • integrations with chat platforms (like Slack), could be mounted under /api/v1/communications/*

All these APIs and integrations can be separate Vercel apps just to keep things more cleanly isolated. That way we can decouple deployments of our different services.

Concepts
πŸ’‘ This is just a concept section of APIs that we can implement at some point. As this section is commented, the dot diagram code below is not rendered into the infra overview at the top of this section.

Proxy/Rewrite

We started off by directing all of our domains to a rewrite/proxy service where we resolved paths. This allowed us to gradually build out our stack with different services that we can mount under arbitrary paths on our target domains but was challenging to develop with. On , I had a call with Hassan about DX and we decided to move all rewrites to the prototype project directly.

By using the rewrites facility in next.config.js, we should be able to reference environment variables which can be defined using .env.local files during development (and pointed towards any arbitrary destination) and through the Environment Variables platform facility on a per-branch level in the Vercel platform. This should provide us the needed control and a tenable DX.

Some of the rewrites that we conduct are rewrites to external services. The last defined rewrite and therefore our fallback path is the prototype app for now.

Eventually, we can use the rewrites/proxy mechanism to mount all of our own services and 3rd-party services under friendlier paths, for example (and bear with me, this is just me musing):

  • /api/v1/auth/* for auth services
  • /api/v1/data/* to for data services,
    • /api/v1/data/forge/ for forge integrations (to GitHub, GitLab, Gitea, etc.)
  • /api/v1/nlp/* for NLP services
  • /api/market for 3rd-party marketplace services

DNS

We have the following domain names:

  • formation.tools
  • forto.dev
  • f7n.io

We use the Rewrite (Proxy) facility from Vercel to manage these domains through a single Vercel project -- a convenient drop-in point to serve as our nice traffic-direction point.

Plausible

We use Plausible for analytics. On the Plausible end, only two artifacts are relevant:

  • the event API and
  • the Plausible client script

In order to minimize the potential for loosing data due to ad-blockers, we proxy our own paths onwards to Plausible as suggested in the Plausible documentation on dealing with missing data (link captured on ).

DONE

Stub Vercel project that provides a rough outline of this infrastructure

- State "DONE"       from "TODO"       [2022-09-14 Wed 11:19] \\
  Resolved with the help of Sebastian from Vercel. Core point was that we only defined a non-root rewrite =/:match*= instead of defining a root rewrite =/=.

Stubbed vercel-config but it is broken. We debugged this together to no avail and I just filed Case #00097879 with Vercel Customer Support.

TODO

Proxy Plausible paths

In order to ensure that we hit our analytics back-end, we can define some rewrite rules towards Plausible.

TODO

Parameterize Plausible magic strings

@vidbina

The Plausible config is hard-coded in the codebase and pollutes our analytics because it also records localhost or staging traffic.

TODO

Think about how to solve payments

- State "TODO"       from "TODO"       [2022-09-14 Wed 11:20] \\
  Not actively tackling this since we are too early stage to worry about payments at the moment. We're focusing on the big picture bits for now.

Wants:

  • Reporting: Single transaction overview at end-of-the-month to simplify accounting
  • Invoicing: Capability to automatically issue PDF invoices to customers in-platform
    • nice-to-have support to direct these invoices to a billing address to simplify processing for companies

Decision

DONE

Develop marketplace concept

product

This is a product concern so I'm signing this over to the product scope which is in the shared intel repo.

Principles

Use these as tools to guide you through every engineering choice.

Minimize Toil

We are a small team and probably always will be (kind of like a small formation of synchronized dancers 🩰 or a well-tuned squad of special forces operatives). Because of our size, we have to build solutions that minimize manual effort on remedial tasks.

Refer to [[https://sre.google/sre-book/eliminating-toil/][Eliminating Toil [Google SRE Handbook]​]].

Design First, Implement After

For anything we attempt to engineer, we make it a habit to design first by drafting a document that outlines the problem, potential solution candidates with reasons for why the candidates are preferred or not. This document is already an example of such a text because we have been "thinking out" ideas in this doc. πŸ˜…

πŸ’‘ Really try to not initiate implementation work without first going through a design effort!

πŸ‘¨πŸΏβ€πŸ’» License to Hack: Sometimes a quick and dirty prototype is arguably part of the "design effort" and maybe even a required step in order to understand a problem and related constraints (for example tooling constraints) better. Somethings are simply hard to sketch out in advance or phrase in clear terms, so, in the spirit of show/hack-and-tell, feel free to whip up the occasional prototype to serve as "documentation" to communicate a design.

When engineering software, start with a README in a repository and document the problem and some solution candidates.

The following is an example of a minimal README for an imaginary project.

#+begin_src org
,#+TITLE: Org API

,* Problem

- Org is a rather extensive (complex) standard
- Changing source on a per-character basis is leaves room for "breaking the format" (e.g.: when necessary punctionation is altered
- From a user-perspective, Org operations happen on an Org-primitives (e.g.: we add text to a heading, we remove a link, we boldface a phrase of text) which a renderer or editor will need to understand

,* Solution Options

- find Org parser that is usable with JavaScript (because we're building something for the browser)
  - org-parser (ClojureScript) based on instaparse (ClojureScript)
    - built with instaparse (seems like a mature parsing toolbox)
    - written in ClojureScript (should be compileable to JavaScript and reuseable in other apps, need to explore this)
    - implemented through expression of *BNF grammar* (a practical abstraction for the parsing problem)
    - developed by 200ok.ch (respected consulting firm with plenty of Org and ClojureScript experience)
- implement own Org parser
  - from scratch (too expensive, not core business, little value to be added to project)
  - in unified ecosystem tooling (implementation of large Org syntax would introduce a lot of work that doesn't directly add value to the project)
#+end_src

Keep single sources of truth

When any topic needs to be documented, find the one place where most will think to look for this information.

For a software project, this will be the git repositories or a more general architecture document in a dedicated knowledge base or another repository.

For a product idea, this will be the place were such ideation typically happens like a product management or project management tool.

Try to avoid replicating information because synchronizing multiple sources is a hard chore. Think: DRY!

Be creative and brief in documenting

We don't all process information the same way. When documenting information consider the following modalities:

  • text
  • image (for example: diagrams, graphs and illustrations/sketches)
  • video

Write in a manner that feels natural to hear such that folks using TTS (text-to-speech) tools can follow along.

Since search mostly functions through text, try to accompany every non-text artefact (e.g.: video, diagram, photograph, audio snippet, etc.) with human-readable description.

Since we don't all parse information best through text alone, try to provide diagrams when possible for those who need visual cues. Complement your diagram entries with sufficiently descriptive alt-texts or summaries such that the idea within the diagram is communicated even if the summary is read without awareness of the image. Do the same for any non-text artefact that you may add to a document.

With complex diagrams, it may be easier to define high level relationships in the diagram and then follow them with a bit of elaboration up to an arbitrary level. In the spirit of time, we can attempt to keep these descriptions brief. We're hoping that someday there will be better tooling to help us convert such artefacts to prose for the folks who will need it.

Example Org files