This place is where we track our engineering tasks and notes. We are dogfooding 🌭🐕 at the moment and working towards a point where this tool is friendly enough for others to use. Cleaning up the UI and providing some basic edit capability are some steps towards that goal.
TL;DR: Effectively this page is just a scrappy plaintext reader (of Org file to be precise, for the non-Org-mode readers on our team). You should be able to edit notes and tasks in a Confluence/Notion-like manner soon while simultaneously allowing teammembers on your team to interface with this knowledge-base and project-management setup through git and thus keep a full history of your project-management and knowledge-base on their machine. I, for example, use this file inside of Emacs but am well aware that it would be torture to ask my fellow devs, let alone non-engineering colleagues, to fire up Emacs to get any work done. 😅 Using git underneat will allow us to facilitate nifty diffing features, time-machine like features or proper roll-backs when shit hits the fan. *Git is a helluva tool* and its time to expose more folks to the magic (magit users will appreciate the pun here) of git without exposing them to git mechanics.
Scroll down to section Background for more information and use the top-right perspective-switch to switch between Prose and Kanban view on this document.
🙊 It's very scrappy but it's a little easier than sending ox-publish HTML or LateX exports over the wire (Slack or mail) or by asking folks to learn Emacs 🙊😅.
Offer something that is way more fun/pleasant to use than Confluence and which is more tolerable than Notion. We want to bring the joy back in collaboration and are approaching this from a developer-first angle.
Knowledge bases and project management tools are currently a source of frustration 😡 and devs are largely keeping notes in separate silos: in different note-taking tools or just on folders on their local machines for easy access simply because the current tools are clunky.
There is a much intel lost which hurts organizations! 📉
Removing the barriers to share knowledge helps teams keep relevant intel together 🧠 and should minimize knowledge loss when people are out-of-office or move on to other challenges. People should ideally ideate in their knowledge management/project-management tools and not be forced to document ideas there as a rote task after conception. We're out to break the barriers to make documenting and ideating together much easier -- challenge accepted. 😅
- Bring play back into collaboration. Collaboration should be like dancing 💃 or playing a game of ball 🏀.
There should be a sense of flow that one can enjoy while at it. We are figuring out how to bring that back into this category of tooling.
Sometimes this means less rigor in the way we work together. There are no fixed lines within which to color! You can start with a basic text file and then gradually start jotting down ideas which grow to become tasks and then can get split up as we need in a very organic manner. Like claymolding -- but at a deskjob and without the sticky fingers.
- Facilitate people in their optimal modality as we don't all consume information very well in the same way.
Some of us need long texts to absorb ideas, some need to hear a voice (audio) and many we need visual/spatial representations to process information. Let's empower teams by providing them the modalities that make sense for whatever they're trying to do as the same data can be presented in different ways so we're doing that.
You should be able to consume a knowledge base while strolling through the park listening to it podcast-style. You should be able to look at a document with tasks in a Kanban-like perspective or as a task dependency graph. We want knowledge-bases and project-management tooling to be more like a Rubik's cubes that you can turn around at study from different perspectives to improve your understanding. 🧩
👨🏿💻 If you're a dev, we'll make it dead-easy to collaborate through the tools you already know very well like plaintext and git, so those notes of yours have no excuse to stay just on your box. Adding them to the knowledge base should be easier than stubbing a prototype and should happen with the toolbox within 🧰 which you play to your strengths.
- Simplify how we work together by introducing some text assist capabilities for conveniences we haven't had access to before.
When engineers draft complex technical texts, because this comes natural to them, we can utilize language models to attempt to paraphrase them for more junior, or perhaps, non-technical team members. Think Reddit's ELI5 r/ExplainlikeImFive or Wired's One Concept in 5 Levels of Difficulty.
Relieving folks from having to write for multiple audiences should lower the barrier for team members to enter their thoughts as quickly as they have them 💡. Less curation, more ping-ponging 🏓, as in, bouncing ideas back and forth but maybe also actual ping-pong if much time is being spared by bringing that flow back.
This is the scrappy app (scr-app 🤣) that we are using to validate some early ideas.
⚒️ Mason's code: Mark tasks as =DONE= only when they are merged into branch =hack=.
- State "DONE" from "TODO" [2022-09-14 Wed 11:40] \\ Already live!
Based on HN input by moeffju, we do readers a service by not displaying non-functional anchors and nudging them into trying to use them and getting lead astray. This is totally on-point so, let's remove all anchors until we sort out the link following.
- State "DONE" from "TODO" [2022-09-25 Sun 21:01] \\ Done by @tijan and already merged. Had something to do with me resolving a merge to use a element instead of divs. 🤦🏿
The TOC is broken in Storybook and this Stack Overflow thread may provide some explanation of the issue at hand. Let's either fix the Storybook or figure out of this case is not testable in Storybook and then just make a note of this issues such that future developers know not to attempt testing click-through inside of Storybook.
- State "DONE" from "TODO" [2022-09-25 Sun 21:03] \\ Mark headline linking as done for now.
Headlines ids can not be randomly generated on a per-session basis (to map between ToC entries and headlines) as these id are not guaranteed to be the same during another session and therefore not shareable.
Bookmarked links need to survive non-destructive diffs (when just new content is added).
If the headline is removed then notify the user that the link has been removed at a given version, timepoint
$t$, and provide an option to navigate to state
$t$ to view the headline anyways
- id must contain at least 1 character
- id must be unique within the entire document
- id must not contain ASCII whitespace
- are there other mechanisms to define/mark/reference entities?
Rough outline how we will deal with headline identifiers.
This is the scenario where a headline is completely created within our tool.
⚠️ Any headline created in another tool is not guaranteed to satisfy the characteristics of a headline populated by us so, we have to be graceful in how we handle headlines. 😅
- If headlines are created within the tool:
- populate a unique
CUSTOM_IDthat can will always resolve to the headline as long as the headline or the
CUSTOM_IDis not removed (see Internal Links)
- populate a unique
- If headlines are renamed through the tool
CUSTOM_IDif already present
CUSTOM_IDmatching what may have been considered the previous headline by either of the following methods (first one wins):
- dedicated target, https://orgmode.org/manual/Internal-Links.html
- naive heading-to-id conversion logic
This operation does not mutate the document in any form.
CUSTOM_IDis defined, use this value
- If radio target (see Org Manual) is set for the heading, use this value
- Iterate over all headings and use naive heading-to-id conversion logic to identify headline
- if found, index this link for this particular document version
With indexing we keep track of headline identifiers throughout the lifetime of a document. This indexing should enable us to provide guidance to the user when a link is followed to a headline that may no longer be there or may have been modified (for example when a CUSTOMID is defined for a heading that was previously referenced through the id obtained by the heading-to-id conversion logic).
Org implements its own <<<heading-to-id conversion logic>>> that roughly works as follows:
org-store-linkcreates a link that can be inserted with function
Controls whether to create/use an
IDproperty for the link
See https://emacs.stackexchange.com/questions/51755/org-mode-link-files-with-ids-and-not-filenames for some relevant background information on how headline linking works in Emacs Org mode.
uuid, the default
- State "DONE" from "TODO" [2022-09-25 Sun 21:02] \\ Mark link following on headline links as done.
Following links is a core feature of any document. Let's provide some basic implementation. In the long-tail, we will need to keep track of heading references over the lifetime of a document just to ensure that we can provide useful guidance on how to resolve links (for example, inform a user to go back to a previous version of a document).
- State "DONE" from "TODO" [2022-09-23 Fri 17:37] \\ Index can be populated but we are populating it at the end of parsing instead of populating it on an ongoing basis. The reason for this is that I simply couldn't figure it out fast enough. Recursing over a document leaves the problem of sharing some document-global state to verify that human-friendly slugs do not collide. Trying to do this with map/reduce like mechanisms is problematic because the index state is copied along and there is a change that competing branches settle on the same slugs, allowing for collisions. Sleep deprivation and tunnel-vision are almost certainly in the mix here, so after a chat with @stefano we opted for the dead-simple approach of generating unique ids for now and then indexing them post-parse. This sucks because we now have to figure out how to lookup the human-friendly slugs in the rendering step. This will require us to pass some document-local state along to all the render functions. Just ideas. Maybe this is all BS and we'll find a better design after we've enjoyed quality sleep. 🤷🏿♂️
Links such as
[[*Headline]] should basically work and resolve to the first match. So if there are multiple headlines with the same text, the first one is always the one reached by such a link.
We can link to a headline by providing the headline text sans the keyword, statistics cookie and tags. Observe list item 1 in the snippet below that demonstrates a valid link to the headline defined at the bottom of the snippet.
- remove all statistics cookies
- collapse multiple whitespace characters to single character
- State "DONE" from "WIP" [2022-09-25 Sun 20:37] \\ Done with latest merge to hack (see https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/88) - State "WIP" from "TODO" [2022-09-23 Fri 17:43] \\ We are currently generating these ids through a helper function which runs through the flat list of headings and produces all slugs that we need. In an ideal world, I imagine populating an index while the document is being transformed (to our internal representation). Fundamentally, I'm torn on whether this information belongs in the internal representation of the headings. Part of me says that slugs, ids and linkTexts belong in the heading while another part says that we should just keep these in indices and provide a fast reference to the data (the heading).
Multiple headings which resolve to the same plaintext, should be numbered such that the id uniqueness constraints remains honored (a given id may only exist once).
We cannot use PRNG-generated heading ids because such ids will be different between sessions.
- State "DONE" from "TODO" [2022-09-23 Fri 17:46] \\ Consider the scrappy part done!
The availability of unique ids for every parsed headline may simplify the implementation of linking. Within a session we can just use the session-scoped unique ids.
These ids will not be stored as data as they are only meant to be used within the intermediate representation to provide us quick references that are "guaranteed" to not collide within a session scope.
With such unique ids, we simplify the implementation of linking as we can just implement a lookup table that points headline links,
ID links or other linking methods to a single reference.
We'll stub a scrappy implementation and will optimize later. 😅
Remove emphasis from Visit Source button and add a clear Follow URL CTA (Call-to-Action).
As a user, I press the Visit Source button expecting to be lead to the URL entered inside of the URL input box. This is a usability problem. We need a strong CTA near the URL input box that does exactly what a typical end-user would expect: lead one to the page that matches the entered URL.
💡 Try to enter
https://gitlab.com/formation.tools/eng/proto-01/-/raw/hack/README.org into the URL input control to see what we mean. When pressing the blue button it only navigates to the source of the current page. The lack of a connection between that prominent button and the input text is a bit confusing. Following the entered text is only possibly by pressing
- leads a user out of the app to the authoritative source on the internet
- leads the user to a in-app rendering of the URL that was provided (core use-case of the app)
💡 The URL input textbox will very likely disappear once we allow people to login with their own GitHub and GitLab accounts. At that point, users will just navigate to the files they want through the sidebar menu. Since we're not there yet, the URL input box is an interim solution for now so it doesn't have to be perfect but should be at least "usable enough" -- as in, not confusing even the people who made it. We improvised input bar without the guidance from design so this explains the poor UX. Now we're trying to improve since we're hitting a point where the previous solution is no longer good enough.
Users can currently press enter to continue upon entering some text into the URL text input. This isn't always very intuitive since there is a big blue button begging to be clicked right next to it. 😅 This button doesn't quite do what one is expecting, so we want there to be a button for the folks who wouldn't intuitively go for the enter key that does exactly what 80%+ would expect from such a a button without reading its label.
I'm thinking that a good starting point would be to move the Visit Source button to an icon-only source button. Perhaps we can add it as an option to the perspective-selection button groun (where we can switch between Prose and Kanban views). But we're also open to a different design since I would understand if you argue it doesn't belong there since Kanban and Prose are both in-app perspectives and this Visit Source action would lead one out of the app and perhaps deserved a different level of emphasis for that reason.
For reference, you can see how GitHub approached a similar problem on page https://github.com/rasendubi/uniorg/blob/master/README.md?plain=1 in the bar with the Raw and Blame buttons.
This bar has Display the source blob and Display the rendered blob buttons as well. The Raw button does what we previously did with the Visit Source button: it leads one to a different place on the internet that serves as the source of content. The Display the source blob button just displays the source of the file in-app (never leaving the main app).
I'm not sure on whether we eventually 1) lead people out of our app to the source or if we 2) offer them a source-only representation in-app, but since we already handled option 1 with the Visit Source button, we can just model the behavior of GitHub's Raw button which does more or less the same thing. This approach keeps the implementation step easier as we already have the Visit Source bit implemented and would simply need to restyle it. Rendering the source in-app, however; may require us to design another perspective for that behavior that we should schedule as a follow-up task and complete at a later point in time.
In Add a clear Follow URL CTA, we touched upon eventually providing an in-app source-only view which would allow folks to see and copy the source code conveniently.
As a user, I want to be able to quickly copy the source code for a given page.
We currently enable this by allowing users to follow a link to the Raw representation (the exact version of the file that we read for parsing but this is technical detail so just ignore this). Going to the raw page, leads a user out of the app and the user has to copy the content by selecting it all (which is easy enough with keybindings) and copying it (also easy enough with keybindings).
We want there to be a perspective where the end-user can:
- view the raw representation of the file without leaving the app
- press a button to copy the raw representation (for future pasting)
- press a button to leave the app and go-to the raw file at the authoritative source
- optionally, press a button to leave the app and go-to the git repo that contains the file
We just want an idea of what this could look like. We are likely not close to implementing anything like this yet.
Right now, the home-page has been the thing we focused on, but we basically want the same behavior that we have on the home page on reader pages.
Handle SSR on the reader pages on par with the home-page and handle indexing on the reader pages on par with the home-page.
Linking is a little bit easier now, but we are still lacking some visual aids/cues that can make is considerably easier for readers.
- State "DONE" from "TODO" [2022-09-27 Tue 13:00] \\ @tijan came to the rescue 🏆 with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/91
As a user, I want to be able to obtain the link to a heading by simply clicking it or by clicking some paragraph-icon (like ¶) or link icon alongside it (let's start with towards the end of the heading text).
Clicking a headline link shoulf link to itself and update the URL to that specific heading.
Clicking the link icon (whether it is ¶ or something else like the link icon to the left of headings on GitHub README pages) should link to the heading by id.
Currently, I can only obtain this link by through the TOC. This is not very convenient when I am already scrolled to a certain part of the text and want to obtain some context by simply refering back to the ToC.
Link for a heading can be obtained by passing looking up the heading id in the
headingIdToSlugIndex index on the document (type
FDocument). See the TOC, where the lookup is being performed.
- State "DONE" from "TODO" [2022-09-28 Wed 21:45] \\ Merging with 4d5dcbc3542b254fcff861ff5f14e1631202b13c
When we navigate to a given heading through clicking an entry in the TOC or by following a link such as
heading-id is the heading id, we want the active id to be highlighted in the view such that it is easier for the user to spot it.
Sometimes entries highlighted in the previous task are hidden because the parent may be collapsed. As a user, I want all parents highlighted until the topmost that isn't folded away. This should allow us to follow the trace when we are unfolding the ToC tree to find our intended entry.
💡 It would be an idea to automatically unfold the tree to show the highlighted entry but this may be too disruptive if the user has folded the tree into a particular view for better overview. The idea of tracing the parents by higlighting them would be less disruptive to a reader's workflow.
Images in the document need to be rendered correctly. We have no images in this document yet to demonstrate this but this task is here to expand on this point.
Upon sorting out a headline linking strategy, we can implement functioning anchors or buttons wherever we have links. The links listed in the document and in the TOC would need to be updated to facilitate linking.
- State "DONE" from "TODO" [2022-09-23 Fri 17:47] \\ The TOC now links through to the unique ids that we generate with the help of =nanoid= at the moment.
As an example for the later implementations.
:COOKIE_DATA: checkbox recursive
Internal document links in Org can refer to a headline by
- the headline text
- the headline
Implement the indexing logic to facilitate heading lookup by the tokens mentioned above.
:COOKIE_DATA: checkbox recursive
The headline links in Org texts are actually called fuzzy links and they fuzzy link values are derivatives of actual heading text.
The following points are copied from the subpoints below:
- all starting asterisks are collapsed to a single asterisk
- all brackets are escaped with
- all statistics cookies are removed
- emojis remain in-tact
Observe that headline fuzzy links are formatted as follows: Verify how [this] heading with an asterisk* links*, where:
- all starting asterisks are collapsed to a single asterisk
- all brackets are escaped with
Which means that we need to keep these changes in mind when indexing fuzzy links for headings.
Observe that headline fuzzy links are formatted as follows: Verify how heading with cookie links 🍪
- all statistics cookies are removed
- emojis remain in-tact
Link following in Prose view requires an index to be populated which is a blocking task.
Upon navigating to a headline, we want a breadcrumb to indicate the level at which the document is being observed. This breadcrumb view can be used as a method of narrowing the Kanban view as well.
In the spirit of keeping things simple and not trusting any input, the user should expect some basic checks to prevent them from doing things that must not work.
Loading of files such as https://raw.githubusercontent.com/sachac/emacs-news/master/index.org currently times out with status 413 and the following logging output. 😰
This problem may be solveable by incrementally parsing larger pages and gradually streaming chunks of parsed content back to the client but I'm not sure how easy it would be to make this happen.
Avoid people from shooting themselves in the foot by implementing basic heuristics to check that GitHub and GitLab links are raw links:
- For GitLab that means:
- For GitHub that means:
In case the links are not raw, present an alert for now with the error. We'll show something nicer in the future when we have time.
Right now we run HTML input through the Org parser if the URL was invalid and pointing towards a regular webpage. We should at the very least check if we can discern plaintext input from HTML input by checking the mimetypes in the response.
As a developer, I sometimes catch snapshot breakage when I'm trying to work on a new feature and this should be caught much earlier by the build pipelines.
Failures in test should not be discovered by developers but should be discovered by the build pipelines.
⚠️ The current =hack= branch now contains a gitlab CI pipeline that runs in parallel to the external Vercel job. Let's figure out if we: 1. delay the Vercel job until the test stage in the GitLab CI pipeline passes. That way we don't waste compute on Vercel if the test pipeline fails and we block deploying code that has failing tests. I don't know if this is possible since this job is an external one and perhaps it is hard to influence it. 2. If idea 1 is too tricky, we can setup the Vercel token inside of Gitlab CI and build and deploy the app directly from GitLab CI (as opposed to from Vercel through an external job). Open to ideas here though especially compelling cons for either of the approaches.
As a team, we need to have some tooling in place for E2E testing. Better to set this up sooner rather than later such that we can already start stubbing the first E2E cases to minimize the cost of manual click-through during development.
The first test case, should complete the following steps:
- navigate to the home page
- enter a https://gitlab.com/formation.tools/eng/proto-01/-/raw/hack/README.org into the URL input and submit
- wait for the new page to get rendered
- click on Resources item in TOC (table of contents)
- verify that prose view scrolls to the Resources section
Setup Codecov for project because it is a FLOSS-friendly option so we'd happily give them our money.
See https://docs.codecov.com/docs/gitlab-tutorial for implementation details
The https://app.codecov.io/gitlab/formation.tools:eng/proto-01 page is not listing our coverage (it is only listed by going to commit-specific pages). My hunch is that codecov is having some trouble cataloging our coverage data because of our merging workflow which heavily uses fast-forward merges.
See @vidbina Tweet to @codecov for a different phrasing of the issue as I observed it then.
We have a
main branch which is deploys to production and a
hack branch which deploys to staging.
We merge feature branches to
hack (staging) and when this is done in some forge like GitLab (in our case) or GitHub, it a non-fast-forward merge into
hack is made which will trigger a deployment to staging.
We maintain a single line between
hack (staging) and
main (production) so
main is trailing hack until we decided that we can fast-forward its position. This means that we don't have unique revs for every event but often find our
main branches sharing the exact same point on the graph when everything is caught up.
Here is a visual breakdown:
Such a merge results into the following situation (note how the feature branch was merged into
On a day-to-day basis, however; I don't always use GitLab's merge button to trigger merges but I may, start a
WIP branch on my machine (actually called
vidbina/merge.hack for me as I prefix all my merge-related branches with
WIP is shorter for the purpose of this description) starting from the upstream
hack and then perform a non-fast-forward merge (like the forges do) locally. This workflow allows me to resolve conflicts more conveniently and run some tests/experiments thereafter.
I may throwaway these local
WIP branches if I mess things up or want to approach the merge slightly differently.
Anyways, after confirming that the results are okay, I fast-foward merge (using
git merge --ff)
hack such that they share the same rev and then I push
After the staging build has been out there without too much trouble, we catch up
main through a fast-forward merge.
Once the task is done and considering that the merging workflow write-up is not too shabby, we should file it were devs can reference it more easily (like in the proto repo itself).
Currently evaluating Auth0, NextAuth, Clerk.dev and ory.sh and optimizing for best ease of use and minimal ops/management load.
We first need to made a decision on how we deal with non-git data.
For collaborative editing (multiplayer mode) we will need to offer a storage overlay that facilitates ephemeral data (like collab session interactions) until these are solidified enough to convert into a change request/MR/PR by way a git commit.
Some of the options in consideration are replicache, liveblocks and Yjs (which is P2P but may still require some information overlay to coordinate session discovery).
Just grabbed coffee with Marijn Haverbeke from ProseMirror and CodeMirror to explore how to approach this problem and whether to focus on collaborative editing right away.
ProseMirror offers OT-based collaborative editing. See https://marijnhaverbeke.nl/blog/collaborative-editing.html and https://prosemirror.net/docs/guide/#collab for relevant background information.
💡 We can implement authoring/editing without auth or persistence by just allowing folks to copy/paste the source. Really bad UX but I'm just saying that we can implement the UX side of this experience without blocking it with back-end-ish concerns (auth and persistence/storage).
defaultOptions for the uniorg parser sets
DONE (see src) which is in line with Org's defaults. Custom TODO keywords that are defined through the
TODO keyword or its variants
TYP_TODO are not being honored and thus these keywords are parsed as "just text".
Examine how to make the parser options region dependent such that we can parse differently within limited scopes (for example parsing with certain keywords known for the scope of a section where these keywords are defined).
💡 Perhaps we can study the streaming API of uniorg to figure out to which extend we can modify the options data structure and whether this mechanism implements some stack such that we can pop out of a given context in order to continue parsing of a document.
Note that defaultOptions in parse-options.ts defines TODO and DONE as the default values for
todoKeywords. According to a code-base grep, the only place to change the defaultOptions seems to be in the constructor of the Parser in parser.ts. The implementation in parser.ts doesn't seem to allow for updating of
options during parsing.
In order to simplify development and to avoid us having to switch between storybook tabs and dev page tabs, it would be nice to have a story for Prose and Board where the content is the raw README file that we are displaying on the home page right now.
- State "DONE" from "TODO" [2022-08-29 Mon 13:18] \\ Merged with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/50
- FallbackBlock, becomes
- for example for tables
- FallbackInline, becomes
- see TailwindCSS example
- for example for links
- for block
- show example of the table (above)
- show just some random code
- for inline
- show example of a link (in list above)
Currently many blocks remain unrendered. This should be resolved by rendering all block-like components in a Fallback component as a interim quick-fix.
- State "DONE" from "TODO" [2022-09-04 Sun 16:21] \\ Just merged to mainline and added some notes in-source tagged to @tijan about restructuring the architecture to optimize DRY-ness of the setup.
- also maybe button
- State "DONE" from "TODO" [2022-08-31 Wed 15:51] \\ Merged into mainline a few days ago
- maybe button
- URL [+ label] [+ description]
- State "DONE" from "TODO" [2022-08-31 Wed 15:51] \\ Merged into mainline along with the Link component
- reference: see Badges on TailwindCSS
- just a string
- State "DONE" from "TODO" [2022-09-04 Sun 16:22] \\ Implemented and merged to mainline
In order to populate the Kanban view (where the first TODO may not be part of a first-level heading), we want to be able to extract all top-level tasks from a document.
For the following snippet snippet we can list 3 top-level tasks, being:
- Task 1: Think about what to do
- Something in the future, not sure yet
- Task 2: Procrastinate
- Task 3: Figure out what is in the future
Any of the subtasks should only be visible when navigating into that level by clicking on the parent card.
:COOKIE_DATA: todo recursive
Not sure if the following links are relevant here:
Emojis are currently not consistently rendered. On my Linux box, I see a painful mess of just monochrome Wingdings-style emojis and the more colorful icons. Setting up an emoji font to ensure that everyone sees the same thing would improve consistency here and should be a small enough win for us to lock in.
- State "DONE" from "TODO" [2022-09-28 Wed 13:57] \\ Completed with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/93
On different platforms the selected serif font will look different. In order to control the quality of the reading experience, let's pick a decent open font to use here.
It is important the the fonts have varying weights such that we can still distinguish from bold text when it is presented inside of a Headline which is already presented in bold.
- State "DONE" from "TODO" [2022-09-28 Wed 13:56] \\ Merged with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/93
Here are the options approved by @siarhei:
- State "DONE" from "TODO" [2022-09-28 Wed 13:56] \\ Merged with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/93
Here are the options approved by @siarhei:
Implement a section component that uses something like headless's Disclosure to make the entire section foldable.
Currently the renderer doesn't render anything for a section (we just return an empty list). We may need to think through what rendering sections will "mean" since there are a lot of nested sections in a doc, so once we start rendering sections we should expect some fireworks. 🎆
Let's not worry about the explosions for now and just get this implemented! Once it's there it'll be my chore to figure out how to clean things up when merging the changes.
Minimize the noise in the docyment by folding all sections that are marked as
DONE, same as Fold completed headings in ToC, above.
- State "DONE" from "TODO" [2022-09-27 Tue 13:04] \\ Implemented by @tijan with https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/92
Minimize the noise in the ToC by folding all the headings that are marked as
todoKeyword from type
- State "DONE" from "TODO" [2022-09-25 Sun 20:57] \\ Block comments are already being rendered albeit not in an optimal presentation.
I know what this means but essentially I want to be able to parse comments that are formatted as follows:
and appears as:
This is a comment
- State "DONE" from "TODO" [2022-09-25 Sun 21:10] \\ Just verified that this works!
Readers need to be able to read inline comments such as the comment below:
This is an inline comment and should be visible on the page!
A comment component should open up as a drawer. Recorded a video (on YouTube) to demonstrate the capability.
My thinking is that a third pane (to the right), like an aside could function like a side-notes/aside/footnotes section where we can present the comments (however long they are). This would keep the flow of the main text clean. In this case, we would just show some line in the main Prose view to indicate that there is some content hidden away and then the user can click on some icon on that line (like the plus or expand icon) to open the full comment block in the third and right-most pane -- the aside.
- State "DONE" from "TODO" [2022-09-29 Thu 16:42] \\ Implemented by rendering source in a fallback component since https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/96. Future work will be to render this in a slicker code component. Battle for another day tho.
As a reader, I want to see the contents of source blocks presented as code. We don't expect any highlighting or any nifty features but we simply want to be able to read the contents of the code blocks.
We should be able to read the source block in section Parse block comments which is currently hidden.
The following code block should be parsed and displayed correctly (open as plaintext just to see what is in the source block).
- State "DONE" from "TODO" [2022-09-15 Thu 12:53] \\ Looking good and merged into hack [2022-09-15 Thu 12:53].
- State "DONE" from "TODO" [2022-09-29 Thu 16:53] \\ Wrote this down and realized the table renders in a fallback-like style. Good enough for now.
We can't wait forever on this one, so meanwhile we need a way to render the following table in a fallback component and then touch it up into a slicker representation at a later point:
| key | value | |---------------+--------------------------------------------------------------------------------------| | name | formation.tools | | type | devtool | | stage | prototype | | current niche | that Emacs crowd | | eventual goal | |
In order to display tables, we will need to parse them first. 😅 Right now the underlying datastructure is completely unclear (although I do have some ideas). Implementing the parsing of tables will inform which attributes and properties we need to keep in consideration.
For user to switch between the different modes/perspectives/views, let's just call it perspective, of a document we need something better than the toggle so let's define a button group that looks a bit like the Preview/Code tablist on pages such like https://tailwindui.com/components/application-ui/navigation/tabs to make it much clearer which view is currently active.
I guess perspective is clear enough as it provides a perspective on a document. /Modes/ may introduce confusion if we were to support vim-style modal navigating/editing at some point and /views/ already has a meaning with regards to interface development that could confuse us, as a team, or readers. /Perspective/ seems like a word that isn't used too often within the realm of text-editing and interface development to allow for easy misunderstanding -- I hope 😅.
- State "DONE" from "TODO" [2022-09-25 Sun 22:12] \\ Top bar is the place for now, but we will revisit this at a later point to localize the perspective-switch in the perspective view itself.
Reach out to @siarhei and team to figure out where to best place the perspective control.
- State "DONE" from "TODO" [2022-09-25 Sun 22:13] \\ Not very clear. We need to implement images to clarify this even more.
The opening section of this document points people towards the Toggle to switch views. Once's we've been able to rip that out, we can update the text to point viewers towards the new controls.
- State "DONE" from "TODO" [2022-09-14 Wed 11:15] \\ Already implemented and live.
The current ToC is looking very cluttered. Let's show the TODO tag, the headline text (for majority of the width of the ToC pane) and then the tags. I'm also open to dropping the tags from the TOC just to keep things cleaner because I believe that this is very likely just too much information.
I did something stupid in the rush of dropping the build. When you study the classes you'll find a few classes named
undefined littering the codebase. 💩 Clean this up!
Currently the entire Kanban view is y-scrolled as a whole but we want end-user to scroll individual columns for better control. Especially alter when we will want to facilitate DnD.
- State "DONE" from "TODO" [2022-09-07 Wed 21:53] \\ More or less done, albeit scrappy 😅
- The Kanban cards currently display an edit button and a deadline component. We want to hide the edit button and the deadline tag-like component and instead display the headline tags.
- Kanban cards may need to provide some context by indicating which heading they belong to.
For readability's sake, we want to color code all the tags such that the same value tags always appear in the same color. This should allow readers to visually associate related items provided that the readers/observers are capable of discerning the colors.
As a follow-up point, we should define a task to break inventorize which colors are fair to use from an a11y perspective in order to maximize the serviced audience by this feature.
Grep codebase for
@tijan and you'll find some notes on our Tailwind usage and DRY-ness (DRY as in "don't repeat yourself"). Let's stub an InlineContainer and a BlockContainer which we can reuse without having to redefine borders, shadows, background colors and basic typography.
Spoke with @tijan about this and we can look at extracting classes the Tailwind way https://tailwindcss.com/docs/reusing-styles#extracting-classes-with-apply to minimize duplication.
For now we have a
Block element that also exposes
blockClasses in case we need block-like styling anywhere but this point is still outstanding for refactoring.
- State "DONE" from "TODO" [2022-09-07 Wed 22:09] \\ All handled by @tijan 🏆🥳
When expanding/collapsing the ToC sections, the disclosures are not smoothly animated. The setup seems to be unaware of the changing height of the children and only effects a height-related change when the child elements are removed altogether which creates a bit of a visual shock as the space for the children instantly disappears from one moment to another.
I very likely lack the lingual skills to explains this very clearly, so just have a look in the storybook and then hop into my Google Meet room for us to talk it through for clarification.
💡 I'm an idiot! Whatever animation I currently implemented there is horrible. Especially on the longer list displayed in the index, the ugliness is very pronounced. It all looked decent in the shorter lists in Storybook but now it's clear that we can remove whatever I did there. It's so ugly it's technically criminal! My "engineering" license should be revoked for this one. 😅 Good thing I don't do this for a living.
- State "DONE" from "TODO" [2022-09-05 Mon 22:33] \\ Implemented with vidbina/parse-lists, ready for @tijan to work on the component.
List can be parsed as follows:
where you can use the following raw text samples:
- unordered list
- ordered list
- description list
- mixed list
💡 All of the previously listed samples are verified to parse correctly in the parser spec (core/parser.test.ts).
- State "DONE" from "TODO" [2022-09-14 Wed 11:42] \\ Merged into mainline but the question of folding/unfolding lists is yet to be solved.
In order to render lists correctly, we need to stub out a List component that can handle the inputs demonstated in Implement list parsing.
- Note taken on [2022-09-05 Mon 22:30] \\ Just wired up the home page to display the Engineering doc with branch vidbina/wire-up-layouts. Consider this a babystep to the end-goal.
Page reader[url] will be using the NarrowSidebar which is already demonstrated in a story for reference's sake. The following inputs are needed for the new layout:
- a ReactNode that is wired up to handle typography and dark mode state
- empty list for starters
- empty list for starters
- empty list for starters
- static, hard-coded vals for starters
- Note taken on [2022-09-05 Mon 22:32] \\ Instead of creating a UI flow, we are improving and simply rendering the Engineering document at index. This is a quick-fix to buy us some time. 😅
Page /reader/index needs to be restyled to align stylistically with the main reader, otherwise users may be confused into thinking that these are different apps.
What basically has to happen:
- browse from the Tailwind catalog which UI layouts are best suited for this view
- copy Tailwind templates over into the codebase
- delegate follow-up implementation
See the following snippet:
- State "DONE" from "TODO" [2022-09-29 Thu 16:40] \\ Just merged into Hack as part of wrapping up https://staging.formation.tools/#implement-source-blocks (https://gitlab.com/formation.tools/eng/proto-01/-/merge_requests/96)
The use of
Buffer-related errors within storybooks and other more limited environments (iirc unit tests may be struggling with this as well but I could be wrong here) than the browser or Node.js:
We can either
- refactor code to rip out the
Buffer-depending logic from the views or
- mock the
base64encodecalls to provide some behavior that Storybook can work with
For naming consistency, we should drop the
Type prefix. I borrowed this from the reference implementation in uniorg but it is a bit more verbose than needed and actually confusing since I don't follow the pattern for ElementType in a consistent manner.
unpackElementType returns an incorrect fallback. It returns a
e-typed fallback, which is reserved for Element types, for a GreaterElement element which should actually unpack into an
Furthermore, there are a bunch of elements for which this function just returns nothing (denoted by the empty list) instead of a fallback entry so a bunch of information goes missing.
Note that every section can contain its own definition of title
Perhaps we even remove
unpackElementType in order to fully rely on
convert which we may rename to
unpack in its new role.
The reasons being:
- there is quite of repetition between the two functions
unpackElementTypeand this logic could sensibly be considered part of the same concern
- converting into a Element alone may not be sufficient as elements tend to exist in the context of a document so we may have to convert an Element into a DocumentPart instead which can be a richer container for some helpful additional context for an Element to be understood correctly
- a heading with a particular TODO state exists in the context of a section which outlines what that state means in the context of that section (the section may define the workflow)
- footnote references don't exist without
The components of this syntax can be divided into two classes: “objects” and “elements”. To better understand these classes, consider the paragraph as a unit of measurement. Elements are syntactic components that exist at the same or greater scope than a paragraph, i.e. which could not be contained by a paragraph. Conversely, objects are syntactic components that exist with a smaller scope than a paragraph, and so can be contained within a paragraph. Elements can be stratified into “headings”, “sections”, “greater elements”, and “lesser elements”, from broadest scope to narrowest. Along with objects, these sub-classes define categories of syntactic environments. Only headings, sections, property drawers, and planning lines are context-free1, 2, every other syntactic component only exists within specific environments. This is a core concept of the syntax. Expanding on the stratification of elements, lesser elements are elements that cannot contain any other elements. As such, a paragraph is considered a lesser element. Greater elements can themselves contain greater elements or lesser elements. Sections contain both greater and lesser elements, and headings can contain a section and other headings.
- context free
Read https://en.wikipedia.org/wiki/Context-free_language as I will not pretend to understand this but the following elements should be parseable context-free and all other elements exist within the context of either of the following:
- planning elements
- property drawers
which is parsed into the following rough structure (in pseudocode)
Note that the first heading is space terminated otherwise, Emacs doesn't seem to recognize it as a heading. The example snippet copied from the referenced link on does not contain a whitespace so the copied content was not recognized as the author intended.
- State "DONE" from "TODO" [2022-09-07 Wed 21:53] \\ It's GPL3 now. It's a prototype build so we'll have to get over it. The idea of keeping this GPL3 is a fun challenge as well.
At the moment, the front-end will have to be released under GPL3 as far as we understand. This represents a risk for the commercial interests of the project and thus it is important to consider how we design the system going forward. Some approaches would be to:
- eliminate all GPL3 code and opt for more permisssive parsers if we can find any or write one ourselves
- design the architecture of the overal system such that we have a clear strategy about what is GPL3 and what we can continue to develop as IP
The uniorg is licensed under GPL3 and possible must be because it represents a translation (see GPL3 text for the definition of a translated program) of the org-element.el implementation in Emacs also under a GPL license. The alternative org-parser is also licensed as GPL3 so, apart from the effort to integrate, this would not be a valid option either.
If we cannot find a permissive parser, we will have to license the front-end project under GPL3.
In order to avoid having to bundle everything into a GPL3 app, we can provide all base front-end logic in the app and then take the marketplace approach and offer integrations with forces, communication tools and other third-party services (or our own services) through server-side facilities.
Use https://www.npmjs.com/package/uuid instead of https://www.npmjs.com/package/uuidv4 as per UUIDv4's notice https://github.com/thenativeweb/uuidv4#please-note.
Switching from Grommet to Tailwind because:
- the Grommet UI is a bit more playful than we need our app to look
- we will be bringing in some dev help that is pretty fluent with Tailwind and we want to be efficient
- TailwindUI provides paid component options that we can use to fast-track development of our UI
- State "DONE" from "TODO" [2022-08-23 Tue 17:36] \\ Sorted by @purcy on [2022-08-23 Tue] so we no longer have to worry about this. Future point would be to look into honoring Tailwind styles in the Storybook views as the current setup renders the components without any styling.
To simplify development of UI parts, we would like to be able to review components in Storybook. For this, we need to configure the project in order for
npm run storybook.
Fix the configuration and provide links to the relevant documentation that explains the design choicese in that configuration.
The following error is produced when running
npm run storybook:
- State "DONE" from "WIP" [2022-08-23 Tue 17:37] \\ We need HeadlessUI for the React components and some of the dynamic behavior associated with them (think Dialogs for Modals, Disclosures for accordion-like components, etc.). Some of the TailwindUI components seem to depend on this library and other third-party component libraries such as Flowbite 😠 (yeah, I'm not amused about this product ATM) seem to do the same.
In branch vidbina/setup-tailwind-headlessui, we are trying out incorporating HeadlessUI components in our setup. If this works, it may simplify building components in React that are based on top of TailwindCSS.
The HeadlessUI repository is pretty small, as in, only few components.
Flowbite has a UI blocks (based on TailwindCSS) that we can build on such as:
We have to explore to which extend we need this and can bring this in.
If we start a branch e.g.: setup-tailwind-flowbite, start it from vidbina/setup-tailwind.
The Next.js docs outline in https://nextjs.org/docs/advanced-features/custom-document how we can set
body level classes from within the
pages/_document.tsx file with the caveat that these are server-side rendered and therefore cannot support dynamic behavior (for example the use of event handlers such as
The blog post https://smnh.me/add-class-to-body-tag-in-nextjs outlines some ways to deal with html/body-level dark-mode settings which are dynamic in the sense that they are change during run-time and will need to be updated. It isn't quite clear if the instructions here are relevant at this moment so we will need to investigate what the issue here is.
The reason why this information is relevant is because some of the TailwindUI components such as https://tailwindui.com/components/application-ui/application-shells/multi-column will require the following updates to our template:
The use of class
bg-white is for non-dark-mode templates. Toggling dark mode will therefore require changes to the
body classList at runtime in order to work adequately.
@vidbina has copied over a few Application UI/Application Shells from TailwindUI and has made the necessary translations from HTML to JSX just to have something that looks okay-ish as a starting point.
We now have to take these compositions, basically UI dumps, and then decompose them into the larger subparts such as (following are just an example):
- SideBar which may contain
- Menu which contains
- Menu which contains
Note that the example in components/Layout.tsx is not complete in that some of the animations are broken and state is lacking which is why menus or other state-dependent elements will not behave correctly. All of these capabilities will need to be implemented.
💡 As a reference, look at component ToggleDarkMode which represents a basic TailwindCSS component which was translated from the TailwindUI code listing and then retrofitted to contain the correct images and to behave as expected by implementing event handling and by binding state into the component.
- State "DONE" from "TODO" [2022-09-25 Sun 14:37] \\ Basic TOC is implemented. Some improvements like linking are still pending but that is to be considered future work.
- State "DONE" from "TODO" [2022-08-23 Tue 17:39] \\ @purcy stubbed this component today [2022-08-23 Tue] and I just integrated it so we should be good to go. We still want to incorporate HeadlessUI's Disclosure components in order to facilitate folding but this is a nice-to-have that we can address later. First we have to address some higher-level layouting concerns because I just plopped the ToC on a page without much though or any grace. 😅
- State "DONE" from "TODO" [2022-09-25 Sun 14:36] \\ This issue is actually unresolved pending because we've refactored the TOC to show in dark-mode for better contrast during the light-mode. In the current form, we are not exposing the dark-mode functionality to end-users and will be recommending the use of a plugin like DarkReader to have a computed browser-wide dark-mode instead of one explicitly designed by us.
Switching between light and dark mode with the ToC rendered doesn't really change the text color. The ToC text should follow the classes for the main text which is altered during light/dark mode transitions. Not sure which TailwindCSS classes need to be set.
- State "DONE" from [2022-09-25 Sun 14:34] \\ Already merged into mainline (work done by @tijan)
- State "DONE" from [2022-09-25 Sun 14:34] \\ Implemented a naive approach (unique link ids per session)
See Design headline linking strategy for a characterization of the headline labelling problem. Once we've resolved this, we should roll an implementation for the ToC (and also the Prose but that is another concern).
- app, for app ui
- doc, for document ui
- app, for app ui
We still use Col, Row, Main, MainContent and AppContainer from components/View.tsx
I'm upgraded this task (instead of a child of the [[*Setup TailwindCSS][Setup TailwindCSS]] task it is now a sibling thereof) since it doesn't block any of the other tasks. We can therefore move along on the Storybook or TailwindCSS concerns without this affecting us at all. Having it here may allow @purcy to prototype things quickly and then we do the translation afterwards so I'm deprioritizing this as well by moving it to the bottom of the task list.
With Tailwind coming in, we should remove gradually remove Grommet. It isn't clear where all Grommet dependencies are, in terms of files that are Grommet-related, but we would need to identify them and then rid them from the dependency (remove Grommet dependency from core components) and also clean up the dependency manifest (packages.json).
Note that _app.tsx imported
GlobalStyle from styles/global.js. We have already removed the import but the styles/global.js file is still in place and will need to be removed.
👋🏿 This is an Org file, that markup format associated to Org-mode the so-called killer feature of Emacs 🐃🙊 but that's not the point. I've hated Confluence, Notion and consorts 😠 with a passion for a while and was keen to have a way to collaborate through plaintext with folks. It's obviously unfair to ask folks to learn Emacs, so I figured building a tool that allows folks to read Org files is at least something I can do to stop my whining about Confluence, Notion, et. al in the "shut up, fix it" spirit. This is the scrappiest prototype that I could whip up to start flirting with the idea. 🌱
The cool thing is that such Org files can be presented differently to provide a better overview to you, the reader. We can look at them as prose, as a kanban board (which displays the entries in this document that are tasks), as an calendar (for a view of everything that has dates/times associated to them) and more.
The toggle top-right can be flipped to switch between prose and Kanban view on this doc. Yeah, it's a horrible UX but it was the quick-and-dirty thing that we could built in there. 🙈
The "big idea" 💡 here is that most of this is kept as plaintext and that a git repo is the conduit to keep track of your content and changes. For most users, this is not interesting so we can provide them a web tool that abstrats all this git and Org stuff away from them but will allow the other weirdo hackers like myself to just live in their editor of choice and not worry about context-switching between their editor and slow web-tools.
Many of us probably agree that git is a great way to collaborate and track versions of ever-changing text. We don't need our data stored somewhere in someone else's database and request permissions through clunky APIs to access it when you can just have it on a repo on your machine. We want to offer a an easy API and good utils to just do whetever the heck you want with the files on your box. Honestly, with the development in tooling recently, decent search or other capabilities should be easily serviceable by your own machine through different utils (some perhaps built in Rust). Our computers are definitely packing enough heat to handle all the compute needed to for a local knowledge base.
Depending on how poorly (or warmly) this is received, I will decide how to move forward, so that feedback is needed for me to know if this is a moderately sensible idea or totally ludicrous. 😅 We're thinking about supporting collaborative editing in the future. I'm already setting up a coffee meet with the author of ProseMirror to explore this. Imagine being able to look at a prose, kanban or calendar diffs on the state of an Org-file when you come back from vacay 🏖️ to get a quick sense of all the things that have changed. This is also on the wishlist.
- React because of large eco-system (think: Storybook, Next.JS, etc.)
- good xs to docs and examples
- ability to deploy easily to Vercel
🧭️ Everything that is dotted and in red is on the way out! Things that are gray and dashed are ideas to likely be introduced Everything green and filled is live at the moment and everything else is just there but not highlighed to keep the diagram from being too cluttered. - red and dotted :: deprecating - green and filled :: live - gray and dashed :: idea
We use Vercel as our compute provider because it is easy.
The prototype app is our current drop-in point -- the one place where you end up for everything. Since we're in an early testing/validation stage, we are keeping this scrappy for now until we know what to really build.
We anticipate needing different APIs at a later stage for different functionality, e.g.:
- integrations with forges data hosting platforms (like GitLab or GitHub)
- integrations with auth services
- integrations with NLP-services to solve QA tasks or offer text paragraphing features (like an automated variant of Reddit's ELI5)
- integrations with chat platforms (like Slack), could be mounted under
All these APIs and integrations can be separate Vercel apps just to keep things more cleanly isolated. That way we can decouple deployments of our different services.
💡 This is just a concept section of APIs that we can implement at some point. As this section is commented, the dot diagram code below is not rendered into the infra overview at the top of this section.
We started off by directing all of our domains to a rewrite/proxy service where we resolved paths. This allowed us to gradually build out our stack with different services that we can mount under arbitrary paths on our target domains but was challenging to develop with. On , I had a call with Hassan about DX and we decided to move all rewrites to the prototype project directly.
By using the rewrites facility in next.config.js, we should be able to reference environment variables which can be defined using .env.local files during development (and pointed towards any arbitrary destination) and through the Environment Variables platform facility on a per-branch level in the Vercel platform. This should provide us the needed control and a tenable DX.
Some of the rewrites that we conduct are rewrites to external services. The last defined rewrite and therefore our fallback path is the prototype app for now.
Eventually, we can use the rewrites/proxy mechanism to mount all of our own services and 3rd-party services under friendlier paths, for example (and bear with me, this is just me musing):
/api/v1/auth/*for auth services
/api/v1/data/*to for data services,
/api/v1/data/forge/for forge integrations (to GitHub, GitLab, Gitea, etc.)
/api/v1/nlp/*for NLP services
/api/marketfor 3rd-party marketplace services
We have the following domain names:
We use the Rewrite (Proxy) facility from Vercel to manage these domains through a single Vercel project -- a convenient drop-in point to serve as our nice traffic-direction point.
We use Plausible for analytics. On the Plausible end, only two artifacts are relevant:
- the event API and
- the Plausible client script
In order to minimize the potential for loosing data due to ad-blockers, we proxy our own paths onwards to Plausible as suggested in the Plausible documentation on dealing with missing data (link captured on ).
- State "DONE" from "TODO" [2022-09-14 Wed 11:19] \\ Resolved with the help of Sebastian from Vercel. Core point was that we only defined a non-root rewrite =/:match*= instead of defining a root rewrite =/=.
Stubbed vercel-config but it is broken. We debugged this together to no avail and I just filed Case #00097879 with Vercel Customer Support.
In order to ensure that we hit our analytics back-end, we can define some rewrite rules towards Plausible.
The Plausible config is hard-coded in the codebase and pollutes our analytics because it also records localhost or staging traffic.
- State "TODO" from "TODO" [2022-09-14 Wed 11:20] \\ Not actively tackling this since we are too early stage to worry about payments at the moment. We're focusing on the big picture bits for now.
- Reporting: Single transaction overview at end-of-the-month to simplify accounting
- Invoicing: Capability to automatically issue PDF invoices to customers in-platform
- nice-to-have support to direct these invoices to a billing address to simplify processing for companies
This is a product concern so I'm signing this over to the product scope which is in the shared intel repo.
Use these as tools to guide you through every engineering choice.
We are a small team and probably always will be (kind of like a small formation of synchronized dancers 🩰 or a well-tuned squad of special forces operatives). Because of our size, we have to build solutions that minimize manual effort on remedial tasks.
Refer to [[https://sre.google/sre-book/eliminating-toil/][Eliminating Toil [Google SRE Handbook]]].
For anything we attempt to engineer, we make it a habit to design first by drafting a document that outlines the problem, potential solution candidates with reasons for why the candidates are preferred or not. This document is already an example of such a text because we have been "thinking out" ideas in this doc. 😅
💡 Really try to not initiate implementation work without first going through a design effort!
👨🏿💻 License to Hack: Sometimes a quick and dirty prototype is arguably part of the "design effort" and maybe even a required step in order to understand a problem and related constraints (for example tooling constraints) better. Somethings are simply hard to sketch out in advance or phrase in clear terms, so, in the spirit of show/hack-and-tell, feel free to whip up the occasional prototype to serve as "documentation" to communicate a design.
When engineering software, start with a README in a repository and document the problem and some solution candidates.
When any topic needs to be documented, find the one place where most will think to look for this information.
For a software project, this will be the git repositories or a more general architecture document in a dedicated knowledge base or another repository.
For a product idea, this will be the place were such ideation typically happens like a product management or project management tool.
Try to avoid replicating information because synchronizing multiple sources is a hard chore. Think: DRY!
We don't all process information the same way. When documenting information consider the following modalities:
- image (for example: diagrams, graphs and illustrations/sketches)
Write in a manner that feels natural to hear such that folks using TTS (text-to-speech) tools can follow along.
Since search mostly functions through text, try to accompany every non-text artefact (e.g.: video, diagram, photograph, audio snippet, etc.) with human-readable description.
Since we don't all parse information best through text alone, try to provide diagrams when possible for those who need visual cues. Complement your diagram entries with sufficiently descriptive alt-texts or summaries such that the idea within the diagram is communicated even if the summary is read without awareness of the image. Do the same for any non-text artefact that you may add to a document.
With complex diagrams, it may be easier to define high level relationships in the diagram and then follow them with a bit of elaboration up to an arbitrary level. In the spirit of time, we can attempt to keep these descriptions brief. We're hoping that someday there will be better tooling to help us convert such artefacts to prose for the folks who will need it.