Kplex 0.0.6 teaser

This is pretty exciting… but I won’t have time enough today to reach a release :frowning:

4 Likes

I believe that where Excalibrain occupied my second screen, K-Plex will occupy the first :star_struck:

1 Like

You can toggle the top right corner to switch between modes:

2 Likes

same with markdown and I guess bases and canvas as well… only I don’t have any bases and canvas files to test with in my main vault.

2 Likes

Just letting you know I haven’t forgotten about this… Over the weekend, I completely reworked the solution’s architecture to make it more future-proof. Since then, I’ve been testing the indexing and improving its performance. It feels like a never-ending story.

My goal is to build a solution that can handle very large vaults. Eventually, I’d like to convince Jerry Michalski, the owner of Jerry’s Brain, to do a test migration of his notes to Obsidian. He has over 600,000 nodes in his Brain, so that sets a high bar.

3 Likes

Thank you for the update! Really looking forward to the enhancements in the coming release!

1 Like

Just chatted with AI-Claude about how to generate a similar vault than the one of Jerry’s Brain.

I have just briefly tested the Node.js code below and generated a 75,000 note vault by:

node --max-old-space-size=6144 generate-vault.js --out ~/big-vault_75k --count 75000 --seed 42

which took about 7.0 seconds.

Opening the folder as vault took Obsidian about 3 to 4 minutes for indexing.
K-Plex worked just fine in about 1.5 minute.

The size could be increased in steps towards 600,000.

Maybe this is of help for testing?

Edit: just updated the code with the ontology now in the YAML block and comments etc. now in English.

#!/usr/bin/env node
/**
 * generate-vault.js - generates a large Obsidian test vault with a TheBrain-like structure.
 *
 * Relations are stored in the YAML frontmatter of each note:
 *
 *   Parent:
 *   - "[[Note name]]"
 *   Child:
 *   - "[[Note name]]"
 *   Friend:
 *   - "[[Note name]]"
 *
 * Example:
 *   node --max-old-space-size=6144 generate-vault.js --out ./big-vault --count 600000 --seed 42
 *   (for a first test use --count 5000)
 *
 * Options (all optional):
 *   --out <dir>              Target folder (default ./test-vault; must be empty or missing)
 *   --count <n>              Number of notes (600000)
 *   --seed <n>               Random seed; same seed = identical vault (42)
 *   --per-folder <n>         Notes per folder (1000)
 *   --max-children <n>       Upper limit of children per hub note (5000)
 *   --extra-parent-prob <p>  Probability of additional parents (0.15)
 *   --friend-prob <p>        Probability that a note has friends (0.25)
 *   --empty-ratio <p>        Share of notes without body text (0.35)
 *   --dup-title-ratio <p>    Share of duplicate titles (0.03)
 *   --force 1                Write into a non-empty folder as well
 */
"use strict";
const fs = require("fs");
const path = require("path");

// ---------- Options ----------
function parseArgs(argv) {
  const o = {
    out: "./test-vault", count: 600000, seed: 42, perFolder: 1000, maxChildren: 5000,
    extraParentProb: 0.15, friendProb: 0.25, emptyRatio: 0.35, dupTitleRatio: 0.03, force: 0,
  };
  for (let i = 2; i < argv.length; i++) {
    const a = argv[i];
    if (a === "--help" || a === "-h") {
      console.log(fs.readFileSync(__filename, "utf8").split("*/")[0]);
      process.exit(0);
    }
    if (!a.startsWith("--")) continue;
    const key = a.slice(2).replace(/-([a-z])/g, (_, c) => c.toUpperCase());
    const val = argv[++i];
    if (!(key in o) || val === undefined) {
      console.error(`Unknown or incomplete option: ${a}`);
      process.exit(1);
    }
    o[key] = typeof o[key] === "number" ? Number(val) : val;
  }
  return o;
}
const o = parseArgs(process.argv);

// ---------- Random numbers (seedable, deterministic) ----------
function mulberry32(a) {
  return function () {
    a |= 0; a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const rnd = mulberry32(o.seed);
const int = (n) => Math.floor(rnd() * n);
const pick = (a) => a[int(a.length)];
const chance = (p) => rnd() < p;
const cap = (s) => s.charAt(0).toUpperCase() + s.slice(1);

// ---------- Vocabulary ----------
const TOPICS = ["network", "memory", "idea", "project", "learning", "design", "history", "music",
  "people", "systems", "philosophy", "research", "culture", "economy", "health", "nature",
  "language", "software", "education", "art", "science", "community", "trust", "complexity",
  "future", "money", "media", "energy", "city", "story", "method", "question", "pattern",
  "value", "practice", "theory", "tool", "book", "event", "place"];
// Words with non-ASCII characters to test unicode handling
const ACCENTED = ["café", "naïve", "résumé", "façade", "über", "señor", "smörgåsbord", "jalapeño", "fiancée"];
const EMOJI = ["💡", "📚", "🌍", "🔥", "✅", "🧠"];
// Characters that are problematic in file names
const SPECIAL = [":", "?", "/", "#", "*", "\"", "|", "<", ">", "[", "]", "^", "&", "'"];
const SYL = ["ka", "lo", "mi", "ra", "ten", "sol", "vi", "dor", "an", "el", "u", "pra", "gen", "ma", "tis", "bo", "ni", "ver", "qua", "lin"];
const TAGS = ["idea", "todo", "people", "reading", "work", "archive", "research", "personal", "project", "reference"];
const TYPES = ["thought", "person", "place", "event", "concept", "reference", "project"];

function word() {
  const r = rnd();
  if (r < 0.5) return pick(TOPICS);
  if (r < 0.52) return pick(ACCENTED);
  let w = "";
  for (let k = 2 + int(2); k > 0; k--) w += pick(SYL);
  return w;
}

function makeTitle() {
  const r = rnd();
  const n = r < 0.35 ? 1 : r < 0.7 ? 2 : r < 0.9 ? 3 : r < 0.99 ? 4 + int(3) : 15 + int(15);
  const t = [];
  for (let k = 0; k < n; k++) t.push(word());
  t[0] = cap(t[0]);
  let s = t.join(" ");
  if (chance(0.03)) s += pick(SPECIAL) + " " + word();   // problematic characters
  if (chance(0.02)) s = pick(EMOJI) + " " + s;
  if (chance(0.01)) s += ".";                            // trailing dot
  if (chance(0.01)) s += " ";                            // trailing space
  return s;
}

// ---------- File names ----------
function toFileName(title) {
  let s = title.replace(/[\\/:*?"<>|#^\[\]]/g, "-").replace(/\s+/g, " ").trim();
  s = s.replace(/[. ]+$/, "");
  if (!s) s = "Untitled";
  if (/^(con|prn|aux|nul|com\d|lpt\d)$/i.test(s)) s = "_" + s;
  if (Array.from(s).length > 150) s = Array.from(s).slice(0, 150).join("").trim();
  return s;
}
const used = new Set();
const counter = new Map();
// Makes a file name unique (case-insensitive) by appending " (2)", " (3)", ...
function unique(base) {
  let name = base;
  let n = counter.get(base.toLowerCase()) || 1;
  while (used.has(name.toLowerCase())) { n++; name = `${base} (${n})`; }
  counter.set(base.toLowerCase(), n);
  used.add(name.toLowerCase());
  return name;
}

// ---------- Body text ----------
function sentence() {
  const n = 5 + int(12);
  const w = [];
  for (let k = 0; k < n; k++) w.push(word());
  return cap(w.join(" ")) + (chance(0.1) ? "?" : ".");
}
function paragraph() {
  const s = [];
  for (let k = 2 + int(4); k > 0; k--) s.push(sentence());
  return s.join(" ");
}
function makeBody() {
  if (chance(o.emptyRatio)) return "";
  const v = rnd();
  const out = [];
  if (v < 0.6) {                      // short
    const s = [];
    for (let k = 1 + int(3); k > 0; k--) s.push(sentence());
    out.push(s.join(" "));
  } else if (v < 0.9) {               // medium
    out.push(paragraph());
    out.push("## " + cap(word()));
    const b = [];
    for (let k = 2 + int(5); k > 0; k--) b.push("- " + sentence());
    out.push(b.join("\n"));
    if (chance(0.3)) out.push(`[${word()}](https://example.org/${word()}/${int(100000)})`);
  } else {                            // long (1% very long)
    const paras = v < 0.99 ? 8 + int(23) : 150 + int(450);
    for (let k = 0; k < paras; k++) {
      if (chance(0.15)) out.push("## " + cap(word()) + " " + word());
      out.push(paragraph());
    }
  }
  if (chance(0.2)) out.push(Array.from({ length: 1 + int(3) }, () => "#" + pick(TAGS)).join(" "));
  if (chance(0.02)) out.push(`![[${word()}-${int(9999)}.pdf]]`);   // intentionally missing attachments
  return out.join("\n\n");
}

// ---------- Phase 1: titles and file names ----------
const N = o.count;
console.log(`Generating ${N} notes in ${o.out} (seed ${o.seed}) ...`);
if (fs.existsSync(o.out) && fs.readdirSync(o.out).length && !o.force) {
  console.error("Target folder is not empty (use --force 1 to write into it anyway).");
  process.exit(1);
}
const titles = new Array(N);
const files = new Array(N);
for (let i = 0; i < N; i++) {
  titles[i] = i === 0 ? "Home" : chance(o.dupTitleRatio) ? titles[int(i)] : makeTitle();
  files[i] = unique(toFileName(titles[i]));
}

// ---------- Phase 2: graph (preferential attachment, multiple parents, friends) ----------
const parents = new Array(N);
const children = Array.from({ length: N }, () => []);
const friends = Array.from({ length: N }, () => []);
const pool = [0];   // each note appears once plus once per child, so hubs attract more children
parents[0] = [];
for (let i = 1; i < N; i++) {
  const ps = [];
  const extra = chance(o.extraParentProb) ? 1 + (chance(0.3) ? 1 : 0) + (chance(0.1) ? 1 : 0) : 0;
  for (let k = 0; k <= extra; k++) {
    let p = -1;
    for (let t = 0; t < 10; t++) {
      const c = chance(0.8) ? pool[int(pool.length)] : int(i);
      if (!ps.includes(c) && children[c].length < o.maxChildren) { p = c; break; }
    }
    if (p < 0) { p = int(i); if (ps.includes(p)) continue; }
    ps.push(p);
  }
  if (!ps.length) ps.push(int(i));
  parents[i] = ps;
  for (const p of ps) { children[p].push(i); pool.push(p); }
  pool.push(i);
}
for (let i = 0; i < N; i++) {
  if (!chance(o.friendProb)) continue;
  for (let k = 1 + int(4); k > 0; k--) {
    // 70% of friends are "near" in creation order, the rest are random
    const j = chance(0.7) ? Math.max(0, Math.min(N - 1, i + int(10001) - 5000)) : int(N);
    if (j === i || friends[i].includes(j) || parents[i].includes(j) || parents[j].includes(i)) continue;
    friends[i].push(j);
    friends[j].push(i);               // friend relations are mutual
  }
}

// ---------- Phase 3: write files ----------
const DATE0 = Date.UTC(2000, 0, 1);
const DATE1 = Date.UTC(2026, 8, 30);
const iso = (ms) => new Date(ms).toISOString().slice(0, 19);
const link = (j) => `[[${files[j]}]]`;

function renderNode(i) {
  const created = DATE0 + rnd() * (DATE1 - DATE0);
  const modified = created + rnd() * (DATE1 - created);
  const fm = ["---", `id: ${i}`, `title: ${JSON.stringify(titles[i])}`, `type: ${pick(TYPES)}`,
    `created: ${iso(created)}`, `modified: ${iso(modified)}`];
  if (chance(0.4)) fm.push(`tags: [${Array.from({ length: 1 + int(3) }, () => pick(TAGS)).join(", ")}]`);
  // Relations as YAML lists of quoted wikilinks
  for (const [key, arr] of [["Parent", parents[i]], ["Child", children[i]], ["Friend", friends[i]]]) {
    if (!arr.length) continue;
    fm.push(key + ":");
    for (const j of arr) fm.push(`- ${JSON.stringify(link(j))}`);
  }
  fm.push("---");
  const body = makeBody();
  return fm.join("\n") + "\n\n" + (body ? body + "\n" : "");
}

// Two-level folder structure: 000/0000, 000/0001, ...
const dirOf = (i) => {
  const f = Math.floor(i / o.perFolder);
  return path.join(String(Math.floor(f / 100)).padStart(3, "0"), String(f).padStart(4, "0"));
};

async function main() {
  const made = new Set();
  let bytes = 0;
  const t0 = Date.now();
  const BATCH = 256;
  for (let s = 0; s < N; s += BATCH) {
    const jobs = [];
    for (let i = s; i < Math.min(N, s + BATCH); i++) {
      const dir = path.join(o.out, dirOf(i));
      if (!made.has(dir)) { fs.mkdirSync(dir, { recursive: true }); made.add(dir); }
      const text = renderNode(i);   // called in order, so the output stays deterministic
      bytes += Buffer.byteLength(text);
      jobs.push(fs.promises.writeFile(path.join(dir, files[i] + ".md"), text));
    }
    await Promise.all(jobs);
    if (((s / BATCH) | 0) % 40 === 0) console.log(`  ${Math.min(N, s + BATCH)} / ${N}`);
  }
  const multi = parents.filter((p) => p.length > 1).length;
  const hub = Math.max(...children.map((c) => c.length));
  console.log(`Done in ${((Date.now() - t0) / 1000).toFixed(1)} s, ${(bytes / 1e6).toFixed(0)} MB of content.`);
  console.log(`Notes with multiple parents: ${multi}, largest hub: ${hub} children.`);
}
main();

1 Like

I currently use a synthetic Vault of 20k notes with 10% of large notes of 1MB size and 100+ links for automated testing.

I found a bug with the indexer - it reindexes whenever the ontology changes. This turns out to require a more invasive fix than I had hoped… and the worst of all, is along the way I just have more and more ideas what I want to build, so the backlog is getting longer :slight_smile:

1 Like

Hi @Zsolt
I’ve been working with K-plex for a week and it’s already wonderful. I get to an interesting workflow with Tasknotes. The only thing I would eventually like to have is to be able to display all the tasks (#task for example) in the same place… a bit like the “siblings” notes are at the top right, one could imagine that the tasks are at the top left (e.g.). The status would be indicated to me by the label of the link of connection… And I could filter them with the graph lenses

Tasks are coming. Not in the immediate next update, but soon. I think, I finally have cracked the instant-on issue, meaning, my target is you get a meaningful graph displayed 1-2 seconds after opening kplex, even if there is no index, and when you make changes those all take immediate visible affect. Plus there are uncountable enhancements that I’ve been holding back because indexing was problematic. I really strongly hope I can have a release before the end of today. :hand_with_index_finger_and_thumb_crossed:

5 Likes

Wonderful news :heart_eyes: @Zsolt

I’ve been playing around a bit with the test vault for ontology resolutions made available on GitHub by @Zsolt.

In doing so, I noticed that, in my view, the information displayed in the graph cannot be unambiguously resolved in some cases.

As I understand it, one reason for this is that the gates on the node box are assigned to fixed positions for lateral relationships. In addition, the relationship fields are permanently mapped to these positions. Furthermore, the four lateral relationship fields are aggregated into two gates (right and left) on the node box, resulting in a loss of structured information.

Please don’t get me wrong; I’m not saying that anything here is incorrect or not working. I simply come across cases that I find difficult to understand when reading the graph.

I would therefore like to present a possible solution in which the representation of the gates is handled quite differently and given greater significance.

As a first step, I translated the ontology’s relationship fields into—hopefully—expressive visuals, before reducing them to simple graphic forms:

I don’t see a major change needed in how the lines representing the links are displayed, except that thinner lines and smaller symbols should be used for cross-links to make them easier to distinguish. Hidden links should also be given a visual representation that is as clear and distinct as possible:

Using this vocabulary, the relationships between the notes can be represented in general terms as follows:

The idea behind the use of color is that only the symbols on the focus note are filled with color. This creates a contrast with the light-colored focus note.

The amount of links of a specific type—for example, “Friend”—can be indicated as small numbers within the graphical forms, see discussed cases below.

I will present a few cases that are difficult for me to interpret in the graph and then apply my proposal to them:

Case_1:

Implementation using the proposed graphical vocabulary:

Of course, things become much clearer without the cross-links. But for me, the Challenger relationship remains somewhat harder to read. In my proposal, however, the nature of the relationship can be grasped more directly if one understands the symbolism.

Case_2:

It’s again about a cross-link which hopefully could be resolved better by the proposed symbols:

Case_3:

When the siblings are activated, a hidden link also becomes visible, although it is currently difficult to distinguish it from a derived link:

In summary, I propose using symbols to indicate the relationship type, rather than relying on the position of the relationship line or a numerical indicator on the note box.

Of course, it is possible that I have overlooked something and the situation is actually different, or that there are cases that cannot be resolved—or cannot be resolved well—using the proposed symbolism. I’m looking forward to your feedback in these cases!

And, of course, I am very interested to know whether this proposal could actually be implemented in K-Plex at all – ???

2 Likes

Really excited for the update, I wish I can test soon. I’m withholding some issues that I think may be resolved with the update.

The next release also has tons of UI improvements. Indexing is getting better but it is still not full there where I want it to be. Aim is for the graph to be instantly available for navigation and interaction.

Regarding @Kai’s ideas, I have plans to split friend, previous, next and challengers. Your suggestion about the two gates but a single landing area actually works better then what I had in mind. I’ll first get the current update released, then start to look at this, and at task management support.

5 Likes

Same! Constantly checking for updates :joy:

1 Like

Good news!
I am just about ready. Doing a final run through and cleaning up some rough edges. I think the engine is now good.

I will likely publish tonight, and add a video walkthrough tomorrow.

If anyone has time at around 2026-10-08T16:00:00Z and would be open to part take in a YouTube video as “the audience” asking questions, let me know. I think a walkthrough of K-Plex would feel more natural and approachable if it were a dialog.

1 Like

I might be able to join, I can let you know how my work schedule is looking right away tomorrow morning.

1 Like

0.1.0 is live in Community Plugins

Given the many changes, I decided to call the new release 0.1.0. Still lots to work on, but I think this is an important milestone: K-Plex 0.1.0 - Many UI/UX improvements, better indexing

6 Likes