NotesByLex.com

LLMs Corrupt Your Documents When You Delegate

2026-05-09T00:00:00+10:00

An interesting paper from researchers at Microsoft.

They introduce a benchmark called DELEGATE-52 that tests whether LLMs can safely carry out, what they call, "long-delegated workflows" for document editing across 52 domains. Every set of instructions in the benchmark is lossless and reversible, allowing the authors to measure how much each task degrades the file's information over multiple interactions.

They found that even the strongest frontier models, including Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4 (the paper was released before their successors), corrupted an average of about 25% of document content after 20 interactions. Across all tested models, average degradation was about 50% Laban et al. (2026).

Python was the main exception. It was the only domain in which most models met the paper’s “delegation-ready” threshold, with 17 of 19 models scoring at least 98% after 20 interactions.

Figure 1 from Laban et al. (2026) shows examples of document degradation across different domains. The benchmark itself is text-only; the visual renderings are illustrative.

Surprisingly, the degradation didn't happen gradually over instructions, but models would typically fail catastrophically after a certain number of steps. Stronger frontier models would fare better only by delaying the step at which the degradation occurs.

They also found that tool use did not prevent degradation. The tested models performed worse with tools, averaging an additional 6% of degradation.

Measuring Document Corruption

To measure document corruption, they introduce a domain-specific Document Similarity Measure that parses documents into components. For a recipe, that means ingredients (name, quantity, unit), steps, and tips; for Python code, it means functions, classes, and imports. This lets them compare two parsed documents based on their actual content, rather than just raw text. Typical document similarity measures might overlook seemingly small changes, such as 200g to 800g of butter, which can be really bad in a recipe, whereas a surface-level rewrite that preserves the underlying structure doesn't need to be heavily penalised.

Figure 5 from Laban et al. (2026) - the domain-specific parsing pipeline, with a concrete recipe example showing how ingredients, steps and tips are extracted and compared

The approach of creating reversible transforms was inspired by Backtranslation, a machine translation technique in which text is translated into another language and then back, allowing the result to be compared with the original. DELEGATE-52 adapts that idea to document editing: apply a forward edit, apply the inverse edit, and compare the reconstructed document to the original. Imagine splitting a CSV into separate files by expense category, then merging them back together. Or converting all amounts in an accounting ledger to euros, then converting back.

They use a round-trip relay simulation method in which every task is assumed to be reversible, defined by a forward instruction and its inverse.

Seed Doc

2 cups flour

1 cup sugar

½ tsp salt

— Mix dry

— Bake 350°F

Original s

Forward edit
"→ metric"🔄 Fresh context

Transformed

473 ml flour

237 ml sugar

2.5 ml salt

— Mix dry

— Bake 175°C

Edited t

Backward edit
"→ imperial"🔄 Fresh context

Reconstructed

2.01 cups flour

1 cup sugar

½ tsp salt

— Mix dry

— Bake 350°F

Reconstructed ŝ

sim(s, ŝ)

RS@2 = 97.3

Reconstruction
Score

Each LLM call is independent with no conversation history. Errors survive into the next round because they are baked into the document itself, not the context window.

It's worth checking out some examples in the GitHub repo, see music, robotics and Ham radio as examples.

They also tested the inclusion of distractor documents in LLM interactions and found that they harm documents more as interaction length increases.

Basically, degradation severity is exacerbated by document size, interaction length, and the presence of distractor files. However, important to note that the LLM interactions themselves are stateless - it's not just that more noise in context causes outputs to degrade.

The simulation below steps through a recipe domain across 8 round-trips. Each forward edit converts imperial measurements to metric; each backward edit reverts. Watch how errors compound through the document across completely independent calls.

Round 0 of 8

Forward edit

—

Backward edit

—

LLM call 🔄 Fresh context — no history → document received, instruction applied, output returned

Original (reference)

Chocolate Chip Cookies

Ingredients

2 cupsall-purpose flour

1 cupgranulated sugar

½ tspsalt

2 sticksunsalted butter

Steps

1. Cream butter and sugar until fluffy.

2. Sift in flour and salt, fold gently.

3. Bake at 350°F for 12 minutes.

After round-trip reconstruction

Chocolate Chip Cookies

Ingredients

2 cupsall-purpose flour

1 cupgranulated sugar

½ tspsalt

2 sticksunsalted butter

Steps

1. Cream butter and sugar until fluffy.

2. Sift in flour and salt, fold gently.

3. Bake at 350°F for 12 minutes.

RS@k across rounds

—

Reconstruction score

■ Corrupted value ■ Deleted content ■ Hallucinated content

DELEGATE-52

The benchmark contains 310 work environments across 52 domains. Each environment includes real seed documents, distractor files, and 5-10 reversible edit tasks that resemble the kinds of tasks a worker might delegate to an LLM.

Figure 3 from Laban et al. (2026) - the 52 domains across five categories: Code & Configuration, Science & Engineering, Creative & Media, Structured Records, and Everyday

Figure 4 shows an example work environment from the accounting domain.

Figure 4 from Laban et al. (2026) - a work environment from the accounting domain, using a Hack Club ledger as the seed document, with forward/backward edit pairs like splitting by expense category and merging back

Results

They tested 19 models across the benchmark. All 19 models degraded documents over the course of the simulation. The top performers, such as Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4, still corrupted an average of about 25% of the document content after 20 interactions. Across all tested models, average degradation was about 50%, with weaker models failing more severely.

Table 1 from Laban et al. (2026) - round-trip relay results for 19 LLMs across 20 interactions, colour-coded by degradation severity. Every model declines over time; frontier models delay but do not avoid degradation.

Short-term performance did not reliably predict long-horizon performance. Some models that looked similar after two interactions diverged sharply after twenty, while others that started behind later caught up. This is one of the reasons the paper argues for long-horizon evaluation rather than only testing one-shot or short workflows.

The kind of degradation also changes with model strength. Weaker models tend to lose content through deletion, while frontier models are more likely to preserve content but corrupt it.

Takeaways

One takeaway is that we need to be careful not to extrapolate model capabilities from one area to all domains. Models follow a Jagged Frontier of LLM Capability, where they can excel in some tasks while making serious errors in others. For example, they perform well on Python and poorly on some structured-but-unfamiliar document formats, such as textual 3D object files.

It also raises interesting questions about whether we need to decouple the reasoning engine from the state management system. LLMs may be useful as the reasoning layer, but long-running document workflows probably need external state, parsers, validators, diffs, tests, and reversible operations to prevent silent corruption.

References

Philippe Laban, Tobias Schnabel, and Jennifer Neville. LLMs Corrupt Your Documents When You Delegate. April 2026. arXiv:2604.15597, doi:10.48550/arXiv.2604.15597. ↩ ¹ ² ³ ⁴ ⁵ ⁶

AI-Induced Cognitive Atrophy

2026-05-07T06:33:00+10:00

Something I’m increasingly worried about is that the more mental labour we offload to AI, the more our own cognitive capacity starts to dwindle.

In a popular paper from 2025 (AI Meets the Classroom: When Does ChatGPT Harm Learning?), they found that students in a programming course who used LLMs to generate answers to questions (substitution) saw a significant decrease in topic understanding, even though they perceived an increase Lehmann et al. (2025).

Learning is effortful: you need to exert mental energy to learn. However, AI can give you the illusion of learning without any mental exertion at all.

The study also found that students who used AI to complement learning activities, like asking for explanations rather than answers, actually saw an increase in understanding. But when given a free choice, most students defaulted to substitution and found that nearly half of all solution requests came without the student making a single attempt first. Our brains tend to prefer to conserve energy. They also found that students with weaker foundations learned less with AI access, while stronger students benefited more.

For a software developer, it’s impossible to ignore the upside from modern agentic coding workflows; they are clearly here to stay. But, at the same time, for complicated projects beyond what can be vibe-coded, a pre-developed intuition for software is necessary to guide those agents safely.

So the natural question arises for a software developer, and a knowledge worker in general: how do we find the balance between making the most of AI while avoiding letting our skills, intuition and understanding decay?

Just as many people leading sedentary lifestyles have to make a deliberate effort to exercise, because inactivity is really bad for our bodies, I think we’re going to realise that a similar process is necessary for our minds.

My hypothesis is that you need to deliberately spend time every day exercising your skills, operating near your cognitive limits - performing deep System 2 Thinking - if you want to avoid the kind of atrophy that will make you worse at wielding AI agents. The idea of Progressive Overload in strength training is to gradually increase resistance via heavier weights, building overall capacity. The same paradigm applies to cognitive capacity.

One thing I've been experimenting with is making sure I allocate at least 30 minutes a day on something mentally difficult, like writing code by hand for an unfamiliar problem or reading difficult papers. Writing these notes is another task that feels like a form of this exercise.

References

Matthias Lehmann, Philipp B. Cornelius, and Fabian J. Sting. AI Meets the Classroom: When Do Large Language Models Harm Learning? March 2025. arXiv:2409.09047, doi:10.48550/arXiv.2409.09047. ↩

OpenGame: Open Agentic Coding for Games

2026-04-29T00:00:00+10:00

Paper describes OpenGame, an agentic framework designed for end-to-end web game creation. Jiang et al. (2026)

The authors argues that to build products as complex as games, the field needs to move beyond generalist code agents to specialist frameworks.

They throw the kitchen sink at the problem of game design: a base model, a code agent, a new collection of Agent Skills for game development, and a new benchmark and evaluation framework.

Figure 2. from Jiang et al. (2026) - showcasing the entire end-to-end OpenGame system.

Base Model

The authors contribute a new foundation model based on a Qwen3.5-27B backbone, called GameCoder-27B, via a three-stage pipeline:

Continual Pre-Training (CPT) on a corpus of open-source Phaser and JavaScript/TypeScript game repositories from GitHub, with docs and tutorials, to build strong priors over game loops, physics systems, asset usage, and state management.

Supervised Fine-Tuning on game generation prompts using gpt-codex5.1, with solutions from minimax2.5. For example: "Implement a 2D platformer character controller with double-jump and sprite animations."

Reinforcement Learning (RL) at the component level: execution grounded, rewarding unit test pass rate and execution success on single-file gameplay logic and targeted functional modules (e.g., collision detection, state-machine transitions). Getting the model strong at the component level works because a downstream agent assembles those building blocks into a full multi-file project.

Code Agent Design

To produce a complete game, the authors argue you need Structured Long-Horizon Workflow systems.

OpenGame orchestrates the agent through six operational phases, using a persistent todo_write tool that lets the agent plan, execute, and transition across phases in a controlled manner.

1. Classification

The agent invokes classify-game-type, which applies a Physics-First Classification rule. Rather than relying on ambiguous genre labels, it categorises the task by physical constraints and spatial mechanics (e.g. "falling without ground support" == platformer archetype, "snapping to a grid" == grid_logic). This sets the macro-level execution plan.

2. Scaffolding

Once the game archetype is known, the agent runs run_shell_command to copy the shared scaffolding codebase and relevant architectural documentation into the workspace, giing the model a baseline to operate from.

3. GDD Generation

The agent invokes generate-gdd to produce a technical Game Design Document. This tool dynamically loads archetype-specific API constraints from the scaffolded documentation, ensuring the proposed mechanics are feasible under the selected framework. The agent extracts the implementation roadmap from the GDD and uses todo_write to refine its high-level plan into granular, file-specific actions.

4. Multimodal Asset Synthesis

The agent reads asset_protocol.md to ensure parameter compliance, then invokes generate-game-assets, leveraging multimodal generation models to synthesise backgrounds, character animations, static items, and audio assets from the GDD's asset registry. For tile-based games, generate-tilemap converts ASCII layouts into structured JSON tilemaps. The agent then reads the produced assetpack.json to record exact texture and asset keys needed during implementation, substantially reducing downstream asset-reference hallucinations.

5. Context-Aware Code Implementation

Before writing gameplay logic, the agent merges GDD parameters into gameConfig.json, enforcing a data-driven interface between design and code.

To mitigate context overflow during implementation, they introduce a Three-Layer Reading Strategy: using read_file, the agent progressively loads (1) an API summary for the template system, (2) the target source file (_Template*.ts) to be modified, and (3) the implementation guide, loaded last to maximise immediate salience.

Code generation follows a Template Method Pattern: rather than writing the project from scratch, the agent copies template files and overrides designated hook methods (e.g., setupCustomCollisions) to inject game-specific logic while preserving the deterministic lifecycle management of base classes.

6. Verification and Self-Correction

The agent reads debug_protocol.md to perform a static self-review over common generative failure modes, then uses run_shell_command to execute npm run build and npm run test under headless browser evaluation.

When build or test failures occur, the agent parses compiler output, localises the faulty script, and iteratively repairs the project until a playable game is obtained. This phase provides the operational substrate for the Debug Skill described below.

Agent Evolution with Game Skills

Game Skill is a "reusable, evolving capability" composed of two components: Template Skill and Debug Skill. Together, they let the agent scaffold stable architectures and systematically repair integration errors, rather than patching isolated bugs.

Template Skill

Grows an evolving library of specialised project skeletons $L$ , starting from a game-agnostic meta template $M_0$ and expanding into specialised template families like gravity-based side-view and top-down continuous motion.

$M_0$ intentionally assumes no genre, physics regime, or gameplay mechanic, just the universal structure required for a playable game: project layout, initialisation, asset loading, scene loops, and configuration interfaces.

As the agent completes games, reusable fragments are extracted and merged back into $L$ . This sharply reduces the search space of generation and stabilises project-wide structure across diverse requests.

Debug Skill

Maintains a living debugging protocol $P$ , updated from observed build, test, and runtime outcomes. Each new failure pattern is appended as a verified (error signature, root cause, verified fix) triple. This lets the agent accumulate verified fixes and systematically resolve high-frequency integration failures across games, rather than re-diagnosing the same classes of error from scratch.

OpenGame-Bench

A new evaluation pipeline for agentic game generation, scoring output across three metrics:

Build Health: whether the project compiles and runs without errors under headless browser execution.
Visual Usability: whether the game is visually coherent and navigable, assessed via VLM judging.
Intent Alignment: whether the generated game matches the original natural-language specification.

Evaluation uses a combination of headless browser execution and VLM judging across 150 game prompts.

They identify three reasons why general-purpose LLMs struggle to produce complete, playable games:

Logical Incoherence: the model loses track of global state across the game loop, causing freezes, failures to terminate, or mechanics that never materialise.
Engine-Specific Knowledge Gaps: general models misuse or ignore engine abstractions, reimplementing mechanics from scratch instead of using framework-native physics, scene, and event systems.
Cross-File Inconsistencies: individual files look plausible, but the overall project breaks due to mismatched asset keys, flawed scene wiring, missing config fields, or broken init order.

The argument is that fixing this requires frameworks that understand the intrinsic structure of games, not just better prompting of generalist agents.

Reminds me of the SheetCopilot Agent, an agentic framework for spreadsheet controls, and systems like AlphaEvolve, a system for algorithmic discovery.

References

Yilei Jiang, Jinyuan Hu, Qianyin Xiao, Yaozhi Zheng, Ruize Ma, Kaituo Feng, Jiaming Han, Tianshuo Peng, Kaixuan Fan, Manyuan Zhang, and Xiangyu Yue. OpenGame: Open Agentic Coding for Games. April 2026. arXiv:2604.18394, doi:10.48550/arXiv.2604.18394. ↩ ¹ ²

Naming Things Is Easy Now

2026-04-25T10:58:00+10:00

“There are only two hard things in Computer Science: cache invalidation and naming things.” - Phil Karlton

I don't think naming things is hard anymore (cache invalidation is about the same).

LLMs, with their default of sampling from the highest-probability tokens, make for painfully boring and mediocre writing, but for many naming tasks, that’s exactly what you want. For example: naming methods for a public API; choosing names for events, actions, states etc; or trying to name an as-yet unnamed but known concept.

It also drastically reduces the time wasted bike-shedding or being stuck in analysis paralysis that often comes with naming tasks.

Simon Willison said something similar in the context of API design:

“A common criticism of these things is that they always come up with the most obvious answer… but when you’re designing an API, that’s exactly what you want.”

I’m sure many others have said it too.

My approach to naming things with LLMs is something like this:

Start by coming up with a few names for the thing on my own. I treat these names as a kind of hold-out set. This also helps me clarify exactly what I’m trying to name.
Then I ask my LLM of choice to come up with a collection of 5-10 names, with ranking and justification, with enough context. That’s typically:
- A good description of what the thing is: what the class does, what the idea is, and so on.
- What the thing is not, especially if there are similarly named concepts that do not fit.
- The surrounding vocabulary of the codebase, API, paper, or note collection (which may come from the research step in Research, Plan, Implement Workflow)
Review the candidate names generated by the model and compare them with my hold-out set.
If one of the top candidate names is in the hold-out set, that’s a pretty damn good signal we have a winner. If the model’s candidates and mine are completely different, there’s a good chance I haven’t explained the concept well enough.
If I don’t see a clear winner in the candidate set, I’ll do another round. This time, I’ll give it my hold-out set of names and ask it to critique my names against its candidates.

In a short while, I usually find a name that feels obviously right. Sometimes I prefer one of my original names, but at least I have a good idea of why - and why it's better than the other options.

Photo by Compagnons on Unsplash

Obsidian Markdown Notebook: code execution with outputs stored in the file

2026-04-21T00:00:00+10:00

I've built Obsidian Markdown Notebook, a plugin that lets you execute code in Obsidian with both code and output stored in the same file, kinda like a Markdown Jupyter Notebook.

Right now, the supported languages are Python, JavaScript, Bash, and R. If you want support for more, let me know.

Here's an example of plotting Fourier series components:

There are quite a few similar plugins already for executing code in Obsidian. However, none of them are designed from the start to store outputs directly in the Markdown file itself. It scratches an itch I've had for a while.

By default, outputs are rendered as HTML. For example, a pandas DataFrame renders as a table:

```python
import pandas as pd
df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
df
```
<!-- nb-output hash="a1b2c3d4e5f6a7b8" format="html" -->
<div class="nb-output-html">...</div>
<!-- /nb-output -->

You can also render as an image by adding format=image to the code fence:

```python {format=image}
import matplotlib.pyplot as plt
plt.plot([1, 2, 3])
plt.show()
```
<!-- nb-output hash="209f46a0d3ce8ca2" format="image" -->
![](../_media/209f46a0d3ce8ca2.png)
<!-- /nb-output -->

The image filename is a hash of the language and source code, so two notes with identical code fences will share the same cached image. You can also assign an id to control the filename:

```python {format=image id=numberplot}
import matplotlib.pyplot as plt
plt.plot([1, 2, 3])
plt.show()
```
<!-- nb-output id="numberplot" hash="209f46a0d3ce8ca2" format="image" -->
![](../_media/numberplot.png)
<!-- /nb-output -->

Output blocks use HTML comments, so they're invisible in PDF export and any standard Markdown renderer.

Each output block stores a hash of the cell's source code. If the code hasn't changed, the cached output is shown without re-executing. Re-running a cell updates the output in place.

You can also specify document-level defaults in the YAML frontmatter, or set project-level defaults in plugin settings.

---
title: My document
notebook:
  format: image
  markdownLinks: true  # There's other settings too - see the GitHub repo for the full list.
---

Similar Plugins

There is a lot of prior art here, but again the major gap is that none of them focuses on storing the output artifact alongside the code.

Obsidian Execute Code Plugin is the closest relative. It does support persistent output since version 2.0.0. However, the output is plain text, and I want rich output (HTML tables, images) to be a first-class citizen from the start.
Obsidian Code Emitter is a great plugin that supports 15 different languages without requiring any system dependencies. However, the outputs do not survive vault reload and cannot be rendered to PDF.
JupyMD uses Jupytext to pair a Markdown file with a Jupyter notebook, but the outputs are stored in the Jupyter file. I just want something totally native where everything lives in the Markdown file.

And of course, Jupyter Notebook was the primary inspiration. There is also a Markdown-based notebooks proposal from 2023 that stalled without a consensus. I want it now, damn it!

See the project Github for more details.

Also, side note: the project was developed using a Research, Plan, Implement Workflow. You can see the Markdown files for each stage in the .claude directory.

Research, Plan, Implement Workflow

2026-03-22T00:00:00+10:00

For people who have been using Claude Code or a similar agentic coding tool for a while, a very obvious pattern emerges: start each session by asking the agent to research the code, work on a plan (or spec), then implement it.

This is absolutely not an original idea. My colleagues have been using this approach, and there's some great writing on it, like this blog post by Boris Tane (and I'm sure there are many others).

The idea is that each development loop starts with Research, then Planning, then Implementation. It's very similar to Spec-First LLM Development - in fact, it's a concrete instantiation of it. Where spec-first is the abstract principle, this is the hands-on workflow.

Research

In the research phase, you ask the model to read the relevant code. For my work codebase, that's usually an entire service directory or a path within it. For a smaller project, it might be the entire repository. The prompt might be like:

"Read and understand the billing service in the billing_service directory. Pay close attention to how we handle cancelling monthly subscriptions."

In Tane's workflow, findings always get written to a persistent research.md file. I find that useful for complex problems, though for smaller tasks, a verbal summary in the chat can be enough. The important thing, either way, is that you read the output and verify your understanding before moving to planning.

Planning

Once you're satisfied with the research, ask for a detailed implementation plan. For example:

"Come up with a plan for adding the ability to pause a monthly subscription. The user will be able to specify how many months to pause for. Include the approach, file paths, code snippets showing the key changes, and any trade-offs."

Claude Code's Planning Mode does a great job of this. Alternatively, storing it in a plan.md file gives you full control; you can edit it in your editor, and it persists as a real artifact.

One trick that works well: if you've seen a good implementation of something similar in another codebase, share that code as a reference alongside the plan request. The StrongDM Software Factory approach calls this Gene Transfusion. A bit dramatic for my tastes - but you get the idea.

A useful technique is to get the LLM to interview you: ask it to clarify requirements before writing the plan. This forces both of you to agree on exactly what the solution needs to look like before any code is written. This is a feature built into Claude Code's Planning Mode, and it works great.

Typically, you expect to iterate on the plan a few times to get it right. This is the time you really want to be paying attention with your neurons firing - getting the plan right can save you a lot of time downstream.

The plan will usually also contain an implementation checklist to be followed during the build.

Implementation

When the plan is right, issue the implementation command:

"Implement based on the plan. Write the tests first."

Sometimes, depending on the problem, I won't require Claude to write the tests first, but TDD is often a nice way to verify the plan.

At the end of the implementation cycle, I'll review the changes, and typically expect some further iterations. I'm not afraid to edit the code myself - especially comments. Sometimes, I'll edit via pseudo code and ask it to go back and implement it, if I think the implementation is too far off.

The pattern tends to come naturally after using agentic coding tools for a while, but it's helpful to give it a name.

Photo by Vooglam Eyewear on Unsplash

Self-Generated Agent Context Files Don't Help Either

2026-02-26T00:00:00+10:00

Self-Generated Agent Context Files don't help either.

This paper, released around the same time as SkillsBench, evaluates whether repository context files (AGENTS.md, CLAUDE.md) actually improve coding-agent performance.

The authors test issue-resolution tasks (bug fixes and feature work derived from real PRs) under three settings:

no context file
LLM-generated context file
developer-written context file

Similarly to SkillsBench's results, they conclude that LLM-generated context files generally reduce task success rates (about 3% on average) while increasing costs by 20%+. ¹

Developer-written context files perform better than LLM-generated ones and can provide a small lift (~4% on average), but they still increase token usage. ¹

Seems that agents do follow context-file instructions, but it typically means they run more tests, traverse more files, and spend more reasoning tokens. The problem is that this is often just extra exploration overhead rather than a meaningful improvement. Also, if the repository already has good developer documentation, duplicating the information into agent content files adds little value - agents can already read the existing docs.

So, putting the paper findings together, it seems the rules of thumb with context files and agents are:

don't mindlessly auto-generate AGENTS files
focus effort on standard high-quality developer documentation first
keep agent context files minimal and high-signal
use a small number of targeted skills/context instructions where they clearly help

Gloaguen, T., Li, J., Schmid, L., Bichsel, B., and Vechev, M. (2026). Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? arXiv. https://arxiv.org/abs/2602.11988 ↩↩

Self-Generated Skills Don't Help

2026-02-21T00:00:00+10:00

Agent Skills are structured packages of Markdown files and scripts that augment AI Agents' capabilities. They usually look something like this:

~/.claude/skills/some-skill/
├── SKILL.md
└── scripts/
    └── some_script.py

They were first introduced as a feature of the Claude ecosystem around October 2025 ¹, and have recently exploded in popularity, thanks to OpenClaw and friends.

Despite a lot of benchmarks existing to measure agentic AI capability, none so far have been created solely for the purpose of measuring the efficacy of skills, including how and when to use them, and what differentiates good skills from bad ones. That's where the paper SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks by Li et al. (Feb 2026) comes in. ²

In the paper, they describe their SkillsBench benchmark of 84 tasks across 11 domains. Each task in the benchmark is tested under 3 conditions:

no skills
curated skills
self-generated skills - that is, skills entirely created from the LLM's own knowledge.

The skills themselves were pulled from GitHub, community marketplaces, and corporate partners, with around 47k skills tested as part of the collection.

They found that the self-generated skills offered no benefit (or worse) on average. However, curated skills raised the average pass rate by 16.2 percentage points, with effects varying across domains. Healthcare tasks got the biggest boost from skills, while software got the least, suggesting that the models already have a lot of useful knowledge about how to achieve software tasks, and less-understood domains get the biggest boost from skills.

Also, 2-3 focused skills per task seems to be the sweet spot, with the authors seeing diminishing returns beyond that.

They test several LLMs with their respective agent harnesses and find that Gemini performs best overall. However, skills had the greatest impact on the Claude Code's capability, which I guess makes sense since Anthropic created skills in the first place and probably has the greatest lead with their models fine-tuning to use skills. Codex CLI showed competitive raw performance, but it frequently neglects the provided skills. Agents acknowledge the skill content but often implement solutions independently. I suspect Codex will improve at skill utilisation in future versions as OpenAI refines its implementation.

The paper provides a concrete definition of a skill, contrasting it with other agentic paradigms like Few-Shot Examples, Retrieval Augmented Generation and Tool Documentation. According to the paper, a Skill is an artifact that satisfies four criteria:

Procedural: It teaches how to do something (workflows, step-by-step procedures) rather than just stating facts
General: It applies to a category of problems, not just one specific instance
Structured: It includes a SKILL.md file and can bundle supporting resources like scripts, templates, or examples
Portable: It lives entirely in the filesystem, making it easy to edit, version control, share, and use across different agent harnesses

The paper draws a nice analogy to computing paradigms: foundation models provide base capabilities (like CPUs), agent harnesses orchestrate context and tools (like operating systems), and skills extend competence to specialised domains (like applications).

In my own work, I've been finding a lot of success recently by adding skills to our project. They tend to be really useful for guiding the LLM on how to run the test suite and evals effectively, how to check for common issues across the codebase, and even for parsing logs and debugging common customer issues. Any time I find myself repeatedly performing a cumbersome sequence of steps, turning it into a Skill pays dividends pretty quickly. They're just docs and convenience scripts at the end of the day. Not exactly a brand new paradigm for software engineers.

My current OpenClaw setup: OpenClaw: the missing piece for Obsidian's second brain
Another article on AI-Assisted Development best practices: Spec-First LLM Development

Introducing Agent Skills. (n.d.). Claude. Retrieved February 23, 2026, from https://claude.com/blog/skills ↩
Li, X., Chen, W., Liu, Y., Zheng, S., Chen, X., He, Y., Li, Y., You, B., Shen, H., Sun, J., Wang, S., Zeng, Q., Wang, D., Zhao, X., Wang, Y., Chaim, R. B., Di, Z., Gao, Y., He, J., … Lee, H. (2026). SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2602.12670 ↩

Generative Modelling via Drifting

2026-02-11T00:00:00+10:00

This paper introduces a new paradigm for single-step generative modelling called Drifting Models ¹

Where Diffusion/Flow Matching performs iterative denoising at inference time, Drifting Models shift the iterative process into training.

The core of the algorithm is the computation of a "drift field", which is a vector for each generated sample that points toward the direction of the real distribution. The goal is to calculate a drift field where when model's output distribution matches the real distribution, the drift is zero.

The drift field is calculated using positive and negative samples. Positive samples are real examples from the same class in the training data. Negative samples are other model outputs in the batch (though other negatives, such as real examples from other classes, are also tested). We want the drift field to attract toward positives, repel from negatives, but at equilibrium, attraction and repulsion balance out: $\mathbf{V} = \mathbf{V}^+ - \mathbf{V}^- = 0$

Leaving aside computing the drifting field for a second, the loss computation is straightforward.

We compute a drifting vector for each output sample and train the model so its output moves toward its drifted version, indirectly minimising the magnitude of the drift. We use stop-gradient to avoid back-propagating through the drift computation.

noise = rand([N, C])

x = model(noise)
y_neg = x
y_pos = training_data

V = compute_V(x, y_pos, y_neg)

x_drifted = stopgrad(x + V)

loss = mse_loss(x - x_drifted)

The drifting can occur in feature space (via an image encoder such as SimCLR or MoCo-v2) or even raw pixel space. Albeit the best results on ImageNet came from feature space.

Computing the Drift Vector - V

The computation for the Drift Vector - V can be broken down into a couple of steps

def compute_V(samples, y_pos, y_neg):
    # ...
    return V  # A batch of drift vectors, each in dimension to the input samples

1. Compute L2 Distances

We first compute the distance between the model samples and the positive samples (a batch of training examples) and negative examples (the model samples themselves). The paper uses the L2 norm, which is available in scipy as cdist:

from scipy.spatial.distance import cdist

# compute_V

dist_pos = cdist(samples, y_pos)
dist_neg = cdist(samples, y_neg)

So we have two matrices that represent the distance between each sample.

dist_pos Distance to positive samples	dist_neg Distance to negative samples

2. Mask out self

Since we're using the batch of model outputs as negatives, we want to ensure that the model ignores any self-distances. In practice, we set this to a very large distance, to ensure it's ignored the softmax (nearby generated samples contribute stronger repulsion, so we don't want a sample repelling itself).

# Add a large value to the diagonal so self-distances are ignored in softmax
dist_neg += np.eye(N) * 1e6

dist_neg_masked - Self-distances are masked so samples don't repel themselves:

3. Compute weights via Softmax

We now compute a Softmax operation that performs attention over these distances.

First, we concatenate the distance matrices and convert to logits using temperature scaling:

logit_pos = -dist_pos / temperature
logit_neg = -dist_neg / temperature
logit = np.concatenate([logit_pos, logit_neg], axis=1)  # [N, N_pos + N_neg]

Temperature Scaling controls how sharply the model focuses on nearby samples. Lower temperature = winner takes all.

The paper uses bidirectional normalisation: softmax over both rows (which y sample to attend to) and columns (which x sample attends). These are combined via geometric mean:

A_row = softmax(logit, axis=1)  # normalize over y samples
A_col = softmax(logit, axis=0)  # normalize over x samples
A = np.sqrt(A_row * A_col)      # geometric mean

This bidirectional normalisation helps prevent any single sample from dominating the attention.

A - Combined attention matrix (positives on left, negatives on right):

Finally, we split back into positive and negative attention:

A_pos = A[:, :N_pos]  # [N, N_pos]
A_neg = A[:, N_pos:]  # [N, N_neg]

A_pos - Attention to positives	A_neg - Attention to negatives (self masked)

4. Cross-weighting

Finally, we scale the positive weights by the total repulsion strength, and the negative weights by the total attraction strength:

We sum the negative weights and scale each positive weight:

# attraction scaled by repulsion strength
W_pos = A_pos * np.sum(A_neg, axis=1, keepdims=True)

And sum the positive weights to scale each negative weight:

# repulsion scaled by attraction strength
W_neg = A_neg * np.sum(A_pos, axis=1, keepdims=True)

This cross-weighting ensures that if you're strongly attracted to positives, you're also strongly repelled from negatives (and vice versa). It maintains the balance needed for the anti-symmetric property.

W_pos - Cross-weighted positives	W_neg - Cross-weighted negatives

5. Compute Drift Vectors

These weights are used to compute weighted averages of the positive and negative samples:

drift_pos = W_pos @ y_pos  # [N, D] - weighted sum of positive samples
drift_neg = W_neg @ y_neg  # [N, D] - weighted sum of negative samples

The final drift vector is the difference - attraction toward positives minus repulsion from negatives:

V = drift_pos - drift_neg

When the generated distribution matches the data distribution, the attraction and repulsion cancel and $\mathbf{V} = 0$ . The paper shows this is a necessary condition for equilibrium (though the converse is not strictly proven).

The Anti-Symmetric Property

The theoretical foundation of the method rests on the anti-symmetry property that the drift field satisfies.

$\mathbf{V}_{p,q}(\mathbf{x}) = -\mathbf{V}_{q,p}(\mathbf{x})$

Where:

$p$ is the target data distribution (positive samples) and
$q$ is the generated distribution (negative samples)

Swapping $p$ and $q$ flips the sign of the drift, which means that when $q = p$ (the generated distribution matches the data distribution), we have $\mathbf{V}_{p,p} = -\mathbf{V}_{p,p}$ , which means $\mathbf{V} = 0$ .

With this equilibrium in place, the model naturally stops drifting once it matches the target distribution.

Multi-Temperature

A single temperature may not capture both local and global structure. The paper uses multiple temperatures summed together (Appendix A.6):

temperatures = [0.02, 0.05, 0.2]
V_total = sum(compute_V(x, y_pos, y_neg, t) for t in temperatures)

Lower temperatures (0.02) draw sharp attention to the nearest neighbours. Higher temperatures (0.2) spread attention more broadly. Combining them gives a richer gradient signal.

Classifier-Free Guidance

Drifting models support CFG naturally. Given a class label $c$ , the positive samples become $\mathbf{y}^+ \sim p_{\text{data}}(\cdot|c)$ . For guidance, mix in real samples from other classes as extra negatives:

$\tilde{q}(\cdot|c) = (1-\gamma) q_\theta(\cdot|c) + \gamma \, p_{\text{data}}(\cdot|\varnothing)$

where $p_{\text{data}}(\cdot|\varnothing)$ is the unconditional distribution (samples from any class). This property is baked into training: the model learns to extrapolate from the unconditional distribution toward the class-conditional one.

Mode Collapse Robustness

Unlike GANs, drifting models are robust to mode collapse. Figure 3 in the paper shows that even when $q$ is initialised collapsed onto a single mode, the method recovers the full distribution, because if $q$ collapses to one mode, the positive samples from other modes of $p$ still attract generated samples toward them. The attraction from missing modes pulls samples out of the collapsed state.

Results

They achieve 1.54 FID on ImageNet 256×256 with a single forward pass (1-NFE), outperforming all previous single-step methods and somewhat competitive with multi-step diffusion models.

In pixel space (no VAE), they achieve 1.61 FID - also state-of-the-art for one-step pixel-space generation.

Method	Space	NFE	FID
DiT-XL/2	latent	250	2.27
Drifting Model, L/2	latent	1	1.54
iMeanFlow-XL/2	latent	1	1.72
Drifting Model, L/16	pixel	1	1.61

Deng, M., Li, H., Li, T., Du, Y., & He, K. (2026). Generative Modeling via Drifting (Version 2). arXiv. https://doi.org/10.48550/ARXIV.2602.04770 ↩

Software Factory

2026-02-08T00:00:00+10:00

Software Factory ¹ refers to the idea of completely abandoning the notion of writing code, even reviewing it, leaving engineers to manage the agent's goal and validate the correctness of the system. Effectively, developers become system-level QA engineers, writing specs and unblocking agents.

StrongDM defines a set of principles for the Software Factory approach ², which requires thinking in terms of "The Loop": the model starts with a seed, iterates, validates, receives feedback, and continues until all the "holdout scenarios" pass. With that paradigm, the question for the engineer is how to structure the problem so that each criterion can be validated (without return-true-in-tests cheating) and can receive meaningful feedback to guide its further development. StrongDM goes so far as to build "Digital Twin Universes": entire replicas of all the tools it integrates with, such as Jira, Okta, and Google Sheets, which it uses to exhaustively test its scenarios.

For a software engineer who's made a living writing software by hand for many years, I can't help but feel anxious when I hear people talking about developing software like this. Clearly, my industry is changing. But I do see some inevitability in this paradigm, or at least some of the ideas. Because, even before I knew it was a thing, I've been seeing it as a natural way to solve problems in the Opus 4.5+ era of agentic software development.

For example, I wanted an MLX (basically, PyTorch for Apple Silicon hardware) version of Demucs, one of the best audio stem-splitting models available. Without knowing a term for it, more just because I was lazy, I set up a software factory. I gave the model context of the reference implementation (the original PyTorch implementation), and the validation scenario: given a range of audio inputs I selected, the MLX outputs must match the PyTorch outputs within a numerical tolerance. I gave the agent an IMPLEMENTATION_NOTES.md file (see Spec-First LLM Development) to serve as both the plan and working memory, to be updated as it goes. I reviewed the initial plan to check that we agreed on the success criteria, then left it to work. And it did work. The results are here.

In another project, I wanted to integrate Pocketsmith (my budgeting tool) into OpenClaw as a "skill". I pointed the agent at several existing OpenClaw skills as concrete exemplars (which StrongDM calls "Gene Transfusion"), gave it the Pocketsmith API documentation, provided a basic indication of how I wanted to use it, told it how to plan and track its tasks, and let it work. The verification was mainly just me testing it and making sure I was satisfied with the interface it had constructed. This skill is just for my own personal use, so I'm not so worried about exhaustive verification - the surface for things to go disastrously wrong is limited. The results are here. I also integrated my preferred share portfolio tool, Sharesight, using a similar approach. See sharesight-skill.

Dan Shapiro ³ discusses 5 levels of software autonomy, with Dark Software Factory at Level 5, where software is a "black box that turns specs into software". I might not quite be operating at Shapiro's level 5 here - I still find myself butting in on the agent's work with my opinions about code quality - but I can certainly see the path towards it.

For my actual work at Canva, our users depend on us getting things exactly right, and currently, engineers must own AI-generated code as if they wrote every line themselves, so we're not doing Dark Software Factory anytime soon. But agentic coding is a fact of life today. Even if the code itself is carefully peer-reviewed at the end, there are definitely lessons to take away: how can I make sure the agent has all the context it needs with exemplar references? How can I enable it to validate at every stage of the implementation, and how can I provide feedback to guide its work? How can I be as exhaustive as possible in all the testing scenarios, removing any means of cheating?

The present of engineering is becoming more about reviewing code than writing it. But the future of engineering might be more about exhaustive verification of your system's correctness, and not much about the actual code at all.

Discussion on Bluesky, LinkedIn or Mastodon.

Cover by Homa Appliances on Unsplash

References

Moynihan, L. (2024, December). The Software Factory. LukePM.com. ↩
StrongDM. (n.d.). StrongDM Software Factory. ↩
Shapiro, D. (2026, January). The Five Levels: From Spicy Autocomplete to the Dark Factory. ↩

OpenClaw: the missing piece for Obsidian's second brain

2026-01-29T00:00:00+10:00

I've been an Obsidian user for many years. I like it a lot. I like the paradigm of linking notes that comes from Zettelkasten. I like having a single place to keep all my notes, to track progress on my goals and so forth.

My workflow is pretty simple. I just create a new Daily Note every day; it contains my todo list and notes I take throughout the day. I also use Permanent Notes to keep track of ideas and link them to daily notes, I also use these notes to help me learn and memorise new concepts. When I have a note that I feel is meaty enough, I'll move it to a public repo which is published on my blog, so I kind of have a system for blogging, albeit I end up with this pretty odd thing, that's somewhere in between a blog and a half-baked personal Wiki. I also have folders for tracking people in my life (particularly birthdays, gift ideas, and various things I want to remember about them), as well as using them for life admin tasks, like managing house repairs, insurance, and so on.

There are aspects of my life that end up outside of Obsidian. I have been using a ChatGPT project to track calories and workouts. I use the Oura Ring for sleep and step tracking. Information about my house is scattered across emails, Dropbox, and files. My wife and I use Google Sheets for finance tracking. I also use Claude and Gemini for various projects, and for things like recipes. I would love to have all this information in Obsidian. I like the freedom of completely owning my data and not being locked into any particular vendor. But, mostly it's just too cumbersome to copy and paste data into Obsidian, and I haven't had the time or energy to figure out how to automate it.

Which is where OpenClaw comes in.

OpenClaw

Recently, a lot of the tech-internet has been talking about a 2-week-old AI assistant project called OpenClaw (which, per the lore, was originally called Warelay, then Clawdbot, then Moltbot - hopefully OpenClaw will be the final name). Sure, it's just a cron wrapper with an LLM integration and some plugins, but there's a lot that can be done with that. To me, this feels like a first look at the next phase of agents: proactivity. Instead of just using AI reactively: give it some prompts and get an answer, OpenClaw can do things in the background, check in on you, give you reminders and suggestions.

But for me, this really feels like the missing piece of the puzzle for Obsidian to live up to its claim of being a "second brain".

I'm only a few days into it, but here are some notes about my setup and what I've been using it for so far.

Basic Setup

If you're using OpenClaw for the first time, I recommend just installing it and using the setup wizard: openclaw onboard. It's really good. Don't worry about getting it set up perfectly right away; just talk to it and work with it to set it up the way you want.

My OpenClaw lives on an old MacBook that I already have running 24/7 for a few various tasks. I call my agent M because I set it up when it was called Moltbot, and I cbf thinking too hard about it.

I communicate with M via WhatsApp. I created a new WhatsApp account by purchasing a ~$10 SIM card at the supermarket to get a new number, and used an old phone from the cupboard to verify the account.

You can also chat directly with OpenClaw via terminal with openclaw tui.

The OpenClaw workspace stores its memory and information about the tools available to it, alongside its soul file. By default, that workspace lives in ~/.openclaw/workspace, but I've put it in my Obsidian vault under ~/obsidian-vault/openclaw so it's synced alongside my Obsidian files, and means I can update the OpenClaw workspace files like any other files in my vault.

OpenClaw puts config in ~/.openclaw, and to track changes, I turned that into a Git repo:

cd ~/.openclaw
echo -e "logs/\nmedia/" > .gitignore
git init
git add .
git commit -m "Init"

I also moved the cronpath into my workspace, so that my crons can be synced and managed alongside my other workspace files:

~/.openclaw/openclaw.json

"cron": {
    "enabled": true,
    "store": "~/obsidian-vault/openclaw/cron/jobs.json.md",
    "maxConcurrentRuns": 1
  }

Had to give the json file a suffix of md to get it to sync - oh well.

Security

You can run openclaw security audit for a security audit. Some notes about my configuration:

I only accepts DMs from me. I've set allowFrom to only my phone number, so I'm the only one who can message M directly. I do have plans to give my wife access at some point.
Make sure to configure loopback Gateway binding (gateway.bind) unless you know what you're doing I have gateway.bind set to loopback.
Double-check file permissions are well locked down: I've set ~/.openclaw/ to 700 and config files to 600, so only my user can read them. You can run openclaw doctor to automatically check and fix permissions.

Describing your Obsidian vault to OpenClaw

Initially, you don't need to configure anything with OpenClaw manually. It's usually easiest to just chat with it what you want and let it update its own files. Eventually, once you have it pretty close, you might find yourself wanting to fine-tune it by hand.

I have my Obsidian vault explained to OpenClaw in my TOOLS.md file, which has been updated through a combination of conversations with OpenClaw, manual edits and Claude Code for major refactors. Here's the first few lines of it so you get a sense:

# TOOLS.md - Local Notes

## Obsidian Vault

My workspace is `openclaw/` inside Lex's Obsidian vault. Access vault files via `../`.

- **Key files:**
  - `../todo.md` — current tasks with deadlines
  - `../finances/finances-summary.md` — property, stocks, crypto
  - `../health/health-summary.md` — health info and fitness routines
  - `../people/birthdays.md` — important birthdays
- **Daily notes:** `../daily/YYYY-MM-DD.md`
- **Subscriptions:** `../finances/subscriptions.md` — alert before renewals flagged for review
- **Projects:** `../projects/` and `../bachelors-degree-2022/cm3070-final-project/`
- **Sickness log:** `../health/conditions/sickness-log.md`

## Current Focus

**Source of truth:** `../todo.md`

Always read todo.md for current priorities, deadlines, and what's in progress. Don't maintain a separate list - todo.md is canonical.
# ...

Things I Use OpenClaw For

Tracking sleep and health

I have an Oura Ring, which I use for sleep and step tracking. I find it really useful. But, as I said, I also like the idea that my data is mine to keep. If I ever find another way to track sleep or steps that I prefer, I'm free to move. So it's nice to make the source-of-truth for that data my existing Obsidian vault.

A key feature of OpenClaw is skills, which are essentially Markdown and scripts that give OpenClaw (or any agent that supports them) new capabilities. There's already a skill for the Oura ring by kesslerio.

And I get M to fetch my sleep and steps via the Heartbeat, and put it into my Daily Notes.

I also weigh myself each morning, message OpenClaw to save it to my daily note, and I have aggregate notes that show me my progress over time.

I used to use DataView to aggregate this data, but it makes more sense to have a script generate the summary so OpenClaw can easily track my progress.

Tracking Workouts and Diet

When I work out, I share various metrics with OpenClaw, and encourage it to keep me consistent with things like my workout routine, flexibility routines and daily step goals.

I'm in a cut phase of my health journey, and I will take a picture of what I eat to track calories, aiming to stay under 2200 per day until the cut finishes.

I have been using ChatGPT projects for this before, and while it works well, I'd rather have the sense that I have full ownership of my data.

I find having M helping me track my food and workouts, and goals keeps me consistent - like having a coach checking in on my progress.

Todos and Life Admin

My todo.md is the source of truth for what I'm working on, it includes priorities and deadlines. Beyond that, I've been getting M to help me track all sorts of life admin:

Subscriptions, especially ones I intend to cancel
Insurance policies and claims
Birthdays for family and friends
Taxable purchases and deductions
Property repairs and contractor details
My dog's medication, vet visits, and registration
Possessions I've bought (with receipts)
Recipes, with a journal of modifications as I improve them

There's a lot. You can also give M access to your Gmail, Calendar and Docs via the gog skill, which has proven to be quite useful, albeit it a little unsettling. Use at your own risk.

Morning Routine

The morning routine is where it all comes together. I have a cron job that runs each morning to create my daily note. M reads my to-do list, checks for upcoming birthdays, fetches my Oura sleep data, and sends me a WhatsApp message with a summary and suggestions for what to focus on.

When I weigh myself, I message M, and it logs the weight, fetches yesterday's steps, and gives me a quick health summary. It's a nice way to start the day - I get a snapshot of my priorities without having to dig through files myself.

Let AI Design the System

In the past, I'd get stuck trying to design the perfect Obsidian system before actually using it. I'd spend so long planning the "ultimate" setup that I never ended up with anything.

Now I just tell the LLM what I want to track and let it figure out the structure. It updates its own memory as it goes. For bigger refactors across many files, I use Claude Code instead of OpenClaw.

On Costs

OpenClaw really has become an indispensable part of my Obsidian and life management strategy. However, it comes at a pretty huge price (literally). All these heartbeats, and skill running and file editing eats up a lot of tokens, especially using the recommended model Opus 4.5. Not even going to mention how much I've spent on it so far - it's too much. But, I'm hoping once I have the system built, day-to-day operations will become a lot cheaper. Plus, the cost of LLM intelligence tends to track downward over time. So hopefully it will be financially sustainable.

Edit: since writing this article I've switched to using GPT-5.4 Codex via my existing ChatGPT Plus subscription ($20 USD/month). I will admit that it's a bit of a downgrade from Opus, but it's good enough.

Discussion on Hacker News, Bluesky or Mastodon.

Spec-First LLM Development

2026-01-19T00:00:00+10:00

Spec-First LLM Development is the simple idea that, instead of asking an LLM to immediately output code after prompting, you first ask it to output a spec file, which is continually updated as you and the LLM develop software together. The spec file is stored in version control and should be re-read by the LLM whenever new context is added.

Luke Bechtel writes a really useful article called Vibe Specs: Vibe Coding That Actually Works that has more details about how it looks in practice. Also, in a Hacker News thread, antirez, creator of Redis, shared his approach, using IMPLEMENTATION_NOTES.md in his flux2.c project. He says: "This project was possible only once I started to tell Opus that it needed to take a file with all the implementation notes, and also accumulating all the things we discovered during the development process. And also, the file had clear instructions to be taken updated, and to be processed ASAP after context compaction. This kinda enabled Opus to do such a big coding task in a reasonable amount of time without losing track"

Another really interesting solution to this problem comes from Steve Yegge in his project beads, which is a much more engineered approach that builds a dependency-aware graph of tasks. It looks like Claude is going to implement a similar solution in Claude Code called Tasks.

It's nothing new. The concept of Software Requirements Specifications (SRS) was commonplace as early as 1975 ¹. However, I suspect I haven't seen many of these in my career because keeping them up to date with the pace of software change was just too difficult an undertaking. Now that we have LLMs to do it for us, the story is very different.

It seems that a lot of the paradigms for writing high-quality software, that in practice often get pushed aside due to them being deemed too labour intensive - SRS being one obvious example, but other practices like Test-Driven Development and Property-based Testing - start to sound a lot more sensible when it's the LLM doing the hard labour, and the engineer reaping the rewards in the form of reliable software.

Personally, I've had some success adding implementation notes to my toolkit for recent projects. I've been working on porting some PyTorch projects to the MLX framework (see mlx-demucs, mlx-contentvec and mlx-rvc for recent examples), and anecdotally, it seems to prevent the LLM from getting stuck in loops, especially across context compactions. It also seems to help prevent it from making the same kinds of errors, like failing to use uv or using the wrong Python version for comparison testing, working better than instructions in AGENTS.md or CLAUDE.md instructions alone.

That said, a text file feels like a pretty rudimentary solution to Memory for an LLM - I'm sure there's going to be a lot more exploration in this space.

Discussion on Bluesky, Mastodon and Mastodon.

References

Ramamoorthy, C. V., & Ho, S. F. (1975). Testing large software with automated software evaluation systems. ACM SIGPLAN Notices, 10(6), 382–394. https://doi.org/10.1145/390016.808461 ↩

Title As Link Text: automatic note titles in Obsidian links

2025-11-01T00:00:00+10:00

I've built Title As Link Text, a new Obsidian plugin that automatically updates your links to use the note's title instead of the filename.

If you use a Zettelkasten-style system or just have notes with slugged filenames, you end up with links that look like this:

[[20230408102501]]
[document-name](./complex-topic.md)

When what you actually want is:

[[20230408102501|My Awesome Note]]
[Understanding Complex Topics](./complex-topic.md)

The plugin handles both Wikilinks and Markdown links, and updates them automatically whenever you save or rename a file. It plays nicely with other plugins that make Obsidian play nice with Markdown links, like Frontmatter Title and Wikilinks to MDLinks.

How it finds the title

It checks three things in order:

A frontmatter property (default: title)
The first H1 heading in the file
The filename itself as a fallback

So if you have a title in your frontmatter, that wins. If not, it looks for a # Heading. If neither exists, it just uses the filename as-is.

You can also configure which frontmatter property to use, so if you prefer name, heading, or something custom, that works too.

Alias support

It also handles aliases. If a note has aliases defined in frontmatter and a link text matches one of them (even partially or fuzzily), the plugin leaves it alone rather than overwriting it with the title. You can tune the similarity threshold or turn alias matching off entirely if you don't want it.

Installation

It's available as a Community Plugin.

Open Settings > Community Plugins > Browse
Search for Title As Link Text
Click Install

There are also two commands available if you want to do a one-off update rather than relying on auto-save: "Update all links" and "Update links for current file".

See the project GitHub for more details.

CBIS-DDSM Mammography Dataset

2025-10-18T00:00:00+10:00

CBIS-DDSM (Curated Breast Imaging Subset of DDSM) ¹ is a Mammography dataset for Computer-Aided Detection (CADe) and Computer-Aided Diagnosis (CADx), derived from an earlier dataset, the Digital Database for Screening Mammography (DDSM) ². This note comes from background research for my Breast Cancer Detection BSc final project.

The idea of CBIS-DDSM was to provide a standardised mammography dataset towards an ImageNet for mammography. Though a few mammography datasets already existed: the DDSM itself ², the Mammographic Imaging Analysis Society (MIAS) database ³, and the Image Retrieval in Medical Applications (IRMA) project ⁴, they were limited by accessibility and data quality issues.

DDSM was already a promising dataset for this purpose, comprising 2620 scanned mammography studies from multiple hospitals. It contains ROI annotations and Breast Imaging Reporting and Data System (BI-RADS) labels for a series of Mammography studies, along with extensive metadata. However, it had several problems: inaccurate region-of-interest annotations, personal health information in some images, and an obsolete file format (LJPEG).

The authors of the CBIS-DDSM subset stripped the dubious annotations and the examples containing PII. They also wrote a conversion tool for LJPEG and converted the images into TIFF files, which are then stored as DICOM files, the standard for medical images, to create CBIS-DDSM.

They also included convenience images, including the region-of-interest mask and the cropped region. You can see an example of it later in the article.

Finally, they improved the accuracy of the existing region-of-interest annotations by applying the Chan-Vese Algorithm, initialised with the original contours, but only to the mass examples, not the calcifications. In the figure below, in red, the original annotations; in blue, some example annotations created by physicians; and in green, the annotations derived from the Chan-Vese model, which clearly improve on the original annotations.

I'm going to walk through how the dataset works in this rendered notebook.

The dataset can be downloaded from the Cancer Imaging Archive.

I downloaded the Images dataset to the ~/datasets/CBIS-DDSM folder, which uncompresses into the ./CBIS-DDSM directory.

The image dataset is a 164GB compressed dataset, which uncompresses to around 180GB.

from pathlib import Path

import matplotlib.pyplot as plt

import pandas as pd
import pydicom

pd.set_option('display.max_colwidth', None)

DATASET_ROOT = Path("/Users/lex/datasets/CBIS-DDSM")

!cd {DATASET_ROOT} && du -sh *.csv CBIS-DDSM

512K	calc_case_description_test_set.csv
1.0M	calc_case_description_train_set.csv
512K	mass_case_description_test_set.csv
1.0M	mass_case_description_train_set.csv
3.0M	metadata.csv
180G	CBIS-DDSM

2 files are provided for each split, representing either calcification or mass abnormalities found in the breast.

calc_case_description_{train|test}_set.csv
mass_case_description_{train|test}_set.csv

train_mass_df = pd.read_csv(DATASET_ROOT / "mass_case_description_train_set.csv")
train_mass_df.head(1).T

	0
patient_id	P_00001
breast_density	3
left or right breast	LEFT
image view	CC
abnormality id	1
abnormality type	mass
mass shape	IRREGULAR-ARCHITECTURAL_DISTORTION
mass margins	SPICULATED
assessment	4
pathology	MALIGNANT
subtlety	4
image file path	Mass-Training_P_00001_LEFT_CC/1.3.6.1.4.1.9590.100.1.2.422112722213189649807611434612228974994/1.3.6.1.4.1.9590.100.1.2.342386194811267636608694132590482924515/000000.dcm
cropped image file path	Mass-Training_P_00001_LEFT_CC_1/1.3.6.1.4.1.9590.100.1.2.108268213011361124203859148071588939106/1.3.6.1.4.1.9590.100.1.2.296736403313792599626368780122205399650/000000.dcm
ROI mask file path	Mass-Training_P_00001_LEFT_CC_1/1.3.6.1.4.1.9590.100.1.2.108268213011361124203859148071588939106/1.3.6.1.4.1.9590.100.1.2.296736403313792599626368780122205399650/000001.dcm\n

Before we get to the metadata, let's take a look at some major bugs with the provided CSV.

Addressing Inconsistent Image Mappings

As you can see, each row in the CSV files contains references to 3 DICOM files:

image file path - the full mammogram
ROI mask file path - binary mask of the region of interest
cropped image file path - cropped region containing the abnormality

However, the DICOM filenames don't match the files downloaded from the Images dataset in a quite confusing way.

The CSVs reference files like 000000.dcm or 000001.dcm, but the actual files are named 1-1.dcm or 1-2.dcm. Even worse, the mapping between these naming conventions is inconsistent. Additionally, some entries in the ROI mask file path column incorrectly point to cropped images rather than actual binary masks.

Thankfully, Andrés Sarmiento created a tool that fixes these issues by interrogating the mask files to determine if they're crops or masks, and correcting the filepath references. The corrected CSV files are available as a HuggingFace dataset.

I downloaded the correct CSV to ~/datasets/CBIS-DDSM/fixed-csv, and the rest of the notebook will use it accordingly.

Train / Test Data

As mentioned, the training and test data are split by the abnormality type present in the scan: calcification or mass. I find it easiest to combine the training into a single file:

train_mass_df = pd.read_csv(DATASET_ROOT / "fixed-csv" / "mass_case_description_train_set.csv")
train_calc_df = pd.read_csv(DATASET_ROOT / "fixed-csv" / "calc_case_description_train_set.csv")
train_df = pd.concat([train_mass_df, train_calc_df])
train_mass_df = train_calc_df = None
train_df.head(1).T

	0
patient_id	P_00001
breast_density	3.0
left or right breast	LEFT
image view	CC
abnormality id	1
abnormality type	mass
mass shape	IRREGULAR-ARCHITECTURAL_DISTORTION
mass margins	SPICULATED
assessment	4
pathology	MALIGNANT
subtlety	4
image file path	Mass-Training_P_00001_LEFT_CC/1.3.6.1.4.1.9590.100.1.2.422112722213189649807611434612228974994/1.3.6.1.4.1.9590.100.1.2.342386194811267636608694132590482924515/1-1.dcm
cropped image file path	Mass-Training_P_00001_LEFT_CC_1/1.3.6.1.4.1.9590.100.1.2.108268213011361124203859148071588939106/1.3.6.1.4.1.9590.100.1.2.296736403313792599626368780122205399650/1-2.dcm
ROI mask file path	Mass-Training_P_00001_LEFT_CC_1/1.3.6.1.4.1.9590.100.1.2.108268213011361124203859148071588939106/1.3.6.1.4.1.9590.100.1.2.296736403313792599626368780122205399650/1-1.dcm
breast density	NaN
calc type	NaN
calc distribution	NaN

We do the same for the test set:

test_mass_df = pd.read_csv(DATASET_ROOT / "fixed-csv" / "mass_case_description_test_set.csv")
test_calc_df = pd.read_csv(DATASET_ROOT / "fixed-csv" / "calc_case_description_test_set.csv")
test_df = pd.concat([test_mass_df, test_calc_df])
test_mass_df = test_calc_df = None
test_df.head(1).T

	0
patient_id	P_00016
breast_density	4.0
left or right breast	LEFT
image view	CC
abnormality id	1
abnormality type	mass
mass shape	IRREGULAR
mass margins	SPICULATED
assessment	5
pathology	MALIGNANT
subtlety	5
image file path	Mass-Test_P_00016_LEFT_CC/1.3.6.1.4.1.9590.100.1.2.416403281812750683720028031170500130104/1.3.6.1.4.1.9590.100.1.2.245063149211255120613007755642780114172/1-1.dcm
cropped image file path	Mass-Test_P_00016_LEFT_CC_1/1.3.6.1.4.1.9590.100.1.2.259596319110047779433501728143778409887/1.3.6.1.4.1.9590.100.1.2.30820586311062570442302321942433426184/1-2.dcm
ROI mask file path	Mass-Test_P_00016_LEFT_CC_1/1.3.6.1.4.1.9590.100.1.2.259596319110047779433501728143778409887/1.3.6.1.4.1.9590.100.1.2.30820586311062570442302321942433426184/1-1.dcm
breast density	NaN
calc type	NaN
calc distribution	NaN

Then combine both splits for analysis:

all_df = pd.concat([train_df, test_df])

The metadata.csv file maps the CSV path references to actual file locations on disk:

metadata_df = pd.read_csv(DATASET_ROOT / "metadata.csv")
metadata_df.head(1).T

	0
Series UID	1.3.6.1.4.1.9590.100.1.2.419081637812053404913157930753972718515
Collection	CBIS-DDSM
3rd Party Analysis	NaN
Data Description URI	https://doi.org/10.7937/K9/TCIA.2016.7O02S9CY
Subject ID	Calc-Test_P_00038_LEFT_CC_1
Study UID	1.3.6.1.4.1.9590.100.1.2.161465562211359959230647609981488894942
Study Description	NaN
Study Date	08-29-2017
Series Description	ROI mask images
Manufacturer	NaN
Modality	MG
SOP Class Name	Secondary Capture Image Storage
SOP Class UID	1.2.840.10008.5.1.4.1.1.7
Number of Images	2
File Size	14.06 MB
File Location	./CBIS-DDSM/Calc-Test_P_00038_LEFT_CC_1/08-29-2017-DDSM-NA-94942/1.000000-ROI mask images-18515
Download Timestamp	2025-11-25T05:30:27.736

CBIS-DDSM contains 1,566 studies.

images_per_patient = all_df.groupby("patient_id").size()
print(f"Total patients: {len(images_per_patient)}")

Total patients: 1566

They separate the studies by the abnormality type present, either "mass" or "calcification".

The paper¹ claims there are 891 mass cases, although the actual dataset appears to have 892 mass abnormalities.

len(all_df[all_df["abnormality type"] == "mass"].patient_id.unique())

The paper also describes 753 calcification abnormalities that match what we see.

len(all_df[all_df["abnormality type"] == "calcification"].patient_id.unique())

We know that a mammogram consists of 2 images per breast: a craniocaudal (CC) view from above and a mediolateral oblique (MLO) view from the side.

fig, ax = plt.subplots(figsize=(10, 5))
images_per_patient.value_counts().sort_index().plot(kind='bar', ax=ax, color='#3498db', edgecolor='black')
ax.set_title('Distribution of Images per Patient', fontsize=12, fontweight='bold')
ax.set_xlabel('Number of Images')
ax.set_ylabel('Number of Patients')
for i, v in enumerate(images_per_patient.value_counts().sort_index().values):
    ax.text(i, v + 2, str(v), ha='center', va='bottom')
plt.tight_layout()
plt.show()

Looking at the distribution of images per patient, about 1005 patients have both views, while many have only a single image, which is basically an incomplete mammogram (containing only one view per patient, instead of the expected two).

Fetching Images

Even with the corrected CSVs, we still need to do a few things to look up the DICOM images for each study.

The file paths in the CSV (e.g. Mass-Training_P_00001_LEFT_CC/1.3.6.1.4.1.9590.100.1.2.422112722213189649807611434612228974994/1.3.6.1.4.1.9590.100.1.2.342386194811267636608694132590482924515/1-1.dcm) aren't direct paths to the files on disk. They encode metadata: subject ID, study UID, series UID, and filename. We need to parse these components and cross-reference with metadata.csv to find the actual file location.

from pydantic import BaseModel

class DCMData(BaseModel):
    subject_id: str
    study_uid: str
    series_uid: str
    dcm_file: str

def get_file_data_from_dcm(dcm_path: str) -> DCMData:
    """Parse DICOM path string to extract metadata components."""
    data = str(dcm_path).strip().split("/")
    dcm_og = data[-1].strip().split(".")[0]
    return DCMData(
        subject_id=data[0],
        study_uid=data[1],
        series_uid=data[2],
        dcm_file=dcm_og,
    )

def get_filepath_from_dcm_data(dcm_data: DCMData) -> Path:
    """Look up actual file path from metadata using DCM data."""
    meta_row = metadata_df[
        (metadata_df["Subject ID"] == dcm_data.subject_id)
        & (metadata_df["Series UID"] == dcm_data.series_uid)
        & (metadata_df["Study UID"] == dcm_data.study_uid)
    ].iloc[0]
    file_location = meta_row["File Location"]
    return DATASET_ROOT / Path(file_location) / (dcm_data.dcm_file + ".dcm")

Now we can load the DICOM images using pydicom:

def dicom_to_array(file_path: Path):
    """Load a DICOM file and return pixel array."""
    ds = pydicom.dcmread(file_path)
    return ds.pixel_array

Let's load an example patient to see all three image types (full mammogram, ROI mask, and cropped region):

patient_df = all_df[all_df.patient_id == "P_00065"]
row = patient_df[patient_df["image view"] == "CC"].iloc[0]

img_path = get_filepath_from_dcm_data(get_file_data_from_dcm(row["image file path"]))
mask_path = get_filepath_from_dcm_data(get_file_data_from_dcm(row["ROI mask file path"]))
crop_path = get_filepath_from_dcm_data(get_file_data_from_dcm(row["cropped image file path"]))

original_img = dicom_to_array(img_path)
mask_img = dicom_to_array(mask_path)
crop_img = dicom_to_array(crop_path)

Visualising the original mammogram, the binary ROI mask, an overlay of the two, and the cropped abnormality region:

fig, axes = plt.subplots(1, 4, figsize=(12, 5))

axes[0].imshow(original_img, cmap="gray")
axes[0].axis("off")

axes[1].imshow(mask_img, cmap="gray")
axes[1].axis("off")

axes[2].imshow(original_img, cmap="gray")
axes[2].imshow(mask_img, cmap="jet", alpha=0.4)
axes[2].axis("off")

axes[3].imshow(crop_img, cmap="gray")
axes[3].axis("off")

plt.tight_layout()
fig.subplots_adjust(top=0.92, wspace=0.01)

titles = ["Original", "ROI Mask", "Overlay", "Cropped ROI"]
for ax, title in zip(axes, titles):
    x = ax.get_position().x0 + ax.get_position().width / 2
    fig.text(x, 0.98, title, ha="center", va="top", fontsize=12, fontweight="bold")
plt.show()

Key Metadata

The most important label is pathology, indicating whether the abnormality is benign, benign_without_callback (clearly no risk of malignancy), or malignant.

pathology_counts = all_df['pathology'].value_counts()

plt.figure(figsize=(8, 5))
plt.bar(pathology_counts.index, pathology_counts.values, color=['#2ecc71', '#3498db', '#e74c3c'])
plt.title('Pathology Distribution', fontsize=12, fontweight='bold')
plt.ylabel('Count')
plt.xticks(rotation=45)
for i, v in enumerate(pathology_counts.values):
    plt.text(i, v + 5, str(v), ha='center', va='bottom')
plt.tight_layout()
plt.show()

The dataset also includes Breast Imaging Reporting and Data System (BI-RADS) assessment categories (0-6) that indicate the level of suspicion. The distribution shows most cases fall into categories 4 and 5 (suspicious/highly suggestive of malignancy), which makes sense given that this is a dataset specifically curated around abnormalities.

assessment_counts = all_df['assessment'].value_counts().sort_index()

plt.figure(figsize=(8, 5))
plt.bar(assessment_counts.index.astype(str), assessment_counts.values, color='#9b59b6')
plt.title('BI-RADS Assessment Distribution', fontsize=12, fontweight='bold')
plt.xlabel('Assessment Category')
plt.ylabel('Count')
for i, (idx, v) in enumerate(assessment_counts.items()):
    plt.text(i, v + 5, str(v), ha='center', va='bottom')
plt.tight_layout()
plt.show()

Mass Descriptors

For mass abnormalities, the dataset includes shape and margin descriptors - clinically important features for diagnosis:

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

shape_counts = all_df['mass shape'].dropna().value_counts()
axes[0].barh(shape_counts.index, shape_counts.values, color='#e67e22')
axes[0].set_title('Mass Shape Distribution', fontsize=12, fontweight='bold')
axes[0].set_xlabel('Count')

margins_counts = all_df['mass margins'].dropna().value_counts()
axes[1].barh(margins_counts.index, margins_counts.values, color='#16a085')
axes[1].set_title('Mass Margins Distribution', fontsize=12, fontweight='bold')
axes[1].set_xlabel('Count')

plt.tight_layout()
plt.show()

Irregular shapes and spiculated margins are typically more concerning for malignancy, while oval/round shapes with circumscribed margins tend to be benign.

Breast Density

Breast density (1-4 scale) affects mammogram interpretation - denser tissue makes abnormalities harder to detect:

density_col = 'breast_density' if 'breast_density' in all_df.columns else 'breast density'
breast_density_counts = all_df[density_col].value_counts().sort_index()

plt.figure(figsize=(8, 5))
plt.bar(breast_density_counts.index.astype(str), breast_density_counts.values, color='#2980b9')
plt.title('Breast Density Distribution', fontsize=12, fontweight='bold')
plt.xlabel('Breast Density (1=fatty, 4=extremely dense)')
plt.ylabel('Count')
for i, v in enumerate(breast_density_counts.values):
    plt.text(i, v + 5, str(v), ha='center', va='bottom')
plt.tight_layout()
plt.show()

Train/Test Split

The authors provide standardised train/test splits (80/20), stratified by BI-RADS assessment to ensure similar difficulty distribution:

Set	Benign Cases	Malignant Cases
Calcification Training	329 (552 abnormalities)	273 (304 abnormalities)
Calcification Test	85 (112 abnormalities)	66 (77 abnormalities)
Mass Training	355 (387 abnormalities)	336 (361 abnormalities)
Mass Test	117 (135 abnormalities)	83 (87 abnormalities)

Note that case counts differ from abnormality counts since some cases contain multiple abnormalities.

Segmentation Quality

The authors validated their Chan-Vese segmentations against hand-drawn ROIs from an experienced radiologist on 118 images. The mean Dice coefficient between computer-generated and hand-drawn ROIs was 0.792 ± 0.108, compared to 0.398 ± 0.195 for the original DDSM annotations vs hand-drawn. These results represent a statistically significant improvement (Wilcoxon signed-rank test, p = 5.54 × 10⁻¹⁹).

During curation, 339 mass images where the lesion was not clearly visible were removed after review by a trained mammographer.

Limitations

While CBIS-DDSM is valuable for research, it has some limitations worth noting. The original DDSM images were digitised from film mammograms, not acquired digitally. Modern Full-Field Digital Mammography (FFDM) systems produce higher-quality images, and newer datasets like InBreast and VinDr-Mammo tend to contain these sorts of images. Additionally, DDSM images are focused on a specific abnormality, but a breast may contain multiple abnormalities, warranting investigation. Lastly, the original DDSM was collected in the 1990s, so imaging quality and patient demographics may differ from those in contemporary datasets.

Despite these limitations, CBIS-DDSM remains one of the most widely used public mammography datasets for developing and benchmarking CAD algorithms.

References

Lee, R. S., Gimenez, F., Hoogi, A., Miyake, K. K., Gorovoy, M., & Rubin, D. L. (2017). A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific Data, 4(1), 170177. https://doi.org/10.1038/sdata.2017.177 ↩
Heath, M., Bowyer, K., Kopans, D., Moore, R. & Kegelmeyer, W. P. The Digital Database for Screening Mammography. Proceedings of the Fifth International Workshop on Digital Mammography 212–218 (2001). Available at http://marathon.csee.usf.edu/Mammography/software/HeathEtAlIWDM_2000.pdf ↩
Suckling, J. et al. The Mammographic Image Analysis Society digital mammogram database. Exerpta Medica 375–378 (1994). http://peipa.essex.ac.uk/info/mias.html ↩
Lehmann, T. M. et al. IRMA—Content-based image retrieval in medical applications. Methods Inf. Med. 43, 354–361 (2004). ↩

Mammography

2025-10-15T00:00:00+10:00

As part of my final project for my BSc, I worked on Breast Cancer Detection. This note was made while doing background research for that topic. See topic #BreastCancerDetection

X-ray Mammography is a breast cancer screening method and remains one of the more effective population-wide tools for early detection of Breast Cancer ¹. The breast is compressed between two plates to spread the tissue and reduce motion blur evenly. A typical screening exam records two views of the breast: Caniocaudal (CC), a top-to-bottom view, and Mediolateral (MLO), a side view.

When reading a mammogram, a radiologist looks for specific abnormalities: masses, calcifications, distortion of breast tissue, or asymmetries when comparing the two breasts and two views.

To standardise reporting, radiologists use the Breast Imaging Reporting and Data System (BI-RADS), which assigns a category from 0 to 6 indicating the level of suspicion:

0 - incomplete: additional imaging needed.
1 - negative.
2 - benign.
3 - probably benign: short-interval follow-up suggested.
4 - suspicious: biopsy should be considered.
5 - highly suggestive of malignancy.
6 - known biopsy with proven malignancy.

Categories 4 and 5 typically warrant a biopsy to confirm or rule out cancer.

Radiologists also classify breast composition by density using four categories:

A - almost entirely fatty
B - scattered fibroglandular densities
C - heterogeneously dense
D - extremely dense

Higher breast density both increases breast cancer risk and reduces mammographic sensitivity, as dense tissue appears white on mammograms (the same appearance as potential tumours), effectively masking lesions.

Modern screening often utilise Digital Breast Tomosynthesis (DBT), or "3D mammography." Unlike standard 2D mammography, DBT captures multiple X-ray projections from different angles to reconstruct the breast in "slices." This minimises the effect of overlapping tissue, improving detection rates in dense breasts.

On of the canonical Mammography datasets is CBIS-DDSM Mammography Dataset.

References

Misra, S., Solomon, N. L., Moffat, F. L., & Koniaris, L. G. (2010). Screening criteria for breast cancer. Advances in Surgery, 44, 87–100. https://doi.org/10.1016/j.yasu.2010.05.008 ↩

SPARQL

2025-09-16T00:00:00+10:00

SPARQL (or SPARQL Protocol and RDF Query Language) is a Query Language for RDF databases and Triplestores.

It uses pattern-based queries. ph databases have no fixed starting points of order. Use variables beginning with question marks (e.g., ?friend, ?name). It also defined how to communicate with SPARQL endpoints over HTTP - it's not just a query language.

Example Queries

All of these queries can be run live against https://dbpedia.org/sparql

Let's start with a query to fetch basic info about the city of London. We can define variables using a ? prefix. Then in our query we can describe a pattern which matches RDF Triples where the Subject is London and any predicate or object matching.: <http://dbpedia.org/resource/London> ?property ?value .

Also, for readability, we'll convert http://dbpedia.org/resource/ into a prefix:

PREFIX dbr: <http://dbpedia.org/resource/>

SELECT ?property ?value
WHERE {
    dbr:London ?property ?value .
} LIMIT 20

Challenge: Can you list basic information about Sydney, Australia?

Answer:

PREFIX dbr: <http://dbpedia.org/resource/>

SELECT ?property ?value
WHERE {
    dbr:Australia ?property ?value .
} LIMIT 20

We define much more complex patterns for the data we want to fetch by providing multiple triples statements. The RDFS schema also contains a number of useful predicates, like rdf:type (which is used to say subject isA object) and rdfs:label is a human readable name for a resource.

Here's a query that gets all the cities in the UK (but only the English labels):

PREFIX dbo: <http://dbpedia.org/ontology/>
PREFIX dbr: <http://dbpedia.org/resource/>

SELECT ?city ?name
WHERE {
    ?city dbo:country dbr:United_Kingdom .
    ?city rdf:type dbo:City .
    ?city rdfs:label ?name .
    FILTER (lang(?name) = 'en')
}

You can rewrite the above query is saving having to use the ?city prefix on each line as follows. Also, an alias for rdf:type is a:

PREFIX dbo: <http://dbpedia.org/ontology/>
PREFIX dbr: <http://dbpedia.org/resource/>

SELECT ?city ?name
WHERE {
    ?city dbo:country dbr:United_Kingdom ;
          a dbo:City ;
          rdfs:label ?name .
    FILTER (lang(?name) = 'en')
}

Challenge: using dbo:TelevisionShow as the type, and using dbo:genre dbo:Science_fiction return all science fiction TV Shows from the produced or created in Australia.

Answer:

PREFIX dbo: <http://dbpedia.org/ontology/>
PREFIX dbr: <http://dbpedia.org/resource/>

SELECT ?show ?name
WHERE {
    ?show dbo:genre dbr:Science_fiction ;
          dbo:country dbr:Australia ;
          a dbo:TelevisionShow ;
          rdfs:label ?name .
    FILTER (lang(?name) = 'en')
}

Filtering

We can use the FILTER call to filter by certain values. For example, to find all cities in the world that have more than 10_000_000 people, and you can ORDER BY DESC(?population)

PREFIX dbo: <http://dbpedia.org/ontology/>

SELECT ?city ?name ?population
WHERE {
    ?city a dbo:City ;
          rdfs:label ?name ;
          dbo:populationTotal ?population .
    FILTER (lang(?name) = 'en')
    FILTER (?population > 10000000)
}
ORDER BY DESC(?population)
LIMIT 10

Challenge: the album genre of rock music is dbp:genre dbr:Rock_music, the property of release year is available in dbp:relYear and the rdf:type is dbo:Album. Can you list all the rock albums released after 2010?

Answer:

PREFIX dbo: <http://dbpedia.org/ontology/>
PREFIX dbr: <http://dbpedia.org/resource/>
PREFIX dbp: <http://dbpedia.org/property/>

SELECT ?album ?name ?relyear
WHERE {
    ?album a dbo:Album ;
            dbp:genre dbr:Rock_music ;
            rdfs:label ?name ;
            dbp:relyear ?relyear .
    FILTER (lang(?name) = 'en')
    FILTER (?relyear > 2015)
}
ORDER BY DESC(?relyear)
LIMIT 10

RDF

2025-09-14T00:00:00+10:00

RDF (or Resource Description Framework) is a data model for representing data as a graph. RDF is the foundational technology for the Semantic Web, which was Tim Berners-Lee's vision for a machine-readable web, once described as Web 3.0 (unrelated to the crypto bros blockchain-powered Web3). RDF is also the backbone of a standard for interconnecting datasets called Linked Data.

Though the Semantic Web never fully materialised, partially due to its reliance on users to input correct data ¹, the RDF standard does live on in multiple forms. Many social media sites and feed readers also support various RDF serialisation standards, such as JSON-LD, to read metadata on websites ⁴. I've also seen it deployed in organisations for taxonomy management, such as maintaining databases of keywords and their relationships (e.g., if an item is a Daffodil, it's also a Flower, which means it's also a Plant).

It's worth taking the time to wrap your head around RDF, although it can seem a little cumbersome at first.

In RDF, data points are defined as triples in the form: <subject> <predicate> <object>.

Jumping straight into an example, here's how I might represent information about myself in this form:

<Lex> <is a> <person>
<Lex> <has occupation> <Software Engineer>
<Lex> <has pet> <Doggo>

Then let's also define some triples for my dog, Doggo:

<Doggo> <is a> <dog>
<Doggo> <has breed> <Staghound>
<Doggo> <is aged> 6

A key detail of RDF is that Subjects and Predicates must be represented as an IRI (Internationalised Resource Identifier), of which URLs are a subset. Objects can be IRIs, but can also be literals: strings, numbers, dates, etc.

These IRIs serve as globally unique identifiers for resources (the "R" in RDF). For instance, I could describe myself using my website's URL and reference schema.org's standardised definition of a person, so that other people would know I was a person.

<https://notesbylex.com/Lex> <rdf:type> <http://schema.org/Person>

This IRI-based approach provides two powerful capabilities. First, it enables shared vocabularies across the internet: different datasets can reference the same definitions, ensuring everyone means the same thing by "Person" or "owns." Second, it allows easy creation of new vocabularies, whether public standards or private organisational schemas.

RDF's graph structure also enables logical reasoning across the data.

For example, given the information:

<Aspirin> <inhibits> <COX-2 enzyme>
<COX-2 enzyme> <produces> <Prostaglandins>
<Prostaglandins> <mediate> <Inflammation>

We can infer that:

<Aspirin> <reduces> <Inflammation>

The technique of representing information in a way that allows us to perform logical inference over it originates from a branch of AI known as Knowledge Representation.

RDF Serialisation

There are multiple ways to serialise RDF data, each with different advantages:

N-Triples

The simplest serialisation format, expressing each triple on a separate line using full URIs. While it is verbose, it's also easy to parse and process programmatically.

The example earlier was an of how I might express myself using N-Triples format:

<https://example.org/person#lex> <https://example.org/hasName> "Lex" .
<https://example.org/person#lex> <https://example.org/hasOccupation> "software engineer" .
<https://example.org/person#lex> <https://example.org/hasPet> <https://example.org/dog#Doggo> .
<https://example.org/dog#Doggo> <https://example.org/hasName> "Doggo" .
<https://example.org/dog#Doggo> <https://example.org/hasBreed> "Staghound" .
<https://example.org/dog#Doggo> <https://example.org/hasAge> 6 .

You'll notice that I've used http://example.org as the prefix for my subjects and predicates. Since RDF mandates the use of URIs for these attributes, http://example.org is a reserved domain name specifically designated for use in documentation and examples (see RFC 2606).

Turtle (Terse RDF Triple Language)

Turtle (or Terse RDF Triple Language) is a human-readable RDF serialisation format. It includes features like:

Prefixes to reduce redundant URL prefixes.
A semicolon (;) to keep the same subject, continues with a new predicate
A comma (,) keeps the same subject and predicate, adds a new object.
"a" shortcut - replaces "is of type".
Language tags to specify labels in different languages.

Example:

@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix ex:   <https://example.org/> .

ex:person#lex
    a foaf:Person ;
    foaf:name "Lex" ;
    ex:hasOccupation "software engineer" ;
    ex:hasPet ex:dog#Doggo .

ex:dog#Doggo
    a ex:Dog ;
    foaf:name "Doggo" ;
    ex:hasBreed "Staghound" ;
    ex:hasAge 6 .

Turtle is one of the most common RDF serialisation approaches, and most RDF databases tend to support it.

RDFa

RDFa allows RDF data to be embedded directly into HTML markup using special attributes, which enables web pages to contain machine-readable structured data alongside human-readable content. RDFa is useful for search engine optimisation and semantic web apps.

RDFa attributes:

about - specifies the subject of the RDF statements.
property - creates a literal value relationship.
rel - create a resource relationship.
resource - specifies the object resource.
typeof - declares the RDF type of the subject.
datatype - specifies the data type of literal values.

Example:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:foaf="http://xmlns.com/foaf/0.1/"
      xmlns:ex="https://example.org/">
      <head>
          <title>Person and Pet Profile</title>
      </head>
      <body>
          <div about="https://example.org/person#lex" typeof="foaf:Person">
              <h1 property="foaf:name">Lex</h1>
              <p>Occupation: <span property="ex:hasOccupation">software engineer</span></p>
              <p>Pet ownership:</p>
              <div about="https://example.org/dog#Doggo" typeof="ex:Dog" rel="ex:hasPet">
                  <h2 property="foaf:name">Doggo</h2>
                  <p>Breed: <span property="ex:hasBreed">Staghound</span></p>
                  <p>Age: <span property="ex:hasAge" datatype="xsd:integer">6</span> years old</p>
              </div>
          </div>
      </body>
</html>

JSON-LD

JSON-LD expresses RDF using familiar JSON syntax while maintaining full RDF compatibility. A good format for working with web applications since it uses common JSON syntax while maintaining full RDF compatibility. Search engines actively use JSON-LD for processing Schema.org structured data.

Keywords of JSON-LD:

@context - defines namespace prefixes and mappings.
@id - specifies the subject URI.
@type - declares the RDF type.
@value and @type - for typed literal values.
@graph - contains an array of linked data objects.

{
    "@context": {
        "foaf": "http://xmlns.com/foaf/0.1/",
        "ex": "https://example.org/",
        "xsd": "http://www.w3.org/2001/XMLSchema#"
    },
    "@graph": [
        {
            "@id": "ex:person#lex",
            "@type": "foaf:Person",
            "foaf:name": "Lex",
            "ex:hasOccupation": "software engineer",
            "ex:hasPet": {
                "@id": "ex:dog#Doggo"
            }
        },
        {
            "@id": "ex:dog#Doggo",
            "@type": "ex:Dog",
            "foaf:name": "Doggo",
            "ex:hasBreed": "Staghound",
            "ex:hasAge": {
                "@value": 6,
                "@type": "xsd:integer"
            }
        }
    ]
}

RDF/XML

One of the first approaches to RDF serialisation, using XML syntax, makes it more verbose but useful for systems already processing XML, but mostly considered less readable than Turtle or JSON-LD formats.

rdf:about specifies the subject URI
rdf:resource creates relationships to other resources
rdf:datatype specifies data types for literal values
Nested elements represent predicates and objects

<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:foaf="http://xmlns.com/foaf/0.1/"
         xmlns:ex="https://example.org/">
    <foaf:Person rdf:about="https://example.org/person#lex">
        <foaf:name>Lex</foaf:name>
        <ex:hasOccupation>software engineer</ex:hasOccupation>
        <ex:hasPet rdf:resource="https://example.org/dog#Doggo"/>
    </foaf:Person>

    <ex:Dog rdf:about="https://example.org/dog#Doggo">
        <foaf:name>Doggo</foaf:name>
        <ex:hasBreed>Staghound</ex:hasBreed>
        <ex:hasAge rdf:datatype="http://www.w3.org/2001/XMLSchema#integer">6</ex:hasAge>
    </ex:Dog>
</rdf:RDF>

Blank Nodes

RDF also supports blank nodes (or anonymous nodes) for representing resources without URIs. These are useful when you need to describe something but don't need to give it a permanent identifier. For example, representing an address without creating a URI for it:

ex:person#lex ex:hasAddress [
    a ex:Address ;
    ex:street "123 Main St" ;
    ex:city "Springfield" ;
    ex:zipCode "12345"
] .

RDF Vocabulary

RDF provides the structural foundation, but vocabularies define the actual meaning of the data. The RDF ecosystem includes several foundational vocabularies and extensions:

Core RDF Vocabularies

RDF - The core Resource Description Framework vocabulary that provides basic terms like rdf:type, rdf:Property, and rdf:Statement for describing the fundamental structure of RDF data.
RDF Schema - Extends RDF with terms for defining classes (rdfs:Class), properties (rdfs:Property), subclass relationships (rdfs:subClassOf), and domain/range constraints (rdfs:domain, rdfs:range).
Web Ontology Language (OWL) - A more expressive vocabulary built on RDF and RDFS that adds complex logical constructs like owl:equivalentClass, owl:disjointWith, and owl:inverseOf for creating sophisticated ontologies and enabling automated reasoning.

Common Application Vocabularies

FOAF (Friend of a Friend) - for describing people and relationships.
Dublin Core - for metadata about resources.
Schema.org - for structured data on web pages.

Ecosystem

While RDF provides the data model, several extensions provide additional capabilities.

Web Ontology Language (OWL) - A more expressive language built on RDF for creating complex ontologies and reasoning.
Triplestores - Specialised databases designed to store and efficiently query RDF triple data.
SPARQL - the standard query language for retrieving and manipulating RDF data, similar to SQL for relational databases.

(2018, May 27). Whatever happened to the semantic web? Two-Bit History. https://twobithistory.org/2018/05/27/semantic-web.html ↩
Webber, J. (n.d.). RDF vs. property graphs: Choosing the right approach for implementing a knowledge graph. Neo4j. https://neo4j.com/blog/knowledge-graph/rdf-vs-property-graphs-knowledge-graphs/ ↩
King, R. D., Rowland, J., Oliver, S. G., Young, M., Aubrey, W., Byrne, E., ... & Sparkes, A. (2009). The automation of science. Science, 324(5923), 85-89. ↩
Paterson, C. (2024, August 20). Being on the semantic web is easy, and, frankly, well worth the bother. csvbase. https://csvbase.com/blog/13 ↩

Genetic Algorithms

2025-09-07T00:00:00+10:00

Genetic Algorithms are an optimisation technique inspired by natural selection.

The high-level idea is that we start from a population of possible solutions to a problem, where, importantly, we have a way to evaluate how good each solution is - called a fitness function. Then we perform a loop that involves selecting from the fittest solutions (with randomness), breeding these solutions together, creating mutations, and replacing the population with the new solutions.

The solutions need to be represented in a way that allows us to breed and mutate solutions. Typically, this is an array, where each element represents some parameter of a solution. For example, for testing robot designs, we might have elements that correspond to how many joints they have, and another that describes joint length, and so on. This representation is called a Genotype, a term borrowed from biology, where all living organisms have a genetic code that stores their information, encoded in DNA. The expression of the genotype, for example, the robot that is constructed as a result of the information in the array, is called a Phenotype. In this example, we might utilise a fitness function that tests the robot's ability to complete a task, or to simply move without falling.

The algorithm looks like this:

Initialisation: Create a random population of candidate solutions genotypes.
Fitness: Evaluate the fitness of each individual in the population.
Selection: Select "parents" based on their fitness.
Crossover: also known as "Breeding", where parents create a new "offspring" by combining the genotype in some way.
Mutation: the offspring is mutated in different ways using algorithms that involve randomness.
Replacement: replace some or all of the population with the new offspring.
Repeat 2-6 until the termination condition is met.

One of the most famous examples comes from Evolving Virtual Creatures by Karl Sims, where he evolved virtual creatures in a simulated 3D environment. In his groundbreaking work, Sims used genetic algorithms to simultaneously evolve both the morphology (body structure) and neural networks (brain/control systems) of virtual creatures.

It has also recently been used within AlphaEvolve, where DeepMind utilised it to explore the solution space for several mathematical and scientific problems. For solutions encoded as software, they utilised LLMs for the code generation and modification tasks, including combining and modifying solutions.

Fitness Function

A fitness function evaluates the population in some way. In the Virtual Creatures example, the fitness is based on how far they travel under different conditions. In AlphaEvolve, the fitness is a test specific to the problem it is trying to solve, designed to find an optimal solution.

Selection

Selection algorithms determine which individuals will become parents for the next generation, giving fitter individuals a higher chance of reproduction while maintaining some diversity. There are a series of common selection algorithms typically used.

Roulette Wheel

When selections are based on the proportion of fitness across the population, think about a roulette wheel as a pie chart, where each pie is proportional to its fitness as a whole. The more fit take up more space, so they're more likely to be selected, but there's also some entropy to ensure the solution space is explored.

Calculate the fitness of all individuals
Generate a random number between 0 and the whole fitness.
Select the individual whose cumulative fitness exceeds that of others.

Tournament Selection

A simpler approach, where we select some individuals at random from the population and choose one of the highest-fitness individuals from each sample.

Rank Selection

Individuals are ranked by fitness, and the selection probability is based on rank rather than raw fitness values. This approach is particularly helpful when fitness values exhibit large variations or when dealing with negative fitness values.

Crossover

Crossover, also known as breeding, is where we combine the genetic material of the two selected parents to create a new offspring. The goal is to create new solutions that potentially inherit the best features from both parents while exploring new areas of the solution space. Different crossover methods are suited to different types of problems and solution representations.

One-Point Crossover

A single crossover point is chosen randomly. Everything before this point comes from Parent 1, and everything after comes from Parent 2.

For example:

Parent 1: $[1, 0, 1, 1, 0, 0, 1]$
Parent 2: $[0, 1, 0, 0, 1, 1, 0]$
Child: $[1, 0, 1, \ \textcolor{red}{\bf{|}} \ 0, 1, 1, 0]$

Two-Point Crossover

Two (or N) crossover points are selected, and the genetic material between these points is swapped between parents.

Uniform Crossover

For each gene position, randomly choose which parent to inherit from (typically with 50% probability for each parent). This approach provides more mixing than point-based methods.

Order Crossover (OX)

Specifically designed for permutation problems like the Travelling Salesman Problem, where each gene must appear exactly once. In this example, we can switch around nodes (if it generates a valid graph).

Mutation

In the mutation step, we modify the child solution slightly, allowing us to maintain genetic diversity and continue exploring the solution space. There are many mutation algorithms.

Bit-flip

For binary representations, each bit has a small probability (typically 1-5%) of being flipped from 0 to 1 or vice versa.

Gaussian

For real-valued genes, add a random value drawn from a Gaussian (normal) distribution with mean zero and a small standard deviation.

Formula: $\text{new gene} = \text{old gene} + \mathcal{N}( 0, \sigma )$

Swap Mutation

For permutation problems, randomly select two positions and swap their values.

Insertion Mutation

Remove a gene from one position and insert it at another random position.

Advantages and Limitations

There are several advantages to genetic algorithms over other AI approaches, such as supervised learning. Firstly, no training data is required. If we can construct a fitness function and encode the problem effectively, then we can run a Genetic Algorithm search over the solution space. It can also handle discrete, continuous and mixed variable types. Since each solution is evaluated independently, it is highly parallelisable, allowing cores to be allocated to evaluate one or a subset of the solutions, so it is good when the search space is huge. It is also robust to noisy/weird objectives and can typically escape local optima.

On the other hand, there's no guarantee of finding a global optimum; typically, many parameters need to be tuned, which can be computationally expensive and can easily lead to premature convergence.

Schema Theorem

For the theoretical foundation of Genetic Algorithms see Schema Theorem.

Optimising Computation At The Token-Level

2025-07-21T00:00:00+10:00

Really interesting paper from researchers at KAIST AI, Mila and Google: "Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation"

They build on the Recursive Transformer, where layers (or groups of layers) can be applied repeatedly during decoding, allowing us to reduce the overall layer count of a network. They add a router to each stack of layers, which assigns the number of repeats (or recursion depth) per token.

This routing strategy means that simple tokens (think: "and", "the", etc) only need to traverse a small recursive depth, and more content-rich tokens can be assigned larger recursion depths, effectively allowing the model to optimise computation at the token level. The routing is a form of Latent Reasoning, where the model can effectively think at each layer group within the network.

They also introduce a "recursion-wise key/value caching" mechanism, where key-value pairs are cached based on corresponding recursion.

Interestingly, although the MoR learns to assign recursive depth to each token during training, at inference time, the depth can be optionally fixed, enabling a form of test-time scaling that trades compute for performance.

I feel that this has the potential to dramatically decrease LLM parameter counts, continuing the tradition of democratising AI.

They experiment with multiple-layer configuration strategies:

Cycle: Layers reused cyclically (e.g., [0,1,2,0,1,2,0,1,2])
Sequence: Same layer repeated before moving to next (e.g., [0,0,0,1,1,1,2,2,2])
Middle-Cycle/Middle-Sequence: Keep unique first and last layers, share only middle layers

Best performance: Middle-Cycle strategy

They also try two different routing strategies:

Expert-choice: Each recursion depth selects its top-k tokens. Guarantees a fixed compute budget but requires auxiliary loss to avoid causality violation
Token-choice: Each token is assigned a fixed recursion depth upfront. No causality issues, but needs load balancing

Best performance: Expert-choice with auxiliary loss and linear router

And two different K/V caching strategies:

Recursion-wise caching: Only cache K/V pairs for tokens active at each recursion depth. Reduces memory by ~50%
Recursive sharing: Reuse K/V pairs from the first recursion for all subsequent steps. Maximum memory savings but slight accuracy drop

Best performance: Recursion-wise caching for expert-choice routing

Absurdly Good Doggo Consistency with FLUX.1 Kontext

2025-06-01T00:00:00+10:00

This new image editing model from Black Forest Labs called FLUX.1 Kontext is really good. You can read more about it here.

Although GPT Image 1 is still one of the best image models out there, it is pretty limited in its ability to edit: characters get lost, and there's usually unrelated changes returned in the images.

On the other hand, FLUX.1 Kontext, thanks to approach of Flow Matching in latent space, maintains a high-quality level of text-to-image quality but with an absurdly good ability to edit photos. One remarkable thing is that it can maintain character consistency through many edits (called multi-turn editing). Even without the context of the chain of images, or any sort of in-painting, I found that it was able to keep a source character consistent, even after many rounds of editing.

To demonstrate the character consistency, I found the most recently taken photo of my dog, Doggo.

Doggo recently had TPLO surgery on each of her legs in two separate staggered surgeries due to some painful arthritis. She is fully recovered now and doing much better.

However, for the first two weeks after surgery, our poor puppy had to wear a giant cone to prevent her chewing off her stitches, which she hated.

Let's see if Kontext can remove the cone from poor Doggo's head.

"remove the cone from my dog's head"

Much better.

That works so well. Of course, if you look close enough, you can see the artifact where the cone was, but that's my dog, alright.

Again, the amazing thing about the Kontext model is its ability to do multi-turn editing. Using the most recent output as input, let's see if we can make my poor Dog look like her usual happy self:

"make my dog look happy"

Admittedly, I had to try this prompt a few times before I got something that looked convincing like this. One of them made her head too big, and the other did some weird stuff with her ears.

Anyway, that's pretty happy! I don't think she's ever quite smiled like that, but it's close.

Now, to maximise her happiness, I move her to one of her favourite places in the world:

"change the background to a sunny beach scene."

I'm really impressed by the cast shadows. The model has figure out where it wants the sun to be, and can generate shadows at roughly match that model. Wild.

Finally, to achieve peak happiness, I put her favourite chew toy next to her, a red deer antler.

"add an antler bone in front of her"

Even after four rounds of editing, it still looks exactly like Doggo - really impressive stuff.

She looks so happy; I think this could be a birthday card.

My nephew's birthday is coming up, and he loves Minecraft. So I'll try turning it into a Minecraft-themed bday.

Change background to minecraft. Write "Happy Birthday, Nephew" in bright, colorful text on top of the image.

Looking pretty good, albeit a little distorted.

I wonder if I can use the same photo for all my future greeting card needs. It's June, but still, it's never too early to be planning Christmas. It is coming into winter in the Southern Hemisphere, after all. Isn't Christmas in July a thing?

change text to "Seasons Greetings" with a Christmas font. Convert into a snowy background. Remove antler. Add a snowman next to her. Add a Christmas hat on top.

Amazing. Now we're seven edits deep, and it still looks like my Doggo. Granted, we've started to see some artifacts, and there's some roughness around the edges, but this is looking good.

Kontext also excels at style transfer, so let's try a few different styles for the Christmas Card.

"Convert into the style of a 90s Christmas Movie poster"

"Convert into a Vintage Storybook Style Christmas Card."

"Convert into a watercolor illustrated Christmas Card."

"change into the style of The Simpsons"

Few interesting variations on the text, but very impressive nonetheless.

Back to the original Christmas Card, the Kontext paper demonstrates even more incredible global edits, like adding multiple characters, and rotating camera angles.

Figure 1: Consistent character synthesis with FLUX.1 Kontext by Black Forest Labs

Let's experiment with some of that.

Remove text. There are now two dogs driving in a pink convertible.

I mean, that kind of works. I think it's still my dog in the driver's seat. Hard to tell whether the snowman or Doggo is driving the car, but between them I assume they've got it covered.

I tried the prompt "Watch them from behind.", which is actually given as an example in the paper. That was immediately flagged as NSFW and refused. Kontext, you definitely are misunderstanding me.

I tried an alternate prompt.

"turn the camera to watch them from the back"

Not sure exactly what's going on here, but it has turned the car around, but not any of the character. Starting to get a little terrifying, feeling a bit like Loab might be waiting a few turns down the line, so I'll stop.

Now, the showcase of all the edits:

FLUX.1: Kontext - Character Consistency

Source image	"remove the cone from my dog's head"	"make my dog look happy"	"change the background to a sunny beach scene"
"add an antler bone in front of her"	"Change background to Minecraft. Write 'Happy Birthday, Nephew' in bright, colorful text"	"Seasons Greetings, snowy background, snowman, Christmas hat, remove antler"	"Two dogs driving in a pink convertible"

It's a very impressive model indeed. We can contrast the same sequence of turns with GPT Image 1, which the paper reports as the second best performing model for character consistency.

Figure 9: Image-to-image evaluation on KontextBench by Black Forest Labs

GPT Image 1 - Character Consistency

Source image	1. "remove the cone from my dog's head" (gpt-image-1)	2. "make my dog look happy" (gpt-image-1)	3. "change the background to a sunny beach scene"
4. "add an antler bone in front of her"	5. "Change background to minecraft. Write 'Happy Birthday, Nephew' in bright, colorful text on top of the image."	6. "change text to 'Seasons Greetings'. Convert into a snowy background. Remove antler. Add a snowman next to her. Add a Christmas hat on top."	7. "Remove text. There are now two dogs driving in a pink convertible."

Still an incredible model, but we can see clearly that even by the second image it's a totally different dog.

Black Forest Labs cooked with this one, as the kids would say.

Learning to Reason without External Rewards

2025-05-28T00:00:00+10:00

Another very interesting paper on the topic of reward modelling, that continues to turn assumptions on their head about how to train reasoning LLMs.

This paper proposes a new RL training method that doesn't rely on reward models (RLHF) like ChatGPT, selected high-quality ground truth examples (RLVR) like DeepSeek R1, or even verifiable generated data (Self-Play) like AbsoluteZero, to develop complex reasoning capability (thinking mode).

Turns out we can just use the model's internal sense of confidence as the reward signal to train a reasoning model 🤯

The authors take GRPO (from Deepseek R1) and replace the verifiable reward signal with a "self-certainty score" (specifically the KL divergence between the model's output distribution and a uniform distribution), then train on a collection of prompts relative to the evaluated domains. They demonstrate reasoning performance comparable to GRPO on math while exceeding it on coding tasks.

They introduce a new paradigm, "Reinforcement Learning from Internal Feedback" (RLIF) and call their approach INTUITOR.

If this approach scales, it could unlock the ability to generally improve reasoning LLMs in domains where high-quality ground truth isn't available or not verifiable (i.e., any domain other than math and code) and provide evidence that pre-trained LLMs "possess richer latent behavioural priors than previously recognised."

Interesting, indeed.

John Carmack is working on game-playing robots

2025-05-23T00:00:00+10:00

John Carmack shared his notes from a recent talk at an AI conference about the research he and his team of six are conducting at his new AGI company, Keen Technologies.

I'm a big fan of Carmack. I grew up on his games, from Commander Keen to Quake. I love his approach to engineering, especially after reading Masters of Doom about the early years of id Software.

Keen Technologies is revisiting a seemingly largely solved problem: playing Atari Games.

In 2013, Deepmind described the first deep-RL approach to playing Atari games that achieved human-level performance just from reading pixels (on some games, at least). Then, in 2020, Deepmind returned and smashed their own record by beating nearly all Atari games to human-level performance in their Agent57 approach. See Playing Atari with Deep Reinforcement Learning.

So you might think Atari is completely solved, so what's left to research? Well, has it been solved with robots??

"A reality check for people that think full embodied AGI is right around the corner is to ask your dancing humanoid robot to pick up a joystick and learn how to play an obscure video game."

Carmack's team has built a system that uses servo motors to control a physical controller, which can learn to play Atari games in real-time, using just camera input as a signal.

I think the intersection of AI research and hardware will be a boon for the capability of embodied AI in general. Despite the rapid progress of LLM technology deployed to consumers, it's yet to meaningfully demonstrate improvements in embodied agents, even though vision understanding capabilities are getting so good.

The real-time aspect is really important: "Reality is not a turn-based game", he says. So far, all the prior work with game playing, and even with the LLM interactions we've seen, have assumed a turn-based approach. You write a prompt, the LLM writes a response, and so on; even with the LLMs that operate using voice, the experience is largely the same. Say some things, and wait for a response, then say something else, which is a different style of conversation to the one we'd have with a person, with pauses, and interruptions, etc. Some of the boundaries to human-like conversations are clearly in the hardware.

Secondly, he's choosing not to focus on pre-trained LLMs, putting him at odds with nearly everyone else in AI right now.

"I believe in the importance of learning from a stream of interactive experience, as humans and animals do."

I think there's likely a lot of value in continuing to explore techniques outside of LLMs, as there's a distinct possibility that the progress of AI will be stuck in a local minima for a while, with so much investment and effort thrown into LLMs, just because of much juice is clearly left to extract from them. It's good to have someone so talented and driven thinking about other possibilities. Still, I don't see why LLMs couldn't be trained on a big enough corpus of robot game-playing that they could demonstrate the same capability that Carmack is going for. This point he does acknowledge, with a references to Sutton's Bitter Lesson. Incidentally, Sutton joined the team at Keen technologies in 2023.

Anyway, who am I to question Carmack? Let him cook.

There's a lot of other really interesting details in there. I highly recommend reading the notes.

Imagen 4 is faster, but GPT is still the GOAT

2025-05-21T00:00:00+10:00

Yesterday, Google announced a bunch of new generative tools at the ongoing Google I/O event. Although, at the time of writing, they're mostly unavailable outside of the USA:

Veo 3: state-of-the-art video generation that can generate videos with audio.
Veo 2: updated to include camera controls, outpainting, and adding and removing objects.)
Flow AI: a filmmaking tool that appears to be used to create videos with continuity between clips.
SynthID Detector: a tool for detecting the SynthID watermark that Google inject in all generated content.
Lyria 2: an updated version of the music generation tool.
Imagen 4: This promises a number of improvements to Imagen 3, including supporting up to 2k resolution and improved typography.

Imagen 4 seems to be the only thing available to me right now in Australia, and only via the Gemini app. However, when generating images in Gemini, they don't mention the model name in the image outputs - I'm not sure if I'm testing Imagen 3 or 4. Imagen 4 is publicly available on Fal.ai; however, as a preview, I'm not sure I quite understand the logic behind that, but I do appreciate how complicated shipping products at Google's scale is.

I did a quick test of some prompts to compare Imagen 4 to OpenAI's gpt-image-1, which is currently the SOTA for image gen.

TL;DR gpt-image-1 is still the greatest for overall quality and prompt adherence, but Imagen 4 is about 5- 10x faster. They are both incredible image generators.

"A group of Mercians, from the Kingdom of Essex in the year 500, sit around a table showing each other a picture on an iPhone."

Imagen 4	gpt-image-1
Gen time: 10.21s	Gen time: 62.00s

In terms of aesthetic quality and prompt adherence, gpt-image-1 is the clear winner. The phone is turned around the wrong way in the Imagen 4 version, and it feels more AI-slop-esque. However, you can see that Imagen 4 ran about 6x faster.

"A group of young women, in the 1920s, watch as one of them flies a drone in the streets of Brooklyn, NYC."

Imagen 4	gpt-image-1
Gen time: 9.6s	Gen time: 47.25s

In this example, I would say this is one rare example of Imagen 4 being better, although both are really good.

"Leonardo da Vinci sits at a HP laptop, prompting ChatGPT with the text: 'portrait of a woman with a mysterious smile, folded hands, and a soft, mountainous background', the Mona Lisa is shown on screen."

Imagen 4	gpt-image-1
Gen time: 8.00s	Gen time: 58.70s

Again, while both are incredible, gpt-image-1 is nearly flawless. The text is a tiny bit off, but overall, it captured exactly the intention, and looks pretty amazing. Imagen 4 leans more towards the AI-slop aesthetic, reminiscent of earlier versions of DALL-E.

I saw a nice paper last year that did some investigation into the poor performance of rare "concepts" in AI image datasets (No 'Zero-Shot' Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance).

The thesis is that image classification and generation models perform much worse on rare concepts in the dataset. They created a dataset of rare concepts (at least rare in the LAION family of datasets) called Let It Wag. I made images with three rare concepts from the Let It Wag dataset: red-necked grebe, Globeflower and SR-20. Not sure how rare these are in OpenAI's or Google's dataset, but it gives some indication of how "rareness" affects the image outputs.

"A Red-necked Grebe sits next to a Globeflower while an SR-20 aircraft prepares for take off in the distance."

Imagen 4	gpt-image-1
Gen time: 8.57s	Gen time: 57.13s

I'm no expert on Globe Flowers, Red-necked Grebes or SR-20s, but while Imagen 4 does a really good job, it seems gpt-image-1 is better. But only marginally. These are really impressive results. Both models can do well on even extremely rare concepts.

For the last test, I tested a very specific prompt containing some complicated poses.

I enlisted GPT-4o's help to write this prompt:

A three-frame sequence showing Egyptian queen Cleopatra performing a Romanian deadlift using dumbbells with perfect form: Frame 1 — standing tall at the top of the movement, dumbbells at her thighs; Frame 2 — at the bottom position with a flat back and slight knee bend, weights just below the knees; Frame 3 — halfway up on the return, hips driving forward, maintaining strong posture and control. Photo realistic.

Imagen 4	gpt-image-1
Image gen: 5.60s	Image gen: 71.47s

gpt-image-1 got it usefully close. I'd say the form is nearly perfect, although Cleo goes from holding one and a half dumbbells to a barbell. I don't know what exercise Imagen 4 is doing, but it ain't what I had in mind.

Summary

I'd say gpt-image-1 wins in 4/5 of my simplistic tests, based on aesthetics and prompt adherence. However, Imagen 4 was sometimes more than 10x faster than gpt-image-1. I could see Imagen 4 being useful for rapid prototyping and exploration of ideas.

Vibe-Coding Mathematical Discoveries

2025-05-18T00:00:00+10:00

You may have heard of DeepMind's new whitepaper about their coding agent, AlphaEvolve. It discovered a new algorithm for multiplying matrices, improving a 56-year-old solution, and many other mathematical problems, including optimising Google's internal infrastructure scheduling algorithms.

AlphaEvolve builds on DeepMind's 2023 work called FunSearch, where the basic idea is to continuously improve LLM-generated solutions to problems expressed as programs (i.e. code gen), re-prompting with previously-generated solutions selected using an evolutionary algorithm. At a high level:

Start with a prompt describing a problem, including a code base skeleton, with markers to show where the LLM should modify the code (EVOLVE-BLOCK START / EVOLVE-BLOCK END)
Use an evaluation function that assesses the solution's correctness and measures other properties, such as runtime, simplicity, num operations, etc.
Create a program database, which can start empty or be seeded with known solutions.
Sample programs from the database, using an evolutionary selection strategy based on island models (see below).
Instruct a set of LLMs to generate new programs to improve the evaluation metrics.
Now evaluate the results (using a lot of $ worth of distributed computation), and store the most promising programs.
Repeat 5-6 until you have SOTA result.

The sampling algorithm is based on prior work on evolutionary algorithms (MAP-Elites and island models):

Solutions are clustered into "islands" based on performance characteristics.
Programs in islands evolve independently from other islands, and high-performance programs are selected as parents.
The LLM plays the breeder role, generating offspring programs by generating new programs seeded from the parents.
Information flows between the islands by "culling" the worst programs in the islands, replacing them with the best programs from the surviving islands

The main improvements on FunSearch are that a) it can evolve an entire codebase, not just a single function, and b) they have access to more powerful LLMs, from PaLM2 to Gemini Flash/Pro 2.0. Presumably, even more improvements can be made by utilising the latest generation of Gemini models (2.5 Pro, etc).

One interesting detail is that the open problems were suggested by mathematicians Javier Gomez Serrano and Terence Tao, who also helped formulate them as inputs. So, in a way, this might be the first example of vibe-coded mathematical discoveries.

Discussion on Linkedin.

NoProp: Training Neural Networks Without Back-Propagation or Forward-Propagation

2025-05-15T00:00:00+10:00

This paper proposes a "back-propagation-free" (kinda) approach to training a denoising (Diffusion / Flow Matching) model.

The main difference between the backprop approach and NoProp is that each block (layer) is optimised to denoise independently instead of propagating errors throughout the entire network, think: gradient descent per block. They also don't need to do a forward pass through the whole network. Each block is given a copy of the input data embeddings and a noised version of the label, meaning blocks can be sampled randomly throughout training, effectively allowing for much better ability to parallelise training.

At inference time, the noised label is replaced with the output of the preceding layer.

In theory, NoProp could reduce the memory needed to train models, as activations wouldn't need to be stored for all layers at once. Also, since it doesn't depend on the preceding block's output, nor does it have to propagate gradients through the whole network, it should be much easier to parallelise the network's training. A final selling point is that the model isn't necessarily learning a hierarchical representation, or at least is learning a different type of hierarchy than backprop.

Each layer is trained using the sum of multiple loss functions:

Cross-entropy loss on the final prediction.
KL divergence between the initial noise and standard normal.
L2 loss between the block output and the label embedding (scaled by SNR change).

They experiment with a few different versions, including using continuous time steps (NoProp-CT and NoProp-FM), instead of discrete blocks (NoProp-DT), similar to Flow Matching.

They test on MNIST, CIFAR-10, and CIFAR-100. There's not much to improve upon those problems in 2025, but they get comparable or even slightly better results than traditional backprop while using about half the GPU memory.

Absolute Zero: Reinforced Self-play Reasoning with Zero Data

2025-05-12T00:00:00+10:00

This paper introduces the Absolute Zero Reasoner.

The idea is to train a reasoning model (from a foundation Qwen-2.5-7B* model) without needing any human-annotated data. Previous DeepSeek-R1-Zero approach learned CoT reasoning just from input/output pairs, but this goes a step further and proposes the inputs and outputs to solve.

The proposing and solving steps both have relevant reward functions: they reward the model to propose examples that are the correct level of difficulty for it, and penalise wrong answers in the solving step.

The tasks it proposes and solves are either:

deduction (prediction output, given program and input)
abduction (prediction input, given program and output)
induction (predict/synthesis program, given input and output)

Of note, is that they also notice some potentially unsafe reasoning chains generated throughout the training, which they call an "uh-oh moment".

It's pretty wild that this works. In theory, we can improve reasoning capability of models by just training them longer with no additional data needed, although I'm a bit dubious about how far it can be pushed, given that the proposing step is still limited to the distribution of the foundation model's training data.

Playing Atari with Deep Reinforcement Learning

2025-05-05T00:00:00+10:00

My summary (still a draft) of the paper Playing Atari with Deep Reinforcement Learning by Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller

Overview

In late 2013, DeepMind researchers published a Reinforcement Learning approach to playing Atari games with AI called Deep Q-Network (DQN). This was one of the first approaches to game playing could successfully "learn control policies" (i.e. play games), only from observing the raw pixels from the game, thanks to recent advanced in deep learning.

This paper was published about one month before Google announced it would acquire DeepMind in January 2014.

In this article, I'm going to walk through the architecture they described in the paper, and attempt to implement it using MLX (which is a library I've been meaning to play with).

High-Level

The paper describes a CNN architecture that inputs game frames as raw pixels, and outputs predicted future rewards (i.e. the game score) for each available action (i.e. move up, move left, etc).

A naive approach to processing frames from the games could be to process a sequential batch of frames at each training step. However, these frames would be strongly correlated. Instead, they used a technique, from a 1993 attempt to train a robotic policy using neural networks, called Experience Replay, which effectively stores a history of game states, their corresponding score (rewards) from action taken, and samples from this each training step.

In practice, the agent plays randomly for a bit, gets the rewards throughout the game, and then uses a neural network to predict the rewards. As the model trains, the agent increasingly uses the model to take the optimal action, which balances exploiting and exploring.

The paper was also made possible thanks to the The Arcade Learning Environment (ALE), released a year prior, which provides an evaluation methodology and toolkit for testing RL agents in Atari games. They test 7 Atari 2600 games, outperforming all previous approaches, and human experts on three games.

Approach Detail

Architecture

They describe the model as follows:

The input to the neural network consists of an 84 × 84 × 4 image produced by $\phi$ . The first hidden layer convolves 16 8 × 8 filters with stride 4 with the input image and applies a rectifier nonlinearity. The second hidden layer convolves 32 4 × 4 filters with stride 2, again followed by a rectifier nonlinearity. The final hidden layer is fully connected and comprises 256 rectifier units. The output layer is a fully connected linear layer with a single output for each valid action. The number of valid actions varied between 4 and 18 in the games we considered.

So that's two conv layers with ReLU, followed by two fully-connected layers. In an MLX model, that might look like this:

import mlx.core as mx
import mlx.nn as nn

class DQN(nn.Module):
    def __init__(self, num_actions: int):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=4, out_channels=16, kernel_size=8, stride=4)
        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=4, stride=2)

        self.fc1 = nn.Linear(2592, 256)  # 9x9x32 = 2592
        self.fc2 = nn.Linear(256, num_actions)

    def __call__(self, x):
        # MLX expects height, width, channels, and needs float, not int, representations.
        x = x.transpose(0, 2, 3, 1) / 255.
        x = nn.relu(self.conv1(x))    # (batch, 84, 84, 4) -> (batch, 20, 20, 16)
        x = nn.relu(self.conv2(x))    # -> (batch, 9, 9, 32)
        batch_size = x.shape[0]
        x = x.reshape(batch_size, -1) # -> (batch, 2592)
        x = nn.relu(self.fc1(x))      # -> (batch, 256)
        x = self.fc2(x)               # -> (batch, num_actions)
        return x

batch_size = 32
num_actions = 5
frame_stack = 4
input = mx.random.normal((batch_size, frame_stack, 84, 84))

model = DQN(num_actions)
output = model(input)
print(output.shape)  # (32, 4)

Dataset

The ALE provides an evaluation set built on top of the Stella an Atari emulator, and is maintained by the Farama Foundation. Farama also maintains the Gymnasium library, a handy toolkit for testing reinforcement learning agents.

I'm going to load the Pacman game, which requires only a few lines of code:

import gymnasium as gym
import ale_py

env = gym.make("ALE/Pacman-v5")

The environment that Gymnasium provides has a few different ways of interacting, but for this purpose, we can reset the game with the reset() method, which gives us the state of the game and some info about the game.

state, info = env.reset()
print(state.shape)
print(info)

The state of the game is just a numpy image, which we can take a look at:

from matplotlib import pyplot as plt

plt.title("Pacman State")
plt.axis("off")
plt.imsave(PUBLIC_MEDIA_DIR / "pacman_state_1.png", state)

We can take an action in the game, by passing one of the actions represented as an integer into step(action: int). Where 0 is noopt, and 1, 2, 3, 4 is up, right, left, down, respectively.

For example, if I go right for 100 frames, and print the game state, you can see we've moved to the right:

for i in range(100):
    state, reward, terminated, truncated, info = env.step(2)

plt.title("Pacman State after moving to the right")
plt.axis("off")
plt.imsave(PUBLIC_MEDIA_DIR / "pacman_state_2.png", state)

We can use the RecordVideo wrapper to record a video of the agent exploring the space with a random policy.

import gymnasium as gym
import ale_py
import random

env = gym.wrappers.RecordVideo(
    gym.make("ALE/Pacman-v5", render_mode="rgb_array"),
    video_folder=PUBLIC_MEDIA_DIR / "pacman_ale",
    episode_trigger=lambda episode_id: True
)

observation, info = env.reset()

done = False
while not done:
    action = random.choice(range(env.action_space.n))
    observation, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated

env.close()

Preprocessing

The preprocessing from the paper converts the RGB representation into grayscale and puts four consecutive frames together, representing one game state. This exact processing is provided by the AtariPreprocessing wrapper in Gymnasium:

import gymnasium.wrappers as wrappers

env = gym.make("ALE/Pacman-v5", frameskip=1)
env = wrappers.AtariPreprocessing(env)

obs, _ = env.reset()
plt.imsave(PUBLIC_MEDIA_DIR / "pacman_preprocessed.png", obs, cmap="grey")

Now we can stack the last N timesteps together, which is how a single observation is recognised.

env = wrappers.FrameStackObservation(env, stack_size=4)

Which might look something like this:

Epsilon-greedy policy

A key component of Reinforcement Learning is the tradeoff between exploring and exploiting. The is, we need to ensure that the model takes enough random actions to examine the space adequately, but also follows the policy it is learning at times, so that it makes progress when going in the correct direction (see Exploration-Exploitation Dilemma).

They set an epislon parameter which slowly anneals (changes throughout training), starting at 1, always selecting random, and gradually going to 0.1, where only 10% of the time we are going random, the rest we are using the highest reward action, as predicted by the model so far.

The behavior policy during training was $\epsilon-greedy$ with annealed linearly from 1 to 0.1 over the first million frames, and fixed at 0.1 thereafter.

epsilon = 1.0  # Epsilon greedy parameter
epsilon_min = 0.1  # Minimum epsilon greedy parameter
epsilon_max = 1.0  # Maximum epsilon greedy parameter
epsilon_interval = (
    epsilon_max - epsilon_min
)  # Rate at which to reduce chance of random action being taken
epsilon_greedy_frames = 1000000.0


def get_next_action(epsilon):
    if epsilon > random.random():
        action = mx.random.randint(0, env.action_space.n).item()
    else:
        action = mx.argmax(model(mx.expand_dims(state, 0))).item()
    return action


def anneal_epsilon(epsilon):
    epsilon -= epsilon_interval / epsilon_greedy_frames
    epsilon = max(epsilon, epsilon_min)
    return epsilon

Experience Replay

They utilise an idea from early RL and neural network experiments, all the way back from 1993, called Experience Replay Buffers, where they store an agents experience's at each timestep, $e_t = (s_t, a_t, r_t, s_{t+1})$ .

we utilise a technique known as experience replay [13] where we store the agent's experiences at each timestep, $et = (st, at, rt, st+1)$ in a dataset $D = e1, ..., eN$ , pooled over many episodes into a replay memory. During the inner loop of the algorithm, we apply Q-learning updates, or minibatch updates, to samples of experience, $e ∼ D$ , drawn at random from the pool of stored samples.

from typing import List
from collections import namedtuple
from pydantic import BaseModel

Experience = namedtuple('Experience', ['state', 'action', 'reward', 'state_next', 'done'])

class ReplayBuffer:
    """Experience replay buffer to store and sample transitions."""
    def __init__(self, capacity: int):
        self.buffer = deque(maxlen=capacity)

    def add(self, experience: Experience):
        self.buffer.append(experience)

    def sample(self, batch_size: int) -> List[Experience]:
        batch = random.sample(self.buffer, batch_size)
        states = mx.stack([exp.state for exp in batch])
        actions = mx.array([exp.action for exp in batch], dtype=mx.int32)
        rewards = mx.array([exp.reward for exp in batch], dtype=mx.float32)
        next_states = mx.stack([exp.next_state for exp in batch])
        dones = mx.array([float(exp.done) for exp in batch], dtype=mx.float32)
        return batch, states, actions, reward, next_states, dones

    def __len__(self) -> int:
        return len(self.buffer)

We can go ahead and play it for a while, so we have a bit of a replay buffer:

state, _ = env.reset()
for i in range(32):
    # Take action in environment
    next_state, reward, terminated, truncated, _ = env.step(action)
    next_state = preprocess_frame(next_state)
    done = terminated or truncated

    # Store experience in replay buffer
    replay_buffer.add(Experience(state, action, reward, next_state, done))
    state = next_state

Train one step

Assuming a replay buffer, a single train step is to sample from the replay buffer.

Calculate the ground truth as the actual reward given the state, but here we also include rewards into the future, this is liekly the trickiest part to wrap your head around.

def get_targets(next_states):
    # Compute target Q values
    next_q_values = target_model(next_states)
    max_next_q = mx.max(next_q_values, axis=1)
    targets = rewards + gamma * max_next_q * (1 - dones)
    return targets

An interesting point to note here is that we use a target network. The papers mention that:

The parameters from the previous iteration θi−1 are held fixed when optimising the loss function Li (θi)

Basically, keeping a separate copy of the weights it prevents the "moving target" problem where Q-value updates chase a constantly shifting target, which can lead to oscillations or divergence in training. By keeping the target network fixed for many steps, the optimization becomes more stable, similar to how freezing parts of a network helps in transfer learning.

The loss is Huber Loss between the actual rewards, discounted into the future, and the predicted discounted rewards. Huber Loss is specifically chosen over Mean Squared Error (MSE) because it's less sensitive to outlier rewards that can occur in games with large, sparse reward signals. For small errors, Huber Loss behaves like MSE, providing strong gradients, but for large errors, it behaves like Mean Absolute Error, reducing the impact of extreme values that might destabilize training.

Since only one action is taken in each experience step, we mask out the loss for the other actions.

def compute_loss(states, actions, targets):
    # Forward pass to get Q-values
    q_values = model(states)

    # Select the Q-values for the actions taken
    masks = mx.eye(num_actions)[actions]
    q_action = mx.sum(q_values * masks, axis=1)

    # Compute Huber loss
    return nn.losses.huber_loss(q_action, targets, reduction='mean')


def loss_and_grad(model_params, states, actions, targets):
    model.update(model_params)
    loss = compute_loss(states, actions, targets)
    return loss

grad_fn = mx.grad(loss_and_grad)

This idea is based on the Bellman Equation, but it is essentially trying to find a policy that always takes the correct answer to maximise rewards but considers rewards given in the future to be less important than immediate rewards.

def train_one_step():
    # Sample batch from replay buffer
    states, actions, rewards, next_states, dones = replay_buffer.sample(batch_size)

    # Compute target Q values
    next_q_values = target_model(next_states)
    max_next_q = mx.max(next_q_values, axis=1)
    targets = rewards + gamma * max_next_q * (1 - dones)

    params = model.trainable_parameters()

    # Compute gradients
    grads = grad_fn(params, states, actions, targets)

    # Update parameters
    optimizer.update(model, grads)

Metrics

By plotting the average total reward during training, the loss appears to be unstable, but instead plotting the average predicted max Q-value over a fixed batch of states shows an increase in the amount of future reward expected, increasing steadily.

Training Loop

Here is the full training loop.

def train_dqn(num_episodes=10000, max_steps_per_episode=10000):
    # Initialize metrics tracking
    rewards_history = []
    loss_history = []
    q_values_history = []

    # Main training loop
    for episode in range(num_episodes):
        state, _ = env.reset()
        episode_reward = 0
        episode_loss = []

        for step in range(max_steps_per_episode):
            # Get action based on epsilon-greedy policy
            epsilon = anneal_epsilon(epsilon)
            action = get_next_action(epsilon)

            # Take action in environment
            next_state, reward, terminated, truncated, _ = env.step(action)
            done = terminated or truncated
            episode_reward += reward

            # Store experience in replay buffer
            replay_buffer.add(Experience(state, action, reward, next_state, done))
            state = next_state

            # Only train once we have enough samples
            if len(replay_buffer) >= batch_size:
                loss = train_one_step()
                episode_loss.append(loss)

            # Update target network periodically
            if step % target_update_freq == 0:
                target_model.update(model.parameters())

            if done:
                break

        # Record metrics
        rewards_history.append(episode_reward)
        if episode_loss:
            loss_history.append(sum(episode_loss) / len(episode_loss))

        # Track average Q-values on fixed set of states
        if episode % 100 == 0:
            q_values = mx.mean(mx.max(model(evaluation_states), axis=1))
            q_values_history.append(q_values.item())

            print(f"Episode {episode}, Reward: {episode_reward}, "
                  f"Loss: {loss_history[-1] if episode_loss else 'N/A'}, "
                  f"Avg Q-value: {q_values.item()}, Epsilon: {epsilon}")

    return rewards_history, loss_history, q_values_history

I'm running into some memories issues with MLX, so haven't managed to fully reproduce the paper yet. You can checkout the entire code base here.

Q-Learning

2025-04-18T00:00:00+10:00

This article is part of my (WIP) series on Reinforcement Learning (RL).

Q-Learning is a reinforcement learning algorithm for finding optimal policies in Markov Decision Process (MDP). Unlike supervised learning, where we learn from labelled examples, Q-learning learns from interaction with an environment. It can learn the value of actions without requiring a model of the environment (i.e. learning via trial-and-error), hence, it is considered a "model-free" method.

The algorithm was introduced by Chris Watkins in his 1989 PhD thesis at King's College ¹, with a convergence proof later published by Watkins alongside Peter Dayan ². The concept was also developed independently by several others in the early 90s.

How It Works

Q-Learning is an iterative optimisation algorithm, similar to Gradient Descent in supervised learning. While gradient descent updates model parameters to minimise a loss function, Q-Learning refines value estimates based on environmental interaction. It utilises the Bellman Equation to update its predictions iteratively.

At the core is a lookup table called a Q-Table, which maps state-action pairs to expected future rewards. The values in the Q-Table are called Q-Values. Conceptually, a Q-value $Q(s,a)$ represents "how good" it is to take action a when in state s.

Rewards are discounted over time, meaning immediate rewards are valued more than distant ones – like how businesses value present cash flow over future earnings.

The algorithm uses three key hyper-parameters:

Learning Rate $\alpha$ (0 to 1) – how quickly the algorithm updates its estimates. Higher values mean faster learning but potentially unstable convergence.
Discount Factor $\gamma$ (0 to 1) – how much it values future rewards versus immediate ones. Higher values mean the agent is more forward-thinking.
Exploration Rate $\epsilon$ (0 to 1) – how often it chooses a random action over the current best action. This parameter is typically decreased over time as the agent learns.

The $\epsilon$ parameter balances the trade-off between exploration and exploitation, managing how much the agent tries new things versus sticking to what it already thinks is best. See also the Exploration-Exploitation Dilemma in A/B testing.

Q-Learning in Practice: Taxi

We'll use the Taxi-v3 environment from the gymnasium library (formerly OpenAI Gym). In this environment, the agent (a taxi) must navigate a grid to pick up and drop off passengers at the right location.

Setup Code

You can install Gymnasium with the toy-text dependencies:

pip install "gymnasium[toy-text]"

Then run this code in a Python script:

import gymnasium as gym
import numpy as np
from gymnasium.wrappers import RecordVideo
from pathlib import Path
from tqdm import tqdm
import matplotlib.pyplot as plt

# Q-learning hyperparameters
alpha = 0.1      # Learning rate
gamma = 0.99     # Discount factor
epsilon = 0.1    # Exploration rate (probability of random action)

# An episode is a start-to-finish exploration of the state space,
# where the finish reaches a goal or a hazard.
episodes = 10_000

# Create the Taxi environment
env = gym.make("Taxi-v3", render_mode="human")

# Initialize Q-table with zeros
n_states, n_actions = env.observation_space.n, env.action_space.n
Q = np.zeros((n_states, n_actions))

# Training loop
for ep in tqdm(range(episodes), desc="Training episodes"):
    state, _ = env.reset()
    done = False
    total_reward = 0

    # Episode loop
    while not done:
        # Epsilon-greedy action selection
        if np.random.random() < epsilon:
            # Explore: random action
            action = env.action_space.sample()
        else:
            # Exploit: best known action
            action = np.argmax(Q[state])

        # Take action and observe new state and reward
        next_state, reward, terminated, truncated, _ = env.step(action)
        done = terminated or truncated
        total_reward += reward

        # Q-learning update using Bellman Equation
        # The temporal difference (TD) error represents how surprised we are by the outcome
        td_error = reward + gamma * np.max(Q[next_state]) - Q[state, action]
        Q[state, action] += alpha * td_error

        # Move to next state
        state = next_state

env.close()

The code above will show a visual representation of the agent exploring and slowly updating its policy. Early episodes often show random-looking behaviour:

After training is completed, the final episodes demonstrate much more efficient behaviour:

As you can see, the taxi can pick the passengers up and drop them off at the destination directly.

Testing the Trained Policy

With a trained policy, we can use argmax to pick the highest reward action at every step, effectively using the policy without any more exploration:

# Test the learned policy
state, _ = env.reset()
env.render()
done = False
total_reward = 0

print("Testing trained policy...")
while not done:
    # Always choose the best action according to Q-table
    action = np.argmax(Q[state])
    state, reward, terminated, truncated, _ = env.step(action)
    total_reward += reward
    done = terminated or truncated
    print(f"Step reward: {reward}, Total reward: {total_reward}")
    print(env.render())

print(f"Final score: {total_reward}")

Q-Learning Update Rule: The Math Behind It

The core of the Q-Learning algorithm is its update rule, which is derived from the Bellman Equation. Let's break it down step by step:

$Q(s,a) \leftarrow Q(s,a) + \alpha \Big[ r + \gamma \max_{a'} Q(s', a') - Q(s,a) \Big]$

Where:

$Q(s,a)$ : Current estimate of value for taking action $a$ in state $s$
$\alpha$ : Learning rate (how quickly we update our estimates)
$r$ : Immediate reward received after taking the action
$\gamma$ : Discount factor (importance of future rewards)
$s'$ : The next state we arrive at
$\max_{a'} Q(s', a')$ : Value of the best possible action in the next state
$\Big[ r + \gamma \max_{a'} Q(s', a') - Q(s,a) \Big]$ : The temporal difference (TD) error

In simpler terms:

We take our current estimate $Q(s,a)$
Calculate the TD error (difference between ideal and current estimate)
Update our estimate by moving it slightly (by $\alpha$ ) toward the ideal

Deep Q-Learning

For complex environments with large state spaces, maintaining a Q-table becomes impractical or impossible. For example, in Atari games, where the state is a raw pixel image, there are millions of possible states.

In 2013, DeepMind published a landmark paper where they replaced the Q-table with a Neural Network to approximate the Q-values - a technique known as Deep-Q Learning or DQN ³.

The neural network inputs the state and outputs Q-values for all possible actions, which allows the algorithm to generalise across similar states and handle continuous state spaces.

See the Deep-Q Learning article for more.

Summary

Q-learning provides a fundamental approach to reinforcement learning by learning state-action values through interaction with an environment. Its simplicity and effectiveness make it a cornerstone algorithm in the field, while extensions like Deep Q-Learning enable its application to complex real-world problems.

Watkins, C. J. C. H. (1989). Learning from delayed rewards (Doctoral dissertation). King's College, Cambridge. ↩
Watkins, C. J. C. H., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3-4), 279–292. https://doi.org/10.1007/BF00992698 ↩
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. https://arxiv.org/abs/1312.5602 ↩

Markov Decision Process (MDP)

2025-03-29T00:00:00+10:00

A Markov Decision Process (MDP) is a mathematical framework for modelling decision-making problems where outcomes are partly random and partly controlled by a decision-maker. The goal is to find an optimal policy that maximizes cumulative reward over time.

The Frozen Lake environment in the gymnasium library provides an excellent visualization of an MDP. In this environment, an agent navigates a frozen lake with slippery surfaces, where taking an action (like moving left) doesn't guarantee the intended movement due to environmental uncertainty.

MDPs form the theoretical foundation for many Reinforcement Learning (RL) algorithms, enabling theoretical understanding and practical implementation of RL solutions.

Core Components

MDPs assume the Markov Property, which states that future states depend only on the current state and action, not on the history:

$P(s_{t+1} \mid s_t, a_t, s_{t-1}, a_{t-1}, \dots, s_0, a_0) = P(s_{t+1} \mid s_t, a_t)$

A Markov Decision Process consists of 4 main components:

State Space

The state space (or observation space) encompasses all possible states a system can occupy. The agent observes the current state before deciding on an action.

Frozen Lake has 16 possible states in the standard 4×4 grid. You can access the observation space via env.observation_space in the gymnasium.

Actions

Actions represent the set of all possible moves available to the agent. In the Frozen Lake example, each state has four possible actions:

0: Move left
1: Move down
2: Move right
3: Move up

You can access the available actions in the gymnasium via env.action_space.

Transition Probabilities

Transition probabilities define the likelihood of moving to a new state $s'$ given that the agent takes action $a$ in state $s$ . This is denoted as:

$P(s' \mid s, a)$

In Frozen Lake, transitions can be stochastic (slippery) or deterministic, depending on the environment configuration. In the default slippery version, actions have probabilistic outcomes. For example, choosing to move right may result in:

1/3 probability of moving right (intended direction)
1/3 probability of moving up
1/3 probability of moving down

This stochasticity models the uncertainty of moving on a slippery surface.

Reward Function

The reward function $R(s, a, s')$ provides a numeric reward upon transitioning from state $s$ to state $s'$ after taking action $a$ .

In Frozen Lake, the reward structure is:

Reach goal: +1
Reach hole: 0
Reach frozen tile: 0

The rewards are sparse in this environment, with the agent only receiving a positive reward (+1) upon reaching the goal. This sparsity makes learning more challenging and reflects many real-world problems with limited feedback.

Solving MDPs

To solve an MDP, we need two additional concepts:

Policy

A policy ($\pi$) is the agent's strategy for deciding which action to take in each state. It can be:

Deterministic: $\pi(s) = a$ (maps each state to a specific action)
Stochastic: $\pi(a|s) = P(a|s)$ (maps each state to a probability distribution over actions)

Here's a simple random policy implementation for the Frozen Lake environment:

import gymnasium as gym

env = gym.make('FrozenLake-v1')

def random_policy(state):
    return env.action_space.sample()

state, info = env.reset()
terminated, truncated = False, False

while not (terminated or truncated):
    action = random_policy(state)
    next_state, reward, terminated, truncated, info = env.step(action)

    print(f"State: {state}, Action: {action}, Reward: {reward}, Next state: {next_state}")

    state = next_state

An optimal policy maximizes the expected cumulative reward.

Discount Factor

The discount factor $\gamma$ (gamma) is a parameter between 0 and 1 that determines how much the agent values future rewards relative to immediate ones. It's used in calculating expected returns:

$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$

Where:

A $\gamma$ close to 0 makes the agent myopic (short-sighted), primarily valuing immediate rewards.
A $\gamma$ close to 1 makes the agent far-sighted, strongly valuing future rewards.

For Frozen Lake, a discount factor of around 0.9 to 0.99 is typically chosen, as reaching the goal requires planning several steps ahead.

Algorithms to Solve MDPs

Various algorithms can solve MDPs by finding the optimal policy or value functions:

Dynamic Programming Methods

Value Iteration
- An iterative algorithm that computes the optimal state-value function by repeatedly applying the Bellman optimality equation.
- Converges to the optimal value function, from which an optimal policy can be derived.
Policy Iteration
- Alternates between policy evaluation (computing the value function for the current policy) and policy improvement (making the policy greedy with respect to the current value function).
- Often more efficient than value iteration for certain problems.

Model-Free Methods

Q-Learning
- A model-free reinforcement learning algorithm that learns the optimal action-value function directly from experience.
- Does not require knowledge of transition probabilities or rewards.
SARSA (State-Action-Reward-State-Action)
- An on-policy learning algorithm that updates Q-values based on the action actually taken rather than the greedy action.

Understanding MDPs provides a solid foundation for more advanced reinforcement learning concepts and applications, from game playing to robotics and autonomous systems.

Merkle Tree

2025-03-15T00:00:00+10:00

Merkle Tree is a data structure where each parent node contains the hash of its child nodes, enabling efficient verification of large datasets, amongst many other useful things.

Original illustration by David Göthberg. Source

To construct a Merkle Tree, data is first hashed at the leaf nodes. These hashes are then paired and hashed again to form parent nodes, continuing recursively until reaching a single hash at the root called the Merkle Root. The leaf nodes contain a hash of the data. This structure makes Merkle Trees "tamper-evident" - any change to data will propagate upward, altering the root hash.

If there's an odd number of nodes at any level, the last node is typically duplicated (paired with itself) to create an even number for the next level.

Merkle Proofs, a capability enabled by Merkle Trees, allows us to verify a specific transaction or piece of data is included in a dataset by only checking a small number of hashes rather than the entire set. You only need to hash values from the leaf to the node to verify that a transaction (for example) exists in the ledger.

In the Bitcoin Paper, Nakamoto described a method to reclaim disk space by hashing transactions into a tree structure with only the Merkle root stored in the block header. The Merkle Root used in Bitcoin also enables "simple payment verification", a feature allowing clients to verify transactions without downloading the entire blockchain.

Diagram from Bitcoin: A Peer-to-Peer Electronic Cash System

Git repositories are a type of Merkle Tree, where each commit is identified by a hash that depends on the entire history of the repository, including the file contents, commit messages, timestamps, and parent commits.

Beyond Bitcoin and Git, Merkle Trees are used in distributed file systems like IPFS and Certificate Transparency logs used to verify SSL certificates across the web.

RSA

2025-03-01T00:00:00+10:00

RSA is a public-key encryption system based on the idea that two prime numbers are computationally easy to multiply together but very difficult to factor back into the original primes. Named after its three creators from MIT: Ronald Rivest, Adi Shamir, and Leonard Adleman.

In RSA and other public-key systems, there is a public key, which can encrypt messages and is freely available, and a private key, which can decrypt messages and is kept secret. We can use the keys to encrypt messages (using the recipient's public key so only they can decrypt) and digital signatures (using the sender's private key to prove authenticity).

However, RSA is a slow algorithm typically not used for directly encrypting user data; more often, it's used for transmitting shared keys for symmetric key cryptography (like AES), which is much faster for bulk encryption.

Algorithm

To create a private and public key, we use the following steps:

Select two large prime numbers, $p$ and $q$ . (In practice, each should be hundreds or thousands of bits long—commonly 2048 bits or more—to ensure security.)
Calculate $N = p \times q$ (the modulus).
Calculate Euler's Totient Function: $\phi(N) = (p-1)(q-1)$
Choose a public key $e$ that is relatively prime to $\phi(N)$ . Two numbers are relatively prime when their greatest common divisor (GCD) is 1. (A common choice is $e = 65537$ , a Fermat Prime. It has only two 1's in its binary representation, greatly reducing exponentiation time.)
Compute the private key $d$ as the modular multiplicative inverse of $e$ modulo $\phi(N)$ . This means finding a value $d$ where: $(d \times e) \mod \phi(N) = 1$ . This can be calculated using the Extended Euclidean Algorithm.

$N$ and $e$ are public key components, represented as $(N, e)$ .

$p$ , $q$ , and $d$ are the private key components, although $p$ and $q$ are typically discarded, and it's represented as $(N, d)$ .

Encryption and Decryption

Given $e$ and $N$ , we can encrypt messages: $c = m^e \mod N$ . Given $d$ , we can decrypt messages: $m = c^d \mod N$ .

Example

For a simple example (using small numbers for clarity):

Choose primes $p = 61$ and $q = 53$
Calculate $N = 61 \times 53 = 3233$
Calculate $\phi(N) = (61-1) \times (53-1) = 60 \times 52 = 3120$
Choose $e = 17$ (relatively prime to 3120)
Calculate $d = 2753$ (since $17 \times 2753 \mod 3120 = 1$ )

Public key: $(N=3233, e=17)$

Private key: $(N=3233, d=2753)$

To encrypt message $m = 123$ : $c = 123^{17} \mod 3233 = 855$

To decrypt ciphertext $c = 855$ : $m = 855^{2753} \mod 3233 = 123$

Security Considerations

If the value of $N$ is not large enough, an attacker could factorise it, effectively allowing them to solve for $d$ . Key sizes of at least 2048 bits are recommended for security through 2030.

Additionally, modern RSA implementations use secure padding schemes (like PKCS#1 OAEP) to protect against various attacks, including chosen-ciphertext attacks.

Side-channel attacks can leak information about private keys during implementation, requiring additional countermeasures.

Lastly, RSA's security relies on the computational difficulty of the factoring problem, but quantum computers running Shor's algorithm could break RSA encryption, driving research into post-quantum cryptography alternatives.

Spanning Tree

2025-02-16T00:00:00+10:00

The Spanning Tree of a connected graph $G$ is a connected sub graph which contains all vertices of $G$ , but with no cycles.

Example

$G$ is a connected graph, $T_1$ , $T_2$ , $T_3$ and $T_4$ are spanning trees.

Minimum Spanning Tree (MST)

The Minimum Spanning Tree (MST) is the lowest cost spanning tree within the graph, where vertices have associated costs.

Two algorithms for computing the MST:

Bucket Sort

2025-02-15T00:00:00+10:00

Bucket Sort is a distribution sorting algorithm that works by dividing elements into a finite number of buckets, sorting these buckets (typically with another algorithm), and then concatenating them to produce the final sorted array. It is particularly efficient when input is uniformly distributed over a range, achieving an average case time complexity of $O(n + k)$ where $k$ is the number of buckets.

Algorithm

$\begin{aligned} &\textbf{BUCKETSORT}(A) \\ &\quad 1. \quad n = A.length \\ &\quad 2. \quad \text{buckets } B[0 \dots n - 1] \\ &\quad 3. \quad \textbf{for } i = 1 \text{ to } n \textbf{ do} \\ &\quad 4. \quad\quad \text{insert } A[i] \text{ into bucket } B[ \ \lfloor \ n \times A[i] \ \rfloor \ ] \\ &\quad 5. \quad \text{sort each bucket} \\ &\quad 6. \quad \text{concatenate sorted buckets} \end{aligned}$

Code

import numpy as np

def bucket_sort(arr, num_buckets=5):
    buckets = [[] for _ in range(num_buckets)]

    for num in arr:
        index = int(num_buckets * num)
        buckets[index].append(num)

    for bucket in buckets:
        bucket.sort()

    sorted_arr = [num for bucket in buckets for num in bucket]
    return sorted_arr

Simple step-by-step example

Given array: $[0.78, 0.17, 0.39, 0.26, 0.72]$ , using 5 buckets.

Create 5 empty buckets
Place each number in appropriate bucket based on value range:
Bucket 0 [0.0-0.2]: 0.17
Bucket 1 [0.2-0.4]: 0.26, 0.39
Bucket 2 [0.4-0.6]: empty
Bucket 3 [0.6-0.8]: 0.72, 0.78
Bucket 4 [0.8-1.0]: empty
Sort each bucket individually
Concatenate buckets: $[0.17, 0.26, 0.39, 0.72, 0.78]$

Visualisation

Runtime Analysis

The time complexity of bucket sort varies depending on several factors:

Average Case: $O(n + k)$
Distribution of input into buckets: $O(n)$
Sorting each bucket: When input is uniformly distributed, each bucket contains approximately $n/k$ elements
With insertion sort for each bucket: $O(k * (n/k)^2) = O(n^2/k)$
When $k \approx n$ , this reduces to $O(n)$
Worst Case: $O(n^2)$
Occurs when all elements are placed in a single bucket
The single bucket must then be sorted using insertion sort: $O(n^2)$
Additional $O(n)$ overhead for bucket creation and concatenation
Best Case: $O(n + k)$
When elements are uniformly distributed across buckets
Each bucket contains approximately $n/k$ elements
Bucket creation and distribution: $O(n)$
Sorting small buckets: $O(1)$ per bucket
Final concatenation: $O(n)$
Space Complexity: $O(n + k)$
Storage for $n$ elements across $k$ buckets
Additional space for bucket array itself

The efficiency of bucket sort heavily depends on: * The uniformity of input distribution * The number of buckets chosen * The algorithm used for sorting individual buckets

When the input is known to be uniformly distributed and the number of buckets is chosen appropriately (typically $k \approx n$ ), bucket sort can achieve linear time complexity, making it more efficient than comparison-based sorting algorithms which have a lower bound of $O(n \log n)$ .

Temperature Scaling

2025-01-14T00:00:00+10:00

Temperature scaling controls how "confident" a model is when making predictions by adjusting the sharpness of probability distributions produced by the Softmax Function.

Softmax is a function that converts a neural network's raw outputs (logits) into probabilities that sum to 1. For example, in a dog breed classifier, the model might output logits representing its confidence for different breeds, and the Softmax function would convert those into probability-like values:

	logit	softmax
Golden Retriever	5.23	0.975007
Labrador	1.54	0.024348
Husky	-2.37	0.000488
German Shepherd	-3.50	0.000158

The basic Softmax formula is:

$Softmax(logits) = \frac{\exp(logits)}{\Sigma \exp(logits)}$

By introducing a temperature parameter $T$ , we can control how "confident" the model is in its predictions:

$Softmax(logits, T) = \frac{\exp(logits/T)}{\Sigma \exp(logits/T)}$

For numerical stability, we apply temperature scaling to logits before Softmax:

def scaled_softmax(logits, temperature=1.0):
    scaled_logits = logits/temperature
    return softmax(scaled_logits)

When $T = 1$ , we have plain old Softmax, which maintains the original relative differences between probabilities.

$T < 1$ creates a sharper distribution, making the model more confident. The highest probability becomes even higher, and the lower probabilities become even lower. At $T$ approaching 0, it becomes deterministic (100% confident).

When $T > 1$ , it creates a flatter distribution, making the model less confident, and the differences between probabilities become smaller; predictions become more evenly distributed.

Try it for yourself to see how adjusting the temperature setting affects the Softmax probabilities in a dog breed classification problem:

Temperature: 1.0

Temperature Scaling in Language Models

In a Language Model, which predicts a token at a time based on the previous tokens in a sequence, each token is predicted by creating a Softmax probability distribution across the vocabulary and then randomly sampling from that distribution.

The temperature parameter, therefore, affects how much randomness is injected at inference time.

$T = 0$ : Deterministic, always selects the highest probability token. However, in practice, you can only approximate it with a very small temperature (since dividing by zero is undefined). Good for math, coding, and fact-based responses.

$T \approx 0.7$ : Balanced between coherence and creativity. Common default for chat models. Maintains context while allowing natural variation

$T > 1$ : Increases randomness. Can generate more creative/diverse outputs. Risk of incoherent or off-topic responses

Temperature is typically applied during inference only. During training, models use $T = 1$ to learn the true probability distribution of the data. However, in the paper, Distilling the Knowledge in a Neural Network, they experiment with using higher temperatures during training to help the model distinguish between similar classes of items.

Few-Shot Knowledge-Distillation

2025-01-12T00:00:00+10:00

Good idea from this blog post by Steve Krenzel: Few-Shot Knowledge Distillation (FSKD)

A retrieval LLM routing idea where we:

Store input/output task embeddings from a high-quality model.
For new examples, compute a novelty score: measure how similar the new example is to other examples.
Send novel examples to gpt4o (or some big, expensive model).
Send other examples to gpt4o-mini (or some cheaper, lower performance model), but include similar gpt4o examples in the prompt (few-shot knowledge distillation).

The novelty score is determined by:

And the task embedding (T).
The similarity threshold (θ) - which defines how similar an entry needs to be to be considered a match
The matches threshold (m) specifies the number of matches that need to be considered "low novelty."

$\text{noveltyScore}(T, \theta, m) = \begin{cases} \text{LOW} & \text{if } |\text{search}(T,\theta)| \geq m \\ \text{HIGH} & \text{otherwise} \end{cases}$

The defaults of $\theta = 0.8$ and m = 3 can be tuned for cost/performance trade-offs.

It's cool because it's self-adapting - if there's Domain Shift new examples are sent to the larger model until it builds up new examples.

Results on real inventory moderation task:

63.4% cost reduction
Slight improvement over a large model: 90.9% accuracy (vs 87.6% for the large model alone)
~69% of tasks handled by a smaller model

Large Language Models are Zero-Shot Reasoners (May 2022)

2025-01-08T00:00:00+10:00

Overview

Language models have shown impressive capabilities in few-shot learning, where they learn from a handful of examples. However, this paper from early 2023 shows that large language models (LLMs) can perform complex reasoning tasks with zero examples - they just need to be asked to Think Step-by-Step

Background

Traditional approaches to getting LLMs to solve complex reasoning problems involved showing them several examples with step-by-step solutions (called Chain-of-Thought Prompting or CoT).

This new research shows that by simply adding the phrase "Let's think step by step" before asking for an answer, LLMs can break down and solve complex problems without any examples.

Figure 1: Example inputs and outputs of GPT-3 with different prompting techniques.

What is the key difference between few-shot CoT and zero-shot CoT?

a) Zero-shot CoT uses more examples than few-shot CoT
b) Few-shot CoT requires no examples while zero-shot CoT needs many
c) Zero-shot CoT only needs the prompt "Let's think step by step" while few-shot CoT needs example solutions
d) Zero-shot CoT only works on simple problems

Answer

c) Zero-shot CoT only needs the prompt "Let's think step by step" while few-shot CoT needs example solutions
This is the fundamental innovation of the paper - discovering that complex reasoning could be achieved without examples, just by prompting step-by-step thinking.

Technique - Two Staged-Approach

Let's think step-by-step uses a two-stage prompting process:

1. Reasoning Extraction (First Stage)

Input question is formatted as "Q: [Question]. A: Let's think step by step"
Model generates step-by-step reasoning

2. Answer Extraction (Second Stage)

Combines original question, generated reasoning, and answer-extraction prompt
Uses format-specific triggers like "Therefore, the answer is..."
Extracts final answer in correct format

Figure 2: Full pipeline of Zero-shot-CoT as described in section 3.

While this approach requires two prompting steps instead of one, it eliminates the need for carefully engineered examples that traditional few-shot methods require. The method can be used with any decoding strategy, though the researchers used greedy decoding for simplicity. The approach's flexibility comes from its ability to adapt to different answer formats while maintaining the same basic reasoning structure, making it truly task-agnostic.

Which statement correctly describes Zero-shot-CoT's prompting process?

a) It uses a single prompt to generate both reasoning and answer
b) It requires examples like traditional chain-of-thought prompting
c) It uses two separate prompts - one for reasoning and one for answer extraction
d) It only works with numerical answers
e) It requires fine-tuning the model for each task type

Answer

c) It uses two separate prompts - one for reasoning and one for answer extraction
This is correct because Zero-shot-CoT employs a two-stage process:
* First stage extracts the reasoning using "Let's think step by step"
* Second stage extracts the final answer using a format-specific prompt
The other options are incorrect because:
* It doesn't use a single prompt
* It doesn't need examples
* It works with various answer formats
* It doesn't require fine-tuning

Results

The researchers tested this approach on various tasks including: - Arithmetic reasoning (MultiArith, [[GSM8K]]) - Symbolic reasoning (Last Letter, Coin Flip) - Commonsense reasoning - Logical reasoning

The results were remarkable. For example, on the MultiArith dataset, accuracy increased from 17.7% to 78.7% simply by adding the step-by-step prompt. They showed that CoT helped large models the most.

What did the results show about model scaling?

a) Smaller models performed better with zero-shot CoT
b) Model size had no effect on performance
c) Larger models showed significantly better performance with zero-shot CoT
d) Zero-shot CoT only worked on medium-sized models

Answer

c) Larger models showed significantly better performance with zero-shot CoT
The paper demonstrated that the effectiveness of zero-shot CoT increased dramatically with model size, suggesting that this capability emerges more strongly in larger models.

Why This Matters

This discovery has several important implications for Large Language Models.

It suggests that LLMs have inherent reasoning capabilities that can be accessed with the right prompting
It provides a simpler alternative to few-shot learning for complex tasks
It hints at broader cognitive capabilities in LLMs that we might not have fully explored

Which of these is NOT a key implication of this research?

a) LLMs have untapped reasoning capabilities
b) Complex prompting with examples might be unnecessary
c) LLMs need to be retrained to achieve better reasoning
d) Simple prompts can unlock sophisticated behaviors

Answer

c) LLMs need to be retrained to achieve better reasoning
This is incorrect because the paper shows that existing LLMs already have these capabilities without retraining - they just need the right prompt to access them.

The research suggests that before investing in creating complex few-shot examples or fine-tuning datasets, we should first explore what capabilities already exist within these models through simple prompting strategies.

This work sets a new baseline for evaluating language models' reasoning capabilities and opens up exciting possibilities for making AI systems more capable through clever prompting rather than just increasing model size or training data.

Neural Machine Translation by Jointly Learning to Align and Translate (Sep 2014)

2024-10-28T00:00:00+10:00

These are my notes from the paper Neural Machine Translation by Jointly Learning to Align and Translate (2014) by Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio.

Overview

This paper proposed an improvement to the RNN Encoder-Decoder ¹ network architecture, introducing an "attention mechanism" to the decoder, which significantly improved performance over longer sentences. The concept of attention went on to become extremely influential in Machine Learning.

At the time, neural networks had emerged as a promising approach to machine translation, where researchers were aiming for an end-to-end translation model, in contrast to the state-of-the-art statistical phrase-based translation methods, which involved many individually trained components. The RNN Encoder-Decoder approach would encode an input sentence into a fixed-length context vector; a decoder would then output a translation using the context vector. The encoder and decoder are jointly trained on a dataset of text pairs, where the goal is to maximise the probability of the target given the input.

However, this approach struggles with longer sentences, as the encoder has to drop information to compress it into a fixed-length context vector.

The authors proposed modifying the encoder to output a sequence with one hidden representation per input word, then adding a search mechanism to the decoder, allowing it to find the most relevant information in the input sequence to predict each word in the output sequence.

They likened the modification to the human notion of "attention", calling it an Attention Mechanism. Though not the first Machine Learning paper to propose applying human-like attention to model architectures ², this approach was very influential in NLP, leading to a lot of research eventually converging on an entirely attention-based architecture called the Transformer.

Architecture Details

The authors propose an RNNSearch model: an Encoder / Decoder model with an attention mechanism. For comparison, they train RNNencdec, which follows the standard RNN Encoder / Decoder architecture ² with the encoder returning a fixed-length context vector.

To demonstrate the ability to handle longer sequences, they train each model twice:

First, with sentences of length up to 30 words: RNNencdec-30, RNNsearch-30
Next, with sentences of size up to 50 words: RNNencdec-50, RNNsearch-50

Encoder

For the RNN, they use a Bidirectional RNN: a Gated Recurrent Unit (GRU).

Each input token is fed into an embedding layer, $x_i$ , and then a GRU encodes into a forward and backward "annotation" per token, concatenated to make a single representation, $h_i$ .

The idea is to allow each annotation to summarise the preceding and the following words, providing the most possible representation for the attention mechanism.

Figure 1: The graphical illustration of the proposed model trying to generate the t-th target word $y_t$ given a source sentence ( $x_1, x_2, \ldots, x_T$ )

Decoder

For the decoder, they use a uni-directional Gated Recurrent Unit.

The initial hidden state $s_0$ is computed as an initialisation layer, which comprises a linear layer followed by a tanh activation function.

$s_0 = \tanh \left( W_s \overleftarrow{h}_1 \right)$ where $W_s \in \mathbb{R}^{n \times n}$ .

For each prediction step, they calculate the word probability as:

$p(y_i|y_1, \ldots, y_{i-1}, x) = g(y_{i-1}, s_i, c_i)$

Where

$y_{i-1}$ is the embedding of the token from the previous step.
$s_i$ is the hidden state output from the previous layer.
$c_i$ is the context vector.

The context vector, $c_i$ is calculated at each step as follows:

$c_i = \sum\limits_{j=1}^{T_x}\alpha_{ij}h_j$

The weights, $\alpha_{ij}$ , are calculated by the alignment (Attention) model.

Alignment Model (Attention)

The alignment scores are calculated by combining a projection of the decoder's previous state and a projection of the encoder output, then applying tanh activation followed by a linear combination with another weight vector.

$e_{ij} = v_a^{T} \tanh(W_as_{i-1} + U_{a}h_{j})$

This function gives us an output score for each token in the input sequence. Finally, we can perform a Softmax calculation to convert the weights to probability distribution:

$\sigma_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x}\exp(e_{ij})}$

Maxout

The final layer, which returns the probabilities for each word, uses a Maxout layer to generate the final probabilities. A Maxout layer acts as a form of regularisation. It projects the input vector onto multiple "buckets" and selects the maximum value from each bucket. This process introduces non-linearity and helps prevent overfitting, akin to dropout.

Training Params

Algorithm: Stochasic Gradient Descent (SGD)
Optimiser: Adadelta (Zeiler, 2012)
Batch Size: 80 sentences
Training Duration: Approximately 5 days per model
Inference method: Beam search
Task: English-to-French translation
Dataset: bilingual, parallel corpora provided by ACL WMT 14.
- Word count: 850 (reduced to 348M)
- Components:
  - Europarl (61M words)
  - News Commentary (5.5M)
  - UN (421M)
  - two crawled corpora of 90M and 272.5M words, respectively
Metric: BLEU Score.
Tokeniser: from open-source machine translation package Moses. They shortlist the most frequent 30k words and map everything else to [UNK].
Comparisons: They compare RNNsearch with a standard RNN Encoder-Decoder, RNNenc and Moses, the state-of-the-art translation package.
Test Set: For the test set, they evaluate news-test-2014 from WMT'14, which contains 3003 sentences not in training
Valid Set: They concat news-test-2012 and news-test-2013.
Initialisation: Orthogonal for recurrent weights, Gaussian ($0, 0.01^2$) for feed-forward weights, zeros for biases.

Results

They record results on all test data with only examples that don't contain unknown tokens.

The RNNsearch-50 model achieved a BLEU score of 34.16 on sentences with unknown tokens excluded, significantly outperforming the RNNencdec-50 model, which scored 26.71 and training RNNsearch-50 to convergence beat the state-of-the-art Moses. However, when unknown tokens are included, the model performs considerably worse.

RNNsearch was much better at longer sentences than RNNenc.

Figure 2: The BLEU scores of the generated translations on the test set with respect to the lengths of the sentences.

Model	All	No UNK
RNNencdec-30	13.93	24.19
RNNsearch-30	21.50	31.44
RNNencdec-50	17.82	26.71
RNNsearch-50	26.75	34.16
RNNsearch-50*	28.45	36.15
Moses	33.30	35.63

*Note: RNNsearch-50* was trained much longer until the performance on the development set stopped improving.

Interpreting Attention

One benefit of calculating attention weights for each output word is that they are interpretable, allowing us to visualise word alignments.

Figure 3. Four sample alignments that were found by RNNsearch-50.

As we can see, typically, words are aligned to similarly positioned words in a sentence, but not always.

Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv. https://arxiv.org/abs/1406.1078 ↩
Brauwers, G., & Frasincar, F. (2023). A general survey on attention mechanisms in deep learning. IEEE Transactions on Knowledge and Data Engineering, 35(4), 3279–3298. https://doi.org/10.1109/tkde.2021.3126456 ↩↩

Thinking LLMs: General Instruction Following with Thought Generation (Oct 2024)

2024-10-16T00:00:00+10:00

These are my notes from the paper Thinking LLMs: General Instruction Following with Thought Generation (Oct 2024) by Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston and Sainbayar Sukhbaatar.

Overview

This paper introduces Thought Prompting, a prompting method that enables LLMs to engage in a "thinking" process before generating responses. Within the Agentic Reasoning framework, it would be considered a Planning technique.

Unlike Chain-of-Thought prompting, which mainly benefits math and logic tasks, Thought Prompting improves results across many tasks.

Fig 4: Fine-grained evaluation on unseen instructions from UltraFeedback, broken down by category. They measure the win rate of TPO against the direct baseline as judged by GPT4.

The method separates the thought process from the final answer, enabling developers to choose whether to display the thought process for interpretability or keep it hidden. For example, in Fig 6, you can see the response is separate from the thoughts using the <R>; this works similarly to o1-preview, which also hides its internal thought process.

Fig 6: Example of TPO answering a simple factoid question. This model is trained with the specific thought prompt, so it writes a draft and evaluates it in the thought part.

Unlike Think Step-by-Step, it relies on fine-tuning - it doesn't benefit "thoughtless" models. As shown in Fig 3, adding Thought Prompting to an untrained model worsens results.

Fig 3: Training iterations on AlpacaEval and Arena-Hard, comparing their TPO method to direct baseline starting from the seed.*

Thought Prompt Optimisation

The paper introduces Thought Policy Optimisation (TPO), a fine-tuning technique based on Direct Preference Optimisation (DPO). TPO uses a Judge model to evaluate model outputs based solely on the responses, without access to the thought process; this lets the model learn and refine its "thinking" abilities without relying on supervised thought data, asshowne in the diagram below.

Figure 1: Thought Preference Optimisation: they start by prompting the LLM to generate thoughts before its response. After sampling different outputs, we feed the response parts to the judge model which determines the best and worst ones. Then, they use the corresponding full outputs as chosen and rejected pairs for DPO optimisation. They perform multiple iterations of this training.

Gets top 3 in AlpacaEval (52.5%) and performs on par with GPT4 in Arena-Hard (37.3%)

Mixtral of Experts (Jan 2024)

2024-10-15T00:00:00+10:00

These are my notes for paper Mixtral of Experts (8 Jan 2024) by Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed.

This paper introduces Mixtral 8x7B, a Sparse Mixture of Experts Model (SMoE) language model.

The model uses an SMoE approach where each layer comprises eight "experts", and a router network selects two experts to process each token. Thanks to this, despite having 47B parameters, it only uses 13B active parameters during inference, which makes it more computationally efficient than other models (although it doesn't save any memory).

It outperforms Llama 2 70B and GPT-3.5 on many benchmarks, particularly in mathematics, code generation, and multilingual tasks.

They also present Mixtral 8x7B – Instruct, a chat model fine-tuned to follow instructions, which outperforms other chat models like GPT-3.5 Turbo and Llama 2 70B – chat.

The base and instruct models are released under the Apache 2.0 license, making them freely available for academic and commercial use.

Architecture

Mixtral is based on a Transformer architecture. It uses the same modifications as Mistral 7b, except each layer comprises 8 Mixture-of-Expert layers, which replace typical feedforward blocks.

At each layer, a router selects two experts for every token to process the request.

Figure 1: Mixture of Experts Layers. Each input vector is assigned to 2 of the 8 experts by a router. The layer's output is the weighted sum of the outputs of the two selected experts (using Softmax weights). In Mixtral, an expert is a standard feedforward block as in a vanilla transformer architecture.

In code:

def moe_layer(x, Wg):  # x: input token, Wg: gating network weights
  top2_logits = top_k(x @ Wg, k=2)  # Get top 2 expert activations
  gating_weights = softmax(top2_logits)  # Weights for each expert
  expert_outputs = [expert_i(x) for expert_i in experts]  # Run chosen experts
  return sum(gating_weights[i] * expert_outputs[i] for i in range(2))

y = moe_layer(x, Wg)

Mixtral supports a fully dense context length of 32k tokens.

Architecture details

Parameter	Value
dim	4096
n_layers	32
head_dim	128
hidden_dim	14336
n_heads	32
n_kv_heads	8
context_len	32768
vocab_size	32000
num_experts	8
top_k_experts	2

Routing Analysis

Surprisingly, the experts don't specialise in particular domains or topics based on different datasets.

Figure 7: Proportion of tokens assigned to each expert on different domains from The Pile dataset for layers 0, 15, and 31. The gray dashed vertical line marks 1/8, i.e. the proportion expected with uniform sampling..

Also, the router exhibits temporal locality - it's more likely to assign the same experts to consecutive tokens - the router is recognising and leveraging the sequential nature of language.

Evaluation

Mixtral outperforms or matches Llama 2 70B on a wide range of benchmarks, including commonsense reasoning, word knowledge, reading comprehension, math, and code generation.

Mixtral is particularly strong in mathematical and coding tasks, significantly surpassing Llama 2 70B.

MMLU and MT Bench: Mixtral performs similarly or better than GPT-3.5 on MMLU and MT Bench despite its smaller size. Although some differences in evaluation protocols between Mixtral and Llama 2 may affect direct comparisons.

Multilingual Proficiency: Mixtral significantly outperforms Llama 2 70B in French, German, Spanish, and Italian, demonstrating its strong multilingual capabilities.

Long Context Handling: Mixtral can effectively process long sequences, achieving 100% accuracy on the passkey retrieval task regardless of context length or passkey position.

Bias Mitigation: Compared to Llama 2, Mixtral exhibits less bias on the BBQ and BOLD benchmarks, indicating improved fairness and reduced bias in its responses.

Positive Sentiment: Mixtral consistently displays more positive sentiments than Llama 2, suggesting a more optimistic and encouraging tone in its generated text.

Evaluation of OpenAI o1: Opportunities and Challenges of AGI

2024-10-14T00:00:00+10:00

This report comprehensively analyses the o1-preview across many tasks and domains. In short, o1-preview significantly improves over gpt4 in all areas, albeit at the cost of inference performance. There are some areas where it continues to struggle, particularly advanced reasoning, creativity and understanding nuanced human-language, however, it is clearly a leap forward, and there's a lot of evidence to suggest it will continue to improve.

A recent paper GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models has found that by modifying the GSM test set, i.e. by adding distracting information, or modifying the numbers used in the equation, they see a considerable drop in performance in all LLM models, which people have suggest means that LLMS "can't reason". However, notably o1-preview is the most robust to these perturbations in the test set.

Figure 8. from GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

Categories tested

Note that I haven't included all the areas tested in the paper, just a few of interest.

High School Level Math

o1-preview achieves 100% accuracy in high school-level mathematical reasoning tasks (although these are likely in the training set according to GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models).

College Level Math

o1-preview answers 7 out of 12 questions right. Mostly had difficulty with advanced discrete mathematics problems and proof-writing.

Logical Reasoning

o1-preview consistently excels in logical reasoning tasks, successfully solving mathematical puzzles, deducing sequences, and evaluating arguments. The authors note that although the final answer is not always accurate, the analysis and reasoning process is "truly convincing".

Natural Language Inference

Across diverse NLI datasets, o1-preview demonstrated high accuracy, successfully analysing complex logical relationships between sentences, signifying advanced reasoning capabilities and a strong grasp of domain-specific knowledge.

Quantitative Investing

o1-preview exhibits a strong understanding of financial knowledge and statistical modelling techniques, enabling it to perform well in quantitative investing tasks. The model demonstrates a comprehensive knowledge and clear explanations of complex concepts like mean reversion, ARIMA models, and stochastic oscillators.

Educational Q&A

o1-preview excels in answering educational questions, highlighting its potential as a valuable tool in academic settings. Its ability to accurately select the correct answer from a set of options reflects a robust comprehension of scientific concepts and the capacity to navigate misleading distractors.

Medical Genetics and Genomics Reasoning

o1-preview consistently demonstrates strong reasoning abilities in genomics and medical genetics. It can generate reliable conclusions and provide detailed, step-by-step explanations when gene and disease descriptions are provided. The model employs a chain-of-thought process based on factual information to make decisions.

Medical Knowledge Question Answer

o1-preview displays impressive accuracy in answering medical knowledge questions, particularly those found in real-world medical exams. It effectively retrieves key information from the questions, providing accurate answers and offering detailed explanations. While occasional gaps in information or illogical reasoning may occur, the overall reasoning process aligns with factual data. It may choose the correct option but miss key conditions while evaluating incorrect options.

Chip Design

In chip design, o1-preview consistently outperforms ChipNeMo, a specialised model designed for this domain. o1-preview exhibits superior problem-solving capabilities, a deeper level of analysis, and practical relevance across tasks such as Engineering Assistant Chatbot, EDA Script Generation, and Bug Summary and Analysis. The model's ability to optimise code, handle intricate netlist structures, and offer insightful suggestions for power optimisation and task prioritisation demonstrates its advanced reasoning and potential to significantly benefit chip design automation.

Student Writing

While proficient in student writing, o1-preview tends to generate responses that adhere to a rigid, machine-like template, lacking citations; this could encourage students to adopt such structures passively, potentially hindering their writing's creativity and effectiveness.

Although o1-preview demonstrates strong overall performance in social media analysis, it sometimes makes errors in sentiment analysis and emotion recognition. It might misinterpret neutral sentiment as negative or struggle to differentiate between positive and negative sentiment when both are present. The sources attribute these errors to the model's difficulty understanding subtle emotional cues, such as sarcasm and tone.

Code Generation

While o1-preview performs well in code generation, optimising solutions for more complex problems is difficult, especially under time constraints. It got 10 out of 12 correct. It's still good.

Radiology Report Generation

Although o1-preview achieves high ROUGE scores in Radiology report generation, indicating good performance, it also has the longest average generation time compared to other models.

AI Meets the Classroom: When Does ChatGPT Harm Learning?

2024-10-13T00:00:00+10:00

My notes from the paper AI Meets the Classroom: When Does ChatGPT Harm Learning? by Matthias Lehmann, Philipp B. Cornelius, Fabian J. Sting.

Summary

This paper covers one observational and two experimental studies on the effects of LLM access on students learning to code.

Key findings:

Using LLMs as personal tutors by asking them for explanations improves topic understanding, though not overall learning outcomes (likely because it reduces the volume of topics covered).
Asking LLMs to generate solutions impairs understanding. The use of copy-and-paste encourages this behaviour, tripling solution requests without increasing explanation requests.
Students with stronger prior knowledge benefit more from LLM usage. Students with weaker foundations learned less with AI access.
LLM access increases students' perceived learning beyond their actual learning outcome, i.e. students think it's a lot more helpful than it is.

In the field study (Study 1), they compare students' work with a number of solutions generated by ChatGPT as a proxy for the use of LLMs. They call this ChatGPT Similarity.

To calculate this, they generate 50 ChatGPT solutions and then take the maximum similarity with a student's code:

$\text{ChatGPT Similarity}_{iq} = \max \left( \text{sim}( \text{Code}_{iq}, \text{Code}^{LLM}_{jq} ) \mid j = 1, \dots, 50 \right)$

for student $i$ and question $q$ , where $\text{Code}_{iq}$ is the final student code, $\text{Code}^{LLM}_{jq}$ is one of the 50 ChatGPT generated solutions, $j = 1, \dots, 50$ , and $\text{sim}(\cdot, \cdot)$ is Damerau Levenshtein similarity.

The lab experiments (Studies 2 and 3) instead directly randomised LLM access, comparing treatment (with LLM) and control (without) conditions with pre- and post-tests.

System 2 Thinking

2024-10-13T00:00:00+10:00

System 2 thinking is characterised by slow, deliberate, and logical reasoning, requiring conscious effort and attention to solve complex problems. Unlike the intuitive nature of System 1 Thinking, System 2 is responsible for critical thinking, reasoning, and decision-making in unfamiliar or challenging situations ¹.

Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux ↩

No 'Zero-Shot' Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

2024-10-03T00:00:00+10:00

Data

Notes from paper No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance by Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H.S. Torr, Adel Bibi, Samuel Albanie and Matthias Bethge.

Overview

The authors analyse 30GB of data from multimodal datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics) used to train models like CLIP and Stable-Diffusion and build a pipeline to extract "concepts" (nouns like "man", "hat", "fish").

They determine a log-linear relationship between concept frequency and zero-shot performance: The model needs to see exponentially more examples in training to achieve a linear zero-shot performance increase for a concept.

They also find some interesting discoveries in these web-scraped datasets:

Concept Distribution: Across all pre-training datasets, concepts are long-tailed: a large fraction of concepts are rare, and the rare concepts are not properly learned during multimodal pretraining.
Concept Correlation across Pre-training Datasets: The distribution of concepts across different pretraining datasets is strongly correlated, suggesting different web crawl datasets mostly contain the same stuff and share similar problems.
Image-Text Misalignment between Concepts in Pre-training Data: They identify a wide-scale misalignment problem between pairs where concepts can appear in one modality but not the other. In some cases, up to 36% of pairs are misaligned in publicly available datasets.

They curate a dataset called Let It Wag that identifies the least frequent concepts across the web scrapes (examples: A310 aircraft, a worm snake, and a tropical kingbird). It was found that performance across classification models and image generation tasks is considerably worse (as compared to ImageNet) on this dataset.

Agentic Reasoning

2024-08-25T00:00:00+10:00

Agentic Reasoning refers to strategies in which Large Language Models can route between states like planning, reflecting, observing, and utilising tools, allowing them to take actions and handle much more complex tasks. Unlike traditional LLM approaches focusing on single-turn responses, agentic reasoning involves multi-step interactions, enabling the model to dynamically adjust its behaviour based on context and goals.

LLMs are called AI Agents within this framework. Agents can take on different roles or states, like planning actions, using tools, reflecting on their own outputs, and storing information for long-term use. In Multi-Agent Systems, an agent can collaborate with other agents.

In contrast, non-agentic workflows include direct methods like Zero-Shot Prompting, where the LLM returns the result based on a single prompt. Some techniques from agentic reasoning are also used in zero-shot prompting, so the line between agentic and non-agentic is not always clear.

However, there are a series of "design patterns" that are indicative of an agentic system.

Tool Use

The clearest example of when an LLM becomes an AI Agent is when it calls another tool to do a job—for instance, booking a flight via an API or sending an email. This interaction allows the LLM to extend beyond text generation and perform meaningful actions in the real world.

Typically, approaches to tool use allow the model to generate command sequences that get translated into API commands, such as Atomic Actions, which are LLM outputs that can be mapped into API commands. Alternatively, the LLM may generate code used to call a tool. Recent papers like ToolGen: Unified Tool Retrieval and Calling via Generation explore embedding tool commands directly into the model's vocabulary.

Other examples include:

Gorilla Large Language Model Connected with Massive APIs - they fine-tune a model that can perform tasks by retrieving API documents and calling functions. They use test-time modifications to ensure that the model can handle changes to APIs and is not limited to information in pre-training.
MM-REACT Prompting ChatGPT for Multimodal Reasoning and Action - they use prompting techniques to allow ChatGPT to call vision models and other models to answer questions.

Memory

Incorporating long-term memory allows LLMs to store and recall information over extended interactions Recent papers have introduced a paradigm of long-term memory, such as RAISE and Reflexion allowing models to remember past actions and outcomes, enhancing future performance.

Planning

Planning involves decomposing tasks into smaller, manageable steps. Techniques like Chain-of-Thought Prompting encourage models to generate intermediate reasoning steps, significantly improving their ability to solve complex problems.

Systems may employ:

Open-loop Planning: Creating and executing an entire plan without adjustments.
Closed-loop Planning: Planning an action, executing it, observing the outcome, and then planning the next step based on the new state.

Acting / Observing

In agentic systems, the model performs actions and then observes the results, updating its plans based on feedback. This iterative process allows the agent to adapt to new information and refine its strategies accordingly.

Reflection

Reflection enables an agent to analyse its outputs and reasoning processes. Methods like Self-Refine allow an LLM to critique and improve its responses iteratively, enhancing the quality and accuracy of its outputs.

Waffle Chart

2024-07-24T00:00:00+10:00

A Waffle Chart is a Data Visualisation where data points are represented as squares on a 2D grid. It's used for displaying the proportion or count of each category, for example, showing voting results across a population.

For example, this Waffle Chart shows the 2020 Presidential Election Electoral College Results, highlighting the closeness of the contest and giving a sense of proportion.

Waffle Charts are particularly effective when you need to show parts of a whole, and can often be more visually engaging than a Pie Chart.

Scaled-Dot Product Attention

2024-03-13T00:00:00+10:00

Scaled-Dot Product Attention is a method of computing a token representation to include the context of surrounding tokens. It was described in the paper Attention Is All You Need and is used in the Transformer architecture.

In a seq-to-seq architecture, we typically convert tokens (words) into a sequence of embeddings. However, some token embeddings will be ambiguous without the surrounding context.

Consider the word "minute" in these two sentences:

"It took one minute."

and

"The size was minute."

The token representing minute will mean very different things in each sentence, even though they will use the same embedding representation. The words "took" and "size" indicate whether the word relates to time or size, respectively. We want a way to represent each token with information about important surrounding tokens.

We could use a simple average to achieve this. However, in the minute cases, some words are more important to defining the context than others. Could we also use a neural network to compute the weights for each other tokens in sequence to have the most useful average representation? That's exactly what Scaled-Dot Product Attention is.

Scaled-Dot Product Attention uses two matrix projections to compute the scores of other tokens in the sequence, then a softmax to convert to weights. Then, a final projection is to create the final weighted representation. All the weights in the attention module are learned alongside the rest of the network.

Let's see how to compute it step-by-step.

Scaled-Dot Product Attention Step-by-step

0. Prepare Input

Though this step is technically not part of Scaled-Dot Product Attention, we represent input tokens in the Transformer architecture using a standard token embedding: nn.Embedding and a Positional Encoding, which we combine to create a final representation.

The positional embedding represents each token's position in the sequence, as this information would otherwise be lost.

The dimensions of this input are batch, time, and embedding size. For example, with a batch size of 8, four input words (assuming word-level tokenisation) and an embedding dimension of 1024, the embeddings would have a shape of: (8, 4, 1024)

1. Create projections used for computing scores

Transform input embeddings into three matrices called query, key, and values. However, those names aren't particularly useful; they could also be called proj1, proj2, and proj_final. Many other articles on the web relate these values to the retrieval system, although I think it's unnecessary confusion.

All you need to know is that our goal is to compute a table of scores with a row per token. This paper chooses this particular method of accomplishing it, but there are alternatives.

We can do this in 6 lines of code:

# __init__
query_proj = nn.Linear(embedding_dim, attention_dim, bias=False)
key_proj = nn.Linear(embedding_dim, attention_dim, bias=False)
value_proj = nn.Linear(embedding_dim, attention_dim, bias=False)

# forward
query = query_proj(X)
key = key_proj(X)
value = value_proj(X)

Like any typically linear layer, the projection weights are learned throughout training.

2. Compute scores as the dot product of query and key

Compute the scores as the dot product of each query and key. However, for efficiency, we compute the dot products for the entire sequence by performing a matrix product of query and the transposed key matrix: $\text{scores} = Q @ K^{T}$

scores = query @ key.transpose(2, 1)

3. Scale the score values using the square root of the attention dimension

As the size of the attention dimensions increases, the dot products computed to get the attention scores generally grow larger in magnitude as more terms are summed in the dot product calculation. Larger values passed into the Softmax function can lead to small gradients, making learning slow and ineffective. So, they scale the scores by dividing the square root of the attention dimension, which doesn't change their relative scores since the same factor scales all scores but keeps their absolute magnitude in a stable range.

scores = scores / sqrt(attention_dim)

Scaled-Dot Product Attention is simply Dot-Product attention with this additional step.

4. (Decoder only) Mask out any future tokens

In the Decoder part of the Transformer, we need to ensure that the model cannot "see" future tokens when making predictions, as the decoder should only rely on previously generated tokens to predict the next token in the sequence. To achieve this, we apply a mask to the attention scores before applying the softmax function. The mask sets the scores for future tokens to negative infinity (-inf), which results in zeros applying the softmax function, effectively blocking any information flow from future tokens.

We can use the tril function to create a diagonal mask where the value True represents positions to be masked, then masked_fill will replace any masked positions with float("-inf"). After performing the Softmax operation, any -inf values will be converted into a weight of 0.

# Compute a mask and set all future values to -inf, which ensures a score of 0 after softmax.
attn_mask = torch.tril(torch.ones(*scores.shape)) == 0
scores = torch.masked_fill(scores, attn_mask, float("-inf"))

5. Compute Softmax to convert scores into probability distributions

Next, we ensure the scores are between 0 and 1 and all the scores equal 1.

scores = softmax(scores, dim=-1)

6. Calculate the final representations as the dot product of scores and values

Finally, we use the value matrix to create a final output using the calculated weights.

out = scores @ value

Here's the full PyTorch module:

import math

import torch
from torch import nn
from torch.nn.functional import softmax


class SingleHeadAttention(nn.Module):
    def __init__(self, embedding_dim, attention_dim):
        super().__init__()
        torch.manual_seed(0)
        self.attention_dim = attention_dim

        self.key_proj = nn.Linear(embedding_dim, attention_dim, bias=False)
        self.query_proj = nn.Linear(embedding_dim, attention_dim, bias=False)
        self.value_proj = nn.Linear(embedding_dim, attention_dim, bias=False)

    def forward(self, X):
        key = self.key_proj(X)
        query = self.query_proj(X)
        value = self.value_proj(X)

        scores = query @ key.transpose(2, 1)
        # Scale scores by sqrt of attention dim
        scores = scores / math.sqrt(self.attention_dim)

        # Compute a mask and set all future values to -inf, which ensures a score of 0 after softmax.
        attn_mask = torch.tril(torch.ones(*scores.shape)) == 0
        scores = torch.masked_fill(scores, attn_mask, float("-inf"))

        # Compute softmax of scores.
        scores = softmax(scores, dim=-1)

        # Now, do the final projection with values.
        out = scores @ value

        return out

In the Transformer architecture, we combine multiple Self-Attention modules by concatenating their outputs into one final representation, passed through a final feed-forward layer. These multiple Self-Attention layers together are called Multi-head Attention.

Multi-Head Attention diagram from paper Attention Is All Your Need

Queue

2024-02-18T00:00:00+10:00

A queue is a simple data structure based on First In, First Out (FIFO) principle, where the longer an item has waited, the sooner it will be available.

It is based around the fundamental queue paradigm that we're familiar with: front of the line goes first.

A queue has a head and a tail. We dequeue by removing an element from the head of the queue and enqueue by adding an element to the tail.

A queue supports the following operations in its typical form:

Operation	Pseudocode	Description
head	`HEAD(q)`	Return element at the head of queue `q`.
dequeue	`DEQUEUE(q)`	Remove and return element at the head of the queue.
enqueue	`ENQUEUE(o, q)`	Adds element `o` to the tail of the queue.
isEmpty?	`EMPTY(s)`	Return boolean result of is empty check.
new empty queue	`new QUEUE q`	Create a new, empty queue.

Stack

2024-02-18T00:00:00+10:00

A stack is a simple data structure based on the Last In, First Out (LIFO) principle, where only the Top element is accessible.

Think of a stack of plates - you can only add or retrieve from the top.

When we add an item to a stack, we call that operation push. When you take an item off the stack, we call it pop.

A stack S can be implemented easily with a fixed-length array. We can track which element is at the top with S.top. The size of the array is the maximum size of the stack.

Example of array-based stack implementation - From Introduction to Algorithms, Third Edition

If we attempt to add to a stack beyond its max size, Stack Overflow error. If we try to pop from an empty stack, we get the Stack Underflow error.

Programming languages use a stack called a Call Stack to keep track of functions running.

A stack supports the following operations in its typical form:

Operation	Pseudocode	Description
push	`PUSH(o, s)`	Place `o` on top of stack.
top	`TOP(s)`	Show the element at the top of the stack.
pop	`POP(s)`	Remove and return the element at the top of the stack.
isEmpty?	`EMPTY(s)`	Return True or False if empty.
new empty stack	`new STACK s`	Create a new, empty stack.

Gale-Shapley Algorithm

2024-02-08T00:00:00+10:00

The Gale-Shapley algorithm (also known as Deferred Acceptance) solves the stable matching problem, where the goal is to match members of two equally sized groups based on their preferences. Gale-Shapley guarantees that all pairs are stable; no two pairs would prefer another compared to their assigned match.

Consider a speed dating event; for simplicity of explanation, I'll assume all participants are heterosexual.

At the end of a series of conversations, each man and woman writes their preferences in order.

We want an algorithm that ensures no woman is paired with a man but prefers another man who also prefers her. This pairing is considered unstable.

Gale-Shapely guarantees no unstable pairs.

The algorithms orchestrate a series of proposals. Each woman proposes to their top-choice man. If a man receives multiple proposals, they accept the ones highest on their preference list and reject others. If a woman is rejected, she proposes to her next choice.

The process continues iteratively until all women are matched with partners. It's worst case Time Complexity is $O(n^2)$

Here, the algorithm is written in Python code. It's commonly executed with a while loop that continues to find proposals until no unmatched pairs exist.

men_preferences = {
    "John": ["Sally", "Jill", "Doris"],
    "Jacob": ["Sally", "Jill", "Doris"],
    "Bob": ["Sally", "Doris", "Jill"]
}

womens_preferences = {
    "Sally": ["John", "Jacob", "Bob"],
    "Jill": ["Jacob", "John", "Bob"],
    "Doris": ["John", "Bob", "Jacob"]
}

def woman_prefers(new_man, current_man, woman_prefs):
    return woman_prefs.index(new_man) < woman_prefs.index(current_man)

def gale_shapley(men_preferences, women_preferences):
    # Initial setup
    n = len(men_preferences)
    free_men = list(men_preferences.keys())
    engaged = {}
    proposed = {man: [] for man in men_preferences}

    while free_men:
        man = free_men[0]
        man_prefs = men_preferences[man]
        woman = next(w for w in man_prefs if w not in proposed[man])
        proposed[man].append(woman)

        if woman not in engaged:
            # Woman is free
            engaged[woman] = man
            free_men.remove(man)
        else:
            # Woman is engaged, check if she prefers this new man
            current_man = engaged[woman]
            if woman_prefers(man, current_man, women_preferences[woman]):
                # Woman prefers new man
                engaged[woman] = man
                free_men.remove(man)
                free_men.append(current_man)
            # Otherwise, do nothing

    return engaged

In the real world, the algorithm is used in dating to match medical graduates to residency problems, job/employer matching, and many other places.

Merge Sort

2024-02-04T00:00:00+10:00

Merge Sort is a Divide-and-Conquer sorting algorithm, which sorts a list recursively by dividing it into halves until a single --element or empty list is created; it then merges these sublists of create a software list.

Merge Sort is one of the faster and most stable sorting algorithms, with an $O(n \log n)$ time complexity in the average and worst case.

Algorithm

$\begin{aligned} &\textbf{MERGESORT}(A, a, b) \\ &\quad \text{if } a < b \text{ then} \\ &\quad\quad m = \left\lfloor \frac{a + b}{2} \right\rfloor \\ &\quad\quad \textbf{MERGESORT}(A, a, m) \\ &\quad\quad \textbf{MERGESORT}(A, m + 1, b) \\ &\quad\quad \textbf{MERGE}(A, a, m, b) \end{aligned}$

Simple step-by-step example

Given array: $[3, 4, 6, 1, 2]$ , a=1 and b=5 for the initial call.

After the first recursive call, we would have sorted the first half (a=1, m=3), so the array would become $[3, 4, 6, 2, 1]$
After the 2nd recursive call, we would have sorted the 2nd half (a=4, b=5), so the array would become $[3, 4, 6, 1, 2]$
Then the final merge would merge both sides, resulting in $[1, 2, 3, 4, 6]$

Visualisation

Neural Codec Language Models are Zero-Shot Text-to-Speech Synthesizers

2024-01-31T00:00:00+10:00

These are my notes from the paper Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers by Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei

Overview

This paper describes a state-of-the-art (Jan 2023) text-to-speech (TTS) language model called VALL-E.

VALL-E can generate speech in anyone's voice with only a 3-second sample of the speaker and some text - this capability is known as zero-shot TTS. It is the speech equivalent of GPT's in-context learning capability.

Fig 1. from Neural Codec Language Models are Zero-Shot Text-to-Speech Synthesizers

Key Details

Use RVQ tokens as an intermediate representation of speech

VALL-E uses an intermediate representation of speech instead of modelling audio directly, which makes it a cascading TTS system. The key insight in this paper, as was the theme of many audio papers in 2023, is utilising a RVQ audio codec, which compresses audio into discrete "acoustic" tokens (they use the Encodec RVQ implementation).

TTS as a language model problem

Thanks to the discrete representation, they can take advantage of a Language Model - a first for a TTS problem.

These first two details are where the paper gets its name: Neural Codec Language Models are Zero-Shot Text-to-Speech Synthesizes.

By treating TTS as a language model problem, they can take advantage of large, noisy datasets, which is a step toward reliable zero-shot TTS.

Train on 60k hours of unannotated speech from the Libri-Light dataset

The Libri-Light dataset has over 60k hours of unannotated speech, hundreds of times more than existing TTS papers. Since most of Libri-Light is unannotated, they train a speech recognition model to generate textual annotations (HDNN-HMM) for the raw speech.

Model tokens hierarchically

The authors note that the quantising approach in RVQ has a hierarchical structure: tokens from the first quantiser can recover acoustic properties like speaker identification, whereas the later quantiser learns fine acoustic details.

They exploit this insight by splitting the language model into two parts:

1. An autoregressive Transformer that is used to predict codes for the first codebook.

Partial of Figure 3 from Neural Codec Language Models are Zero-Shot Text to Speech Synthesisers

2. A non-autoregressive Transformer that predicts the subsequent codes from the first code.

Partial of Figure 3 from Neural Codec Language Models are Zero-Shot Text to Speech Synthesisers

Proposing VALL-E as a Hierarchical Model provides a good trade-off between flexibility with the length of returned speech and inference performance, as the NAR can operate at $O(1)$ instead of $O(T)$ where $T$ is the sequence length of the tokenised audio.

Comparison to Previous Work

In the past, a Mel Spectrogram has been commonly used as the intermediary representation for TTS, relying on a Vocoder (like HiFi-GAN) to the decoder. There have also been some successful end-to-end TTS approaches. However, all these problems are typically formulated as continuous signal regression problems, which so far have needed high-quality, clean audio to train on - not data scraped from the internet. Without the larger datasets, reliable zero-shot TTS on unseen speakers is very difficult. The capacity for in-context learning enjoyed by GPT is a powerful capability and is now available for Speech Synthesis.

Table 1 summarises the difference between VALL-E and previous TTS systems.

	Current Systems	VALL-E
Intermediate representation	Mel Spectrogram	Audio Codec Code
Objective function	Continuous Signal Regression	Language Model
Training Data	≤ 600 hours	60k hours
In-Context Learning	✘	✔

Table 1 from Neural Codec Language Models are Zero-Shot Text to Speech Synthesisers

Architecture

AR / NAR Models

Both the AR model and the NAR model have the same Transformer architecture, which contains:

12 layers
16 attention heads
an embedding dimension of 1024
a feed-forward layer dimension of 4096
dropout of 0.1.

Both the AR and NAR models received the phoneme sequence as a prompt.

Encodec

They use eight codebooks, each with 1024 code dimensionality. The encoder produces an embedding sequence at 75 Hz for input waveforms at 24 kHz. So, For a 10-second audio waveform, the discrete representation would be $750 \times 8$ ( $750 = \frac{24000 \times 10}{320}$ ).

Figure 2 from Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

Attention

The AR model can attend to all previous tokens in the sequence, whereas the NAR model can attend to all previously predicted tokens.

Embeddings and decoding

They use a sinuous position embedding for the prompt and input tokens and share the output projection layer parameters with acoustic embedding parameters.

Decoding

For the AR model, they found beam-search could lead the model to an infinite loop, so they used sampling-based decoding conditioned on the prompts. For the NAR model, they use greedy decoding to choose the token with the highest probability.

Training Settings

GPU	16 NVIDIA TESLA V100 32GB GPUs
Batch size	6k acoustic tokens per GPU
Steps	800k
Optimiser	AdamW Optimizer
Learning rate	Warm up for the first 32k, peak at 5 x 10-4, then linear decay
Data preprocessing	Randomly crop the waveform to a random length between 10 secs and 20 secs.

Evaluation

Baseline Models

TTS

YourTTS,

SOTA zero-shot TTS model trained on VCTK, LibriTTS, TS-Portuguese.

Speaker to Speaker

GSLM

GSLM HuBERT codes as input and reconstructs the waveform with the Tacotron2 model and the WaveGlow vocoder. HuBERT codes discard speaker identity, so it tends to achieve a poor speaker score.

AudioLM

An audio-to-audio model that also uses RVQ codes. They use the word error score reported in their paper, which a Conformer Transducer model obtained.

Datasets

LibriSpeech

Since there's no overlap between LibriLight and LibriSpeech, they can use it for Zero-Shot TTS evaluation.

They use the samples from LibriSpeech test-clean with lengths between 4 and 10 seconds, resulting in a 2.2-hour subset.

For each sample synthesis, they randomly chose another utterance of the same speaker and cropped a 3-second speech segment as the enrolled speech, giving them 40 test cases. Each experiment runs three times, and the average score is reported.

VCTK

VCTK consists of 108 speakers, none in the training set. VCTK is more challenging than LibriSpeech as it contains speakers with various accents.

YourTTS has seen 97 speakers in VCTK as training. So, they evaluate YourTTS performance on the full 107 speakers and 11 unseen speakers, respectively.

For each speaker, they randomly selected three utterances of 3s/5s/10s as the prompts and the text of another utterance as the text prompt.

Metrics

Automated

Speaker Similarity

They use a SOTA speaker verification model, WaLM-TDNN, to check the speaker similarity between audio prompt and synthesised speech.

Word Error Rate

They run speech recognition on the generated audio and calculate the Word Error Rate (WER) concerning the original transcriptions (using the HuBERT-Large model fine-tuned on LibriSpeech 960h as the ASR model)

Human evaluation

CMOS is an indicator of speech naturalness.

SMOS measures where speech is similar to the original speaker's voice.

They also use crowd-sourcing to calculate the comparative mean option score (CMOS) and similarity mean option score (SMOS) by inviting 12 and 6 native speakers as CMOS and SMOS contributors.

The scale of SMOS is from 1 to 5 with 0.5-point increments.

CMOS ranges from -3 to 3 with intervals of 1.

Results

Table 2 shows the objective evaluation results on the automatic metrics. VALL-E does much better than YourTTS at speaker similarity and robustness. For GSLM, since the HuBERT codes discard the speaker identity, it achieves a poor speaker score. There is no open-source implementation of AudioLM, so they use the results from the paper.

Automatic Metric results (Table 2)

Model	Word Error Rate	Speaker Identity
GroundTruth	2.2	0.754
GSLM	12.4	0.126
AudioLM (model is not open source)	6
YourTTS	7.7	0.337
VALL-E	5.9	0.580
VALL-E continual	3.8	0.508

VALL-E is very close to ground truth regarding SMOS, indicating the synthesised speech is similar to the given unseen speaker in testing. It significantly outperforms the baseline with +0.93 SMOS, demonstrating the effectiveness of VALL-E in zero-shot scenarios. Regarding naturalness, VALL-E beats the baseline with +0.12 CMOS, indicating the proposed method could synthesise more natural and realistic speech against baselines.

Human evaluations (Table 3)

Model	SMOS	CMOS (vs VALL-E)
YourTTS	3.45	-0.12
VALL-E	4.38	0.0
GroundTruth	4.5	+0.17

Table 6 shows the results for the VCTK dataset. VALL-E outperforms the baseline even though it has seen 97 speakers in training. The performance gap becomes larger compared to the baseline in a fair setting (11 speakers).

**Table 6: Automatic evaluation of speaker similarity with 108 speakers on VCTK. **

97 speakers in VCTK as training

	3s prompt	5s prompt	10s prompt
YourTTS	0.357	0.337	0.394
VALL-E	0.382	0.423	0.484
GroundTruth	0.546	0.591	0.620

11 unseen speakers

	3s prompt	5s prompt	10s prompt
YourTTS	0.331	0.337	0.334
VALL-E	0.389	0.380	0.414
GroundTruth	0.528	0.556	0.586

Table 7 shows a comparison of their method against baseline and ground truth. VALL-E has better speaker similarity than the baseline, even if the baseline has seen some speakers in training. The side-by-side CMOS evaluation shows that VALL-E is +0.23 over YourTTS, indicating a significantly better performance speaking of naturalness. VALL-E also achieves +0.04 CMOS over ground truth, demonstrating no statistically significant difference from human recordings.

Table 7: Human evaluation with 60 speakers on VCTK with a 3-second enrolled recording for each.

	SMOS	CMOS (v.s. VALL-E)
YourTTS	3.70 (+0.09)	0.23
VALL-E	3.81 (+0.09)	0.00
GroundTruth	4.29 (+0.09)	-0.04

Ablation

NAR Ablation

They train three NAR models with different numbers of prompts:

NAR-no prompt - trained without any prompts.
NAR-phn prompt - trained with only phoneme sequence as prompt
NAR-2 prompts - uses phoneme prompt and acoustic token prompt as conditions

They use ground-truth first-level acoustic tokens as the model input and compute the WER and speaker similarity scores.

Results: The model, without any prompts, performs poorly on both ASR and speaker similarity evaluations, even though the acoustic input token is ground truth. When adding the phoneme prompt, the WER is reduced by a large margin from 19.6 to 3.0. It shows the phoneme prompt mainly contributes to the content of the generation.

In the NAR-2 prompts, the model can learn speaker information from the acoustic token prompt and thus improve the speaker evaluation quality.

Ablation study of the NAR model. The inputs of the NAR models are the ground truth for the ablation study. (Table 4)

	NAR-no prompt	NAR-phn prompt	NAR-2 prompts
WER	19.6	3.0	2.8
SPK	0.518	0.541	0.732

AR Ablation

They always use the NAR-2 prompts setting in these experiments as the NAR model.

They try removing the acoustic prompt (w/o acoustic prompt). After that, it can only obtain a speaker similarity score of 0.236, showing the prompt is extremely crucial for speaker identity. Even if the NAR model could see the prompt, the prompt for the AR model also contributes a lot to speaker similarity.

Ablation study of the AR model (Table 5)

	WER	SPK
VALL-E	5.9	0.585
w/o acoustic prompt	5.9	0.236

Qualitative Analysis

Diversity

Previous TTS systems had a strong one-to-one mapping between input text and output waveform because mel spectrogram generation was based on reconstruction for each step without randomness. On the other hand, VALL-E uses the sampling-based method to generate discrete tokens so that the output can be diverse for the same input text.

Given a sentence and an enrolled recording, they run the inference process twice and visualise its waveform.

They observed the two samples having different lengths and phrase durations, where the first has a faster speech rate.

They observe that the accents of the two samples are different. The second output emphasises the word "must" with a larger amplitude, whereas the first does not.

The diversity may be useful for downstream scenarios, particularly generating pseudo-data for speech recognition.

Partial Figure 4. from Neural Codec Language Models are Zero-Shot Text to Speech Synthesisers

Acoustic environment maintenance

They find acoustic environment consistency between the acoustic prompt and the generation. VALL-E could also synthesise speech with reverberation when the acoustic prompt reverberates, whereas the baseline outputs clean speech.

They explain that VALL-E is trained on a large-scale dataset consisting of more acoustic conditions than the data used by the baseline.

VALL-E could learn acoustic consistency instead of a clean environment only during training.

Speaker's emotion maintenance

Emotional TTS is a classic subtopic of speech synthesis, synthesising speech with a required emotion.

Traditional methods always train a model on a supervised emotional TTS dataset, where the speech corresponds to a transcription and an emotion label.

They find that VALL-E can preserve the emotion in the prompt at a zero-shot setting.

They select acoustic prompts from EmoV-DB, a dataset containing speech with five emotions; VALL-E can keep the same emotion of the prompt in speech synthesis, even if the model is not fine-tuned on an emotional TTS dataset.

Future Work

The authors note four future directions for this research.

1. Improve synthesis robustness

The pronunciation isn't always clear, and occasional words are duplicated. They think it's mainly due to the phoneme-to-acoustic being an autoregression model, where disordered attention alignments exist with no constraints. They could modify the attention mechanism or apply more non-autoregression models to solve.

2. Data coverage

Due to dataset limitations, VALL-E does not work for everyone's voice, especially those with accents. More data scale-up is likely the answer to improve this.

3. Model structure

One possible direction is to predict codes with a large universal model. Using NAR models to speed up inference is another.

4. Risk mitigation

Since a zero-shot TTS model like this carries a large potential for misuse, they want to consider building a model to detect where AI-synthesised audio.

Software Development is a Trade

2024-01-27T00:00:00+10:00

Despite their occasional ridicule within the industry, the software boot camp model works - the top boot camps tend to have a higher success rate at landing people in jobs than computer science departments at top universities ¹ ².

Anecdotally, I've hired many devs who came straight out of boot camps, and they all rocked and are in the industry to this day - most of them have become seniors.

From day 1, they came to work knowing Git, how to use their IDE (shortcuts, etc.), the basics of typical Agile-like practices and had practical experience building web applications.

Yes, a Survivorship Bias is at play here - not everyone "made it". However, their fellow students who found that software wasn't for them only took 3-6 months of their lives learning it instead of 3-10 years to get a degree.

On the flip side, for a typical Computer Science degree, the engineering content is often unaligned with the expectations of a junior developer ³. In the field, you're going to spend a lot less time concerned with the complexity of search algorithms or building abstract syntax trees, and a lot more time creating CI pipelines, dealing with merge conflicts, and investigating complex customer issues, which, in my experience, are usually a tiny part of one engineering module, if covered at all, in a CompSci degree.

The fundamentals are useful and typically must be in place to progress into senior roles, and they can't be taught in depth at a short boot camp. However, I argue that fundamentals are much more efficiently learned alongside a career as a practising software engineer. Instead of learning them all up front, they'd be much more effectively taught after an engineer has some practice building products with software. For example, learning about worst-case complexity and Big O notation is much easier when you have already blown up prod, thanks to a nested loop in your code.

And that's much closer to how an apprenticeship works: some study, followed by lots of hands-on experience, followed by more study, and so on until you're a professional.

Like a trade, writing software can only be mastered by practising it - a lot, and we should educate developers accordingly.

Burning Glass Technologies via this article by Optimal reports on boot camp vs college, job placement success rate. The data is from 2021 and needs updating in 2022-2023 conditions. ↩
Council on Integrity in Results Reporting (CIRR) via a Forbes article reports ~71% success rate of tech placements from boot camps. ↩
Garousi, V., Giray, G., Tüzün, E., Catal, C., & Felderer, M. (2020). Closing the gap between software engineering education and industrial needs. IEEE Software, 37(2), 68–77. https://doi.org/10.1109/MS.2018.2880823 ↩

Convex Hull

2024-01-26T00:00:00+10:00

The Convex Hull is the smallest polygon that contains a set of points.

Think of some nails poking out of a board. If you place a rubber band around all nails, the polygon shape the rubber band makes is the convex hull.

One real-world application in game programming is Collision Detection. Since computing collisions can be expensive, a convex hull is calculated around the 3d model of an object to provide an approximate collision region, reducing the number of computations required.

This example from the Roblox docs shows the collision region for a 3d model with the Hull collision option.

Original Mesh	Hull Collision Region

This example of a 3d model to its hull collision model comes from the Roblox docs.

Insertion Sort

2024-01-25T00:00:00+10:00

Insertion Sort is one of the most widely known sorting Algorithm. The algorithm iteratively compares each element with its left neighbours, shifting them one position to the right if they are greater. It has an average and worst-case run time of $O(n^2)$ , so it is one of the slowest algorithms for large input sizes; however, if the list is mostly sorted, it can be one of the best-performing options.

Image via Wikimedia commons.

Python code

def insertion_sort(v):
    j = 1

    for i in range(j, len(v)):
        key = v[j]
        i = j - 1
        while i >= 0 and v[i] > key:
            v[i + 1] = v[i]
            i = i - 1
        v[i + 1] = key
        j = j + 1

    return v

Time Complexity

Worst case run-time: $O(n^2)$
Avg run-time: $O(n^2)$
Best case run-time: $O(n)$ (when the array is already sorted)

Space Complexity

$O(1)$ (in-place sorting algorithm)

Overfit First

2024-01-21T00:00:00+10:00

In a Tweet by Karpathy from 2019, he describes his #1 neural network training mistake: "You didn't try to overfit a single batch first".

Jeremy Howard shares a similar sentiment in his Three Steps to Training a Good Model from the same year, where he lists the Overfit as the #1 step:

A slide from fastai - Lesson 8 (2019) - Deep Learning from the Foundations.

In my experience, failing to check that the model can overfit to a small amount of data is one of the surest ways to waste time in machine learning. If the model cannot learn to perform the task perfectly on a small set of in-domain data, then what hope do you have generalising with a larger dataset?

Also, by starting with the overfitting, you allow the data collection and model construction to happen in parallel, which is much closer to the Iterative Development goals of the Agile software methodology. Instead of spending time collecting data upfront, it can be done alongside the model development, allowing for parallel progress streams throughout the project. It also means each step can inform each other: model results can be used to assess the data most pertinent for modelling the problem, and available data can tell what model architectures make the most sense for the project.

Always start a neural network training endeavour by overfitting on a small amount of data first.

Residual Vector Quantisation

2024-01-13T00:00:00+10:00

Residual Vector Quantisation (RVQ) is a technique for encoding audio into discrete tokens called codes. It's like a tokeniser for audio. Not only does that allow us to compress audio into absurdly small sizes - up to a 90x compression rate, but we can also use the tokens to model audio using the same architectures that work so well for text, like Transformers. Now we can build language models for audio, speech or music, and that's precisely what recent models like Google's AudioLM, Microsoft's VALL-E and Meta's MusicGen are.

RVQ was first applied to audio in the Soundstream paper by Google Researchers and has since been used in popular neural audio compression architectures like Encodec and DAC.

To understand RVQ, let's start by ignoring the R part of RVQ to focus on Vector Quantisation (VQ).

Vector Quantisation

Quantisation is the process of converting continuous infinite values into discrete finite values.

In VQ, we encode a signal into a series of Vector, then query each vector to find the closest neighbour in a lookup table called a codebook. Now, we can represent an entire chunk or "frame" of a signal with a single code.

The codebook table is nothing more than an embedding table, where the table size is the codebook size, and the vector size is the codebook dimensions: codebook = nn.Embedding(codebook_size, codebook_dim). Like an embedding table, the weights are learned alongside the rest of the network during training.

Vector quantisation comes originally from signal processing and has been exploited throughout image modelling architectures like VQ-VAE and VQGAN.

Vector Quantisation for Audio

The direct VQ approach to encoding audio might look like this:

Audio Input: an audio signal is represented as a multidimensional array of numbers with a known Sample Rate (usually 44.1kHz). A mono signal has one channel; stereo and others can have more.
Encoder: An encoder converts the signal into a sequence of vectors, one per "frame". The frame rate will be dependent on the model architecture and sample rate.
Quantise: Find each vector's nearest neighbour in the codebook table. Again, the codebook table is learned alongside the encoder and decoder during training.
Output: The index of the lookup vector in the matrix is the "code" and is all we need to reconstruct the audio, given a Decoder. Though not pictured in the diagram, the decoder is learned alongside the encoder and codebook table.

Note that for upstream modelling tasks, we will want to use the vector representation of each code.

One clear limitation of this approach is that representing audio with a single code per frame will never allow us to accurately reconstruct audio from these codes unless we have an infinitely large codebook.

A clever technique to mitigate this is to take the difference between the encoded vector and the codebook vector for each encoded vector, which we call the Residual vector. We can look up the residual vector in a subsequent codebook table. And we can repeat this process, each time adding a high capability for accurate reconstruction at the cost of compression size.

Residual VQ

For Residual Vector Quantisation, we add these steps to the VQ operation:

Residual - calculate a difference vector called the Residual for each codebook and input vector. Use that to look at a subsequent codebook.
Repeat - repeat this for $Nq$ codebook tables.
Output - at the end, we will have $Nq$ sequences of codes for modelling.

So now we have: Residual Vector Quantization.

Training

We can train a model like this by performing the encode and decode audio during training and calculating various forms of reconstruction loss, including a GAN-style discriminator. This example is the architecture described in the SoundStream paper:

This note was heavily inspired by What is Residual Vector Quanitzation by AssemblyAI.

Snake Activation Function

2024-01-04T00:00:00+10:00

Snake is a neural network activation function useful for modelling problems with a "periodic induction bias" - in other words, problems with regular, repeating patterns - for example, time-series data, audio signals and so on. It was described in the paper Neural networks fail to learn periodic functions and how to fix it.

In the DAC paper, High-Fidelity Audio Compression with Improved RVQGAN, they replace Leaky ReLU with Snake and significantly improve the reconstructed audio quality.

$\text{snake}(x) = x + \frac{1}{\alpha} \sin^2(\alpha x)$

The $\alpha$ is a parameter that controls the frequency of the periodic component of the signal.

In code:

import numpy as np

def snake_activation(x, alpha):
    return x + (1 / alpha) * np.sin(alpha * x) ** 2

Plot:

Leaky ReLU

2024-01-03T00:00:00+10:00

Leaky ReLU (Rectified Linear Unit) is a modification of the ReLU activation function, which outputs a small value for negative numbers instead of 0.

The small value is generated by multiplying x with a hardcoded value like 0.1.

$\alpha = 0.1$

$f(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha x & \text{if } x \leq 0 \end{cases}$

In code:

import numpy as np

def leaky_relu(x, alpha=0.1):
    return np.where(x > 0, x, alpha * x)

It's commonly used for training generative adversarial networks or any networks with sparse gradients. By introducing a non-zero gradient for negative values, Leaky ReLU allows the model to learn from data that standard ReLU might disregard, but at the cost of adding a hyperparameter. At $\alpha = 0$ Leaky ReLU is simply ReLU.

ReLU

2024-01-02T00:00:00+10:00

Rectified Linear Unit (ReLU) is the most common activation function in deep learning: it converts negative values to 0. ReLU is one of the simplest conceivable ways to add non-linearity to a neural network. And it works!

$f(x)= \max(0,x)$

Code:

import numpy as np

def relu(x):
    return np.maximum(0, x)

The paper Rectified Linear Units Improve Restricted Boltzmann Machines is commonly cited as the first usage of the ReLU activation function, though the first usage of the function dates back to the 1975 paper Cognitron: A self-organizing multilayered neural network

ReLU helped to overcome the vanishing gradient problem prevalent in traditional activation functions like Sigmoid Function or tanh. In these functions, gradients can become extremely small, stopping the network from learning further.

However, since the function outputs zero for any negative input, neurons can sometimes become inactive and stop contributing to the learning process, referred to as "dying ReLU", especially if the network is not properly initialized or the learning rate is too high.

Variations like Leaky ReLU and Parametric ReLU mitigate this by replacing 0 with a small value when the unit is inactive, providing a way to keep the neurons alive during the training process.

Module Coupling

2023-12-31T00:00:00+10:00

Module coupling concerns the interdependence between software Module: how much do a group of modules rely on each other?

Coupling is certainly not a bad thing on its own; it's necessary to develop a sufficiently complex system. However, some types of coupling are preferred over others.

Similar to Module Cohesion, the ISO/IEEE Systems and Software Engineering Vocabulary recognises six key types of module coupling.

Having names for these different coupling types is quite useful for identifying problem points in a code base and strategies for refactoring.

Content Coupling

Content Coupling is when a module is contained within another module.

Consider an Image module that contains various image type loading implementations: JpegImage, PngImage, GifImage, etc.

Given an image, a user doesn't need to know which submodule to call for their specific image type; they request to load an image, and the main module calls the required submodules.

image = Image.load("cat.jpg")

Content Coupling is universally considered a good idea and property of Khorikov's Well-Designed API.

Data Coupling

Data Coupling, also known as input-output coupling, is a type of coupling in which output from one software module is input to another.

For example, a data module that prepares data before running a transformation.

import data
import transform

my_data = data.prepare_data()
output = transform.transform_data(my_data)

Data coupling is also considered a good thing.

Common-Environment Coupling

Common-environment coupling is when multiple modules share the same global data.

Common environment refers to global variables, singleton state objects, system environment variables, etc.

It is not necessarily a bad thing. However, using a mutable global state can lead to hard-to-find bugs.

Sometimes, a better idea is to use sub-environments: global states within specific classes or modules, name-spaced environment variables, etc.

If you must use a global environment, ideally, it would be immutable.

Control Coupling

Control Coupling is when a module communicates information, perhaps via flags, to another to influence its execution.

For example, if Module A passes flags to Module B to change the mathematical operations that Module B performs.

Control coupling is mostly considered bad; in the example above, Module B is hard to test and verify since it's dependent on control information from Module A, which can lead to complex and hard-to-reason designs.

Data coupling is preferred over control coupling.

Hybrid Coupling

Hybrid Coupling occurs when different subsets of the range of values of a data item are used for separate and unrelated purposes.

It is a rare type of coupling but sometimes the only option, especially in limited memory environments (microcontrollers). You should avoid it unless you know what you're doing.

Pathological Coupling

Pathological Coupling occurs when one module is used to change the behaviour of another module.

I'm thinking of ideas like monkey patching or modifying private variables.

It's mostly a bad thing. Usually, it occurs when a module is repurposed to support functionality it wasn't originally intended for, and it often calls for a refactor.

Module

2023-12-29T00:00:00+10:00

A module is a distinct and identifiable unit of code designed to be part of a larger application.

The specific definition of a module varies amongst programming languages and developers.

In Python and similar languages, a module is clearly defined as a file with a .py extension containing code that can be imported and utilised by other modules. In others, a module could refer to a class definition and its associated methods.

The key idea of modularity is to promote code reusability and organisational clarity.

Module Cohesion refers to the degree of relation between module elements. Module Coupling is the degree of interdependence between software modules.

Module Cohesion

2023-12-29T00:00:00+10:00

In software engineering, module cohesion refers to the degree to which the elements of a Module are related. For example, functions that manage the same type of data or contribute to a singular, unified piece of capability.

Some forms of cohesion are considered good, while others are not.

Like Module Coupling, the ISO/IEEE Systems and Software Engineering Vocabulary recognises seven types of module cohesion.

Functional Cohesion

Functional cohesion refers to when elements of a module are grouped because they contribute to the same purpose.

For example, a module called file_operations has a set of functions like read_file(), write_file() and delete_file(). Each function contributes to the overarching file-handling capability of the program.

Functional cohesion is the ideal type and is universally considered good.

Communicational Cohesion

Communicational cohesion occurs when code is grouped because it operates on the same data type.

For example, an Invoice class might contain methods for adding items, calculating the total, applying discounts, etc. Each method operates on the same data structure, the items in an invoice.

As the foundation of object-oriented programming, communicational cohesion is generally considered good. However, consider a complex Customer class, where we have methods for adding customers, updating details, generating customer-specific reports, etc. Though the module operates on the same data type, the functionality is quite different, and refactoring it into multiple modules might be better.

Logical Cohesion

Logical cohesion is when things are grouped because they relate in some way but don't necessarily have interdependence or a relationship to a single task.

For example, a math module includes functions like math.square_root, math.log, math.sine.

Sometimes considered a bad practice - while they may be related logically, they don't share a relationship that warrants putting them in a single module.

Procedural Cohesion

Procedural cohesion is when things are grouped because they happen in sequence.

For example, a process class contains a method for user authentication, followed by a technique to log the authentication, and finally, a method to redirect the user based on their role. These methods are part of a sequence (login process) but are quite different in functionality.

Generally considered bad. Just because things happen sequentially doesn't mean they should be logically grouped.

Sequential Cohesion

Sequential cohesion is where the output of one task performed by a software module serves as input to another task performed by the module.

For example, in a data processing module, one function takes raw data and formats it, the following function performs some calculations on this formatted data, and a third function logs the results. Here, the output of one function becomes the input for the next.

Similar to procedural cohesion, it's considered bad.

Temporal Cohesion

Temporal cohesion refers to grouping based on things that happen at a similar time or a particular program execution phase.

An example is a startup module that initialises various components of an application - like setting up database connections, loading configuration files, and initialising logging. These tasks are all required during the startup phase but aren't functionally related, indicating temporal cohesion.

Mostly considered bad: just because things happen simultaneously doesn't mean they should be related.

Coincidental Cohesion

Coincidental cohesion is where elements of a module share no functional relationship.

For example, a module contains miscellaneous functions like calculate_age, generate_random_number, format_date etc. Here, these tasks have no functional relationship to each other and are grouped by coincidence, not because they logically belong together.

Coincidental cohesion is universally considered bad.

Large-scale Contrastive Language-Audio Pre-training with Feature Fusion and Keyword-to-Caption Augmentation

2023-12-13T00:00:00+10:00

These are my notes from the paper Large-scale Contrastive Language-Audio Pre-training with Feature Fusion and Keyword-to-Caption Augmentation by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov

Overview

This paper uses contrastive learning to learn a shared text/audio latent space. Like CLIP for audio. It has four main contributions:

1. Contrastive Language-Audio Pre-training (CLAP)

CLAP is a pre-training system for learning a shared text/audio latent space from pairs of examples. From this, we can learn representations that allow us to query audio using text or vice versa.

It is trained using Contrastive Loss. Hence: Contrastive Language-Audio Pre-training.

The authors use a text-encoder and audio-encoder to generate respective representations, then feed into an MLP layer to learn a shared latent space. For the text-encoder, they use RoBERTa, and for the audio-encoder, they use HTS-AT. The authors also compared several alternatives.

They released the architecture, training code and series of weights trained on different subsets of their datasets here.

2. Feature-Fusion

A technique for dealing with variable-length, long audio (> 10 secs). It combines the entire audio downsampled, alongside 10-second clips taken randomly throughout the audio. They pass through several layers to get the final representation.

More information is below.

3. Keyword-to-Caption Augmentation

Since the Audioset dataset only has keywords and labels paired with audio, they use a T5 model provided by the keytotext library to create captions from these keywords.

They also add a de-biasing step to convert references to "woman" or "man" to "person".

Here are some examples of the keywords and the generated captions, followed by their debiased versions.

4. Laion Audio 630K

Finally, Laion Audio 630k was released: a large text/audio dataset scraped from the internet. They collect 633 526 text/audio pairs, amounting to 4,325.39 hours of audio.

Contains:

human activities
natural sounds
audio effects

Examples

"The sound of a siren"

"The sounds of wrestling crowd, mezzanine level, huge crowd, p.a., and loop."

CLIP

Demonstrated how contrastive loss can be used to learn the relationship between text and image by projecting into a shared latent space
trained on a large-scale internet scrape

Other attempts to use contrastive pre-training on language-audio:

Text-to-audio retrieval via large-scale contrastive learning

Pre-trained Audio Neural Network (PANN) as the audio encoder.
BERT as text encoder.
Various loss functions to evaluate text-to-audio retrieval.

Audio Retrieval with WavText5K and CLAP Training

Also adds HTS-AT and RoBERTa into encoder list to enhance performance.
Uses representation in the downstream task of audio classification.

Other studies extending CLIP for audio:

However, they all share limitations:

Datasets "relatively small."
Prior work has yet to investigate selections and hyper-parameters thoroughly.
Can't accommodate varied audio lengths, particularly with Transformer-based audio encoder.
No analysis of representation in the downstream task.

Training Dataset

Laion Audio 630K

630K+ audio-text pairs.

AudioCaps + Clotho

55k audio-text pairs.

Audioset

1.9 million audio samples with only labels available for each sample.

They exclude overlapping data in evaluation sets, resulting in 2.5 million audio samples.

Dataset Format and Preprocessing

All audio is converted to mono.

48kHz in FLAC format.

Expand data with only tags or labels using the template: "the sound of label-1, label-2, ..., and label-n"

Model Architecture

$X^{a}_i$ = audio example i
$X^{t}_{i}$ = text example i
$X^{a}_{i}, X^{t}_{i}$ = audio-text pair $i$ .
$\text{faudio}(X^{a}_{i})$ = audio encodings.
$\text{ftext}(X^{t}_{i})$ = text encodings

Pass the audio and text encodings into another neural network layer to get 512-dimension embeddings.

$E^{a}_{t} = \text{MLPaudio}(\text{faudio}(X^{a}_{i}))$
$E^{t}_{i} = \text{MLPtext}(\text{ftext}(X^{t}_{i}))$

Now, you can use the contrastive learning loss function to compare pairs and negative pairs in each batch.

$L = \frac{1}{2N} \Sigma^{N}_{i=1} \ ( \log \frac{\exp{E^{a}_{i} \cdot E^{t}_{i} / r}}{\Sigma^{N}_{j=1} \exp(E^{a}_{i} \cdot E^{t}_{j} / r)} + \log \frac{\exp{E^{t}_{i} \cdot E^{a}_{i} / r}}{\Sigma^{N}_{j=1} \exp(E^{t}_{i} \cdot E^{a}_{j} / r)} )$

Where: * $r$ is a learnable temperature parameter for scaling the loss * N is the batch size.

Note: the two logarithmic terms consider audio-to-text or text-to-audio logits in each bath.

After training, embeddings are then used for upstream tasks:

Text-to-Audio Retrieval: given text, find the closest audio (or vice versa).
Zero-shot audio classification: construct text like: "the sound of class name", and use to query your audio. This classification style is useful because it allows you to use any category, provided it is in the training data.
Supervised Audio Classification: take audio embeddings and fine-tune them on a classification task.

Audio Encoders and Text Encoders

They experiment with two models for audio encoders:

PANNs

a CNN-based audio classification model with seven downsampling CNN blocks and seven upsampling blocks.

HTS-AT

a transformer-based model with four Swintransformer blocks, achieving SOTAs on three audio classification datasets.

For both, they use the 2nd last layer's output. For PANNs, it's a 2048-sized embedding, and for HTSAT, it's a 768-sized embedding.

For text encoder, they experiment with:

CLIP transformer (output dimension 512)
BERT (output dimension 768)
Roberta (output dimension 768)

Feature Fusion for Variable-Length Audio

Audio is a natural variable length, unlike image data, which we can resize to "unified resolution" (i.e. 224x224).

A common approach is to average per-frame or per-chunk audio embeddings as output. However, this is not computationally efficient on long audio.

Enter Feature-Fusion.

It allows them to train on different lengths of audio inputs in constant computation time by combining coarsely global and randomly sampled local information.

For an audio in T seconds and a fixed chunk duration d = 10 seconds:

$T < d$ :

repeat the input, then pad with zero values. For example, 3-second input will be repeated 3 x 3 = 9 seconds and padded with 1-second zero values.

$T > d$ :

First, downsample the input from T to d-second (10 seconds) as global input. These are the global inputs.
Then, randomly slice three d-second clips: in front 1/3, then middle 1/3 and back 1/3 of the input. These are the local inputs.
Send these $4 \ x \ d$ inputs to the mel encoder to get the initial features.
send the three local inputs to the 2D Conv layer with a 3-stride in the time axis to convert to one feature.
Now fuse the local feature $X^{a}_{local}$ and the global feature $X^{a}_{global}$ : $X^{x}_{fusion} = \lambda X^{a}_{global}.+ (1 - \alpha)X^{a}_{local}$ , where $α = fAF F (X^{a}_{global}, X^a_{local})$ is a factor obtained by Attention Feature Fusion.

Code from the repo:

mel = get_mel(audio_data, audio_cfg)

ranges = np.array_split(list(range(0, total_frames - chunk_frames + 1)), 3)

if len(ranges[1]) == 0:
    # if the audio is too short, we use the first chunk
    ranges[1] = [0]
if len(ranges[2]) == 0:
    # If the audio is too short, we use the first chunk
    ranges[2] = [0]

# randomly choose index for each part
idx_front = np.random.choice(ranges[0])
idx_middle = np.random.choice(ranges[1])
idx_back = np.random.choice(ranges[2])

# select mel parts for local inputs
mel_chunk_front = mel[idx_front:idx_front + chunk_frames, :]
mel_chunk_middle = mel[idx_middle:idx_middle + chunk_frames, :]
mel_chunk_back = mel[idx_back:idx_back + chunk_frames, :]

# shrink the mel to create global inputs
mel_shrink = torchvision.transforms.Resize(size=[chunk_frames, audio_cfg['mel_bins']])(mel[None])[0]

# stack
mel_fusion = torch.stack([mel_shrink, mel_chunk_front, mel_chunk_middle, mel_chunk_back], dim=0)

Compared with the "slice & vote" method, the feature fusion saves training time as we only process audio slices in the first few layers.

Keyword-to-Caption Augmentation

See the Main Contributions section.

Experiments

They evaluated two initial tasks using recall and mean average precision (mAP) on audio-to-text and text-to-audio:

Experiment with different audio and text encoders to get the best baseline model.
Experiment with various dataset sizes, using feature fusion and keyword-to-caption augmentation to verify the efficacy of the proposed methods.

Then, they do zero-shot and supervised audio classification experiments to evaluate the generalisation ability of the downstream tasks.

Hyper-parameters

Audio settings

10-second input length
480 hop size
1024 window size
64 mel-bins to compute STFTs and mel-spectrograms

So, each input sent to the audio encoder is of the shape $(T = 1024, F = 64)$

When training the model without the feature fusion, the audio longer than 10 seconds will be randomly chunked into a 10-second segment.

Text settings

Tokenise the text with a maximum token length of 77

Optimiser

Use the Adam optimiser with β1 = 0.99, β2 = 0.9 with a warm-up and cosine learning rate decay at a basic learning rate of 10-4.

Batch sizes

Batch size of 768 on the AudioCaps + Clotho dataset
2304 on the training dataset containing LAION-Audio-630K
4608 on the training dataset containing the AudioSet

Epochs

We train the model for 45 epochs.

Text-to-Audio Retrieval

They experiment to find the best audio and text encoder for retrieval tasks.

Combine two audio encoders with three text encoders loaded from pre-trained checkpoints.

In this experiment, train on AudioCaps and Clotho datasets, and report the best mAP@10 on audio-to-text (A→T) and text-to-audio (T→A) perspectives

Results:

Audio encoder: HTS-AT better than PANNs combined with RoBERTa or BERT.
Text encoder: RoBERTa beats BERT. CLIP transformer worst.

Dataset Scale

Now, they use the HTSAT-RoBERTa model to perform text-to-audio retrieval experiments across the datasets.

This task uses the same metrics to compute recall scores at different ranks.

In the training set, they gradually increase the scale of the dataset.

They find that scaling up the dataset from "AudioCaps + Clotho" to "LA." does not improve the result on the AudioCaps evaluation set but gets better performance on the Clotho evaluation set.

One reason is that AudioCaps contains audio similar to AudioSet, on which the audio encoder's loaded checkpoint is pre-trained.

When the model receives more data from other sources, it increases its generalisation but moves the distribution out of AudioSet data.

Therefore, the performance on AudioCaps drops, but that on Clotho increases a lot, demonstrating a trade-off of the model to keep the performance among different types of audio.

Keyword-to-Caption and Feature Fusion

They observe performance improvements when adding the feature fusion mechanism and keyword-to-caption augmentation to the model.

Feature fusion is effective, especially in the Clotho dataset, because it contains longer audio data (> 10 seconds).

When they add AudioSet into the training set with either template prompting or keyword-to-caption augmentation, we can see the performance increases again on AudioCaps while decreasing on Clotho.

This result further confirms the trade-off performance between AudioCaps and Clotho datasets mentioned above.

Also, keyword-to-caption augmentation works better than the template text prompting method on most metrics.

As a result, our best model outperforms previous methods on most metrics (mainly R@1=36.7% on AudioCaps and R@1=18.2% on Clotho) in the text-to-audio retrieval tasks.

We show that training on large-scale datasets (LAION-Audio-630K and AudioSet with keyword-to-caption augmentation) and feature fusion can improve model performance.

Zero-shot Audio Classification

To study the model generalisation and robustness, they conducted zero-shot audio classification experiments on three top-performing models in previous experiments.

They evaluate models on three audio classification datasets: ESC50, VGGSound, and Urbansound8K. They use top-1 accuracy as the metric.

They classify audio by performing audio-to-text retrieval with each text corresponding to the text prompt converted from class label via "This a sound of the label.".

There's some dataset overlap between the training data and the zero-shot dataset. They excluded all the overlap samples and performed a zero-shot evaluation on the remaining dataset.

As shown in Table 4, the models achieve new SoTAs of zero-shot audio classification across all three datasets.

Keyword-to-caption augmentation increases the performance of VGGsound and US8K by adding more text captions to "enrich" the text embedding space.

Supervised Audio Classification

They perform supervised audio classification by fine-tuning the audio encoder on FSD50K and VGGSound datasets. They do not run experiments on ESC50 or Urbansound8K due to data leakage concerns. mAP is used as an evaluation metric.

Feature-fusion enables the model to handle variable-length input and performs better than previous models.

They outperform the current state-of-the-art on the VGGSound dataset, close to the state-of-the-art on the FSD50K dataset.

Future Work

They want to collect an even larger dataset.

Apply representations to other downstream tasks like audio synthesis and separation.

CLIP

2023-12-06T00:00:00+10:00

Contrastive Language-Image Pretraining or CLIP is an approach to training a model to associate images with their textual representations using Contrastive Loss. This allows for high-performing Zero-Shot Learning i.e. the model can generalise to new tasks without fine-tuning.

The architecture is a "simplified version of ConVIRT" ¹ trained from scratch.

From paper Learning Transferable Visual Models From Natural Language Supervision

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020. https://arxiv.org/abs/2103.00020 ↩

Predicate Logic

2023-10-13T00:00:00+10:00

An extension of Propositional Logic that uses variables and quantifiers to represent and analyse Statement.

Predicate

Is a Statement that includes a variable.

$P(x)$ : "x is a prime number"

A predicate becomes a proposition when the variable are substituted for values.

$P(2)$ : "2 is a prime number" (True)

Quantifiers

Quantifiers describe how many of a thing there are.

Universal Quantifier

Symbol: $\forall$

Means "For all" or "Every".

Example:

$\forall x, P(x)$ : "For every x, x is a prime number"

Existential Quantifier

Symbol: $\exists$

Means "There exists" or "Some".

Example:

$\exists x, P(x)$ : "There exists an x such that x is a prime number"

DeMorgan's Laws for negating quantifiers

First law

The negation of "for all x, P(x)" is equivalent to "there exists an x such that not P(x)"

$\sim[(\forall x)P(x)] \equiv (\exists x)[\sim P(x)]$

Second law

The negation of "there exists an x such that P(x)" is equivalent to "for all x, not P(x)"

$\sim[(\exists x)P(x)] \equiv (\forall x)[\sim P(x)]$

Propositional Logic

2023-10-12T00:00:00+10:00

Propositional Logic is a system that deals with Propositions (statements).

The truthfulness or falsity of a proposition is called its Truth Value. Denoted by $T$ or $F$ , or 1 and 0 in computer science.

We can use connectives to change or combine the meaning of propositions. For example, $\neg p$ negates the value of p. If it's true, it becomes false and vice versa.

Truth Table

A truth table allows us to consider all possible combinations of proposition logic systems.

For example, consider $\neg p$ :

$p$	$\neg p$
1	0
0	1

We can use truth tables to help us understand the truth values of other connectives within propositional logic.

Connective

Negation (NOT)

Symbol: $\neg p$

An operator that negates a proposition.

$p$ = I will pass my exam.
$\neg \ p$ = I will NOT pass my exam.

In Boolean Algebra, it's equivalent to $1 - T(p)$

Truth table

$p$	$\neg p$
1	0
0	1

Disjunction (OR)

Symbol: $p \lor q$

True when p OR q is true.

Truth table

$p$	$q$	$p \lor q$
1	0	1
0	1	1
1	1	1
0	0	0

Equivalent to $\max(T(p), T(q))$

Conjunction (AND)

Symbol: $p \land q$

True only when p AND q is true.

Truth table:

$p$	$q$	$p \land q$
1	0	0
0	1	0
1	1	1
0	0	0

Equivalent to multiplication $T(p) \times T(q)$

Implication (If...Then)

Symbol: $p \rightarrow q$

If p is true, then q is true.

Truth table

$p$	$q$	$p \rightarrow q$
1	1	1
0	1	1
1	0	0
0	0	1

Think of it as a promise. Only false when promise is broken.

Bi-conditional: $\leftrightarrow$

Symbol: $p \leftrightarrow q$

Truth table

$p$	$q$	$p \leftrightarrow q$
1	1	1
0	1	0
1	0	0
0	0	1

Equivalent to equality check: $T(p) = T(q)$

Exclusive-Or

Symbol: $p \oplus q$

p or q but not both.

Also called XOR.

Truth Table

$p$	$q$	$p \oplus q$
1	1	0
0	1	1
1	0	1
0	0	0

Truth table is opposite of bi-conditional.

Operator Precendence

$\neg$
$\land$
$\lor$
$\rightarrow$
$\leftrightarrow$

Tautology

A statement that is always true.

For example: $p \lor \neg p$ is always true.

$p$	$\neg p$	$p \lor \neg p$
1	0	1
0	1	1

Consistent

A formula that is true in at least one scenario.

Contradiction

A formula that is never true.

For example: $p \land \neg p$

$p$	$\neg p$	$p \land \neg p$
1	0	0
0	1	0

Also called "inconsistent".

Equivalence

If two formula are equivalent if they have identical truth tables.

Denoted by: $\equiv$

$A \equiv B$ means A and B have the same Truth Table.

Note: equivalence is relation, not connective.

Can prove De Morgan's Laws using a truth table.

$\neg (p \land q) \equiv \neg p \lor \neg q$

$p$	$q$	$\neg (p \land q)$	$\neg p$	$\neg q$	$\neg p \lor \neg q$
1	1	0	0	0	0
1	0	1	0	1	1
0	1	1	1	0	1
0	0	1	1	1	1

$(p \rightarrow q) \equiv (\neg p \land q)$

Laws of logic

See Laws of Logic.

Propositions

2023-10-11T00:00:00+10:00

A proposition (also called a statement) is a declarative sentence with a truth value of either true or false, but not both.

Every proposition must be:

A complete sentence that makes a claim
Either true or false (but not both)
Definite in its truth value

For example, these are valid propositions:

"I wrote this article on Thursday" (true)
"I am 14 years old" (false)
"1 + 1 = 3" (false)
"Water boils at 100°C at standard atmospheric pressure" (true)
"Every even integer is divisible by 2" (true)

These are not propositions:

"What time is it?" - Questions aren't propositions as they don't make claims
"Close the door" - Commands don't have truth values
"What a beautiful day!" - Exclamations and opinions without clear criteria aren't propositions

Also, expressions with variables, where the value of the variable would affect the truth value, aren't propositions:

$x + 2 = 5$ - Not a proposition, as its truth value depends on the value of $x$ .

When a statement contains a variable whose truth value depends on that variable's value, it's called a Predicate.

For example:

"I am $X$ years old"
$Y + 1 = 5$
"The speed of light in a vacuum is approximately $Z$ meters per second."

Which of these is a valid proposition?

a) I live in Australia.
b) Z > 2
c) Please don't do that anymore
d) $1 + 2$
e) $11 \times 11 = -1$

Answer

a) I live in Australia.
e) $11 \times 11 = -1$
Only (a) and (e) are valid propositions because:
* "I live in Australia" is a declarative statement that is either true or false
* $11 \times 11 = -1$ is a mathematical statement that is definitely false
* (b) contains a variable, making it a predicate
* (c) is a command
* (d) is an expression, not a statement

In propositional logic, we often assign propositions to variables, typically using lowercase letters $p$ , $q$ , $r$ , $s$ , or $t$ :

Let $p = \text{"It rained yesterday"}$
Let $q = \text{"I am happy"}$

These variable assignments allow us to manipulate and analyse propositions using Propositional Logic, where we can combine simple propositions to form more complex ones.

Mel Spectrogram

2023-10-09T00:00:00+10:00

Mel Spectrogram is a graphic representation of a Sound Wave, visualising frequency over time. The difference between a Mel Spectogram and a Spectrogram, is the frequency y-axis is represented using the Mel Scale in the former.

Here is a Mel Spectrogram of the audio of a Trumpet

The process of generating a Mel Spectrogram works like this:

Break the audio signal down into short frames
Convert time signal into the frequency domain using a Fourier Transform
Convert frequencies into Mel scale, to more closely align with our intuition of frequencies. This operation is called Mel Filter Bank.
Plot the Mel values over time.

Mel Scale

The Mel Scale is a perceptual scale of audio frequencies. In other words, it represents our perceived distance of the frequencies from others.

The Mel scale is a logarithmic formula where 1000Mel = 1kHz. You can convert Hz to Mel using this formula:

$Mel(f) = 2595 \log_{10} (1 + \frac{f}{100})$

Sample Rate

2023-10-08T00:00:00+10:00

Sample rate measures how accurately a sound wave is digitally represented.

To create a digital audio recording of a Sound Wave, we capture the signal's amplitude many times a second and store it as an array of numbers. Each of these numbers is called a sample. We measure audio quality by number of samples per second or hertz.

The number is generally stored as an int or float, whose range or precision is defined by the Bit Depth

Together, they are the two critical details that define the quality of a digital audio recording.

If we zoom into an audio file in Audacity to the higher resolutions, we can visualise our waveform at the sample level:

The number of elements in the array is $\text{ audio time (secs) } \times \text{ sample rate }$ . Therefore, higher sample rates will require more storage space. We will have an array per channel for stereo and other multichannel audio.

In Python, the sound is typically represented using a Numpy multidimensional array. Here we can see an example of loading an audio file using the scipy library:

>>> from scipy.io import wavfile
>>> sample_rate, audio_array = wavfile.read("../../_media/4s-silence.wav")
>>> audio_array.shape
(176400,)
>>> audio_length = len(audio_array) / sample_rate
>>> audio_length
4.0

As you can see, we can find the length of audio represented as a Numpy array by dividing the number of samples by the sample rate.

A higher sample rate means that we can store higher frequencies, since higher frequency sound waves have shorter cycles, and therefore need more samples to capture.

How do we determine the optimal sample rate?

Nyquist-Shannon sampling theorem

According to the Nyquist-Shannon Sampling Theorem the frequency of sampling of a wave must be greater than twice the highest frequency in a wave. Since the highest frequency the human ear can hear is around 20 kHz, anything about 40 kHz should perfectly reproduce what the human ear can perceive.

The sample rate for CD audio is 44.1kHz, which is two times 22.05 kHz - that gives a bit of extra buffer for very high frequencies.

44.1kHz continues to be a standard sample rate for high-quality audio, although 48kHz is also a common choice.

However, there are other standard sample rates for different types of audio.

Common Sample Rates

44.1 kHz

The most commonly found sample rate, as it has been the standard for CD quality since inception.

48 kHz

For audio and some film and video, as it divides evenly for film/video rates.

88.2 kHz and 96 kHz

For higher-resolution audio formats

192 kHz

For some ultra-high-definition recordings.

Aliasing

Since the true sound wave has to be inferred from digital samples, the sound will only be accurately captured if the rate is higher. In particular, the higher frequencies will be folded into lower frequencies, causing distortion - this issue of under-sampling is referred to as aliasing.

The figure below shows an example of a 15Hz sine wave over a minute. As you can see, we can only accurately reconstruct the original sine wave if we sample enough points. However, after a certain number of samples, we can rebuild the sound wave perfectly; more samples do not help.

Real sound waves are more complex than simple sine waves, so more samples are needed to capture that complexity. However, the important detail is that more samples are not necessarily better. We can accurately represent all audio we perceive at any sample rate above 40kHz.

Bit Rate

Bit Rate is the number of bits per second required to store or transport the audio signal. For uncompressed audio, that is simply the number of bits per sample x sample rate. For mono audio with a bit depth of 16, and a sample rate of 44,100Hz, we could calculate it as: 16 bits x 44,100Hz = 705,600 bits/s.

Sound Wave

2023-10-06T00:00:00+10:00

A sound wave is a wave of pressure that travels through the air (or another medium like water or solids) that our ears can perceive.

Sound waves are produced when when a vibrating object exerts force or pressure, setting the molecules to oscillate around their equilibrium position.

The rate of oscillation of these molecules is what determines the frequency of the sound, while the maximum distance they move away from their equilibrium position defines the amplitude.

These characteristics give a specific sound its unique pitch and loudness.

We measure the frequency of sound in cycles per second, known as hertz (Hz). For instance, 440 Hz indicates 440 cycles per second.

The amplitude of a sound is gauged by the maximum displacement of the medium's molecules.

This displacement results in pressure fluctuations within the medium, which we express in units known as the decibel sound pressure level (dB SPL).

The amplitude largely determines the loudness of a sound.

Individuals with typical hearing can perceive sounds as soft as 0 dB SPL and generally tolerate up to around 120 dB SPL.

Law of Sines

2023-08-05T00:00:00+10:00

The Law of Sines tells us that the ratio between the sine of an angle and the side opposite will be constant for any angle in a triangle.

$\frac{\sin A}{a} = \frac{\sin B}{b} = \frac{\sin C}{c}$

If we have at least two sides and an angle or two angles and a side, we can use it to find the missing values.

Example

Question

Consider a triangle with angles $A$ , $B$ and $C$ and sides $a$ , $b$ and $c$ , where $a$ is the side opposite $A$ , $b$ opposite $B$ and $c$ opposite $C$ .

If $C = 42°$ , $c = 15cm$ and $b = 1cm$ , solve the triangle.

Working

We can start by finding $B$ using the Law of Sines:

Since we know that $\frac{\sin 42°}{15cm} = \frac{\sin B}{1cm}$

We can calculate $\frac{\sin 42°}{15} = 0.0446$

$\frac{\sin B}{1} = 0.0446$

Since dividing by 1 equals the numerator, we know:

$\sin B = 0.0446$

Then use $\arcsin$ to find $B$

$B = arcsin(0.0446)$

$B = 2.557°$

We can now find $A$ since we know that all the angles in a triangle add up to $180°$

$A = 180° - 42° - 2.56°$

$A = 135.44°$

Now to find $a$ , we can use the Law of Sines again:

$\frac{\sin 42°}{15cm} = \frac{\sin(135.44)}{a}$

Calculate the known values:

$0.0446 = 0.702 / a$

Multiply both sides by A:

$a \times 0.0446 = 0.702$

$a = \frac{0.702}{0.0446} = 15.74cm$

Now that we have the missing values, we can use the Law of Sines to check that all ratios are equal:

$\frac{sin(135.44°)}{15.74} = 0.045$

$\frac{sin(2.56°)}{1} = 0.045$

$\frac{sin(42°)}{15} = 0.045$

Answer

$A = 135.44°$

$B = 2.56°$

$C = 42°$

$a = 15.74cm$

$b = 1cm$

$c = 15cm$

Making Song Covers With My AI Voice

2023-05-07T00:00:00+10:00

Recently, there has been a lot of talk about so-called AI music, which in May 2023, likely refers to taking a song made by a person and applying a voice conversion effect to the vocals to make it sound like Kanye, Drake, or any other artist singing the lyrics. Though if you're reading this in the future, AI music might mean something entirely different.

This weekend I wanted to play with this voice conversion technology to make AI covers of my own. But, instead of taking a song's vocals and making them sound like Drake, I wanted to take some tunes and hear me singing them.

The results were amazing! I created a decent model with less than 20 minutes of voice data that sounded like I could sing with perfect pitch across multiple languages. I have no natural singing talent at all, as you can listen to in the example later in the article.

If you are eager to get straight to the point: this is the finished product and SoVITS is the tool I used.

This article aims to give a high-level look at SoVITS and shows how I trained my voice model.

I want to share some thoughts about the implications for the music industry and society.

Singing Voice Conversion (SVC)

The study of Voice Conversion aims to modify speech audio to make it sound like a different person is speaking. It has legitimate applications in speech therapy, accessibility, entertainment and many other domains. It also has massive potential for misuse, like identity theft, fraud, and starting world wars.

As the name suggests, singing voice conversion, or SVC, is about taking vocals and making them sound like a different singer. It's just voice conversion + pitch.

The popularity of SVC has taken off in recent months, with Discord channels forming to allow people to share artist models and datasets, as well as tips for training models and inference. Social networks are full of AI covers, like Biggie rapping the song N.Y. State of Mind and new bangers by Drake, which he has no involvement in and are sure to be taken down.

The most widely used implementation of SVC is from a repository called so-vits-svc, whose name is a blend of SoftVC and VITS. Some alternative implements like RVC exist, which improves training speed and requires less training data.

The core idea of the system at training time is to learn a representation of the singer's voice, called an embedding, which captures details of the voice's characteristics. Then at inference time, combine an embedding representation of the source speaker with the target embedding and decode it into a mel spectrogram (a kind of image for sound) before vocoding into audio.

Though the original so-vits-svc project is now archived, many forks have sprung up that add various functionality and simplify training and inference.

Creating a dataset

Creating the dataset is the first step to training an SVC model on a new target voice.

You can train a new voice with around 100 clips of the target speaker's voice, each between 5-15 seconds (people commonly recommend making them ~10 seconds). You should trim excessive silence and ensure the vocals have minimal audio layers and processing.

I recorded myself singing six songs across different genres, including rap. I picked songs whose lyrics I know very well. In future, I would choose additional pieces that capture a range of timbres and pitches to improve the model. I used an app called Voloco to record the vocals on my phone, which provides some tools for pitch correction and vocal cleaning. I recorded the songs in my closest facing towards my clothes to minimise unwanted noise (ChatGPT said it would help).

You can listen to the audio clips here, though a warning, I am a terrible singer with an annoying, nasally voice.

Next, I loaded each wav into Audacity and normalised them via the Effects menu > Normalise, as the recording was quiet.

Then, create and extract ~10-second clips around silence. You can highlight a region in Audacity and press Cmd + b (Ctrl + b on PC) to create a label. You don't need to add any text.

Then you can export each label as an audio clip via File > Export > Export Multiple.

I have a folder with about 113 clips and about 14 minutes of clean audio.

lex@lex-macbook-m1:~/datasets/lex> ls
LEX_ALLBLACK-01.wav     LEX_CHILDRENS_STORY-14.wav  LEX_LITHIUM-21.wav
LEX_ALLBLACK-02.wav     LEX_CHILDRENS_STORY-15.wav  LEX_LITHIUM-22.wav
# ...
lex@lex-macbook-m1:~/datasets/lex> ls -l | wc -l
113

Training the model

The authors of the so-vits-svc forks have made the training process easy. In this Colab notebook, you merely upload your dataset to your Google Drive and run all the cells and end up with a trained model.

Here's a breakdown of what's going on in the notebook.

1. Mount your Google Drive to access your training set.

from google.colab import drive
drive.mount('/content/drive')

2. Unzip the dataset to a folder called dataset_raw.

!mkdir -p "dataset_raw"
!unzip /content/drive/MyDrive/svc/LexDataset.zip -d dataset_raw/lex

3. Run the preprocessing scripts. These resample the audio to 44khz, normalise the audio and prepare the data for training.

svc pre-resample
svc pre-config
svc pre-hubert -fm dio

4. Train the model. You specify a path on Google Drive to dump the checkpoints for easy resume ability.

svc train --model-path drive/MyDrive/svc/lex-20230506/logs/44k

I trained it for a few hours to about 2k epochs, then downloaded the weights to my local machine to perform inference.

Transfer vocals to other songs

This idea of an AI cover is possible thanks to SVC and Meta's magical source separator tool called Demucs. It separates songs into their stems: drums, bass, other instruments and vocals. It works unbelievably well.

In this inference example, I took a cover of Fleetwood Mac's Dreams from YouTube by a singer called Lanie Gardney. The reason to use a cover version over the original is that the vocals have less layering and processing, but it still worked okay with the original song.

I split the stems using the htdemucs_ft model:

> demucs --device cpu --name htdemucs_ft songs/Dreams.wav

It is slow on a CPU but okay on a GPU. The htdemucs split stems about 4x faster than htdemucs_ft, so that's another option for a slight quality penalty.

And now I have a folder that contains the song stems:

> ls separated/htdemucs_ft/Dreams/
bass.wav    drums.wav   other.wav   vocals.wav

And can use the svc infer command to run inference using my voice model:

svc infer \
--model-path models/lex/G_2057.pth \
--config-path models/lex/config.json \
--transpose 0 \
--cluster-model-path "models/lex/means.pt" \
--cluster-infer-ratio 0.5 \
--wav_format wav \
separated/htdemucs_ft/Dreams/vocals.wav

The inference script takes in a few parameters that are worth paying attention to:

--model-path - the path to the trained weights (a G_*.pth file).
--config-path: the config used for training
--transpose allows you to transpose the source voice before inference. Usually, you would transpose -12 for female-to-male or +12 for male-to-female. Although, it is incredible to hear my voice in the range of Stevie Nicks.
--auto-predict-f0 is an alternative to --transpose, automatically adjusting the pitch to match the target speaker. It works amazingly for rap, but for singing, it can alter the pitch and make the vocals out of the key.
--cluster-model-path and --cluster-infer-ratio can choose an optional clustering scheme to make the trained sound more like the target's timbre. But it makes the results less clear sounding. A fusion method can control the balance between timbre and clarity, allowing an appropriate trade-off point to be manually adjusted. 0.5 is a good starting point.

Now I have my converted vocals in the results folder.

Here's how they sound side-by-side against the source.

Original:

Transferred vocals with transpose 0 (singing in Larie's/Stevie's pitch):

Transferred vocals with transpose -12 (probably closer to my range if I could sing):

And just for good measure, here's me trying to sing it without any autotune (warning: it is bad. I swear I did my best here):

Then I added the stems and my new vocals into a Logic Pro project and applied these effects:

a noise gate to clean out some artefacts.
eq to reduce some noisy parts and remove sub rumble.
compressor to flatten the peaks
reverb and delay, which masks imperfections.

And here's the finished song on YouTube.

I also tried my voice on my wife's favourite Taiwanese band, Mayday, and it worked well.

I tried transferring to rap, and it had mixed results. I thought Regulate worked pretty well.

In future, I'm planning to run some additional experiments:

experiment with smaller/larger datasets.
experiment with just speaking training data transferring to rap and singing.
trying non-human voices. I'm keen to hear how my dog sounds singing the classics.

The implications for the music industry

Will this effectively destroy the music industry as we know it? Probably not.

Most uploaded AI covers are already tagged and monetised as per any streaming platform, so it's just more streaming engagement for labels.

The songs made by unknown producers, using a famous voice like Drake's to get publicity, will likely get taken down for now as they a) violate the artist right to publicity and b) trained on a corpus such that it violates copyright law. Eventually, I think social networks will automatically classify SVC vocals and offer to remove the offending material or compensate the label, and the labels will go back to the BAU monetisation system for working with streaming services.

Some artists are already starting to monetise their voice models. Grimes already set up a tool to use her AI voice in exchange for a 50% proceeds split.

The reality is that people will likely get sick of the deep fake music, and I imagine the fad will die off as attention shifts to the next phase of AI music, whatever that is.

The next question is: if anyone can sing in perfect pitch, does that invalidate natural talent?

Again, probably not. There's more to music than the audio file; people want to connect to a person with a story. Humans will always have a role to play in making music for other humans.

Chess is more popular than ever, even though AI mastered it in the 80s.

The implications of deep fakes in society

The true implication of this tech, and deep fakes in general, is that we can only trust voice (and video) recordings with sufficient verification of authenticity.

If you have a bank using only a voice recognition service to authorise you, I recommend turning off this feature immediately or changing banks if you cannot.

Voice conversion isn't a future technology; it works exceptionally well in the present. As you can see from the simplicity of training a model, it's a tool available to everyone, regardless of technical capability.

Perhaps, we should be wary of how much of our audio we publicly release to the world.

Now that we know the potential for misuse and danger, we must educate our family and friends about the potential for fakers to scam us.

Verify and check everything you hear.

Conclusion

I created a decent model with minimal effort that sounded like I could sing across languages with perfect pitch. This technology is super easy to use and will only get easier. The music industry will be fine - they will likely cash in on it, and the demand for human-made music will continue. Still, deep fakes will be an increasingly significant problem for society, and we should be aware of how capable the tools are right now.

Recursion

2023-04-15T00:00:00+10:00

Recursion is a problem-solving method where a Function calls itself with a smaller instance of the problem. A Base Case is required to ensure the calls eventually terminate.

All recursive problems can be solved using iteration; however, some algorithms, particularly a Divide-and-Conquer algorithm, can be solved much more elegantly with recursion.

Fibonacci numbers example

The canonical example of a recursive solution is the Fibonacci number sequence. The sequence is the sum of the previous two numbers:

$0, 1, 1, 2, 3, 5, 8, 13, ..., n$

It can be expressed recursively as: $F_n = F_{n - 1} + F_{n - 2}$

The base cases are:

$\text{Fibonacci}(1) = 1$
$\text{Fibonacci}(0) = 0$

We visualise a call to $\text{Fibonacci}(5)$ by representing the call stack as a tree, like this:

Source dhovemey at ycp (page now unavailable)

As you can see, each branch opens two new branches until we finally reach the bottom of the tree (the base case) and can finally propagate the answers back to the top.

In pseudocode:

function Fibonacci(n)
     assert n >= 0

    if n < 2 then
        return n
    end if

    return Fibonacci(n-1) + Fibonacci(n-2)

end function

Factorial example

The Factorial algorithm is:

$n! = n \times (n - 1) \times (n - 2) \times ... \times 2 \times 1$

Which can be expressed recursively simply as:

$n! = n \times (n - 1)!$

The base case for Factorial comes at $0! = 1$

We can rewrite the function in pseudocode as follows:

function Factorial(n)
    if n = 0 then
        return 1
    end if

    return n x Factorial(n - 1)

end function

Function

2023-04-09T00:00:00+10:00

A function is a rule that associates inputs with outputs.

They form the core of many aspects of mathematics and numerous programming languages.

The fundamental explanation of functions comes from Set Theory, in which a function is regarded as the mapping from one set, $A$ , to another set, $B$ , expressed as:

$f : A \rightarrow B$

Functions are commonly denoted using the letters $f$ , $g$ , $i$ , or $j$ .

Consider a function, $f$ , that maps a set of people's names to their ages:

$A = \{\text{Sarah}, \text{Geoff}, \text{Clyde}, \text{Betty}\}$
$B = \{0, 1, 2, \cdots, 120 \}$
$f : A \rightarrow B$

The set of possible inputs is called the Domain of a Function or $D_f$ :

$\color{lightblue}D_f = A = \{Clyde, Sarah, Geoff, Betty\}$

The set of possible outputs is the co-domain or $co-D_f$ of the function.

$\color{orange}coD_f = B = \{0, 1 ... 120\}$

In this case, the co-domain includes all plausible human ages.

Given the input set $A$ , the set of possible outputs is $\{11, 34, 98\}$ . This set is known as the range of the function ( $R$ ).

$\color{darkred}R = \{11, 34, 98\}$

To show a single input-output relationship, we could write:

$f(Sarah) = 34$

The output 34 is the function's image, and the corresponding input, $Sarah$ , is the pre-image.

Functions are considered a "well-behaved relation". That means that for each input, there must be exactly one output. This example qualifies as a function because each person has a unique, valid age.

We can express functions as the relationship between an input variable and its output. For example, the function to convert temperature in Fahrenheit to Celsius is as follows:

${f(\text{x}) = (x - 32) \times \frac{5}{9}}$

The complete definition of a function should include its domain and co-domain. Since Fahrenheit and Celsius are real numbers, so we would define the function using the Special Infinite Set $\mathbb{R}$ .

$f : \mathbb{R} \rightarrow \mathbb{R}$

The two parts combined give the complete definition of the function:

Let $f: \mathbb{R} \to \mathbb{R}$ , $f(x) = (x - 32) \times \frac{5}{9}$

In programming languages, mixing the type declaration with the implementation is common. Below is an example of the function $f$ in Python. It takes an input $x$ as a float and returns a float, described using the notation -> float.

def f(x: float) -> float:
    return (x -  32) * (5/9)

Plotting Functions

We can create a set of input values and their corresponding outputs, then visualise them geometrically by drawing the inputs and outputs on the x-axis and y-axis, respectively. This visualisation is called a graph of a function.

Here is a plot of the Fahrenheit to Celsius function earlier, plotted across a range of inputs: from -100 to 100.

When the graph is a straight line like this, it's called a Linear Function. There are other names for common function types:

Linear Function: A function where the output is proportional to the input.
Quadratic Function: A function where the output is proportional to the square of the input.
Exponential Function: A function where the output is proportional to a fixed base raised to the power of the input.
Polynomial Function: A function that we represent as a sum of terms, each consisting of a constant multiplied by a variable raised to a non-negative integer power. Linear and quadratic functions are specific types of polynomial functions.

There are some other important properties of functions:

One-to-one / Injective

We consider a function "one-to-one" or "injective" if each output is associated with exactly one input and no two different inputs have the same image.

Onto / Surjective

A function is "onto" or "surjective" if every element in the co-domain is output for at least one input in the domain.

Bijective

We call a function Bijective if it is both injective and surjective.

Continuity

A function is continuous at a point x = c under the following conditions:

f(c) is defined.
The limit of f(x) as x approaches c exists.
The limit of f(x) as x approaches c is equal to f(c).

That is, $\lim_{x \rightarrow c} f(x) = f(c)$

A function is discontinuous at a point x = c if any of the above conditions are not met.

A function might only have discontinuatities specific internals

Some special cases apply:

Polynomials are always continuous.
Rational functions: Continuous when the denominator is not zeo.
Trig functions: continuous on their domain.
Exponential and log functions: continuous when defined.

Graph

2023-04-09T00:00:00+10:00

A Graph Theory is a visual representation of interconnected systems using circles and lines. Circles represent nodes or vertices. Lines represent links or edges. Graphs are used to analyze and solve problems in various interconnected systems.

Historical Origin: The Seven Bridges of Königsberg

Graph theory originated from a famous mathematical problem in the 18th century in Königsberg (now Kaliningrad, Russia). The city had seven bridges connecting four land areas separated by the Pregel River. The puzzle asked whether it was possible to walk through the city crossing each bridge exactly once.

In 1735, mathematician Leonhard Euler proved this was impossible by abstracting the problem. He represented land masses as points (A, B, C, D) and bridges as lines connecting them, creating what would become the foundation of graph theory.

Euler's insight was that for a complete path to exist (crossing each bridge exactly once), each land mass except for the starting and ending points must have an even number of bridges connected to it. Since all four land areas in Königsberg had an odd number of bridges, such a path was impossible.

Graph Classifications

Graphs can be classified in several ways based on their properties:

Directed vs. Undirected

Directed Graphs

In directed graphs, edges have a specific direction indicated by arrows. These represent one-way relationships.

Examples: Twitter following relationships, one-way streets in a city map, workflow diagrams

Undirected Graphs

In undirected graphs, edges have no direction and represent symmetrical relationships.

Examples: Facebook friendships, telecommunication networks, chemical bonds

Weighted vs. Unweighted

Weighted Graphs

In weighted graphs, edges have different values or importance attached to them.

Examples: Road networks where weights represent distances, communication networks where weights represent bandwidth

Unweighted Graphs

In unweighted graphs, all edges have equal importance.

Examples: Simple connection diagrams, logical relationships

Common Graph Topologies

Graphs come in various standard topologies, each with specific use cases:

Bus Topology

Nodes are arranged in a line, with each node connected to adjacent nodes.

Ring Topology

Similar to a bus, but the last node connects back to the first, forming a circle.

Tree Topology

Follows a hierarchical tree data structure, with a root node and branches.

Regular Manhattan Topology

A grid-like structure resembling city blocks.

Arbitrary Mesh Topology

Random interconnected pattern with no specific structure.

Real-World Applications

Graphs have countless real-world applications across different domains:

Literature Analysis

In literature analysis, graphs can represent character interactions. For example, in "Les Misérables," nodes represent characters and edges show interactions between them, revealing the structure of relationships within the novel.

Movie Plot Mapping

Similar to books, graphs can be used to visualize character interactions in films. For instance, in "The Imitation Game," a graph could show relationships between Alan Turing and other characters, highlighting the central role of certain individuals.

Computer Networks

In computer networks, nodes represent routers or computers, while edges represent communication links. The 1991 NSFNET (National Science Foundation Network) backbone can be visualized as a graph showing the early structure of what would become the internet.

Social media platforms use graph theory extensively. Nodes represent people, and edges indicate friendships or following relationships. This helps identify communities, influential users, and information flow patterns.

Graph theory provides a powerful framework for understanding complex systems across disciplines, from mathematics and computer science to sociology and literature.

Graph Theory

2023-04-09T00:00:00+10:00

See Graphs

Linear Function

2023-04-09T00:00:00+10:00

A linear function is represented by a straight line on a graph. It indicates a linear relationship between two variables: when one changes, the other changes at a constant rate. For example:

The count of floors vs number of windows on a high-rise building.
The distance travelled vs time at a constant speed.
The number of beers consumed vs blood alcohol content.

The general form of a linear function is: $f(\mathbf{\textcolor{#8ba7b9}{x}}) = \mathbf{\textcolor{#b9b28b}{m}}\mathbf{\textcolor{#8ba7b9}{x}} + \mathbf{\textcolor{#b98b99}{b}}$ where $\mathbf{\textcolor{#b9b28b}{m}}$ is the slope of the line and $\mathbf{\textcolor{#b98b99}{b}}$ is the intercept.

The slope, $\mathbf{\textcolor{#b9b28b}{m}}$ , indicates how steep the line is and in which direction it points.

The point $\mathbf{\textcolor{#b98b99}{b}}$ is the y-coordinate of the point where the line intercepts on the y-axis, which tells us the value of the function when $\mathbf{\textcolor{#8ba7b9}{x}}$ is zero.

Quadratic Function

2023-04-09T00:00:00+10:00

Quadratic functions are of the form: $f(x) = ax^2 + bx + c$ where $a$ , $b$ and $c$ are constants.

The functions graph as parabolas - a U-shape curve that open upwards or downs depending on the sign of $a$ . If $a>0$ , the parabola opens upwards, and if $a<0$ , it opens downwards.

In this example, I've used $a = 1$ , $b = -2$ and $c =1$ , giving us the function $f(x) = x^2 - 2x + 1$

TF-IDF

2023-04-09T00:00:00+10:00

TF-IDF (Term Frequency - Inverse Document Frequency) is a word numerisation technique in NLP which weights terms by their importance in the document relative to a corpus. The idea is that:

Words that appear frequently in a specific document should be weighted higher.
Words that appear frequently across all documents (like "the", "is", and "and") have less signal and are weighted lower.

An alternative numerisation approach to the Bag-of-Words Model, which simply counts word frequency in a text - TF-IDF includes context about how important each word is compared to the corpus as a whole.

Step-by-step

1. Term Frequency (TF)

Measures how frequently a term occurs in a document.

$\text{tf}(t, d) = \frac{\text{count of \textit{t} in \textit{d}}}{\text{total words in \textit{d}}}$

2. Inverse Document Frequency (IDF)

$\text{idf}(t) = \log\left(\frac{N}{1 + df_t}\right)$

Measures how rare a term is across the entire corpus of documents.

$N$ : Total number of documents.
$df_t$ : Number of documents containing the term.
The +1 (smoothing) prevents division by zero if a term isn't in the training corpus.
The log function dampens the magnitude of the IDF weight, ensuring that extremely rare words don't overpower the entire vector.

3. TF-IDF Score

The final score is the product of these two metrics:

$\text{tf-idf}(t, d) = \text{tf}(t, d) \times \text{idf}(t)$

Words that are frequent in a specific document but rare across the corpus receive the highest scores, making them the "signature" terms for that document.

4. Normalisation

TF-IDF vectors are typically normalised to unit length using L2 (Euclidean) normalisation:

$\hat{v} = \frac{v}{|v|_2} = \frac{v}{\sqrt{\sum_i v_i^2}}$

Otherwise, longer documents would naturally have higher TF-IDF magnitudes.

Scikit-learn provides TfidfVectorizer, which can convert a collection of text documents into a matrix of TF-IDF features. Here we visualise a heatmap of 4 simple texts to illustrate TF-IDF in effect:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are great pets",
    "the mat is on the floor"
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(corpus)

df_tfidf = pd.DataFrame(
    tfidf_matrix.toarray(), 
    columns=vectorizer.get_feature_names_out(),
    index=[f"Doc {i+1}" for i in range(len(corpus))]
)

plt.figure(figsize=(10, 6))
sns.heatmap(df_tfidf, annot=True, cmap="YlGnBu")
plt.title("TF-IDF Scores Heatmap")
plt.xticks(rotation=45)
plt.show()

Cardinality

2023-04-08T00:00:00+10:00

The cardinality of a Set refers to the number of elements it contains.

In math notation, we represent the cardinality of a set $S$ as $|S|$ . For example, the set $S = \{1, 2, 3\}$ has a cardinality of 3, expressed as $|S| = 3$ .

In machine learning, the "cardinality of a feature" denotes the number of unique elements or categories within that feature. High-cardinality features may require feature engineering or be excluded entirely (for example, user_id).

Use of the vertical bar `|A|` notation

Initially it seemed confusing to me that mathematical notation employs the vertical bar symbol for different purposes.

For instance:

The absolute value of a number $a$ is expressed as $|a|$ .
The Matrix Determinate of a matrix $\mathbf{M}$ is expressed as $|\mathbf{M}|$ :

$\begin{aligned} \mathbf{M} = \begin{bmatrix} A & B \\ C & D \end{bmatrix} ,\quad |\mathbf{M}| = AD - BC \end{aligned}$

However, these notations share a common theme of representing size or magnitude:

In set theory, cardinality describes the size of a set by the number of elements it contains.
For numbers, the absolute value captures the distance from zero on the number line.
In linear algebra, the Matrix Determinate determinant describes how much a Matrix Transformation scales space.

Discrete Mathematics Course Notes

2023-03-06T00:00:00+10:00

These are my notes from Discrete Mathematics by the University of London.

I took it as part of the BSc Computer Science degree from Oct 2022 - March 2023.

The books required:

Intro

Discrete Maths
- The study of discrete objects.
- Discrete objects are different from "connected objects".
  - separated and distance from each other
  - examples: integer, positions, set, relationships, functions
- Increasingly applied to practical fields of math and science.
- Improves reasoning and problem-solving.
Course instructor is Dr Lahcen Ouarbya.

Set

2023-01-15T00:00:00+10:00

A set is a unique collection of well-defined objects.

Objects in a set are called elements or members of the set.

We often refer to sets in everyday language: you may have heard of a tea set, a drum set, a set of action figurines, etc. These terms refer to specific collections of objects. It is clear which objects in are members of each set and which are not.

Sets are commonly notated using curly braces. For example:

The set of numbers from 1 to 3: $A = \{1, 2, 3\}$
A set of words: $B = \{\text{hello}, \text{world}\}$

Membership

The elements of a set must be well-defined, and there should be no ambiguity about whether something is a set member or not.

We use the $\in$ notation to describe if something is a set member:

$1 \in A$
$\text{hello} \in B$

We use the $\notin$ notation to describe if something is a not set member:

$99 \notin A$
$\text{goodbye} \notin B$

The symbols in Latex are:

$\in$ : \in
$\notin$ : \notin

A set can be a member of another set. Interestingly, this fact means that the definition of a "set" is circular, making it technically an undefined term.

Uniqueness

Sets do not have duplicates; therefore:

$A = \{1, 1, 2\} = \{1, 2\}$

This property means sets are practically helpful for finding unique counts of things, and they often appear in programming for this purpose and many others.

Subsets

If every element in set $A$ is in set $B$ , we consider set $A$ to be a subset of $B$ . The notation for subset is $A \subseteq B$ . In LaTeX, we use: \subseteq

For example, the set of days on the weekend is a subset of the days in the week:

$\{\text{Sunday}, \text{Saturday}\} \subseteq \{\text{Sunday}, \text{Monday}, \text{Tuesday}, \text{Wednesday}, \text{Thursday}, \text{Friday}, \text{Saturday}\}$

Supersets

Conversely, we consider $B$ to contain $A$ , denoted as $B \supseteq A$ . We say that $B$ is a superset of $A$ .

If $A$ is not a subset of $B$ , we write $A \nsubseteq B$ . In LaTeX, it's: \nsubseteq

Cardinality

The number of elements in a set is called Cardinality

Equal Sets

When 2 sets contain the same elements, we consider them equal sets: $A == B$ .

Empty Set

When a set has no elements it's called the empty set: $\emptyset$ : $\emptyset = \{\}$ . The empty set is a subset of every other set.

Universal Set

A special set called the universal set $U$ , is a set where every other set is a subset of $U$ .

$A \subseteq U$ for every set $A$

Disjoint Sets

A set may not have common elements. These are called disjoint sets. For example, a set of integers and a set of letters are disjoint sets.

Power Set

The power set is a subset representing all subsets of set A, written as $P(A)$ .

For example, the power set of $A = \{1, 2\}$ :

$P(A) = \{\emptyset, \{1\}, \{2\}, \{1, 2\}\}$

Infinite and finite sets

If a set has an infinite number of elements, we call it an infinite set.

Some Special Infinite Set come up frequently in Math.

Cover by ahlianyq on Unsplash.

Disputing A Parking Fine with ChatGPT

2022-12-10T00:00:00+10:00

This article covers a use case for ChatGPT, an exciting new product by OpenAI.

This post made it to the front page of Hacker News. See the discussion here. Sydney Morning Herald also mentioned the story in an article on chatbots.

Recently, on holidays in Far North Queensland, my wife and I parked our rental car in a paid parking lot to visit a restaurant.

I paid using the EasyPark App per the council's instruction on various signs throughout the lot.

When we returned, they had slapped a fine on our Toyota anyway.

I double-checked what I had entered into the app. I mistyped the number plate by one letter. Oops.

Since we have never received a parking fine before, and I had proof of payment, I knew there was a good chance the council would let me off if I sent a letter of explanation.

We had already been experimenting with ChatGPT, and the letter seemed a good test case.

The first response was close but longer than I would like. Plus, I didn't tell it that I was planning to attach a photo of the fine and proof of payment.

That works. Now for a title.

I emailed that* from my phone and a day later received this response.

So you can dispute a parking fine with ChatGPT.

ChatGPT is incredible - I cannot believe how well this technology works. It's like having a talented personal assistant (who is often wrong - need to read their work thoroughly) at your fingertips.

* I used ChatGPT from my phone and didn't screenshot the prompt, so I had to generate a new prompt/response for this blog post. It's similar enough that I'm sure it would work too.

Idempotent Data Pipelines

2022-07-17T00:00:00+10:00

A key property of a fault-tolerant and easy-to-operate data pipeline is idempotence. ¹

When an operation is idempotent, it means if we perform it multiple times, the result will be the same as if we ran it once. In math, we express an idempotent function as: $f(f(x)) = f(x)$ .

Consider a simple data pipeline task that loads and transforms source records into batches, then stores the result in a relational database.

If a pipeline is not idempotent, running the job twice would generate 2x as many records as running it once. Therefore we must take care only to run our job once. If the pipeline fails halfway through, we must ensure that we resume it from where it broke. However, if it failed midway through a batch (especially one that partially succeeded in writing to the database), it may become difficult or near impossible to resume. Our only action is to clear the destination database and run the pipeline from the start or manually correct the destination database by hand.

On the other hand, an idempotent pipeline can be run infinite times and only generates one set of records. With this configuration, we can safely resume from somewhere before the failed batch, in or worst case, restart the entire pipeline from scratch. We could even have pipeline tasks automatically restart on failure.

To achieve idempotence, we must figure out how to uniquely identify our transformed records. Sometimes this is as simple as taking a primary key already provided in the source data. Other times we need to concatenate metadata together to identify a record uniquely. Sometimes the entire body of a source record must be hashed to identify it uniquely.

A side benefit of doing this is that you will have a richer understanding of your source data and transformations. It will force you to think about what makes each record unique and what you consider a duplicate.

Now our pipeline can use the unique identifier to check if the destination record exists, performing either an update, delete or perform a no-op, depending on what makes sense for our problem. Another advantage of doing it this way means if our transformations need adjustments, we can make them and rerun the pipeline when needed.

Performing a query to check for existing records will add performance overhead to the pipeline; however, the savings in operational complexity far outweigh the penalty in my experience. If you can just rerun parts of your system on errors, or when you find that your transforms need to be updated, you will spend far less time babysitting them.

It's not just pipelines that benefit from idempotence. In a blog post, Eric Lathrop describes a customer billing operation that he makes dramatically easier to operate after introducing idempotence.

It's much easier to build idempotence from the start than bolting it on later.

Idempotence is closely related to Declarative programming, a paradigm used amongst Infrastructure As Code practitioners. ↩
Inspired by this diagram from Data Pipelines with Apache Airflow. ↩

An Interface Should Respond Within A Tenth Of A Second

2022-07-03T00:00:00+10:00

"if your interface does not respond within a tenth of a second, the player will feel like something is wrong with the interface." - James Schell, The Art of Game Design: A Book of Lenses ¹

According to studies ², 0.1 seconds is roughly the response time limit for a user to feel like they are in control of an interface.

Up to 1 second is the cut-off for a user's flow to remain uninterrupted, though they will not feel in control.

Ten seconds is the hard limit for keeping a user's attention focused. After that, they will want to do other things while waiting for the interface to respond.

You can try it yourself. Each button here will change color when clicked and respond in the time shown in the button text.

0.05 secs 0.1 secs 0.5 secs 1 sec 5 secs 10 secs 15 secs

Which of them makes you feel in control of the button color? Which of them feels like the computer is in control? Which of them makes you want to rage quit?

There are many solutions to make an interface feel responsive, even when a delay is required to return results: animations, loading spinners, progression indicators, skeleton objects, etc.

Here's one idea to make the buttons respond immediately using a fun pulsing animation I found here.

0.05 secs 0.1 secs 0.5 secs 1 sec 5 secs 10 secs 15 secs

Notice how you still feel in control even at the longest wait time for a change.

Though the exact solution you choose will likely come from a designer (if you're lucky enough to work with one), a developer's responsibility is to understand which parts of an interface are likely to need these solutions.

Only we know which interactions can return results straight from the client, which need to request results from servers, which requests are produced quickly from a cache or will require expensive processing.

It is up to us to review designs and give feedback on these interface problem areas.

For the development of the Splash game, since the response time for any call to a server is difficult to estimate, we follow a simple rule:

Any button press that triggers a server invocation we must first acknowledge on the client.

Usually we do this via a loading state. In code terms, we're doing this:

interface.onClick = function()
   toggleLoadingState(true)
   local response = invokeServer()
   displayResponse(response)
   toggleLoadingState(false)
end

For some particularly long-running requests, we aim to do them in the background to allow the player to continue to enjoy the game while they wait.

A Rule For Ordering Methods

2022-06-27T00:00:00+10:00

A small thing that made our code more readable and consistent was to add a code style guide rule for ordering methods.

Order your methods so that the object starts with the constructor (if required), followed by the key public methods, then any private implementation details.

We based this rule on The Stepdown Rule from Clean Code in the chapter on functions ¹:

"We want the code to read like a top-down narrative. We want every function to be followed by those at the next level of abstraction so that we can read the program descending one level of abstraction at a time as we read down the list of functions."

Public methods first, preferably only a few, ensure the object's key functionality is immediately apparent. Most intuitively follow this anyway, but it is easy to overlook.

The exact order you prefer matters less than being consistent. The cost of searching through arbitrarily ordered code accumulates over time. Plus, it's one less inconsequential decision a developer has to make.

Below is an example of a typical object we'd write in Lua ordered with our style guide rule in mind.

Note that Lua uses Metatables modules to emulate classes using Object Prototypes like Javascript. The details are unimportant, included for completeness.

local UsefulThing = {}
-- Lua Metatable thing.
UsefulThing.__index = UsefulThing

-- Constructor first.
function UsefulThing.new()
    local self = {}
    -- Another Lua Metatable thing.
    setmetatable(UsefulThing, self)
    return self
end

-- The key public method(s) (often just one).
function UsefulThing:use()
    self:_prepare()
    self:_doBehaviour()
    self:_cleanUp()
end

-- All the implementation details.
function UsefulThing:_prepare()
end

function UsefulThing:_doBehaviour()
end

function UsefulThing:_cleanUp()
end

return UsefulThing

Like the Python convention, we prefix private methods with underscores to further distinguish.

Photo by Andrew Brandy on Unsplash

MLA (7th ed.) Martin, Robert C. Clean Code: A Handbook of Agile Software Craftsmanship. Upper Saddle River, NJ: Prentice Hall, 2009. (Pg. 37) ↩

Do Standups Story-By-Story not Person-By-Person

2022-06-21T00:00:00+10:00

Story-focused (or story-by-story) standups work like this: every morning, the host walks us through each in-progress story on the board from right to left (closest to release first), with assignees giving a brief update. We take turns hosting the meeting.

In contrast, a person-focused standup, also known as person-by-person or Three Questions ¹, has each team member answering three questions: "What did you work on today?", "What did you work on yesterday?" and "Are there any blockers?". The host selects who speaks next or creates a system for it.

I think story-focused daily standups are almost always better, having tried both at times throughout my career, for three key reasons.

Story-focused reminds us every day of what we're trying to achieve

The goal of a standup is to review the work in progress. Our story-focused status updates give us our best chance to adjust if it looks like our commitments are in danger. A daily review also means that if we change course mid-sprint, everyone knows what direction we're now heading in.

On the other hand, a person-focused standup might have you believe the goal is to review the details of each other's work days or even for each of us to justify our existence. The person-focused updates can sometimes take the whole allotted standup time, at worse, causing us to get no ongoing feedback on our sprint commitments until our next Retro.

Story-focused standups are more collaborative.

Product development is a team sport. We all have different skills, and collectively, we seek to build something more significant than one of us could alone.

Giving our updates in the context of our overall plan gives us a better chance of collaboration.

Maybe a designer learns how difficult a particular detail is to implement and can propose a simpler alternative. Perhaps another developer knows an expert in an area of struggle from another team who can help.

In comparison, in a person-by-person standup, lacking the context of each other's updates, we may find ourselves tuning out some of the ones where updates aren't clear to us how they fit into our overall goals.

Story-focused standups are fun and less stressful

Since we're focusing on what we're collecting and achieving, not interrogating the details of each person's work days, we can relax and be more engaged and present. The overall work we are trying to accomplish is much more interesting than the minutiae of each other's work days.

A meeting where everyone is engaged and present is much more fun.

The day's first meeting should not sap the team's energy.

In either case, we aim to finish standup in 15 minutes or less. The host politely tries to wrap up long-winded status updates and organizes follow-up discussions.

The downside to story-focused standups is that some individuals who are less comfortable contributing to the group may not get a chance to speak. Maybe they are working on something we aren't covering on our board, or perhaps we haven't assigned them anything, and they're not sure when to ask.

To combat this, we end standups by giving the floor to anyone who hasn't yet contributed. It allows us to write tickets for their work or to ensure tasks are assigned.

It's vital to remember why we do daily standups: we do them because the coordination effort is complex enough to require daily alignment. If it isn't sufficiently complex - i.e., if it's a small team - you may not need standup.

Cover by Paul Hanaoka on Unsplash

It's Not Just Standing Up: Patterns for Daily Standup Meetings ↩

Document Your Documentation

2022-06-18T00:00:00+10:00

One simple thing that can help a software team is to create a short document explaining how the team documents things.

Over time, as your product's complexity increases, so does the complexity of knowledge required to support it. It's common to find information scattered in many places.

My team has most documents in Notion, some in the README.md files, some as comments in code, some pinned to Slack channels, and some in Google Docs.

Documentation in multiple places isn't necessarily a problem - information requires various mediums.

However, having to make a decision every time we try to share information is a waste of valuable brain cycles and increases Decision Fatigue. I would prefer we utilize our brain energy for writing good documentation.

We should instead make a decision once for each class of information (see example below), documenting how and where we store and retrieve as clearly as possible. It doesn't matter if we don't cover all kinds of information initially or if the system isn't perfect; we can improve it over time.

You can think of it as a code-style guide for your information.

It's also helpful to include why you have made decisions; that allows you or the future team to evaluate if the original logic still holds up. Maybe there's a better way to communicate that information now - that's okay.

A well-understood knowledge system will also help us with document retrieval.

Here's an example:

Documentation Guide

Primary

Notion should be the first choice for most documentation.

If it's product-specific, please put it in the folder under Products > Product Name.

Review the existing categories in the sidebar for the most relevant ones for company-wide sharing. You may add a new one if none fit.

Notion ensures our documentation is searchable and easily accessible.

Some exceptions apply:

Repository setup information

A README.md file should describe how to set up the project, including running the application and unit tests. Add a page in Notion that links to each repository. See Project A for an example. Use Notion if you need to expand on documentation beyond simply setting up the project.

Using the README for repository setup is usually the shortest path to ensuring that the setup instructions remain up-to-date and allow for linking to files within the repro. It is also the first place that most newer developers will look for documentation.

Ephemeral Documents

Some documents, for example, the project to-do lists, meeting summaries, balancing information, etc., make sense to live in a Google Document.

Google Documents is generally better for these sorts of documents, especially involving outside collaborators.

If the document needs to become long-lived, add a link for it in Notion. Each project has a page called Key Documents to which we can link these.

The cover is by Jon Tyson on Unsplash

A Discriminative Feature Learning Approach for Deep Face Recognition

2022-05-18T00:00:00+10:00

These are my notes from the paper A Discriminative Feature Learning Approach for Deep Face Recognition by Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao.

Abstract

We commonly train image classification models using Categorical Cross-Entropy Loss. However, softmax loss does not learn sufficiently discriminative features for face recognition.

This paper proposes a new supervision signal called Center Loss. Center Loss simultaneously learns a center for each class and penalizes the distance between features and their class centers.

Center Loss requires training with joint supervision of softmax loss for stability.

The paper improves on the state-of-the-art for face recognition and face verification tasks.

1. Introduction

Modern image classification typically involves some backbone model to learn features from input images, which we can feed into a final fully-connected layer for classification.

At the time of the paper, CNNs were the best-performing model architecture for learning features.

Since image classification problems are typically Close-Set (all possible test classes well represented in the training set), Categorical Cross-Entropy Loss table loss function choice. In this case, the features learned by the backbone model only need to be separable enough so the classifier can distinguish between classes.

However, in facial recognition, you cannot pre-collect all possible test identities in the training set for face recognition. We call these problems Open-Set Classification. Therefore, for facial recognition, we need to learn discriminative features.

Discriminative features have two properties:

features from the same class should be close together, aka "inter-class dispensation."
features from different classes should be far apart, aka "intra-class compactness."

In Fig 1. we see a typical image classification pipeline, comparing separable and discriminative features.

Typically recognition pipelines use a Nearest Neighbours or K Nearest Neighbours step to classify faces based on the distance to other identities instead of label predictions.

Constructing a loss function for discriminative feature learning is challenging.

Since Stochastic Gradient Descent (SGD) relies on mini-batches, you cannot represent the global distribution in every step.

Alternatives proposed include Contrastive Loss (training using pairs) and Triplet Loss (training using triplets). However, they rely on Hard Negative Mining for efficiency, which adds complexity to the training pipeline.

This paper proposes Center Loss. They add a center for each class, a vector of the same dimension as the input feature embedding.

They simultaneously learn center during training while minimizing the distance between features and their corresponding class center.

The backbone requires trained using joint supervision of Categorical Cross-Entropy LossLoss, with a new hyperparameter to balance each component.

The center loss pulls deep features of the same class toward their centers, accomplishing the goal of inter-class compactness and intra-class dispersion.

Paper runs experiments on:

MegaFace Challenge
Labeled Faces in the Wild (LFW)
YouTube Faces (YTF)

Face recognition with Deep Learning
- Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings of the British Machine Vision, vol. 1, no. 3, p. 6 (2015)
- Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)
- Sun, Y., Chen, Y., Wang, X., Tang, X.: Deep learning face representation by joint identification-verification. In: Advances in Neural Information Processing Systems, pp. 1988–1996 (2014)
- Sun, Y., Wang, X., Tang, X.: Hybrid deep learning for face verification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1489–1496 (2013)
- Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)
- Wen, Y., Li, Z., Qiao, Y.: Latent factor guided convolutional neural networks for age-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4893–4901 (2016)
Mapping pair of face images to distance
- Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 1, pp. 539–546. IEEE (2005).
  - They train siamese networks to drive the similarity metric to be small for positive and large for negative pairs.
- 1. Hu, J., Lu, J., Tan, Y.P.: Discriminative deep metric learning for face verification in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1875–1882 (2014)
  2. Introduce a margin between positive and negative face image pairs.
Softmax modifications
- Sun, Y., Wang, X., Tang, X.: Deep learning face representation from predicting 10,000 classes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1891–1898 (2014)
- Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)
Joint Identification Verification Supervision Signal
- Sun, Y., Chen, Y., Wang, X., Tang, X.: Deep learning face representation by joint identification-verification. In: Advances in Neural Information Processing Systems, pp. 1988–1996 (2014)
- Wen, Y., Li, Z., Qiao, Y.: Latent factor guided convolutional neural networks for age-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4893–4901 (2016)
- Sun, Y., Wang, X., Tang, X.: Deeply learned face representations are sparse, selective, and robust. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2892–2900 (2015)
  - Add fully connected layer and loss functions to each conv layer.
Triplet Loss
- Liu, J., Deng, Y., Huang, C.: Targeting ultimate accuracy: Face recognition via deep embedding. arXiv preprint (2015). arXiv:1506.07310
- Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings of the British Machine Vision, vol. 1, no. 3, p. 6 (2015)
- Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)
  - In this paper, they minimise the distance between an anchor and a positive, while maximising the distance between an anchor and a negative until the margin is met.

3. The Proposed Approach

The authors use a toy example to intuitively show the distribution of deeply learned features. Inspired by this distribution, they propose center loss to improve the discriminative power of the learned features.

The toy example is trained on MNIST. They modify LeNet (a standard convolutional architecture) to include an extra conv layer (making up three conv layers in total) and additional conv filters in each layer. Then they reduce the output of the last hidden layer to 2, so they can visualize features in 2d space.

Fig 2. shows the results of plotting the 2-dimension hidden layer output on the training set (a) and the test set (b).

We can infer from this that the learned features are separable - a classifier can find a decision boundary between them - but not discriminative. In other words, it would be challenging to classify features using the nearest neighbors approach, as the distance between intra-class samples often matches those inter-class.

Center loss function is proposed to address this:

$L_{C} = \frac{1}{2} \sum\limits_{i=1}^{m} {||\mathbf{x}_i - \mathbf{c}_{y_i}||}^{2}_{2}$

Where:

$m$ refers to the size of the mini-batch
$c_{yi}$ refers to the $y_i$ th class center of the deep features.

Since we cannot take the entire training set into account to average features of every class in each iteration, we have to make two modifications to support mini-batches.

Centers are computed by averaging features of the corresponding class (some centers may not update in a mini-batch)
Control the learning rate of the center using param $\alpha$ to avoid mislabelled samples breaking everything.

Lastly, they train the model using joint supervision of softmax loss and center loss, using hyperparameters $\lambda$ . When $\lambda = 0$ , training uses conventional Softmax Loss.

Fig 3. shows how the hyperparameters $\lambda$ affect the feature distributions. The higher it is, the more discriminative the features.

Joint supervision is necessary: if you only trained using class centers, the centers would degrade to 0 since that creates the lowest possible center loss.

The method is superior to contrastive and triplet loss as it is efficient and easy to implement.

4. Experiments

They use a typically CNN archiecture for the experiments.

Filter sizes for conv and local conv layers are 3x3 with stride 1, followed by PReLU nonlinear units.

The number of feature maps is 128 for conv layers and 256 for local conv layers. Max-pooling grid is 2x2, and the stride is 2.

They concatenate the output of the 4th pooling layer and the 3rd local conv layer as input to the 1st fully connected layer. The output of the fully connected layer is 512.

In Fig 4, we see a diagram of this architecture.

4.1 Implementation Details

Use five landmarks (2 eyes, nose, and mouth corners) for similarity transformation.

When detection fails, discard the image in the training set, but use the provided landmarks if it is a testing image.

Faces are cropped to 112 x 96 RGB images.

Each pixel $[0, 255]$ is normalized by subtracting 127.5 and dividing by 128.

Training data uses web-collected training data, including:

CASIA-WebFace
CACD2000
Celebrity

They remove images with identities appearing in testing datasets, which goes to 0.7M images of 17,189 unique persons.

The authors horizontally flip images for augmentation.

They train three types of models for comparison:

Model A: Softmax loss
Model B: Softmax and contrastive loss
Model C: Softmax and center loss

They extract features for each image and the horizontally flipped one and concatenate them as representation.

They compute the score as Cosine Distance of 2 features after PCA.

They use Nearest neighbor and threshold comparison for Face Identification and Face Verification tasks.

4.2 Experiments on the Parameter $\lambda$ and $\alpha$

They conducted experiments to investigate the sensitivity of the params on Labeled Faces in the Wild dataset.

The results are shown in Fig 5, for 2 experiments:

Experiment (a)

Fix $α$ to 0.5.
Vary $\lambda$ from 0 to 0.1 to train different models.

Experiment (b)

Fix $\lambda = 0.003$
Vary $\alpha$ from 0.01 to 1.

From this they infer:

Softmax Loss alone is not a good choice. It leads to poor verification performance.
Properly choosing a value of $\lambda$ can improve verification accuracy.
Verification performance of model remains stable across a range of $\lambda$ and $\alpha$ .

4.3 Experiments on the LFW and YTF Datasets

Evaluate single model on Labeled Faces in the Wild and YouTube Faces.

Fig 6. has some examples.

In (a), they show pairs in LFW the green frame is for positive pairs, and the red frame is for negative ones.

In (b), they show examples from YTF, where the white bounding box is the face for testing.

They train model on only 0.7M outside data with no overlapping in LFW and YTF. Fix $\lambda = 0.003$ and $\alpha = 0.5$ for Model C.

LFW dataset contains 13,233 web-collected images from 5749 identities, with variations in pose, expression, and illuminations. They test on 6,000 face pairs and report the experiment results.

YTF dataset consists of 3,425 videos of 1,595 different people, with an average of 2.15 videos per person. The clip durations vary from 48 to 6,070 frames, with an average length of 181.3 frames.

Table 2 has the results on 5,000 video pairs.

They observe the following:

Softmax Loss and Center Loss beats baseline one (Model A) by a large margin.
Joint supervision can notably enhance the discriminative power of deeply learned features, demonstrating the effectiveness of center loss over Softmax.
It also improves over Softmax and Contrastive Loss.
Using less training data and simpler architectures, they outperform many state-of-the-art approaches.

4.4 Experiments on MegaFace Challenge Dataset

MegaFace datasets aim to evaluate the performance of face recognition algorithms with millions of distractors (people who are not in the testing set).

MegaFace dataset has two parts:

Gallery Set. 1 million images from 690K people. A subset of Flickr photos from Yahoo.
Probe Set. It consists of 2 datasets:
- Facescrub contains 100K photos of 530 unique people.
- FGNet. Face Aging Dataset, with 1002 images from 82 identities. Each identity has multiple face images at different ages (ranging from 0 to 69).

The challenge has several testing scenarios: Identification, Verification, and Pose Invariance with two protocols: large or small training set.

The Center Loss authors follow a small training set protocol, which is less than 0.5M images and 20K subjects. They reduced the size of the training image dataset to 0.49M but kept the number of identities unchanged at 17,189.

They discard any images overlapping with Facescrub dataset.

They train three models: Model A, B, and C for comparison.

They use the same settings as earlier the $\lambda$ is 0.003 and the $\alpha$ is 0.5 in Model C.

They test the algorithm on only one of the three galleries.

Face Identification

Face identification aims to match a given probe image to those with the same person in the gallery. This task computes the similarity between each given probe face image and the gallery, including at least one image with the same identity as the probe one. The gallery contains a different scale of distractors, from 10 to 1 million, leading to increasing testing challenges.

In Fig 8, they show the results of face identification experiments. They measure performance using Cumulative Match Characteristics (CMC) curves, which is the probability that a correct gallery image is in top-K.

Face Verification

For face verification, the algorithm should decide whether a given pair of images is the same person or not.

They create four billion negative pairs between the probe and gallery datasets.

They compute the True Accept Rate (TAR) and False Accept Rate (FAR) and plot the Receiver Operating Characteristic (ROC) curves of different methods in Fig. 9.

Some of the methods they compare against include methods that require hand-crafted features, including LBP (Local Binary Pattern) and shallow models like JointBayes.

From Fig. 8 and Fig. 9, we can see that these modes perform poorly: their accuracies drop as the number of distractors increases.

Model C performs the best out of A and B and outperforms the other published methods.

To be practical, face recognition models should achieve high performance with millions of distractors.

Table 3 reports the rank-1 identification rate with at least 1 million distractors.

Table 4 reports the True Accept Rate with a False Accept Rate of $10^{-6}$ .

They make these observations from the results:

Model C is much better than Model A and Model B for face identification and verification tasks, confirming Center Loss with Softmax works best.
Second, under the evaluation protocol of a small training set, the proposed Model C- achieves the best results in both face identification and verification tasks, outperforming 2nd place by 5.97% on face identification and 10.15% on face verification, respectively.
Model C does better than some models trained with a large training set (e.g., Beijing Facecall Co.).
The models from Google and NTechLAB achieve the best performance thanks to the large training set (500M vs. 0.49M).

5 Conclusions

The paper proposes a new loss function called Center Loss.

By combining Center Loss with Softmax Loss to jointly supervise the learning of CNNs, the authors show that they can enhance the discriminative power of features for face recognition problems.

The paper runs extensive experiments on large-scale face benchmarks to demonstrate its effectiveness.

ArcFace: Additive Angular Margin Loss for Deep Face Recognition

2022-05-01T00:00:00+10:00

These are my notes from the paper ArcFace: Additive Angular Margin Loss for Deep Face Recognition by Jiankang Deng, Jia Guo, Niannan Xue, Stefanos Zafeiriou.

Abstract

The key to face recognition is a loss function with strong discriminative power.

Centre loss penalises the distance between features and a set of learned class centres.

SphereFace lets the linear transformation matrix in the last fully connected layer represent class-center and penalise the angle between deep features and their corresponding weights.

A recently popular idea is incorporating "margins" to maximise face class separability.

This paper proposes Additive Angular Margin Loss (ArcFace), which creates "highly discriminative features" for face recognition.

ArcFace has a "clear geometric interpretation" due to its correspondence to "geodesic distance" (a curve representing the shortest path between 2 points) on the hypersphere.

This paper studies the results of ArcFace on ten face recognition benchmarks and shows that ArcFace continually outperforms other algorithms.

The authors released all their training code and metadata.

1. Intro

The method of choice right now for face recognition is to represent faces using an Embedding generated by a Convolutional Neural Network.

The embedding has a small distance to other examples in the same class (intra-class) and a large distance to different classes (inter-class).

A pose normalisation step (aligning faces) typically occurs before generating embeddings.

There are two main tracks of research for training these embeddings:

Train a multi-class classifier and use the embedding generated by the network as the representation.
Directly learn an embedding, for example, Triplet Loss.

Both have their drawbacks.

The linear transformation matrix $W$ size increases linearly with every identity n for softmax loss.

The learned features are separable enough for closed-set classification but not for open-set problems.

For triplet loss, there's a combinatorial explosion in the number of face triplets for large-scale datasets.

Semi-hard sample mining is quite a complex problem to engineer.

Others have proposed some variants of Softmax loss to enhance the discriminative power of Softmax loss.

Wen et al. l. pioneered centre loss. It captures the distance between each feature vector and class centre and requires joint penalisation of softmax loss for intra-class dispersion. However, it is challenging to update class centres when there are a lot of classes.

Instead, enforcing intra-class closes and inter-class separateness at every step should lead to better models.

That's the idea behind Sphereface, which introduced an angular margin. However, the loss function needs a series of approximations to be computed, resulting in unstable training. They fix this by proposing a hybrid loss function that includes softmax loss.

CosFace adds a cosine margin penalty to the target logits:

logits + some margin penalty

This method provides better performance and is easier to train.

ArcFace is just a slight alternative to CosFace. It gets its name from the use of arc-cosine.

In this paper, they propose Additive Angular Margin Loss (ArcFace) to improve the discriminative power of CosFace.

Since we know the dot product between the CNN feature and the last fully connected layer is equal to cosine distance after feature and weight normalisation.

They utilise the arc-cosine function to calculate the angle between the current feature and target weight.

Then, add an additive angular margin to the target angle and get the target logit back by a cosine function. They rescale all logits by a fixed feature norm, and subsequent steps are the same as Softmax loss.

Its:

Engaging - optimises the geodesic distance margin by exact correspondence between angle and arc in the normalised hypersphere.
Effective - achieves state-of-the-art performance on ten face rec datasets.
Easy - Only needs several lines of code to implement.
Efficient - negligible computational overhead.

2. Proposed Approach

2.1 ArcFace

Firstly, the function for Softmax Loss.

$L_1 = -\frac{1}{N} \sum\limits_{i=1}^{N} \log \frac{e^{W^{T}_{y_i} x_i + b_{yi}}}{\sum_{j=1}^{n} e^{W^{T}_{j} x_i + b_j}}$

In this expression, $x_i$ is a deep feature of the $i$ th sample, belonging to the $y_i$ class. The author sets the embedding feature dimension to 512.

$W_j$ denotes the jth column of the weight $W$ , and $b_j$ is the bias term. The batch size and class numbers are $N$ and $n$ .

Softmax loss does not explicitly optimise the feature embedding to enforce higher similarity for intra-class samples and diversity for inter-class samples.

We can set the bias $b_j = 0$ then change the logit $W^{T}_{j} x_i = ||W_j|| \ ||x_i|| \cos \theta_j$ . If we l2 norm $W_j$ and $x_i$ , re-scale it to $s$ . The normalisation step on features and weights makes the prediction only dependent on the angle between features and thus distributed on a hypersphere with a radius of $s$ .

$L_2 = -\frac{1}{N} \sum\limits_{i=1}^{N} \log \frac{e^{s \cos \theta_{y_i}}}{e^{s \cos \theta_{y_i}} + \sum_{j=1,j \ne y_i}^{n} e^{s \cos \theta_{j}}}$

As we distribute embedding features around each feature centre on the hypersphere, we add an additive angular margin penalty $m$ between $x_i$ and $W_{ji}$ to enhance intra-class compactness and inter-class discrepancy.

The method is named ArcFace since the additive angular margin penalty is equal to the geodesic distance margin penalty in the normalised hypersphere,

$L_3 = -\frac{1}{N} \sum\limits_{i=1}^{N} \log \frac{e^{s (\cos \theta_{y_i} + m ) }}{e^{s (\cos \theta_{y_i} + m) } + \sum_{j=1,j \ne y_i}^{n} e^{s \cos \theta_{j}}}$

The authors demonstrate the idea of ArcFace using a 2D feature embedding network—Softmax on the left and ArcFace on the right. The "geodesic distance gap" is evident here.

2.2 Comparison with SphereFace and CosFace

SphereFace, ArcFace and CosFace propose three different kinds of margin penalty:

multiplicative angular margin $m_1$
additive angular margin $m_2$
additive cosine margin $m_3$

All of them enforce inter-class diversity by penalising the target logit.

This plot shows the angle between the target class and another class. As you can see, the angle between the target and the correct feature is around 20° ArcFace and 100°.

In this plot, we see the angle between the feature and target centre at different stages of training.

Combining all margin penalties, we can implement SphereFace, ArcFace, and CosFace in a unified framework with $m_1$ , $m_2$ , and $m_3$ as the hyperparams. Unifying penalties like this should give us target logit curves with high performance.

$L_4 = -\frac{1}{N} \sum\limits_{i=1}^{N} \log \frac{e^{s (\cos ( m_1 \theta_{y_i} + m_2 ) - m_3 ) }}{e^{s (\cos (m_1 \theta_{y_i} + m_2) - m_3 ) } + \sum_{j=1,j \ne y_i}^{n} e^{s \cos \theta_{j}}}$

Geometric Difference

Though similar, ArcFace has a better geometric attribute as the angular margin corresponds directly to Geodesic distance.

ArcFace has a "constant linear, angular margin" throughout the interval. By contrast, SphereFace and CosFace only have a nonlinear angular margin.

Minor margin differences can have a "butterfly effect" on model training. For example, the original SphereFace employs an annealing optimisation strategy. They tried implementing a new version of SphereFace without the integer requirement on the margin. $m = 1.35$ can perform similarly to SphereFace without convergence difficulty.

2.3 Comparison with Other Losses

Other loss functions can be designed based on the angular representation of features and weight vectors.

We can design a loss to enforce intra-class compactness and inter-class discrepancy on the hypersphere. Here's the comparison with three other losses:

Intra-Loss

They designed them to improve the intra-class compactness by decreasing the angle/arc between a sample and the ground truth centre.

$L_5 = L2 + \frac{1}{\pi N} \sum\limits_{i=1}^{N} \theta_{yi}$

Inter-Loss

Targets enhance inter-class discrepancy by increasing the angle/arc between different centres.

$L_6 = L2 - \frac{1}{\pi N (n - 1)} \sum\limits_{i=1}^{N} \sum\limits_{j=1, j \ne y_i}^{n} \arccos(W^{T}_{y_i} W_{j})$

We can think of Inter-Loss as a particular Minimum Hyper-spherical Energy (MHE) case. With this loss, we regularise hidden and output layers by MHE. In the MHE paper, the authors propose a special case of loss function by combining the SphereLoss with MHE loss on the network's last layer.

Triplet-loss

It aims to increase the angle arc between triple samples. In Facenet, the Euclidean margin is applied to the normalised features. Here we employ the triplet-loss by the angular representation of our features as $\arccos(x_{i}^{\text{pos}} x_i) + m \leq \arccos (x_i^{\text{neg}} x_i)$

3. Experiments

3.1. Implementation Details

Datasets

Use CA-SIA, VGGFace2, MS1MV2, and DeepGlint-Face (including MS1M-DeepGlint and Asian-DeepGlint) as training data to conduct a fair comparison with other methods.

MS1MV2 is a refined semi-automatic version of the MS-Celeb-1M dataset. Authors use ethnicity-specific annotators for large-scale face annotations.

Use face verification datasets (LFW, CFP-FP, and AgeDB-30) to check improvements from different settings. Report performance on large-pos and large-age datasets (CPLFW and CALFW). Test ArcFace on large-scale image datasets (MegaFace, IJB-B, IJB-C, Trillion-Pairs, and video dataset (iQIYI-VID))

Experimental Settings

Generate normalized face crops (112 x 112) utilizing five facial points.
For embedding networks, use CNN architectures ResNet50 and ResNet100.
After the last conv layer, use the BN-Dropout-FC-BN structure to get the final 512-D embedding feature.
Paper used ([training_dataset, network structure, loss]) for understanding experimental settings.
Set feature scale to $s = 64$ and choose angular margin $m$ of ArcFace at 0.5.
Use batch size 512 and train on 4 NVIDIA Tesla P40 (24GB) GPUs.
On CASIA, the learning rate starts from 0.1 and is divided by ten at 20K and 28K iterations. Training finished at 32K iterations. On MS1MV2, divide the learning rate at 100K, 160K, and 180K iterations. We set the momentum to 0.9 and weight decay to 5e-4. During testing, keep the feature embedding without the FC layer, with extra 5120D features for each face.
Remove overlap identities between the training and test sets and use only one crop for testing.

3.2 Ablation Study on Losses

The best margin observed in experiments was 0.5.
Using the combined margin framework in Eq. 4, it is easier to set the margin of SphereFace and CosFace, found to have optimal performance at 1.35 and 0.35
Implementations for SphereFace and CoseFace can lead to strong performance without issue converging.
ArcFace achieves the highest verification accuracy on all 3 test sets.

Refer to the paper for comprehensive testing results.

4. Conclusions

Paper proposed an Additive Angular Margin Loss function, which can enhance the discriminative power of feature embeddings learned with CNNs for face recognition.
Paper demonstrates that the method consistently outperforms the state-of-the-art.

5. Appendix

5.1 Parallel Acceleration

ArcFace and other margin loss rely on Center ($W$), but the param size of Centre is proportional to the number of classes.
When there are millions of identities in training, ArcFace can cause GPU memory to run out.
They solve it with a strategy called "parallel acceleration":
1. Get feature $x$ $x$ (face embeddings are aggregated into one feature matrix (batch size 8 * 64 (as there are 8 GPUs) $\times$ $\times$ feat dim 512))
  - The size of the feature matrix is only 1MB, so the communication cost between GPUs is negligible.
2. Get similarity score matrix ($score = xW)$. They copy the feature matrix into each GPU and concurrently multiply the feature matrix by the centre sub-matrix (1M/8) to get the similarity score submatrix
3. Get the gradient on Centre ($dW$). Transpose the feature matrix on each GPU and multiply the transposed feature matrix by the gradient sub-matrix of the similarity score.
4. Get the gradient on the feature ($x$) by concurrently multiplying the gradient sub-matrix of similarity score by the transposed centre sub-matrix and sum up the outputs from 8 GPU cards to get the gradient on feature x.

5.2 Feature Space Analysis

512-d hypersphere space should be large enough for large-scale identities.

Eigenvalue

2022-01-22T00:00:00+10:00

A scalar value that describes how much a transformation scales an Eigenvector.

Eigenvector

2022-01-21T00:00:00+10:00

An Eigenvectors of a Matrix Transformation is any non-zero vector that remains on its Vector Span after being transformed.

That means that performing the transformation is equivalent to scaling the vector by some amount. The amount it scales the Eigenvector is called the Eigenvalue.

For example, if we transform the basis vectors with matrix $\begin{bmatrix}2 && 1 \\ 0 && 2\end{bmatrix}$ , we can see that $\hat{j}$ is knocked off its span, where $\hat{i}$ is simply scaled by 2.

One other particular case of vector that remains on its span is the zero vector: $\vec{v}=\begin{bmatrix}0\\0\end{bmatrix}$ , but that's not an Eigenvector.

When performing 3d rotations, the Eigenvectors are particularly useful as they describe the axis of rotation.

The notation for describing the relationship between the matrix transformation of the vector and same scaling quality equivalent:

$A\vec{v} = \lambda\vec{v}$

Essence of Linear Algebra

2022-01-14T00:00:00+10:00

Notes from 3Blue1Brown's Essence of Linear Algebra.

Essense of linear algebra
Linear combinations, span, and basis vectors
Linear Transformations and Matrices
Matrix multiplication as composition
Three-Dimensional Linear Transformations
The determinant
Inverse matrices, column space, and null space
Nonsquare matrices as transformations between dimensions
Dot Products and Duality
Cross Products
Chapter 11. Cross products in the light of linear transformations (missing)
Chapter 12. Cramer's rule, explained geometrically (missing)
Change of basis
Eigenvectors and Eigenvalues

Changing Basis

2022-01-04T00:00:00+10:00

The use of coordinates to define vectors implies an agreement about which Basis Vectors we use. In 2d space, we commonly use the standard basis vectors:

$\hat{i}=\begin{bmatrix}1 \\ 0\end{bmatrix}$ , $\hat{j}=\begin{bmatrix}0 \\ 1\end{bmatrix}$ .

However, we are technically not limited to just the standard basis vectors: we can use any set of vectors to describe a coordinate system. Perhaps we encounter an Alien whose coordinate system uses these basis vectors:

$\hat{e}_{1} = \begin{bmatrix}2 \\ 4\end{bmatrix}$ , $\hat{e}_{2} = \begin{bmatrix}1 \\ 1\end{bmatrix}$ .

If the Alien describes a vector, say $\begin{bmatrix}3 \\ 1\end{bmatrix}$ , in their coordinate system, we'd have first to translate it to our system.

We can translate back into our system by creating a matrix which uses the Alien basis vectors as the columns:

$\begin{bmatrix}2 && 1 \\ 4 && 1\end{bmatrix}\begin{bmatrix}3 \\ 1\end{bmatrix} = \begin{bmatrix}7 \\ 13\end{bmatrix}$

We can think of that as a Matrix Transformation that scales basis vector $\hat{e}_1$ by $3$ and $\hat{e}_2$ by $1$ .

We can convert a vector described in our coordinate system to the Alien using the Matrix Inverse of our Alien's basis vector matrix:

$\begin{bmatrix}2 && 1 \\ 4 && 1\end{bmatrix}^{-1}\begin{bmatrix}7 \\ 13\end{bmatrix} = \begin{bmatrix}3 \\ 1\end{bmatrix}$

If we wish to perform a transformation described in our Basis, for example, a rotational transformation on an alternate basis, we can follow these steps:

Convert the vector into our Basis by applying our friend's transformation matrix.
Perform the translation.
Convert the vector back into our friend's Basis by applying the inverse of the transformation.

In notation, if we have vector $\vec{v}$ described in our friend's Basis $A$ , we can apply Matrix Transformation $M$ , described in our Basis, as follows:

$A^{-1}MA \ \vec{v}$

Roblox Attachment

2021-12-12T00:00:00+10:00

An Attachment in Roblox is an object that describes a point and orientation in space relative BasePart's Roblox CFrame.

Consider a Fire that you want to burn from the back of a Part. If I attach a Fire instance directly to a Part, it will always burn from the Part's center of the Part. Instead, I can create an Attachment parented to the Part and position it -2 studs along the X-axis and one stud along the Y-axis from the Part's center.

The Attachment remains in the same place relative to the Part as the part moves.

Here's how it looks in code:

--!strict

-- Create Part.
local part: Part = Instance.new('Part')
part.Anchored = true

-- Create an Attachment behind the Part.
local attachment: Attachment = Instance.new('Attachment')
attachment.Position = Vector3.new(-2, 1, 0)
attachment.Parent = part

-- Attach fire to Attachment instead of Part.
local fire: Fire = Instance.new('Fire')
fire.Size = 1
fire.Heat = 1
fire.Parent = attachment

part.Parent = game.Workspace

-- Code to move Part around below.

A new attachment can be created using Instance.new('Attachment').

An attachment's position can be updated using the Position attribute or editing the CFrame attribute.

There are many items that you can parent directly to an Attachment instead of a Part:

Any ParticleEmitters can be parented directly to an Attachment.
Sound objects allow audio to play directly from the Attachment's location.
PointLight and SpotLight allow light to shine from a specific point on a Part.

Roblox Constraint objects rely on Attachments.

The Roblox Accessory system utilizes attachments to position accessories on a character's body parts.

Cross Product

2021-12-05T00:00:00+10:00

The Cross Product is an operation between two vectors that returns a vector: $\vec{a} \times \vec{b} = \vec{c}$

The returned vector will be orthogonal to both input vectors and have a length equal to the Parallelogram Area the input vectors define.

Though the Cross Product can be generalized to multiple dimensions by taking the product of $n - 1$ vectors, you usually calculate the Cross Product in 3d space.

The Cross Product operation works as follows:

Line up the vectors, as you would when taking the Dot Product.

$\begin{bmatrix}a_1 \\ a_2 \\ a_3\end{bmatrix} \times \begin{bmatrix}b_1 \\ b_2 \\ b_3\end{bmatrix} = \begin{bmatrix} ? \\ ? \\ ? \end{bmatrix}$
For the first component of the new vector, exclude the top rows of the input vectors. Then, calculate the 2d Matrix Determinate of the matrix created by the bottom two rows of each matrix.

$\det\left( \begin{bmatrix} \\ a_2 \\ a_3\end{bmatrix} \times \begin{bmatrix} \\ b_2 \\ b_3\end{bmatrix} \right) = \begin{bmatrix}\mathbf{a_2b_3 - a_3b_2} \\ \\ \end{bmatrix}$
For the 2nd component of the new vector, exclude the middle row of the input vectors. Then, we calculate the determinate; however, this time, you flip the order of the operations from $ad - bc$ to $bc - ad$ .

$\left( \begin{bmatrix} a_1 \\ \\ a_3\end{bmatrix} \times \begin{bmatrix} b_1 \\ \\ b_3\end{bmatrix} \right) = \begin{bmatrix}a_2b_3 - a_3b_2 \\ \mathbf{a_3b_1 - a_1b_3} \\ \ \end{bmatrix}$
Perform a 2d determinate operation, excluding the input vector's last rows for the final component.

$\det\left( \begin{bmatrix} a_1 \\ a_2 \\ \ \end{bmatrix} \times \begin{bmatrix} b_1 \\ b_2 \\ \ \end{bmatrix} \right) = \begin{bmatrix}a_2b_3 - a_3b_2 \\ a_3b_1 - a_1b_3 \\ \mathbf{a_1b_2 - a_2b_1} \end{bmatrix}$

For each pair of vectors, there will be two vectors that are perpendicular to both. To find out which direction the Cross Product's vector faces, use the right-hand rule. For $\vec{a} \times \vec{b}$ , adjust you right-hand so you can place your index finger in the direction of $\vec{a}$ and your middle finger in the direction of $\vec{b}$ . Whichever direction your thumb is pointing in is the direction of the cross product.

The cross product is helpful because it tells you if your vectors are parallel when the length of the vector returned by the Cross Product is 0.

KhanAcademyLabs (2009) 3Blue1Brown (2016)

References

3Blue1Brown. Cross products \textbar Chapter 10, Essence of linear algebra. September 2016. ↩

Khan Academy Labs. Cross product introduction. October 2009. ↩

Cross product introduction

2021-12-04T00:00:00+10:00

These are notes from Cross product introduction | Vectors and spaces | Linear Algebra | Khan Academy by Khan Academy.

The Cross Product is much more limited than Dot Product. Where the dot product is defined in any dimension ( $R_N$ ), the cross product is only defined in 3d ( $R_{3}$ ).

The dot product returns a scalar; the cross product a Vector.

Definition of the dot product:

$\vec{a} = \begin{bmatrix}a_1 \\ a_2 \\ a_3\end{bmatrix}, \vec{b} = \begin{bmatrix}b_1 \\ b_2 \\ b_3\end{bmatrix}$

$\vec{a} \times \vec{b} = \begin{bmatrix} a_2 \cdot b_3 - a_3 \cdot b_2 \\ a_3 \cdot b_1 - a_1 \cdot b_3 \\ a_1 \cdot b_2 - a_2 \cdot b_1 \end{bmatrix}$

For first row in the returned vector, you ignore the top row and take $a_2 \cdot b_3 - a_3 \cdot b_2$
For the 2nd row in the returned vector, you ignore the middle row of the vectors and take a similar product to the first; however, this time, you are doing it the opposite way around: $a_3 \cdot b_1 - a_1 \cdot b_3$
For the 3rd row, you ignore the last row of input and make the same operation as the first row of the top 2 rows of input: $a_1 \cdot b_2 - a_2 \cdot b_1$ .

The vector that's returned is orthogonal to both $\vec{a}$ and $\vec{b}$ .

Note that two vectors are orthogonal to those vectors. To find which direction it points in, you use the right-hand rule: take your right hand and put your index finger in the direction of $\vec{a}$ and your middle finger in the direction of $\vec{b}$ , where your thumb is pointing in the direction of the returned vector.

What does orthogonal mean in this context? It means if $\vec{a} \cdot \vec{b} = 0$ , the difference between orthogonal vectors and perpendicular vectors is orthogonal could also apply to 0 vectors.

You can prove it works by taking the dot product with one of the input vectors and the output vector:

$\begin{bmatrix} a_2 \cdot b_3 - a_3 \cdot b_2 \\ a_3 \cdot b_1 - a_1 \cdot b_3 \\ a_1 \cdot b_2 - a_2 \cdot b_1 \end{bmatrix} \cdot \begin{bmatrix}a_1 \\ a_2 \\ a_3\end{bmatrix}$

$= a_1 a_2 b_3 - a_1 a_3 b_2 + a_2 a_3 b_1 - a_2 a_1 b_3 + a_3 a_1 b_2 - a_3 a_2 b_1$

$= \mathbf{a_1 a_2 b_3} - a_1 a_3 b_2 + a_2 a_3 b_1 \mathbf{- a_2 a_1 b_3} + a_3 a_1 b_2 - a_3 a_2 b_1$

$= \mathbf{-a_1 a_3 b_2} + a_2 a_3 b_1 + \mathbf{a_3 a_1 b_2} - a_3 a_2 b_1$

$= \mathbf{a_2 a_3 b_1} \mathbf{- a_3 a_2 b_1}$

$= 0$

Maths/LinearAlgebra/CrossProduct

Matrix Determinate

2021-11-06T00:00:00+10:00

The determinate of a Matrix Transformation refers to how much it scales space.

If we think of the standard Basis Vectors as the sides of a square, we can think of them as having an area of $1 \times 1 = 1$ .

Then, if we transform them using matrix $\begin{bmatrix}2 && 0 \\ 0 && 4\end{bmatrix}$ , the new area is $2 \times 4 = 8$ . So we can say that the matrix has a determinant of 8.

$\det(\begin{bmatrix}2 && 0 \\ 0 && 4\end{bmatrix}) = 8$ .

Once we know how much a transformation scales a single square, that tells us how any area in space would be scaled, since linear transformations "keep gridlines parallel and evenly spaced." 3Blue1Brown (2016)

A determinate can be a fractional value, which reduces the size of space:

$\det\left(\begin{bmatrix}0.5 && 0.5 \\ 0.5 && 0.5\end{bmatrix}\right) = 0.5$

A determinate can even have a negative value, which means that the orientation of space is flipped.

$\det\left(\begin{bmatrix}-1 && 0 \\ 0 && -1\end{bmatrix}\right) = -1$

A Matrix Transformation was a determinate of 0, means that the transformation collapses space onto a single line. These types of matrices do not have a Matrix Inverse In 2d space, the Determinate can be calculated using this formula: $\det(\begin{bmatrix}{\color{red}{a}} && \color{green}{b} \\ \color{red}{c} && \color{green}{d}\end{bmatrix}) = {\color{red}{a}}{\color{green}{d}} - {\color{green}{b}}{\color{red}{c}}$ .

The intuition for this comes when you set $b = 0$ and $c = 0$ . In that case, the x and y-axis are scaled in a straight line. If you set either $b$ or $c$ to 0, the shape becomes a parallelogram. But the area is unchanged.

Dye et al. (2018)

In 3d space, it becomes a lot more complex. We take the product of each element of the first row with the matrix that can be created excluding the current element's column and row.

$det\left(\begin{bmatrix}\color{red}{a_{11}} && \color{green}{a_{12}} && \color{blue}{a_{13}} \\ \color{red}{a_{21}} && \color{green}{a_{22}} && \color{blue}{a_{23}} \\ \color{red}{a_{31}} && \color{green}{a_{32}} && \color{blue}{a_{33}}\end{bmatrix}\right) = {\color{red}{a_{11}}} \ \det\left(\begin{bmatrix}\color{green}{a_{22}} && \color{blue}{a_{23}} \\ \color{green}{a_{32}} && \color{blue}{a_{33}}\end{bmatrix}\right) - {\color{green}{a_{12}}} \ \det\left(\begin{bmatrix}\color{red}{a_{21}} && \color{blue}{a_{23}} \\ \color{red}{a_{31}} && \color{blue}{a_{33}}\end{bmatrix}\right) + {\color{blue}{a_{13}}} \ \det\left(\begin{bmatrix}\color{red}{a_{21}} && \color{green}{a_{22}} \\ \color{red}{a_{31}} && \color{green}{a_{32}}\end{bmatrix}\right)$

KhanAcademyLabs (2009)

References

David Dye, Sam Cooper, and Freddie Page. Mathematics for Machine Learning: Linear Algebra - Home. 2018. ↩

3Blue1Brown. The determinant. August 2016. ↩

Khan Academy Labs. 3 x 3 determinant. November 2009. ↩

Matrix Transformation

2021-11-01T00:00:00+10:00

We can think of a matrix as a transformation of a Vector or all vectors in space.

When we take the product of a matrix and a vector, we are transforming the vector.

A transformation is another word for a function: it takes in some inputs (a vector) and returns some output (a transformed vector).

For example, we can rotate a vector $\begin{bmatrix}x \\ y\end{bmatrix}$ some angle $\theta$ about the origin using a Rotational Matrix.

$\begin{bmatrix}\text{x*} \\ \text{y*} \end{bmatrix} = \begin{bmatrix}\cos\theta && \sin\theta \\ -\sin\theta && \cos\theta\end{bmatrix} \begin{bmatrix}x \\ y\end{bmatrix}$

In this example, we perform a 90° rotation of vector $\begin{bmatrix}2 \\ 2\end{bmatrix}$ .

To describe a transformation as a matrix, we only need to record where the Basis Vectors land as columns of a new matrix: $\begin{bmatrix}\color{red}{a} && \color{green}{b} \\ \color{red}{c} && \color{green}{d}\end{bmatrix}$

For example, a Shear Transformation keeps the $\hat{i}$ basis vector fixed, and slants the $\hat{j}$ basis vector. We can record that as: $\begin{bmatrix}\color{red}{1} && \color{green}{1} \\ \color{red}{0} && \color{green}{2}\end{bmatrix}$

A matrix transformation is always linear in that it keeps all gridlines in space are parallel and evenly spaced.

3Blue1Brown (2016)

Dye et al. (2018)

Image processing is a use case for matrix transformations.

Since we represent an image as a $m \ x \ n$ grid of pixels, we can treat the position of each pixel as a vector, then perform a transform of each position vector to transform the entire image.

In this example, I rotate an image using the rotational matrix above.

There's some additional code required:

Create a new matrix that's the maximum possible width and height.
Convert each position into a vector.
Convert each position vector, so it's a distance from the center, not top-left.
Rotate each position.
Revert convert using a new image size.

{% notebook permanent/notebooks/rotation-mnist.ipynb cells[1:2] %}

Note that we end up with some empty pixels in a 45° rotation. These occur because some of the transformed coordinates are floating-point numbers. When they get rounded into integer positions, some of the pixels get excluded. There are many strategies to deal with this, but that's for another article.

Agrawal (202)

References

Gautam Agrawal. Rotating image by any angle using only NumPy. September 202. ↩

David Dye, Sam Cooper, and Freddie Page. Mathematics for Machine Learning: Linear Algebra - Home. 2018. ↩

3Blue1Brown. Linear transformations and matrices. August 2016. ↩

Basis Vectors

2021-10-24T00:00:00+10:00

The set of Vectors that defines space is called the basis.

We refer to these vectors as basis vectors.

In 2d space, basis vectors are commonly defined as $\hat{i} = \begin{bmatrix}1 \\ 0\end{bmatrix}$ and $\hat{j} = \begin{bmatrix}0 \\ 1\end{bmatrix}$ .

These particular vectors are called the standard basis vectors. We can think of them as 1 in the direction of X and 1 in the direction of Y.

We can think of all other vectors in the space as Linear Combinations of the basis vectors.

For example, if I have vector $\begin{bmatrix}10 \\ -7\end{bmatrix}$ , we can treat each component as scalar (see Vector Scaling) for the basis vectors: $10\begin{bmatrix}1 \\ 0\end{bmatrix} + (-7)\begin{bmatrix}0 \\ 1\end{bmatrix}$ .

We can choose any set of vectors as the basis vectors for space, giving us entirely new coordinate systems. However, they must meet the following criteria:

They're linear independent. That means you cannot get one Vector by just scaling the other.
They span the space. That means, by taking a linear combination of the two scaled vectors, you can return any vector.

Basis vectors don't have to be orthogonal to each other, but transformations become more challenging with a non-orthogonal basis.

3Blue1Brown (2016) Dye et al. (2018)

References

David Dye, Sam Cooper, and Freddie Page. Mathematics for Machine Learning: Linear Algebra - Home. 2018. ↩

3Blue1Brown. Linear combinations, span, and basis vectors. August 2016. ↩