Project summary

Building a new type of language model based on discrete rules.

What are this project's goals? How will you achieve them?

1: Improve language modeling interpretability and safety:

It can be a significant challenge to understand what exactly is going in a modern Transformer after token embedding. The embedding travels along the residual stream for many layers, and with each layer the stream is transformed based on the all previous token's previous stream at that layer. The transformations and weights are often 16 bit values or more.

2: Improve efficiency:

Character level tokenization is already replaced by BPE, a discrete lookup. Also, arithmetic and traversing a knowledge graph are discrete, and current LLMs often do not find generalizing solutions.

Goals are to be accomplished by thorough study and analysis of discrete language models.

How will this funding be used?

Funding will be used to support myself and for compute.

Who is on your team? What's your track record on similar projects?

Just me. I did research at university, applying machine learning to improve classification of electrical signal outputs from medical sensors. I was initial member at a data observability startup, researching core ML algorithms behind the platform (went on to be YC funded).

What are the most likely causes and outcomes if this project fails?

It may be that for some reason, the current models that are not easily interpretable have a significant advantage over any interpretable alternative.

How much money have you raised in the last 12 months, and from where?

None.