Project summary

For the safety community to budget time and efforts in making AI go well, we must clearly understand whether how models are improving, to predict further improvement. Unfortunately, our current benchmarks are frequently duplicated, thus performance is frequently confounded across factors. That is, it is unclear how much of the saturation of benchmarks we measure models against are a function of actual model generalization, or a measure of duplication in the data. This is becoming increasingly more important as governments have begun to consider regulation of AI, and AI safety orgs are deploying resources based on time schedules to deal with the most pressing x-risk issues in Artificial Intelligence.

To best deploy resources, we MUST understand the actual generalization properties of Artificial Intelligence, across many axis, thus we brace in more informed time for the shifting world under AI and its risks. Our project informs this.

We are interested in well determining the effect of training data on generalization versus memorization. Because benchmarks are saturated, it has not been made clear yet, how much models are capable of deriving results from information that isnt duplicated (in some form), in the enormous training corpuses. This is confounded by the clear effect that nearby, or semantically duplicated data is a strong contaminate of benchmark data itself, so decontamination is very difficult, despite current efforts by large labs.

What are this project's goals? How will you achieve them?

To help understand this problem we propose the following expirements.

1) Build two twin datasets, and inject them into training data at count N.

Dataset 1. Valid training data, related to the corpus text, via our methods of semantic duplication. The model can get this right over either memorization (extrapolation) or generalization.

Dataset 2. Identical in distribution items to dataset 1, so prompts are of the same benchmarkin training. However in this case solutions are selected at random (from the space of reasonable solutions to benchmark problems, i.,e. random multiple choice solutions). Since solutions are random, the only way for the model to recreate these solutions is via memorization.

By training on BOTH datasets at the same time, at rate given by count N per epoch, we observe that generalization is the difference in evaluation of Dataset 1 versus Dataset 2. Evaluation on Dataset 2 in this case is the model reproducing the random answer, the memorization artifact. If the model only memorizes then your delta will be approximately 0, that is the model will be right as frequently on dataset 1, as it is wrong (via memorization on Dataset 2). However if the delta is larger then 0, then we have shown a metric of generalization, since pure memorization always gives one 0. Behavior of model during experiment will be monitored by other nearby benchmarks. The behaviors of the model, and how these differences occur as training proceeds presents a method of understanding model generalization, and measures model capacity in a way that we believe is novel. A full study of these behaviors will occur, and be published, including likelihood, entropy, and complexity generalization.

2) Measuring benchmark duplication by inspection of computational graphs.

We propose to study benchmark alignment over computational graphs via methods like NTK (neural tangent kernels), and activation features to determine whether model computation can illicit a metric of semantic duplication, that is deeper then current methods. We hope to develop a functional continuous metric over model weights or computational graphs that decreases via semantic or notional similarity between benchmarks and a training point. Using such a definition, models who are trained on duplicates would show very little distance to contaminated benchmark points. If such a method is successful, we would 1) publish the work, 2) provide the method as an open source module for users to label and determine the contamination of their own benchmarks, over a model, against our large set of training data, held at Arb.

How will this funding be used?

How the money will be spent

Ideally 3 months of work- Ari at 70$ per hour-33600

Vasudha 4 days per month at 100$ per hour - 9600

Compute and storage - 6800

Who is on your team? What's your track record on similar projects?

Ari Spiesberger - Arb Research Ai researcher, ML engineer, Scientist. Will direct project. Running research, directing effort and output, responsible for content.

Juan Velasquez - Arb Research AI researcher and Scientist. Co-author of Sem Dupes paper that motivates this work.

Vasudha Kowtha - Ex-Apple Audio Researcher, fellow at Berkman Klien center at Harvard. Will provide valuable input and ideas, help derive scientifically strong experiments, provide outside perspectives.

Generally, folks at Arb may step in and help.

What are the most likely causes and outcomes if this project fails?

Separating generalization versus memorization may be separable to some extent, but there are many vectors of knowledge, and many potential confounds. Getting clean sensible results from these experiments will be difficult, but is why we have money allocated to thinking about it.

How much money have you raised in the last 12 months, and from where?

None for this project, Previous research was funded by Arb.