blank

Reinforcement Learning from Self-Feedback for Graphics Program Synthesis

2025-06-26T00:00:00+00:00

DeTikZify is a family of multimodal large language models that automatically convert hand-drawn sketches or images of existing scientific figures into editable, semantics-preserving TikZ graphics programs. It aims to simplify the process of creating and editing high-quality scientific figures, which traditionally is time-consuming and challenging. Here is an example workflow showing how to interact with the model:

Background

The video also showcases an iterative inference algorithm based on Monte Carlo Tree Search (MCTS), which enables DeTikZify to continuously refine its outputs without additional training. The reward scores required by MCTS are computed entirely using DeTikZify’s vision encoder, by visually assessing the similarity between input figures and compiled generated outputs. External reward models are not only unnecessary but often have lower correlations with human judgments, as the vision encoder was fine-tuned end-to-end with the entire model, optimizing it for evaluating this specific task. We refer readers to the DeTikZify and TikZero papers for further details. These self-computed rewards have been effective in enhancing model outputs during inference. With reinforcement learning algorithms like Group Relative Policy Optimization, this reward signal could also be used for the model to improve itself during a post-training step, i.e., reinforcement learning from self-feedback (RLSF). In this post, we investigate how post-training DeTikZify_v2, the previously state-of-the-art DeTikZify model, improves performance. We call the resulting model DeTikZify_v2.5.

Model Training

Our post-training setup does only require figures, and does not require aligned code as in supervised fine-tuning, granting us more flexibility in selecting training data. 50% of the training data comes from the subset of DaTikZ_v3, that was filtered out during the training of DeTikZify_v2. The remaining 50% is sampled from the SPIQA dataset, which contains image labels for figures extracted from arXiv. We exclude all figures from papers included in DaTikZ_v3. We sample this split so that 60% of these figures are labeled as schematics, 20% as plots, and 20% come from other categories. Since these figures are not necessarily created from TikZ, they may aid in enhancing the model’s generalization capabilities. As with DeTikZify_v2, input figures are randomly converted into synthetic sketches using image transformations and UltraSketch. Using this dataset, we post-train DeTikZify_v2 with RLSF, employing a batch size of 16. For each image, 32 outputs are generated, resulting in the model being trained on 512 outputs per step. We train for a total of 500 steps which takes 5 days to complete on eight Nvidia H200 GPUs. We keep the vision encoder frozen to mitigate reward hacking.

Experiments and Results

We evaluate DeTikZify_v2.5 (8b) on the test split of DaTikZ_v3 and compare it to DeTikZify_v2 (8b). The metrics employed include DreamSim (DSim), Kernel Inception Distance (KID), CrystalBLEU (cBLEU), TeX Edit Distance (TED), Mean Token Efficiency (MTE), and Mean Sampling Throughput (MST). Refer to the DeTikZify paper for further details. All scores except MST are multiplied by 100.

Sampling-based Inference

Results for sampling-based inference. Arrows indicate metric directionality. Overall, DeTikZify_v2.5 demonstrates the best performance.
	Reference Figures					Synthetic Sketches
Model	DSim_↑	KID_↓	cBLEU_↑	TED_↓	MTE_↑	DSim_↑	KID_↓	cBLEU_↑	TED_↓	MTE_↑
DeTikZify_v2 (8b)	80.503	0.626	6.105	54.946	93.326	74.584	0.751	3.356	58.32	93.858
DeTikZify_v2.5 (8b)	84.6438	0.298	4.202	52.939	100	78.257	0.577	1.551	56.121	100

In sampling-based inference (i.e., accepting the first output that compiles successfully) using reference figures and synthetic sketch inputs, DeTikZify_v2.5 (8b) outperforms DeTikZify_v2 (8b) on most metrics, demonstrating that RLSF can effectively enhance performance. The considerably increased DreamSim scores indicate that DeTikZify_v2.5 (8b) generates outputs that are much more visually similar to the reference figures. Furthermore, it is much less likely to produce outputs that do not compile, as evidenced by its perfect MTE score. Interestingly, while it scores lower on the code-based metric CrystalBLEU, it performs better on the code-based TED. DeTikZify_v2.5 (8b) tends to generate more concise programs with less syntactic noise. While this likely reduces the n-gram overlap with the reference code, it also decreases the number of edits necessary to convert one into another, explaining this phenomenon. Generally, more concise programs are beneficial as long as the semantics are preserved.

MCTS-based Inference

Results for MCTS-based inference. Arrows indicate metric directionality. Overall, DeTikZify_v2.5 demonstrates the best performance.
	Reference Figures					Synthetic Sketches
Model	DSim_↑	KID_↓	cBLEU_↑	TED_↓	MST_↑	DSim_↑	KID_↓	cBLEU_↑	TED_↓	MST_↑
DeTikZify_v2 (8b)	89.020	0.016	6.593	52.466	52.723	81.482	0.313	3.344	56.405	53.586
DeTikZify_v2.5 (8b)	90.889	-0.047	4.646	51.824	68.12	83.74	0.61	1.976	55.239	78.908

We observe similar trends when using our MCTS-based inference algorithm with a time budget of 10 minutes. Compared to sampling-based inference, DeTikZify_v2.5 (8b) noticeably improves its scores, illustrating that MCTS on top of RLSF can still lead to additional gains. Additionally, within the same timeframe, DeTikZify_v2.5 (8b) generates 25 more outputs than DeTikZify_v2 (8b), supporting our hypothesis that the generated programs are more concise. On reference figures, DeTikZify_v2.5 (8b) scores better on both DreamSim and KID, with the KID score even being slightly negative due to the high similarity of distributions. For synthetic sketches, it achieves a higher DreamSim score but performs worse on KID, indicating a prioritization of faithfulness to the reference figure over just focusing on general aesthetics.

Inference with TikZero Adapters

Results for sampling-based inference with TikZero adapters. Arrows indicate metric directionality.
	Captions
Model	DSim_↑	KID_↓	CLIP_↑	cBLEU_↑	TED_↓	MTE_↑
DeTikZify_v2 (8b)	52.829	5.103	10.051	1.603	65.51	82.291
DeTikZify_v2.5 (8b)	53.564	7.471	7.968	0.732	62.189	100

TikZero adapters integrate into the vision encoder of DeTikZify models, enabling them to be conditioned on text in addition to images. Since we keep the vision encoder frozen, we can evaluate DeTikZify_v2.5 (8b) on adapters trained for DeTikZify_v2 (8b). Compared to our previous experiments, the results are more varied. While DeTikZify_v2.5 (8b) achieves a better DreamSim value and maintains a perfect MTE, it performs worse on CLIPScore, suggesting difficulties in reproducing text from captions. This could be due to an increased modality gap, as RLSF further refines the model for image-only inputs. We plan to address this in future work by incorporating caption inputs into RLSF training.

Summary

Overall, RLSF greatly enhances model performance for most tasks. For image and sketch inputs, DeTikZify_v2.5 (8b) emerges as the clear leader. For text inputs via TikZero adapters, the choice between model versions depends on the specific use case, given the trade-offs involved. If you’d like to experiment with the model yourself, feel free to try it in our interactive playground.

Acknowledgments

This model was trained using computational resources provided by the bwForCluster Helix, as part of the bwHPC-S5 project. The authors acknowledge support from the state of Baden-Württemberg through the bwHPC initiative and the German Research Foundation (DFG) under grant INST 35/1597-1 FUGG. This post was inspired by the paper Rendering-Aware Reinforcement Learning for Vector Graphics Generation and is based on the DeTikZify_v2.5 (8b) model card.

Building a USB converter for a Vintage Siemens PC16-11 Keyboard

2024-12-17T00:00:00+00:00

I found this Siemens PC16-11 keyboard at a flea market. I had been searching for a Siemens keyboard for quite some time because I like their funky keycaps and overall aesthetics. Finally, I stumbled upon one. Here’s how it looked when I first brought it home:¹

It was a bit dirty but otherwise in good condition, with no missing keycaps! The first thing I did when I got home was disassemble the keyboard to take a look inside:

Everything looked good! Fortunately, there were no obvious signs of damage. The most interesting part of the circuitry is definitely the upper-left corner. Notable components include the Intel 8031 micro controller (1), a 2732 EPROM (2), and, most intriguingly, a beeper speaker (3):

Another highlight for me were the switches, the original Siemens STB 11 switches. Someone did a review on their (seemingly disappointing) successors, but I couldn’t find much information on these earlier models:

I really wanted to use this keyboard with a modern computer, but it uses a proprietary connector and protocol, so reverse-engineering seemed inevitable. I had previously done an internal conversion on a keyboard of similar vintage, but this time, I wanted to try an external conversion, leaving the original keyboard functionality intact. This approach meant that I would have to reverse-engineer the entire protocol, which would probably be a more extensive project in comparison. However, before making any assumptions I thought I’d start by looking for the original PC16-11 manual. Perhaps someone had scanned it, and it might contain useful information for the project.

To my surprise, I found that someone had indeed scanned the entire manual and uploaded it for archival purposes. A huge thank you to that kind soul! The 250-page manual (written in German) detailed the workings of the entire system, including how it communicates with the keyboard:

The manual revealed that the keyboard communicates with the computer via a unidirectional serial protocol on Pin 11. It also outlined how the protocol worked: the keyboard simply sends a keycode to the computer with each keypress, and that’s it:

Armed with this valuable information, building the hardware for the converter became much easier. Within hours of receiving the parts, I had a working prototype:

The only thing I changed from the schematic was adding a 2.2 kOhm resistor between pin 4 and ground, since we don’t need the -RxD signal. Finalizing the software took a bit longer, as I had to account for various edge cases when mapping the keycodes to actual keys. Fortunately, the excellent TMK Keyboard Firmware Core Library made this process much smoother. After working on it for about five evenings, I had a working converter. The source code is available here for anyone interested. The repository also includes a dump of the EPROM for archival purposes. After a thorough cleaning, here’s a typing demo with and without the buzzer:

I was really surprised by the switches. They’re tactile, which explains why the buzzer was necessary to reintroduce the noise level people were used to at the time. The tactile bump is gentle but noticeable, and the force required to bottom out increases sharply, perhaps reminding me a bit of Cherry MX Clears. I’ll definitely pull this keyboard out from time to time to type on it.

A downside of how the protocol works is that it only sends keypress events, not key release events. While this is fine for typing, it’s not at all suited for things like gaming (if you hold a key down for a while, it sends repeated keypress events). That said, it does have 2-key rollover, so it won’t satisfy your gaming needs in any case. Here’s a final photo of the cleaned-up keyboard, complete with the converter:

Overall, this was a fun and rewarding project. It also highlighted how valuable archiving things can be. I’m not sure how long the project would have taken without the manual. I wish modern computers still came with similarly detailed documentation. Even apart from the technical details, the manual was a fun read. I really enjoyed learning the German equivalents of terms I usually hear only in English; and just look at how happy this guy from the manual looks with his new PC16-11 keyboard! Sure looks like me.

Originally posted on Deskthority. ↩

Vintage ITT Courier 110169 Hall Effect Keyboard USB Conversion

2020-11-18T00:00:00+00:00

I did this conversion quite a while ago, but since this was a fun process, the keyboard is pretty interesting and because this could potentially be of interest to others, I still decided to write about it.¹

I found this keyboard on a local flea market, it was an instant must-have. When I got it it looked like this:

Maybe a little bit yellowed, but overall in good condition, nothing a good old retr0bright can’t fix. But before that I wanted to get that keyboard up and running, so the first thing I did was to disassemble the unit to get a glimpse at the PCB:

You can easily spot the Hall Effect sensors, which are arranged in a matrix of 13 columns and 8 rows. Other than that there isn’t much going on here, the big IC in the middle is an Intel 8048 compatible microcontroller. 4 of its pins are connected to a demultiplexer located to its right which is used to strobe the columns of the switch matrix. The demultiplexer expects the column index through a binary encoding from its 4 input pins. This means that the microcontroller can strobe all columns with only 4 of its pins being used! For sensing the 8 rows of the matrix are directly connected to the microcontroller.

A nice side-effect from this design is the fact, that this board supports inherent N-Key Rollover. This is nothing I really need, but it’s cool to see and for a Hall Effect keyboard I think such a feature is quite rare. Another useless but nevertheless really cool feature is the solenoid located on the lower right of the board.

In order to record my findings and to make working with this keyboard easier, I created a schematic of the circuit:

My first idea to get the keyboard working was to build a converter that can be plugged into the original cable. The original cable used a Micro D-Sub connector, something which I have never seen before, look how tiny it is compared to a regular D-Sub connector!

Since (at the time) I didn’t have any advanced measuring equipment, which I could have used to reverse-engineer the protocol, I decided to desolder the microcontroller and read its ROM directly to figure out how everything works. Thankfully the MCS-48 line of microcontrollers has extensive documentation, so building a circuit with an Arduino to read it was straightforward.

After that I used a disassembler to convert the binary into a readable format. Up to this point everything went pretty well but unfortunately understanding the firmware was much harder than expected. I figured out, that I could use one of the pins of the connector to get the microcontroller to jump to the interrupt routine were it looked like some kind of IO could happen but from there on the code was much harder to understand because one instruction used an indirect jump. After some time I gave up on that approach and decided to do the conversion internally.

I decided to utilize a Teensy 2.0, because it has so many IO ports, that I only had to reroute a handful of signals and could leave most of the traces untouched. I also repurposed the connector on the board for a custom USB cable. Software-wise I decided to use the TMK Keyboard Firmware Core Library, which made everything pretty easy, especially when compared to reading antiquated 1970s assembler code. In the end the board looked like this:

After that the only thing left to do was retr0brighting and putting everything back together. Here is a picture of the keyboard after everything was done:

Someone already did a fantastic job reviewing the unit so I’ll leave a link to the video here. I agree that the switches are incredibly smooth, but unfortunately a little bit too heavy for me for daily use. The solenoid was a bit surprising, though. It not only is loud as hell when activated, but it also makes the whole keyboard slightly vibrate which makes the switches feel almost clicky. Overall this was a very fun and instructive undertaking and if there’s someone else on this planet who owns this keyboard and is looking to convert it I hope this is a helpful resource. For the interested reader, I created a GitHub Repository for the code I wrote along with more detailed install instructions. Finally, here is a typing demo:

Originally posted on Deskthority. ↩