I further improved my line scan camera image processing!

Earlier, I wrote a blog post about various image processing methods in my previous blog post.

Line scan camera image processing

However, as you may recall from that post, some things are still not satisfactory.

• Vertical stripes removal was bad
• Speed estimation still isn’t perfect

Thanks to modern advancements in vibe coding, I was able to implement some of the ideas I wanted to implement in that blog post, to significantly perfect those things. (Spoiler alert: still not perfect, but a lot better)

1 Speed estimation by human

At the end of the day, doing subpixel correlation between the two channels of the Bayer array is stil super sussy. As mentioned in the previous blog post:

Hmm, I think my speed estimation still isn’t perfect. It could be off by about 10%. For future work, I think I might be able to extract feature correspondences such as SIFT or LightGlue. Trains are full of repeating elements that are supposed to be evenly spaced. I can detect those, and add a cost function to evenly space them, and optimize. Another idea is to use a circle Hough transform to find circles.

Instead of the fancy feature-based methods or Hough transform strategy, I implemented “meatbag dewarping” by using Codex to vibe code a Python GUI app for me, a human, to draw constraining features:

• Ellipses, that ought to be circles
• Horizontal segments that ought to be the same width
• Cut positions to cut up the super long image into chunks (e.g. between train cars)

So I fired up Codex with the following prompt:

In the python directory, let’s implement a new GUI tool to help fine tune speed estimation. Right now, as you may know from reading preview.py, speed estimation is done by subpixel correlation of the two green channels of the Bayer array line scan camera. In most cases, it works well, but in some featureless scenes, noisy scenes, scenes where the sample rate is much higher than the motion, or sometimes frankly I have no idea why, it is off by a little bit. The idea is that the GUI tool will allow me, a human, to carefully label two kinds of constraints: 1. draw ellipses that ought to be true circles, e.g. train wheels, London Underground roundel, etc. 2. draw horizontal segments that ought to be the same length after speed correction. For example, if the subject is a train, I can draw horizontal segments to constrain the width of each train window to be equal, or the width of each train car to be equal. This GUI tool will show a couple of raw, completely uncorrected frames, in a scrollable sliding window. e.g. if it shows 8 frames, then it would start out with 0000.bin to 0007.bin, and then we can increment or decrement the sliding window start/end by 1 by pressing left or right. Then, we can select which of the two tools to use, and draw directly on the raw frames. In a side panel, it shows a list of circle features added thus far, and a list of “horizontal width groups”. The horizontal line segments in each horizontal width segment group shall indicate that in real life they are all the same width. For ellipse drawing, just clicking and dragging the top left and bottom right of the bounding rectangle of the ellipse will work. The user may select on a previously drawn ellipse and resize it as needed. The ellipse shall be thin 1 px in a bright #00ff00 green color for easy visibility, but the draggable handles may be semitransparent and bigger (e.g. 8 px) for ergonomic clicking. For equal-width constraints, clicking and dragging shall highlight a full-height slice, with a thin 1 px vertical line on either side. The user may select, out of the list of equal width constraint groups, to add a new constraint to an existing group, modify an existing constraint, or to add a new group. Next, the user shall also indicate the start, end, and “cutting positions” of the sequence. This will populate a third list of things, which is again editable. Each “cutting position” is simply an x position; the first one indicates the start of the region of interest, the last one the end, and the middle ones are cutting positions. For example, for a train, we may want to add the cutting regions at the couplers in between the train cars. The UI for cut positions will simply show a thin 1 px vertical line that the user may place or modify. Finally, the output of this tool saves to a JSON file in the same directory as the raw data. Don’t worry about actually using the new constraint data in preview.py. All we want to do is to output the JSON. Also, when running the script, if the JSON already exists, it will load it from disk and populate the state of the tool already, so that the user may easily add or modify existing speed correction jsons. To display the raw chunks, we can simply bin the red, two green channels, and blue channel into a half size (e.g. 2048 x 1 rgb, from the original 4096 x 2 bayer) image for fast performance. Implement the GUI tool in a new python script in the python directory and add whatever dependencies you need to pyproject.toml. Make your own judgment about using the most modern, cleanest, minimal dependencies. Use a dark mode.

Anyway, it basically one-shotted the task, although I had to iterate a couple of times to iron out things I didn’t specify clearly, and add new features like the navigator on the bottom.

2 Vertical stripes removal

Vertical stripes continued to plague many of my line scan captures. So I kicked off Codex with the following prompt:

Take a look at the preview.py script. The problem we are trying to solve now is the vertical stripes showing up in line scan camera captures. The line scan camera captures one column at a time and has a 2 x 4096 px sensor comprising a Bayer array (one column is red-green and the other is green-blue), and it seems that the columns have different brightnesses, causing the stripe artifacts.

Here’s the description of the problem and the original approach to solve it:

Currently, the “jitter correction” in column_exposure_jitter_correction is still unsatisfactory. For a brief explanation of the algorithm, see the “Getting rid of the vertical stripes” section of /home/dllu/proj/daniel.lawrence.lu/blog/2025-09-21-line-scan-camera-image-processing/index.md. Of course, that is just a documenting blog post for your reference, but you should fix the code in preview.py in this project without touching that blog post.

Here’s what’s wrong with the original approach:

1. Smooth gradients may naturally occur in trains. This causes incremental steps that keep stacking and result in a big incorrect correction.
2. The current algorithm is neither aware of overexposed areas and underexposed areas.
3. The current algorithm is sensitive to tuning, such as the sigma, which varies greatly by the overall exposure of the photos.

Here are some weird camera quirks I know of:

• The brightness offset causing the stripes seems to be nonlinear and row position-dependent. The hue of the stripes also changes, suggesting that it may vary between the channels.
• The stripes are much more visible in the darker regions.
• The two green channels of the Bayer array are different from each other in a weird, nonlinear, and row position-dependent way. As such, it’s possible that one of them is zeroed out (clipping at zero) but the other isn’t. This particularly exacerbates the jitter correction’s problems and the correction may introduce strong striping that wasn’t previously there due to applying an offset to the zeroed out regions.
• There’s a brightness offset associated with the overall brightness of the whole line. In other words, if a column contains parts which are naturally very dark, it is likely to be overexposed, and conversely, if a column contains parts which are naturally very bright, the exposure for the whole column is decreased.
• The brightness offset effect is particularly strong if parts of the column are clipped. For an example, take a look at the capture /mnt/dataz/pictures/linescan/2026-01-13-09-49-37. The roof of the train is catching the sunlight at such an angle that it has many clipped highlights, which are causing the rest of the image to have strong vertical stripes.
• Due to the poor quality of the current algorithm, jitter correction may introduce new vertical stripes that mess with downstream algorithms such as speed estimation (which does a highly sensitive subpixel correlation), so I’ve turned it off by default.

Here are some ideas that we CANNOT do:

• We cannot assume that the top or bottom of the image are background. Hence, we cannot simply perform a calibration based on the top row and bottom row. This is because sometimes, the captured subject does in fact extend to the top and bottom edges. Also, even if it doesn’t, it can have subtle influences on the top and bottom of the image, such as casting a shadow. Furthermore, the photometric calibration we do has a nonlinear dependency on both the row position and the brightness, so typically calibrating based on a single row is not sufficient. Finally, as many of the captures involve trains, train bodies themselves often contain flat regions that are important to eliminate vertical stripes on.
• We cannot blindly assume that high frequency stripes are all noise. High frequency, high contrast vertical stripes may naturally occur in trains, for example, in ventillation grills.

Here are some ideas that we might try:

• Feel free to try out any lightweight machine learning models, for example, to run segmentation models or whatever. This machine is equipped with a powerful NVIDIA GB10 which has 128 GB of unified memory shared between the GPU and the host. You may add dependencies such as pytorch to the pyproject.toml, if it is helpful. We have $HF_TOKEN in the environment in case you need to download models from HuggingFace.

Here are some suggestions for captures that we can iterate on:

• /mnt/dataz/pictures/linescan/18-10-06-13-52-16-utc contains a Spanish AVE Class 102 “Pato” which has both smooth gradients and overexposed areas.
• /mnt/dataz/pictures/linescan/18-10-18-02-01-33-utc contains a Shanghai maglev which is very underexposed and noisy. The speed estimation has failed badly due to the noisy data even though the train is in fact moving at a nearly constant speed during the capture.
• /mnt/dataz/pictures/linescan/2026-01-13-09-49-37 contains a Hong Kong MTR which has overexposed elements on the roof.
• BONUS: /mnt/dataz/pictures/linescan/2026-01-13-09-32-44 contains a rotating line panorama. As such, it has no fixed (horizontally striped) background. Yet, vertical stripe artifacts are still visible, e.g. when inspecting the bottom right of the image, we see subtle stripes caused by large changes in brightness in the scene near the top caused by high contrast edges between tall buildings and the sky.

(note that the ones starting with 18 should have FLIP_OUTPUT_180 = False and the ones starting with 2026-01 and later should have FLIP_OUTPUT_180 = True)

While implementing:

• You can treat the various capture directories as scratch spaces to dump debug artifacts or visualizations as needed. However, take care to avoid modifying or overwriting the raw data (*.bin and capture_settings.txt, if it exists) and best not to overwrite existing artifacts.
• Feel free to refactor the code or add arguments to make your experiments more ergonomic instead of the original approach which is to manually modify the list of hardcoded globs and the hardcoded settings.
• Instead of iterating on the whole capture, which may be more than 100,000 px wide and very slow to process, you should hone in on a couple of known bad chunks for rapid iteration. In the same vein, avoid loading the full capture data into memory which would be likely to cause OOM. Best to cherry pick some known chunks and use them as a quick test harness. As vertical stripe correction happens prior to speed estimation and skew correction, it is fruitful for your test harness to iterate rapidly on known chunks without running speed estimation, skew correction, or high quality sampling. Bear in mind that the final success criteria still requires these to be enabled, however.
• Use your vision capabilities as much as possible to inspect the images for any remaining artifacts or obvious issues. Deploy subagents if necessary to quickly find relevant parts of large captures.
• We are now on the branch jitter-correction. Create a new commit whenever you’ve made meaningful progress towards eliminating the vertical stripes.
• Don’t modify this document.
• Jot down your experiments, ideas, thoughts, and results by creating new Markdown documents in python/experimental-docs. Create the docs in chronological order, e.g. 2026-07-26-blah-blah-blah.md, and ensure that each document is coherent by appending new results in chronological order within it.
• Read the earlier experiments and ideas (if any) in python/experimental-docs and inspect the current state of the code by looking at the commits ahead of master. The current state may have changed (hopefully improved) from the description given in this document.

This took Codex several iterations to implement correctly.

First, after an hour, it triumphantly concluded that it was perfected, but in fact it had caused a serious regression. Then, I told it that it had caused a regression, and it spent another six hours, and declared victory after erasing all the detail. Then it implemented a weird multi pass approach to erase all the detail and then rescue detail later.

Whatever, at least it’s better than before.

3 DNG export

It’s nice to process photos in Darktable using stuff like:

• Global tone mapping with AgX
• color calibration
• sharpening, defringing, and denoising
• tone curves

The line scan camera outputs 14 bit raws so it’s not bad. To that end, I vibe coded the ability to output reasonable-looking DNGs.

4 The overall correction pipeline

As you know, the pipeline is intentionally designed in these phases:

• vertical stripes removal
• speed estimation
• skew correction

To understand this, we need to clarify:

• Raw data before speed estimation: horizontal axis is time
• Resampled data after speed estimation: horizontal axis is space

The vertical stripes happen in the raw image data as each column just physically has different brightness in the time domain.

Speed estimation estimates a unitless ratio (pixel/pixel) that maps the time domain pixels to the space domain pixels. Vertical stripes really mess with the speed estimation, which uses subpixel correlation between columns to estimate speed, so it’s really sensitive to subtle differences in brightness.

Finally, skew correction needs to happen after speed estimation and vertical stripes correction, because first of all the skew is due to a physical tilt in the camera, so it happens in the space domain, and secondly, the stripes also introduce false edges that mess with the Hough transform.

4.1 AI shenanigans

While giving Codex several sample pictures, I expected that correcting the vertical stripes better would impact the speed estimation. In particular, if it weren’t done correctly, speed estimation by subpixel correlation would be borked.

So I told it to try to make sure that the speed estimation works.

But instead, it did a few stupid booboos:

• it added an artificial cap to the max and min speeds. This happened to work for the test cases I mentioned but broke other captures.
• it disabled vertical stripes removal in speed correction and skew correction. This eliminated “regressions” compared to not having vertical stripes correction in the first place, but failed to bring the improvements of the vertical stripes correction.

Also, gpt-5.6-sol max is generally basically blind. Relying on its vision capabilities is extremely flaky. It cannot notice obvious flaws in the image.