2025-01-13

On Prompt Engineering

I have a visceral reaction to people talking about "prompt engineering". This is an attempt to reflect on why this phrase elicits such a strong negative emotional response in me. There are a lot of things to criticize about the current AI hype and its impact on the web, the environment, the economy and society at large. That’s not my focus here. I want to instead evaluate what I understand good (software) engineering to mean and contrast that to the activity of writing prompts.

What are some properties you want a software system to exhibit? I’d argue the list should include at least:

  • Predictable. My mental model should be aligned with the software so that I can with reasonable certainty predict the outcome of calling any one function in the system. This is very much not the case for LLM prompts. Replacing a word with a synonym, fixing a spelling mistake or changing the order of statements can all have dramatic and unpredictable effects on the result.
  • Deterministic / Repeatable. The software should be deterministic. Providing the same inputs to the system should reliably produce the same outputs. LLMs have built-in randomness euphemistically referred to as “temperature”. Execute the same prompt multiple times, get a different result each time.
  • Inspectable / Debuggable. I should be able to pop open the hood, look at the code, and figure out what it is doing. If I experience bugs or unexpected outputs I should be able to trace back through the code to understand where and why the logic went wrong or differed from my expectations. None of that is true for LLMs. If I get an unexpected result all I can do is permute my prompt and hope for the best. I can’t inspect the inner workings of the network to figure out which part of the input was responsible for the deviation.
  • Composable. Software systems should be building blocks I can combine to build something bigger than the parts. LLMs are all or nothing. Their interface is effectively all of human language as input and output. Multi modal models go even further to include other modalities like images, video and audio. This is too wide and deep an interface. By its very design, an LLM is not only a monolith but also perfect spaghetti. Everything potentially affects everything else. There’s no isolation in the network, no re-use of just parts of it is possible.
  • Stable. Software should be stable across versions. LLMs change in random unpredictable ways from one release to the next. A prompt that provides good results now may become entirely useless with the next generation of the model. Numeric embeddings produced with one version are meaningless and incomparable to ones produced with a different version. Great fun to start over each time.
  • Testable. Software should be (unit-)testable with fairly high coverage. LLMs offer a single entry point for a huge set of functionality. The input and output space is enormous. It is utterly hopeless to achieve anything approaching good test coverage. All you can do is shine a flashlight in a vast dark ocean and hope that your tiny collection of training examples covers all cases relevant to your problem.
  • Efficient. Software should be resource efficient. The principle of least power applies. LLMs are massive resource hogs and you activate large fractions of them no matter how simple or complex your prompt is.
  • Fast. Most software counts QPS - queries per second and latency in milliseconds. LLMs count QPM - queries per minute and latency in seconds. Maybe fast enough for an interactive chat bot. Painfully slow for working with large data sets. Especially if conjuring up prompts is so random that responsible development involves a ton of repeated experimentation and testing on large validation sets.
  • Precise. Software interfaces should be precise. Human language is not that. It is inherently ambiguous, redundant and open to interpretation. There’s no way to ensure the LLM will choose any one particular interpretation of a statement.
  • Secure. Software should be robust against injection attacks, leaking data and other safety concerns. Given the properties listed so far, it seems highly doubtful that this can ever be guaranteed for prompting LLMs. How can we pretend to secure a system we don’t understand and can’t test comprehensively?
  • Useful. This is arguably the one point where LLMs make up for not ticking even a single one of the other boxes. They do solve problems where we don’t currently have any alternative approach for a solution.

Prompt engineering is hard for all the wrong reasons. Don’t take my word for it. Take a recent (December 2024) paper The Prompt Report: A Systematic Survey of Prompting Techniques which surveys more than 1500 other papers to compile a taxonomy of prompting techniques. Everyone is just blindly pushing and prodding the machinery, hoping to tickle it in just the right ways to provide useful results. Consider choice quotes like “exemplar order can cause accuracy to vary from sub-50% to 90%+” or “providing models with exemplars with incorrect labels may not negatively diminish performance. However, under certain settings, there is significant performance impact” or the summary “prompt engineering is fundamentally different from other ways of getting a computer to behave the way you want it to: these systems are being cajoled, not programmed, and, in addition to being quite sensitive to the specific LLM being used, they can be incredibly sensitive to specific details in prompts without there being any obvious reason those details should matter”. This is utterly ridiculous and shows our fundamental lack of understanding of cause and effect with these things.

Prompt engineering is performing alchemy in a chemistry lab. It is an insult to the chemists.

So yeah. Devising good prompts is hard and occasionally even useful. But the current practice consists primarily of cargo culting and blind experimentation. It is lacking the rigor and explicit trade-offs made in other engineering disciplines.

Gemini 1.5 Flash: "Draw a brightly colored cheerful cartoon robot with an idiotic grin on its face and bulging bloodshot eyes. It should swing a hammer and wrench in dangerous and clumsy ways."

2024-11-16

Brüschbüchel (1817m), Chruter (1881m)

Afroz suggested to go hiking on the weekend. He had a nice trail in Valais in mind. I was up for spending time outside, but the proposed route would imply a four hour one way car commute, tilting the balance of commute to hike entirely in the wrong direction in my opinion. So we spontaneously compromised on a trail near lake Klöntal. My idea was to link some smaller unremarkable mountains that I hadn't been to yet. The trick was chosing a Southerly aspect for the hike and not go too high - both to avoid getting into snow. It worked out beautifully! We enjoyed great November weather. In the sun high above the perpetually shaded Klöntal valley lying still in hoar frost. We ticked two of my three planned summits. The third looked a bit too intimidating after we had just scrambled over some karst landscape and steep slippery grass. It's most definitely doable, but at the same time easy to get stuck in these conditions when you never know whether you'll encounter a patch of snow or ice in the wrong spot. So we watched the local chamois population show off their skills in difficult terrain and enjoyed each others company and conversation on the way back to the car.

Smoke from a wood fire hangin low in the valley.
Sun!
Lake Klöntal with the mighty Glärnisch North face. That'd be a dream to climb... an intimidating prospect.
Frozen pond.
Afroz.
Sören.
Some balance required.
View towards the Wägitalersee.
Some scrambling required.
Steep slope cleared of trees. Maybe by an avalanche a long time ago?
Back into the frozen valley.

2024-09-15

Cragging at Gallerie Amden

Fresh snow in the mountains on a weekend in September. A perfect weather forecast for Sunday. So where does one go? Gallerie Amden is the easy to reach, go-to crag for folks from Zürich. So this where Björn and I headed. And what felt like half of the rest of the city as well. The atmosphere was almost like a day on the beach. People enjoyed just hanging out with beautiful views of lake Walen. We stuck to the easier routes in the 6a range. As is typical for crags with a wide range of grades the easy ones ended up being the most polished and in any case presented a good challenge for us. The gallerie is notorious for crimpy small holds and fairly long routes - we needed the full length of my 70m rope. Good times.

The views and atmosphere aren't too shabby - especially considering that we are literally right on top of a road (the routes start on the roof of a half tunnel built as avalanche protection).
Björn preparing to come back down.
A common theme: climbers in t-shirts, belayers in down jackets ;-P
Sören.

2024-09-07

Barglen (2669m) via "Jesi", 190m, 6a

Björn and I decided to explore a new (to us) climbing area up at Melchsee-Frutt. The approach is a bit longer than usual, so we took Björn & Sandra's "Büssli" (VW van) and mounted our bicycles on the backrack. I have been to Melchsee a bunch of times, but got there by either walking up or taking the cable car. This time we drove the winding mountain road. It requires good timing as it changes direction every hour, going uphill on even hours and downhill on odd ones. Even before we reached the small road we passed crowded parking lots and lots of tents. The Modular Music Festival in full swing - drugged or hung over people walking about even in the early morning when we arrived ;-)

In the gravel on the approach.
Björn's macro photography.
At the start of our climb.

We mounted our bikes at the Melchsee and proceeded to pedal up towards the Tannensee. A surprising number of people walked around there too. Drawn by the "Plauschfischen", a invitation by the local fishery club. We kept biking until the road turned to gravel and eventually became too steep and loose for my bike. At that point only a few hundred meters of road were left anyways, so we proceeded by foot in pathless terrain. A long traverse under the looming cliff of the Hohmad. We couldn't find an obvious approach to gain it, so at some point we just scrambled up the steep gravel. We chose a spot that made us head straight towards a climbing party already on the wall. On noticing us, one of the climbers shouted down to us from about two pitches up. He appeared to be the local expert and was keen to demonstrate it. After he learned what route we were aiming for he lectured us on how it wasn't the best choice for that grade and recommended a different one. Björn and I consulted for a bit but in the end decided to stick with our original plan. We still learned of www.barglen.ch from the encounter, a great resource on routes in the area. Thanks!

Warmup.
Björn.
Nice chimney.

By 11 o'clock we had finally reached the start of our route and started climbing. This went very smooth and we did enjoy the route. If the others are even better as the guy claimed, we should definitely return for more! Our rappel line crossed a few other, harder, routes and we eye balled them, trying to gauge whether they'd be within our abilities. "Dä Burner" (topo1, topo2) looked particularly nice.

This was one of the harder pitches. Even slightly overhanging in places.
Björn walking up the ramp. The only section with a bit questionable loose rock.
Björn on the sharp end.

We made it back down by 3 pm and this time decided to traverse directly at the base of the cliff. There are a bunch of caves cut into the rock and quite a few of them were busy with climbers preparing their bivouacs for the night. Nice spot! Sheltered from the elements with great mountain panorama to fall asleep to. Unfortunately the weather forecast for the next day predicted thunderstorms and lots of rain. Wishing people the best of luck we reached our bikes and sped down the mountain back to the Melchsee village. Unfortunately (fortunately?) our timing was almost maximally bad wrt the road opening times. So we bridged the better part of an hour by sitting in the sun and enjoying a beer on the terrace of a café ;-)

Family picture!
Coming back down.
Quite steep!

A clean on-sight for the both of us. On a great weather day. In great atmosphere and mood. What more could you wish for?

Flowery meadows in September.
Random munition lying around.
Some sections were a bit swampy. Bare minutes before we got back onto the gravel road and our bikes I made a single misstep onto grass that was less solid than it looked ;-/

2024-08-31

Läged Windgällen West (2'572m) via "Langi Zyt", 200m, 6a

Björn and I went up to the Läged Windgällen. I had been there just a few weeks prior with Arne. At the time we had to bail before we even started climbing - just when we had reached the base of the wall, a thunderstorm was rolling in and got everything thoroughly wet. This time the weather was great. In fact, up high was where it was bearable, the valleys were way too hot. The one successful climb I had already done on this mountain was the Zentralpfeiler. I remembered that as a pretty old school mountaineering route, requiring strong nerves on so-so rock and sparse protection. Luigi had recently returned with somebody else, climbing yet another route, and reported that it was "mental": sparsely protected and with a lot of exposure. So it was that Björn and I came prepared, bringing extra trad gear and expecting the worst.

It's a pretty wide wall with lots of (potential) routes. But apparently the rock quality isn't consistently great.
Björn.
Sören.

We would be surprised! It turned out to be a complete plaisir climb. The rock quality, while not bomber everywhere, was solid enough for an alpine climb. And the protection and diffiulty grade were well within our comfort zone so that we didn't place any additional gear. We made good time up and down and were surprised that the route book, although being shared between two routes, had only few entries dating all the way back to 2005. Great day out! Can I haz moar of these pleaz?

IIRC This might have been the crux pitch?
Super fun dihedral.
The second but last 5c pitch.
Quite overhanging for the grade, but on friendly jugs. Fun!
"Langi Zyt" and "Linke Südwand" share a route book and merge near the top.
Family photo.
Final pitch to the top. At this point three routes converge and we weren't 100% sure we climbed the right one. They are all similar in grade, so I guess it doesn't really matter.
Almost there!
Preparing to rappel.
Björn celebrating.
Lunch break on a nice ledge on the way down.
I eye-balled the other routes that we crossed during the rappel. They look eminently doable and worth returning for.
This guy (or his cousins) accompanied us for quite some time.
Mission accomplished.

2024-07-20

Sweden

For our family summer vacation this year we went exploring in Sweden. We took the nighttrain from Zürich to Hamburg, rented an RV there and then drove North to Sweden via Denmark. We had no plan on where to go. To the extent that at some point we told the kids that we'd be crossing a long bridge in a few minutes - and then the bridge turned out to be a ferry. Talk about a vacation planned out in meticulous detail ;-P

The night train from Zürich to Hamburg. Operated by the Austrian railways. Obviously.
Leonie.
Lukas.

Our modus operandi was to just sort of eyeball the map for landmarks that looked interesting to us and then drive to the nearest campground in the area. We stayed for two nights on every campground so that we'd have a full day for exploring on each. The kids were great and quickly made friends wherever we went. Often finding other German or Swiss kids. But even if they didn't share a language things worked out.

First night in the RV.
Sea Life at the Timmendorfer Strand.
Captain Lukas the Cruel.

We had loose plans of meeting up with five different groups of friends. Two of which actually panned out. Thomas and Stine live in Hamburg but had existing commitments on the two weekends we passed through. Linus and Gintare live in Stockholm but had family emergencies preventing travel. Björn and Sandra are friends from Zürich who were driving around Norway at the time. We thought we might meet up on their way back "down", but the timing didn't work out. Bastian and Rebecca and kids were doing their traditional summer vacation in a rented country house. We stayed in their front yard for a few nights and had a great time together. Including visits to the nearby lake and Astrid Lindgren park. Finally Bernd and Anna and kids were coming down from the very North of Sweden where they spent a few weeks hiking and camping in the wilderness. At this point they are Sweden professionals, having thoroughly fallen in love with Lapland and travelling there often. We spent several nights together on different campgrounds on our shared journey back. This was particularly valuable time for me as they are old and good friends from Münster who we hadn't seen in enough years that we hadn't even met each others younger kids yet. Great to finaly remedy that! Hopefully there won't be many years before we see each other again!

Fehmarn Sound Bridge.
The "bridge" that turned out to be a ferry. Clueless vacation planners unite!
Vemmetofte Strand Camping in Denmark.

This was the first time we travelled in an RV and it turned out pretty well. I'd not rent the exact model again - it had a fatal design flaw where a foldable shower (nobody needs or uses that in a cramped RV!) caused one of the beds to be too short and worse, have a hard wooden board right through the middle of it. Whoever came up with that design clearly has never tried to sleep in it. Such a stupid and unnecessary mistake.

Cirkus Bella Donna giving a free performance on the campground.

We of course couldn't have known this ahead of time, but it turned out that we had fantastic weather for pretty much the entire duration of our visit. A small handful of rainy days, nothing but sunshine and blue skies for the rest. So we could easily have stayed in tents as well. The RV was meant as our rainy day backup solution - you don't want to be stuck in a tiny tent with cranky kids for an entire day.

The kids found new friends on every single campground we stayed at. Not least because of natural meeting points like this one ;-P
Highway cruising mode. Kids in the back, watching cartoons on the tablet.
Øresund bridge.

Overall a great vacation. I think we'll do this kind of road tripping again. And making a point of visiting old friends along the way!

First Camp Tylösand. Surprisingly the North Sea was much warmer and more inviting than the Baltic. The kids really loved the waves.
We could walk out a long distance - just had to keep the phone dry.
Lukas covered in a warming blanket of sand.
Anita and Leonie during our family mini golf championship.
In the dunes.
Elinge Älkpark. Small, but very nice and hands-on.
Habo Camping at lake Vättern.
Beautiful weather. Lots of people sunning on the sandy beach. Yet nobody in the water. Why?
Answer: water temperature of 16.5°C! Brrr!
Our home on the road.
Another batch of new friends.
Lukas and Leonie.
Borås Zoo. Lukas competing with Tyrannosaurus Rex for who's more intimidating.
The zoo had regular live musical shows.
Marstrands Familjecamping. Probably my favorite of the entire trip. The Swedish Schären archipelago is beautiful.
Carlsten Fort.
Exciting adventure to discover hidden passages in the old fortress.
Kids in the dungeon. I'll get one of those for our home. So convenient to lock the kids away for an hour or two. Or ten.
On the top of the inner citadel.
The kids getting acqainted with medieval torture and execution methods.
Barfotens Camping at lake Vänern.
The kids improvised their own tent.
That's not how you use a tractor!
Nässjö citi park. Very nice playground.
Art exhibit in the Nässjö library.
Cruising Swedish highways. Super relaxing driving. Especially with adaptive cruise control and an automatic gear shift. I had to be careful not to fall asleep at the wheel.
Ronja rövardotter.
Astrid Lindgren's world in Vimmerby. An amazing experience for both kids and adults. Lots of open air musicals where the stages double as playgrounds.
Nils Karlsson-Pyssling
Emil i Lönneberga
Pippi Långstrump
We visited the Basse family in their vacation home and stayed with them for a few days.
Filmbyn Småland in Mariannelund. Our own little Lönneberga getting his head stuck in the soup bowl.
Bolmens Camping. Exploring the islands in lake Bolmen.
Found a rope swing on one of the islands.
My crew of pirates.
Leonie and Rappa.
Exploring lake Bolmen by canoe. This time with Anna and kids as the bounty to be boarded and captured.
An island entirely to ourselves.
Tofu sausages.
Lena and Lukas. Not quite daring enough to fully go into the cold water, but still playing around.
At least one of our kids is turning into a climber.
Testing the life jackets...
...turned out to be great fun!
The gang. Leonie, Lukas, Jonathan, Anna, Thore, Bernd.
The Northern Sweden gang.
And the Southern Sweden gang (with a few extras) ;-)
Swimming off the coast of Fehmarn.
The kids enjoying free vehicles on the Niobe campground.
A horse riding lesson turned into just taking care of the horses because it was raining too much.
Killing time in Hamburg, waiting for the night train.
Quite a playground! Live music and a magician entertaining the kids.
Lukas trying on my clothes for size.