Model Scaffolding > Model Quality*

* Terms and Conditions Apply

TLDR;

• Models good, scaffolding gooder.

• Embeddings based search available to anyone with access to a laptop and who can run a jupyter notebook, coming soon.

• I’ll try to map Indonesian aquaculture with this workflow to prove it scales.

• Thank you Earth Genome for releasing data + code!

Model Scaffolding

In ‘AI’ world, model scaffolding is any post-training structure you wrap around a foundation model to guide or enhance it’s behavior. Prompts, memory stores, tool-calling or even calls to other models can all be qualified as scaffolding.

What is scaffolding for geospatial models? Embeddings-based search and human in the loop labeling and ML is an area that many have been investigating for a long time, yet is somehow still in my opinion under-explored. Humans are still state-of-the-art (SOTA) when it comes to end-to-end geospatial tasks like “find all the aquaculture ponds in Bali”, so any tool that can allow humans to inject their vision, reasoning and domain intuition is extremely high leverage.

Geospatial search is a great example of geospatial model scaffolding: it lets you

1. Click a point on the map.

2. Use embeddings to surface visually similar locations nearby.

3. Label positives and negatives on‑the‑fly to spin up a training dataset.

   What happens if the user needs more information than is available in the basemap/imagery to make a decision though?

The Earth Genome, released a notebook for embeddings-based crop classification recently. I’m currently working on modifying it to work with any custom set of embeddings, but I wanted to share a great example of additional model scaffolding to answer this question.

Street View

In the video below, I show how really simple model scaffolding can enable the incorporation of external information: the user can directly open Google Maps at the location, and use Street View imagery to confirm whether what they are looking at in the satellite imagery is aquaculture or not.

In the video (sound on to hear) I narrate how I:

1. Use the scaffolding to open a street view tab in my browser.

2. Explore the area to confirm that it indeed contains aquaculture.

3. Label a positive and begin searching

4. Find other instances of aquaculture in Bali.

With a single click, humans can now look at ground-level imagery before labeling satellite imagery: a trick inspired by the classic QGIS Street‑View plugin and by Soler et al. 2023’s clever crop‑type mapping in Thailand. The key here is integrating these existing tools with embeddings-based search, which allows users to scale their vision and reasoning abilities as well as their domain expertise on geospatial data.

This is an extremely simple thing to implement (~3 lines of code), and I can personally attest to how high leverage it is having used it to build datasets and perform mapping experiments:

import webbrowser
center = Map.center
url = f"https://www.google.com/maps/@{center[0]},{center[1]},15z" webbrowser.open(url, new=2)

Conclusions

• Human‑in‑the‑loop remains unbeaten for nuanced geospatial tasks; tiny scaffolds like this one multiply expert time. You have to look at the data, this is the cardinal rule of geospatial data science.

• Low‑hanging fruit first: rather than chase ever‑larger foundation models (which still lag supervised baselines on pixel‑wise segmentation), focus on interfaces that scale human level vision + reasoning + experience with geospatial data. This is how we will get adoption.

• I’ll release the notebook & backend soon so anyone can build an embeddings DB and start mapping with a set of embeddings parquet files.

• ipyleaflet’s look‑and‑feel is … serviceable. If you know a slicker JavaScript or Python framework or hacks to prettify ipyleaflet drop me a line!

• A big thank you to the Earth Genome for releasing both the notebook and the embeddings.