TIL search: search
Django 3.1 introduces PostgreSQL `search_type="websearch"` - which gives you search with advanced operators like `"phrase search" -excluding`. James Turk [wrote about this here](https://jamesturk.net/posts/websearch-in-django-31/), and it's also in [my weeknotes](https://simonwillison.net/2020/Jul/23/datasette-copyable-datasette-insert-api/).
I decided to add it to my Django Admin...
I figured out how to use a JSON API to run a very limited Google search today in a legit, non-screen-scraper way.
Google offer a product called [Programmable Search Engine](https://programmablesearchengine.google.com/), which used to be called [Google Custom Search](https://en.wikipedia.org/wiki/Google_Programmable_Search_Engine).
It's intended for creating a search...
I came up with this trick today, when I wanted to run a `LIKE` search against every column in a table.
The trick is to generate a SQL query that does a `LIKE` search against every column of a table. We can generate that query using another query:
```sql
select
'select * from "' || :table || '" where ' || group_concat(
'"' || name || '" like ''%'' || :search || ''%''',
' or...
You can search for a string in any resource loaded by a page (including across HTML, JavaScript and CSS) in the Debugger pane by hitting Command+Shift+F.
<img alt="Screenshot of search interface" src="https://raw.githubusercontent.com/simonw/til/main/firefox/search-across-all-resources.jpg" width="600">
This view doesn't search the body of any JSON...
...https://github.com/readthedocs/readthedocs.org/blob/master/docs/development/search.rst
It looks like the key bits of the server-side implementation live here:
- https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/api.py
- https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/faceted_search.py
The [GitHub Code Search beta](https://docs.github.com/en/search-github/github-code-search) is really powerful - it allows advanced search - including regular expression matches - against every public repo on GitHub.
It's still in a preview (December 2022) - you can [request access here](https://github.com/features/code-search-code-view/signup).
Today I used it to figure...
...Tailing the Heroku logs showed that it was getting a barrage of traffic to the `/search/` page, which implements faceted search and hence has an almost unlimited set of possible combinations - `/search/?tag=python&year=2023` and suchlike.
I have this in my `robots.txt` file precisely to prevent bots from causing problems here:
```
User-agent: *
Disallow: /search/
```
Evidently something...
I use Cloudflare's CAPTCHA (they call it a "Managed Challenge") on [simonwillison.net/search/](https://simonwillison.net/search/) to prevent crawlers from following every single possible combination of my [faceted search](https://simonwillison.net/2017/Oct/5/django-postgresql-faceted-search/) UI.
This was getting pretty annoying, since I had to wait for the challenge every time I searched...
...At first glance there isn't a GraphQL field for that page - but it turns out you can access it using a GitHub search:
topic:git-scraping sort:updated-desc
An oddity of GitHub search is that sort order can be defined using tokens that form part of the search query!
Here's a GraphQL query [tested here](https://developer...
...It turns out the `search()` operation can be used for this, given 100 repos at a time. The trick is to use the `repo:` search operator, e.g `repo:simonw/datasette repo:django/django` as demonstrated by [this search](https://github.com/search?q=repo%3Asimonw%2Fdatasette+repo:simonw/sqlite-utils&type=Repositories).
Here's the GraphQL query, tried out...
...It gains access to three new actions:
* `wikipedia: <search term>` - search Wikipedia and returns the snippet of the first result
* `simon_blog_search: <search term>` - search my blog and return the truncated text of the first result
* `calculate: <expression>` - evaluate an expression using Python's `eval()` function (so dangerous! This should absolutely use something like [a WebAssembly sandbox](https://til...
...Array.from(table.querySelectorAll("tr"))
.slice(1)
.map((tr) => {
return Array.from(tr.querySelectorAll("td")).map((cell) =>
cell.textContent.trim(),
);
}),
};
});
}
"""
def test_search(ds_server, page):
page.goto(ds_server + "/-/search?q=cleo")
# Should show search results, after fetching them
page.wait_for_selector("table")
tables = page.evaluate(table_js)
assert tables == [
{
"headers": ["rowid", "name", "description"],
"rows": [["1", "Cleo", "A...
...Here's how I did that using [ripgrep](https://github.com/BurntSushi/ripgrep):
```
$ rg --multiline --encoding none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .
```
The `--multiline` option means the search spans multiple lines - I only want to match entire files that begin with my search term, so this means that `^` will match the start of the file, not the start of...
...https://til.simonwillison.net/macos/atuin
autoload -U history-search-end
zle -N history-beginning-search-backward-end history-search-end
bindkey "^[[A" history-beginning-search-backward-end
```
...That one claims 45 searches rather than 77, I asked Claude Code and it noted that some of those `"type": "web_search_preview"` calls were `"action": "search"` but others were `"action": "open_page"`. I don't know if `"open_page"` calls count the same as searches for pricing, but the tool shows 45 searches and 24 pages visited (and 12...
...application/json" \
-d "$(jq -c -n --arg query '
{
search(type: REPOSITORY, query: "user:simonw topic:git-scraping", first: 100) {
repositoryCount
nodes {
__typename
... on Repository {
nameWithOwner
description
defaultBranchRef {
name
target {
... on Commit {
committedDate
url
message
}
}
}
}
}
}
}' '{"query":$query}')" \
| jq .data.search.nodes | sqlite-utils insert /tmp/github.db repos - --flatten
```
The line doing the work at the end is:
| jq .data.search...
...Algolia search
The official Hacker News search uses Algolia, with a constantly updated index.
If you search for `story:35111646`, filter for comments and then order by date you'll get the comments for that particular story, most recent at the top. Problem solved!
Here's the URL - edit the story ID in that URL (or on the page) to...
I wanted to add the new tutorials on https://datasette.io/tutorials to the search index that is used by the https://datasette.io/-/beta search engine.
To do this, I needed the content of those tutorials in a SQLite database table. But the tutorials are implemented as static pages in [templates/pages/tutorials](https://github.com/simonw/datasette.io...
...I used this search pattern:
`#(\d+) <https://github.com/simonw/datasette/issues/\1>`__
And this as the replacement value:
:issue:`$1`
Note that the search pattern uses `\1` as the group reference for the captured number, but in the replacement value you use `$1` to re-use that value.
Here's [the commit](https://github.com/simonw/datasette/commit/d2d53a5559f3014cccba2cba7e1eab1e5854c759...
...Response.redirect("/tils/feed.atom", status=301)),
(
"^/til/search$",
lambda request: Response.redirect(
"/tils/search"
+ (("?" + request.query_string) if request.query_string else ""),
status=301,
),
),
)
```
...data"
# It should have a search form
assert page.query_selector('form[action="/-/search"]')
def test_search(ds_server, page):
page.goto(ds_server + "/-/search?q=cleo")
# Should show search results, after fetching them
assert page.locator("table tr th:nth-child(1)").inner_text() == "rowid"
# ... assertions continue
```
There are two new tricks in here:
1. I'm using the...
...So, if all else is equal, SQLite will choose Idx1 with the hope of narrowing the search to as small a number of rows as possible.
I tried that on my global-power-plants table, which has an index `owner` and another on `country_long`:
```sql
explain query plan
select
country_long,
count(*)
from
[global-power-plants]
where
owner = 'Cypress...
...It allows users to search for names of functions and classes that occur at the top level of a Python file. It can search within specific files or directories and their subdirectories. It can also search for multiple symbols at once and support wildcards. The tool can be used in combination with `llm` command to grab specific class or function...
...I used the new [GitHub Code Search](https://cs.github.com/) to figure out how to do this. I searched for:
3.11 path:workflows/*.yml
And found [this example](https://github.com/urllib3/urllib3/blob/7bec77e81aa0a194c98381053225813f5347c9d2/.github/workflows/ci.yml#L60) from `urllib3` which showed that the version tag to use is:
3.11-dev
> **Update 28th Noveber 2022...
...Photos Library.photoslibrary/database/Photos.sqlite
301924352 /Users/simon/Library/Reminders/Container_v1/Stores/Data-A8BA15F3-C80E-43B3-9E21-F7D1CD42AB6A.sqlite
51343360 /Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite
47349760 /Users/simon/Library/Application Support/Firefox/Profiles/ljj897b8.default-release/webappsstore.sqlite
47185920 /Users/simon/Library/Application Support/Firefox/Profiles/ljj897b8.default-release/places.sqlite
33779712 /Users...
The `URLSearchParams` class, in [every modern browser](https://caniuse.com/?search=URLSearchParams) since IE 11, provides a sensible API for manipulating query string parameters in JavaScript. I first used it to build Datasette's column action menu, see [table.js](https://github.com/simonw/datasette/blob/0.50a0/datasette/static/table.js) and [issue 981](https://github.com/simonw/datasette...
...So where do those icons come from?
I looked at the HTML source code and searched the PyPI website source code for one of the CSS classes I found there: https://github.com/pypa/warehouse/search?q=fa-bug&unscoped_q=fa-bug
This lead me to the macro that defines the custom icons, which shows what custom keys or...
I ran a Google search this morning for `s3-credentials simon willison` and when I clicked on the top result it jumped me to a highlighted snippet of text on the page, despite that page not having relevant anchor links.
 Datasette instance carrying a SQL query that searches various types of content.
Result:
```bash
./blog.sh "Has Simon implemented GraphQL...
...I've implemented it at least a dozen times in my career, using everything from Solr and Elasticsearch to PostgreSQL (see [Implementing faceted search with Django and PostgreSQL](https://simonwillison.net/2017/Oct/5/django-postgresql-faceted-search/)) or SQLite (see [Datasette Facets](https://simonwillison.net/2018/May/20/datasette-facets/)).
The Django admin almost has facets out of the...
...Frequently check Reddit for new links to any page on `simonwillison.net` or `datasette.io`, using the unofficial search API - the Reddit search page with `.json` appended to the URL.
2. Record the results to a GitHub repository
3. Submit any new results to my personal Datasette Cloud instance via the Write API
4. Use [datasette-atom](https://datasette.io...
...My `decode_matchinfo()` function runs against a binary data structure returned by SQLite - more details on that in [Exploring search relevance algorithms with SQLite](https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/).
SQLite doesn't change the binary format depending on the endianness of the system, which means that my function here works correctly on little-endian...
I've been trying to get my head around [ColBERT](https://github.com/stanford-futuredata/ColBERT).
> ColBERT is a fast and accurate retrieval model, enabling scalable BERT-based search over large text collections in tens of milliseconds.
But what does that mean?
I understand [embedding models](https://simonwillison.net/2023/Oct/23/embeddings/) pretty well now. They let you take...
...I also installed [datasette-block-robots](https://github.com/simonw/datasette-block-robots) to prevent search engines from crawling the site and waking it up more than strictly necessary.
## Using it from an Observable Notebook
My [Search for Faucets with CLIP, API edition](https://observablehq.com/@simonw/search-for-faucets-with-clip-api) Observable notebook makes calls to my newly...
...HTML I like two space indents.
Today I figured out how to teach VS Code those defaults.
1. Hit `Shift+Command+P` to bring up the action menu
2. Search for the `Preferences: Configure Language Specific Settings...` item
3. Select the language, e.g. `HTML` or `Python`
4. Add `"editor.tabSize": 2` to the corresponding JSON object
5. Save that...
Search and replace with regular expressions in VS Code
...I figured someone must have solved this, so I ran a GitHub code search for [purpleair.com map json](https://github.com/search?q=purpleair.com+map+json&type=code) and found [zakj/scriptable](https://github.com/zakj/scriptable) with [code for decoding that](https://github.com/zakj/scriptable/blob/d42a2d7727a2642ad6ee880c15ae65dca4232fe4/purpleAir.js). I adapted it to the following:
```javascript...
...Start with the + menu next to the chat prompt and navigate through it like this:
![The Claude prompt box with the + menu open, showing Add files or photos, Take a screenshot, Add to project, Add from GitHub, Skills, Connectors (highlighted), Add plugins and Web search. The Connectors submenu lists Add connector, Manage connectors and toggles for Descript, Gmail, Google Calendar...
...find /tmp/iPhone | grep 'wal'
This searches for files with names like `downloads.28.sqlitedb-wal` - which indicate a SQLite database that has been opened in WAL mode.
On my iPhone I get these:
```
/tmp/iPhone/Downloads/downloads.28.sqlitedb-wal
/tmp/iPhone/Books/MetadataStore/BookMetadataStore.sqlite-wal
/tmp/iPhone/Books/Sync/Database/OutstandingAssets_4.sqlite-wal
/tmp/iPhone/Radio...