Annie Sexton

Annie Sexton is a Developer Educator at ngrok with a pas­sion for nerd-snip­ing de­vel­op­ers. She also has over a decade of ex­pe­ri­ence work­ing at PaaS com­pa­nies such as Heroku, Render, and Fly.io.

For help please visit help.ft.com. We apol­o­gise for any in­con­ve­nience.

The fol­low­ing in­for­ma­tion can help our sup­port team to re­solve this is­sue.

A white na­tion­al­ist with neo-Nazi links who hurled racist abuse at the Indian prime min­is­ter dur­ing his visit last month is be­ing paid by Facebook’s par­ent com­pany, Meta, to pro­duce con­tent.

An ABC NEWS Verify in­ves­ti­ga­tion into sev­eral con­tro­ver­sial Australian pages on Facebook has found that some are ben­e­fit­ing di­rectly from so­cial me­dia mon­eti­sa­tion pro­grams run by Meta.

Hugo Lennon is a far-right ag­i­ta­tor who was for­mally moved on by Victoria Police for shout­ing pro­fan­i­ties and racist abuse at Narendra Modi at a Melbourne ho­tel.

He has been re­ceiv­ing rev­enue through Facebook’s “Content Monetization” pro­gram since September 2025.

White na­tion­al­ist Hugo Lennon be­ing con­fronted by Victoria Police af­ter yelling racist abuse at India’s prime min­is­ter. (Supplied)

Mr Lennon has been pho­tographed with known neo-Nazis and has in­ter­acted with them on­line.

His re­cent Facebook videos in­clude his out­burst at the Indian PM, and oth­ers ref­er­enc­ing white su­prema­cist con­cepts like re­place­ment the­ory and rem­i­gra­tion.

The Noticer, a far-right Australian news web­site that reg­u­larly pro­motes white su­prema­cist and neo-Nazi ide­olo­gies, has been mak­ing money through the same Facebook rev­enue pro­gram since November 2025.

There is a short pe­riod from January 11 to February 15 when it stopped earn­ing money for un­known rea­sons.

The Noticer is cur­rently sus­pended from its X ac­count with a no­tice that the ac­count is “temporarily un­avail­able be­cause it vi­o­lates the X Hateful Profile Policy”.

A page pro­mot­ing the anti-im­mi­gra­tion group “March for Australia”, which also has close ties to neo-Nazis, reg­is­tered for the pro­gram in December 2025.

Monica Smit, the founder of the anti-vac­cine and anti-lock­down group Reignite Democracy Australia, joined Facebook’s con­tent mon­eti­sa­tion pro­gram in September 2025.

She ap­pears to have been earn­ing ad­ver­tis­ing rev­enue through Facebook from as far back as 2017.

Her page fea­tures vac­cine mis­in­for­ma­tion and pro­motes the sale of “radiation pro­tec­tion” bracelets through her per­sonal web­site.

Hugo Lennon, Monica Smit, The Noticer and March for Australia did not re­spond to re­quests for com­ment.

Facebook told ABC NEWS Verify “it is not Meta’s role to po­lice of­fen­sive­ness”. (Reuters: Dado Ruvic/illustration)

Invitation only

Facebook dis­trib­uted nearly $US3 bil­lion ($4.27 bil­lion) to an es­ti­mated 16.2 mil­lion mon­e­tised ac­counts in 2025.

Meta has pub­lished reg­u­lar dis­clo­sures re­lat­ing to these “partner-publishers” since 2019.

Tech pol­icy and ac­count­abil­ity non-profit What To Fix col­lates this raw data into a search­able archive that was used by ABC NEWS Verify for this in­ves­ti­ga­tion.

According to Facebook, the mon­eti­sa­tion scheme that Hugo Lennon, Monica Smit, March for Australia and The Noticer ap­pear to be prof­it­ing from is “an in­vi­ta­tion-only pro­gramme that lets you earn money from the per­for­mance of your el­i­gi­ble pub­lic reels, pho­tos, sto­ries and text posts.”

What To Fix ex­ec­u­tive di­rec­tor Victoire Rio said it meant that, in ef­fect, Facebook was in a busi­ness re­la­tion­ship with these peo­ple and or­gan­i­sa­tions.

“If we con­sider that Meta is in a di­rect com­mer­cial re­la­tion­ship with its pub­lish­ers, in­so­far that they’re pay­ing them roy­al­ties and that they have a mon­eti­sa­tion agree­ment with them, then you could ar­gue that they are li­able for the con­tent that gets pro­duced by their busi­ness part­ner in some ways.”

Ms Rio said she was no longer sur­prised by some of the pages Meta was choos­ing to mon­e­tise.

“We’ve ac­tu­ally en­coun­tered ac­counts af­fil­i­ated with sanc­tioned en­ti­ties that were part of those pro­grams,” she said.

“It’s quite com­mon that we see ac­counts that are very clearly in vi­o­la­tion of their poli­cies get­ting through.

“I think the ques­tion here re­ally is: is the qual­ity of the en­force­ment of those poli­cies that they have.”

RDA founder Monica Smit de­liv­ers a speech out­side Daniel Andrews’ elec­torate of­fice on Tuesday. (Facebook)

Policy Violation

The pages ex­am­ined by ABC NEWS Verify all ap­peared at times to be in di­rect vi­o­la­tion of Facebook’s own poli­cies around con­tent mon­eti­sa­tion.

They state that “debated so­cial is­sues” such as race “may face re­duced or re­stricted mon­eti­sa­tion”, while “misleading med­ical in­for­ma­tion” is “ineligible to mon­e­tise”.

Independent right-wing ex­trem­ism re­searcher Kaz Ross said pay­ing con­tro­ver­sial cre­ators was a de­lib­er­ate strat­egy from Meta.

“Their fi­nan­cial model is to re­ward con­tent cre­ators who get en­gage­ment, and as we know, the best way of get­ting en­gage­ment is to pro­duce rage bait, ex­trem­ist ma­te­r­ial, ag­gra­vat­ing ma­te­r­ial: any­thing that will get peo­ple up­set and ar­gu­ing is the best way to get en­gage­ment,” Dr Ross said.

Meta did not an­swer spe­cific ques­tions about these pages and their con­tent but pro­vided a gen­eral state­ment.

“We have clear poli­cies in place which any­one us­ing our mon­eti­sa­tion tools must com­ply with.“When Content Monetisation Pages vi­o­late our Community Standards, we ap­ply penal­ties such as tem­porar­ily or per­ma­nently dis­abling their abil­ity to earn on our plat­forms.“Cre­ators or pub­lish­ers who re­peat­edly breach these poli­cies will be re­moved from our plat­forms.”

“We have clear poli­cies in place which any­one us­ing our mon­eti­sa­tion tools must com­ply with.

“When Content Monetisation Pages vi­o­late our Community Standards, we ap­ply penal­ties such as tem­porar­ily or per­ma­nently dis­abling their abil­ity to earn on our plat­forms.

“Creators or pub­lish­ers who re­peat­edly breach these poli­cies will be re­moved from our plat­forms.”

The com­pany also said it was, “important to dis­tin­guish be­tween speech that is of­fen­sive and con­tent that could po­ten­tially lead to of­fline vi­o­lence”.

“Such speech may be of­fen­sive to many, but it is not Meta’s role to po­lice of­fen­sive­ness,” it said.

‘Any amount is too much’

It is un­clear ex­actly how much money these spe­cific cre­ators have made from their Facebook pages as this in­for­ma­tion is not made pub­lic by Meta.

Dr Ross said any amount was too much.

“It’s morally bank­rupt to do that, to know that these peo­ple are pro­duc­ing con­tent that’s in breach of your own reg­u­la­tions, yet you’re ac­tu­ally not just al­low­ing them, but you’re en­cour­ag­ing them by pay­ing them,” she said.

Meta is also one of the few so­cial me­dia gi­ants that dis­closes any in­for­ma­tion about who it is pay­ing. Other plat­forms, like YouTube, TikTok and Twitter, have very lit­tle trans­parency in this space.

“I think in gen­eral, Meta has been more forth­com­ing with trans­parency and that’s some­thing that should be lauded,” What To Fix’s Victoire Rio said.

“It’s still a very low bar, and the ques­tion here is to re­ally push for all plat­forms to at least meet that bar and if not ac­tu­ally go higher,′ she said.

It’s 2020. You’re the most se­nior per­son on your team, in charge of code qual­ity and ar­chi­tec­ture. You’ve set up good en­gi­neer­ing prac­tices, you thor­oughly re­view PRs from peo­ple who are less ex­pe­ri­enced than you and work hard to main­tain a healthy code­base.

Then at some point, you go on hol­i­day. When you come back, the code­base is a mess. Everyone merged each oth­er’s PRs with­out re­ally pay­ing much at­ten­tion, some­one added a bunch of new ta­bles to the data­base to de­nor­malise it be­cause it was eas­ier and they added server­less or Kafka to the stack with­out any solid ev­i­dence that they needed ei­ther.

It’s okay. You can fix this.

Fast for­ward to 2026. You haven’t been on hol­i­day. It’s just a nor­mal Monday morn­ing. You make your­self a nice cof­fee, open your com­puter and find your­self with 7 PRs to re­view. You open the first one: +24506 – 3938 lines, ac­com­pa­nied by some AI-generated de­scrip­tion of what they’re sup­posed to do. Somehow, your team has made more changes since Friday than they used to make while you were away for a few weeks.

AI re­moved the speed limit

AI makes pro­jects with weak en­gi­neer­ing cul­ture fail much faster.

There used to be a time when peo­ple sat down and talked about how they’d do some­thing. Now they can just prompt an agent for a few hours and open a PR.

The most tragic as­pect of this way of work­ing is that, to the un­trained eye, it works.

If you pull the branch and test it, you’ll prob­a­bly get some­thing some­what func­tional. So what do they do? They keep go­ing. Again and again. Until the pro­ject reaches a point where no one knows how any­thing works.

Just like some­one buy­ing a new lux­ury car on a credit card. You don’t see the debt. You just see the car that looks great.

But then users start to re­port a weird bug. It’s the 4th time your team has been try­ing to fix it. I mean… ask­ing AI to fix it. Unfortunately, it seems like not even Fable can fig­ure it out.

You go talk to the per­son who worked on this fea­ture.

“So where does the data come from?”

“Hmm… ac­tu­ally I don’t know. Let me ask Claude.”

You sit next to each other watch­ing an end­less wall of text ap­pear on the screen. Neither of you has any idea whether any of it is true but Claude seems very con­fi­dent.

“Let’s just turn on ul­tra­code and ask it to dou­ble-check?”

This one will take a while. You start talk­ing about the lat­est drama on X.

You fi­nally get an an­swer back.

“Does this make any sense to you?”

“I’m not sure.”

“Didn’t you build this like… last week?”

Silence.

This pro­ject has be­come so con­vo­luted, with so many lay­ers and ser­vices, that no one on your team could pos­si­bly start to un­der­stand what’s go­ing on.

So, what do you do?

Fixing it would re­quire such a colos­sal amount of work that it would be im­pos­si­ble to even start jus­ti­fy­ing it to any­one in man­age­ment.

And what are you even think­ing about? It would end up in the ex­act same state again in just a few months any­way.

“Let’s just ask Claude to fix it.”

“Okay. I’ll cre­ate a loop and goal so it does­n’t stop un­til it’s checked that every­thing works.”

“Sounds good”

“Actually, I ran out of Fable us­age for to­day so I’ll run it to­mor­row”

You grab an­other cof­fee and walk back to your com­puter. You now have 13 PRs left to re­view. You see some­thing you don’t quite un­der­stand, so you mes­sage the per­son who wrote it.

“Why are we do­ing this here?”

They send you a link. It’s a Claude con­ver­sa­tion.

Somewhere in that con­ver­sa­tion, buried be­tween Claude con­fi­dently rec­om­mend­ing one ar­chi­tec­ture, apol­o­gis­ing, chang­ing its mind, your coworker ask­ing it to re­con­sider again and an­other 15 rounds of changes, is ap­par­ently the de­sign de­ci­sion be­hind this code.

“Which part should I read?”

“Probably all of it.”

Does this sound fa­mil­iar?

Whenever I talk about this, some­one even­tu­ally tells me that no­body ever fully un­der­stood large sys­tems any­way. It’s true.

You were never ex­pected to un­der­stand every ser­vice and every data­base. But at least some­one did and would ex­plain it to you.

Now they ask an LLM be­cause they don’t ac­tu­ally know them­selves.

You can’t af­ford bad en­gi­neers any­more

In every team, there are com­pe­tent peo­ple who make the pro­ject pos­si­ble. There are also peo­ple who es­sen­tially make it harder for every­one else. And now any­one can pro­duce more code in a day than they used to in a year.

In the story above, every­one is fail­ing:

The en­gi­neer open­ing a 25,000-line PR should have stopped the agent long be­fore it got there. They should have un­der­stood what it was do­ing, bro­ken the work into smaller pieces and ques­tioned every new ab­strac­tion it in­tro­duced.

The per­son re­view­ing it should have re­fused to re­view some­thing that large in­stead of giv­ing in.

The per­son adding Kafka should have been able to ex­plain ex­actly why it was needed.

The per­son who built the fea­ture should have been able to ex­plain where the data came from with­out send­ing a link to a Claude con­ver­sa­tion.

But what’s the prob­lem then? Just use AI to fix it. Well, it’s not that easy…

Before any­one jumps on this, none of this means tech­ni­cal debt is al­ways bad. The im­por­tant part is that you know it’s a short­cut.

Anyway, re­vert­ing a bad de­ci­sion is hard. Very hard.

For ex­am­ple, how long would it take an LLM to add a bunch of ta­bles and columns to the data­base? 10 min­utes?

But once you start stor­ing data there, you can’t just re­move them. You have to come up with a mi­gra­tion plan, make sure you don’t dis­rupt the sys­tem be­cause peo­ple are pay­ing to use this every day. You have to think about what you’ll do if the mi­gra­tion fails. Make sure you don’t end up with or­phaned for­eign keys. It’s just so much harder to fix. Even with the best model you can get.

And while you’re fix­ing it, more PRs keep com­ing in. More code, more ab­strac­tions, more de­ci­sions. A per­son can gen­er­ate 20,000 lines of code in an af­ter­noon, but you still have to sit there and un­der­stand what those lines ac­tu­ally do.

By the time you’ve un­tan­gled one bad de­ci­sion, five more have been merged.

The new AI econ­omy

Of course, bad en­gi­neers were al­ways a li­a­bil­ity.

It has been like this for decades, well be­fore OpenAI or Anthropic ex­isted. Bad de­ci­sions com­pounded, un­nec­es­sary com­plex­ity ac­cu­mu­lated and teams ended up main­tain­ing sys­tems no­body re­ally un­der­stood.

The dif­fer­ence is that there used to be a limit to how fast you could do it.

Today, im­ple­men­ta­tion is cheap. You are paid to make good de­ci­sions. To build soft­ware that will scale while man­ag­ing com­plex­ity.

Ask your­self why com­pa­nies are pay­ing six-fig­ure salaries for en­gi­neers in London or San Francisco in the first place.

If all they needed was some­one who could turn a spec­i­fi­ca­tion into work­ing code, why were they pay­ing that much when they could al­ready get it done cheaply else­where?

Why are the tech com­pa­nies claim­ing that “software is solved” still pay­ing top salaries to at­tract the best peo­ple they can?

My bet is that AI pushes salaries fur­ther apart. To be em­ploy­able, there’s a bar you have to clear and that bar is what­ever the cur­rent best model du jour can do.

Good en­gi­neers have be­come more valu­able be­cause AI lets them move much faster. They don’t need as many peo­ple around them just to do the im­ple­men­ta­tion work any­more.

At the same time, bad en­gi­neers have be­come much more ex­pen­sive to hire.

I wrote about this be­fore when I said the vibe coder ca­reer path is doomed.

You need to con­tribute be­yond what every­one al­ready gets by giv­ing an agent a prompt.

If you lack the judg­ment re­quired to eval­u­ate the LLM’s rec­om­men­da­tion, ask­ing for more judg­ment does­n’t solve the prob­lem.

At some point, some­one still has to know what is go­ing on. And that’s the most valu­able per­son on the team.

The peo­ple who don’t will be­come much cheaper to hire or get re­placed en­tirely while the money gets fun­nelled to­wards an in­creas­ingly smaller num­ber of peo­ple who can ac­tu­ally be trusted.

I don’t think this is go­ing to be lim­ited to soft­ware en­gi­neer­ing ei­ther. I be­lieve the same thing is go­ing to hap­pen across most knowl­edge work. AI will make the best peo­ple much more pro­duc­tive and the bad ones al­most im­pos­si­ble to hire. Before, there was a good chance some­one would catch their bad de­ci­sions be­fore they went too far. Now they can make changes faster than any­one around them can re­al­is­ti­cally re­view or un­der­stand them.

At the end of last year, our up­time was pretty shaky. You can see this trend on our sta­tus page, and that in­sta­bil­ity con­tin­ued into the new year. Many of these out­ages were caused by a sin­gle bug, deep in SQLite. It took months of in­tense foren­sics to track it down.

Now we’re in sum­mer, we’re con­fi­dent that we’ve found the bug, that we un­der­stand it—and more im­por­tantly, that we’ve fixed it.

We know our cus­tomers ex­pect Tailscale to be a re­li­able ser­vice, and for sev­eral months we did­n’t live up to that promise. That’s dis­rup­tive, and we’re sorry. We’re pub­lish­ing this blog post to ex­plain what went wrong, how we re­sponded, and how we ul­ti­mately helped to un­cover a long-stand­ing bug in the heart of the SQLite data­base.

Tailscale’s data­base ar­chi­tec­ture

While our clients in­ter­act with our con­trol plane as a sin­gle pub­lic end­point (controlplane.tailscale.com), in­ter­nally, our con­trol plane is split into a se­ries of co­or­di­na­tion servers (or “shards”). Each tail­net lives on one in­ter­nal shard at a time, but can mi­grate seam­lessly from one to an­other. These shards are an in­ter­nal im­ple­men­ta­tion de­tail: you don’t know what shard your tail­net is on, and you never need to.

Each shard has an SQLite data­base that holds all the in­for­ma­tion about the tail­nets on that shard. A sin­gle Go process ex­clu­sively ac­cesses that data­base, and serves the con­trol plane for those tail­nets. This sin­gle-writer de­sign is ex­actly how SQLite is meant to be used.

We’ve used SQLite as our pri­mary data­base since 2022, and we chose it be­cause it’s well-known, re­li­able, and widely used. SQLite is “boring tech­nol­ogy”—in a good way. Many com­pa­nies use SQLite in much larger de­ploy­ments with­out is­sue, and we ex­pected the same stress-free us­age.

In our cur­rent backup pipeline, we take a com­plete snap­shot of the data­base every few min­utes, then up­load the en­tire SQLite file to an S3 bucket. We’d been run­ning this setup with­out in­ci­dent since early 2023.

Fast for­ward to August last year, when a data pipeline that reads those S3 back­ups re­ported an er­ror in one of our data­bases. We ran SQLite’s PRAGMA in­tegri­ty_check com­mand against the backup, and found it was in­deed cor­rupted. SQLite cor­rup­tion is pos­si­ble, but it’s highly un­usual and not some­thing you should en­counter in nor­mal op­er­a­tion. We re­paired the af­fected data­base, and in­ves­ti­gated the cause, but to no avail.

When op­er­at­ing at scale, even rare events can oc­cur with some fre­quency, so we should have been un­sur­prised when it hap­pened again—and again, and again, and again. In to­tal, we faced 19 sep­a­rate in­stances of data­base cor­rup­tion over six months be­fore we fi­nally re­solved the un­der­ly­ing bug.

When you hear the phrase “database cor­rup­tion”, it’s nat­ural to worry about data loss. Because our con­trol plane only han­dles con­fig­u­ra­tion data, these data­bases con­tain meta­data about your tail­net and de­vices, but never your pri­vate en­cryp­tion keys or net­work traf­fic. In the ear­li­est in­ci­dents, the re­cov­ery process meant a hand­ful of newly added de­vices or con­fig­u­ra­tion changes did­n’t per­sist, and a small amount of meta­data had to be re-en­tered.

Whenever cor­rup­tion oc­curred, we had to stop the con­trol plane process on the shard while we re­paired or re­stored the data­base. This was painful for tail­nets on that shard, be­cause their en­tire con­trol plane dis­ap­peared dur­ing that re­cov­ery win­dow. In the early in­ci­dents, that down­time was over an hour, but we grad­u­ally sped up the re­cov­ery process over sub­se­quent in­ci­dents.

Each tail­net is a mesh net­work, where de­vices make peer-to-peer WireGuard® con­nec­tions to each other. When a de­vice joins the tail­net, it has to get a list of other de­vices from the con­trol plane be­fore it can es­tab­lish new con­nec­tions—so if a de­vice came on­line dur­ing the SQLite down­time, it could­n’t con­nect. While the data­base was be­ing re­paired, de­vices al­ready on­line re­mained con­nected to each other, but they could­n’t learn about changes to the net­work. Those tail­nets also tem­porar­ily lost ac­cess to the web-based ad­min con­sole and the Tailscale API.

There’s also a broader im­pact on trust. We post a global in­ci­dent on our sta­tus page even when only a small num­ber of tail­nets are af­fected. Many peo­ple saw a sta­tus page event for an in­ci­dent that did­n’t af­fect them. Indeed, the ma­jor­ity of shards and tail­nets were never in­volved in a data­base cor­rup­tion in­ci­dent! Nonetheless, re­peated down­time erodes trust, whether or not you’re di­rectly af­fected.

From the very first in­stance of cor­rup­tion, we knew this was a se­ri­ous threat to our re­li­a­bil­ity, and we threw a lot of en­gi­neer­ing time at the prob­lem—but the fix was­n’t easy.

Trying to find the fault

This bug re­sisted all our ini­tial at­tempts to find it.

We looked at re­cent changes, but there weren’t any that seemed rel­e­vant. Nobody had been work­ing on our low-level code that in­ter­acts with SQLite, be­cause it had all been writ­ten years ago and pre­sented no is­sues up un­til that point. We re-re­viewed all of that code with a fine-toothed comb to look for pre­vi­ously missed bugs, but we did­n’t find any­thing that would cause the cor­rup­tion we were see­ing.

We looked for com­mon fac­tors be­tween cor­rup­tion in­ci­dents, but we could­n’t find any. It was­n’t tied to a sin­gle shard, or cus­tomer, or tail­net fea­ture, or time of day, or load level. We were at a loss for what might be trig­ger­ing the be­hav­iour.

This lack of re­li­able trig­ger con­di­tions meant we could­n’t re­pro­duce the bug syn­thet­i­cally. Instead, we had to rely on de­ploy­ing pas­sive, foren­sic teleme­try in our live en­vi­ron­ment to catch the cor­rup­tion red-handed. Gathering live di­ag­nos­tics for a data­base is­sue is the last thing we wanted to do, but we had no choice.

As an ad­di­tional com­pli­ca­tion, the cor­rup­tion did­n’t oc­cur on a reg­u­lar sched­ule. Sometimes in­ci­dents would be hours apart, other times weeks. This made it dif­fi­cult to pre­dict progress or plan fur­ther work, be­cause we were never sure when we’d get our next di­ag­nos­tic dump. We had a six-week pe­riod be­tween October and December when there were no cor­rup­tion in­ci­dents, be­fore they re­turned as an un­wel­come Christmas pre­sent.

Because this would­n’t be a quick or easy fix, we reached out to the SQLite de­vel­op­ers for a pro­fes­sional sup­port con­tract. This was a great de­ci­sion. It gave us di­rect ac­cess to their deep ex­per­tise and ex­pe­ri­ence, and we had many de­tailed tech­ni­cal con­ver­sa­tions about our ar­chi­tec­ture and our in­ci­dents.

Between Tailscale en­gi­neer­ing and the SQLite core de­vel­op­ers, we mapped out sev­eral the­o­ries for what might be caus­ing the cor­rup­tion—in­clud­ing bro­ken POSIX locks on close(), mis­man­ag­ing mem­ory owned by SQLite, or ac­ci­den­tally us­ing SQLite from mul­ti­ple threads while dis­abling thread safety. After every in­ci­dent, we gath­ered more data, added more di­ag­nos­tics, and sys­tem­at­i­cally ruled out these the­o­ries. We were grad­u­ally con­verg­ing on the true bug.

The trans­ac­tions that did­n’t bark

While we were in­ves­ti­gat­ing the root cause, we still had a live plat­form to run. We took ag­gres­sive steps to au­to­mate re­cov­ery and min­i­mize down­time:

Configuring our con­trol plane shards to hard-stop im­me­di­ately upon en­coun­ter­ing cor­rup­tion

Deploying an au­to­mated backup mon­i­tor that con­tin­u­ously ran PRAGMA in­tegri­ty_check over our back­ups

Improving our run­books and on-call train­ing

These ef­forts cut our re­sponse time to un­der an hour—and then we dis­cov­ered an un­ex­pected clue.

We wanted a way to re­store ser­vice that did­n’t in­volve rolling back to the last known-good backup (which would lose a lot of data) or re­pair­ing the known-cor­rupted data­base (which was po­ten­tially risky).

To do this, we built a trans­ac­tion log­ging pipeline. We streamed every SQL state­ment that mod­i­fied the data­base to a sep­a­rate log file. Because SQLite is a sin­gle-writer data­base with se­ri­al­is­able trans­ac­tions, our trans­ac­tion his­tory was com­pletely lin­ear and de­ter­min­is­tic. (This would­n’t be true in a multi-writer data­base like Postgres or MySQL.) Replaying those trans­ac­tions against the lat­est known-good backup should re­store the data­base to its most re­cent state, safely by­pass­ing the cor­rup­tion.

This pipeline worked, but then it did some­thing even bet­ter: it gave us a clue.

In two in­ci­dents, our trans­ac­tion logs failed to re­play cleanly. Upon closer in­spec­tion, we dis­cov­ered that data writ­ten and com­mit­ted by one trans­ac­tion was in­ex­plic­a­bly in­vis­i­ble to later trans­ac­tions. A write had van­ished into thin air with­out rais­ing an er­ror. That should be im­pos­si­ble!

The writ­ing on the WAL

As these in­ci­dents were on­go­ing, the SQLite de­vel­op­ers had been de­vel­op­ing a new de­bug­ging tool. For a while, we’d sus­pected that the bug was some­where in the check­point process. They were build­ing a new tool to give bet­ter vis­i­bil­ity into what was hap­pen­ing dur­ing check­points.

To un­der­stand what this tool found, we need to briefly ex­plain how SQLite check­points work.

A SQLite data­base is made of a se­ries of “pages”, tiny blocks of in­for­ma­tion. When you up­date the data­base, some of those pages need to be re­placed with new pages with the up­dated in­for­ma­tion.

For bet­ter per­for­mance and greater con­cur­rency, we run SQLite with Write-Ahead Logging, which means new pages aren’t writ­ten di­rectly to the data­base file. Instead, they’re writ­ten to the “write-ahead log” or “WAL file”.

New pages can’t be writ­ten to the WAL file in­def­i­nitely; at some point they have to be copied back to the main data­base file. This process is called “checkpointing”.

In most de­ploy­ments, SQLite it­self de­cides when to do a check­point, and the process is in­vis­i­ble to the end user and de­vel­oper. In our con­trol plane, we take man­ual con­trol of the check­point process so we can run fast and con­sis­tent back­ups. This non-stan­dard ap­proach seemed sus­pi­cious as we steadily elim­i­nated po­ten­tial causes.

One clue was that dur­ing cor­rup­tion in­ci­dents, our met­rics showed that SQLite would re­port copy­ing more pages from the WAL file than were ac­tu­ally avail­able. If there are 10 pages in the WAL file and 20 pages get copied to the data­base, some­thing is clearly wrong.

To un­der­stand what was hap­pen­ing dur­ing these faulty check­points, the SQLite de­vel­op­ers cre­ated a new de­bug­ging tool for the vir­tual filesys­tem layer.

SQLite is split into sev­eral lay­ers. The top layer is the parser and code gen­er­a­tor, which con­verts SQL state­ments into SQLite’s in­ter­nal data struc­tures. These data struc­tures get passed to the pager, which splits them into the in­di­vid­ual pages to be writ­ten to disk. Actually writ­ing them to disk is han­dled by the OS in­ter­face, or “virtual filesys­tem”. Currently SQLite has two main­stream vir­tual filesys­tem im­ple­men­ta­tions—Unix and Windows.

If you’re in­ter­ested in a deeper dive on these in­ter­nals, I rec­om­mend this lec­ture by Richard Hipp, the pri­mary au­thor of SQLite.

This ap­proach al­lows you to re­place dif­fer­ent lay­ers with dif­fer­ent im­ple­men­ta­tions, or wrap an ex­ist­ing layer to get more in­for­ma­tion. To help di­ag­nose our prob­lem, the SQLite de­vel­op­ers cre­ated a wrap­per around the vir­tual filesys­tem that writes ad­di­tional trac­ing in­for­ma­tion and logs about changes to the data­base. This wrap­per is called the tm­stm­pvfs shim, and the source code is avail­able in the SQLite pub­lic repos­i­tory.

We de­ployed the shim into our live en­vi­ron­ment, and waited for the next cor­rup­tion to oc­cur. Fortunately, we did­n’t have to wait long.

The WAL-Reset bug

After our next cor­rup­tion in­ci­dent, the ad­di­tional logs from the new tm­stm­pvfs shim al­lowed the SQLite de­vel­op­ers to find and fix the bug: a rare data race in the SQLite source code be­tween a check­point and a write trans­ac­tion.

In par­tic­u­lar, if a write oc­curs at a spe­cific time dur­ing a check­point, the check­point­ing process gets con­fused—it thinks some of the pages have been copied from the WAL into the main data­base file, but they haven’t. Those pages never get writ­ten to the data­base file, and that data is per­ma­nently lost. The data­base file be­comes cor­rupt, be­cause other pages which ref­er­ence those pages—such as an in­dex—are writ­ten to the data­base.

The SQLite de­vel­op­ers named this the “WAL-Reset bug”, and they es­ti­mate it was pre­sent in SQLite for at least 16 years. It could ex­ist that long be­cause it was rare—so rare, the SQLite de­vel­op­ers had to add code to de­lib­er­ately trig­ger it in their test­ing en­vi­ron­ments. Their fix adds an ad­di­tional check to the check­point­ing func­tion which de­tects when the WAL has been re­set by an­other thread.

They con­firmed that this bug caused all of the baf­fling be­hav­iour we’d seen. It ex­plained the cor­rup­tion, the trans­ac­tion logs that would­n’t ap­ply cleanly, and the in­con­sis­tent check­point sta­tis­tics. They also ex­plained why we were more likely to hit the bug than other SQLite users: we take man­ual con­trol of the check­point­ing process, and we check­point very ag­gres­sively. Even a bug trig­gered by a rare con­di­tion was bound to hit us even­tu­ally.

This was an ex­cit­ing mo­ment. After months of con­fu­sion and un­cer­tainty, we fi­nally had a plau­si­ble the­ory for why the cor­rup­tion was oc­cur­ring, and a fix we could de­ploy to pre­vent it.

The SQLite de­vel­op­ers re­leased the fix as SQLite 3.52.0, and we pre­pared to de­ploy it as soon as it was avail­able.

Fixed, with a false alarm

We rolled out SQLite 3.52.0 care­fully—first to a few ca­nary shards, then, when we saw it run­ning smoothly, we de­ployed it to the rest of the con­trol plane.

Our backup mon­i­tor promptly turned red, and re­ported cor­rup­tion in 13 dif­fer­ent data­bases. This was ex­tremely alarm­ing, but we fol­lowed our re­cov­ery pro­ce­dures to fix all the sup­posed cor­rup­tion, and every­thing was happy. It turned out these data­bases had not suf­fered real cor­rup­tion, but were sub­ject to a sec­ond prob­lem in the ver­sion of SQLite.

We shared our er­rors with the SQLite de­vel­op­ers, which un­cov­ered a bug in SQLite re­lated to stale ex­pres­sion in­dexes. If you cre­ate an in­dex on a com­puted value, and then the com­pu­ta­tion changes, the in­dex will con­tain mis­matched val­ues, which gets re­ported as cor­rup­tion by PRAGMA in­tegri­ty_check.

In our case, we were stor­ing some high-pre­ci­sion time­stamps as text, con­vert­ing them to a float­ing-point num­ber in a VIRTUAL gen­er­ated col­umn, and the SQLite 3.52.0 re­lease that fixed our data race also made an op­ti­mi­sa­tion that sub­tly changed the round­ing be­hav­iour for text-to-float­ing-point con­ver­sions. Our ca­nary shards did­n’t have any time­stamps that trig­gered the changed round­ing be­hav­iour, so we missed this in our phased roll­out.

Because this change caused false cor­rup­tion warn­ings, the SQLite de­vel­op­ers with­drew the 3.52.0 re­lease and in­stead pub­lished 3.51.3, which only con­tained a fix for the WAL-Reset bug.

We fixed the is­sue on our side by re­duc­ing the pre­ci­sion of our time­stamps to in­te­ger sec­onds; text-to-in­te­ger con­ver­sions are un­am­bigu­ous. Meanwhile, the SQLite de­vel­op­ers cre­ated an au­to­mated, self-heal­ing in­dex fea­ture in 3.53.0, which pre­vents the stale ex­pres­sion in­dex prob­lem.

Party time!

With the fix rolled out to our en­tire con­trol plane, we were ready to de­clare vic­tory, but we were still cau­tious. An ab­sence of cor­rup­tion in­ci­dents does­n’t mean things are fixed—we’d al­ready had one six-week pe­riod of de­cep­tive calm.

We wanted pos­i­tive proof that this data race was ac­tively oc­cur­ring in our pro­duc­tion en­vi­ron­ment. Now that we un­der­stood the cause of the bug—a col­li­sion be­tween a write trans­ac­tion and a WAL-reset—we patched our SQLite dri­ver to log a warn­ing when these two op­er­a­tions over­lap. If the warn­ing fired but the data­base re­mained un­cor­rupted, we’d know the fix had saved us from a po­ten­tial cor­rup­tion in­ci­dent.

We de­ployed the warn­ing, and we waited. And we waited. And waited. And waited. As weeks slipped by, we be­gan to won­der why we did­n’t see it. Was the warn­ing bro­ken? Was our the­ory wrong? Was the true bug still lurk­ing in the dark­ness?

Then, two months later, the alert we were wait­ing for fi­nally fired:

This alert proved that the pre­cise con­di­tions for the WAL-Reset bug do oc­cur in our pro­duc­tion en­vi­ron­ment, which means it was the likely cul­prit for our six months of shaky up­time.

Since that weirdly joy­ous alert fired, we’ve run for an­other four months with­out any data­base in­ci­dents, as of this writ­ing. Finally, we could breathe a sigh of re­lief.

Off the well-trod­den path

Nobody wanted us to spend six months look­ing for bugs in SQLite. This was an im­mensely frus­trat­ing ex­pe­ri­ence for both our cus­tomers and staff, and we’re all glad to put this in­sta­bil­ity be­hind us.

This in­ves­ti­ga­tion is a use­ful re­minder: run­ning bor­ing tech­nol­ogy in a non-stan­dard way is a risk. The com­mon paths and stan­dard con­fig­u­ra­tions are in­cred­i­bly well-tested and re­li­able. Most peo­ple use SQLite in a stan­dard con­fig­u­ra­tion and never face this sort of is­sue. Everything we were do­ing was a pub­lic, doc­u­mented, sup­ported con­fig­u­ra­tion—but by tak­ing man­ual con­trol of the check­point­ing process and run­ning at our own ag­gres­sive pace, we stepped off the well-trod­den op­er­a­tional path.

Resolving these in­ci­dents was a mas­sive, cross-func­tional ef­fort in­volv­ing dozens of peo­ple—in­clud­ing Tailscale’s en­gi­neer­ing and sup­port teams, and the core main­tain­ers of SQLite. It is to all of their credit that the im­pact of these in­ci­dents was not much worse.

We know that re­peated down­time erodes trust, no mat­ter how many peo­ple are af­fected, and we’re grate­ful to our cus­tomers for their pa­tience and sup­port while we chased this down.

Frustrating as this pe­riod was, we’re left in a stronger po­si­tion than we were be­fore. The long-stand­ing bug in SQLite has been patched, and we fixed dozens of other in­ci­den­tal is­sues that we spot­ted while look­ing for it. We funded the open-source SQLite VFS shim that helped iso­late the race con­di­tion al­most im­me­di­ately, and will help track down sim­i­lar bugs in the fu­ture. Finally, we’ve re­fined our data­base backup and re­cov­ery processes, and live-tested them over a dozen times.

Hopefully there won’t be an­other data­base in­ci­dent like this—but if there is, we’ll be ready.

Today, the Mojo lan­guage of­fi­cially reaches 1.0: a mile­stone the lan­guage has been build­ing to­ward since its first re­lease in 2023. Mojo has grown into a gen­eral-pur­pose lan­guage with a vi­brant de­vel­oper com­mu­nity writ­ing their own li­braries, tools, and ap­pli­ca­tions on top of it. With Mojo 1.0, de­vel­op­ers can now build for the long-term on a sta­ble, pro­duc­tion-ready lan­guage foun­da­tion.

Mojo 1.0: A sta­ble foun­da­tion for ecosys­tem growth

Modular has rapidly evolved the Mojo lan­guage through ex­ten­sive in­ter­nal use. But that pace of progress has come with a trade­off: fre­quent changes have made it dif­fi­cult for the com­mu­nity to main­tain long-term pro­jects.

As we stated when we first an­nounced the path to Mojo 1.0, its pri­mary goal is to pro­vide a sta­ble foun­da­tion de­vel­op­ers can build on. We are mak­ing that com­mit­ment to­day be­cause Mojo is ready: it is no longer just a lan­guage we are de­vel­op­ing; it is a lan­guage we rely on every day in pro­duc­tion as the foun­da­tion of our com­mer­cial in­fra­struc­ture, MAX and Modular Cloud.

Importantly, Mojo 1.0 does not mark the end of the lan­guage’s evo­lu­tion, but it is an im­por­tant mile­stone on a longer jour­ney. During the 1.x time­frame, changes should pri­mar­ily be ad­di­tive, giv­ing de­vel­op­ers con­fi­dence that the lan­guage will not con­tin­u­ally shift be­neath them. Breaking changes may still be made, but will be man­aged with care, fol­low­ing the stan­dards of how ma­ture lan­guages (e.g. C++) evolve over time.

Yet, this mile­stone be­longs just as much to our in­cred­i­ble com­mu­nity as it does to us. Since we open-sourced the stan­dard li­brary, nearly 200 con­trib­u­tors have landed more than 1,100 pull re­quests, chang­ing over 200,000 lines of code, and more than a thou­sand oth­ers have filed is­sues that shaped the lan­guage. To every de­vel­oper who filed an is­sue, opened a pull re­quest, wrote a lan­guage pro­posal, or built a pack­age: thank you for be­ing the ar­chi­tects of this lan­guage along­side us.

Mojo im­prove­ments in 26.5

Much of this re­lease is fo­cused on com­plet­ing the work re­quired for Mojo 1.0 — a through­line across our last sev­eral re­leases as we’ve worked to make the lan­guage more con­sis­tent, pre­dictable, and ap­proach­able.

Where Mojo of­fered mul­ti­ple ways to ex­press the same idea, we’ve con­verged on one. Variables are now con­sis­tently de­clared with var, clo­sures have been uni­fied, there is a sin­gle Pointer type, and a num­ber of re­nam­ings have made the Mojo lex­i­con more pre­cise and con­sis­tent.

This re­lease com­pletes that fi­nal round of lan­guage sim­pli­fi­ca­tion and cleanup, giv­ing Mojo 1.0 the sta­ble, co­her­ent foun­da­tion we want de­vel­op­ers to be able to build on for years to come.

Beyond this foun­da­tional work, Mojo 1.0 also in­cludes sev­eral new fea­tures and im­prove­ments since the last beta re­lease:

Mojo now sup­ports Python-style “lambda” syn­tax for in­line clo­sures.

The Mojo LSP server is far more sta­ble and re­li­able, greatly im­prov­ing your every­day ex­pe­ri­ence with VS Code and other ed­i­tors.

The Mojo AI Skills are now “1.0 ready”, cov­er­ing new pro­ject cre­ation, GPU pro­gram­ming, port­ing from other lan­guages, etc.

Mojo now di­ag­noses mem­ory safety prob­lems in­volv­ing ref­er­ence in­val­i­da­tion, e.g. notic­ing when List.append in­val­i­dates a ref­er­ence into the list.

“where” clauses are more con­sis­tently used across the stan­dard li­brary, and al­low a de­scrip­tive mes­sage to make fail­ures more ac­tion­able.

These are only a few of the high­lights. See the full Mojo changelog on mo­jolang.org for the com­plete list of changes.

Where Mojo goes from here

Mojo 1.0 is a ma­jor mile­stone, but there’s so much more we are plan­ning for the lan­guage. Mojo has al­ready es­tab­lished it­self as a pow­er­ful lan­guage for writ­ing high-per­for­mance code across mod­ern CPUs, GPUs, and ac­cel­er­a­tors. The next phase of its evo­lu­tion is to broaden that foun­da­tion and make Mojo a truly great gen­eral-pur­pose sys­tems pro­gram­ming lan­guage.

That means con­tin­u­ing to in­vest in the core lan­guage and de­vel­oper ex­pe­ri­ence, with ma­jor ca­pa­bil­i­ties ahead in­clud­ing a ro­bust asyn­chro­nous pro­gram­ming model, pat­tern match­ing and unions, and much more. You can see what we are work­ing to­ward in the Mojo roadmap.

Finally, we will con­tinue to pro­gres­sively open-source more of the Mojo lan­guage, as well as com­po­nents in MAX that we have built with it. Our com­mit­ment re­mains un­changed — we will open source the Mojo com­piler and tool­chain in 2026.

MAX en­hance­ments in 26.5

While Mojo 1.0 is the high­light of this re­lease, 26.5 brings im­prove­ments to MAX, too.

Installing MAX is now eas­ier: use max[“serve”] and max[“bench­mark”] (max-serve and max-bench­mark with conda) to in­stall only the de­pen­den­cies you need, or max[“all”] to in­stall every­thing. The mod­u­lar pack­age will be re­tired in 26.6.

MAX also adds sup­port for two new model fam­i­lies: GLM-5.2 and Nemotron-H, both hy­brid Mamba-2 mod­els. And Kimi 2.5 now works with Module V3, our stream­lined model-au­thor­ing path.

Last, our col­lec­tion of open source agent skills is a great way to get started with this re­lease. We’ve used these skills in­ter­nally to speed up full model life­cy­cle bring-up, and they’ve picked up 7.2K+ down­loads through skills.sh.

For the full list of up­dates, see the MAX changelog.

Get started with 26.5 and Mojo 1.0

Install or up­grade to get started in min­utes:

bash

uv pip in­stall –upgrade mojo

uv pip in­stall max[all]

Mojo changelog

MAX changelog

mo­jolang.org

GitHub

Modular fo­rum

1.0 is just the be­gin­ning, and we’ll share more on our plans for Mojo, MAX, and open source at ModCon on August 18th in San Francisco. Tune in vir­tu­ally via the livestream or join the in-per­son wait­list.

🌑 Totality be­gins in …

📹 Reaches first we­b­cam in …

made by jonty

So while I work with po­lice de­part­ments reg­u­larly, I think it is crit­i­cally im­por­tant that tech­nol­ogy be used rea­son­ably.

While this may be off-putting to some of my clients, I worked with the Institute for Justice as an ex­pert wit­ness in their trial Schmidt v City of Norfolk. (Any opin­ions herein are my own and not those of IJ, to be clear.) The gist of that case was whether searches of his­tor­i­cally cached ALPR data (automated-license-plate-reader) con­sti­tuted an il­le­gal search.1

The judge ruled against plain­tiffs in that case. Here is a quote from the judg­ment:

Consistent with Plaintiffs’ claims in this case and con­trol­ling prece­dent in­volv­ing mass sur­veil­lance in pub­lic spaces, ALPR sur­veil­lance could be­come too in­tru­sive and run afoul of [constitutional pri­vacy stan­dards] at some point. But when? While a de­fin­i­tive an­swer to that ques­tion is elu­sive, what is read­ily ap­par­ent to this Court is that, at least in Norfolk, Virginia, the an­swer is: not to­day.

Consistent with Plaintiffs’ claims in this case and con­trol­ling prece­dent in­volv­ing mass sur­veil­lance in pub­lic spaces, ALPR sur­veil­lance could be­come too in­tru­sive and run afoul of [constitutional pri­vacy stan­dards] at some point. But when? While a de­fin­i­tive an­swer to that ques­tion is elu­sive, what is read­ily ap­par­ent to this Court is that, at least in Norfolk, Virginia, the an­swer is: not to­day.

The im­por­tant point to note about this quote is “not to­day”. This will be a long winded post, but to try to keep it sim­ple:

I think cam­eras will be­come ubiq­ui­tous in the fore­see­able fu­ture. So the ques­tion is not if this data will re­quire a war­rant, it is when. It is go­ing to hap­pen even­tu­ally un­der cur­rent case law.

I think cam­eras are good, and can be used to re­duce crime in a cost ef­fec­tive man­ner.

There is a dif­fer­ence be­tween ac­tive flags (e.g. this car is stolen and it pings the PD when it dri­ves past a cam­era) vs his­tor­i­cal searches (e.g. look to see where li­cense plate XYZ1000 was the last 30 days).

Requiring a war­rant for his­tor­i­cal searches will not se­ri­ously im­pede po­lice in­ves­ti­ga­tions.

The cur­rent sta­tus quo of not re­tain­ing data is VERY BAD; it does not pre­vent il­le­gal searches, and cur­rently lim­its the util­ity of ac­tu­ally us­ing that data for le­git­i­mate in­ves­ti­ga­tions.

Current stan­dards to pre­vent abuse of the search­ing ALPR data sys­tems are laugh­able.

Long story short in my opin­ion every­one would be bet­ter off if states just man­dated war­rant pro­ce­dures through state statutes.

Current Legal Standards

To try to not get too much into the weeds of what his­tor­i­cally con­sti­tutes a search, I think the eas­i­est place to start is via Carpenter vs US. So cur­rent US case law re­quires po­lice de­part­ments to ob­tain a war­rant to re­quest cel­lu­lar providers pro­vide law en­force­ment with cell phone tower pings (cell-site lo­ca­tion in­for­ma­tion, CSLI).

This de­vi­ated from his­tor­i­cal prece­dent in re­quir­ing a war­rant mainly be­cause it was pri­vate com­pa­nies that had the in­for­ma­tion. Before Carpenter, mostly it was ar­gued you did not have a rea­son­able ex­pec­ta­tion of pri­vacy if a pri­vate com­pany could ac­cess the same data. The court in Carpenter ba­si­cally made a de­ter­mi­na­tion that cell phone data was so com­pre­hen­sive it jus­ti­fied a dif­fer­ent stan­dard — that you could track the whole of a per­son’s move­ments with the de­tailed CSLI data. And that this level of in­va­sive­ness vi­o­lated a rea­son­able per­son’s ex­pec­ta­tion of pri­vacy. Even if Google had all that info, you did not ex­pect them to give it away.

This opin­ion was reaf­firmed with the re­cent Chatrie de­ci­sion (for ge­ofence war­rants, e.g. give me a ping for all cell phones in area X and date­time-range Y). Another rel­e­vant de­ci­sion to be aware of is also Beautiful Struggle v Baltimore, in which search­ing his­tor­i­cal aer­ial im­agery via drones also con­sti­tuted a search.

So this is why I am say­ing the ques­tion is when, not if, ALPR data will re­quire a war­rant. If a city hap­pened to have a cam­era on lit­er­ally every in­ter­sec­tion (which I think will hap­pen in the fu­ture), un­der cur­rent case law it would clearly be the same sit­u­a­tion as you have for your cell phone data.

Cameras are Good

To be brief, again I mostly work with po­lice de­part­ments in my ca­reer and was a for­mer crime an­a­lyst. I do think ALPR cam­eras are good in­vest­ments, mainly be­cause they are cheap enough to have a rea­son­able re­turn on in­vest­ment. (Note I do not think this about all po­lice tech, I am par­tic­u­larly crit­i­cal of the price tag for acoustic-gun-shot-de­tec­tion.)

So ALPRs are well un­der $3,000 per cam­era. The ma­chine learn­ing mod­els, cam­era, and com­pu­ta­tion nec­es­sary to flag a plate when it passes can eas­ily fit on cur­rent cell phones. (The harder part is pow­er­ing the phone and pro­tect­ing it from the el­e­ments.) ALPRs for the most part just take sta­tic im­ages and then ex­tract out the li­cense plate (and for some ven­dors ex­tract out ad­di­tional in­for­ma­tion, like car make and color).

The over­all ev­i­dence that ALPRs re­duce crime is pretty meh at the mo­ment (see my slides at a Wake Libertarian talk I did in 2024), but be­cause they are so cheap they re­ally only need to in­crease a few ar­rests per cam­era to likely have a pos­i­tive re­turn on in­vest­ment.

It is pretty hand-wavy, as we do not have es­ti­mates for the value of in­creased clear­ances I find per­sua­sive. But I think say­ing “I would pay $500 to help solve one case” is on the low side if any­thing. So a sin­gle cam­era if it helps catch just a hand­ful of crimes a year is likely in my opin­ion to be a pos­i­tive ROI.

I think cam­eras in all pub­lic spaces are go­ing to hap­pen. Imagine Ring comes out with a nicer cam­era sys­tem for home­own­ers that has more com­pre­hen­sive views around your house and is just as cheap. And we will ul­ti­mately be safer for it. So even for folks ad­vo­cat­ing that cities do not pay for Flock, this is com­ing any­way in the near fu­ture.

Historical Searches vs Active Flags

ALPRs have been around a long time. The first ones I worked with at Troy, NY when I was an an­a­lyst were in-car cam­eras. Basically a go pro at­tached to the win­dow that alerted when an of­fi­cer drove by a stolen plate.

While ALPRs ini­tial use was al­ways pitched as this ac­tive flag­ging of stolen ve­hi­cles, they were used right away to retroac­tively search the his­tor­i­cal lo­ca­tions of plates. They had a log of every plate, lat/​lon, and time­stamp of when that car passed a cam­era.

So imag­ine you are con­duct­ing an in­ves­ti­ga­tion of Joe Schmo, you know his li­cense plate, and then you can type in his plate and see where his car passed a cam­era. Based on this in­for­ma­tion, same as CSLI data, you can ba­si­cally trace where Joe went, where he re­peat­edly vis­ited, where he likely slept, etc. (The first time I used this at Troy, we fig­ured out a par­tic­u­lar in­di­vid­ual we were ac­tively in­ves­ti­gat­ing was liv­ing with his girl­friend for ex­am­ple. I was hon­estly amazed how densely filled in the map was of hits for a sin­gle plate based on the in-car cam­eras.)

You tech­ni­cally do not need to cache any data at all to ac­com­plish this “flag a stolen ve­hi­cle” (or any other sce­nario where you are ac­tively look­ing for a spe­cific li­cense plate). There are le­git­i­mate sce­nar­ios though where ALPR searches for re­cent data in a real time con­text can be very help­ful.

One of the more com­mon ex­am­ples — some­one robs a gas sta­tion, and they drove a ve­hi­cle. You don’t know the plate, but can look at the im­ages that passed by the fixed lo­ca­tion ALPRs in the time range, and then es­pe­cially if you have a car de­scrip­tion from the gas sta­tion at­ten­dant can fig­ure out the plate as­so­ci­ated with the ve­hi­cle.

To be clear I am not a lawyer, but in my opin­ion I think ex­i­gent cir­cum­stances make search­ing a few min­utes of cached ALPR lo­ca­tion data to­tally rea­son­able. In prac­tice, New Hampshire’s 3 minute data re­ten­tion is far too short. I could see ar­gu­ments for sev­eral hours (imagine “I found a dead body on the side of the road”, that re­quires more time for it to be re­ported.) But we are me­an­der­ing into the ter­ri­tory where it is not an ac­tive emer­gency “need to find some­one who may have a gun and hurt peo­ple” that would jus­tify those ex­i­gent cir­cum­stances. Those are the sce­nar­ios where get­ting a war­rant is rea­son­able (no dif­fer­ent than a ge­ofence war­rant if you do not have a plate and want to just search what cars passed by a cam­era within a cer­tain date-time win­dow, or no dif­fer­ent than a CSLI war­rant if you have an ac­tive sus­pect and want to search for a spe­cific li­cense plate).

Most states are re­tain­ing ALPR data for longer pe­ri­ods. While the Norfolk case was on­go­ing, Virginia set a stan­dard across the state at 21 days. Before that it was up to the in­di­vid­ual agency. It varies state by state, but states of­ten man­date data re­ten­tion around 30 days, or leave it up to the dis­cre­tion of the po­lice de­part­ment.

Deleting Data does not pre­vent abuses

These data re­ten­tion statutes are ar­gued as a mech­a­nism to pre­vent abuse. They do not ac­com­plish this.

If you look through the cases in which of­fi­cers abused the sys­tem to search for in­di­vid­u­als, all of them searched for spe­cific plates over-and-over again, some­times hun­dreds of times.

If you re­tain data for 20 days, you can just go and do a search every 20 days, keep notes on the data as you so wish, and then do an­other search 20 days later. Getting rid of old data, in-and-of-it­self, does noth­ing to pre­vent that abuse. In fact if some­one is ac­tively stalk­ing a per­son, you would ex­pect them to reg­u­larly do searches, see­ing where their vic­tim is go­ing on a reg­u­lar ba­sis while they have ac­cess to the sys­tem.

Simultaneously, delet­ing data does pre­vent its le­git­i­mate use in long term law en­force­ment in­ves­ti­ga­tions. It is to­tally nor­mal for a mur­der in­ves­ti­ga­tion to take more than 30 days to iden­tify a sus­pect. Gosh, sure would be nice to be able to then query the ALPR data to show whether a per­son was in the vicin­ity of the mur­der. Simultaneously it could be used by the de­fense for ex­cul­pa­tory pur­poses (which as­suredly would take longer than 30 days).

So folks ad­vo­cat­ing for delet­ing data as a mech­a­nism to pre­vent abuse are mak­ing things worse. It does not pre­vent abuse, and lim­its the util­ity of ALPR for his­tor­i­cal in­ves­ti­ga­tions. The only way data re­ten­tion by it­self pre­vents abuse is if you do not cache data at all (like in New Hampshire), and only use ALPRs for the ac­tive alert sit­u­a­tion.

What Smart Regulation Looks Like

One of the rea­sons I say that the cur­rent stan­dards to pre­vent abuse are laugh­able is that data re­ten­tion poli­cies and in­ter­nal PD poli­cies on when the data should be searched have been in place in most de­part­ments for years (if not a decade) at this point. The ex­am­ples where search­ing ALPR data to stalk an in­ti­mate part­ner were ob­vi­ously not pre­vented via data re­ten­tion poli­cies.

Alas, my sug­ges­tion that some data is cached for real time in­ves­ti­ga­tions (longer than 3 min­utes), and that a war­rant should be re­quired out­side of this win­dow, does not pre­vent that type of abuse ei­ther. Most de­part­ments have in place rea­sons why a search can be con­ducted, and some states have spe­cific statutes iden­ti­fy­ing im­per­mis­si­ble rea­sons for con­duct­ing searches. In the Norfolk IJ case, of­fi­cers, when en­ter­ing a rea­son for a search (which was of­ten omit­ted), some­times sup­plied rea­sons that ap­peared prima fa­cie il­le­gal, such as “protest”.

Departments, even if they have a stan­dard to do in­ter­nal au­dits, of­ten do not fol­low them. It took Tyler Dukes ask­ing Raleigh PD for their au­dit re­sults for them to even con­duct their first au­dit.

This is a long stand­ing prob­lem for PDs, not just with ALPRs, but also with search­ing crim­i­nal his­tory il­le­gally. IJ col­lat­ing a dozen cases of ar­rests of ALPR mis­use across the coun­try is not ev­i­dence these sys­tems are work­ing, as it is likely the case that only the most egre­gious abuses are ever caught.

In ad­di­tion to cre­at­ing state statutes to man­date that a war­rant be used for his­tor­i­cal ALPR searches, states should, at a min­i­mum, have clear pun­ish­ments for il­le­gal searches. These should in­clude at a min­i­mum los­ing your job, and be­ing banned from ac­cess­ing the sys­tem for­ever. When I was a crime an­a­lyst in New York (and ditto for when I worked at DCJS), this was the stan­dard for mis­us­ing the crim­i­nal his­tory search data­base.

If there is a stan­dard for just re­tain­ing ac­tive search data for less than 24 hours, it does pre­sent a po­ten­tial sim­ple check that should be flagged — if a spe­cific plate or spe­cific cam­era is searched twice within 2 days, it should be flagged to re­view more closely. Flock does have their own sys­tem to iden­tify sus­pi­cious search his­tory.

The big­ger is­sue to me though is who is do­ing the re­view­ing. It does not make sense to put this on ven­dors, and PDs just have not se­ri­ously de­voted re­sources to this, even in re­sponse to pub­lic crit­i­cism. This au­dit mech­a­nism should be del­e­gated to a third party, ei­ther a spe­cific group in the state at­tor­ney gen­er­al’s of­fice, or a state crim­i­nal jus­tice agency (like DCJS in New York).

So that of course needs to be ex­plic­itly set by state statute as well. Who is do­ing the au­dit­ing?

My fo­cus so far has been on abuses via po­lice de­part­ments them­selves, but smart reg­u­la­tion should also spec­ify au­dit­ing of the ven­dors them­selves, as well as pun­ish­ments if they fail to meet data stan­dards. (I am not think­ing so much TEMPEST at­tacks here, but more so “I left an unau­then­ti­cated end­point willy nilly on the in­ter­net”.)

Indeed, many of the re­quire­ments I am sug­gest­ing are likely al­ready on the books; the prob­lem is that the en­tity re­spon­si­ble for au­dit­ing is of­ten un­spec­i­fied or lacks the re­sources to do the work. (Also it is of­ten un­clear what the pun­ish­ments are for fail­ing to abide by statutes. That also needs to be specif­i­cally stated.)

The Future

So while I hope (although I have no ex­pec­ta­tion) that my blog post can some­how in­flu­ence cur­rent stan­dards across the coun­try, I think it is im­por­tant to keep in mind sur­veil­lance not just as the world ex­ists now, but how it may look in the fore­see­able fu­ture.

I think states should just pull the band aid off and cre­ate statutes that re­quire a war­rant to search the his­tor­i­cal ALPR data. (And this makes data shar­ing be­tween agen­cies mostly moot, the real time searches only need to be done within your own ju­ris­dic­tion.) Like I said at the be­gin­ning, the cur­rent case law on be­ing able to re­con­struct the whole of a per­son’s move­ments (which I think is quite rea­son­able) will even­tu­ally be met if the ALPR cam­eras be­come dense enough. So states can ei­ther cre­ate the statutes to dic­tate that a war­rant is nec­es­sary them­selves, or even­tu­ally have the court sys­tem thrust it upon them.

In a world filled with pri­vately owned cam­eras in pub­lic spaces, I think these sug­ges­tions are still rel­e­vant. So sim­i­lar to Carpenter for CSLI data, and Chatrie for ge­ofence war­rants, there should just be war­rant stan­dards for his­tor­i­cally search­ing any sur­veil­lance footage. There need be no spe­cial dis­tinc­tion be­tween ALPR data (public or pri­vate) or video cam­eras.

Even if the groups call­ing for the ban­ning of Flock cam­eras get their way, this does not stop pri­vate own­ers from col­lect­ing the data. So ban­ning Flock, by it­self, does not pre­vent abuse of search­ing pri­vate cam­eras. Again I think it is bet­ter to just let the gov­ern­ment re­tain the data (same as pri­vate ven­dors will re­tain the data), and have con­sis­tent war­rant stan­dards for po­lice to ob­tain that his­tor­i­cal data.

This, of course, is a bur­den to de­tec­tives. I be­lieve that trade-off in pro­tect­ing our per­sonal lib­er­ties while still al­low­ing po­lice ef­fec­tive means to in­ves­ti­gate cases is a rea­son­able one.

There are some tech­ni­cal­i­ties be­tween whether just col­lect­ing the data is a search (which was the sce­nario in the Norfolk case) or whether do­ing an ac­tive search (e.g. an of­fi­cer query­ing the sys­tem for li­cense plate ABC1234). The Norfolk case was the for­mer, but for this post I am fo­cus­ing on of­fi­cers ac­tu­ally search­ing the data (the lat­ter sce­nario).↩︎

There are some tech­ni­cal­i­ties be­tween whether just col­lect­ing the data is a search (which was the sce­nario in the Norfolk case) or whether do­ing an ac­tive search (e.g. an of­fi­cer query­ing the sys­tem for li­cense plate ABC1234). The Norfolk case was the for­mer, but for this post I am fo­cus­ing on of­fi­cers ac­tu­ally search­ing the data (the lat­ter sce­nario).↩︎

Why have I been blocked?

This web­site is us­ing a se­cu­rity ser­vice to pro­tect it­self from on­line at­tacks. The ac­tion you just per­formed trig­gered the se­cu­rity so­lu­tion. There are sev­eral ac­tions that could trig­ger this block in­clud­ing sub­mit­ting a cer­tain word or phrase, a SQL com­mand or mal­formed data.

What can I do to re­solve this?

You can email the site owner to let them know you were blocked. Please in­clude what you were do­ing when this page came up and the Cloudflare Ray ID found at the bot­tom of this page.