Over the past cou­ple weeks I’ve been do­ing agent-dri­ven re­verse en­gi­neer­ing of pe­riph­er­als that hap­pen to be within ar­m’s reach. From those de­vices, I’ve come away with a full plain­text com­mand shell in­side my mi­cro­phone, a we­b­cam whose ac­tiv­ity LED I can switch off while it records, and a key light that hands out mem­ory writes to any­one on the WiFi. Peripherals have proven to be an ideal tar­get for agen­tic RE - they’re tiny com­put­ers at­tached to my com­puter, with a data con­nec­tion to the host and usu­ally a firmware up­date mech­a­nism, so an agent has some­thing to it­er­ate against. The net out­come is bet­ter con­trol and un­der­stand­ing of my ma­chine.

My process was pretty much the same for each of these de­vices: grab a copy of the de­vice’s firmware and as­so­ci­ated up­date tool from the man­u­fac­turer, throw it into my re­verse en­gi­neer­ing en­vi­ron­ment, tell Claude Opus 5 what my goals are, and let it churn. Depending on the de­vice, the goals were some­what dif­fer­ent, but they usu­ally looked some­thing like:

In this di­rec­tory is the firmware and up­date util­ity for ___. The de­vice is also at­tached to this com­puter, and you may in­ter­act with it in non-mu­tat­ing ways. Exhaustively doc­u­ment and cross-val­i­date the en­tire firmware, in­clud­ing the fol­low­ing goals:

* re­verse en­gi­neer the firmware up­date for­mat and up­date pro­to­col * im­ple­ment our own up­date util­ity * de­ter­mine the se­cu­rity prop­er­ties of the up­date pro­to­col, in­clud­ing check­sums, sig­na­ture val­i­da­tion, se­cure boot * use sta­tic and dy­namic analy­sis to de­ter­mine all pro­to­col sur­faces and com­pletely enu­mer­ate func­tion­al­ity * find any hid­den or de­bug func­tion­al­ity in the prod­uct and how to ac­cess it

Depending on the re­sults, there were dif­fer­ent di­rec­tions of fol­low-up, but you should get the gen­eral idea. Let’s run through the list - each de­vice links to a GitHub repo full of gen­er­ated-slop docs and scripts, most of which have been val­i­dated live against real hard­ware. I’ve also in­cluded the ef­fort each de­vice took, pulled out of the Claude Code ses­sion tran­scripts. “Churn” is the time Claude was ac­tu­ally work­ing, with the long idle gaps re­moved. “Prompts from me” is every mes­sage I typed, in­clud­ing the one-word ones telling it to keep go­ing. All five de­vices to­gether came out to about 13 hours of churn and 98 prompts, spread across two weeks of evenings.

Everything I own

Insta360 Link we­b­cam

GitHub repo - 3.7 hours of Claude churn, 33 prompts from me

I use an Insta360 Link we­b­cam, which is a nice gim­baled pan-tilt-zoom cam­era that does face track­ing for au­to­mat­i­cally fram­ing the shot. I wanted to know if it was pos­si­ble to sub­vert the ac­tiv­ity LED, like in the clas­sic iSeeYou ex­ploit.

Interestingly, it was im­me­di­ately ob­vi­ous that this cam­era has a lot go­ing on in­side it. It turns out that it runs a whole RTOS (ThreadX) sourced from the up­stream SoC ven­dor, Ambarella. The RTOS hosts sev­eral small vi­sion mod­els that pro­vide things like the afore­men­tioned face track­ing, as well as ges­ture de­tec­tion for con­trol­ling set­tings. Pretty amaz­ing com­plex­ity in­side a tiny we­b­cam, but it also means there’s some ex­cit­ing at­tack sur­face here.

Over the USB Video Class in­ter­face, there’s an XU (Extension Unit) com­mand that kicks the de­vice into “mass stor­age” mode. This then lets us trans­fer a staged firmware up­date to the de­vice’s in­ter­nal FAT filesys­tem, which the de­vice then ap­plies to it­self on re­boot. This route does re­quire user in­ter­ven­tion to re­boot with a re­plug, but there’s ac­tu­ally an­other com­mand chan­nel that ex­poses ar­bi­trary read/​write of files and a re­boot com­mand over the USB ven­dor class. With this, we can fully flash the de­vice with­out any user in­ter­ac­tion. Once the firmware is in the right place, there’s ef­fec­tively no anti-tam­per, just an ap­pended MD5 hash to en­sure in­tegrity.

The in­di­ca­tor LED turns out to have a well-struc­tured set of “patterns” in the firmware that dic­tate color, blink pat­tern, etc. that are in­dexed into for var­i­ous de­vice states. I had Claude write a tool to patch out the table en­try for cam­era ac­tiv­ity, fix up the in­tegrity hash, and flash it to the cam­era. A quick test showed that the green LED that nor­mally il­lu­mi­nates while record­ing no longer turned on. Horrifying! On this de­vice, the gim­bal it­self also de­flects down when not record­ing, so it’s not com­pletely stealth, but it still does­n’t feel great.

The LED be­hav­ior be­fore and af­ter patch­ing.

ASUS ROG Swift PG42UQ mon­i­tor

GitHub repo - 1.2 hours of Claude churn, 13 prompts from me

My ASUS ROG Swift PG42UQ mon­i­tor was ac­tu­ally where I started, be­cause I got an­noyed at the pop-up over­lay that comes up every once in a while that tells me to run “pixel clean­ing”. I have never in­ten­tion­ally run pixel clean­ing on this mon­i­tor and I never will, I don’t care, and I would like for that over­lay to go away for­ever. Maybe there’s a de­bug menu or some­thing that can turn it off, or worst case we patch a branch in the firmware?

Claude found that the firmware has ef­fec­tively no pro­tec­tion what­so­ever - there’s a two-slot A/B scheme and a sim­ple check­sum, but ul­ti­mately we can write what­ever we want to the thing. Firmware up­dates run over an I2C bus bridged over USB.

The pixel clean­ing warn­ing turns out to have no na­tive way to dis­able it, and it’ll al­ways show up af­ter 8 hours of run­time. Oh well. Claude did find the ap­pro­pri­ate area to patch to kill the func­tion­al­ity though. I haven’t ac­tu­ally been brave enough to write a mod­i­fied firmware to the thing yet - it’s a pretty ex­pen­sive mon­i­tor - but I’ll get there at some point.

Another neat thing was ex­plor­ing the DDC/CI in­ter­face. This is the con­trol chan­nel avail­able over the dis­play ca­ble it­self, al­low­ing the host to change in­puts and other set­tings. I be­lieve ASUS of­fers this through their Windows util­ity, DisplayWidget, but that does lit­tle for me on Linux. So, now I have a shell script that can flip through some of the DDC/CI fea­tures like the hard­ware crosshair or zoom over­lays, FPS counter, and count­down timer. I might set up some of these on hotkeys in the fu­ture for easy ac­cess.

Shure MV7 mi­cro­phone

GitHub repo - 4.2 hours of Claude churn, 32 prompts from me

At this point, there’s less ac­tual in­cen­tive to keep pop­ping these de­vices and more just mor­bid cu­rios­ity. My mi­cro­phone, the Shure MV7, con­nects over USB and ob­vi­ously has some amount of smarts to it, with on-de­vice dig­i­tal vol­ume con­trols and such.

The firmware for this one turned out to be hid­den in­side the Windows soft­ware, MOTIV Mix, so Claude in­stalled that in Wine, found the up­date server, and pulled it down. I was­n’t on the lat­est, so there was ac­tu­ally a rea­son­able in­cen­tive here to get this work­ing just to up­date my mi­cro­phone from Linux. The firmware turned out to con­tain both DSP and MCU firmware, and was hon­estly pretty bor­ing as you might ex­pect. Again, no real se­cu­rity on the firmware flash it­self.

However, the up­date pro­to­col re­vealed that the en­tire thing ac­tu­ally runs over a USB HID ven­dor class pro­to­col that im­ple­ments a full plain­text com­mand shell, with 48 dif­fer­ent com­mands. Since it’s HID, we can ac­tu­ally hit this over WebHID from a web­page in Chrome, so I had Claude build a web in­ter­face for us­ing the shell. There’s all sorts of in­ter­est­ing set­tings in here in­clud­ing a dozen DSP knobs, ar­bi­trary mem­ory read/​write, LED con­trol, and a 4-tier user priv­i­lege sys­tem whose en­tire au­then­ti­ca­tion is a string com­par­i­son against the name of the tier you asked for. su sup just works, and the top tier can dis­able the touch panel so you can’t mute at the de­vice, and drive the mute LED in­de­pen­dently of whether the mi­cro­phone is ac­tu­ally muted. It’s the we­b­cam LED trick again, on a mi­cro­phone. Obviously, be aware that you could prob­a­bly break your de­vice if you use that UI and do some­thing stu­pid with it.

The WebHID shell in­ter­face. The DSP knobs on the left are the de­vice’s own set­tings; the con­sole on the right is the plain­text com­mand shell talk­ing over HID.

Elgato Cam Link 4K video cap­ture

GitHub repo - 1.5 hours of Claude churn, 10 prompts from me

The Elgato Cam Link 4K is just an HDMI video cap­ture de­vice, and hon­estly was just more of the same. The in­ter­est­ing thing for this one was that I let it go fully un­at­tended - I lit­er­ally kicked off the process be­fore go­ing to sleep and woke up to a tear­down and func­tion­ing firmware up­dater. The firmware con­tains an MCU im­age and an FPGA bit­stream for the ac­tual HDMI han­dling, so you could po­ten­tially do some­thing fun with the FPGA if you went deep enough into the re­verse en­gi­neer­ing there. There’s no pro­tec­tion on the firmware up­date path.

I was able to pull out all the EDID in­for­ma­tion used for ne­go­ti­at­ing video pa­ra­me­ters, so we know ex­actly what res­o­lu­tions, re­fresh rates, color spaces, and chroma sub­sam­pling op­tions are of­fered to de­vices.

The ven­dor HID pro­to­col does in­clude tun­neled ac­cess to the in­ter­nal I2C bus, which is kinda neat as you can poke the in­ter­nal HDMI re­ceiver reg­is­ters.

Elgato Key Light Mini

GitHub repo - 2.4 hours of Claude churn, 10 prompts from me

Finally, I poked at some­thing that was­n’t con­nected over USB but WiFi in­stead, the Elgato Key Light Mini. This one turned out to be way more in­ter­est­ing than I ex­pected: it’s the only one with mean­ing­ful firmware in­tegrity pro­tec­tion. Elgato signs the firmware up­dates with Ed25519 over a SHA-512 hash of the firmware pay­load, and re­jects firmware that does­n’t val­i­date. This makes sense to do, as the de­vice ba­si­cally con­nects to a WiFi net­work and then pro­vides unau­then­ti­cated ac­cess to any­one on the same net­work, so the threat model is in­her­ently dif­fer­ent.

Unfortunately, while that’s an im­prove­ment over all of the other de­vices we’ve looked at, it pro­tects the firmware at ex­actly one point in time: when an up­date is hap­pen­ing. It’s not a boot time check en­forced by the boot­loader or any other kind of se­cure boot scheme, and the up­dater hap­pens to be run­ning while every­thing else in the de­vice is still op­er­at­ing, mean­ing there’s huge at­tack sur­face to try to dis­able that sig­na­ture val­i­da­tion. I asked Claude to look for an ex­ploit that might en­able this, and it found a doozy: an HTTP POST re­quest that drops a pay­load straight into the in­ter­nal UART, which in­cludes a mem­ory poke com­mand. This means that a sin­gle HTTP POST of ATSE=0200ED94,0E001009 turns the sig­na­ture check into a no-op, and we can freely up­date to a firmware im­age with­out a le­git­i­mate sig­na­ture. I suc­cess­fully tested this with a sim­ple patch that changed the name of the de­vice, so uh, yeah, don’t put these on an un­trusted net­work.

…, owned

I have a lot of feel­ings about this whole thing. As I wrote back in March, this is in­cred­i­ble for in­ter­op­er­abil­ity and fix­ing things that don’t work how we want them to. Hardware is al­most uni­ver­sally “open” for tin­ker­ing at this point with just a cou­ple hours of mostly hands-off ma­chine-dri­ven la­bor each, and I look for­ward to a near fu­ture where I can add fea­tures to my we­b­cam firmware as eas­ily as I can to soft­ware that runs on my Linux ma­chine it­self.

On the other hand, as a se­cu­rity pro­fes­sional, this scares me for sev­eral rea­sons. I would work from the op­er­at­ing as­sump­tion that any de­vice at­tached to a com­puter could have had a ma­li­cious firmware im­plant per­formed, where pre­vi­ously that re­quired sig­nif­i­cant per-model in­vest­ment and was stereo­typed as a “state ac­tor” kind of ac­tiv­ity. Operating sys­tems aren’t re­ally equipped to work with the user to en­sure that a mi­cro­phone stays a mi­cro­phone, and does­n’t spon­ta­neously turn into a key­board that hits Win+R and drops a pay­load to steal all your data when the room is quiet enough that it can as­sume you aren’t watch­ing. And the ex­is­tence of WebUSB, WebHID, and WebBluetooth mean that for some de­vices, de­pend­ing on the specifics of which classes are used, a mo­ment of user in­dis­cre­tion in ac­cept­ing a per­mis­sions prompt could per­ma­nently back­door one of their at­tached de­vices.

Network-connected de­vices seem near uni­ver­sally fucked at this point? There are a few oth­ers I’ve poked at that I haven’t doc­u­mented here, but I’ve got­ten a root shell on a com­mer­cial Dell dis­play, and RCE on an Eaton UPS. Obviously it was never best prac­tice to let un­trusted clients touch these things, but the speed and scale at which this can be ex­e­cuted makes the risk so much higher now.

Finally, I can’t help but think about what an AI-equipped au­to­mat­i­cally-re­verse-en­gi­neer­ing worm could do to­day. It’s only a tiny leap to imag­ine that some­one could make a self-repli­cat­ing piece of mal­ware that probes its en­vi­ron­ment, re­lay­ing re­con­nais­sance back to a smart com­mand-and-con­trol that ac­tively works to push it­self into ac­ces­sories and IoT de­vices and in­dus­trial equip­ment found ad­ja­cent to an in­fected tar­get. Two things have kept this from hap­pen­ing: every de­vice model needs its own re­verse en­gi­neer­ing, and val­i­dat­ing any of it needs the hard­ware in hand. The first is the la­bor I just handed to an agent. The sec­ond is free to mal­ware al­ready sit­ting on an in­fected host. Honestly, I would­n’t be sur­prised if this al­ready ex­ists, and I think the next few years are go­ing to be ex­tremely in­ter­est­ing. 🫠

Lectronz is a mar­ket­place for open-source hard­ware mak­ers and DIY elec­tron­ics. Most of our sell­ers are not fac­to­ries or well-funded start-ups. They are en­gi­neers, in­de­pen­dent de­sign­ers, and hard­ware en­thu­si­asts work­ing from spare rooms, garages, and tiny work­shops.

Some earn a liv­ing from their prod­ucts. Some sell only a hand­ful of boards each year. Others build ten units sim­ply be­cause they cre­ated some­thing use­ful and want to share it with the com­mu­nity. Occasionally, one of those ex­per­i­ments grows into a real busi­ness. Every Arduino be­gins some­where.

But the European Union’s new pack­ag­ing rules now threaten to kill the world of mak­ers and mi­cro-en­tre­pre­neurs, putting jobs, liveli­hoods and an en­tire ecosys­tem of in­no­va­tion at risk.

And this threat is not just lim­ited to mak­ers and en­gi­neers. It af­fects artists, crafts­peo­ple and other mi­cro-en­tre­pre­neurs sell­ing their work across the EU.

A good idea, a ter­ri­ble im­ple­men­ta­tion

The EU has re­quired pro­duc­ers to take re­spon­si­bil­ity for pack­ag­ing waste for many years through Extended Producer Responsibility (EPR) schemes. The new Packaging and Packaging Waste Regulation (PPWR), which gen­er­ally ap­plies from 12 August 2026, aims to har­monise pack­ag­ing rules across the European Union and re­duce waste.

The main idea of EPR is sen­si­ble: busi­nesses that place pack­ag­ing on the mar­ket should help fi­nance its col­lec­tion and re­cy­cling.

For mak­ers, this means tak­ing re­spon­si­bil­ity for the boxes, en­velopes, plas­tic bags and other pack­ag­ing used to de­liver their prod­ucts. This is an idea we can all get be­hind.

Unfortunately, in­stead of cre­at­ing a sin­gle European sys­tem, the PPWR pre­serves a frag­mented na­tional model. A busi­ness sell­ing di­rectly to cus­tomers across the EU must reg­is­ter and ful­fil its oblig­a­tions sep­a­rately in every Member State where its pack­ag­ing be­comes waste. For large com­pa­nies, this is part of the cost of do­ing busi­ness; for mi­cro-busi­nesses sell­ing only a hand­ful of prod­ucts into each coun­try, the cost and ad­min­is­tra­tive bur­den can be wildly dis­pro­por­tion­ate to the amount of pack­ag­ing in­volved.

Imagine an en­gi­neer in Greece who de­signs a €25 open-source sen­sor board…

During the first year, he sells five to Germany, two to France, two to Austria and one to Belgium. Each ships in a small an­ti­sta­tic bag and a padded en­ve­lope. The amount of pack­ag­ing gen­er­ated for each sale is prob­a­bly around 50 grams.

He has just be­come a pack­ag­ing waste pro­ducer in four coun­tries.

Based on in­dica­tive prices cur­rently quoted by na­tional schemes and com­pli­ance providers, the an­nual cost for France alone can look like this:

Registering for a pack­ag­ing scheme, to­talling €110 in fees per year.

Using the ser­vices of an Authorised rep­re­sen­ta­tive, adding €190 to €300 in costs per year.

Spending time reg­is­ter­ing, doc­u­ment­ing, and re­port­ing waste cre­ated.

These in­dica­tive costs con­tinue to add up for each coun­try:

Belgium: €50 to €100 ad­min­is­tra­tive fees per year, plus the ser­vices of an au­tho­rised rep­re­sen­ta­tive (approx. €250 to €450).

Germany: reg­is­tra­tion is free, but pack­ag­ing-scheme par­tic­i­pa­tion starts at ap­prox­i­mately €10 per year, plus an au­tho­rised rep­re­sen­ta­tive cost­ing around €190 per year.

Austria: €250 ad­min­is­tra­tive fees per year, plus the ser­vices of an au­tho­rised rep­re­sen­ta­tive (approx. €100).

In short, the bar­rier to en­try for these four coun­tries to­tals €1150 per year in an op­ti­mistic sce­nario.

The weight-based en­vi­ron­men­tal con­tri­bu­tion as­so­ci­ated with half a kilo­gram of pack­ag­ing should be mea­sured in cents. The bu­reau­cracy re­quired to ac­count for it is mea­sured in thou­sands of eu­ros.

Now imag­ine you want to sell to all 27 Member States! To make it worth­while, our Greek en­gi­neer needs to sell not 10 boards, not 100, but lit­er­ally thou­sands of boards every year from the very start.

It sim­ply is­n’t worth it any­more.

Killing in­no­va­tion softly

Often, in­no­va­tion does­n’t come from large es­tab­lished cor­po­ra­tions, but from small busi­nesses that start from scratch with new ideas and lit­tle money. Before be­com­ing suc­cess­ful and sell­ing mil­lions of prod­ucts, many com­pa­nies started sell­ing 10, then 100, then 1000. Most busi­nesses never make it there. But there has to be space where ideas can be tested. This is one of the rea­sons Lectronz ex­ists.

In the past year, while some sell­ers on Lectronz sold hun­dreds of prod­ucts, half of our reg­is­tered sell­ers got fewer than 10 or­ders. This is not a bug, but the na­ture of a mar­ket­place like Lectronz where mak­ers are free to ex­per­i­ment with prod­uct ideas. Some ideas don’t work. Some cre­ators on Lectronz only build 10 units and share them with the com­mu­nity with­out mak­ing a profit. But even prod­ucts that “fail” have a value. When hard­ware cre­ators share them with the com­mu­nity, they help oth­ers grow as well. One piece of hard­ware may un­lock the cre­ation of an­other, lead­ing to new prod­uct ideas and in­no­va­tion.

The EPR reg­u­la­tions threaten the ex­is­tence of this in­no­v­a­tive space in the EU.

EU pol­i­cy­mak­ers keep sound­ing the alarm about Europe’s lack of in­no­va­tion, but seem hell-bent on mak­ing it as hard as pos­si­ble for in­no­va­tion to emerge at all, with reg­u­la­tions that cre­ate a dis­pro­por­tion­ate bar­rier to en­try for mi­cro-en­ter­prises and SMEs. It’s an en­vi­ron­ment where only big play­ers like Amazon, Temu, or eBay can ex­ist.

Lectronz is also a mi­cro-en­ter­prise

Lectronz col­lects a 5% fee on every trans­ac­tion it processes. We waive this fee on the first five sales to en­cour­age sell­ers to test our plat­form. After years of work, and with the re­cent surge of new sell­ers join­ing our plat­form in 2026, Lectronz now gen­er­ates roughly the equiv­a­lent of one mod­est salary.

I did not build it to be­come the next Amazon. I built it be­cause in­de­pen­dent hard­ware cre­ators de­serve a mar­ket­place de­signed for them.

If these rules force many of our sell­ers to with­draw from the European mar­ket, they could also make Lectronz it­self un­vi­able. After every­thing we have built to­gether, that would be per­son­ally heart­break­ing.

For now, Lectronz sell­ers should not ex­pect any im­me­di­ate dis­rup­tion. It re­mains un­clear how na­tional au­thor­i­ties will en­force these rules against mak­ers and mi­cro-en­ter­prises, and we will con­tinue mon­i­tor­ing the sit­u­a­tion closely.

What are the so­lu­tions?

If these reg­u­la­tions are ap­plied strictly, the short-term so­lu­tion for mak­ers is sim­ple: stop sell­ing in the EU and ship ex­clu­sively to non-EU mar­kets.

Yes, you read that right. For a French mi­cro-en­tre­pre­neur, it makes more sense to ship prod­ucts to the US than to ship to neigh­bour­ing Germany or Belgium, for ex­am­ple. This is true even with any US tar­iffs in place.

Of course, lim­it­ing sales to the US is not a vi­able so­lu­tion for some sell­ers. It’s also a loss for the European econ­omy it­self. I still hope that we can work out re­al­is­tic so­lu­tions that can help re­store the EU sin­gle mar­ket for mi­cro-en­ter­prises. Here are some ideas.

Solution #1: Introduce an EU-wide de min­imis thresh­old.

Exempt small-vol­ume sell­ers and mi­cro-en­ter­prises from cross-bor­der pack­ag­ing oblig­a­tions. The thresh­old would ap­ply only to pro­duc­ers that are be­low a spe­cific vol­ume of waste and/​or a spe­cific yearly turnover.

Solution #2: Create an EU EPR One Stop Shop.

Create a cen­tralised EU por­tal where sell­ers can reg­is­ter, re­port waste, and pay truly rea­son­able fees at once, for all Member States where they ship prod­ucts. This could mimic the mech­a­nism that al­ready ex­ists for VAT with the One Stop Shop (OSS).

Ideally, since we are in 2026, most of this work should be done through a mod­ern open RESTful API (not web forms) and open-source soft­ware, to be as au­to­mated as pos­si­ble.

Solution #3: Allow mar­ket­places to rep­re­sent and man­age mi­cro-en­ter­prises col­lec­tively as if it were a sin­gle pro­ducer.

A mech­a­nism should al­low mar­ket­places like Lectronz or Tindie to reg­is­ter, re­port waste, and pay rea­son­able fees on be­half of all their sell­ers as if they were col­lec­tively one pro­ducer of waste.

This means that the mar­ket­place would pay ad­min­is­tra­tive fees and other EPR costs cor­re­spond­ing to a sin­gle pro­ducer, that would col­lec­tively rep­re­sent all its sell­ers. For Lectronz, this would have a non-triv­ial im­pact in terms of cost and ad­min­is­tra­tive work, but it might be achiev­able un­der the right con­di­tions.

As stated above, us­ing a com­mon API stan­dard for all coun­tries would help au­to­mate things.

Make your voice heard

Again, to re­it­er­ate, we sup­port the idea of re­duc­ing waste and pro­mot­ing sus­tain­abil­ity. But there’s got to be a bet­ter, sim­pler, and fairer way to do it.

This reg­u­la­tion is hav­ing a mas­sive ef­fect on the en­tire ecosys­tem of mi­cro-busi­nesses, not just mak­ers. It af­fects artists who sell their cre­ations on­line. Local tra­di­tional food pro­duc­ers who ex­port their prod­ucts across the EU. It also af­fects crafts­peo­ple who sell their work on­line through their own web­site or ded­i­cated plat­forms like Etsy. Beyond the small world of mak­ers and DIY elec­tron­ics, this will have an im­pact on the liveli­hood of po­ten­tially hun­dreds of thou­sands of peo­ple in the EU.

And to be clear: these rules af­fect not only busi­nesses in the EU, but any busi­ness that sells to buy­ers in the EU.

Jeanette Koňarčíková, an in­de­pen­dent artist and mi­cro-en­tre­pre­neur from Slovakia, launched an on­line pe­ti­tion to draw the at­ten­tion of pol­i­cy­mak­ers to this is­sue:

https://​www.change.org/​p/​stop-de­stroy­ing-eu-mi­cro-busi­nesses-im­me­di­ate-mora­to­rium-on-cross-bor­der-epr-fees

The pe­ti­tion is thought­ful and well-writ­ten. I en­cour­age you to read and sign it!

The European Commission also has an open pub­lic feed­back page for this is­sue here:

https://​ec.eu­ropa.eu/​info/​law/​bet­ter-reg­u­la­tion/​have-your-say/​ini­tia­tives/​15352-Pack­ag­ing-and-pack­ag­ing-waste-rules-on-na­tional-reg­is­ters-of-pro­duc­er­s_en

Consider leav­ing feed­back there as well.

Recently, the European Commission has be­gun to recog­nise part of the prob­lem and has pro­posed sus­pend­ing the re­quire­ment to ap­point an au­tho­rised rep­re­sen­ta­tive in every des­ti­na­tion coun­try un­til 2035. But this pro­posal has not yet been adopted. Unfortunately, this pro­posal may take time to be voted on and en­ter into force. By then, many small busi­nesses may have closed. More im­por­tantly, re­mov­ing the au­tho­rised-rep­re­sen­ta­tive re­quire­ment would ad­dress only part of the prob­lem. Rules like this risk un­der­min­ing trust in the European pro­ject it­self. What’s the point of the EU if the sin­gle mar­ket no longer ex­ists for mi­cro-en­ter­prises?

Here at Lectronz, we will con­tinue to move for­ward and hope for the best.

But make your voice heard now to make sure pol­i­cy­mak­ers un­der­stand the ur­gency of this is­sue!

Marc Andreessen wants to shape US AI pol­icy. The ven­ture cap­i­tal firm he co-founded and runs, Andreessen Horowitz (abbreviated “a16z”), is a ma­jor player in the de­vel­op­ment of new tech star­tups.

These star­tups in­clude:

A bot farm of fake ac­counts, trick­ing peo­ple and so­cial me­dia plat­forms into think­ing AI-generated ads are posted by real peo­ple

A bot farm of fake ac­counts, trick­ing peo­ple and so­cial me­dia plat­forms into think­ing AI-generated ads are posted by real peo­ple

An AI com­pany that wants to nor­mal­ize cheat­ing on dates, job in­ter­views, and tests with AI

An AI com­pany that wants to nor­mal­ize cheat­ing on dates, job in­ter­views, and tests with AI

AI com­pan­ion apps linked to sui­cide and dis­turb­ing be­hav­ior to­ward chil­dren

AI com­pan­ion apps linked to sui­cide and dis­turb­ing be­hav­ior to­ward chil­dren

A plat­form host­ing thou­sands of deep­fake mod­els — 96% tar­get­ing iden­ti­fi­able women — that have been used to cre­ate AI-generated con­tent sex­u­al­iz­ing chil­dren

A plat­form host­ing thou­sands of deep­fake mod­els — 96% tar­get­ing iden­ti­fi­able women — that have been used to cre­ate AI-generated con­tent sex­u­al­iz­ing chil­dren

Gambling plat­forms that at­tempt to sub­vert ex­ist­ing laws and tar­get vul­ner­a­ble users

Gambling plat­forms that at­tempt to sub­vert ex­ist­ing laws and tar­get vul­ner­a­ble users

Fintech com­pa­nies im­pli­cated in fraud and il­le­gal­ity

Fintech com­pa­nies im­pli­cated in fraud and il­le­gal­ity

Many of these com­pa­nies knew the rules and broke them any­way — or de­signed prod­ucts specif­i­cally to ex­ploit gaps in con­sumer pro­tec­tion. The firms prof­ited, and the pub­lic paid the costs.

There’s a grow­ing pub­lic de­sire to rein in tech com­pa­nies and reg­u­late AI, so a16z is spend­ing tens of mil­lions of dol­lars to shape the de­vel­op­ment of AI pol­icy. The firm helped launch a $100 mil­lion su­per PAC, saw for­mer part­ners take key gov­ern­ment roles, and suc­cess­fully pushed for an ex­ec­u­tive or­der at­tempt­ing to un­der­mine state AI laws. The part­ners want to set the rules of the road, even as they’re al­ready dri­ving reck­lessly.

What fol­lows is The Midas Project’s sur­vey of 18 of Andreessen Horowitz’s most no­to­ri­ous in­vest­ments. This is­n’t a com­pre­hen­sive overview of the fir­m’s larger port­fo­lio, but it in­di­cates a pat­tern of be­hav­ior — one com­pris­ing hun­dreds of mil­lions of dol­lars of in­vest­ment by a16z.

These in­vest­ments re­veal the lines that a16z is will­ing to cross and how the lax reg­u­la­tory en­vi­ron­ment that they fa­vor would ben­e­fit the fir­m’s bot­tom line.

A16z did not re­spond to a re­quest to com­ment for this re­port.

Deception and ma­nip­u­la­tion

A16z has in­vested in prod­ucts de­signed for mass de­cep­tion. Even if these tac­tics don’t ex­plic­itly vi­o­late the law, they can be cor­ro­sive to so­ci­ety.

As tech­nol­ogy like ad­vanced AI im­proves — making it much eas­ier to fake al­most any­thing — de­ci­sion mak­ers may want to en­act new laws or poli­cies that mit­i­gate the so­cial costs. And if a16z gets its way, we might never up­date the rule­book.

Doublespeed

A16z in­vested $1 mil­lion in October 2025 via Speedrun.

Doublespeed sells the ca­pac­ity to trick every­day peo­ple, and so­cial me­dia plat­forms them­selves, into think­ing AI-generated ads are gen­uine hu­man con­tent. Here are some se­lect quotes from the com­pa­ny’s pro­mo­tional video:

“We run the only VC-backed bot farm in America. Because why let Russia and China have all the fun?”

“We run the only VC-backed bot farm in America. Because why let Russia and China have all the fun?”

“We did­n’t break the in­ter­net. It was bro­ken to be­gin with. But now we’re killing it en­tirely.”

“We did­n’t break the in­ter­net. It was bro­ken to be­gin with. But now we’re killing it en­tirely.”

“Welcome to the dead in­ter­net.”

“Welcome to the dead in­ter­net.”

A16z’s Speedrun pro­gram in­vested $1 mil­lion in Doublespeed, a com­pany that was re­cently cov­ered in a blis­ter­ing ar­ti­cle by 404 Media, which re­ported: “Andreessen Horowitz is fund­ing a com­pany that clearly vi­o­lates the in­au­then­tic be­hav­ior poli­cies of every ma­jor so­cial me­dia plat­form.”

Excerpts from Doublespeed’s web­site

The com­pa­ny’s busi­ness model re­lies on de­cep­tion, de­signed to make so­cial me­dia plat­forms and their users be­lieve AI-generated im­ages and videos de­pict real peo­ple.

How do they do this? By sell­ing ac­cess to “phone farms” that cre­ate and man­age thou­sands of fake so­cial me­dia ac­counts to ma­nip­u­late en­gage­ment met­rics. The com­pa­ny’s web­site is ex­plicit, say­ing its prod­uct “mimics” the be­hav­ior of real peo­ple on so­cial me­dia in or­der to “get our con­tent to ap­pear hu­man to the al­go­rithms.”

“Yes, we built a phone farm (and its pretty sick),” said Doublespeed founder Zuhair Lakhani on X. The pur­pose was “replacing hu­man cre­ators with ai, mainly used for mar­ket­ing.”

A photo of Doublespeed’s phone farms, shared by the founder Zuhair Lakhani on X.

They use thou­sands of real phones to pull this off be­cause so­cial me­dia plat­forms like TikTok have poli­cies against and meth­ods to de­tect the mass gen­er­a­tion and de­ploy­ment of fake ac­counts.

The com­pany has the ac­counts im­i­tate hu­man be­hav­ior be­fore post­ing de­cep­tive con­tent. This means the fake ac­counts search spe­cific key­words, scroll their “For You” pages, and use AI to an­a­lyze screen­shots of con­tent to de­ter­mine whether to “repost it, com­ment on it” or “swipe away.”

A feed of AI-generated mar­ket­ing con­tent cre­ated by Doublespeed. Source: Superwall on YouTube

A se­lec­tion of nearly iden­ti­cal Doublespeed-run TikTok ac­counts. Most posts in­volve the AI de­coy com­plain­ing about any one of a num­ber of med­ical is­sues. Then, the ac­count lists a hand­ful of cures, in­clud­ing a foam roller prod­uct from Doublespeed’s client. Source: Tiktok, Doublespeed on loom

This is all de­signed to cir­cum­vent plat­forms’ re­stric­tions on fake con­tent and then serve that fake con­tent to un­sus­pect­ing real peo­ple.

In a pod­cast in­ter­view, Lakhani of­fered de­tails about one of the com­pa­ny’s clients: “They’re hit­ting like the old per­son niche, which is what I think is like the best niche to hit with AI con­tent.”

Polling and re­search have found that older peo­ple are less likely to say they’ve heard about AI and more likely to fall for AI-generated mis­in­for­ma­tion.

Lakhani drew a par­al­lel be­tween this client and his prior work pro­duc­ing AI-generated mar­ket­ing con­tent at scale: “It was all like old per­son niche stuff. So like all sup­ple­ments that would, you know, tar­get old peo­ple, and that’s when the com­mis­sion would go crazy.”

“Those brands would tell you to do like, you know, make some like ex­tremely crazy claims,” he said, “especially with sup­ple­ments.” Lakhani added, “The sup­ple­ment stuff should def­i­nitely be like kind of il­le­gal. I don’t know how that is al­lowed.”

Despite their founder stat­ing that sup­ple­ment ads should be il­le­gal, Doublespeed is­n’t shy­ing away from them. In December 2025, a hacker gained ac­cess to Doublespeed’s en­tire back­end and the leaked data showed what the AI-generated “influencers” were ac­tu­ally sell­ing.

One ac­count, “pattyluvslife,” fea­tured an AI-generated woman claim­ing to be a UCLA stu­dent. The ac­count crit­i­cized the sup­ple­ment in­dus­try and phar­ma­ceu­ti­cal com­pa­nies as fraud­u­lent — while si­mul­ta­ne­ously pro­mot­ing a herbal sup­ple­ment from a brand called Rosabella.

Another ac­count un­der the name “chloedav1s_” had up­loaded some 200 posts fea­tur­ing an AI-generated woman claim­ing to suf­fer from var­i­ous health con­di­tions and of­ten pic­tured in a hos­pi­tal bed. She ul­ti­mately pro­moted a spe­cific com­pa­ny’s foam roller as a so­lu­tion to her ail­ments.

A tweet from DoubleSpeed’s founder shows one of the com­pa­ny’s bot ac­counts mes­sag­ing a user to pro­mote the prod­uct. In the post, Lakhani boasted, “A cou­ple of weeks ago, we gave the [AI] agents ac­cess to dm … This was for an ecom­merce brand - out of 130 dms sent, 15 pointed to a con­ver­sion.”

Another im­age from Doublespeed’s plat­form show­ing their bot ac­count, im­i­tat­ing a hu­man and mes­sag­ing users with med­ical con­di­tions to pro­mote the clien­t’s foam roller prod­uct. Source: Zuhair Lakhani on X.

The Doublespeed hack re­vealed more than 1,100 phones and over 400 TikTok ac­counts op­er­ated by the com­pany. Most of the ac­counts were pro­mot­ing prod­ucts with­out dis­clos­ing that the posts were paid ad­ver­tise­ments — a vi­o­la­tion of both TikTok’s Community Guidelines, which re­quire cre­ators to la­bel AI-generated con­tent de­pict­ing re­al­is­tic scenes, and FTC reg­u­la­tions, which re­quire in­flu­encers to clearly dis­close any “material con­nec­tion” to a brand when en­dors­ing prod­ucts.

Doublespeed and a16z did not re­spond to 404 Media’s re­quests for com­ment. After 404 Media flagged the ac­counts to TikTok, the plat­form said it added la­bels in­di­cat­ing they were AI-generated. However, The Midas Project’s fol­low-up in­ves­ti­ga­tion has re­vealed that while la­bels have been added to some con­tent from some Doublespeed-run ac­counts (including chloe­dav1s_), oth­ers with com­pa­ra­ble reach and near-iden­ti­cal con­tent still re­main un­la­beled (such as lily­w4t­son and mia.gar­c1a), with most com­menters ap­pear­ing to be­lieve the posts are au­then­tic.

Cluely AI

A16z led a $15 mil­lion Series A in June 2025.

Cluely’s of­fi­cial man­i­festo de­clares: “We want to cheat on every­thing. Yep, you heard that right. Sales calls. Meetings. Negotiations. If there’s a faster way to win — we’ll take it… So, start cheat­ing. Because when every­one does, no one is.”

Cluely’s co-founders Neel Shanmugam (left), Roy Lee (center), and Alex Chen (right). Source: Cluely via Bloomberg.

Founder and CEO Roy Lee is no stranger to us­ing AI to cheat. By his own ad­mis­sion to New York Magazine, while study­ing at Columbia, he used AI to cheat on “nearly every as­sign­ment,” es­ti­mat­ing that ChatGPT wrote 80% of every es­say he turned in. “At the end, I’d put on the fin­ish­ing touches. I’d just in­sert 20 per­cent of my hu­man­ity, my voice, into it.”

In early 2025, Lee built Interview Coder, a tool that op­er­ates be­hind-the-scenes dur­ing tech­ni­cal cod­ing in­ter­views and feeds AI-generated so­lu­tions to users in real time. He recorded him­self us­ing it to pass Amazon’s in­ter­view, re­ceived a job of­fer, pub­licly de­clined it with mock­ery, and posted the video to YouTube. He also claimed to re­ceive of­fers from TikTok, Meta, and Capital One. Amazon re­ported him to Columbia. The uni­ver­sity placed him on pro­ba­tion for “facilitation of aca­d­e­mic dis­hon­esty.”

“Even if I say ex­tremely crazy shit on­line,” Lee has ex­plained, “it will just make more peo­ple in­ter­ested in me and the com­pany and it will just drive more down­loads and con­ver­sions and get more eye­balls onto Cluely.”

A mar­ket­ing video for Cluely sug­gests that the prod­uct can be used dis­creetly to “cheat” on dates. Source: YouTube

Cluely’s launch video demon­strated an­other of the pro­duc­t’s in­tended use cases: dat­ing. In it, Lee goes on a blind date and uses the tool to lie about his age, job, and in­ter­ests. It has so far amassed 13 mil­lion views on X.

Under scrutiny, Cluely has qui­etly walked back some of its orig­i­nal po­si­tion­ing. The com­pany scrubbed ref­er­ences to cheat­ing on ex­ams and job in­ter­views from its web­site. By November, the com­pany had repo­si­tioned it­self as an AI meet­ing as­sis­tant and note­taker — en­ter­ing a crowded mar­ket far from its provoca­tive ori­gins. Lee told TechCrunch that Cluely’s “invisibility func­tion is not a core fea­ture” and that “most en­ter­prises opt to dis­able the in­vis­i­bil­ity al­to­gether be­cause of le­gal im­pli­ca­tions.” Despite Lee’s claim that in­vis­i­bil­ity is not a core fea­ture, the very first sen­tence of Cluely’s home­page ad­ver­tises the prod­uct as “undetectable.”

Cluely’s home page at time of pub­li­ca­tion. Source: Cluely

Lee’s stated goal was to “desensitize every­one to the phrase ‘cheating.’” If you say it enough, he ar­gues, “cheat be­gins to lose its mean­ing.” A16z praised Lee’s ap­proach as “rooted in de­lib­er­ate strat­egy and in­ten­tion­al­ity.”

While some com­pa­nies, like Lyft, largely ben­e­fited every­day peo­ple while break­ing rules around taxi reg­u­la­tion, Lee is in­ter­ested in break­ing some­thing more fun­da­men­tal: the shared un­der­stand­ing that ly­ing and cheat­ing is wrong.

Cluely AI an­nounced a $15 mil­lion Series A led by a16z in June 2025. Both Cluely and Doublespeed share a com­mon the­ory: that the ba­sic rules gov­ern­ing so­cial and pro­fes­sional life are ob­sta­cles to be over­come. A16z would seem to agree.

Gambling

Since a 2018 Supreme Court rul­ing, sports bet­ting has pro­lif­er­ated in the U.S. Many of the im­pacts haven’t been pretty. Researchers have found ev­i­dence that the rise of easy ac­cess to gam­bling has pushed peo­ple into greater debt, been linked to vi­o­lence, and in­creased strain on fi­nan­cially vul­ner­a­ble house­holds.

Meanwhile, a16z has in­vested in sev­eral gam­bling com­pa­nies that use reg­u­la­tory loop­holes to reach users who would oth­er­wise be pro­tected by ex­ist­ing gam­bling laws.

Coverd

A16z in­vested via Speedrun.

Coverd is pur­su­ing a novel form of gam­bling. The com­pany an­nounced its app in March 2025, invit­ing users to “bet on your bills — OnlyFans, child sup­port, and last night’s Uber. Wipe them from your credit card by play­ing your fa­vorite casino games.” The app syncs with your bank ac­counts and al­lows you to se­lect in­di­vid­ual trans­ac­tions from your credit card bill and bet against them, gam­bling to po­ten­tially win back the value of the trans­ac­tion (or, more re­al­is­ti­cally, to dou­ble your losses).

The com­pa­ny’s CEO has stated openly, “We did­n’t build Coverd to help peo­ple in­hibit their spend­ing; we built it to make spend­ing ex­cit­ing. We let spenders win twice — the sec­ond time is when they play it back and win.”

A now-deleted ad­ver­tise­ment for the Coverd app. Source: Coverd on X via Archive.is

This mar­ket­ing likely ap­peals to peo­ple who are al­ready stretched thin and des­per­ate. Many cus­tomers may be fi­nan­cially vul­ner­a­ble and will­ing to chase any way to erase ex­penses that they don’t know how to pay off.

But gam­bling is never a good ap­proach to get­ting out of debt, as the lead­er­ship at Coverd and a16z surely know. The core busi­ness model of gam­bling is based around of­fer­ing play­ers neg­a­tive ex­pected value bets, but what keeps them play­ing is that near-miss out­comes ac­ti­vate the brain’s dopamine sys­tem sim­i­larly to ac­tual wins — and gam­bling games are of­ten de­lib­er­ately de­signed to pro­duce these near-misses fre­quently. Combined with cog­ni­tive bi­ases like se­lec­tive mem­ory and the gam­bler’s fal­lacy, one study sug­gests 96% of long-term gam­blers lose money.

Nonetheless, Coverd’s app store de­scrip­tion de­scribes the prod­uct as a way to make the user more fi­nan­cially savvy, sug­gest­ing that the app will help them im­prove their fi­nan­cial health. It reads: “Coverd makes every­day fi­nance more en­gag­ing and in­ter­ac­tive! See your spend­ing habits, play games, and be­come more fi­nan­cially savvy! Win in-game to­kens as you play and stay on top of your fi­nances — all in one easy-to-use app. No pur­chase re­quired, just a fresh take on fi­nan­cial aware­ness. Download Coverd and be­come money-smart to­day!”

The home­page of the app en­cour­ages the user to link their credit card to “bring your spend­ing in­sights to the next level.” An in-app ad­ver­tise­ment for an up­com­ing Coverd-branded credit card sug­gests that users will re­ceive “up to 100% cash back” on their pur­chases.

Coverd raised $7.8 mil­lion in seed fund­ing with a16z par­tic­i­pa­tion and a16z part­ner Anish Acharya sits on the board.

Edgar

A16z in­vested via Speedrun.

The home­page for Edgar. Source: Edgar.co

How do you build a casino that’s not a casino? The com­pany Edgar, a part of a16z’s port­fo­lio, thinks it has found the an­swer in its game BettySweeps, launched in January 2025.

Edgar calls it “America’s #1 so­cial casino for slot lovers!”

This game uses a trick com­mon among sweep­stakes casi­nos — us­ing two dif­fer­ent cur­ren­cies. By mak­ing a pur­chase, play­ers re­ceive “Betty Coins” for en­ter­tain­ment, as well as a “bonus” gift of “Sweepstakes Coins” that can be gam­bled and re­deemed for cash prizes. The com­pany claims no pur­chase is nec­es­sary to play — but mul­ti­ple states have con­cluded that such mod­els con­sti­tute il­le­gal gam­bling re­gard­less.

In August 2025, Arizona’s Department of Gaming is­sued cease-and-de­sist or­ders to BettySweeps and three other sweep­stakes op­er­a­tors. The de­part­ment ac­cused them of op­er­at­ing “felony crim­i­nal en­ter­prises” and or­dered them to “desist from any fu­ture il­le­gal gam­bling op­er­a­tions or ac­tiv­i­ties of any type in Arizona.”

The com­pany ex­ited California ahead of that state’s sweep­stakes ban which took ef­fect in January 2026. BettySweeps is now re­stricted in 15 states: Arizona, California, Connecticut, Delaware, Idaho, Kentucky, Louisiana, Maryland, Michigan, Montana, Nevada, New Jersey, New York, Washington, and West Virginia.

Edgar also op­er­ates a sep­a­rate real-money on­line casino in Ontario, Canada — where it is prop­erly li­censed by the Alcohol and Gaming Commission of Ontario. The com­pany ev­i­dently knows how to ob­tain gam­bling li­censes and com­ply with reg­u­la­tions when it chooses to. In the United States, it chose a dif­fer­ent path.

Cheddr

A16z in­vested via Speedrun.

On a16z’s own Speedrun ac­cel­er­a­tor web­site, Cheddr is de­scribed as “building the TikTok of sports wa­ger­ing.”

The com­pany wants to push the fron­tier of sports bet­ting across the coun­try, tar­get­ing 46 states even though only ap­prox­i­mately 34 have le­gal­ized on­line sports bet­ting. It’s also tar­get­ing its app to users un­der age 21. To do this, the com­pany is ex­ploit­ing the same sweep­stakes law loop­hole that Edgar uses. This lets Cheddr of­fer sports bet­ting that sup­pos­edly is­n’t “gambling” in the eye of reg­u­la­tors.

The pro­mo­tional video shows users swip­ing through rapid-fire prop bets dur­ing live games; “it’s sports wa­ger­ing at the pace of a slot ma­chine,” the video says.

A now-un­listed YouTube ad for Cheddr. Source: Jason Krupat via Youtube

There are good rea­sons law­mak­ers have been re­luc­tant to open up gam­bling to 18-year-olds. Researchers have found that teenagers are roughly twice as likely as adults to de­velop gam­bling dis­or­ders.

But per­haps that’s the point. Just as cig­a­rette and al­co­hol com­pa­nies have been happy to get cus­tomers ad­dicted to their prod­ucts while young, Cheddr may be hop­ing its TikTok-style en­gage­ment me­chan­ics will start form­ing life­long gam­bling habits in their youngest users. Why else com­bine the al­ready ad­dic­tive fea­tures of TikTok with the no­to­ri­ously ad­dic­tive habit of gam­bling?

Concerns about this prod­uct have grown so se­vere that California’s Governor Newsom re­cently signed leg­is­la­tion ban­ning sweep­stakes gam­bling plat­forms such as Cheddr.

Sleeper

A16z led a $20 mil­lion Series B in May 2020 and par­tic­i­pated in a $40 mil­lion Series C in September 2021.

I have been prob­a­bly ob­sessed with two things in the last few years: Nix as a tool to ex­plore in­no­v­a­tive ideas that re­quire the ca­pa­bil­ity to re­build the world and re­plac­ing ELF with SQLite as an ex­e­cutable for­mat. You might have no­ticed that these two ideas are well suited to each other.

I ex­plored the idea dur­ing my PhD the­sis but found feed­back from oth­ers un­mo­ti­vat­ing. Radical ideas are hard to sell, as you are work­ing against the in­er­tia of the es­tab­lished so­lu­tion.

One of the end re­sults of that ex­plo­ration was sqlelf, a tool that lets you ex­plore an ELF file de­clar­a­tively us­ing SQL.11I wrote a pa­per, arXiv:2405.03883, that I failed to get pub­lished and a fol­low-up post on query­ing with it. SELECT name FROM elf_sym­bols in­stead of fid­dling with read­elf and grep. It was re­mark­ably sim­ple by lever­ag­ing vir­tual ta­bles over the ELF: how­ever I found it to be a re­fresh­ing im­prove­ment to ex­plore the ELF file for­mat. I knew how­ever that there is still some­thing much big­ger to be done.

I never let the idea go and with the re­cent im­prove­ments with LLMs, I find it com­pelling to re­visit these ideas to ex­plore fur­ther. Specifically, can we re­place ELF with SQLite as an ex­e­cutable for­mat? 🤔

Not “a data­base that de­scribes an ex­e­cutable”, but the ac­tual file you chmod +x and run.

$ file hello hello: SQLite 3.x data­base, ap­pli­ca­tion id 0x53454c46, user ver­sion 1

$ ./hello Hello, world!

$ sqlite3 hello ‘SELECT son­ame FROM ldd’ libc.so.6

I de­vel­oped a pretty fleshed out pro­to­type. It is called SELF, the Structured Executable & Linkable Format, be­cause I am un­o­rig­i­nal. It is on GitHub if you are in­ter­ested. I’m sur­prised about all the in­ter­est­ing things that fall out of this idea.

§ELF is a data­base that re­fuses to ad­mit it

Working through my PhD, I re­al­ized some­thing that bugged me. ELF is al­ready a data­base. It just im­ple­ments many data­base prim­i­tives by hand, along with a sur­pris­ing num­ber of data struc­tures for per­for­mance, like a bloom fil­ter for sym­bol lookup.

If you ever have to an­a­lyze or parse ELF, the ker­nel, ld.so, binu­tils, LIEF, gob­lin, read­elf, you are re-im­ple­ment­ing the same parser over and over again. Every pro­ducer re-im­ple­ments the same se­ri­al­izer.

The for­mat it­self is in­cred­i­bly terse, de­signed for a world where disk space and net­work band­width was at an ex­treme pre­mium. Modifying the for­mat is hard, you of­ten have to zero out sec­tions and add new ones since it is packed so tightly. There is also no self-de­scrib­ing schema. ELF it­self is a very generic for­mat that sup­ports sec­tions of data that by con­ven­tion are in­ter­preted in spe­cific ways but the for­mat does not en­force it.

SQLite is the counter-ex­am­ple. They are a self-de­scrib­ing for­mat that is ex­tremely sta­ble. It is de­signed to be ex­tended to sup­port new fea­tures with­out break­ing ex­ist­ing con­sumers and sup­port­ing a wide range of queries per­for­mantly.

If we were to re­place ELF with SQLite, what would fall out and can all of the nec­es­sary in­for­ma­tion be rep­re­sented in a SQLite data­base? The an­swer is yes, and it is sur­pris­ingly sim­ple.

§What falls away

A SELF file needs two ta­bles to run: self­_meta is the ELF header as key/​value pairs and seg­ments is the load im­age, one row per pro­gram header with the bytes in a BLOB:

CREATE TABLE seg­ments ( — orig­i­nal phdr in­dex id INTEGER PRIMARY KEY, — ‘load’ | ‘tls’ | ‘stack’ | ‘relro’ type TEXT NOT NULL, — orig­i­nal file off­set off­set INTEGER NOT NULL, vaddr INTEGER NOT NULL, filesz INTEGER NOT NULL, memsz INTEGER NOT NULL, r INTEGER, w INTEGER, x INTEGER, align INTEGER NOT NULL DEFAULT 4096, — the seg­ment bytes; NULL for pure BSS con­tent BLOB );

A sin­gle table for the sym­bol table re­places many of the ELF sec­tions and the .gnu.hash in­dex. It is a sin­gle table with a sin­gle in­dex:

CREATE TABLE sym­bols ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, — ‘GLIBC_2.2.5’ ver­sion TEXT, value INTEGER, size INTEGER, — ‘func’ | ‘object’ | ‘tls’ | … type TEXT, — ‘global’ | ‘weak’ | ‘local’ bind TEXT, de­fined INTEGER NOT NULL, ex­ported INTEGER NOT NULL ); CREATE INDEX idx_sym­bol­s_­name ON sym­bols(name, ver­sion);

Our ca­pa­bil­ity to in­clude an in­dex is equiv­a­lent to .gnu.hash and .hash in ELF, but it is a proper b-tree in­dex main­tained by SQLite in­stead of a hand-rolled bloom fil­ter.22.gnu.hash is a bloom fil­ter plus bucket chains, laid out so ld.so can re­ject a miss with­out touch­ing the chain dur­ing sym­bol dis­cov­ery.

Surprisingly a lot more falls out as well: .dynstr is gone, be­cause name is TEXT and SQLite al­ready in­terns strings, sym­bol ver­sion­ing is a col­umn, not the .gnu.version_r / .gnu.version_d con­trap­tion and there is no need for a strings table.

Other ta­bles ex­ist as well for meta­data which ex­ist for tool­ing: sec­tions, notes, dy­nam­ic_en­tries. Delete them and the pro­gram still runs, which means strip(1) is a trans­ac­tion:

# ldd(1) $ sqlite3 hello ‘SELECT son­ame FROM ldd’ libc.so.6

# nm -D –undefined $ sqlite3 hello ‘SELECT name,ver­sion FROM im­ports LIMIT 3’ __libc_start_main|GLIBC_2.34 _ITM_deregisterTMCloneTable| puts|GLIBC_2.2.5

# read­elf -l $ sqlite3 hello \ “SELECT type,vaddr,memsz,r,w,x FROM seg­ments WHERE type=‘load’” load|0|1744|1|0|0 load|4096|361|1|0|1 load|8192|312|1|0|0 load|15768|640|1|1|0

# strip(1) $ sqlite3 hello ‘DELETE FROM sec­tions; DELETE FROM notes; VACUUM;’ # 57344 -> 49152 bytes

# still runs, the op­tional ta­bles were op­tional $ ./hello Hello, world!

All the tools that op­er­ate on ELF files for read­ing, re­duce to queries over the data­base. Any tool that mod­i­fies an ELF file, like strip, can op­er­ate on the data­base within a trans­ac­tion rather than per­form­ing frag­ile off­set surgery: strip is a DELETE and VACUUM. patchelf is an UPDATE.

Any in­for­ma­tion miss­ing from the schema can be eas­ily ex­posed via a view. For ex­am­ple, ldd is a query over the needed table, which is a join of the sym­bols table with the seg­ments table to find the son­ames of the li­braries needed by the pro­gram.

CREATE VIEW ex­ports AS SELECT name, ver­sion, type, size FROM sym­bols WHERE ex­ported = 1; CREATE VIEW im­ports AS SELECT name, ver­sion FROM sym­bols WHERE de­fined = 0; CREATE VIEW ldd AS SELECT ord, son­ame FROM needed ORDER BY ord;

§How does it work?

SQLite re­serves a 4-byte ap­pli­ca­tion_id at byte off­set 68 of its header, for ex­actly this pur­pose. We stamp it SELF, so an or­di­nary SQLite data­base never matches:

$ xxd -s 64 -l 8 hello 00000040: 0000 0001 5345 4c46 ….SELF

We can now lever­age binfmt_misc, the sub­sys­tem that al­lows you to in­voke any bi­nary as if it were na­tive. We need only to reg­is­ter the magic to trig­ger on and an in­ter­preter that will in­voke our new file for­mat.

On NixOS the reg­is­tra­tion is a few lines match­ing the SQLite magic at off­set 0 and SELF at 68:

boot.binfmt.reg­is­tra­tions.self = { recog­ni­tion­Type = “magic”; off­set = 0; # bytes 0 – 15, 68 – 71 magi­cOrEx­ten­sion = “SQLite for­mat 3\\x00″ + … + “SELF”; # ig­nore the mid­dle mask = “\\xff..\\x00..\\xff”; in­ter­preter = “${self-exec}/bin/self-exec”; };

For now, I have a small tool elf2­self that con­verts an ELF file into a SELF file. It is a sim­ple post­Fixup hook you can opt into per pack­age on NixOS. The tool reads the ELF, ex­tracts the pro­gram head­ers and sym­bol table, and writes them into the SQLite data­base. We could look at ex­tend­ing gcc or ld to emit SELF di­rectly, but for now this is a sim­ple way to ex­plore the idea.

self-exec is the in­ter­preter. It is a small C pro­gram linked against lib­sqlite3. Its im­ple­men­ta­tion is re­mark­ably sim­i­lar to that of ld.so but it fetches the pro­gram head­ers and sym­bol table from the data­base in­stead of read­ing them from the ELF file. It maps the load­able seg­ments into mem­ory, re­lo­cates them, and jumps to the en­try point.

Note self-exec has to stay an ELF file. An in­ter­preter that also matches the reg­is­tra­tion re­curses straight into -ELOOP.

Note self-exec has to stay an ELF file. An in­ter­preter that also matches the reg­is­tra­tion re­curses straight into -ELOOP.

§Dynamic link­ing

Running a sta­tic pro­gram was quick and easy but bor­ing and unimag­i­na­tive. The in­ter­est­ing part is dy­namic link­ing, which is where the data­base shines.

I ex­plored two dif­fer­ent ways to do dy­namic link­ing. The first is to keep ld.so and just re­place the lookup with a SQL query via glibc rtld-au­dit in­ter­face, to quickly it­er­ate on the de­sign. The sec­ond is to re­place ld.so en­tirely with a new dy­namic linker that does the en­tire lookup and bind­ing in SQL.

glibc’s rtld-au­dit in­ter­face lets an au­dit li­brary in­ter­cept every shared ob­ject lookup (la_objsearch) be­fore any filesys­tem search hap­pens, dlopen in­cluded. The au­dit li­brary can then an­swer the ques­tion “which li­brary sat­is­fies this sym­bol?” with a SQL query in­stead of walk­ing the RUNPATH and LD_LIBRARY_PATH. Stock ld.so maps and re­lo­cates it, so the full gamut of glibc fea­tures work: lazy PLT, IFUNCs, TLS and sym­bol ver­sion­ing, while li­brary stor­age are rows and li­brary lookups are queries.

# no ELF li­brary any­where on disk $ rm lib­greet.so.1 $ ./app ./app: er­ror while load­ing shared li­braries: lib­greet.so.1: can­not open …

$ self scan –db sys­tem.db . $ SELF_SYSTEM_DB=system.db LD_AUDIT=libself-audit.so ./app Hello, world, from a SQLite li­brary!

I was cu­ri­ous what a fully SQL dy­namic linker would look like, so I pro­to­typed one. It is called self-ld and it is a small C pro­gram that im­ple­ments the dy­namic linker en­tirely in SQL. It is a proof-of-con­cept, but it works. It maps every ob­jec­t’s seg­ments, pub­lishes their ex­ports, and for each re­lo­ca­tion patches the GOT and jumps to the start.

SELECT s.value + o.load­_bias FROM re­lo­ca­tions r JOIN sym­bols s ON r.sym­bol = s.id JOIN ob­jects o ON s.ob­ject = o.id WHERE r.id = ? ORDER BY o.load­_or­der LIMIT 1;

§Cost & Benchmark

The two things that of­ten mat­ter when re­plac­ing a well-es­tab­lished for­mat are size and la­tency. How much big­ger is a SELF file than an ELF file, and how much slower is it to run?

Size. A SELF file car­ries SQLite’s b-tree over­head and lands at roughly dou­ble the ELF.

Similar to ELF bi­na­ries, most of that is re­cov­er­able, be­cause the over­head is mostly the op­tional ta­bles for de­bug­ging and tool­ing. Stripping them and delet­ing them is a trans­ac­tion. A stripped core­utils SELF is 1,794,048 B against the ELF’s 1,768,632 B, that is within 1%.

We will see though that there are in­ter­est­ing ways to amor­tise the over­head even more which I found very unique and in­ter­est­ing.

Latency. I bench­marked var­i­ous bi­na­ries from a 15 KiB hello to a 42 MiB gdb link­ing 47 li­braries:

There is a fixed ~5 ms to open SQLite and start the in­ter­preter, plus a copy pro­por­tional to the im­age. That copy is worse than it looks, be­cause the b-tree pages are not mapped into mem­ory. Two processes run­ning the same SELF bi­nary do not share text pages the way a nor­mally-mmap‘d ELF does, be­cause the bytes are copied out of the b-tree rather than mapped.33You might no­tice that curl (274 KiB, 27 li­braries) starts slower than ELF git (4.6 MiB, 5 li­braries). That is ld.so do­ing work pro­por­tional to the num­ber of ob­jects rather than the num­ber of bytes, which I have com­plained about be­fore.

§The sys­tem is a clo­sure

A SQLite data­base though need not merely be a sin­gle ex­e­cutable. It can be a clo­sure, a sin­gle file that con­tains a pro­gram and all of its tran­si­tive de­pen­den­cies. The ldd out­put of a pro­gram is am­bigu­ous: it only lists the son­ames of the li­braries it needs, not the spe­cific files that sat­isfy those needs. Nix im­proves upon this by ex­plic­itly re­solv­ing every edge to a spe­cific store path via the use of RUNPATH.44I have writ­ten about RUNPATH on Nix be­fore such as mak­ing it re­dun­dant or speed­ing it up.

We can do the same in SELF by stor­ing the re­solved path of each edge in the data­base:

CREATE TABLE ob­jects (id INTEGER PRIMARY KEY, path TEXT UNIQUE, son­ame TEXT, kind TEXT, is_­root INTEGER); CREATE TABLE needs ( ob­jec­t_id INTEGER REFERENCES ob­jects(id), ord INTEGER NOT NULL, son­ame TEXT NOT NULL, — the FK that kills am­bi­gu­ity re­solved_­path TEXT REFERENCES ob­jects(path) );

self clo­sure packs a bi­nary and its tran­si­tive de­pen­den­cies into one data­base with those edges filled in. Shared li­brary res­o­lu­tion stops be­ing a guess and be­comes a for­eign key and ldd be­comes a JOIN 🤯:

$ self clo­sure “$(readlink -f $(command -v ls))” core­utils.db ls + clo­sure -> core­utils.db

$ sqlite3 -column core­utils.db \ “SELECT n.son­ame, sub­str(n.re­solved_­path, 12, 20) FROM needs n JOIN ob­jects o ON o.id = n.ob­jec­t_id WHERE o.is_­root = 1” libgmp.so.10 rfabf­smwq02s­n94m­b3qg libacl.so.1 x0zgis­s9hdzc­sll3c­swg li­battr.so.1 08nfpyc4qhzdkc37nznv libc.so.6 8kvxvr3pmsypxiypq4g8

This sin­gle data­base is a clo­sure of the ls ex­e­cutable and its five li­braries: six ob­jects, seg­ment bytes and all, in one 4.8 MiB file. There is no son­ame am­bi­gu­ity in­side a clo­sure, be­cause a clo­sure by con­struc­tion con­tains ex­actly one provider per edge.

§How far does this go? One file, one user­land

I hope you’ve been with me so far, be­cause this is where it gets re­ally in­ter­est­ing. We can go even fur­ther and pack mul­ti­ple clo­sures into a sin­gle data­base.

I pointed self clo­sure at every ELF bi­nary on this sys­tem’s PATH: 723 ex­e­cuta­bles, which pull in 400 dis­tinct shared li­braries. 1,123 ob­jects, 346,386 sym­bols, 3,808 de­pen­dency edges, all as one SQLite file.

Turns out when you do that, the data­base is much smaller than you would ex­pect.

611.9 MiB of data­base against 644.4 MiB of ELF files. The whole user­land, as one queryable file, is smaller than the files it came from. The b-tree cost that dou­bled a sin­gle hello amor­tises to nearly noth­ing across 1,123 ob­jects and is roughly 6% over the ac­tual pro­gram bytes.

The li­braries and clo­sure are shared across the ex­e­cuta­bles very sim­i­lar to how Nix might share them across mul­ti­ple clo­sures, if the store-path was the same. If every root shipped its own pri­vate clo­sure (i.e. the AppImage model), the same 723 pro­grams would come to 5.53 GiB but the dedu­pli­ca­tion of li­braries and sym­bols falls out nat­u­rally from the data­base schema.

$ sqlite3 user­land.db \ ‘SELECT count(DIS­TINCT son­ame), count(*) FROM ob­jects WHERE son­ame IS NOT NULL’ 345|399

$ sqlite3 -column user­land.db \ ‘SELECT son­ame, count(*) FROM ob­jects WHERE son­ame IS NOT NULL GROUP BY son­ame HAVING count(*) > 1 ORDER BY 2 DESC LIMIT 4’ lib­sys­temd.so.0 3 libpthread.so.0 3 libgc­c_s.so.1 3 libc.so.6 3

$ sqlite3 user­land.db \ “SELECT count(*) FROM needs WHERE re­solved_­path IS NULL AND son­ame NOT LIKE ‘ld-%’” 4

Many com­mon id­ioms we use in ELF im­me­di­ately fall out of the data­base. For ex­am­ple, LD_PRELOAD is a row in a table rather than an en­vi­ron­ment vari­able. The pre­load table is a list of ob­jects to map last, so their ex­ports win. This means that turn­ing LD_PRELOAD on and off is a trans­ac­tion.

$ ./app.self; echo $? 13

$ sqlite3 sys­tem.db “BEGIN; CREATE TABLE pre­load(ord INTEGER PRIMARY KEY, path TEXT); INSERT INTO pre­load VALUES (0, ‘libmul.so.1.self’); COMMIT;”

# same bi­nary, no env var, no re­link $ ./app.self; echo $? 42

$ sqlite3 sys­tem.db ‘DELETE FROM pre­load;’ $ ./app.self; echo $? 13

We were able to ac­com­plish an atomic LD_PRELOAD across a whole user­land in one file, “interpose a trac­ing mal­loc every­where, then ROLLBACK” is a sin­gle trans­ac­tion. 😈

§Where it stands

The for­mat is done and round-trips be­tween ELF and SELF loss­lessly. The tool­ing is done and can query, mod­ify, and pack clo­sures. Lookup through SQL works on un­mod­i­fied glibc pro­grams per­fectly and the na­tive-SQL loader works enough to ex­plore it as a pos­si­bil­ity for ideas.

The whole thing is at fza­karia/​selfdb. nix run .#self-vm boots a NixOS VM where hello is a SQLite data­base. 🙌

Nix lets us ex­plore rad­i­cal ideas like this. We can re­build the world down to the Linux ker­nel if needed. We need not be con­strained by the ex­ist­ing de­ci­sions and con­straints of the past. We can ex­plore new ideas and see what falls out. I hope you find this idea as in­ter­est­ing as I do.

Reverse en­gi­neer­ing re­veals how Paint and Photos em­bed a server-is­sued GUID into the pix­els of lo­cally gen­er­ated AI im­ages.

TL;DR

Microsoft Paint sup­ports both lo­cal and cloud im­age gen­er­a­tion

Paint and Photos also ship lo­cal AI mod­els

The two apps send the prompt to a re­mote server for mod­er­a­tion

The server re­turns a GUID along with the mod­er­ated prompt

The GUID is em­bed­ded into the lo­cally gen­er­ated im­age as an in­vis­i­ble wa­ter­mark

A sep­a­rate vis­i­ble-wa­ter­mark set­ting does not con­trol this in­vis­i­ble wa­ter­mark

On Copilot+ PCs, im­age gen­er­a­tion is lo­cal but prompt mod­er­a­tion re­mains re­mote

Microsoft dis­closes that Paint adds C2PA meta­data to AI-generated im­ages

AI-generated im­age saves lim­ited to C2PA-preserving for­mats: PNG, JPEG, GIF, and .paint

A cu­ri­ous look at Microsoft Paint

This re­search started with my cu­rios­ity about Paint. I re­cently had some suc­cess look­ing into less-ex­plored Windows fea­tures like UCPD, WHESCVC, and I have long known that Microsoft added a bunch of AI fea­tures into the Paint app. I do not know if any­one ac­tu­ally uses Paint + AI to gen­er­ate im­ages, but I wanted to see how ex­actly the im­age gen­er­a­tion works.

Before I started, I ex­pected that it sim­ply called a re­mote API to do the im­age gen­er­a­tion. However, af­ter I set up Binary Ninja MCP with Codex and started the analy­sis, I soon re­al­ized that Microsoft ac­tu­ally shipped lo­cal mod­els in Windows as part of Copilot.

The Paint App is sit­ting in the fol­low­ing path (yes, they are all Windows Apps now):

C:\Program Files\WindowsApps\Microsoft.Paint_11.2605.71.0_x64__8wekyb3d8bbwe\PaintApp\

And there are four ap­par­ent model files with the .onnxe ex­ten­sion:

seg.on­nxe 23.1 MB in­seg_enc.on­nxe 28.0 MB in­seg_dec.on­nxe 16.5 MB mager.on­nxe 302.4 MB

The for­mat of seg.on­nxe was pre­vi­ously known, i.e., when it is XORed with the string Microsoft_2023, it be­comes a nor­mal ONNX file. However, the for­mat of the other three .onnxe files ini­tially looked dif­fer­ent.

It turned out that Microsoft had not changed the al­go­rithm, only the key. segapi.dll con­tains a small key reg­istry:

ps_enc_key.1.0.80-main -> “Microsoft_2023” ps_enc_key.1.0.81-main -> a 4,096-byte al­phanu­meric string

After de­cryp­tion, onnx.checker.check­_­model() works on all of them:

A vis­i­ble wa­ter­mark

While walk­ing through these files, I found a Watermarker.dll:

This is not su­per sur­pris­ing to me, be­cause while I in­ter­acted with the Paint app, I al­ready dis­cov­ered that it has a set­ting to em­bed a vis­i­ble wa­ter­mark to the im­age that it pro­duces:

The vis­i­ble wa­ter­mark is just a small Copilot logo at the bot­tom right of the im­age, which is to­tally nor­mal.

Then, out of nowhere, I de­cided to ask AI to an­a­lyze the DLL and see if it could also be em­bed­ding an in­vis­i­ble wa­ter­mark. This is part of my in­tu­ition as a re­verse en­gi­neer, be­cause the file is 1.67 MB in size, which is un­usu­ally large for such triv­ial func­tion­al­ity (arguably, the vis­i­ble wa­ter­mark does not even re­quire a sep­a­rate DLL). Apparently, the re­cent Claude Code text-wa­ter­mark an­nounce­ment also played a role in prompt­ing me to think about this pos­si­bil­ity.

An in­vis­i­ble wa­ter­mark

To be­gin with, the vis­i­ble wa­ter­mark is added by AddPerceptibleWatermark:

CPBDoc::Save(…) | `– per­cep­ti­ble-wa­ter­mark save helper(bitmap, WatermarkSetting) | +– WatermarkSetting::Never | `– re­turn the orig­i­nal bitmap | +– WatermarkSetting::AskEveryTime | `– show the Yes / No con­fir­ma­tion popup | +– No: re­turn the orig­i­nal bitmap | `– Yes: con­tinue | `– Always or con­firmed Yes +– Paint::AI::GetPerceptibleWatermarkSvg() `– Paint::AI::AddPerceptibleWatermark(bitmap, SVG stream) `– com­pos­ite the vis­i­ble Copilot logo

Then there is also a dif­fer­ent WmkWriteWatermark func­tion:

Watermarker.dll!WmkWriteWatermark( out­put_pix­els, pay­load, pay­load­_length, width, height, stride, in­put_pix­els, pix­el_­for­mat);

Tracing the call tree, we can see WmkWriteWatermark is called af­ter a lo­cal Stable Diffusion im­age gen­er­a­tion. And if WmkWriteWatermark fails, Paint con­verts the en­tire gen­er­a­tion into an er­ror rather than re­turn­ing the im­age with­out it:

CocreatorViewModel::GenerateImageAsync(…) | `– Paint::AI::StableDiffusionHelpers::GenerateAsync(…, wa­ter­markId, …) | `– Microsoft.ImageCreation.ImageGenerator | `– NPU-generated im­age re­sult | +– out­put safety/​mod­er­a­tion checks | +– Paint::AI::AddWatermark(bitmap, wa­ter­markId) | | | `– Watermarker.dll!WmkWriteWatermark(…) | | | +– suc­cess: re­turn the wa­ter­marked bitmap | `– fail­ure: turn gen­er­a­tion into an er­ror | `– con­struct suc­cess­ful StableDiffusionResult

Then it is nat­ural to ask what the in­com­ing pay­load ac­tu­ally is. It quickly be­comes ap­par­ent that it must be 16 bytes:

if (payload_length < 16) re­turn -6;

if (payload_length > 16) re­turn -5;

It is funny to me that the code is us­ing two dif­fer­ent er­ror codes when the pay­load is too short or too long. The func­tion then ig­nores the length pa­ra­me­ter and uses a hard-coded loop bound when it copies the pay­load:

for (size_t i = 0; i < 16; i++) mes­sage.push_back(pay­load[i]);

We do not yet know what the 16-byte pay­load is, but as we will see later, it is a GUID! WmkWriteWatermark does not em­bed the GUID di­rectly. Its wrap­per con­structs the fol­low­ing 18-byte (144-bit) mes­sage:

0x4c || GUID[0..15] || (sum of the 16 GUID bytes mod­ulo 256)

The core en­coder rounds the us­able im­age di­men­sions down to mul­ti­ples of eight and keeps 144 coun­ters, one for each bit. It re­quires every bit to be placed at least three times.

The en­coder it­self can be sum­ma­rized as:

WmkWriteWatermark(output, guid, 16, width, height, stride, in­put, for­mat) | +– val­i­date point­ers, for­mat, stride, and pay­load length +– re­quire width >= 192 and height >= 192 +– con­struct pay­load | `– 0x4c || GUID || byte-sum check­sum +– ex­pand 18 bytes into 144 in­di­vid­ual bits +– round us­able di­men­sions down to 8-pixel bound­aries +– scan/​se­lect suit­able im­age blocks +– quan­tize se­lected block/​ma­trix val­ues ac­cord­ing to each bit +– re­quire at least three suc­cess­ful place­ments per bit | | | `– in­suf­fi­cient ca­pac­ity -> re­turn -8 `– re­con­struct RGB pix­els into the out­put buffer

The em­bed­ding loop per­forms small quan­tized changes over se­lected im­age blocks. It con­tains 3-by-5 ma­trix op­er­a­tions and a ma­trix-de­com­po­si­tion rou­tine, and it uses con­stants in­clud­ing 24.0, 0.25, 0.5, and 0.2. This looks like a con­tent-adap­tive block-do­main, SVD-style wa­ter­mark.

I am not an ex­pert in im­age wa­ter­mark­ing, but one thing should be clear — this is an in­vis­i­ble wa­ter­mark! AI even wrote some code to call this func­tion di­rectly and tested it with a syn­thetic 512-by-512 BGRA im­age — 193,376 of the 262,144 pix­els changed af­ter adding the wa­ter­mark.

That led to the next ques­tion. Where does the in­put of the wa­ter­mark come from?

a GUID from re­mote prompt mod­er­a­tion

At the WmkWriteWatermark bound­ary, the pay­load is only a pointer and a length. Knowing that it must be 16 bytes was a clue, but many things can be 16 bytes. I there­fore started walk­ing back­ward through its callers. The im­me­di­ate wrap­per in PaintAIManager.dll has this sym­bol­ized sig­na­ture:

Paint::AI::AddWatermark( Gdiplus::Bitmap& im­age, winrt::guid const& wa­ter­markId);

winrt::guid, yikes! Now we know that the 16-byte wa­ter­mark pay­load is in­deed a GUID.

Further track­ing the source, we find that the GUID ac­tu­ally comes from a net­work re­quest. Before Paint runs the lo­cal im­age model, AIServices.dll sends the prompt and style to:

https://​ap­sais­er­vices-a0fqcjc6bzb­hgdcd.b02.azurefd.net/ v1/​paint-cocre­ator/​mod­er­ate-prompt

The re­quest is JSON and con­tains at least these fields:

{ “prompt”: ”…”, “style”: ”…”, “lastPromptGenerationId”: ”…” }

The re­sponse parser ex­pects:

{ “revisedPrompt”: ”…”, “promptGenerationId”: ”…”, “watermarkId”: ”…”, “containsHumanReference”: false }

Static analy­sis is nice, but at this point I wanted to see a real re­sponse from the server. I reused Paint’s own au­then­ti­cated ses­sion and sent the fol­low­ing prompt through the mod­er­a­tion end­point:

a cobalt blue cir­cle above a tiny or­ange square

The server re­turned HTTP 200:

{ “revisedPrompt”: “a cobalt blue cir­cle above a tiny or­ange square”, “promptGenerationId”: “74d9e06b-adea-43ce-85fe-186a26e2e34a”, “watermarkId”: “83424621 – 03cb-40e3 – 9808-a9fae837156d”, “containsHumanReference”: false }

I also tried the prompt a por­trait of a smil­ing per­son wear­ing a blue hat. This time the re­sponse con­tained a dif­fer­ent pair of GUIDs and con­tain­sHu­man­Ref­er­ence was true. The field is there­fore a server-side clas­si­fi­ca­tion of whether the prompt refers to a hu­man. Paint parses and stores it along­side the IDs, al­though I found no ev­i­dence that it con­trols the wa­ter­mark­ing step it­self.

ParseModerateResponse parses both ID strings as GUIDs and re­jects zero val­ues with InvalidPromptGenerationId or InvalidWatermarkId. The server’s wa­ter­markId is what be­comes part of the gen­er­ated im­age:

PaintUI.dll `– IPromptModerationService `– PaintAIManager.dll `– AIServices.dll!ModerateAsync(…) | +– build JSON | +– prompt | +– style | `– last­Prompt­Gen­er­a­tionId | +– HTTPS POST /v1/paint-cocreator/moderate-prompt | `– AIServices.dll!ParseModerateResponse(response) +– re­vised­Prompt +– prompt­Gen­er­a­tionId -> parse as GUID +– wa­ter­markId -> parse as GUID `– con­tain­sHu­man­Ref­er­ence | `– PaintUI stores WatermarkId `– StableDiffusionHelpers::GenerateAsync(…, wa­ter­markId, …) `– lo­cal Stable Diffusion re­sult `– Paint::AI::AddWatermark(bitmap, winrt::guid const&) `– WmkWriteWatermark(…, guid, 16, …) `– mod­i­fied RGB pix­els

In other words, “generated lo­cally” does not mean that the com­plete op­er­a­tion is lo­cal. Microsoft re­ceives and mod­er­ates the prompt, then is­sues the unique GUID that Paint em­beds into the lo­cally gen­er­ated im­age. Paint also sends the pre­vi­ous prompt­Gen­er­a­tionId as last­Prompt­Gen­er­a­tionId with its next mod­er­a­tion re­quest, al­low­ing suc­ces­sive re­quests to be linked ex­plic­itly.

There is an­other piece to this story. Paint does more than al­ter the pix­els. It also at­taches C2PA Content Credentials to the saved file. The code re­spon­si­ble for this lives in ProvenanceHelper.dll, backed by prove­nancesdk.dll.

For the lo­cal Stable Diffusion path, the flow looks like this:

lo­cal Stable Diffusion re­sult | +– Paint::AI::AddWatermark(bitmap, wa­ter­markId) | `– Watermarker.dll!WmkWriteWatermark(…, wa­ter­markId, 16, …) | `– AIServices.dll!SignIngredientOnlineAsync(…, prompt­Gen­er­a­tionId, im­age, …) | +– POST /v1/paint-cocreator/image-sign | +– im­ageMeta­data | | +– PromptGenerationId | | +– GenerationSeed | | +– CreativityLevel | | +– AIFVersion | | `– mod­er­a­tion scores | `– im­age­ToSign.jpg | `– ParseProvenanceResponse(…) `– server-sup­plied C2PA man­i­fest `– ProvenanceHelper::InsertManifestIngredient(…) `– AuthoringFinalizeOutputToBufferAsync(…) `– fi­nal im­age with C2PA meta­data

Notice that the sign­ing re­quest sends PromptGenerationId, while the im­age al­ready con­tains the sep­a­rately re­turned wa­ter­markId. The server as­signed both val­ues dur­ing mod­er­a­tion, so it can as­so­ci­ate the sign­ing re­quest with the wa­ter­mark al­ready pre­sent in the sub­mit­ted pix­els.

I then saved a real im­age di­rectly from Paint’s Image Creator and in­spected its PNG chunks. Immediately af­ter IHDR was an 18,979-byte caBX chunk con­tain­ing a signed C2PA man­i­fest. The in­ter­est­ing part was this:

{ “c2pa.soft-binding”: { “alg”: “com.microsoft.invismark.1”, “blocks”: [ { “scope”: “the en­tire im­age”, “value”: “83424621 – 03cb-40e3 – 9808-a9fae837156d” } ] }, “c2pa.actions.v2”: { “actions”: [ { “action”: “c2pa.watermarked”, “description”: “Content wa­ter­marked by Microsoft Responsible AI” } ] } }

Decoded into some­thing more read­able, the man­i­fest says:

Generator: Microsoft Responsible AI Provenance

AI sys­tem: Azure OpenAI ImageGen

Action: c2pa.wa­ter­marked

Algorithm: com.mi­crosoft.in­vis­mark.1

Watermark value: 83424621 – 03cb-40e3 – 9808-a9fae837156d

Description: Content wa­ter­marked by Microsoft Responsible AI

The server’s wa­ter­markId, the iden­ti­fier em­bed­ded into the pix­els, and the C2PA c2pa.soft-bind­ing.value are the same per-gen­er­a­tion value.

That re­la­tion­ship is im­por­tant. C2PA calls this a soft bind­ing: a value de­rived from, or em­bed­ded into, the con­tent so that the con­tent can still be matched with its prove­nance record af­ter the file-level man­i­fest has been re­moved. For a wa­ter­mark soft bind­ing, the value is the wa­ter­mark’s con­tent iden­ti­fier. Microsoft cryp­to­graph­i­cally signed this as­ser­tion.

Why does Paint wa­ter­mark lo­cally?

At this point, the ex­is­tence of Watermarker.dll started to make more sense. Paint ac­tu­ally has two rather dif­fer­ent gen­er­a­tion paths.

The Image Creator fea­ture I tested above uses Azure OpenAI ImageGen. Generation, wa­ter­mark­ing, and prove­nance pack­ag­ing can all hap­pen in Microsoft’s cloud, and Paint can sim­ply re­ceive a fin­ished im­age that al­ready con­tains both the in­vis­i­ble wa­ter­mark and C2PA man­i­fest:

Image Creator `– Microsoft cloud +– con­tent fil­ter­ing +– Azure OpenAI ImageGen +– in­vis­i­ble wa­ter­mark +– C2PA man­i­fest `– com­pleted im­age re­turned to Paint

Cocreator is dif­fer­ent. On a sup­ported Copilot+ PC, Microsoft says that the NPU gen­er­ates the im­age lo­cally, while Azure on­line ser­vices still per­form the safety checks. The fea­ture there­fore re­quires both a Microsoft ac­count and an in­ter­net con­nec­tion even though the ac­tual Stable Diffusion in­fer­ence runs on the de­vice:

Cocreator on a Copilot+ PC | +– prompt -> Microsoft mod­er­a­tion ser­vice | +– re­vised­Prompt | +– prompt­Gen­er­a­tionId | `– wa­ter­markId | +– re­vised­Prompt + sketch -> lo­cal NPU gen­er­a­tion | +– Watermarker.dll -> em­bed wa­ter­markId lo­cally | `– on­line prove­nance sign­ing -> fi­nal C2PA man­i­fest

This is prob­a­bly the rea­son Paint needs a lo­cal wa­ter­mark im­ple­men­ta­tion at all. A cloud gen­er­a­tor can wa­ter­mark its out­put be­fore re­turn­ing it. A lo­cal gen­er­a­tor can­not rely on that, so Paint has to al­ter the lo­cally gen­er­ated pix­els it­self. It also ex­plains why Paint treats a fail­ure from WmkWriteWatermark as a fail­ure of the en­tire gen­er­a­tion in­stead of qui­etly re­turn­ing an un­marked im­age.

There is an­other sur­pris­ingly vis­i­ble sign that Microsoft de­signed the save path around prove­nance. When I save a gen­er­ated re­sult di­rectly from the Image Creator pane, Paint of­fers ex­actly one for­mat: PNG.

After an AI re­sult is ap­plied to the Paint can­vas, the avail­able for­mats are still re­stricted to PNG, JPEG, GIF, and Paint’s own .paint for­mat. BMP—the clas­sic Paint for­mat—is con­spic­u­ously ab­sent.

This lines up with the for­mats sup­ported by C2PA. PNG stores its man­i­fest in a caBX chunk, JPEG uses one or more APP11 marker seg­ments, and GIF has its own C2PA ap­pli­ca­tion-ex­ten­sion rep­re­sen­ta­tion. The .paint for­mat is con­trolled by Microsoft and can pre­serve what­ever prove­nance state Paint re­quires. By con­trast, the C2PA spec­i­fi­ca­tion ex­plic­itly calls out BMP as a clas­sic for­mat that can­not em­bed ar­bi­trary man­i­fest data with­out us­ing an ex­ter­nal man­i­fest. If Paint al­lowed the im­age to be ex­ported di­rectly as BMP, the file-level C2PA man­i­fest would there­fore dis­ap­pear.

The split also raises an in­ter­est­ing se­cu­rity ques­tion about the cloud path. If the un­der­ly­ing re­mote im­age-gen­er­a­tion end­point can be made to re­turn the gen­er­ated im­age be­fore wa­ter­mark­ing and prove­nance pack­ag­ing—or has an in­ter­nal op­tion that sup­presses those stages—it might be pos­si­ble to ob­tain a cloud-gen­er­ated im­age with nei­ther sig­nal at­tached.

How to clas­sify such a path would de­pend en­tirely on Microsoft’s de­sign goal. It could be in­tended be­hav­ior if the un­der­ly­ing ser­vice is al­lowed to re­turn raw gen­er­a­tions and Paint is merely re­spon­si­ble for ap­ply­ing the prove­nance lay­ers. It could be a prod­uct bug if Microsoft over­looked the pos­si­bil­ity of some­one call­ing the API di­rectly and by­pass­ing Paint’s wa­ter­mark­ing step. Or it could be a se­cu­rity vul­ner­a­bil­ity if Microsoft treats wa­ter­mark­ing as a manda­tory abuse-pre­ven­tion or prove­nance con­trol and the end­point can be made to by­pass it. Without know­ing the in­tended trust bound­ary, all three pos­si­bil­i­ties re­main open.

Photos app does the same thing

While I was try­ing to lo­cate the Watermarker.dll on disk, I hap­pened to no­tice that Microsoft Photos con­tains a DLL with the same name:

C:\Program Files\WindowsApps\ Microsoft.Windows.Photos_2026.11060.2004.0_x64__8wekyb3d8bbwe\Watermarker.dll

There are also lo­cal Stable Diffusion op­er­a­tions be­hind Photos’ Image Creator and Restyle Image fea­tures. Both lead to the same wa­ter­mark wrap­per:

Photos Image Creator `– PerformSDTextToImageAndWatermarkAsync(…, prompt­Gen­er­a­tionId, …) +– run the lo­cal text-to-im­age model `– ApplyWatermark(image, prompt­Gen­er­a­tionId) +– parse prompt­Gen­er­a­tionId as a GUID +– ConvertGUIDtoContiguousByteArray() +– con­vert RGBA to ARGB +– Watermarker.dll!WmkWriteWatermark(…, guid, 16, …) `– con­vert ARGB back to RGBA

Restyle Image takes the par­al­lel path:

Photos Restyle Image `– PerformSDSketchToImageAndWatermarkAsync(…, prompt­Gen­er­a­tionId, …) `– ApplyWatermark(image, prompt­Gen­er­a­tionId) `– Watermarker.dll!WmkWriteWatermark(…, guid, 16, …)

A sub­tle dif­fer­ence be­tween Photos and Paint is fail­ure be­hav­ior. If the wa­ter­mark en­coder re­turns an er­ror, its code logs:

Published Oct 7, 2025  |  Updated Aug 23, 2026

Read by …

Update: As of August 2026, this is­sue is still pre­sent.

I was cre­at­ing a Google Workspace ac­count for my com­pany, as every­one knows, it’s a pretty straight­for­ward sign-up process, un­til I’m hit with a weird, con­fus­ing er­ror when I type my do­main name:

“Enter a valid do­main name in­stead of an email provider”

Huh? An email provider? What does this mean?

I thought maybe my do­main was pre­vi­ously used as an email provider, so I look up the his­tory of the do­main, but nope, that’s not it.

I look up the er­ror and find no of­fi­cial doc­u­men­ta­tion by Google. However, I see that I’m not the only one to have come across it, there are a few com­mu­nity threads posted by peo­ple ask­ing for help.

Believe it or not, one of those threads was by the Ukrainian Ministry of Economy; they were get­ting the same er­ror for their do­main. And of course, the so-called “experts” of the com­mu­nity pro­vide the most use­less re­sponse as al­ways:

Left with no so­lu­tion, I had to con­tact Google Workspace sup­port. After be­ing asked by mul­ti­ple agents “Have you tried to cre­ate this to an­other browser?”, when I had al­ready ex­plained to them that I did, they trans­ferred me to their high-level sup­port spe­cial­ist, Karen, who again asked me to try to sign up us­ing an­other de­vice, and I told her, again, that I have tried.

Then she asked me to record a video and wait for their prod­uct en­gi­neers to re­view it, and I did, and a few hours later I get an email:

Ok, I give it a try, and nope, the is­sue still per­sists.

One week later, I re­ceive an­other email:

So let me get this right: they are sug­gest­ing that I use an­other do­main for my com­pany be­cause their sys­tem WRONGLY flagged my le­git­i­mate do­main, and they don’t even know ex­actly why.

Just for con­text, this is a pre­mium do­main with a very high pre­mium re­newal fee, no his­tory of abuse ob­vi­ously. And the .one TLD is a le­git TLD used by le­git busi­nesses.

It seems like even their prod­uct en­gi­neers gave up and went ahead and gave a lazy re­sponse.

After re­ply­ing with an an­gry re­sponse, left with no so­lu­tion, I was about to ditch Google com­pletely and head over to Microsoft, but I thought I would give it one last try and take the mat­ter into my own hands.

I de­bug the source code of their sign-up page, and I find that the er­ror is trig­gered by a lo­cal in­put val­i­da­tion func­tion. They are check­ing the do­main against a regex list of what seems to be email providers, though look­ing closely, there are a lot of weird en­tries on the list, one of which is web\\..*, the cul­prit pre­vent­ing me from sign­ing up my do­main.

So ap­par­ently, en­gi­neers at Google de­cided that all web.[tld] do­mains are email providers…

And the en­try me\\..* pre­vents the Ukrainian Ministry of Economy’s do­main, me.gov.ua, from be­ing ac­cepted. Just be­cause its sub­do­main is me.

One other en­try is al­ice\\..*-not sure what’s up with that.

Here’s the full code snip­pet from the Google Workspace sign-up page for those in­ter­ested:

var DHa, EHa; DHa = “gmail\\..* hot­mail\\..* ya­hoo\\..* myya­hoo\\.com out­look\\..* aol\\..* in­box\\..* fast­mail\\..* mail\\..{1,3}(\\..{2})?$ red­iff­mail\\..* ly­cos\\..* att\\..* com­cast\\.net ver­i­zon\\..* talk­talk\\..* la­poste\\..* gmx\\..* web\\..* bk\\..* yan­dex\\..* ram­bler\\..* pochta\\..* live\\..* cox\\..* bell­south\\..* me\\..* earth­link\\..* sbc­global\\..* qq\\..* cez­nam\\..* 163\\..* 126\\..* free\\..* sfr\\..* wanadoo\\..* aruba\\..* terra\\..* bt­in­ter­net\\..* al­ice\\..* libero\\..* te­lenet\\..* corp\\.terra\\.com\\.co mac\\.com web­starts\\.com google­mail\\.com msn\\.com tele­worm\\.com is­b­n10\\.net is­b­n13\\.net ymail\\.com maili­na­tor\\.com pook­mail\\.com trashy­mail\\.com dodgit\\.com spam­bob\\.com dotarai\\.co\\.th apps­peo­ple\\.dk in­dia­mart\\.com pkt\\.pl naver\\.com sez­nam\\.cz hem­si­da24\\.se shaw\\.ca ncs\\.com\\.sg rogers\\.com rock­et­mail\\.com ukr\\.net char­ter\\.net uol\\.com\\.br ya\\.ru big­pond\\.com id­web­host\\.com vkios\\.com reg­is­tar\\.com cp­a­sites­o­lu­tions\\.com op­ton­line\\.net irid­i­u­minter­ac­tive\\.com data-typ­ing-earns\\.info smart­pil­lars\\.com nate\\.com list\\.ru o2\\.pl ri­aen­jolie\\.com ii-ii-ii-ii\\.com wp\\.pl ther­a­pysites\\.com mef\\.gob\\.pe the­icon­web\\.com han­mail\\.net nordicweb\\.com havealook\\.com ig\\.com\\.br skepy\\.com nameisp\\.com op­tus­net\\.com\\.au cre­ative­mo­tion­de­sign nifty\\.com ntl­world\\.com peta­matic\\.com telus\\.net msa\\.hinet\\.net sym­pa­tico\\.ca bo­janowski\\.pl bing\\.com zlat­es­tranky\\.cz xtra\\.co\\.nz sina\\.com bluewin\\.ch bol\\.com\\.br cen­trum\\.cz mind­spring\\.com juno\\.com yourha\\.com pho­to­biz\\.com aim\\.com videotron\\.ca blueyon­der\\.co\\.uk email\\.cz tis­cali\\.it tis­cali\\.co\\.uk host­ing\\.com\\.pe in\\.com tea­mu­nify\\.com sky\\.com id­ium\\.no fam­ily-ad­dress\\.ru pel­soft­labs\\.in pobox\\.com sakar­yare­hberim\\.com email\\.com iinet\\.net\\.au web­sites\\.ca globo\\.com code49\\.net xs4all\\.nl win­dowslive\\.com usa\\.net web­tel\\.in big­pond\\.net\\.au vip\\.qq\\.com road­run­ner\\.com pac­bell\\.net sohu\\.com usa\\.com i\\.ua em­bar­q­mail\\.com batesville\\.com go­b­ier­noen­linea\\.gov\\..* fox­mail\\.com in­tern­ode\\.on\\.net abv\\.bg paran\\.com t-on­line\\.de cfl\\.rr\\.com 139\\.com icloud\\.com new­mail\\.ru night­mail\\.ru nm\\.ru pop3\\.ru rbc­mail\\.ru smtp\\.ru hush­mail\\.com al­iceadsl\\.fr ar­cor\\.de cen­tu­ry­tel\\.net chello\\.nl club-in­ter­net\\.fr face­book\\.com freenet\\.de fron­tier­net\\.net het­net\\.nl home\\.nl neuf\\.fr or­ange\\.fr planet\\.nl skynet\\.be tin\\.it vir­gilio\\.it voila\\.fr wind­stream\\.net ya­hoogroups\\.com zon­net\\.nl pro­ton\\.me pro­ton­mail\\.com pro­ton­mail\\.ch pm\\.me .*\\.ocn\\.ne\\.jp ocn\\.ne\\.jp .*\\.plala\\.or\\.jp plala\\.or\\.jp .*\\.biglobe\\.ne\\.jp biglobe\\.ne\\.jp .*\\.so-net\\.ne\\.jp so-net\\.ne\\.jp”.split(” “); EHa = “att.com sfr.com terra.co in­box.kitchen al­ice.app live.space live.lk live.synxis.com live.ink”.split(” “); _.Au = func­tion(a) { re­turn EHa.includes(a) ? !1 : DHa.some(b => (b = a.match(b)) && b.in­dex === 0) } ;

The workaround

I was cu­ri­ous what would hap­pen if I dis­abled this func­tion, so I did, and I was happy to find out that once dis­abled, I was able to con­tinue the sign-up process for my do­main suc­cess­fully. That means that this was only a fron­tend val­i­da­tion, not a server-side one.

Though, I guess that the Ukrainian Ministry of Economy prob­a­bly went for Microsoft 365 in­stead.

“We see a fu­ture where in­tel­li­gence is a util­ity like elec­tric­ity or wa­ter and peo­ple buy it from us on a me­ter and use it for what­ever they want to use it for” - Sam Altman of OpenAI

“We see a fu­ture where in­tel­li­gence is a util­ity like elec­tric­ity or wa­ter and peo­ple buy it from us on a me­ter and use it for what­ever they want to use it for” - Sam Altman of OpenAI

In my pre­vi­ous ar­ti­cle, Agentic Coding is a Trap, I dis­cussed the “skilled or­ches­tra­tor para­dox”, where the skills re­quired to man­age AI agents for cod­ing are the same ones that can be di­min­ished through the con­tin­ued use of said AI agents. Expertise was largely the dif­fer­en­tia­tor; the more ex­pe­ri­enced a de­vel­oper is, the less likely it is that they might ex­pe­ri­ence skill at­ro­phy, as the knowl­edge has had a chance to os­sify af­ter years of ex­pe­ri­ence.

If you look around right now, you’ll find the vast ma­jor­ity of those that are see­ing the most ben­e­fits from these mod­els are those that have had years, if not decades, of ex­pe­ri­ence in the field (which pre­dates AI tool­ing, of course). And any in­dus­try vet­eran will tell you the same: the bedrock of this knowl­edge comes from do­ing the work.

Developers who’ve en­tered the field around the time of LLMs are placed in a po­si­tion where they don’t have the ben­e­fit of longevity, but they are be­ing guided (and some­times man­dated) to ac­cel­er­ate their ef­forts us­ing cod­ing as­sis­tants that re­quire a his­tory of ex­per­tise to wield ef­fec­tively and re­spon­si­bly.

It’s an awk­ward place to be for that de­mo­graphic, as it cre­ates a sce­nario where a novice needs ex­pert-level skills to lever­age the tools and keep pace in the in­dus­try.

The “Expert Novice”

We’re cur­rently send­ing very mixed sig­nals to peo­ple across the in­dus­try. We’re ham­mer­ing in that if you’re not us­ing AI tools, you will be “left be­hind” by your peers who are us­ing them. “AI won’t re­place you, some­one us­ing AI will” has been on re­peat since 2023.

And in the same breath, it’s also said that the way to get the best re­sults from these mod­els is to ap­ply higher-or­der think­ing; “vibe cod­ing” is a dead end; you need to “move up the stack” and cre­ate ro­bust specs, ar­chi­tect with good de­sign pat­terns, and al­ways re­view the out­puts dili­gently so you never ship some­thing you don’t un­der­stand.

The skills to do so, how­ever, are a func­tion of some­one who has ex­pe­ri­enced the fric­tion and chal­lenges over time that cul­mi­nate in “good taste”.

This leads to an­other sit­u­a­tional para­dox: If these tools de­mand ex­per­tise, yet the tools can ac­tively cir­cum­vent the fric­tion that cul­ti­vates ex­per­tise, then what is the path for one to be­come an ex­pert so they can ef­fec­tively use these tools?

Confidence with­out Comprehension

One hope is that these mod­els will end up ac­cel­er­at­ing learn­ing as they are used for code gen­er­a­tion. Junior de­vel­op­ers can work with the same grav­i­tas and con­fi­dence as in­dus­try vet­er­ans with their “personal AI tu­tor”. Knowing syn­tax is in­creas­ingly less im­por­tant, and any knowl­edge or am­bi­gu­ity gaps are filled by the AI tool. The deeper me­chan­ics of the code stay ab­stracted away, since the de­vel­oper sits higher in the stack.

JetBrains, a ma­jor player in de­vel­oper tools, re­cently cited a study ti­tled “The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers”, which painstak­ingly an­a­lyzed in­di­vid­ual be­hav­iors in live cod­ing ses­sions, and tested their abil­ity to learn cod­ing with vary­ing de­grees of AI as­sis­tance. Their main take­away was stark and coun­ter­in­tu­itive:

“Participants thought it was like hav­ing a per­sonal tu­tor. From the data in our study … we ob­served that they did not, in fact, use GenAI tools like a per­sonal tu­tor. In fact, it was quite the op­po­site.”

“Participants thought it was like hav­ing a per­sonal tu­tor. From the data in our study … we ob­served that they did not, in fact, use GenAI tools like a per­sonal tu­tor. In fact, it was quite the op­po­site.”

The par­tic­i­pants that leaned into heav­ier AI as­sis­tance:

“Often skipped cru­cial plan­ning stages, find­ing that be­cause they had­n’t rea­soned them­selves into this po­si­tion, Copilot had.”

“Finished with an ‘illusion of com­pe­tence’ rather than true un­der­stand­ing.”

Counter to that, the par­tic­i­pants that mit­i­gated their us­age of AI:

“Succeeded be­cause they had de­vel­oped ‘negative ex­per­tise’—which is ‘the abil­ity to ig­nore in­cor­rect or un­help­ful GenAI sug­ges­tions’—al­low­ing them to fo­cus on writ­ing their own so­lu­tions rather than be­ing led astray.”

“Were able to use GenAI to ac­cel­er­ate, cre­at­ing code they al­ready in­tended to make.”

The novice de­vel­op­ers who were the most un­re­stricted and con­fi­dent in their AI us­age “had skipped cru­cial steps in the pro­gram­ming prob­lem-solv­ing process, and were now lost.”

Perhaps un­sur­pris­ingly, the novice de­vel­op­ers who per­formed the best were the ones that greatly mit­i­gated or out­right ig­nored the AI cod­ing as­sis­tance.

Inverted Learning

Due to the self-di­rected na­ture of LLMs, the more ex­pe­ri­ence you have, the more ben­e­fit they pro­vide since you can ac­cu­rately steer, au­dit, and ver­ify the out­puts. The less knowl­edge you have, the more they can mis­lead you. Interacting with LLMs for learn­ing new skills takes the shape of an “inverted learn­ing” model, a role re­ver­sal where the stu­dent is ini­tially guid­ing the men­tor, the men­tor re­sponds, and then the stu­dent, again, steers the men­tor.

The process is pre­car­i­ous; LLMs are in­cred­i­bly sen­si­tive to the shape of the prompt. When you’re ex­plor­ing new do­mains, you don’t know what you don’t know, and the mal­leable and ac­com­mo­dat­ing de­sign of an LLM can lead you to be­lieve you know more than you ac­tu­ally do.

If you’re ex­plor­ing ter­ri­tory that is even some­what un­fa­mil­iar, you of­ten don’t even know the ques­tions that you need to ask that could prop­erly guide the model to pro­vid­ing the best an­swers. It be­gins to feel like a com­pass that al­ways points north, wher­ever you sug­gest north might be.

From the same study that JetBrains high­lights, even the most pre­pared stu­dents were de­railed by the AI as­sis­tance due to this type of learn­ing model: One par­tic­i­pant demon­strated good fun­da­men­tal plan­ning and habits, but sud­denly “skipped cru­cial prob­lem-solv­ing plan­ning stages, jump­ing di­rectly to cod­ing and was en­ticed by Copilot into quickly pro­duc­ing code” and had to rely on the LLM to fix the er­ror that the LLM in­tro­duced in the first place.

AI mod­els lack judg­ment, em­pa­thy, and ped­a­gog­i­cal in­tent, and the so­lu­tions pro­vided are not rooted in ex­pe­ri­ence but rather in pat­terns in the train­ing data (LLMs are, at their core, in­cred­i­bly com­plex pat­tern in­ter­po­la­tors).

The in­fi­nite an­swer ma­chine is tempt­ing, and known to be ad­dic­tive. It can un­wind rather quickly, es­pe­cially for in­ex­pe­ri­enced de­vel­op­ers. Once you get deep enough into a gen­er­ated so­lu­tion, you are of­ten be­holden to the AI tool to also fin­ish the job, cir­cum­vent­ing the prob­lem-solv­ing fric­tion that is re­quired for the for­ma­tion of a men­tal model (and to be fair, se­nior de­vel­op­ers are prone to this phe­nom­e­non, as well).

The Friction is a Feature

Expertise and mas­tery don’t hap­pen purely through ob­ser­va­tion and di­a­logue, but through ex­pe­ri­ence, rep­e­ti­tion, and trial and er­ror; you have to fail to suc­ceed. If I wanted to learn how to cook, I could watch a Master Chef work and make end­less in­quiries. After a month, I would be able to de­scribe the per­fectly medium-rare rib­eye but never know what it’s like to cook one, and I’d al­most cer­tainly over­cook it on my first at­tempt.

Coding has end­less mo­ments of trac­ing ob­scure er­rors with no log file to help, ex­pe­ri­enc­ing the sub­tle per­for­mance dif­fer­ences of cer­tain meth­ods, or hav­ing to rewrite an ap­proach when it’s clear it won’t go­ing to scale.

This ap­plied fric­tion is di­rectly what builds “developer in­tu­ition” (or “taste”). The Germans have a great word for this: Fingerspitzengefühl (fingertip feel­ing). It’s the mus­cle mem­ory that trig­gers when a de­vel­oper looks at some­thing and thinks, “yeah…this is prob­a­bly go­ing to cause prob­lems.” By avoid­ing the me­chan­ics of the strug­gle, this in­tu­ition is never built.

In UPenn’s large-scale 2025 study Generative AI with­out guardrails can harm learn­ing, they fol­lowed 1,000 stu­dents us­ing an LLM to learn math­e­mat­ics and found stu­dents used AI as a crutch and ended up per­form­ing 17% worse than stu­dents with just a text­book (and just as with the pre­vi­ous study, the stu­dents us­ing the AI as­sis­tance thought they were ex­celling).

LLMs don’t just have to gen­er­ate code, though.

If lever­aged as Socratic spar­ring part­ners in­stead of an­swer gen­er­a­tors, stud­ies have shown that “dialogic AI sys­tems can mean­ing­fully stim­u­late re­flec­tive, crit­i­cal and in­de­pen­dent think­ing”.

In that same UPenn study, they also tested a “Tutor” ver­sion by hav­ing stu­dents ask for help and then in­de­pen­dently solve the prob­lem. The GPT Tutor group per­formed an as­ton­ish­ing 127% bet­ter in the AI-assisted prac­tice ses­sion (although, in­ter­est­ingly, they scored about the same on the test as the text­book group).

This is ef­fec­tive be­cause the model is no longer be­ing uti­lized as a means of pro­duc­tion, and it shifts the cog­ni­tive work back onto the in­di­vid­ual. It’s when the fric­tion is still pre­sent that it cre­ates a last­ing im­print that leads to ex­per­tise.

Anthropic’s 2026 study “How AI as­sis­tance im­pacts the for­ma­tion of cod­ing skills” came to sim­i­lar con­clu­sions:

For novice work­ers in soft­ware en­gi­neer­ing or any other in­dus­try, our study can be viewed as a small piece of ev­i­dence to­ward the value of in­ten­tional skill de­vel­op­ment with AI tools. Cognitive ef­fort—and even get­ting painfully stuck—is likely im­por­tant for fos­ter­ing mas­tery.

For novice work­ers in soft­ware en­gi­neer­ing or any other in­dus­try, our study can be viewed as a small piece of ev­i­dence to­ward the value of in­ten­tional skill de­vel­op­ment with AI tools. Cognitive ef­fort—and even get­ting painfully stuck—is likely im­por­tant for fos­ter­ing mas­tery.

There’s a cer­tain sense of irony here: the most pro­duc­tive learn­ing that can hap­pen with an AI cod­ing tool is when it is­n’t used to gen­er­ate much of any code at all.

Pipeline Collapse

If LLMs can write code and de­bug code, and agen­tic work­flows can per­form sys­tem de­sign from the abun­dance of pat­terns in the train­ing data, then what is the pur­pose of this knowl­edge in the first place? Programming will be done en­tirely in nat­ural lan­guage, and we can dis­pense with the need to en­gage with the code be­cause the mod­els con­tinue to im­prove and fill in any knowl­edge or am­bi­gu­ity gaps. They will de­bug any is­sues that arise and man­age any com­plex­ity that they in­tro­duce.

The tril­lion-dol­lar bet that is be­ing made is: this knowl­edge won’t mat­ter, be­cause LLMs will take up the slack and ef­fec­tively be­come the new gen­er­a­tion of “developers”. It starts give off an aire of hubris that drove past no-code move­ments, and the fever dreams of CEOs, rather than the re­al­ity on the ground.

Coding/programming/software is a unique in­ter­sec­tion of logic, math, prob­lem-solv­ing, crit­i­cal think­ing, plan­ning, com­mu­ni­ca­tion, and cre­ativ­ity. LLMs can de­tect pat­terns at a scale that no hu­man ever could, but pat­terns only get you so far.

David Cramer, co-founder at Sentry (a per­for­mance and er­ror track­ing plat­form), put it suc­cinctly in a re­cent in­ter­view:

I think there’s a type of per­son … that in­her­ently be­lieves that LLM will get bet­ter enough that they will go back and fix this stuff, that it will be able to clean up all the junk that’s been stacked up along the way. I don’t think that’s true. I think it’s a sci­ence ex­per­i­ment. You want to flex that you can gen­er­ate all of your code and have hun­dreds of things go­ing in par­al­lel, I will flex and show you how bro­ken the code is 100% of the time.

I think there’s a type of per­son … that in­her­ently be­lieves that LLM will get bet­ter enough that they will go back and fix this stuff, that it will be able to clean up all the junk that’s been stacked up along the way. I don’t think that’s true. I think it’s a sci­ence ex­per­i­ment.

You want to flex that you can gen­er­ate all of your code and have hun­dreds of things go­ing in par­al­lel, I will flex and show you how bro­ken the code is 100% of the time.

Will the pipeline col­lapse, or just change?

It re­ally de­pends on whether we make the needed shift to a more ped­a­gog­i­cal us­age of these sys­tems.

By con­tin­u­ing to fo­cus on and pro­mote AI cod­ing work­flows that pri­or­i­tize code gen­er­a­tion above deep un­der­stand­ing, we are not cul­ti­vat­ing the next gen­er­a­tion of ex­per­tise who will in­herit the code that is be­ing cre­ated to­day.

My Approach: Friction First

Joel Spolsky pre­sciently writes (in 2002, no less) in his Law of Leaky Abstractions:

Code gen­er­a­tion tools which pre­tend to ab­stract out some­thing, like all ab­strac­tions, leak. And the only way to deal with the leaks com­pe­tently is to learn about how the ab­strac­tions work … the ab­strac­tions save us time work­ing, but they don’t save us time learn­ing.

Code gen­er­a­tion tools which pre­tend to ab­stract out some­thing, like all ab­strac­tions, leak. And the only way to deal with the leaks com­pe­tently is to learn about how the ab­strac­tions work … the ab­strac­tions save us time work­ing, but they don’t save us time learn­ing.

If a de­vel­oper wants to learn Java, they should prob­a­bly not start with Spring Boot. If they want to learn JavaScript fun­da­men­tals, they should not start with React. If they want to be­come highly adept at CSS, they should not start with Tailwind. LLMs could be con­sid­ered the ul­ti­mate leaky ab­strac­tion.

My ad­vice here is very sim­i­lar to my pre­vi­ous pre­scrip­tion.

If a de­vel­oper wants to be­come an ex­pert in pro­gram­ming, they should largely dis­re­gard the pure code gen­er­a­tion ca­pa­bil­i­ties of these mod­els, and in­stead use them for in­ter­ac­tive doc­u­men­ta­tion, dy­namic tu­to­r­ial gen­er­a­tors, and Socratic ex­er­cises.

It’s not a panacea, of course: Using an AI tool as a tu­tor car­ries its own risks since it is sus­cep­ti­ble to the same hal­lu­ci­na­tions as any other in­ter­ac­tions, and it can­not be re­lied upon solely as a learn­ing source. If you can’t prop­erly au­dit the ac­cu­racy of the gen­er­ated code, they you can’t au­dit the ac­cu­racy of the gen­er­ated con­cept. If you use AI as a men­tor, you must still ver­ify its out­puts against of­fi­cial doc­u­men­ta­tion, hu­man peers, and ac­tual trial and er­ror.

“Coding’s ac­tu­ally a great way to ce­ment un­der­stand­ing. The more you pro­gram, the more you un­der­stand the do­main that you’re work­ing in.” — Kent Beck, cre­ator of Test-Driven Development

“Coding’s ac­tu­ally a great way to ce­ment un­der­stand­ing. The more you pro­gram, the more you un­der­stand the do­main that you’re work­ing in.”

— Kent Beck, cre­ator of Test-Driven Development

Choosing this slower, more de­lib­er­ate path is the best way to grow ex­per­tise, but I’m aware of how hard that is when the sur­round­ing ecosys­tem is ac­tively work­ing against it. AI is be­ing man­dated (often reck­lessly) across com­pa­nies, and baked into most soft­ware de­vel­op­ment tools and IDEs as they cater largely to se­nior en­gi­neers (even with some tools like Cursor tuck­ing away the code view un­less the user specif­i­cally seeks it out). Some com­pa­nies are even forc­ing de­vel­op­ers to only use AI for all cod­ing tasks, re­gard­less of ex­pe­ri­ence level, and these com­pa­nies will have to learn their own lessons.

However, for every­one else who is look­ing to strike a bal­ance be­tween deep learn­ing (no pun) and pro­duc­tiv­ity, there are qual­i­fy­ing ques­tions you can ask to en­sure your us­age of these tools yields long-term ben­e­fits.

My AI-assistance check­list:

If I did not have ac­cess to an AI tool, could I still ac­com­plish this task?

Am I us­ing the model to deepen my un­der­stand­ing, or ex­pe­dite the an­swer?

If I had to au­dit and ver­ify the gen­er­ated out­put, could I ad­e­quately ex­plain what was hap­pen­ing?

If I’m learn­ing a new con­cept, have I done proper re­search to know the right ques­tions to ask?

Have I cross-ref­er­enced and ver­i­fied the ap­proach through other meth­ods (reading doc­u­men­ta­tion, stan­dard search tools, StackOverflow, Reddit)?

Is this a truly rote task that’s been done 100 times be­fore, or a task that re­quires ex­ec­u­tive de­ci­sion-mak­ing some­where in the process?

Even as a de­vel­oper with decades of ex­pe­ri­ence un­der my belt, I am still con­stantly re­fer­ring to them through­out my daily work, es­pe­cially when I am at­tempt­ing to learn some­thing new (which in this field, is nev­erend­ing).

The key is to de­tect the dif­fer­ence be­tween cog­ni­tive debt and cog­ni­tive of­fload­ing: Cognitive debt is ab­di­cat­ing your judg­ment and de­ci­sions, whereas cog­ni­tive of­fload­ing is del­e­gat­ing the me­chan­i­cal or te­dious.

As the Anthropic study men­tioned, get­ting “painfully stuck” is a good thing. It takes dis­ci­pline and ef­fort to not drift back to­wards just gen­er­at­ing an­swers, which might not even be ac­cu­rate in the first place. LLMs did­n’t sud­denly rewrite the fun­da­men­tals of how we learn, but they did give us a new way to do so.

Intelligence is­n’t a Commodity

The re­align­ment I hope to see over the years is the un­der­stand­ing that skills don’t de­velop with­out ac­tive par­tic­i­pa­tion. You must en­gage di­rectly and con­ti­nously to ex­pe­ri­ence the es­sen­tial fric­tion that cul­mi­nates in ex­per­tise (even if it means mov­ing more slowly).

If we stay fix­ated on lines of code and to­kens burned while the ex­per­tise pipeline dries up over the years, Sam Altman’s vi­sion of sell­ing in­tel­li­gence back to us on a me­ter could be­come re­al­ity. Domain knowl­edge could be­come very hard to come by, and when one sits down to do any type of de­vel­op­ment work, there will be a pang of paral­y­sis if that per­son does not have an ac­tive AI tool sub­scrip­tion at their side.

LLMs are a sta­tic data­base of skills. They are in­ter­po­la­tion en­gines. Software en­gi­neer­ing, how­ever, is an ex­er­cise in adap­ta­tion and novel prob­lem-solv­ing. You can­not in­ter­po­late your way through a com­pletely unique sys­tem fail­ure. — François Chollet, cre­ator of ARC-AGI Benchmark

LLMs are a sta­tic data­base of skills. They are in­ter­po­la­tion en­gines. Software en­gi­neer­ing, how­ever, is an ex­er­cise in adap­ta­tion and novel prob­lem-solv­ing. You can­not in­ter­po­late your way through a com­pletely unique sys­tem fail­ure.

— François Chollet, cre­ator of ARC-AGI Benchmark

Meet Varkos, a some­what good boy

A real-time in­tel­li­gent gam­ing com­pan­ion that ac­tu­ally plays along­side you. World agency, lo­cal in­fer­ence, per­sis­tent plans, and an AI char­ac­ter de­signed to cre­ate a strik­ing emo­tional ex­pe­ri­ence. (Currently run­ning in Skyrim, but by de­sign plug­gable any­where, stand­alone too)

See it in ac­tion.

:: Contents ::

The goal

Complex com­mands

Combat

Personality evo­lu­tion

Dog in and out of the game - Void mode

How it is built

A fi­nal word

The goal

Simple: Let’s build a su­per-charged next-level gam­ing com­pan­ion that ac­tu­ally feels good.

There are al­ready mul­ti­ple frame­works that let LLMs con­trol NPC di­a­logue. They are fan­tas­tic for role-play­ing and stay­ing in char­ac­ter, but they have two re­cur­ring prob­lems: weak world agency and la­tency. Good at talk­ing but far less re­li­able at per­form­ing ac­tions and ter­ri­ble at com­plex in­struc­tion sets. You may have also no­ticed how pop­u­lar AI NPC demos of­ten cut be­tween the player speak­ing and the AI re­ply­ing, try­ing to mask la­tency. Can we do bet­ter?

I wanted a com­pan­ion that:

Useful and in­stant. It should fight, fetch, loot, in­spect, carry and give items etc etc, fol­low com­plex multi-step in­struc­tions with­out feel­ing buggy or ex­per­i­men­tal. This mat­ters es­pe­cially in VR, where nav­i­gat­ing menus is cum­ber­some and im­mer­sion-break­ing. it needs to be FAST fast, not just fast

Alive and pre­sent. It should have a fun, en­dear­ing per­son­al­ity, not canned ro­botic pre-writ­ten re­sponses. Remember shared ex­pe­ri­ences and change over time. The mi­cro­phone stays ac­tive while a ses­sion is run­ning: you do not sum­mon Varkos through a di­a­logue menu, you talk to him. When im­mer­sion kicks in, it should feel like you are not play­ing alone.

Local and pri­vate wher­ever prac­ti­cal. The ele­phant in the room is that cloud LLM calls can get quite pricy (especially with multi-thou­sand-to­ken LLM calls) and the added la­tency can be an ex­pe­ri­ence killer. And why turn a pri­vate sin­gle-player game into a me­tered and sur­veilled ex­pe­ri­ence? Let’s try to give as much con­trol to the user as pos­si­ble (bonus it’s a fun tech­ni­cal chal­lenge).

Basically: a sin­gle-player game where you are not play­ing alone.

Complex com­mands

Varkos can han­dle com­mands that ex­tend be­yond one im­me­di­ate ac­tion. Plans can wait for events, pre­serve tar­gets be­tween steps, mon­i­tor progress and re­pair or stop when world state changes. Nothing is pre-scripted.

Let’s see some ex­am­ples

Varkos re­ceives a con­di­tional in­struc­tion in­volv­ing the next ar­row. He reg­is­ters the fu­ture trig­ger in­stead of act­ing im­me­di­ately, waits for the cor­re­lated pro­jec­tile im­pact and then con­tin­ues the plan.

Long-form multi-step com­mand “I want you to wait here and I’m gonna go over there. Once you see the sig­nal, the sig­nal is go­ing to be an ar­row I fire up in the sky, I want you to pick up this po­tion and come and bring it to me. Okay?”

Long-form multi-step com­mand

“I want you to wait here and I’m gonna go over there. Once you see the sig­nal, the sig­nal is go­ing to be an ar­row I fire up in the sky, I want you to pick up this po­tion and come and bring it to me. Okay?”

Item search

Varkos can search the grounded world state for a re­quested item, iden­tify where it is and re­spond us­ing what is ac­tu­ally pre­sent in the game.

“Do you see the cer­e­mo­nial sword any­where?” Varkos picks up a dif­fer­ent sword and brings it to us. We tell him that’s not the one, then he of­fers to be on the look­out.

“Do you see the cer­e­mo­nial sword any­where?”

Varkos picks up a dif­fer­ent sword and brings it to us. We tell him that’s not the one, then he of­fers to be on the look­out.

Hide-and-seek

Hide-and-seek is not a sin­gle API call. It be­comes a per­sis­tent goal with move­ment, wait­ing, mon­i­tor­ing and com­ple­tion con­di­tions.

“Let’s play hide-and-seek again. You wait here and I’m gonna go hide, then count to ten and come and try to find me.”

“Let’s play hide-and-seek again. You wait here and I’m gonna go hide, then count to ten and come and try to find me.”

Loot this chest and give me the po­tion

This com­bines a grounded con­tainer, a fil­tered loot step and an in­ven­tory trans­fer. Each phys­i­cal re­sult ad­vances the next part of the plan.

Pick up all the items

“Pick up all the items and give them to me” be­comes a bounded col­lec­tion plan over real ref­er­ences. Varkos gath­ers them, re­turns and trans­fers them with­out pre­tend­ing that one mag­i­cal ac­tion means “all.”

Combat

Varkos re­ceives grounded events from the game, can warn the player through a fast re­flex path and uses na­tive body con­trol to act. Instruction plans can strate­gize (e.g. at­tack this, then re­treat, etc.), and his emo­tional state can af­fect how and if he chooses to fight.

Personality evo­lu­tion

Varkos is fully cus­tomiz­able. He does not have to be a de­mon dog, and the run­time does not have to con­trol only a sin­gle char­ac­ter. What sys­tems are ap­plied and what they do, is up to open con­fig­u­ra­tion.

One part of my cur­rent build still fully de­pends on big model/​cloud LLM calls: slow per­son­al­ity evo­lu­tion. This work hap­pens away from the real-time ac­tion path. As the player and Varkos travel to­gether, im­por­tant in­ter­ac­tions be­come ev­i­dence for grad­ual changes to his per­son­al­ity.

My demo Varkos be­gins as a de­mon rein­car­nated as a dog. He con­sid­ers his ca­nine in­stincts hu­mil­i­at­ing, his dog body a prison, and is mis­trust­ful, proud and sar­cas­tic. Through shared ex­pe­ri­ences he can be­come more and more do­mes­ti­cated, grow at­tached to the player and starts en­joy­ing be­ing a dog. Eventually he starts bring­ing over toys be­cause he wants to play, run­ning off to chase things and seek­ing af­fir­ma­tion from the player.

Only the start­ing char­ac­ter traits are au­thored. The sys­tem changes both his ex­plicit traits and his emo­tional home­osta­sis. How eas­ily he be­comes ir­ri­tated, fright­ened, af­fec­tion­ate or play­ful, etc etc. He can over­write parts of his vo­cab­u­lary and code. Changes are ver­sioned and re­versible.

I could make it more bounded, but I think there’s some­thing fun about some open world clank­i­ness, so how he evolves is up in the air.

Dog in and out of the game - Void mode

My plan is to make this sys­tem a gam­ing com­pan­ion that can fol­low you across mul­ti­ple dif­fer­ent games, not just Skyrim (Skyrim felt like a good start­ing point due to its mas­sive mod­ding com­mu­nity, VR sup­port and big open world).

For this rea­son he ex­ists out­side the game too. When the game closes, he en­ters “void mode” and can­not see or feel any­thing. How he re­sponds to that de­pends on his per­son­al­ity evo­lu­tion.

This state also works as an in-be­tween for dif­fer­ent games. One mo­ment Varkos could be fight­ing a dragon, then the world goes dark, then he ap­pears be­side you in Microsoft Flight Simulator. Maybe he would be shocked, need time to un­der­stand the new world and slowly learn what its ma­chines and rules mean, or maybe he knows about it al­ready and over­joyed tries to chase the sun.

Let’s talk tech­nol­ogy now

Unfortunately I am bit­ter-les­son pilled. Big model is bet­ter. If we wanted a per­fectly in­tel­li­gent sys­tem then let­ting a coun­cil of hy­per-in­tel­li­gent LLMs con­trol im­pulses, sen­sory pro­cess­ing, think­ing and act­ing at suf­fi­cient re­fresh rate would be best.

In some early ex­per­i­ments this worked in­sanely well, un­for­tu­nately to­day it is too slow and too ex­pen­sive. I do be­lieve this will be the ap­proach of some vague fu­ture.

Until then how­ever we need to hack our way in. Today’s games have pretty cool “AI” (not in the llm sense, more in the be­hav­ioral graph one), games like Red Dead Redemption and Dwarf Fortress have tons of depth and they can run per­fectly on 10 year old hard­ware.

Through this whole AI-craze peo­ple have for­got­ten that we had in­tel­li­gent sys­tems that could process speech since the 1970s, and some­what LLM-like be­hav­ior with chat­bots like SmarterChild in the early 2000s. There’s a lost art that is be­ing over­looked to­day in things like tra­di­tional NLP and be­hav­ioral graphs.

Let’s take a quick look at Varkos tech stack.

The game runs on Windows, the au­dio pro­cess­ing and brain runs on my M4 MacBook. It could all run on Windows (provided there is ded­i­cated ~12gb or more gpu ram for it), but I do de­vel­op­ment on the MacBook and I got so deep in that… eeh.

Audio:

Microphone is al­ways on.

Main voice to text en­gine is (custom ker­nels) op­ti­mized Qwen3-ASR 1.7b. A cus­tom har­ness is built around Qwen3-ASR, that processes and stitches au­dio in rolling par­tials (by de­fault that model does not sup­port stream­ing). The goal is to process au­dio in 40ms-80ms be it a tiny ut­ter­ance like “Hey” on a 1 minute long mono­logue.

VAD-like meth­ods such as turn­pipe and Silero (both op­ti­mized) are used to dis­tin­guish when a turn is open.

Lexical analy­sis also is done on the text try­ing to de­cide if the player has made a point or is not done talk­ing yet (eg think­ing mid-sen­tence). In a per­fect world of suf­fi­ciently fast and smart LLMs, the LLM would per­form bet­ter, but I have to re­sort to more rudi­men­tary but fast NLP ap­proaches.

This is im­por­tant as with the mi­cro­phone al­ways on as we want to start pro­cess­ing the play­er’s ut­ter­ance ASAP. It is also im­por­tant for turn in­ter­rup­tion and barge in and to have the com­pan­ion not speak over the player

I will be open-sourc­ing this Qwen3-ASR har­ness soon (bear with me I have a day job).

Audio gen­er­a­tion.

Optimized ver­sion of PocketTTS called PocketTTS-Raven (open sourced this a while ago. You can see it in ac­tion here - https://​pan­tel.is/​pro­jects/​pocket-tts-raven/?​b=1 or grab its code https://​github.com/​pkalo­giros/​pocket-tts-raven ) is used as the main fast en­gine.

Similarly op­ti­mized qwen-3-tts (write up com­ing soon). Depending on the com­plex­ity of the gen­er­a­tion we ei­ther use qwen-3 since it has bet­ter emo­tional con­trol. If a gen­er­a­tion has taken longer we de­fault to PocketTTS since it is quite fast (20 – 30ms au­dio gen­er­a­tion).

One trick I do, is that I gen­er­ate mul­ti­ple voices for dif­fer­ent emo­tions (angry, sad, neu­tral, con­fused etc) - load them all in mem­ory, and then use them where ap­pro­pri­ate.

“Thinking.”

This is the se­cret sauce and biggest dif­fer­en­tia­tor. I call this sys­tem (ALE - Action Latent Encoder be­cause it’s an ac­tion en­coder in need of a fun acronym). Under the hood, ALE is a hy­brid of em­bed­dings, small clas­si­fiers, ex­plicit rules and tra­di­tional ML. ALE de­tects struc­ture, iden­ti­fies nega­tion, com­mands, con­tin­u­a­tion, pro­nouns, and se­quences. For ex­am­ple, “pick up the sword and bring it to me” be­comes two linked ac­tion slots.

ALE is de­signed to be largely in­vari­ant to phras­ing. You can say pick up, you can say grab, fetch, go get the damn sword you fool - it does­n’t mat­ter, it will still un­der­stand you. If there is not enough con­text it will in­ject from pre­vi­ous dis­cus­sion. If it does­n’t, it will fall­back to ‘clarification’ and the dog will ask what do you mean.

It cre­ates em­bed­dings from the full text as well as its ex­tracted struc­ture, then se­man­ti­cally com­bines and com­pares it with ac­tion pro­to­types. A sep­a­rate clas­si­fier es­ti­mates whether the turn is a com­mand, ques­tion, chat, clar­i­fi­ca­tion or com­plex re­quest. Everything gets merged to­gether.

The main dif­fer­ence be­tween ALE and other such hy­brid-clas­si­fiers is that it ac­cepts the world state as well. It tries to match in­for­ma­tion from the world JSON to the play­er’s re­quest.

ALE needs to have a dif­fer­ent ver­sion for each game Varkos would par­tic­i­pate in. So in a way, it is not fully plug and play but a small prepa­ra­tion and com­pat­i­bil­ity step would need to be im­ple­mented to en­sure ac­tions are ac­counted for and world state is un­der­stood.

ALE can be trained in a few min­utes, so the sys­tem could in the­ory use a beefier LLM of­fline to re­view a ses­sion and re-train it­self let’s say overnight based on the play­er’s ex­pe­ri­ence and im­prove it­self. In my lim­ited in­ter­nal evals, ALE per­forms sur­pris­ingly close to large LLMs at se­lect­ing the right ac­tion and de­com­pos­ing the plan. I do not con­sider this a se­ri­ous bench­mark yet, but it has been re­li­able enough to drive Varkos in prac­tice.

It runs quite fast, around 2 – 20ms on M4 MacBook and es­sen­tially acts as a tool+tar­get func­tion and plan de­com­poser caller.

A lo­cal fine-tuned LLM is then used to fuse, Varkos per­sona, emo­tions, etc etc, re­cent his­tory, + ac­tion cho­sen and lets the brain form and paint the re­sponse. Extra ground­ing is per­formed to weed out hal­lu­ci­na­tions and re-ground it. If it fails maybe he will speak a cached re­sponse, if we have a time bud­get we can re­process. With a smart pre­fill strat­egy, in cer­tain cases the dog can be­gin an­swer­ing in un­der 500ms—from the player stop­ping speak­ing to the dog yap­ping.

Budget break­down

~40 – 80ms for voice to text.

~20 – 60ms for au­dio gen­er­a­tion.

~20ms for ac­tion analy­sis

And 300 – 600ms for cre­at­ing the re­sponse and ground­ing its el­i­gi­bil­ity (since if the dog is afraid of spi­ders and we ask it to at­tack a spi­der, it might be cute for it to stub­bornly deny).

Everything needs to be stream­ing and start as early as pos­si­ble (eg llm speech does­n’t need to wait to be com­pleted for the dog to speak out loud. Prefill early as soon as pos­si­ble, etc).

Limitations

Fast enough and lo­cal mod­els are not very ca­pa­ble at keep­ing the thread across mul­ti­ple turns and a dis­cus­sion over a long pe­riod of time can drift mak­ing the dog ap­pear con­fused. I ex­pect this to im­prove as both hard­ware and soft­ware evolve over time (1 to 2 years my es­ti­ma­tion). Also this runs in real time on con­sumer (albeit higher-end) hard­ware to­day in the near fu­ture it will be com­mon­place.

Surprisingly enough, us­ing re­mote LLM providers does not help much. Big mod­els are still too slow. There are su­per-fast in­fer­ence providers out there such as Cerebras. These work re­ally well in terms of la­tency, and leave head­room for greater con­text, higher in­tel­li­gence and depth. However, the mod­els they pro­vide (gpt-oss-120b and gem­ma31 as of to­day) are also pretty bad at hold­ing a con­ver­sa­tion (Why was GLM and the king of role­play Qwen taken away huh??).

A fi­nal word

Overall I think there is some­thing spe­cial here. Maybe it’s be­cause Varkos is a dog, and who does­n’t like dogs. Maybe it’s the low la­tency and that Varkos is ac­tu­ally use­ful in scout­ing ar­eas for clues and ob­jects or as a pack-mule. Maybe see­ing the small cracks in his per­son­al­ity as he com­plains why he has­n’t been called a “good boy” re­cently, but as I playtest the sys­tem I catch my­self ac­tu­ally hav­ing tons of fun.

Let me be clear that I do not ex­pect LLMs to re­place hand-crafted char­ac­ters and sto­ry­lines. Slop is a real thing, and in­ten­tional de­sign is still king and I be­lieve and want it to re­main so. But I think it is only a mat­ter of time be­fore we start see­ing more such sys­tems. An AI that ac­tu­ally plays with us, not merely talks at us, can be a dif­fer­ent medium.

Plus the philo­soph­i­cal mind­fuck of it all. Pleasingly ridicu­lous to wield the power of thun­der to cre­ate a dif­fer­ent kind of in­tel­li­gence, then forc­ing it to be a dog and go hunt things to­gether (is it bet­ter morally than hav­ing it do never-end­ing work? Well, it’s not alive so it does­n’t mat­ter), but it’s nice to think about. In a world of never-end­ing on­line dis­course around per­ma­nent un­der­classes and world-end­ing rogue agents, it’s nice to deal with align­ment through shared ex­pe­ri­ences and tam­ing the thing to play fetch and eat treats.

In Plato’s cave we may still be alone, but at least we can be hav­ing fun with this weird dis­torted and alien thing that is now deep in the cave with us.

I’ll prob­a­bly be open sourc­ing parts of the sys­tem, and even­tu­ally all of it soon, along with a ver­sion that sup­ports mul­ti­ple NPCs (cloud inf only for now) in­ter­act­ing (actually in­ter­act­ing not just larp­ing) with each other.

Until then fol­low me on @pkalogiros if you’re cu­ri­ous and wanna see (slow) up­dates. Cheers.

We have some dif­fi­cult news to share with the IPFS and wider peer-to-peer com­mu­nity.

Protocol Labs has in­formed us that it will not be re­new­ing Shipyard’s fund­ing. While we’re grate­ful for the sup­port and trust they have placed in us over the past two-plus years, we’re nat­u­rally dis­ap­pointed by this out­come. As a di­rect re­sult, Shipyard will be wind­ing down its IPFS-related en­gi­neer­ing, main­te­nance, and in­fra­struc­ture op­er­a­tions. Our fi­nal day of our IPFS re­lated work will be September 30, 2026.

Over the past three years, it has been our priv­i­lege to help shape the mod­ern IPFS ecosys­tem and em­power users with more re­silient, self-sov­er­eign tech­nol­ogy. You can read more about the im­pact­ful work that we shipped in a fol­low-up post we’ll be shar­ing in the com­ing days, but some high­lights in­clude:

Delivering ver­i­fi­able web­sites and down­loads di­rectly in the browser through in­browser.link.

Re-architecting IPFS gate­way in­fra­struc­ture to han­dle ap­prox­i­mately 3× more traf­fic while re­duc­ing op­er­at­ing and main­te­nance costs by around 80%.

Advancing HTTP-native ap­proaches to IPFS that dra­mat­i­cally sim­plify de­ploy­ment, de­vel­op­ment, and op­er­at­ing costs com­pared with tra­di­tional libp2p-based host­ing.

Maintaining and im­prov­ing many of the core im­ple­men­ta­tions, li­braries, and pub­lic in­fra­struc­ture re­lied upon by the IPFS ecosys­tem every day.

We were ex­cited about de­liv­er­ing the next chap­ter for IPFS: dra­mat­i­cally sim­pler HTTP-native im­ple­men­ta­tions, re­silient and sus­tain­able con­tent rout­ing, sup­port for large na­tive SHA-256 ob­jects, pseu­do­ny­mous host­ing and re­trieval through Tor and onion ser­vices, and many other ideas we be­lieved would make IPFS sig­nif­i­cantly eas­ier to adopt. Unfortunately, we won’t have the op­por­tu­nity to see those ef­forts through our­selves.

The prac­ti­cal im­pli­ca­tions ex­tend well be­yond Shipyard. Among other things:

Projects main­tained by Shipyard will no longer have ded­i­cated main­tain­ers re­spon­si­ble for new fea­tures, bug fixes, re­leases, or long-term stew­ard­ship. These in­clude: Kubo, Helia, Boxo, Rainbow, IPFS Desktop, IPFS Companion, Someguy, Service Worker Gateway, IPFS Check, and oth­ers.

Contributions from Shipyard to up­stream pro­jects such as go-libp2p and js-libp2p will cease.

Our work on IPFS spec­i­fi­ca­tions, stan­dards, and broader ecosys­tem co­or­di­na­tion will come to an end.

Shipyard will cease op­er­at­ing the pub­lic in­fra­struc­ture it cur­rently man­ages, in­clud­ing ipfs.io, dweb.link, check.ipfs.net­work, del­e­gated-ipfs.dev, the IPFS boot­strap nodes, col­lab­o­ra­tive clus­ter in­fra­struc­ture such as Wikipedia-on-IPFS, and re­lated ser­vices. Protocol Labs, as the owner of the as­so­ci­ated do­mains and in­fra­struc­ture, will de­ter­mine their fu­ture.

Our goal over the com­ing weeks is to leave the IPFS ecosys­tem in the best pos­si­ble po­si­tion for what­ever comes next.

We’ll re­main avail­able through the end of September to help with that tran­si­tion. If you main­tain soft­ware, op­er­ate in­fra­struc­ture, or rely on any of the work Shipyard has been re­spon­si­ble for, please don’t hes­i­tate to reach out. We’ll do every­thing we rea­son­ably can to an­swer ques­tions, pro­vide con­text, and help make the tran­si­tion as smooth as pos­si­ble.

If you have a favourite mem­ory of work­ing with Shipyard, or an idea you al­ways hoped IPFS would even­tu­ally achieve, we’d love to hear it. Google Form

Finally, we want to say thank you.

To every­one who con­tributed code, re­viewed pull re­quests, filed is­sues, tested ex­per­i­men­tal fea­tures, ran in­fra­struc­ture, par­tic­i­pated in stan­dards dis­cus­sions, or sim­ply be­lieved in the idea that con­tent should be ad­dressed by what it is rather than where it lives: thank you.

It’s been an ho­n­our to build along­side this com­mu­nity. While this chap­ter of IPFS at Shipyard is com­ing to a close, we re­main proud of what we’ve ac­com­plished to­gether, and we hope the work we’ve done helps pro­vide a strong foun­da­tion for what­ever comes next.

Updated / Monday, 24 Aug 2026 19:41

Getty

New EU rules have come into force de­signed to en­cour­age con­sumers to re­pair rather than re­place prod­ucts to tackle the es­ti­mated 35 mil­lion tonnes of waste the pre­ma­ture dis­posal of con­sumer goods gen­er­ates across the bloc.

The ‘right to re­pair’ reg­u­la­tions in­tro­duce new rights and sup­ports aimed at en­cour­ag­ing re­pair over dis­posal, in­clud­ing a re­pair oblig­a­tion for man­u­fac­tur­ers of cer­tain prod­ucts and the es­tab­lish­ment of a na­tional re­pair plat­form to help con­sumers lo­cate re­pair ser­vices.

The rules ap­ply to house­hold and elec­tronic prod­ucts such as wash­ing ma­chines, vac­uum clean­ers, mo­bile phones, and tablets.

The reg­u­la­tions are de­signed to make it eas­ier for con­sumers to ac­cess re­pair ser­vices when prod­ucts de­velop faults af­ter the sell­er’s guar­an­tee pe­riod.

People now have the right to re­quest that man­u­fac­tur­ers re­pair prod­ucts that are tech­ni­cally re­pairable un­der EU law.

The re­pairs must be done for free or at a rea­son­able price, within a rea­son­able time­frame, to en­cour­age re­pairs.

We need your con­sent to load this rte-player con­tentWe use rte-player to man­age ex­tra con­tent that can set cook­ies on your de­vice and col­lect data about your ac­tiv­ity. Please re­view their de­tails and ac­cept them to load the con­tent.Man­age Preferences

Manufacturers must also pro­vide eas­ily ac­ces­si­ble in­for­ma­tion about re­pair ser­vices, as well as ac­cess to spare parts at a rea­son­able price.

The na­tional re­pair di­rec­tory, RepairMyStuff.ie, will be fur­ther de­vel­oped to ful­fil the role as Ireland’s na­tional re­pair plat­form.

“These reg­u­la­tions will make it eas­ier for con­sumers to choose re­pair over re­place­ment, while also cre­at­ing op­por­tu­ni­ties for Irish busi­nesses op­er­at­ing in the re­pair and re­fur­bish­ment sec­tor,” said Minister for Enterprise, Tourism and Employment Peter Burke.

“In im­ple­ment­ing the Directive, we have sought to strike the right bal­ance be­tween sup­port­ing con­sumers, en­cour­ag­ing sus­tain­able eco­nomic ac­tiv­ity and en­sur­ing that busi­nesses are not faced with un­nec­es­sary reg­u­la­tory bur­dens,” he added.

We need your con­sent to load this rte-player con­tentWe use rte-player to man­age ex­tra con­tent that can set cook­ies on your de­vice and col­lect data about your ac­tiv­ity. Please re­view their de­tails and ac­cept them to load the con­tent.Man­age Preferences

According to the European Commission, the new rules are ex­pected to bring €4.8 bil­lion in growth and in­vest­ment within the EU.

“These new reg­u­la­tions are im­por­tant in help­ing to meet our cli­mate goals by re­align­ing how we pro­duce, con­sume, and value ma­te­ri­als,” said Minister for Climate, Energy and the Environment Darragh O’Brien.

Read more: New EU re­pair rules ‘make sense’, en­gi­neer says