Please en­able cook­ies.

Error 1005

What hap­pened?

The owner of this web­site (videocardz.com) has banned the au­tonomous sys­tem num­ber (ASN) your IP ad­dress is in (14061) from ac­cess­ing this web­site.

Please see https://​de­vel­op­ers.cloud­flare.com/​sup­port/​trou­bleshoot­ing/​http-sta­tus-codes/​cloud­flare-1xxx-er­rors/​er­ror-1005/ for more de­tails.

Was this page help­ful?

Thank you for your feed­back!

Cloudflare Ray ID: a1d29f1e2d­c­c7cb4 •

Your IP:

167.99.127.20 •

Performance & se­cu­rity by Cloudflare

KAISER PERMANENTE NURSES who an­swer ad­vice and triage calls say their duty of care for pa­tients is be­ing in­creas­ingly threat­ened by work­place sur­veil­lance.

Seven cur­rent and for­mer nurses told CalMatters that those who spend more than 15 min­utes on a call with a pa­tient rou­tinely face crit­i­cism from Kaiser man­age­ment or get called into per­for­mance eval­u­a­tion meet­ings. Call time, they said, fac­tors into monthly per­for­mance scores they re­ceive.

In ad­di­tion to track­ing call length, they said Kaiser uses soft­ware that tries to pre­dict on a daily ba­sis whether they’re be­ing un­pro­duc­tive or fail­ing to an­swer calls quickly. Artificial in­tel­li­gence sys­tems have also been used to rate their em­pa­thy and tone of voice.

Their com­ments come as the California Nurses Association be­gins ne­go­ti­at­ing a new con­tract with Kaiser this month with AI a likely is­sue. Kaiser nurses went on strike against AI for one day in March and pick­eted against AI last fall. The CNA is bar­gain­ing for 25,000 nurses, in­clud­ing 1,000 in call cen­ters.

At the same time, California law­mak­ers are con­sid­er­ing sev­eral bills reg­u­lat­ing AI in the work­place, in­clud­ing one that would pro­tect from re­tal­i­a­tion doc­tors and nurses who over­ride au­to­mated care rec­om­men­da­tions.

Kaiser de­fended its use of AI, say­ing it de­ploys the tech­nol­ogy with pa­tient safety in mind and does not use “average han­dle time” to as­sess per­for­mance.

Kaiser Permanente is the largest pri­vate em­ployer in California, pro­vid­ing health­care ser­vices to more than 9 mil­lion peo­ple in the state and to 3 mil­lion other Americans. That means the com­pa­ny’s use of ar­ti­fi­cial in­tel­li­gence could set im­por­tant prece­dents for man­ag­ing work­ers with AI. It could also have a big im­pact on pa­tient care, pro­vid­ing an early ex­am­ple of how the health­care sec­tor bal­ances cost-cut­ting au­toma­tion with hu­man pres­ence or touch.

Raquel Alvarez Sanchez, a Kaiser Permanente ad­vice nurse in Vallejo since 2010, said she was on a call with a pa­tient who was sui­ci­dal last year that took more than an hour be­cause she had to wait for po­lice to ar­rive be­fore hang­ing up. She tried to make the man feel cared for, even though she was cog­nizant that stay­ing on the call that long would throw off her av­er­age call time for weeks and could lead to ques­tions from man­age­ment. Sanchez, a union stew­ard, said she’s ac­com­pa­nied col­leagues to per­for­mance eval­u­a­tion meet­ings, where they were found to have done every­thing right on a call — ex­cept stay­ing on the line for more than 15 min­utes. She said she has­n’t seen nurses get fired for do­ing that, but she fears that con­tin­ued pres­sure can lead nurses to quit or re­tire early.

“I think at some point all of the nurses have been talked to about their av­er­age han­dle time,” she said. “The only thing I can think of is they’re do­ing it for profit.”

Another nurse who spoke with CalMatters on con­di­tion of anonymity due to fear of ret­ri­bu­tion de­scribed how that sur­veil­lance af­fected a call with a pa­tient last year. Initially she thought her pa­tient, an el­derly woman who just re­ceived a ter­mi­nal can­cer di­ag­no­sis, was sui­ci­dal, but quickly came to un­der­stand that she was in shock and re­ally needed some­body to talk to.

The nurse wanted to take time to show com­pas­sion or com­fort to the woman, who acts as a care­taker for her daugh­ter, but she stopped her­self out of fear it would hurt her monthly per­for­mance score and lead to a rep­ri­mand from her man­ager. She be­came a nurse to pro­vide peo­ple with com­pas­sion­ate care, but “I had to ask my­self: Am I go­ing to get dis­ci­plined for go­ing off script or say­ing more than what is nec­es­sary?”

Kaiser Permanente says its per­for­mance eval­u­a­tions help im­prove pa­tient out­comes. A com­pany spokesper­son said, “Kaiser Permanente does not use Average Handle Time to as­sess agent per­for­mance or en­force call time met­rics. Any tools used in con­tact cen­ter set­tings sup­port our qual­ity as­sur­ance ef­forts and have hu­man re­view and over­sight.” In a state­ment pro­vided to CalMatters, spokesper­son Vincent Staupe added that Kaiser uses AI re­spon­si­bly, with hu­man over­sight, and by “prioritizing pa­tient safety, pri­vacy, and eq­uity,” but he said, “As a large or­ga­ni­za­tion, we do not share spe­cific in­for­ma­tion about in­ter­nal tech­nol­ogy sys­tems for se­cu­rity and op­er­a­tional rea­sons.”

Is tech­nol­ogy putting pa­tients at risk?

It’s not clear how pa­tient care is af­fected by al­go­rith­mic man­age­ment, nor is the im­pact of lim­it­ing the length of triage and ad­vice calls on pa­tients. Kaiser call cen­ter nurses can’t say for cer­tain whether the pres­sures they face re­sults in ad­verse out­comes for pa­tients be­cause their con­tact with pa­tients ends af­ter they hang up the phone. A 2024 pub­lic records re­quest by CalMatters to the California Department of Managed Health Care found no com­plaints by pa­tients against Kaiser re­lated to call times. But nurses in­sist the risk to pa­tient safety and qual­ity of care is real.

Consumer Watchdog pa­tient ad­vo­cate Michele Ramos said many Kaiser pa­tients be­gin their care on the ad­vice line. They later com­plain to her, mostly about things that hap­pen in Kaiser fa­cil­i­ties, but “I can see now where a lot of the prob­lems” start, given the call con­straints nurses are un­der.

“Kaiser’s been known through the years to man­age dol­lars over man­ag­ing care, … which is only go­ing to fail pa­tients.”

Michele Ramos, Consumer Watchdog pa­tient ad­vo­cate

“Kaiser’s been known through the years to man­age dol­lars over man­ag­ing care, … which is only go­ing to fail pa­tients.”

Michele Ramos, Consumer Watchdog pa­tient ad­vo­cate

Ramos said the time pres­sures may fit a broader pat­tern at Kaiser of putting costs over qual­ity. The health gi­ant was hit with a record fine, $50 mil­lion, as part of a set­tle­ment over find­ings from the California Department of Managed Health Care that it de­layed be­hav­ioral health ap­point­ments be­yond statu­tory lim­its and too of­ten moved pa­tients into group rather than in­di­vid­ual ther­apy. Kaiser also set­tled with the U.S. Department of Labor after in­ves­ti­ga­tions into its sub­stance use and men­tal health ser­vices. Kaiser faced crit­i­cism in 2002 for pay­ing bonuses to call cen­ter work­ers who aren’t nurses for keep­ing calls short, though call cen­ter nurses who spoke with CalMatters said they en­coun­tered no such prac­tices to­day.

“Kaiser’s been known through the years to man­age dol­lars over man­ag­ing care, and I think this would be a con­trib­u­tor to that, which is only go­ing to fail pa­tients,” Ramos added.

Nurses said they are pres­sured to stay un­der 15 min­utes even for the sorts of calls that of­ten take more time, like di­ag­nos­ing a pa­tient with mul­ti­ple symp­toms, chronic ill­nesses, new par­ents in need of ad­vice and as­sur­ance, peo­ple who de­sire ex­tended health ed­u­ca­tion, or peo­ple who are over­whelmed af­ter re­ceiv­ing life-al­ter­ing news who could use some com­pas­sion. Nurses say calls that in­volve in­ter­preters of­ten take 30 min­utes or more. About four in 10 Californians speak a lan­guage other than English and half of them do not speak English well, ac­cord­ing to a state en­vi­ron­men­tal health agency.

“The amount of time that Kaiser is giv­ing us to com­plete a call is some­times not safe,” said one nurse, who asked to re­main anony­mous due to fear of re­tal­i­a­tion.

“People can get hurt,” said Charlotte Capulong, who has worked in nurse call cen­ters for 22 years and helped or­ga­nize Kaiser nurses against the AI tone-of-voice tool. Capulong said nurses felt ha­rassed by man­agers in meet­ings she at­tended as a union rep, even if they suc­cess­fully car­ried out all other du­ties of their jobs ex­cept com­plet­ing calls within 15 min­utes.

“You aren’t call­ing Comcast. We’re deal­ing with life here,” she said.

Nurses are in­structed to stick to a script on phone calls and give no more than two to three pieces of ad­vice, Capulong and other nurses said, which means they may some­times need to de­cide whether to with­hold ad­vice or face a per­for­mance eval­u­a­tion hear­ing.

The nurses say ar­ti­fi­cial in­tel­li­gence could make the sur­veil­lance nurses en­counter on the job worse.

In sum­mer 2024, Kaiser be­gan test­ing an AI tool that at­tempts to as­sess em­pa­thy and tone in the voices of nurses and pa­tients, ac­cord­ing to nurses who spoke with CalMatters. In re­sponse, nurses cir­cu­lated and signed a pe­ti­tion in fa­vor of the right to pa­tient pri­vacy, more trans­parency,  and the right to ex­er­cise their pro­fes­sional judge­ment and en­cour­aged man­age­ment to in­volve nurse’s in­put and feed­back. The sig­na­ture cam­paign used the same tag line that nurses used at protests out­side San Francisco hos­pi­tals ear­lier that year: “Trust nurses, not AI. The AI tests ended in November 2024, but union rep­re­sen­ta­tives were told that man­agers may bring the pro­gram back in the fu­ture.

Nurses re­ported feel­ing ha­rassed by ex­ist­ing sur­veil­lance, “and that was in­ten­si­fied when they said we’re go­ing to use AI to eval­u­ate our calls and grade us,” said Sanchez.

Another nurse speak­ing on con­di­tion of anonymity said “AI did not un­der­stand our job and would grade us wrong all the time.”

A Kaiser spokesper­son de­clined to re­spond to ques­tions about the AI tool or an­swer ques­tions about the use of AI and other au­to­mated sys­tems in the com­pa­ny’s call cen­ters and health­care fa­cil­i­ties, in­clud­ing for eval­u­at­ing nurse per­for­mance or whether pa­tients were in­formed about the use of AI to eval­u­ate their em­pa­thy and tone.

Nurses also said they get lit­tle time be­tween calls even if that call in­volves speak­ing with a pa­tient who is sui­ci­dal, ex­pe­ri­enc­ing a men­tal health episode, or near death. In years past, nurses got around 10 min­utes to fin­ish writ­ing notes in a pa­tien­t’s chart or col­lect them­selves af­ter a par­tic­u­larly tough call. Today they say they typ­i­cally get 30 sec­onds or less when lines are busy, al­though more at slow times, like late at night, or if they get a man­ager’s per­mis­sion af­ter a par­tic­u­larly chal­leng­ing call. The over­all pace they say, can lead to mis­takes like miss­ing im­por­tant cues into a pa­tien­t’s well­be­ing.

CNA reps de­clined to talk about spe­cific pro­vi­sions they in­tend to seek re­lated to AI ahead of their talks with Kaiser this sum­mer.

How sur­veil­lance and AI shape nurs­ing

Critics say ex­ces­sive work­place mon­i­tor­ing can lead to lower morale as em­ploy­ees feel less trusted and au­tonomous, rel­e­gated to be­ing no more than al­go­rithm mon­i­tors. UC Berkeley Labor Center Technology and Work Program di­rec­tor An­nette Bernhardt has warned that al­go­rith­mic man­age­ment can turn peo­ple into fleshy ro­bots, echo­ing com­plaints from an Ama­zon fac­tory worker who CalMatters in­ter­viewed last year. A 2023 aca­d­e­mic sur­vey of call cen­ters in four de­vel­oped coun­tries found that us­ing AI for man­age­ment or mon­i­tor­ing left work­ers with less time be­tween calls and more likely to feel emo­tion­ally drained by their work. Nearly half of re­spon­dents said that AI tools made their jobs more stress­ful. A prior study by the same re­searchers, Virginia Dolleghast of Cornell University and Sean O’Brady of McMaster University found that per­for­mance mon­i­tor­ing leads to higher rates of emo­tional ex­haus­tion.

Dolleghast, who has stud­ied the im­pact of sur­veil­lance tech­nol­ogy on call cen­ter work­ers for more than a decade, said what Kaiser call cen­ter nurses are ex­pe­ri­enc­ing is part of a broader trend: Across dif­fer­ent in­dus­tries, per­sis­tent sur­veil­lance is in­creas­ing stress lev­els for work­ers who are re­solv­ing com­plex, emo­tion­ally-charged is­sues.

“Stress and burnout can lead to more mis­takes across a range of ar­eas, and in the health­care set­ting that is much higher risk be­cause you’re deal­ing with peo­ple’s lives and their health,” she said.

The con­verse can be true: Workers who are given more dis­cre­tion to de­cide the pace and tim­ing of their work ex­pe­ri­ence higher lev­els of job sat­is­fac­tion and less ab­sen­teeism.

Nurses na­tion­wide are more fre­quently en­coun­ter­ing ar­ti­fi­cial in­tel­li­gence and sim­i­lar soft­ware sys­tems in the work­place. Half of more than 2,000 nurses who re­sponded to a 2024 sur­vey by the National Nurses United union said their em­ployer uses al­go­rith­mic sys­tems to an­a­lyze health records. Such sys­tems can do things like de­ter­mine how frag­ile a pa­tient is or pre­dict how many hours of care they will need. Two-thirds of the sur­veyed nurses said their own as­sess­ments had at some point dis­agreed with a com­puter-gen­er­ated pre­dic­tion. Six out of 10 re­spon­dents said they don’t trust their em­ployer to pri­or­i­tize pa­tient safety when us­ing AI.

Pa Vue has worked as a nurse in call cen­ters for the bet­ter part of the past decade. She said she and other Kaiser nurses rou­tinely have con­ver­sa­tions with man­agers about call ef­fi­ciency and re­ceive eval­u­a­tion scores once a month. She re­calls hav­ing a score re­duced for re­peat­ing ad­vice to a pa­tient that she wor­ried had un­usual symp­toms and pos­si­ble heart is­sues.

As a union rep­re­sen­ta­tive in some per­for­mance meet­ings, Vue has seen man­agers raise ef­fi­ciency ques­tions about calls they deem too long. She’s also seen nurses re­ceive lower per­for­mance scores if they go against soft­ware rec­om­men­da­tions based on their pro­fes­sional opin­ion or make an ap­point­ment for a pa­tient with­out con­sult­ing a doc­tor.

She be­lieves that ef­fi­ciency aims ac­cel­er­ated by tech­nol­ogy can hin­der a nurse’s abil­ity to fo­cus and re­duce the qual­ity of care that pa­tients pay for.

“I’m not against the use of AI as long as it’s ben­e­fi­cial to the pa­tient but in this par­tic­u­lar use [empathy and tone mon­i­tor­ing] it’s to in­crease pro­duc­tiv­ity and im­prove ef­fi­ciency and cut costs. Kaiser is for­get­ting we aren’t just a call cen­ter for cus­tomer sup­port, we’re nurses, and we’re there to take care of pa­tients,” she said.

As AI im­proves and busi­nesses push work­ers to use it, unions are, in turn, in­creas­ingly de­mand­ing that em­ploy­ers ad­dress is­sues raised by AI when bar­gain­ing for new con­tracts. Surveillance tech­nol­ogy has be­come a com­mon way for man­agers to col­lect data about work­ers in a num­ber of in­dus­tries, used for every­thing from im­prov­ing safety to hunt­ing for ways to in­crease profit gains or train AI to do a job.

At Kaiser, AI is a key is­sue not only among nurses but also for men­tal health work­ers, 2,400 of whom are in con­tract ne­go­ti­a­tions in Northern California with Kaiser Permanente. Kaiser ther­a­pists have said they are con­cerned about use of ther­apy ses­sion tran­scripts to train AI mod­els and about the health-care gi­ant us­ing AI to take their jobs. National Union of Healthcare Workers spokesper­son Matt Artz told CalMatters con­tract ne­go­ti­a­tions are on­go­ing.

How Kaiser uses AI

Kaiser Permanente is ex­plor­ing or us­ing AI in many parts of the health­care ex­pe­ri­ence far be­yond nurse call cen­ters. Kaiser uses AI to iden­tify pa­tients in hos­pi­tals at risk of ad­verse events by eval­u­at­ing data on their elec­tronic health records. An AI sys­tem called Preventus is used to de­ter­mine when to dis­charge pa­tients. Doctors and ther­a­pists use Abridge to record in­ter­ac­tions and trans­late speech to text dur­ing in-per­son vis­its with pa­tients in­stead of tak­ing notes. Remote mon­i­tor­ing with AI for pa­tients that need ex­tra care has been tested at Kaiser Permanente fa­cil­i­ties in the Bay Area, ac­cord­ing to nurses who en­coun­tered the tech­nol­ogy in the course of do­ing their jobs.

National Nurses United and CNA President Cathy Kennedy sees the use of AI to de­tect nurse em­pa­thy as part of a long se­ries of steps by Kaiser to limit their au­ton­omy and make them more ef­fi­cient. She be­lieves AI threat­ens to au­to­mate and frag­ment the work that nurses do, and com­pa­nies de­vel­op­ing and de­ploy­ing AI sys­tems should es­tab­lish that those sys­tems are ef­fec­tive and eq­ui­table be­fore de­ploy­ing them.

Notification of new tech de­ploy­ments is part of the nurse union’s con­tract with Kaiser but some­times nurses don’t re­ceive no­ti­fi­ca­tion, CNA says. So union lead­ers are at­tempt­ing to track the num­ber of AI mod­els in use at Kaiser Permanente, ad­vis­ing its mem­bers to in­form them when they en­counter new tech. This paves the way for CNA to push back as it did with the em­pa­thy and tone AI last sum­mer or as it did when it stopped a pi­lot pro­gram that would have re­placed nurses that sit at the bed­side of con­fused pa­tients with cam­eras.

Debru Carthan, a Kaiser ra­di­ol­o­gist, is on the front line of worker-man­age­ment fights over AI at the com­pany. A mem­ber of Service Employees International Union, she is also part of the Coalition of Kaiser Permanente Unions, where she sits on a com­mit­tee to dis­cuss use of AI and emerg­ing tech­nol­ogy at Kaiser. The coali­tion also has a “see some­thing, say some­thing,” cam­paign for front­line work­ers to re­port when they no­tice AI de­ploy­ments; the coali­tion says that too of­ten man­age­ment qui­etly im­ple­ments AI into work­flows with­out no­tice or worker in­put. She wor­ries that the AI tone de­tec­tor used on ad­vice nurses could dis­crim­i­nate against nurses from dif­fer­ent cul­tures and has come to be­lieve that the use of AI in health­care gen­er­ally has more to do with money and cor­po­rate greed than pa­tient care.

California law­mak­ers have re­sponded to worker AI con­cerns both in­side and out­side the health­care sec­tor. They tried and failed last year to ad­dress how AI im­pacts work­ers like call cen­ter nurses. As­sem­bly Bill 1018 and Senate Bill 7, two bills en­dorsed by the CNA, would have re­quired em­ploy­ers to in­form work­ers be­fore us­ing au­to­mated sys­tems on the job to do things like pro­mote or dis­ci­pline work­ers or eval­u­ate job per­for­mance, but Gov. Gavin Newsom ve­toed SB 7, and, fac­ing strong op­po­si­tion from com­pa­nies in­clud­ing Kaiser Permanente, AB 1018 failed to pass for the third con­sec­u­tive year.

Earlier this year, law­mak­ers rein­tro­duced a new ver­sion of Senate Bill 7, now called Senate Bill 947. Another bill would pro­hibit em­ploy­ers us­ing AI to pre­dict the emo­tional state of their em­ploy­ees. Yet another bill would pro­tect doc­tors and nurses from re­tal­i­a­tion if they over­ride rec­om­men­da­tions gen­er­ated by an au­to­mated sys­tem and re­quire health­care providers to sup­ply em­ploy­ees with an in­ven­tory of au­to­mated sys­tems once a year. Kaiser de­clined to share a com­pre­hen­sive list of AI sys­tems in use when asked by CalMatters.

Altogether CNA and the af­fil­i­ated California Labor Federation sup­port roughly half a dozen bills to reg­u­late use of AI in the work­place. Calling AI a cen­tral is­sue in the next pres­i­den­tial elec­tion, mem­bers of the California Labor Federation and la­bor lead­ers from Democratic pri­mary states held a press con­fer­ence in Sacramento ear­lier this year to say that if Newsom wants to be­come pres­i­dent then he needs to pass laws pro­tect­ing work­ers from AI. “It’s an on­go­ing fight, and it’s a fight well worth hav­ing,” Kennedy said. “Whenever there are other unions in dis­cus­sion about ar­ti­fi­cial in­tel­li­gence we are in sol­i­dar­ity with them.”

The nurse that with­held com­pas­sion to a ter­mi­nal can­cer pa­tient she thought was sui­ci­dal said she be­lieves mon­i­tor­ing and scor­ing sys­tems turn nurses into au­toma­tons that check boxes.

“I used to use hu­mor as a way to help pa­tients heal, and I don’t feel com­fort­able do­ing that here be­cause I know the calls are be­ing recorded. You can al­ways tell when a pa­tient ap­pre­ci­ates the hu­mor or your per­sonal com­pas­sion, but I don’t feel like call cen­ters have tol­er­ance for that be­cause that’s not part of the script,” she said. “That re­ally takes away from the whole point of be­ing a nurse and what pa­tients come to know from nurses.”

This story was re­ported with con­tri­bu­tions from Lam Thuy Vo and Ana Ibarra.

This story orig­i­nally ap­peared in CalMatters.

One of the cool fea­tures of JPEG files is that there’s the op­tion to save low fre­quency com­po­nents first. This means that a par­tially down­loaded im­age will be dis­played at low res­o­lu­tion in­stead of be­ing cut off.

In the file, this works by break­ing up the com­pressed data into mul­ti­ple “scans”, each pre­fixed with a header. Here’s the first scan of a rep­re­sen­tive im­age:

FF DA - “start of scan” marker 00 0C - Big en­dian length field (12 bytes) Includes it­self 03 - Number of chan­nels in scan (3) 01 - Global id of first in­cluded chan­nel 00 - Huffman table in­dex #1 (DC: 0, AC: 0) 02 - Global id of sec­ond in­cluded chan­nel 10 - Huffman table in­dex #2 (DC: 1, AC: 0) 03 - Global id of third in­cluded chan­nel 10 - Huffman table in­dex #2 (DC: 0, AC: 0) 00 - Starting DCT bin (DC) 00 - Ending DCT bin (also DC) 01 - Precision: half, no pre-ex­ist­ing data.

f8ad 512d d3f1 cd96 - Huffman coded DCT co­ef­fi­cients bcb0 58df 53d5 5d97 […and a lot more]

… this one in­cludes the low­est (DC) Fourier bin for all three color chan­nels.

The three color chan­nels are YCbCr in­stead of the usual RGB. The lu­mi­nance (Y) seper­ated be­cause it must be high qual­ity, but the color can be fudged quite a bit while look­ing fine.

Very roughly: Y = G, Cb = B - G, Cr = R - G

After it, the file con­tains eight more scans to fill in the rest of the data:

Scan #0 con­tains a very low res­o­lu­tion pre­view of the im­age.

Scan #1 adds some de­tails to the lu­mi­nance.

Scans num­ber two through five con­tain full low pre­ci­sion data.

Scan 4 has an un­usual spec­tral range be­cause it’s fill­ing in the gap left by #1. That way, num­ber 5 has full quar­ter pre­ci­sion data to build on.

Scans six through nine add the fi­nal miss­ing bit to bring the im­age to full qual­ity.

Given what I said about color be­ing less im­por­tant, it might seem weird that my ex­am­ple has the color data first: This works be­cause the the chromi­nance is saved at half res­o­lu­tion (quarter pixel count). As a re­sult, full chromi­nance data (Cr + Cb) only weighs half as much as lu­mi­nance.

Since each scan ex­plic­itly sets its spec­tral range, it should be pos­si­ble to con­struct a JPEG file where fu­ture scans over­write al­ready ren­dered im­age data.

Actually, it’s very easy to do this:

Concatenate mul­ti­ple im­ages with the same res­o­lu­tion and fil­ter out the start-of-im­age, start-of-frame and end-of-im­age mark­ers. This can be done in a hex ed­i­tor, but I used a quick and dirty C pro­gram.

When served over a slow net­work, this con­cate­nated file will switch be­tween mul­ti­ple im­ages:

But, most de­coders will give up af­ter some num­ber of scans: I think this is done to avoid a zip bomb style prob­lem… but it pre­vents this from work­ing on more than 9 frames, which is not enough for a proper an­i­ma­tion.

To do that, I’d have to min­i­mize the num­ber of scans in each frame. The sim­plest idea is to start with base­line JPEGs that only have a sin­gle scan.

… but it does­n’t work:

In pro­gres­sive mode, a scan can’t con­tain both AC (bins above 0) and DC (bin 0) data at the same time. This lim­i­ta­tion does­n’t ex­ist for base­line mode, but the base­line de­coder stops af­ter the first scan.

Since AC data must fol­low DC data, the small­est pos­si­ble “progressive” JPEG con­tains a sin­gle DC-only scan. Because the DCT runs on 16x16 blocks, such an im­age won’t a solid color:

it’ll be 1/16th of the orig­i­nal res­o­lu­tion.

Doing this, I can get Chrome to ren­der around 90 frames be­fore giv­ing up. Other browsers like Firefox have more pa­tience, but a 90 scan im­age seems to work al­most every­where.

As a bonus, this avoids the ghost­ing of the naive at­tempt: that hap­pened be­cause AC scans are sup­posed to re­fine old data. Normally, this al­lows im­ages to in­clude mul­ti­ple pre­ci­sion lev­els with­out in­flat­ing file size… but does­n’t play nicely with my tricks.

If the file only in­cludes DC scans with no ac­tual pro­gres­sion, this is­n’t a prob­lem.

Since a “DC-only” frame is a stan­dards-com­pli­ant im­ages, cre­at­ing them does­n’t re­quire any­thing spe­cial:

cat > frame.scans<<EOF # DC only scan: 0,1,2:0 – 0,0,0; # and noth­ing else EOF jpeg­tran -scans frame.scans -outfile out.jpg in.jpg

Using these, it’s pos­si­ble to pack a whole video in­side a sin­gle im­age:

Besides un­con­ven­tional rick­rolls and other trolling, this has no prac­ti­cal ap­pli­ca­tions: there’s no way to add tim­ing in­for­ma­tion, so play­back is en­tirely de­pen­dent on net­work de­lay.

… al­though there is a lot of fun to be had us­ing par­tial ren­der­ing:

This is a pure HTML video us­ing <dialog> tags: badap­ple.rose.sys­tems

Of course, there’s no rule that the data must be hard­coded: here’s a in­ter­ac­tive sin­gle-page ap­pli­ca­tion with no CSS or JavaScript. (seems slighty bro­ken, I’ll in­ves­ti­gate later)

Related:

/projects/bad_jpeg/merge.c: The code used to gen­er­ate these im­ages

16th July 2026

Chinese AI lab Moonshot AI an­nounced Kimi K3 this morn­ing, de­scrib­ing it as their “most ca­pa­ble model to date, with 2.8 tril­lion pa­ra­me­ters”. It’s cur­rently avail­able via their web­site and API, but an open weight re­lease is promised “by July 27, 2026”.

Moonshot are call­ing this the first “open 3T-class model” (I guess they’re round­ing 2.8 tril­lion up to 3 tril­lion), tak­ing the crown from DeepSeek’s 1.6T v4 Pro. Their self-re­ported bench­marks have K3 mostly beat­ing Claude Opus 4.8 max and GPT-5.5 high, while los­ing out to Claude Fable 5 and GPT-5.6 Sol.

A few high­lights from the Artificial Analysis re­port on the model:

“On our pri­vate long-hori­zon knowl­edge work eval­u­a­tion, Kimi K3 reaches an over­all Elo of 1547, +732 points from Kimi K2.6 and be­hind only Claude Fable 5.”

“Cost per task ($0.94) is sim­i­lar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers”

“Kimi K3’s to­ken us­age on the Artificial Analysis Intelligence Index de­creased sig­nif­i­cantly, us­ing 21% fewer out­put to­kens than K2.6.”

The model is also now the lead­ing model on Arena.ai’s Frontend Code arena, sur­pass­ing even Claude Fable 5.

The new model is no­table for the pric­ing: $3/million in­put to­kens and $15/million out­put to­kens, putting it at the same level as Anthropic’s Claude Sonnet se­ries and mak­ing it the most ex­pen­sive model re­leased by a Chinese AI lab to date. This is a sig­nif­i­cant in­crease on their ear­lier mod­els such as Kimi K2.6 at $0.95/$4. 2.8 tril­lion pa­ra­me­ters is also more than twice the size of that 1T model.

But how does it pel­i­can?

I used OpenRouter (to avoid sign­ing up for a Moonshot API key) with the llm-open­router plu­gin to gen­er­ate an SVG of a pel­i­can rid­ing a bi­cy­cle:

llm -m open­router/​moon­shotai/​kimi-k3 ‘Generate an SVG of a pel­i­can rid­ing a bi­cy­cle’

Here’s the tran­script. It looks like this:

That pel­i­can took 95 in­put to­kens and 16,658 out­put to­kens (13,241 were rea­son­ing to­kens), for a to­tal cost of 25 cents!

Since K3 ac­cepts im­age in­put I ran it against that ren­dered SVG above (with my alt text prompt) and got back (for 0.6 cents):

Cartoon il­lus­tra­tion of a white pel­i­can wear­ing a red scarf, rid­ing a red bi­cy­cle along a gray road with white dashed lines; the pel­i­can has a large or­ange beak and webbed or­ange feet ped­al­ing, with white mo­tion lines be­hind it; the back­ground shows a light blue sky with white clouds, a yel­low sun, two small black birds in flight, and green grass with tiny white flow­ers in the fore­ground

Cartoon il­lus­tra­tion of a white pel­i­can wear­ing a red scarf, rid­ing a red bi­cy­cle along a gray road with white dashed lines; the pel­i­can has a large or­ange beak and webbed or­ange feet ped­al­ing, with white mo­tion lines be­hind it; the back­ground shows a light blue sky with white clouds, a yel­low sun, two small black birds in flight, and green grass with tiny white flow­ers in the fore­ground

What can we learn from the pel­i­can?

My Generate an SVG of a pel­i­can rid­ing a bi­cy­cle test is 21 months old now. It was never a par­tic­u­larly great bench­mark. It started out as a joke on how ab­surdly dif­fi­cult it is to com­pare these mod­els, but then for the first year it turned out to have a sur­pris­ing cor­re­la­tion to how good the mod­els ac­tu­ally were.

That con­nec­tion has been mostly sev­ered now. The GPT-5.6 and Claude Fable 5 pel­i­cans are out­classed by GLM-5.2, and much as I love GLM I don’t think that’s a Fable-class model.

(I’m still not con­vinced that labs are train­ing for the bench­mark—if they were, I’d ex­pect much bet­ter re­sults. There’s a chance that Gemini has op­ti­mized for any com­bi­na­tion of an an­i­mal on a ve­hi­cle though!)

The biggest lim­i­ta­tion of the pel­i­can is that it does­n’t touch at all on the thing that mat­ters most for to­day’s model: agen­tic tool call­ing and the abil­ity to op­er­ate tools re­li­ably as con­ver­sa­tions grow in length.

So don’t go us­ing pel­i­cans to com­pare mod­els!

All of that said, I still get a de­cent amount of value out of run­ning the bench­mark my­self.

Firstly, it’s a forc­ing func­tion for ac­tu­ally try­ing the model. If I show you a pel­i­can, that means I’ve man­aged to run a prompt through it. If the model has an of­fi­cial API I’ll use that, if it’s open weight (and small enough to fit a 128GB M5 MacBook Pro) I’ll try run­ning it on my own ma­chine, usu­ally via llama.cpp or LM Studio or Ollama. I’ll fre­quently use OpenRouter since that usu­ally pro­vides a proxy to an of­fi­cial API with­out me need­ing a new API key.

Most of my pel­i­cans are gen­er­ated us­ing my LLM CLI tool, which helps en­cour­age me to en­sure the lat­est mod­els are sup­ported by that (via one of its plu­g­ins).

More im­por­tantly though, even the act of a sin­gle prompt to “Generate an SVG of a pel­i­can rid­ing a bi­cy­cle” can re­veal in­ter­est­ing model char­ac­ter­is­tics.

Consider the re­sult for Kimi K3 to­day. Running those sim­ple prompts helped em­pha­size sev­eral points about the model.

It only has one rea­son­ing ef­fort right now, “max”—and it shows. The model con­sumed 13,241 rea­son­ing to­kens to out­put 3,417 to­kens of re­sponse. This is ex­pen­sive—the pel­i­can cost 25 cents!

How does the prompt “Generate an SVG of a pel­i­can rid­ing a bi­cy­cle” add up to 95 in­put to­kens? OpenAI’s to­k­enizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 to­kens, sug­gest­ing there may be an 85 to­ken hid­den sys­tem prompt. It re­fused to leak it though.

Vision works well: the alt text it gen­er­ated is very good.

K3 cur­rently only has one think­ing ef­fort level, but I’ve been de­riv­ing quite a bit of value re­cently from run­ning the same pel­i­can prompt through dif­fer­ent ef­fort lev­els to get a quick idea for what im­pact those have. Here’s my ma­trix for the GPT-5.6 model fam­ily, for ex­am­ple.

Really though the main things I gain from the pel­i­can test are:

It’s a “hello world” ex­er­cise for prompt­ing a model

A rough cost and rea­son­ing es­ti­mate for a sim­ple task

Confirmation that the model can out­put valid SVG and has a ba­sic idea of geom­e­try and spa­tial aware­ness. This is a much big­ger deal for the smaller mod­els that run on my lap­top.

It’s still in­ter­est­ing to com­pare pel­i­cans be­tween re­leases in the same model fam­ily. K3’s pel­i­can is a no­table im­prove­ment from Kimi 2.5.

It’s some­thing I can share that demon­strates I’ve tried it. Plus a com­ment with a pel­i­can in it is kind of a tra­di­tion on Hacker News at this point, any time I’m late I get com­ments ask­ing where it is!

If you pay at­ten­tion to AI com­pany brand­ing, you’ll no­tice a pat­tern:

Circular shape (often with a gra­di­ent)

Central open­ing or fo­cal point

Radiating el­e­ments from the cen­ter

Soft, or­ganic curves

Sound fa­mil­iar? It should, be­cause it’s also an apt de­scrip­tion of… well, you know.

A but­t­hole.

The cir­cu­lar AI logo epi­demic

If you ever thought that AI com­pany lo­gos look like but­t­holes, you’re not alone.

FastCompany no­ticed this trend in 2023 and pub­lished an ar­ti­cle about it, but (I could only pre­sume) their ed­i­tors and lawyers did­n’t let them ti­tle the ar­ti­cle the way the wanted it to ti­tle, so it got pub­lished with a more safe for work ti­tle: The AI boom is cre­at­ing a new logo trend: the swirling hexa­gon. They also were care­ful not to men­tion any­thing anatom­i­cal.

I don’t have ed­i­tors or lawyers, so let’s take a closer look at some ex­am­ples:

OpenAI’s logo evo­lu­tion

OpenAI’s orig­i­nal logo was a sim­ple, text-based mark. Then came the re­design: a per­fect cir­cle with a sub­tle gra­di­ent and cen­tral void.

OpenAI’s of­fi­cial ex­pla­na­tion is a mas­ter­class in cor­po­rate eu­phemism:

“The Blossom logo is more than just a vi­sual sym­bol; it rep­re­sents the core phi­los­o­phy that guides our ap­proach to de­sign and in­no­va­tion. At its heart, the logo cap­tures the dy­namic in­ter­sec­tion be­tween hu­man­ity and tech­nol­ogy—two forces that shape our world and in­spire our work. The de­sign em­bod­ies the flu­id­ity and warmth of hu­man-cen­tered think­ing through the use of cir­cles, while right an­gles in­tro­duce the pre­ci­sion and struc­ture that tech­nol­ogy de­mands.”

Sure, Sam.

Translation: “We made a cir­cu­lar shape with some an­gles be­cause it looked nice, then wrote flow­ery lan­guage to jus­tify why our but­t­hole-ad­ja­cent de­sign is ac­tu­ally pro­found.”

The flu­id­ity and warmth of hu­man-cen­tered think­ing through the use of cir­cles is per­haps the most el­e­gant way any­one has ever de­scribed mak­ing a logo that re­sem­bles an anus.

The Big AI com­pa­nies

Looking at the lo­gos of the Big AI com­pa­nies, you can see that they al­most all of them have a cir­cu­lar or snowflake-like shape and a cen­tral open­ing.

Only DeepSeek and Midjourney don’t fol­low the trend. Interestingly, both are sea-re­lated.

Smoking gun: Anthropic’s Claude

Up un­til this point, the lo­gos have been sub­tle. You might say that the lo­gos are sim­ply cir­cu­lar and there’s not much more to it.

But Anthropic’s Claude takes it to the next level.

Here’s a side-by-side com­par­i­son with a draw­ing from Kurt Vonnegut’s book “Breakfast of Champions”. I added Claude’s logo be­low for easy com­par­i­son.

Both the draw­ing and the de­scrip­tion in the book are un­am­bigu­ous. This is not just “a cir­cu­lar shape with a gra­di­ent” any­more, is it?

July 2026 up­date: the smok­ing gun now moves

Since pub­lish­ing this ar­ti­cle, I’ve dis­cov­ered new ev­i­dence. Open claude.ai and click on the Claude logo. Just watch what it does:

The Claude logo, when clicked. I have no fur­ther ques­tions.

Every click makes the logo clench and re­lax. It even re­sponds with a slightly an­noyed “Yes, yes. What can I do for you?”, as if you poked some­thing you weren’t sup­posed to.

At this point, there’s no ar­gu­ment you could make that would per­suade me this is not a but­t­hole 🙂

It’s not just AI com­pa­nies

Even tra­di­tional com­pa­nies aren’t im­mune. Here are a few no­table or funny ex­am­ples. But the per­cent­age of AI com­pany lo­gos that look like but­t­holes is still sig­nif­i­cantly higher than than any other in­dus­try.

I es­pe­cially like the Electrolux one. It’s sim­ple, mem­o­rable, and once you see a butt and bikini, you can’t un­see it.

Why does this keep hap­pen­ing?

There are sev­eral fac­tors at play:

Circular de­sign psy­chol­ogy

Circles rep­re­sent whole­ness, com­ple­tion, and in­fin­ity—con­cepts that align with AI’s promise. They’re also friendly and non-threat­en­ing, qual­i­ties com­pa­nies des­per­ately want to pro­ject when sell­ing po­ten­tially job-re­plac­ing tech­nol­ogy.

Unintentional bio­mimicry

The hu­man brain finds fa­mil­iar pat­terns in ran­dom shapes (pareidolia), like a face on Mars, taken by the Viking 1 or­biter and re­leased by NASA in 1976.

But some­times, de­sign­ers in­ad­ver­tently recre­ate bi­o­log­i­cal forms with­out re­al­iz­ing the… anatom­i­cal im­pli­ca­tions.

The copy­cat ef­fect

Once a few ma­jor play­ers adopted the cir­cu­lar sphinc­ter aes­thetic, every­one fol­lowed suit. Now we have an in­dus­try where stand­ing out means look­ing ex­actly like every­one else’s but­t­hole.

Basically, the same rea­son why so many brands change their lo­gos and look like every­one else.

Design by com­mit­tee

Another fac­tor is how these lo­gos are cre­ated. Important cor­po­rate de­ci­sions in­volve many stake­hold­ers. The re­sult is of­ten the safest, most in­of­fen­sive op­tion, the av­er­age of every­one’s opin­ions. In de­sign meet­ings at AI com­pa­nies, con­ver­sa­tions prob­a­bly sound like:

Can we make it more fu­tur­is­tic?

It needs to feel ad­vanced but ap­proach­able.

Let’s add a sub­tle gra­di­ent to con­vey in­tel­li­gence.

No sin­gle per­son sug­gests mak­ing a logo that re­sem­bles an anus, but when every­one’s feed­back gets in­cor­po­rated, that’s what of­ten emerges. Risk aver­sion in cor­po­rate en­vi­ron­ments nat­u­rally pushes de­signs to­ward fa­mil­iar, “safe” ter­ri­tory, which ap­par­ently means anatom­i­cal open­ings.

What this says about tech brand­ing

This phe­nom­e­non re­veals some­thing deeper about the tech in­dus­try: the fear of stand­ing out too much. Despite claims of in­no­va­tion and dis­rup­tion, there’s tremen­dous pres­sure to look le­git­i­mate by con­form­ing to es­tab­lished vi­sual lan­guage.

When OpenAI’s sphinc­ter-like logo be­came suc­cess­ful, it cre­ated a tem­plate that said, “This is what se­ri­ous AI looks like.” Now, any new AI com­pany that does­n’t re­sem­ble a col­or­ful anatom­i­cal open­ing risks be­ing seen as un­se­ri­ous or un­pro­fes­sional.

Tech de­sign trends through his­tory

This is­n’t the first time tech de­sign has gone through a con­for­mity phase. Consider these pre­vi­ous waves:

1990s-2000s: 3D and Glossy - Remember when every logo needed a drop shadow and a glassy shine? Apple’s aqua in­ter­face set the stan­dard.

2010 – 2013: Skeuomorphism - Digital de­signs mim­ic­k­ing phys­i­cal ob­jects, with stitched leather tex­tures and re­al­is­tic di­als.

2013 – 2018: Flat Design - Reaction to skeuo­mor­phism brought min­i­mal, clean in­ter­faces with bright col­ors and no shad­ows.

2018 – 2022: Neomorphism - Soft shad­ows and semi-flat de­sign cre­at­ing sub­tle, “touchable” in­ter­faces.

2022-Present: The Butthole Era - Circular gra­di­ents with cen­tral fo­cal points dom­i­nat­ing AI brand­ing.

Each era started with in­no­va­tions that were quickly copied un­til the in­dus­try reached sat­u­ra­tion point and moved on to the next trend.

Logos be­come in­creas­ingly in­ter­change­able (one of the bags is real, but they all look the same)

Historic logo dis­as­ters: You’re not alone

AI com­pa­nies can take some com­fort in know­ing they’re not the first to face un­in­tended anatom­i­cal com­par­isons. Logo his­tory is filled with dis­as­ters but to keep this con­sis­tent with the theme of the ar­ti­cle, here’s a cou­ple of them.

Zune logo, when flipped, says some­thing dif­fer­ent. Maybe that’s one of the rea­sons why iPod won?

Brazilian Institute of Oriental Studies is a styl­ized pagoda sil­hou­et­ted against the set­ting sun. Or so the de­sign­ers wanted it to look. The fi­nal re­sult was much more… anatom­i­cal. They since changed it to some­thing less sug­ges­tive.

Maybe com­pa­nies should have a panel of “middle school­ers” on their pay­roll to re­view lo­gos be­fore launch. They’ll find every pos­si­ble in­ap­pro­pri­ate in­ter­pre­ta­tion with ruth­less ef­fi­ciency.

Breaking free from the but­t­hole

For com­pa­nies brave enough to dif­fer­en­ti­ate, here are some al­ter­na­tives:

Embrace sharp an­gles - geo­met­ric shapes with de­fined edges cre­ate a dis­tinct vi­sual iden­tity

Use neg­a­tive space cre­atively - think FedEx ar­row, not bi­o­log­i­cal open­ings

Avoid ra­dial sym­me­try - not every­thing needs to be per­fectly cir­cu­lar

Skip the gra­di­ent - flat de­sign still works

Test with di­verse au­di­ences - if five dif­fer­ent peo­ple in­de­pen­dently say “that looks like a but­t­hole,” it prob­a­bly does (show it to teenagers if you want to un­cover even the most sub­tle anatom­i­cal im­pli­ca­tions)

Conclusion

Does this mean AI com­pa­nies should change their brand­ing? Not nec­es­sar­ily. The fa­mil­iar­ity clearly works in build­ing trust. But per­haps the next wave of AI in­no­va­tion could be ac­com­pa­nied by some vi­sual in­no­va­tion too.

For com­pa­nies look­ing to break the mold, con­sider these ap­proaches that suc­cess­ful tech brands have used:

Embrace mean­ing­ful ab­strac­tion - Slack’s hash­tag-in­spired logo com­mu­ni­cates col­lab­o­ra­tion with­out cir­cu­lar clichés

Leverage let­ter­forms - Netflix’s sim­ple “N” has be­come in­stantly rec­og­niz­able with­out anatom­i­cal con­fu­sion

Tell a story - Stripe’s dis­tinc­tive par­al­lel lines re­flect pay­ment flows mov­ing seam­lessly

Use dis­tinc­tive color com­bi­na­tions - Twitch’s pur­ple brand­ing stands out in a sea of blue tech lo­gos

The chal­lenge for the next gen­er­a­tion of AI com­pa­nies is­n’t just tech­no­log­i­cal - it’s find­ing vi­sual lan­guage that com­mu­ni­cates in­no­va­tion with­out re­sort­ing to the same tired sphinc­ter-in­spired pat­terns.

PS. This post is meant to be hu­mor­ous, but let’s not pre­tend there is­n’t a se­ri­ous point here about the de­press­ing same­ness in mod­ern de­sign. No ac­tual anuses were con­sulted dur­ing this re­search, though sev­eral de­sign­ers were clearly think­ing about them.

If you like what you see, you’ll find more stuff like this on my Twitter.

If you like what you see, you’ll find more stuff like this on my Twitter.

Hello! I’ve been work­ing on a Django site re­cently, and I de­cided to use SQLite as the data­base. When I was get­ting started with us­ing SQLite as data­base for a web­site I read a bunch of blog posts about how it is to­tally fine to use SQLite in pro­duc­tion for a small site and I think it is to­tally fine, but what I did not fully ap­pre­ci­ate is that SQLite is still a data­base, data­bases are com­pli­cated, and I do not know a lot about op­er­at­ing data­bases.

So here are a cou­ple of small things I’ve been learn­ing about run­ning SQLite. This is the 4th web­site I’ve used SQLite for, and I think this one is harder be­cause with the power of the Django ORM I’ve been mak­ing the data­base do more work than I was pre­vi­ously with­out Django.

I started by turn­ing on WAL mode like all the blog posts said to do and hop­ing for the best.

ANALYZE is ap­par­ently im­por­tant

Today I was run­ning a query (using SQLite’s FTS5 for full-text search) on a table with 4000 rows and it took 5 sec­onds. That seemed wrong to me: com­put­ers are fast!

It turned out that what I needed to do was to run ANALYZE! Immediately the prob­lem query went from tak­ing 5 sec­onds to like 0.05 sec­onds (or some other num­ber small enough that I did­n’t care to in­ves­ti­gate fur­ther). I still don’t know ex­actly what went wrong in the query plan, but my best guess is that it was some sort of ac­ci­den­tally qua­dratic thing.

ANALYZE gen­er­ates “statistics” (I guess about the num­ber of rows in each table? and pre­sum­ably other things?) so that the query plan­ner can make bet­ter choices.

Maybe one day I’ll learn to read a query plan.

clean­ing up the data­base is tricky

Occasionally I’ve run into sit­u­a­tions where I ac­ci­den­tally put a bunch of rows in my data­base that I don’t want to be there (for ex­am­ple com­pleted tasks from django-tasks-db), and I want to clean them up.

What’s hap­pened to me a few times in this case is:

I run some kind of com­mand to clean up the rows

The com­mand takes more than 5 sec­onds, since there are a lot of rows (though I still have some ques­tions about why these DELETE state­ments are so slow hon­estly, maybe there’s a bunch of Python code run­ning in­side a trans­ac­tion, I’m not sure)

One of the other work­ers tries to write the data­base while this is hap­pen­ing, and times out af­ter 5 sec­onds (I have a time­out of 5 sec­onds set)

The worker crashes be­cause it could­n’t write to the data­base and the VM shuts down

My ap­proach so far has been to just do these cleanup op­er­a­tions in small batches so that I don’t need to do data­base queries that take more than 5 sec­onds to run. This whole ex­pe­ri­ence has given me more of an ap­pre­ci­a­tion for why some­one might want to use a “real” data­base like Postgres which can have more than one writer at the same time though.

Maybe in the fu­ture I’ll just take the site down for sched­uled main­te­nance in­stead when I need to do this kind of thing, but I haven’t fig­ured out a work­flow for that yet.

no notes on per­for­mance of ORM queries yet

So far I’ve been us­ing Django’s ORM to make any query I want with­out pay­ing any at­ten­tion at all to query per­for­mance and it’s mostly been go­ing okay other than the ANALYZE thing. The data­base is pretty small (maybe 10000 rows?) and I ex­pect it to stay pretty small for­ever, so I’m hop­ing that that plan will keep work­ing.

back­ing up sqlite

I’ve done SQLite back­ups a cou­ple of ways. I don’t think I’ve ac­tu­ally tested restor­ing from my back­ups but I do usu­ally try to mon­i­tor them with a dead man’s switch.

way 1: restic

sqlite3 /data/calendar.db “VACUUM INTO ‘/tmp/calendar.sqlite’” gzip /tmp/calendar.sqlite

# Upload backup to S3 # Sometimes the backup gets OOM killed and so it stays locked, do an un­lock restic -r s3://s3.ama­zon­aws.com/​some_bucket/ un­lock # Do the backup & prune old back­ups restic -r s3://s3.ama­zon­aws.com/​some_bucket/ backup /tmp/calendar.sqlite.gz restic -r s3://s3.ama­zon­aws.com/​some_bucket/ snap­shots restic -r s3://s3.ama­zon­aws.com/​some_bucket/ for­get -l 1 -H 6 -d 2 -w 2 -m 2 -y 2 restic -r s3://s3.ama­zon­aws.com/​some_bucket/ prune

way 2: litestream

I started try­ing out Litestream re­cently be­cause I felt like do­ing in­cre­men­tal back­ups might be more ef­fi­cient: my restic back­ups were some­times get­ting OOM killed, and I was a bit tired of it. Basically I just write a con­fig file and run:

litestream repli­cate -config litestream.yml

I set re­ten­tion: 400h in my con­fig file in an at­tempt to re­tain some amount of his­tory of the data­base but I have no idea if it works.

I’ve been back­ing up to AWS, which is al­ways a pain be­cause it’s an­noy­ing to nav­i­gate the AWS con­sole to gen­er­ate cre­den­tials. Maybe one day I’ll move away to some other S3-compatible al­ter­na­tive.

you can use mul­ti­ple data­bases

My cur­rent pro­ject only has one data­base, but one trick I used with Mess with DNS was to split the ta­bles into three sep­a­rate data­base files be­cause I did­n’t ac­tu­ally need my ta­bles to be in the same db. I think it was help­ful.

Mess with DNS has been run­ning on SQLite for 4 years now (since 2022) and it’s been great, I think the move from Postgres was a great choice for that pro­ject.

that’s all!

It’s al­ways kind of fun to see how long it takes me to learn sort of ba­sic things about the tech­nolo­gies I’m us­ing. I think I used SQLite for a web pro­ject for the first time in 2022 and I only learned that ANALYZE ex­isted to­day! I imag­ine in a year or two I’ll be learn­ing about some other very ba­sic fea­ture.

some ref­er­ences

Some blog posts I’ve looked at, other than the of­fi­cial docs:

The de­fin­i­tive guide to us­ing Django with SQLite in pro­duc­tion

a gist on sqlite per­for­mance tun­ing

Introduction

As of writ­ing, the Zilog Z80 proces­sor was of­fi­cially launched 50 years ago, in July of 1976, less than 4 years af­ter the last hu­man had walked on the moon, decades closer to WWII than to the pre­sent day, roughly at a half way point be­tween the Kennedy as­sas­si­na­tion and the fall of the Berlin wall, closer to the Korean war than to 9/11 which is it­self an event that hap­pened a quar­ter of a cen­tury ago. (Sorry…)

The proces­sor was ex­tremely suc­cess­ful, be­ing used in many 8 bit mi­cro­com­put­ers, in­clud­ing early per­sonal com­put­ers, home & hobby com­put­ers, as well as many em­bed­ded, in­dus­trial ap­pli­ca­tions.

Together with the 8080 & 8085 that it is bi­nary com­pat­i­ble with, it con­tributed to cre­at­ing a de facto hard­ware stan­dard for 8 bit mi­cros, al­low­ing a de facto soft­ware stan­dard of CP/M, and Microsoft BASIC.

The Z80 it­self also spawned many clones and de­rived ar­chi­tec­tures over the years, fa­mously in­clud­ing the Sharp LR35902, used in the orig­i­nal GameBoy. Zilog them­selves even­tu­ally gave up their line of 16 and 32 bit de­rived ar­chi­tec­tures and re­turned to Z80 based mi­cro­con­trollers and vari­ants like the pipelined and higher clocked eZ80, mainly for con­tin­ued use in in­dus­trial ap­pli­ca­tions.

I my­self am much too young to have seen the home com­put­ing side of this (ignoring the afore­men­tioned GameBoy), but the wide­spread use in in­dus­trial ap­pli­ca­tions means that the orig­i­nal Z80 is still around and in use with Zilog fi­nally dis­con­tin­u­ing it mere 2 years ago.

My own first en­counter with the Z80 was as a late teenager, when I was brows­ing an elec­tron­ics com­pany cat­a­log, sur­prised to find them still be­ing sold. I de­signed my own lit­tle Z80 com­puter and con­vinced a school teacher to let me use the photo lab at night, so I could etch some PCBs.

As sev­eral of my for­mer teach­ers got cu­ri­ous what I was up to, I ended up hear­ing a lot of in­ter­est­ing anec­dotes about old home com­put­ers, con­soles and a story about DIY wire wrap com­puter in a Tupperware box, run­ning CP/M and WordStar, hooked up to a “borrowed” IBM ter­mi­nal that was used to write a the­sis on. Over time I ended up be­ing gifted a num­ber of old chips from dusty draw­ers that made it into my own DIY pro­ject, in­clud­ing a bunch of MCS-85 parts, sev­eral Z80s, 8085s, 6502s and 6522s.

The whole thing sure taught me a num­ber of in­ter­est­ing lessons about sys­tems en­gi­neer­ing and some un­ex­pected ones (reliable power-on re­set is sur­pris­ingly hard; writ­ing a linker is a lot harder than writ­ing an as­sem­bler, writ­ing a com­piler is some­thing you can ac­tu­ally do).

Anyway, that is my claim to be­ing al­lowed to rem­i­nisce about the Z80. While I orig­i­nally wanted to limit my­self to some tech­ni­cal de­tails based on my own ex­pe­ri­ence, com­par­ing the Z80 with the 8080 that it was de­rived from, I ended up div­ing down a rab­bit hole of the Computer History Museums oral his­tory panel, where the peo­ple in­volved re­called even more anec­dotes about the de­vel­op­ment of those chips. The whole “I’ll try to write a blog post (again)” idea quickly bal­looned in scope.

From the 2200 to the 8008

Once upon a time, the

Computer Terminal Corporation (CTC) built a new, pro­gram­ma­ble ter­mi­nal, the Datapoint 2200, sport­ing an 8 bit proces­sor con­structed from in­di­vid­ual TTL chips. Intel was sup­ply­ing CTC with shift reg­is­ters and mem­ory chips at the time.

The idea was floated to re­place parts of TTL ceme­tery with cus­tom ICs, even­tu­ally it was con­sid­ered to try and get the en­tire 8 bit CPU on a sin­gle chip. Two dif­fer­ent com­pa­nies were ul­ti­mately con­tracted for this task: Texas Instruments and Intel.

Neither com­pany fin­ished their de­sign in time. When Intel had the chip ready, orig­i­nally named 1201 based on a sys­tem­atic nam­ing con­ven­tion, CTC were al­ready sell­ing ter­mi­nals based on the TTL de­sign.

Engineers at CTC were also un­sat­is­fied with the per­for­mance of the chips and they had al­ready made changes to the ar­chi­tec­ture for the next gen­er­a­tion of the ter­mi­nal any­way.

While TI ul­ti­mately canned their de­sign, Intel went ahead and suc­cess­fully com­mer­cial­ized their ver­sion as the 8008 (like the 4004, re­named by mar­ket­ing).

The 8008 Architecture

___ ___ -9V –-|1 |_| 18|<- IRQ AD7 <->|2 17|<- READY AD6 <->|3 16|<- CLK1 AD5 <->|4 15|<- CLK2 AD4 <->|5 14|-> SYNC __ AD3 <->|6 13|-> S0 | AD2 <->|7 12|-> S1 > State AD1 <->|8 11|-> S2 __| AD0 <->|9 10|– +5V |_________|

The 8008 has 7 reg­is­ters: A, B, C, D, E, H, L. Where A is the des­ig­nated ac­cu­mu­la­tor, the oth­ers can be used as operands or scratch. As the name might im­ply, H and L to­gether form the High and Low part of a mem­ory pointer. Accessing mem­ory is done through an 8th pseudo reg­is­ter M, rep­re­sent­ing the mem­ory byte that HL points to.

The proces­sor in­ter­nally keeps track of ALU state (Cary, Parity, Zero, Sign) in a few flag bits on which it can per­form con­di­tional jumps (including call and re­turn).

The pro­gram counter PC is pretty much never vis­i­ble di­rectly. There are ded­i­cated func­tion call & re­turn in­struc­tions, but the proces­sor uses an in­ter­nal re­turn ad­dress stack that is 8 lev­els deep. The rea­son for this was that the Datapoint 2200 was orig­i­nally sup­posed to use se­r­ial mem­ory, a call stack in mem­ory was con­sid­ered to end up a per­for­mance bot­tle neck.

Memory ad­dresses are 14 bits wide, there is a sep­a­rate I/O ad­dress space with a to­tal of 32 I/O ports (the ad­dresses are al­ways im­me­di­ate and bit-stuffed into the op­code).

For in­ter­rupt han­dling, there is a spe­cial “restart” in­struc­tion that es­sen­tially calls into 1 of 8 slots (0x00, 0x08, 0x10, 0x18, …, 0x38) at the be­gin­ning of the ad­dress space. The slot in­dex is bit-stuffed into the RST op­code it­self. When an in­ter­rupt oc­curs, the proces­sor sig­nals to the pe­riph­ery that it got the hint and then blindly ex­e­cutes the cur­rent con­tents of the data bus that bet­ter be an RST in­struc­tion.

From there, it gets a bit tricky. The CPU does not have a gen­eral pur­pose stack that it can safe reg­is­ters to, all mem­ory ac­cess needs HL, but you don’t want to clob­ber HL in the in­ter­rupt han­dler. The in­tended way to solve this was through ex­ter­nal latches on the I/O bus, serv­ing as scratch reg­is­ters.

All in all, the ar­chi­tec­ture is fairly sim­plis­tic, re­quir­ing about 3500 tran­sis­tors and used a DIP18 pack­age. Address and data were mul­ti­plexed, re­quir­ing ex­ter­nal latch­ing. Internal de­code/​ex­e­cu­tion state was ex­posed that needed to be de­coded to drive latches and fig­ure out what the proces­sor is at­tempt­ing to do (read from or write to mem­ory, or the I/O bus).

The proces­sor needed two phase-shifted clock sig­nals (it ran at 500kHz), a +5V pos­i­tive sup­ply and -9V neg­a­tive sup­ply.

From the 8008 to the 8080

The short­com­ings of the Datapoint 2200 de­rived 8008 ar­chi­tec­ture were known dur­ing de­vel­op­ment, and in typ­i­cal en­gi­neer­ing fash­ion, be­fore de­vel­op­ment was even wrapped up, ideas were thrown around for an im­proved ar­chi­tec­ture.

Federico Faggin, who was brought over from the 4004 pro­ject, was push­ing to start work on an im­proved ver­sion, but man­age­ment in­sisted to first see how the mar­ket would re­act to their two mi­cro­proces­sors. Competitors even­tu­ally an­nounced their own 8 bit de­signs in the mak­ing and the de­lays ended up cost­ing Intel a to­tal of 9 months of their lead time.

Even be­fore the pro­ject was fi­nally ap­proved, Federico Faggin got ap­proval to hire Masatoshi Shima away from Busicom to work on the 8080 de­sign. In many ways sim­i­lar to how CTC had a hand in the de­vel­op­ment of the 8008, Busicom was in­volved in the de­vel­op­ment of the 4004, orig­i­nally want­ing a set of cus­tom chips for their cal­cu­la­tors.

Criticism and feed­back from po­ten­tial cus­tomers that the 8008 was demon­strated to, also in­flu­enced the de­sign of the 8080, and it was de­cided early on to set aside bi­nary com­pat­i­bil­ity.

The 8080 Architecture

The 8080 has in essence the same reg­is­ter set as the 8008, but it re­places the in­ter­nal re­turn ad­dress stack with an ex­ter­nal one that lives in mem­ory and is ac­cessed via a stack pointer reg­is­ter (SP).

The stack pointer can be ex­changed or moved in to/​out of HL, reg­is­ters can be pushed on or popped of the stack pair­wise. Besides HL, the other reg­is­ter pairs are BC, DE, and AF (accumulator and ALU flags), but the 8080 as­sem­bly prefers to call the later the “program sta­tus word” PSW.

Memory ad­dresses are bumped up to full 16 bits, giv­ing the ma­chine a 64k ad­dress space. The I/O ports are bumped up to 256. BC and DE now also al­low rudi­men­tary in­di­rec­tion (loading/storing the ac­cu­mu­la­tor), the ac­cu­mu­la­tor and HL can be loaded/​stored at an im­me­di­ate des­ti­na­tion.

A few dou­ble-byte arith­metic op­er­a­tions are added that can work on reg­is­ter pairs (e.g. in­cre­ment/​decre­ment), mainly to al­low pointer arith­metic and 16 bit coun­ters. Using those in­struc­tions on AF would ac­tu­ally act on SP in­stead.

Interrupt han­dling works much the same way, us­ing restart in­struc­tions, but with the added fea­ture that in­ter­rupts can be en­abled/​dis­abled in soft­ware. An ex­plicit stack also no longer re­quires sav­ing reg­is­ters us­ing I/O hard­ware.

Here is a slightly mod­i­fied mem­ory-copy ex­am­ple from the Wikipedia page, it copies a num­ber of bytes (stored in BC) from DE to HL.

mem­cpy: PUSH B  ; pushes BC PUSH D  ; pushes DE PUSH H  ; pushes HL

loop: LDAX D  ; A := *(DE) MOV M, A  ; *(HL) := A INX D  ; ++DE INX H  ; ++HL DCX B  ; –BC

MOV A, B  ; A := B ORA C  ; A |= C JNZ loop  ; jump if not zero

POP H POP B POP D RET

A few things that are note­wor­thy here: in­struc­tions that act on reg­is­ter pairs al­ways use a sin­gle reg­is­ter as a mnemonic for both, the “X” ion the “INX” dif­fer­en­ti­ates the dou­ble-byte in­cre­ment from an “INC” on a sin­gle reg­is­ter byte. The to Intel 8080 as­sem­bly has an al­most 1:1 map­ping of mnemon­ics to op­codes and is ex­tremely easy to parse, mak­ing an as­sem­bler easy to im­ple­ment. To a de­gree, this comes at the ex­pense of hu­man read­abil­ity.

Furthermore, the dou­ble byte arith­metic has no in­flu­ence on the ALU flags, it is a sep­a­rate, in­de­pen­dent func­tion block. After decre­ment­ing BC, we need to man­u­ally check if both reg­is­ters are zero.

Electrical Interfacing

To im­prove speed, the 8080 used NMOS logic. The down­side of this was the the CPU now needed 3 dif­fer­ent sup­ply volt­ages (-5V, +5V and +12V). It also stuck with us­ing 2 phase-shifted clock sig­nals (in the 9V to 12V range), mak­ing elec­tri­cal de­sign around the chip a bit cum­ber­some.

Thanks to a 40 pin pack­age (something Faggin re­calls as an up­hill bat­tle to get ap­proved), the CPU no longer had to mul­ti­plex data and ad­dress lines. But like the 8008, it still ex­posed in­ter­nal proces­sor state ex­ter­nally, mul­ti­plex­ing the ac­tual con­trol states on the data bus, re­quir­ing ex­ter­nal latch­ing and de­cod­ing.

Intel would of course sell sup­port chips for state de­cod­ing, clock gen­er­a­tion and so on. Additionally, one might also want to buy an ac­com­pa­ny­ing in­ter­rupt con­troller and some­thing like a pro­gram­ma­ble in­ter­val timer, at the very least to drive DRAM re­fresh, pos­si­bly us­ing one of those handy Intel DMA con­trollers.

Intel ad­dressed at least some of those short com­ings with the 8085, only re­quir­ing a sin­gle 5V sup­ply and a sin­gle 5V clock sig­nal. The freed up pins ex­pose a few ad­di­tional con­trol sig­nals. But still need­ing some spe­cial­ized sup­port chips.

Zilog and the Z80

Dissatisfied with his ex­pe­ri­ence at Intel, the de­lays and up­hill bat­tles with man­age­ment to even get the 8080 pro­ject ap­proved in the first place, Federico Faggin fi­nally de­cided to quit Intel and start his own firm to­gether with Ralph Ungermann, then head of the mi­cro­proces­sor di­vi­sion.

Originally some­what di­rec­tion­less, Faggin at first con­sid­ered de­sign­ing a mi­cro­con­troller, but re­al­ized that the mar­gins were too tight to make it eco­nom­i­cal for a fa­b­less semi­con­duc­tor startup.

He even­tu­ally set­tled on de­sign­ing an im­proved ver­sion of the 8080, nick­named “Super 80”, later be­com­ing the Zilog Z80. They se­cured fund­ing from Exxon and also brought over Masatoshi Shima from Intel to work on the de­sign, later in­creas­ing the size of the team to a to­tal of 11 peo­ple to work on lay­out­ing, soft­ware sim­u­la­tion, and so on.

The de­sign of the Z80 was in­tended to be bi­nary com­pat­i­ble with the 8080, while adding reg­is­ters, ad­dress­ing modes, new in­struc­tions, draw­ing in­spi­ra­tion from other con­tem­po­raries like the 6800. It also aimed at sim­pler elec­tri­cal in­ter­fac­ing and higher speed than the 8080.

The en­tire de­vel­op­ment of the proces­sor up to the first, work­ing pro­to­types cost roughly $400k, fin­ish­ing on time and un­der bud­get (they had se­cured $500k from Exxon).

Zilog re­lied on Mostek to man­u­fac­ture their proces­sors (after some hos­til­i­ties with Synertek they ini­tially con­tracted with). They did even­tu­ally se­cure fur­ther fund­ing from Exxon to build their own fab, but kept sec­ond sourc­ing the Z80.

The Z80 Architecture

The Z80 is fully bi­nary com­pat­i­ble with the 8080 in­struc­tion set.

Inspired by the 6800, it adds two in­dex reg­is­ters IX and IY that can be used in place of HL (same en­cod­ing but with an op­code pre­fix) and with an im­me­di­ate off­set.

The AF, BC, DE and HL reg­is­ter pairs are bank switched, al­low­ing sim­pler & faster in­ter­rupt han­dling.

Speaking of in­ter­rupts, the Z80 has 3 dif­fer­ent ways that it can han­dle them: An 8080 com­pat­i­ble way (mode 0), one that al­ways calls to a fixed lo­ca­tion (mode 1), and one that dis­patches through a call table (mode 2), us­ing the num­ber on the bus as an in­dex. An ex­tra reg­is­ter is used to lo­cate the base of the table in mem­ory.

The Z80 also adds a bunch of bit ro­tate, bit test & set in­struc­tion in­struc­tions, BCD arith­metic, along with built-in loop in­struc­tions (using BC as a counter), self-re­peat­ing block trans­fer, block com­pare and string op­er­a­tions.

Because Intel claimed a copy­right on the as­sem­bly mnemon­ics, the Z80 ended up us­ing its own as­sem­bly lan­guage with an ar­guably cleaner syn­tax. The Z80 as­sem­bly ex­presses operands more ex­plic­itly and uses over­loaded vari­ants of ba­sic mnemon­ics.

This is es­sen­tially the same pro­gram from above, in the more ex­pres­sive Z80 as­sem­bly:

mem­cpy: PUSH BC  ; full name of the reg­is­ter pair PUSH DE PUSH HL

loop: LD A, (DE)  ; ex­plicit 2 ar­gu­ment syn­tax LD (HL), A INC DE INC HL DEC BC

LD A, B  ; over­loaded name OR C JP NZ, loop  ; over­loaded name, con­di­tion is an ar­gu­ment

POP HL POP BC POP DE RET

Of course, on the Z80, the en­tire byte copy loop could also be re­placed with a sin­gle, self-re­peat­ing in­struc­tion: LDIR.

Improved Bus Design

The Z80 only needs a sin­gle 5V sup­ply and a sin­gle clock sig­nal. Many of the ex­ter­nally latched/​de­coded states of the 8080 are ex­plic­itly ex­posed by the chip, such as as MREQ or IORQ to in­di­cate mem­ory or I/O ac­cess, RD/WR sig­nals that in­di­cate ex­actly what the name sug­gests, and an M1 sig­nal that in­di­cates the cur­rent mem­ory ac­cess is an in­struc­tion fetch. Those sig­nals can be con­nected to some­thing as sim­ple as a sin­gle 74xx138 to drive an (E)EPROM, some form of RAM and an UART con­troller. Connect the ad­dress and data lines di­rectly to the Z80 and you es­sen­tially have a work­ing com­puter!

If the RAM in use hap­pens to be DRAM, the Z80 can also take care of DRAM re­freshes, us­ing an in­ter­nal re­fresh counter that it puts on the ad­dress bus dur­ing in­struc­tion de­cod­ing cy­cles, as­sert­ing a con­trol line to tell the ex­ter­nal de­cod­ing logic to do a DRAM re­fresh.

With in­ter­rupt mode 1, where the CPU al­ways calls to a hard wired lo­ca­tion, a sim­ple de­sign can get by with­out any ex­ter­nal in­ter­rupt con­troller, hook­ing a sin­gle de­vice up to the in­ter­rupt pin or us­ing some­thing as sim­ple as a 74xx148 (priority en­coder) and a latch.

How the Story Continued

Even be­fore the Z80 was fi­nally re­leased in July of 1976, rough de­sign work on the 16 bit Z8000 ar­chi­tec­ture had al­ready be­gun. The Z8000 was re­leased in 1979, af­ter the Intel 8086, but be­fore the Motorola 68000.

Like the 8086, it used seg­mented mem­ory, but un­like the 8086 ex­posed a seg­ment num­ber on the bus that an ex­ter­nal MMU chip was sup­posed to con­vert to a lin­ear ad­dress (and check bounds & per­mis­sions).

While com­mon her­itage of the 8080 can be clearly seen in the 8086 in­struc­tion set, a num­ber of fea­tures from the Z80 were also car­ried over, such as the self re­peat­ing block and string op­er­a­tion or loop in­struc­tions. The de­sign of the Z8000 MMU also in­flu­enced the de­scrip­tor table based de­sign of the 286′s 16 bit pro­tected mode.

Despite the fact that Zilog aimed their prod­ucts for a com­puter ori­ented mar­ket, even at an early time when mi­cro­proces­sors were con­sid­ered logic re­place­ment, their ties with Exxon ended up be­ing one of the rea­sons why IBM ul­ti­mately de­cided against a Zilog proces­sor for their PC, opt­ing for an Intel 8088 in­stead.

Part of the rea­son Exxon was in­ter­ested in Zilog in the first place was their in­tent to build up a com­put­ing em­pire of their own, ri­val­ing IBM. They had a num­ber of other com­pa­nies they strate­gi­cally in­vested in (e.g. type­writer, word proces­sor or printer man­u­fac­tur­ers), some of whom even de­signed prod­ucts around Zilog parts, all eat­ing away at the mar­ket share of com­pet­ing IBM prod­ucts.

The close ties with Exxon even­tu­ally also caused fric­tion be­tween Faggin and Ungermann, the later leav­ing Zilog be­fore it be­came a full Exxon sub­sidiary in 1980.

Zilog even­tu­ally split off of Exxon again in 1989 and went pub­lic in 1991, sub­se­quently chang­ing hands sev­eral times, bounc­ing around be­tween pri­vate eq­uity and ac­tual elec­tron­ics com­pa­nies, cur­rently owned by Littelfuse.

The Z80, af­ter a long life as an em­bed­ded proces­sor, was even­tu­ally dis­con­tin­ued in June of 2024.

Links and References

Ken Shirriff, Tracing the roots of the 8086 in­struc­tion set to the Datapoint 2200 mini­com­puter

Ken Shirriff, The Texas Instruments TMX 1795: the (almost) first, for­got­ten mi­cro­proces­sor

CHM, oral his­tory panel, Intel 8008

CHM, oral his­tory panel, Intel 8080

CHM, oral his­tory panel, Intel 386

CHM, oral his­tory panel, Zilog Z80

CHM, oral his­tory panel, Zilog Z8000

This past Saturday, @pushcx and I de­ployed the SQLite pull re­quest to pro­duc­tion. We were wait­ing till this morn­ing to see how it would re­act to the Monday traf­fic spike be­fore mak­ing this post. Needless to say, SQLite seems to have passed with fly­ing col­ors: cpu us­age is down, mem­ory us­age is down, site seems to be snap­pier at least for me, 1/2 the vps cost once mari­adb vps is taken down, and fi­nally “We’re hav­ing a quiet Monday.”. Finally #539 Migrate to SQLite was closed this morn­ing.

Let us know if you have any ques­tions about the mi­gra­tion.

Background Story:

I got in­volved with this mi­gra­tion be­cause back in 2019 I stum­bled upon #539 and be­cause I had lots of ex­pe­ri­ence work­ing with, man­ag­ing and mi­grat­ing lar­gish data­bases, I left a com­ment sug­gest­ing MySQL as an al­ter­na­tive, be­cause of the com­pat­i­bil­ity be­tween MariaDB and MySQL. At that time I was­n’t plan­ning on get­ting in­volved since there were al­ready con­ver­sa­tions in place to mi­grate to PostgreSQL.

Fast for­ward to 2025, Rahul left a com­ment men­tion­ing K1′s ac­qui­si­tion of MariaDB. A dis­cus­sion around the de­tails of mi­grat­ing to post­gresql pro­ceeded. Then in February, Rahul asked “Can lob­sters run on sqlite? which in­cluded a very de­tailed post around SQLite.

I of­fi­cially showed in­ter­est in tak­ing on this pro­ject in June 2025. I think this some­how got men­tioned in lob­sters of­fice hours but it has been so long since then that I don’t re­meme­ber for cer­tain.

In August 2025 I opened my first pull re­quest at­tempt when I got busy and could­n’t at­tend to the PR. Github closed it as stale and I could­n’t re­open it so I opened an­other PR. The sec­ond PR at­tempt in­cluded some per­for­mance test­ing, a data­base x to data­base y script (since none of the ex­ist­ing mari­adb/​mysql to sqlite scripts sat­is­fied me), de­bug­ging and think­ing around data in­tegrity.

Then came the first de­ploy on Feb 21st. @pushcx and I got on a call, came up with a check­list for the de­ploy­ment. Everything went right up un­til the de­ploy­ment of the PR. Once de­ployed the site was in read­only mode, but just the read­only traf­fic was spik­ing all the cpus to 100%. We could­n’t fig­ure out what the prob­lem was so we de­cided to re­vert. I did­n’t feel great af­ter that first failed de­ploy since I knew that per­for­mance could be a prob­lem due to not hav­ing ac­cess to the pro­duc­tion data­base.

Two days af­ter the failed de­ploy I opened the 3rd and fi­nal pr at­tempt. I fixed some mi­nor is­sues with search that were dis­cov­ered dur­ing the failed de­ploy, cre­ated a bulk data cre­ation script which took a week to get half of lob­sters’ data set size cre­ated lo­cally, and com­mit­ted the three changes that fixed the per­for­mance is­sues dur­ing the first de­ploy: 1, 2, 3. The per­for­mance is­sues boiled down to SQLite do­ing full table scans on the largest ta­bles in the data­base for 2 of queries and the 3rd one solved an n+1 is­sue. During the morn­ing of the sec­ond de­ploy­ment, I also added a slow query log just in case there were more per­for­mance is­sues dur­ing the de­ploy­ment.

Then came the sec­ond de­ploy on July 11th. @pushcx and I got on a morn­ing call and came up with a de­ploy­ment and re­vert check­list. Everything was go­ing smoothly and then @pushcx merged and de­ployed the PR. Once de­ployed the site was still live and the cpu/​mem­ory us­age was still good. This was a big re­lief for me. We mon­i­tored site met­rics and irc for peo­ple men­tion­ing is­sues, which a cou­ple did and those were promptly fixed: 1, 2. Overall the site seemed to work so we wrapped up the call and waited till Monday when the traf­fic spikes.

Monday came and the site is stil happy so we’re call­ing this a win and mov­ing on.

SQLite lessons:

The SQLite gem sup­ports user de­fined func­tions (udfs) and we used it to im­ple­ment some miss­ing func­tions in SQLite like reg­exp, if and std­dev so that we would­n’t have to deal with too many sql mi­gra­tion workarounds.

SQLite does­n’t sup­port un­signed big­ints. Previously, the mari­adb was us­ing un­signed big­ints for cer­tain ids, so we had to switch those to big­ints for the mi­gra­tion.

Collation in SQLite is rather weak com­pared to MariaDB. Lobste.rs used ut­f8m­b4_­gen­er­al_ci in MariaDB, but used NOCASE in SQLite. The down­side of NOCASE is that it only sup­ports ASCII char­ac­ters, not the full UTF case fold­ing.

Use the pre­ferred Contentless-Delete Tables in SQLite for your full text search ta­bles. These are not the de­fault. I’m con­stantly sur­prised by the de­fault choices of SQLite.

Rails lessons:

The de­fault PRAGMAs in Rails seem to be work­ing for lob­sters.

You typ­i­cally don’t think about it but data­base mi­gra­tions are data­base spe­cific. I had to move the old mi­gra­tions out to an old mi­gra­tions di­rec­tory so that db:mi­grate would con­tinue to work.

Lobsters code­base lessons:

There is a search parser in the lob­sters code­base.

I learned about heinous_in­line_­par­tials which is a hack to speed up ren­der­ing.

The lob­sters test­suite was es­sen­tial in mak­ing sure I could mi­grate to SQLite with­out a ton of man­ual test­ing.

Overall lessons:

I think a key in­gre­di­ent in mak­ing this work was good com­mu­ni­ca­tion from every­one that par­tic­i­pated. I don’t think this would have been pos­si­ble oth­er­wise.

Migrating the un­der­ly­ing data­base with­out hav­ing ac­cess to the pro­duc­tion data­base is re­ally hard to get right. This was my first un­der­ly­ing data­base mi­gra­tion with­out hav­ing ac­cess to pro­duc­tion. One les­son I’ll take away from this is that I’ll make sure to have re­al­is­tic dataset sizes be­fore do­ing an­other un­der­ly­ing data­base mi­gra­tion in the fu­ture.

Wishes:

I wish we could say in a test, “Fail if you en­counter any full table scans”. Which would have caught the perf is­sues we ex­pe­ri­enced dur­ing the first de­ploy.

I wish cre­at­ing a pro­duc­tion like dataset would be much eas­ier than hav­ing to man­u­ally write some­thing and wait­ing a week.

Try it

Paste a Sanskrit verse in any Indian script — the me­ter is de­tected au­to­mat­i­cally.

First chant takes ~10 – 60s while the model warms up. If the demo does­n’t load, use the backup demo ↗.

Listen — sam­ple chants

Six vṛttas ren­dered by this sys­tem — in­clud­ing verses from the shipped de­ploy­ments.

vas­an­tati­lakā Mahābhārata Tātparya Nirṇaya · 1.1

नारायणाय परिपूर्णगुणार्णवाय विश्वोदयस्थितिलयोन्नियतिप्रदाय ।ज्ञानप्रदाय विबुधासुरसौख्यदुःखसत्कारणाय वितताय नमोनमस्ते

śārdūlavikrīḍita Śrīmad Bhāgavatam · 1.1.2

जन्माद्यस्य यतोऽन्वयादितरतश्‍चार्थेष्वभिज्ञः स्वराट् तेने ब्रह्म हृदा य आदिकवये मुह्यन्ति यं सूरयः। तेजोवारिमृदां यथा विनिमयो यत्र त्रिसर्गो मृषा धाम्ना स्वेन सदा निरस्तकुहकं सत्यं परं धीमहि

anuṣṭubh Śrīmad Bhāgavatam · 1.1.5

नैमिषेऽनिमिषक्षेत्रे ऋषयः शौनकादयः। सत्रं स्वर्गाय लोकाय सहस्रसममासत

vaṃśastha Śrīmad Bhāgavatam · 1.3.5

पश्यन्त्यदो रूपमदभ्रचक्षुषः सहस्रपादोरुभुजाननाद्भुतम्। सहस्रमूर्धश्रवणाक्षिनासिकं सहस्रमौल्यम्बरकुण्डलोल्‍लसत्

dru­tavil­am­bita Śrīmad Bhāgavatam · 1.1.4

निगमकल्पतरोर्गलितं फलं शुकमुखादमृतद्रवसंयुतम्। पिबत भागवतं रसमालयं मुहुरहो रसिका भुवि भावुकाः

mālinī Narasiṃha stuti · retroflex tongue-twister

हठलुठ दल घिष्टोत्कण्ठदष्टोष्ठ विद्युत् सटशठ कठिनोरः पीठभित्सुष्ठुनिष्ठाम् ।पठतिनुतव कण्ठाधिष्ठ घोरान्त्रमाला दह दह नरसिंहासह्यवीर्याहितं मे ॥

Get the app

These recita­tions power a free app for the com­plete Śrīmad Bhāgavatam.

॥ भागवतवाणी ॥

Bhāgavata-VāNi

The com­plete Śrīmad Bhāgavatam — all 12 skand­has — with synced au­dio recita­tion and line-by-line karaoke high­light­ing, in 10 Indian scripts. Search any verse in any script, tra­di­tional in­dices, works fully of­fline. Free, no ads.

Practise with it

Vāgdhenu’s chants power a com­pan­ion tool that lis­tens to you.

॥ वाग्बोधिनी ॥

Vāgbodhinī

A Sanskrit chant tu­tor. Paste any śloka (or prose) in any script, hear its me­tre-aware ref­er­ence chant ren­dered by Vāgdhenu, then chant along — a Sanskrit speech model scores every syl­la­ble and shows you what to fix. For clas­si­cal (laukika) Sanskrit.

About

Vāgdhenu maps a met­ri­cal verse to its chanted pārāyaṇa recita­tion. Its voice is a flow-match­ing TTS back­bone re­trained on a pur­pose-recorded, care­fully de­signed sin­gle-speaker Sanskrit chant cor­pus (~5 hours), with a fur­ther voice-steer­ing re­train; the neural vocoder is like­wise fine-tuned for the chant reg­is­ter. Around the trained model sits the ma­chin­ery a faith­ful Sanskrit chant pipeline needs: a script-aware fron­tend that routes Sanskrit through Kannada or­thog­ra­phy (avoiding the Hindi schwa-dele­tion that Devanagari trig­gers); vis­arga sandhi with the ji­hvāmūlīya and upadhmānīya al­lo­phones; the as­pi­ra­tion con­trast; the three sibi­lants and the full retroflex se­ries kept dis­tinct; ho­mor­ganic anusvāra and vo­calic ṝ; and a vṛtta-aware mech­a­nism that de­tects the me­ter and se­lects a matched ref­er­ence un­der the half-ref­er­ence rule. The re­trained model reaches an ex­pert MOS of about 4.6, and dense con­juncts — in­clud­ing retroflex as­pi­rates — ren­der cor­rectly, the class ear­lier ar­chi­tec­tures could not crack.

Deployments

This sys­tem pro­duced two cor­pora at scale.

● Mahābhārata Tātparya Nirṇaya — 32 chap­ters, 5,183 verses (~17.5h) · video se­ries ↗

● Śrīmad Bhāgavatam — ~18,000 verses across 12 books · karaoke-video se­ries ↗

whoa there, pard­ner!

Your re­quest has been blocked due to a net­work pol­icy.

Try log­ging in or cre­at­ing an ac­count here to get back to brows­ing.

If you’re run­ning a script or ap­pli­ca­tion, please reg­is­ter or sign in with your de­vel­oper cre­den­tials here. Additionally make sure your User-Agent is not empty and is some­thing unique and de­scrip­tive and try again. if you’re sup­ply­ing an al­ter­nate User-Agent string, try chang­ing back to de­fault as that can some­times re­sult in a block.

You can read Reddit’s Terms of Service here.

If you think that we’ve in­cor­rectly blocked you or you would like to dis­cuss eas­ier ways to get the data you want, please file a ticket here.

When con­tact­ing us, please in­clude your Reddit ac­count along with the fol­low­ing code:

019f75ec-7a40 – 7958-b297-a2c4ba233131