This is the twenty-sec­ond post in the Fastmail Advent 2024 se­ries. The pre­vi­ous post was Dec 21: Fastmail In A Box. The next post is Dec 23: Ten years of JMAP.

There has re­cently been talk of cloud repa­tri­a­tion where com­pa­nies are mov­ing from the cloud to on premises, with some par­tic­u­larly noisy ex­am­ples.

Fastmail has a long his­tory of us­ing our own hard­ware. We have over two decades of ex­pe­ri­ence run­ning and op­ti­mis­ing our sys­tems to use our own bare metal servers ef­fi­ciently.

We get way bet­ter cost op­ti­mi­sa­tion com­pared to mov­ing every­thing to the cloud be­cause:

We un­der­stand our short, medium and long term us­age pat­terns, re­quire­ments and growth very well. This means we can plan our hard­ware pur­chases ahead of time and don’t need the fast dy­namic scal­ing that cloud pro­vides. We have in house op­er­a­tions ex­pe­ri­ence in­stalling, con­fig­ur­ing and run­ning our own hard­ware and net­work­ing. These are skills we’ve had to main­tain and grow in house since we’ve been do­ing this for 25 years. We are able to use our hard­ware for long pe­ri­ods. We find our hard­ware can pro­vide use­ful life for any­where from 5-10 years de­pend­ing on what it is and when in the global tech­nol­ogy cy­cle it was bought, mean­ing we can amor­tise and de­pre­ci­ate the cost of any hard­ware over many years.

Yes, that means we have to do more our­selves, in­clud­ing plan­ning, choos­ing, buy­ing, in­stalling, etc, but the trade­off for us has and we be­lieve con­tin­ues to be sig­nif­i­cantly worth it.

Of course over the 25 years we’ve been run­ning Fastmail we’ve been through a num­ber of hard­ware changes. For many years, our IMAP server stor­age plat­form was a com­bi­na­tion of spin­ning rust dri­ves and ARECA RAID con­trollers. We tended to use faster 15k RPM SAS dri­ves in RAID1 for our hot meta data, and 7.2k RPM SATA dri­ves in RAID6 for our main email blob data.

In fact it was slightly more com­plex than this. Email blobs were writ­ten to the fast RAID1 SAS vol­umes on de­liv­ery, but then a sep­a­rate archiv­ing process would move them to the SATA vol­umes at low server ac­tiv­ity times. Support for all of this had been added into cyrus and our tool­ing over the years in the form of sep­a­rate “meta”, “data” and “archive” par­ti­tions.

A few years ago how­ever we made our biggest hard­ware up­grade ever. We moved all our email servers to a new 2U AMD plat­form with pure NVMe SSDs. The den­sity in­crease (24 x 2.5″ NVMe dri­ves vs 12 x 3.5″ SATA dri­ves per 2U) and per­for­mance in­crease was enor­mous. We found that these new servers per­formed even bet­ter than our ini­tial ex­pec­ta­tions.

At the time we up­graded how­ever NVMe RAID con­trollers weren’t widely avail­able. So we had to de­cide on how to han­dle re­dun­dancy. We con­sid­ered a RAID-less setup us­ing raw SSDs dri­ves on each ma­chine with syn­chro­nous ap­pli­ca­tion level repli­ca­tion to other ma­chines, but the soft­ware changes re­quired were go­ing to be more com­plex than ex­pected.

We were look­ing at us­ing clas­sic Linux mdadm RAID, but the write hole was a con­cern and the write cache did­n’t seem well tested at the time.

We de­cided to have a look at ZFS and at least test it out.

Despite some of the cyrus on disk data­base struc­tures be­ing fairly hos­tile to ZFS Copy-on-write se­man­tics, they were still in­cred­i­bly fast at all the IO we threw at them. And there were some other wins as well.

When we rolled out ZFS for our email servers we also en­abled trans­par­ent Zstandard com­pres­sion. This has worked very well for us, sav­ing about 40% space on all our email data.

We’ve also re­cently done some ad­di­tional cal­cu­la­tions to see if we could tune some of the pa­ra­me­ters bet­ter. We sam­pled 1 mil­lion emails at ran­dom and cal­cu­lated how many blocks would be re­quired to store those emails un­com­pressed, and then with ZFS record sizes of 32k, 128k or 512k and zstd-3 or zstd-9 com­pres­sion op­tions. Although ZFS RAIDz2 seems con­cep­tu­ally sim­i­lar to clas­sic RAID6, the way it ac­tu­ally stores blocks of data is quite dif­fer­ent and so you have to take into ac­count vol­block­size, how files are split into log­i­cal record­size blocks, and num­ber of dri­ves when do­ing cal­cu­la­tions.

Emails: 1,026,000

Raw blocks: 34,140,142

32k & zstd-3, blocks: 23,004,447 = 32.6% sav­ing

32k & zstd-9, blocks: 22,721,178 = 33.4% sav­ing

128k & zstd-3, blocks: 20,512,759 = 39.9% sav­ing

128k & zstd-9, blocks: 20,261,445 = 40.7% sav­ing

512k & zstd-3, blocks: 19,917,418 = 41.7% sav­ing

512k & zstd-9, blocks: 19,666,970 = 42.4% sav­ing

This showed that the de­faults of 128k record size and zstd-3 were al­ready pretty good. Moving to a record size of 512k im­proved com­pres­sion over 128k by a bit over 4%. Given all meta data is cached sep­a­rately, this seems a worth­while im­prove­ment with no sig­nif­i­cant down­side. Moving to zstd-9 im­proved com­pres­sion over zstd-3 by about 2%. Given the CPU cost of com­pres­sion at zstd-9 is about 4x zstd-3, even though emails are im­mutable and tend to be kept for a long time, we’ve de­cided not to im­ple­ment this change.

We al­ways en­able en­cryp­tion at rest on all of our dri­ves. This was usu­ally done with LUKS. But with ZFS this was built in. Again, this re­duces over­all sys­tem com­plex­ity.

So af­ter the suc­cess of our ini­tial test­ing, we de­cided to go all in on ZFS for all our large data stor­age needs. We’ve now been us­ing ZFS for all our email servers for over 3 years and have been very happy with it. We’ve also moved over all our data­base, log and backup servers to us­ing ZFS on NVMe SSDs as well with equally good re­sults.

The flash mem­ory in SSDs has a fi­nite life and fi­nite num­ber of times it can be writ­ten to. SSDs em­ploy in­creas­ingly com­plex wear lev­el­ling al­go­rithms to spread out writes and in­crease drive life­time. You’ll of­ten see the quoted en­durance of an en­ter­prise SSD as ei­ther an ab­solute fig­ure of “Lifetime Writes”/“Total bytes writ­ten” like 65 PBW (petabytes writ­ten) or a rel­a­tive per-day fig­ure of “Drive writes per day” like 0.3, which you can con­vert to life­time fig­ure by mul­ti­ply­ing by the drive size and the drive ex­pected life­time which is of­ten as­sumed to be 5 years.

Although we could cal­cu­late IO rates for ex­ist­ing HDD sys­tems, we were mak­ing a sig­nif­i­cant num­ber of changes mov­ing to the new sys­tems. Switching to a COW filesys­tem like ZFS, re­mov­ing the spe­cial cas­ing meta/​data/​archive par­ti­tions, and the mas­sive la­tency re­duc­tion and per­for­mance im­prove­ments mean that things that might have taken ex­tra time pre­vi­ously and ended up batch­ing IO to­gether, are now so fast it ac­tu­ally causes ad­di­tional sep­a­rated IO ac­tions.

So one big un­known ques­tion we had was how fast would the SSDs wear in our ac­tual pro­duc­tion en­vi­ron­ment? After sev­eral years, we now have some clear data. From one server at ran­dom but this is fairly con­sis­tent across the fleet of our old­est servers:

# smartctl -a /dev/nvme14

Percentage Used: 4%

At this rate, we’ll re­place these dri­ves due to in­creased drive sizes, or en­tirely new phys­i­cal drive for­mats (such E3. S which ap­pears to fi­nally be gain­ing trac­tion) long be­fore they get close to their rated write ca­pac­ity.

We’ve also anec­do­tally found SSDs just to be much more re­li­able com­pared to HDDs for us. Although we’ve only ever used dat­a­cen­ter class SSDs and HDDs fail­ures and re­place­ments every few weeks were a reg­u­lar oc­cur­rence on the old fleet of servers. Over the last 3+ years, we’ve only seen a cou­ple of SSD fail­ures in to­tal across the en­tire up­graded fleet of servers. This is eas­ily less than one tenth the fail­ure rate we used to have with HDDs.

After con­vert­ing all our email stor­age to NVMe SSDs, we were re­cently look­ing at our data backup so­lu­tion. At the time it con­sisted of a num­ber of older 2U servers with 12 x 3.5″ SATA drive bays and we de­cided to do some cost cal­cu­la­tions on:

Looking at var­i­ous providers, the per TB per month price, and then a yearly price for 1000Tb/1Pb (prices as at Dec 2024)

Some of these (e.g. Amazon) have po­ten­tially sig­nif­i­cant band­width fees as well.

It’s in­ter­est­ing see­ing the spread of prices here. Some also have a bunch of weird edge cases as well. e.g. “The S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive stor­age classes re­quire an ad­di­tional 32 KB of data per ob­ject”. Given the large re­trieval time and ex­tra over­head per-ob­ject, you’d prob­a­bly want to store small in­cre­men­tal back­ups in reg­u­lar S3, then when you’ve gath­ered enough, build a big­gish ob­ject to push down to Glacier. This adds im­ple­men­ta­tion com­plex­ity.

* Pros: No limit to amount we store. Assuming we use S3 com­pat­i­ble API, can choose be­tween mul­ti­ple providers.

* Cons: Implementation cost of con­vert­ing ex­ist­ing backup sys­tem that as­sumes lo­cal POSIX files to S3 style ob­ject API is un­cer­tain and pos­si­bly sig­nif­i­cant. Lowest cost op­tions re­quire ex­tra care­ful con­sid­er­a­tion around im­ple­men­ta­tion de­tails and spe­cial lim­i­ta­tions. Ongoing monthly cost that will only in­crease as amount of data we store in­creases. Uncertain if prices will go down or not, or even go up. Possible sig­nif­i­cant band­width costs de­pend­ing on provider.

Seagate Exos 24 HDs are 3.5″ 24T HDDs. This would al­low us to triple the stor­age on ex­ist­ing servers. Each HDD is about $500, so up­grad­ing one 2U ma­chine would be about $6,000 and have stor­age of 220T or so.

* Pros: Reuses ex­ist­ing hard­ware we al­ready have. Upgrades can be done a ma­chine at a time. Fairly low price

* Cons: Will ex­ist­ing units han­dle 24T dri­ves? What’s the re­build time on drive fail­ure look like? It’s al­most a day for 8T dri­ves al­ready, so pos­si­bly nearly a week for a failed 24T drive? Is there enough IO per­for­mance to han­dle daily back­ups at ca­pac­ity?

As we know, SSDs are denser (2.5″ -> 24 per 2U vs 3.5″ -> 12 per 2U), more re­li­able, and now higher ca­pac­ity - up to 61T per 2.5″ drive. A sin­gle 2U server with 24 x 61T dri­ves with 2 x 12 RAIDz2 = 1220T. Each drive is about $7k right now, prices fluc­tu­ate. So all up 24 x $7k = $168k + ~$20k server =~ $190k for > 1000T stor­age one-time cost.

* Pros: Much higher se­quen­tial and ran­dom IO than HDDs will ever have. Price < 1 year of stan­dard S3 stor­age. Internal to our WAN, no band­width costs and very low la­tency. No new de­vel­op­ment re­quired, ex­ist­ing backup sys­tem will just work. Consolidate on sin­gle 2U plat­form for all stor­age (cyrus, db, back­ups) and SSD for all stor­age. Significant space and power sav­ings over ex­ist­ing HDD based servers

* Cons: Greater up front cost. Still need to pre­dict and buy more servers as back­ups grow.

One thing you don’t see in this cal­cu­la­tion is dat­a­cen­ter space, power, cool­ing, etc. The rea­son is that com­pared to the amor­tised yearly cost of a stor­age server like this, these are ac­tu­ally rea­son­ably min­i­mal these days, on the or­der of $3000/2U/year. Calculating per­son time is harder. We have a lot of home built au­toma­tion sys­tems that mean in­stalling and run­ning one more server has min­i­mal mar­ginal cost.

We ended up go­ing with the the new 2U servers op­tion:

* The 2U AMD NVMe plat­form with ZFS is a plat­form we have ex­pe­ri­ence with al­ready

* SSDs are much more re­li­able and much higher IO com­pared to HDDs

* No un­cer­tainty around su­per large HDDs, RAID con­trollers, re­build times, shuf­fling data around, etc.

* No new de­vel­op­ment re­quired, can use ex­ist­ing backup sys­tem and code

So far this has worked out very well. The ma­chines have bonded 25Gbps net­works and when fill­ing them from scratch we were able to sat­u­rate the net­work links stream­ing around 5Gbytes/second of data from our IMAP servers, com­press­ing and writ­ing it all down to a RAIDz2 zstd-3 com­pressed ZFS dataset.

Running your own hard­ware might not be for every­one and has dis­tinct trade­offs. But when you have the ex­pe­ri­ence and the knowl­edge of how you ex­pect to scale, the cost im­prove­ments can be sig­nif­i­cant.

I’m one of the founders at Corelatus. Our niche is tele­com hard­ware for the E1/T1 and SDH/SONET in­ter­faces. I work with layer 2 sig­nalling, i.e SS7 MTP-2, ATM and frame re­lay, and the Erlang con­trol sys­tem for our em­bed­ded hard­ware.

You can reach me at matthias@core­la­tus.com

Decoding the tele­phony sig­nals in Pink Floyd’s ‘The Wall’

How far can I place a PMP from the tap-in point? (update)

Finding the bad link in an SDH net­work us­ing BIP coun­ters

What does an im­prop­erly placed mon­i­tor point do to a sig­nal?

I have 47 E1s? How much hard­ware do I need to mon­i­tor them?

Decoding sig­nalling on the Abis in­ter­face with Wireshark

How does TCP be­have on an in­ter­rupted net­work?

Perl ex­am­ple code for GTH: SS7 ISUP de­cod­ing and play­back/​record

I like puz­zles. Recently, some­one asked me to iden­tify the tele­phone net­work sig­nalling in The Wall, a 1982 film fea­tur­ing Pink Floyd. The sig­nalling is au­di­ble when the main char­ac­ter, Pink, calls London from a pay­phone in Los Angeles, in this

scene (Youtube).

Here’s a five sec­ond au­dio clip from when Pink calls:

The clip starts with some speech over­lap­ping a dial-tone which in turn over­laps some rapid tone com­bi­na­tions, a ring tone and some pops, clicks and mu­sic. It ends with an an­swer tone.

The most char­ac­ter­is­tic part is the tele­phone num­ber en­coded in the rapid tone com­bi­na­tions. Around 1980, when the film was made, dif­fer­ent parts of the world used sim­i­lar, but in­com­pat­i­ble, tone-based sig­nalling schemes. They were all based on the same idea: there are six or eight pos­si­ble tones, and each digit is rep­re­sented by a com­bi­na­tion of two tones.

SoX, an au­dio edit­ing tool for PCs, can make charts that show the spec­tral com­po­nents of the au­dio over time. The hor­i­zon­tal axis rep­re­sents time, the ver­ti­cal axis fre­quency, and darker sec­tions show more au­dio power, and lighter sec­tions less.

Signalling tones ap­pear as dark hor­i­zon­tal lines in the spec­tro­gram, with the digit sig­nalling vis­i­ble from 0.7 to 1.8 sec­onds. That part of the sig­nalling has tones at roughly 700, 900, 1100, 1300, 1500

and 1700 Hz.

Everyone’s heard DTMF (Dual Tone Multi Frequency). It’s the sound your phone makes when you in­ter­act with one of those “Press 1 if you are a new cus­tomer. Press 2 if you have a billing en­quiry. Press 3…” sys­tems. DTMF is still used by many fixed-line tele­phones to set up a call.

In DTMF, each digit is en­coded by play­ing a “high” tone and a “low” tone. The low ones can be 697, 770, 852 or 941 Hz. The high ones 1209, 1336, 1477 and 1633 Hz.

None of the pairs in the au­dio match this, so it’s not

DTMF. Here’s an au­dio clip of what it would sound like if we used DTMF sig­nalling for the same num­ber, with about the same speed of tones:

CAS R2 uses a two-out-of-six tone scheme with the fre­quen­cies 1380, 1500, 1620, 1740, 1860 and 1980 Hz for one call di­rec­tion and 1140, 1020, 900, 780, 660 and 540 Hz for the other. None of these are a good match for the tones we heard. Besides, Pink is in the USA, and the USA did not use CAS R2, so it’s not CAS.

This is what the digit sig­nalling would have sounded like if CAS were used:

SS5 also uses a two-out-of-six scheme with the fre­quen­cies 700, 900, 1100, 1300, 1500 and 1700 Hz. This matches most of what we can hear, and SS5 is the sig­nalling sys­tem most likely used for a call from the USA to the UK in the early 1980s.

This is what the digit sig­nalling sounds like in SS5, when re-gen­er­ated to get rid of all the other sounds:

It can’t be SS7. Signalling sys­tem No. 7 (SS7) does­n’t use tones at all; it’s all dig­i­tal. SS7 is car­ried sep­a­rately from the au­dio chan­nel, so it can’t be heard by callers. SS7 was­n’t in com­mon use un­til later in the 1980s.

I made a spec­tro­gram which com­bines all three sig­nalling types on the same chart. The dif­fer­ence be­tween DTMF and SS5 is sub­tle, but recog­nis­able. CAS is ob­vi­ously dif­fer­ent.

I in­jected the au­dio file into a times­lot of an E1 line, con­nected it to Corelatus’ hard­ware and started an ss5_reg­is­ter­sig_­mon­i­tor.

The in­put au­dio has a lot of noise in ad­di­tion to the sig­nalling, but these pro­to­cols are ro­bust enough for the dig­i­tal fil­ters in the hard­ware to be able to de­code and time­stamp the di­alled dig­its any­way. Now, we know that the num­ber sig­nalling we hear was 044

1831. The next step is to analyse the fre­quen­cies pre­sent at the start time for each tone. I re-analysed the au­dio file with SoX, which did an FFT on snip­pets of the au­dio to find the ac­tual tone fre­quen­cies at the times there were tones, like this:

sox in­put.wav -n trim 0.700 0.060 stat -freq

At this point, I’m cer­tain the sig­nalling is SS5. It uses the cor­rect fre­quen­cies to trans­mit dig­its. It uses the cor­rect digit tim­ing. It obeys the SS5 rules for hav­ing KP1 be­fore the dig­its and ST af­ter the dig­its. It uses a tone close to 2400 Hz to in­di­cate that the call was an­swered.

I’ve also listed the dial tone at the be­gin­ning, and the 2400 Hz seiz­ing tone at the end. SS5 also uses a 2600 Hz tone, which is in­fa­mous for its use in blue box phreak­ing (telephone fraud) in the 1980s.

My best guess is that, at the time the film was made, callers could hear the in­ter-ex­change sig­nalling dur­ing op­er­a­tor-as­sisted calls in the US. That would have al­lowed the sound en­gi­neer to record a real tele­phone in the US and ac­cu­rately cap­ture the feel­ing of a long-dis­tance call. The num­ber it­self was prob­a­bly made-up: it’s too short and the area code does­n’t seem valid.

The au­dio was then cut and mixed to make the dial tone over­lap the sig­nalling. It sounds bet­ter that way and fits the scene’s tim­ing.

It turns out that an ex­tended ver­sion of the same phone call ap­pears near the end of ‘Young Lust’, a track on the al­bum ‘The Wall’. Other en­gi­neers with ac­tual ex­pe­ri­ence of 1970s tele­phone net­works have also analysed the sig­nalling

in an in­ter­est­ing ar­ti­cle with a host of de­tails and back­ground I did­n’t know about, in­clud­ing the likely names of the peo­ple in the call.

It’s nice to know that I got the digit de­cod­ing right, we both con­cluded it was 044 1831. One sur­prise is that the num­ber called is prob­a­bly a short­ened real num­ber in London, rather than a com­pletely fab­ri­cated one as I sus­pected ear­lier. Most likely, sev­eral dig­its be­tween the ‘1’ and the ‘8’ are cut out. Keith Monahan’s analy­sis noted a very ugly splice point there, whereas I only briefly won­dered why the digit start times are fairly reg­u­lar for all dig­its ex­cept that the ‘8’ starts early and the fi­nal ‘1’ starts late.

This is the fas­ci­nat­ing story of how re­searchers used Bloom fil­ters clev­erly to make SQLite 10x faster for an­a­lyt­i­cal queries. These are my five-minute notes on the pa­per SQLite: Past, Present, and Future (2022). I’ll also ex­plain some data­base in­ter­nals and how data­bases im­ple­ment joins.

SQLite is a B-tree on disk, row-based stor­age. It in­ter­nally uses a VM called VDBE to ex­e­cute queries. It is cross-plat­form, sin­gle-threaded, and runs al­most every­where.

SQLite is a gen­eral-pur­pose data­base, but it ex­cels at OLTP work­loads. However, re­searchers at Buffalo University in 2015 found that most queries are sim­ple key-value lookups and com­pli­cated OLAP queries. So, re­searchers at the University of Wisconsin-Madison set out to make it faster for an­a­lyt­i­cal queries since the trend is chang­ing. To set a base­line, they used DuckDB. They used the in­dus­try stan­dard OLAP bench­mark - Star Schema Benchmark (SSB).

SSB con­tains a bunch of an­a­lyt­i­cal queries called Star Queries, where you have one large fact table and mul­ti­ple smaller di­men­sion ta­bles. e.g., fact table = cus­tomer or­ders, di­men­sion ta­bles = cus­tomer, mer­chant info, de­liv­ery part­ners, etc.

Here is a sam­ple query with the query plan. Like a typ­i­cal an­a­lyt­i­cal query, this one con­tains joins be­tween four ta­bles.

As ex­pected, they found DuckDB to be 30-50x faster than SQLite. Note that they ran DuckDB in sin­gle-threaded mode.

Let’s fig­ure out why SQLite was slow; then, we can de­ter­mine how to make it faster. SQLite has a com­pile time op­tion, VDBE_PROFILE, which mea­sures the num­ber of CPU cy­cles each in­struc­tion takes in VDBE. When they reran the bench­marks, they found two op­codes:

What do these op­codes do?

* SeekRowID - given a rowId, probe the row in B-tree

Since the an­a­lyt­i­cal queries mainly con­tain join op­er­a­tions, let’s un­der­stand how they are im­ple­mented.

Following are some of the ways data­bases im­ple­ment joins:

SQLite does Nested Loop join, which is the sim­plest of all three. Here is an an­i­ma­tion (source) show­ing how Nested Loop join works:

Assume you have two ta­bles: or­ders and cus­tomers. The code here de­picts join­ing both ta­bles us­ing the or­der_id col­umn. The outer loop it­er­ates over every item in the cus­tomers table, and for each item, it it­er­ates over every item in the or­ders table.

For every id it matches, it ap­pends to the re­sult list. The loop op­er­a­tions are equiv­a­lent to prob­ing in B-tree, which is very ex­pen­sive.

Your goal is to re­duce the B-tree probes. Now, stop for a minute and con­sider why this is not good. Can you come up with some ideas where you can make this bet­ter?

Next, the or­der of ta­bles in the join op­er­a­tion mat­ters. Here is a sim­ple il­lus­tra­tion:

Consider there are three ta­bles: Orders, Customers, Date. Our query matches 20 items in Customers and 10 items in Date. We probe the B-tree when­ever a row matches.

Just by flip­ping the or­der, it re­duced to 1M op­er­a­tions! But it is in­cred­i­bly dif­fi­cult to come up with an op­ti­mized query plan. It is an NP-Hard prob­lem.

How do we op­ti­mize the join? The other two join al­go­rithms are bet­ter than Nested Loop join. However, au­thors ar­gue that Hash Join takes sig­nif­i­cant mem­ory, and SQLite mostly runs in mem­ory-con­strained en­vi­ron­ments. Second, adding one more join al­go­rithm would com­pli­cate the query plan­ner.

Here is one way: be­fore you run both loops, you first build the cus­tomer data cache. Then, in the in­ner loop, first, you check this cache. Only if there is a match do you it­er­ate over the loop.

That’s what the re­searchers did! They used a Bloom fil­ter, which is very space ef­fi­cient and fits in a CPU cache line. It was also easy to im­ple­ment.

They added two op­codes: Filter and FilterAdd. At the start of the join op­er­a­tion, we go over all the rows of di­men­sion ta­bles and set the bits in the Bloom fil­ter which match the query pred­i­cate. The op­code is FilterAdd.

During the join op­er­a­tion, we first check if the row ex­ists in the Bloom fil­ter at each stage. If it does, then we do the B-tree probe. This is the Filter op­code.

This is the op­ti­mized query plan.

This is the CPU cy­cle analy­sis post-op­ti­miza­tion. You can see that the large blue bars are al­most gone!

The re­sults of this re­search have been ap­plied to SQLite al­ready and were re­leased in v3.38.0.

tl;dr: Bloom fil­ters were great be­cause: min­i­mal mem­ory over­head, goes well with SQLite’s sim­ple im­ple­men­ta­tion, and worked within ex­ist­ing query en­gine.

1. I pre­sented this pa­per at a lo­cal Papers We Love group. This blog post is a sum­mary of the talk

2. Yes, there is a typo in the ‘Matters Order’ slide

Someone pur­port­ing to be United Healthcare is fil­ing DMCA re­quests to scrub the in­ter­net of artists’ de­pic­tions of the sur­veil­lance video of Luigi smil­ing, par­ody mer­chan­dise of “Deny, Defend, Depose,” and other mer­chan­dise show­ing the al­leged shooter.

If it is re­ally the health in­surer fil­ing these no­tices, it wildly over­steps any le­gal rights the com­pany might have, but if there’s any com­pany will­ing to pre­emp­tively breach the law and dare the other side to spend their lim­ited re­sources try­ing to pro­tect their rights it would be a health in­surer.

404 Media re­ports that a num­ber of DMCA take­down re­quests from “United Healthcare” have hit artists de­pict­ing the news­wor­thy event. The pub­li­ca­tion reached out to the com­pany to con­firm that it’s be­hind the fil­ings but did­n’t hear back.

For those keep­ing score at home, the DMCA pro­vides that a must in­form an in­ter­net en­tity that their in­tel­lec­tual prop­erty is be­ing abused and the ser­vice provider then has an op­por­tu­nity to ex­pe­di­tiously re­move the ma­te­r­ial and avoid pos­si­ble li­a­bil­ity. That said, the DMCA only pro­vides a shield against le­gal ac­tion and if the copy­right claim is — to use the pre­cise le­gal term — “utter bull­shit,” they don’t have to com­ply. Unfortunately, hosts rou­tinely err on the side of cau­tion and re­move con­tent when there’s zero cog­niz­able claim.

If this is some­one act­ing on be­half of United Healthcare, what an irony to lodge a fal­si­fied claim with­out it get­ting de­nied.

An en­tity claim­ing to be United Healthcare is send­ing bo­gus copy­right claims to in­ter­net plat­forms to get Luigi Mangione fan art taken off the in­ter­net, ac­cord­ing to the print-on-de­mand merch re­tailer TeePublic. An in­de­pen­dent jour­nal­ist was hit with a copy­right take­down de­mand over an im­age of Luigi Mangione and his fam­ily she posted on Bluesky, and other DMCA take­down re­quests posted to an open data­base and viewed by 404 Media show copy­right claims try­ing to get “Deny, Defend, Depose” and Luigi Mangione-related merch taken off the in­ter­net, though it is un­clear who is fil­ing them.

It’s a bizarre and grimly ironic turn if United Healthcare wanted to the words “Deny, Defend, Depose.” The take­down re­quest for that piece ob­jects to the artist styling the “D” with el­e­ments of the United Healthcare logo… which would be the very de­f­i­n­i­tion of pro­tected par­ody. The 404 story has im­ages if you want to see what these all look like.

Beyond ty­ing it­self to the gun­man’s catch­phrase, the idea of UHC try­ing to own any and all fix­a­tions of the al­leged shooter’s like­ness would be a wild leap. An artists’ de­pic­tion of Mangione could be­long to the artist (Mangione might be able to as­sert some rights to his like­ness — a du­bi­ous claim un­der the cir­cum­stances and in light of the First Amendment — but in no case would UHC have such a claim).

“What is the cir­cum­stance un­der which United Healthcare might come to own the copy­right to a wa­ter­color paint­ing of the guy who as­sas­si­nated their CEO?” tech rights ex­pert and sci­ence fic­tion au­thor Cory Doctorow told 404 Media in a phone call. “It’s just like, it’s hard to imag­ine” a lawyer think­ing that, he added, say­ing that it’s an ex­am­ple of “copyfraud.”

It is il­le­gal to file DMCA no­tices if you don’t own the copy­right (or at least have a good faith be­lief that you do). The idea that UHC now owns every de­pic­tion of the guy ac­cused of killing their em­ployee is laugh­ably friv­o­lous and one that its le­gal de­part­ment un­der­stands this and these re­quests are com­ing from a third party troll im­per­son­at­ing the car­rier.

An in­de­pen­dent jour­nal­ist post­ing a photo of Mangione with his fam­ily also re­ceived a DMCA re­quest — from a lawyer claim­ing to rep­re­sent a fam­ily mem­ber hold­ing the copy­right — even though the im­age was “originally posted on the cam­paign web­site of Maryland as­sem­bly­mem­ber Nino Mangione.” That site ap­par­ently deleted the im­age and turned around to threaten any­one us­ing it now which is… not how fair use works. But at least this re­quest can claim they have a “good faith” claim, though the sys­tem prob­a­bly should­n’t re­ward peo­ple for try­ing to retroac­tively claim rights af­ter they try to mem­ory-hole their in­ter­net his­tory.

But the dis­turb­ing thread run­ning through all these re­quests is how easy it’s be­come for copy­right trolls to lever­age the DMCA to in­tim­i­date providers into ac­cept­ing fa­cially in­valid re­quests. The statute has given way to a sort of asym­met­ri­cal war­fare over IP where bad faith ac­tors can pep­per sites with own­er­ship claims and trust that their tar­gets will back down rather than deal with the lit­i­ga­tion risk. As 404’s cov­er­age notes, this does­n’t bode well in a coun­try about to in­au­gu­rate an ad­min­is­tra­tion openly en­cour­ag­ing ret­ri­bu­tion against jour­nal­ists in an ef­fort to si­lence crit­i­cism.

This sure looks like a com­pany (and a lo­cal politi­cian) try­ing to use copy­right law as a “square peg-round hole” so­lu­tion to eras­ing any hu­man­iz­ing or sym­pa­thetic de­pic­tion of Mangione. And un­for­tu­nately, every­one seems to be obey­ing in ad­vance.

Copyright Abuse Is Getting Luigi Mangione Merch Removed From the Internet [404 Media]

Joe Patrice is a se­nior ed­i­tor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, ques­tions, or com­ments. Follow him on Twit­ter or Bluesky if you’re in­ter­ested in law, pol­i­tics, and a healthy dose of col­lege sports news. Joe also serves as a Managing Director at RPN Executive Search.

Whose woods these are I think I know.

His house is in the vil­lage though;

He will not see me stop­ping here

To watch his woods fill up with snow.

My lit­tle horse must think it queer

To stop with­out a farm­house near

Between the woods and frozen lake

The dark­est evening of the year.

He gives his har­ness bells a shake

To ask if there is some mis­take.

The only other sound’s the sweep

Of easy wind and downy flake.

The woods are lovely, dark and deep.

But I have promises to keep,

And miles to go be­fore I sleep,

And miles to go be­fore I sleep.

Welcome to twtx­t’s doc­u­men­ta­tion. This doc­u­men­ta­tion is di­vided into mul­ti­ple parts. We rec­om­mend you to get started with the Installation and then head over to the

Quickstart sec­tion. If you don’t know what this is about read the Introduction first. There is a more de­tailed Usage sec­tion about how to use twtxt via the CLI. The in­ter­nals of twtxt are doc­u­mented in the API chap­ter.

Feel free to con­tribute to this pro­ject. The source code is main­tained on GitHub.

Five years ago I was in Berlin for a few months, and my friend Michael in­vited me to a gath­er­ing of his beer afi­cionado friends at a neigh­bor­hood bar. It seemed like a typ­i­cal bar hang — un­til one of the guys planted a lit­tle flag on the table: an in­signia for their group.

That seemed a bit odd: Why do these pals have a flag? Turns out, this was no ran­dom meetup for a pint. It was a Stammtisch.

As I learned, Stammtisch (“shtom-tish”) means “regulars’ table”. It’s an age-old German tra­di­tion where a group of like-minded peo­ple — tra­di­tion­ally men — meet up at a bar or a restau­rant at a reg­u­lar time to have drinks and chat.

Michael and his Stammtisch com­pan­ions clearly en­joyed each oth­er’s com­pany. And as I learned, it was re­ally more than just shar­ing drinks — these monthly meet­ings deep­ened their friend­ships.

And it had an el­e­gant sim­plic­ity: A stand­ing date on the cal­en­dar to see a group of peo­ple you en­joy, out at a bar or restau­rant so no one needs to tidy their home.

These days, I’ve got a tod­dler at home and so­cial­iz­ing sans kid takes more plan­ning. The easy com­fort of meet­ing friends at a bar some­times feels like an ac­tiv­ity from my care­free youth, and I’m al­ready nos­tal­gic for it.

Back home in D. C., I found my­self think­ing re­cently of Stammtisch and what a nice way it is to see peo­ple you like. I men­tioned this to a German friend, Amrai, as we walked around the zoo with our kids, ad­mir­ing the goats and al­pacas.

Amrai laughed and said, “You know there’s a Stammtisch that meets in D. C.? It’s mostly Germans.” Regretting my scant Deutsch, I de­cided to get my­self in­vited.

So on a re­cent cold, dark December evening, I parked my bike out­side a brew­pub called Right Proper, and stepped in­side. I was met by warm con­vivi­al­ity and con­ver­sa­tion — and many peo­ple who know the Stammtisch tra­di­tion well.

Over the din of chat­ter and pop mu­sic, the gath­er­ing’s or­ga­nizer, Andrea Shalal, de­scribed the clas­sic Stammtisch: “In every German vil­lage there is the cor­ner bar, and in the cor­ner is a table. It’s re­served for the sort of el­ders or other reg­u­lars. And they sit in the cor­ner and they drink their beer and smoke their cig­a­rettes and pon­tif­i­cate on the town and all of its crazi­ness.”

Traditionally, many tav­erns in Germany had a big, round table re­served for reg­u­lars — Stammtisch refers to both the table and those who meet there. There are many tales of for­eign trav­el­ers un­wit­tingly tak­ing a seat at this table and be­ing shooed away by the bar­keep.

This Stammtisch in D. C. was a mod­ern take: There were women, not just men. Instead of that big table, peo­ple milled with their drinks be­side a bar. But every­one seemed to agree that the ba­sic for­mat of Stammtisch is a good one.

“Growing up in the ’80s, ’90s, early 2000s, the idea of Stammtisch in Germany’s youth was sort of re­jected as ‘This is some­thing that our par­ents’ gen­er­a­tion, our grand­par­ents’ gen­er­a­tion, would do,′ ” said Robert Christoffel, 45, who moved to the U. S. more than a decade ago.

But he’s no­ticed that’s start­ing to change.

“A lot of my friends, nowa­days, in Germany — maybe be­cause they have got­ten a lit­tle older as well — they are now meet­ing up and sort of es­tab­lish­ing Stammtisch,” he told me.

In mod­ern life, as fewer peo­ple go to church — which in Germany might be fol­lowed by drink­ing at the tav­ern — these gath­er­ings can take on even more sig­nif­i­cance.

Werner Hein, who’s 80 years old, de­scribed the Stammtisch this way: “That’s a place where you meet in per­son. It’s hard to see a Stammtisch on Zoom.”

Men in par­tic­u­lar can find it hard to be vul­ner­a­ble with one an­other, added Hein. But amid the am­biance and re­peated meet­ings of a Stammtisch, some­thing starts to hap­pen.

“You open up on things, on your very per­sonal things,” Hein said. “Nowadays, I have to al­most say it’s more im­por­tant than ever to have a kind of Stammtisch.”

Does a Stammtisch have to in­volve al­co­hol? I asked.

“If you con­sider beer al­co­hol, yes, I think so,” replied Christoffel, pint glass in hand.

“That’s called a Kaffeekränzchen,” Hein ex­plained. “That’s typ­i­cally more like the women when they meet in the af­ter­noon for a cof­fee and cake. Also an­other very tra­di­tional German thing.”

To me, that sounds equally good as a stein of beer — maybe bet­ter.

Turns out I’m ac­tu­ally fa­mil­iar with Kaffeekränzchen, though I’d never heard the term. For the last cou­ple years, my friend Jordan has or­ga­nized a reg­u­lar gath­er­ing she dubbed Coffee Club. A group of us meet at a neigh­bor­hood café at the same time each week and hang out for an hour over caf­feine and baked goods.

At Coffee Club we talk about our jobs, our week­ends — and our lives. On one oc­ca­sion, sev­eral mem­bers brought dresses for me to bor­row for a wed­ding and an im­promptu cat­walk tran­spired.

I can’t make it every week, but that’s OK. The group is large enough that there’s no pres­sure.

Amid the stress of work and the full-court press of par­ent­hood, this reg­u­lar hang with the ladies has been a re­li­able source of good vibes and com­mis­er­a­tion. It’s some­thing to look for­ward to, not an oblig­a­tion — just like the Stammtisch.

I asked Werner Hein if he had any ad­vice for some­one who might want to start a Stammtisch.

It takes a bit of com­mit­ment, he said — “A com­mit­ment to have fun.”

And in re­turn? A warm cir­cle of con­nec­tion, no mat­ter how cold it is out­side.

Aging brings two op­pos­ing trends in can­cer risk: first, the risk climbs in our 60s and 70s, as decades of ge­netic mu­ta­tions build up in our bod­ies. But then, past the age of around 80, the risk drops again — and a new study may ex­plain a key rea­son why.

The in­ter­na­tional team of sci­en­tists be­hind the study an­a­lyzed lung can­cer in mice, track­ing the be­hav­ior of alve­o­lar type 2 (AT2) stem cells. These cells are cru­cial for lung re­gen­er­a­tion, and are also where many lung can­cers get started.

What emerged was higher lev­els of a pro­tein called NUPR1 in the older mice. This caused cells to act as if they were de­fi­cient in iron, which in turn lim­ited their re­gen­er­a­tion rates — putting re­stric­tions on both healthy growth and can­cer­ous tu­mors.

“The ag­ing cells ac­tu­ally have more iron, but for rea­sons we don’t yet fully un­der­stand, they func­tion like they don’t have enough,” says can­cer bi­ol­o­gist Xueqian Zhuang, from the Memorial Sloan Kettering Cancer Center (MSK) in New York.

“Aging cells lose their ca­pac­ity for re­newal and there­fore for the run­away growth that hap­pens in can­cer.”

The same processes were found to be hap­pen­ing in hu­man cells too: more NUPR1 leads to a drop in the amount of iron avail­able to cells. When NUPR1 was ar­ti­fi­cially low­ered or iron was ar­ti­fi­cially in­creased, cell growth ca­pa­bil­i­ties were boosted again.

That po­ten­tially gives re­searchers a way of ex­plor­ing treat­ments that tar­get iron me­tab­o­lism — es­pe­cially in older peo­ple. It could per­haps re­store lung ca­pac­ity in those ex­pe­ri­enc­ing long-term ef­fects from COVID-19, for ex­am­ple.

These find­ings also have im­pli­ca­tions for can­cer treat­ments based on a type of cell death called fer­rop­to­sis, which is trig­gered by iron. This cell death is less com­mon in older cells, the re­searchers found, be­cause of their func­tional iron de­fi­ciency.

This per­haps also makes them more re­sis­tant to can­cer treat­ments based on fer­rop­to­sis that are in de­vel­op­ment– so the ear­lier a fer­rop­to­sis treat­ment can be tried, the bet­ter it’s likely to work.

“What our data sug­gests in terms of can­cer pre­ven­tion is that the events that oc­cur when we’re young are prob­a­bly much more dan­ger­ous than the events that oc­cur later,” says can­cer bi­ol­o­gist Tuomas Tammela, from MSK.

“So, pre­vent­ing young peo­ple from smok­ing, or from tan­ning, or from other ob­vi­ous car­cino­genic ex­po­sures are prob­a­bly even more im­por­tant than we thought.”

There’s lots more to ex­plore here about the ef­fects of NUPR1 and how it re­lates to stem cell func­tion — both healthy re­gen­er­a­tion and can­cer­ous growth — but these are im­por­tant find­ings for fight­ing can­cer at any stage of life.

As al­ways with can­cer treat­ments, mul­ti­ple fac­tors need to be taken into ac­count: the type and stage of can­cer, other med­ical con­di­tions that might be in­volved, and (as this new study shows) the age of the in­di­vid­ual. The more per­son­al­ized we can make these treat­ments, the more ef­fec­tive they can be.

“There’s still a lot that’s un­known about how ag­ing ac­tu­ally changes the bi­ol­ogy of can­cer,” says Zhuang.

The re­search has been pub­lished in Nature.

In the 1990s, Apple strug­gled to bring the orig­i­nal Mac OS—originally writ­ten in 1984 for the re­source-con­strained Macintosh 128K ma­chine—up to mod­ern op­er­at­ing sys­tem stan­dards. The story of how OS X came to be is thrilling in its own right, but suf­fice it to say that Apple ended up buy­ing Steve Jobs’ sec­ond com­puter com­pany, NeXT, and us­ing its NeXTSTEP op­er­at­ing sys­tem as the ba­sis of a new gen­er­a­tion of Macs.

Apple an­nounced the ac­qui­si­tion of NeXT on December 20, 1996, not­ing it wanted NeXT’s ob­ject-ori­ented soft­ware de­vel­op­ment tech­nol­ogy and its op­er­at­ing sys­tem know-how. As part of the deal, Jobs came back to Apple, even­tu­ally tak­ing over as CEO and mak­ing the com­pany into the con­sumer elec­tron­ics gi­ant it is to­day. Six­teen years later, sev­eral tech­nolo­gies de­vel­oped or cham­pi­oned by NeXT still sur­vive in OS X and in its mo­bile cousin, iOS. This week, we re­mem­ber some of those tech­nolo­gies which con­tinue to power key fea­tures of Apple’s de­vices.

We con­sulted sev­eral de­vel­op­ers with knowl­edge of both NeXTSTEP and OS X; they uni­ver­sally agreed that one cru­cial fea­ture of NeXTSTEP that made OS X and iOS what they are to­day is its un­der­ly­ing UNIX roots. Beneath all the slickly styled win­dows, icons, scrolling lists, and but­tons sits a rock-solid, cer­ti­fied UNIX op­er­at­ing sys­tem.

UNIX was orig­i­nally con­ceived in 1969 by Bell Labs’ Ken Thompson for the PDP-7 mini­com­puter. Its de­vel­op­ment through­out the early ’70s led to the de­vel­op­ment of the C pro­gram­ming lan­guage by Thompson’s col­league Dennis Ritchie. UNIX at its core was meant to be a pow­er­ful op­er­at­ing sys­tem for the mas­sive com­put­ers of the day, but it could also be eas­ily pro­grammed and con­trolled by de­vel­op­ers us­ing a sim­ple video ter­mi­nal.

That same ba­sic sys­tem pow­ers both the Mac sit­ting on your desk and the iPhone rest­ing in your pocket. By bas­ing OS X—and later, iOS—on UNIX, Macs were fi­nally able to en­joy im­por­tant op­er­at­ing sys­tem fea­tures like pro­tected mem­ory, pre-emp­tive mul­ti­task­ing, and dae­mon-based ser­vices.