A real-world pro­duc­tion mi­gra­tion from DigitalOcean to Hetzner ded­i­cated, han­dling 248 GB of MySQL data across 30 data­bases, 34 Nginx sites, GitLab EE, Neo4j, and live mo­bile app traf­fic — with zero down­time.

Running a soft­ware com­pany in Turkey has be­come in­creas­ingly ex­pen­sive over the last few years. Skyrocketing in­fla­tion and a dra­mat­i­cally weak­en­ing Turkish Lira against the US dol­lar have turned dol­lar-de­nom­i­nated in­fra­struc­ture costs into a se­ri­ous bur­den. A bill that felt man­age­able two years ago now hits very dif­fer­ently when the ex­change rate has mul­ti­plied sev­eral times over.

Every month, we were pay­ing $1,432 to DigitalOcean for a droplet with 192GB RAM, 32 vC­PUs, 600GB SSD, two block vol­umes (1TB each), and back­ups en­abled. The server was fine — but the price-to-per­for­mance ra­tio had stopped mak­ing sense.

Then we dis­cov­ered the Hetzner AX162-R.

That’s $14,388 saved per year — for a server that’s ob­jec­tively more pow­er­ful in every di­men­sion. The de­ci­sion was easy.

I’ve been a DigitalOcean cus­tomer for nearly 8 years. They have a great prod­uct and I have no com­plaints about re­li­a­bil­ity or de­vel­oper ex­pe­ri­ence. But look­ing at those num­bers now, I can­not help feel­ing a bit sad about all the ex­tra money I left on the table over the years. If you are run­ning steady-state work­loads and not ac­tively us­ing DO’s ecosys­tem fea­tures, do your­self a fa­vor and check ded­i­cated server pric­ing be­fore your next re­newal.

* Several live mo­bile apps serv­ing hun­dreds of thou­sands of users

Old server: CentOS 7 — long past its end-of-life, but still run­ning in pro­duc­tion. New server: AlmaLinux 9.7 — a RHEL 9 com­pat­i­ble dis­tri­b­u­tion and the nat­ural suc­ces­sor to CentOS. This mi­gra­tion was also an op­por­tu­nity to fi­nally es­cape an OS that had­n’t re­ceived se­cu­rity up­dates in years.

The naive ap­proach — change DNS, restart every­thing, hope for the best — was­n’t ac­cept­able. Instead, we de­signed a proper mi­gra­tion path with six phases:

Phase 1 — Full stack in­stal­la­tion on the new server

Nginx (compiled from source with iden­ti­cal flags), PHP (via Remi repo, with the same .ini con­fig files from the old server), MySQL 8.0, Neo4J Graph DB, GitLab EE, Node.js, Supervisor, and Gearman. Every ser­vice had to be con­fig­ured to match the old server’s be­hav­ior be­fore we touched a sin­gle DNS record.

SSL cer­tifi­cates were han­dled by rsync­ing the en­tire /etc/letsencrypt/ di­rec­tory from the old server to the new one. After the mi­gra­tion was com­plete and all traf­fic was flow­ing through the new server, we force-re­newed all cer­tifi­cates in one shot:

Phase 2 — Web files cloned with rsync

The en­tire /var/www/html di­rec­tory (~65 GB, 1.5 mil­lion files) was cloned to the new server us­ing rsync over SSH with the –checksum flag for in­tegrity ver­i­fi­ca­tion. We ran a fi­nal in­cre­men­tal sync right be­fore cu­tover to catch any files changed af­ter the ini­tial clone.

Phase 3 — MySQL mas­ter to slave repli­ca­tion

Rather than tak­ing the data­base of­fline for a dump-and-re­store, we set up live repli­ca­tion. The old server be­came mas­ter, the new server a read-only slave. We used my­dumper for the ini­tial bulk load, then started repli­ca­tion from the ex­act bin­log po­si­tion recorded in the dump meta­data. This kept both data­bases in real-time sync un­til the mo­ment of cu­tover.

Phase 4 — DNS TTL re­duc­tion

We scripted the DigitalOcean DNS API to lower all A and AAAA record TTLs from 3600 to 300 sec­onds — with­out touch­ing MX or TXT records (changing mail record TTLs can cause de­liv­er­abil­ity is­sues). After wait­ing one hour for old TTLs to ex­pire glob­ally, we were ready to cut over in un­der 5 min­utes.

Phase 5 — Old server ng­inx con­verted to re­verse proxy

We wrote a Python script that parsed every server {} block across all 34 Nginx site con­figs, backed up the orig­i­nals, and re­placed them with proxy con­fig­u­ra­tions point­ing to the new server. This meant that dur­ing DNS prop­a­ga­tion, any re­quest still hit­ting the old IP was silently for­warded. No user would see a dis­rup­tion.

Phase 6 — DNS cu­tover and de­com­mis­sion

A sin­gle Python script hit the DigitalOcean API and flipped all A records to the new server IP in sec­onds. The old server re­mained as a cold standby for one week, then was shut down.

The key in­sight: at no point did we have a win­dow where the ser­vice was un­avail­able. Traffic was al­ways be­ing served — ei­ther di­rectly or through the proxy.

This was the most com­plex part of the en­tire op­er­a­tion.

We used my­dumper in­stead of the stan­dard mysql­dump — and it made an enor­mous dif­fer­ence. By lever­ag­ing the new server’s 48 CPU cores for par­al­lel ex­port and im­port, what would have taken days with a tra­di­tional sin­gle-threaded mysql­dump was com­pleted in hours. If you’re mi­grat­ing a large MySQL data­base and you’re not us­ing my­dumper/​my­loader, you’re do­ing it the hard way.

The main dump’s meta­data file recorded the bin­log po­si­tion at the time of the snap­shot:

File: mysql-bin.000004

Position: 21834307

This would be our repli­ca­tion start­ing point.

Once the dump was com­plete, we trans­ferred it to the new server us­ing rsync over SSH. With 248 GB of com­pressed chunks, this was sig­nif­i­cantly faster than any other trans­fer method:

The –compress flag in my­dumper paid off here — com­pressed chunks trans­ferred much faster over the wire.

Being stuck on CentOS 7 meant we were also stuck on MySQL 5.7 — an out­dated ver­sion that had been run­ning in pro­duc­tion for years. Before the mi­gra­tion, we ran mysqlcheck –check-upgrade to ver­ify that our data was com­pat­i­ble with MySQL 8.0. It came back clean, so we in­stalled the lat­est MySQL 8.0 Community on the new server. The per­for­mance im­prove­ment across all our pro­jects was im­me­di­ately no­tice­able — query ex­e­cu­tion times dropped sig­nif­i­cantly thanks to MySQL 8.0’s im­proved op­ti­mizer and InnoDB en­hance­ments.

That said, the ver­sion jump did in­tro­duce one tricky prob­lem.

After im­port, the mysql.user table had the wrong col­umn struc­ture — 45 columns in­stead of the ex­pected 51. This caused mysql.in­fos­chema to be miss­ing, break­ing user au­then­ti­ca­tion.

But this failed the first time with:

ERROR: ‘sys.innodb_buffer_stats_by_schema’ is not VIEW

The sys schema had been im­ported as reg­u­lar ta­bles in­stead of views. Solution:

With both dumps im­ported, we con­fig­ured the new server as a replica of the old one:

Almost im­me­di­ately, repli­ca­tion stopped with er­ror 1062 (Duplicate Key). This hap­pened be­cause our dump was taken in two passes — dur­ing the gap be­tween them, rows were writ­ten to cer­tain ta­bles, and now both the im­ported dump and the bin­log re­play were try­ing to in­sert the same rows.

IDEMPOTENT mode silently skips du­pli­cate key and miss­ing row er­rors. All crit­i­cal data­bases synced with­out a sin­gle er­ror. Within a few min­utes, Seconds_Behind_Master dropped to 0.

Before touch­ing a sin­gle DNS record, we needed to ver­ify that all ser­vices were work­ing cor­rectly on the new server. The trick: we tem­porar­ily edited the /etc/hosts file on our lo­cal ma­chine to point our do­main names to the new server’s IP.

# /etc/hosts (local ma­chine)

NEW_SERVER_IP your­do­main1.com

NEW_SERVER_IP your­do­main2.com

# … and so on for all your do­mains

With this in place, our browsers and Postman would hit the new server while the rest of the world was still go­ing to the old one. We ran through our API end­points, checked ad­min pan­els, and ver­i­fied that every ser­vice was re­spond­ing cor­rectly. Only af­ter this con­fir­ma­tion did we pro­ceed with the cu­tover.

Once mas­ter-slave repli­ca­tion was fully syn­chro­nized, we no­ticed that INSERT state­ments were suc­ceed­ing on the new server when they should­n’t have been — read­_only = 1 was set, but writes were go­ing through.

The rea­son: all PHP ap­pli­ca­tion users had been granted SUPER priv­i­lege. In MySQL, SUPER by­passes read­_only.

We re­voked it from all 24 ap­pli­ca­tion users:

After this, read­_only = 1 cor­rectly blocked all writes from ap­pli­ca­tion users while al­low­ing repli­ca­tion to con­tinue.

All do­mains were man­aged through DigitalOcean DNS (with name­servers pointed from GoDaddy). We scripted the TTL re­duc­tion against the DigitalOcean API, only touch­ing A and AAAA records — not MX or TXT records, since chang­ing mail record TTLs can cause de­liv­er­abil­ity is­sues with Google Workspace.

After wait­ing one hour for old TTLs to ex­pire, we were ready.

Rather than edit­ing 34 con­fig files by hand, we wrote a Python script that parsed every server {} block in every con­fig file, iden­ti­fied the main con­tent blocks, re­placed them with proxy con­figs, and backed up orig­i­nals as .backup files.

The key: prox­y_ss­l_ver­ify off — the new server’s SSL cert is valid for the do­main, not for the IP ad­dress. Disabling ver­i­fi­ca­tion here is fine be­cause we con­trol both ends.

With repli­ca­tion at Seconds_Behind_Master: 0 and the re­verse proxy ready, we ex­e­cuted the cu­tover in or­der:

1. New server: STOP SLAVE;

2. New server: SET GLOBAL read­_only = 0;

3. New server: RESET SLAVE ALL;

4. New server: su­per­vi­sor­ctl start all

5. Old server: ng­inx -t && sys­tem­ctl re­load ng­inx (proxy goes live)

6. Old server: su­per­vi­sor­ctl stop all

7. Mac: python3 do_­cu­tover.py (DNS: all A records to new server IP)

8. Wait: ~5 min­utes for prop­a­ga­tion

9. Old server: com­ment out all crontab en­tries

The DNS cu­tover script hit the DigitalOcean API and changed every A record to the new server IP — in about 10 sec­onds.

After mi­gra­tion, we dis­cov­ered many GitLab pro­ject web­hooks were still point­ing to the old server IP. We wrote a script to scan all pro­jects via the GitLab API and up­date them in bulk.

We went from $1,432/month down to $233/month — sav­ing $14,388 per year. And we ended up with a more pow­er­ful ma­chine:

The en­tire mi­gra­tion took roughly 24 hours. No users were af­fected.

MySQL repli­ca­tion is your best friend for zero-down­time mi­gra­tions. Set it up early, let it catch up, then cut over with con­fi­dence.

Check your MySQL user priv­i­leges be­fore mi­gra­tion. SUPER priv­i­lege by­passes read­_only — if your app users have it, your slave en­vi­ron­ment is­n’t ac­tu­ally read-only.

Script every­thing. DNS up­dates, ng­inx con­fig rewrites, web­hook up­dates — do­ing these by hand across 34+ sites would have taken hours and in­tro­duced er­rors.

my­dumper + my­loader dra­mat­i­cally out­per­forms mysql­dump for large datasets. Parallel dump/​re­store with 32 threads cut what would have been days of work down to hours.

Cloud providers are ex­pen­sive for steady-state work­loads. If you’re not us­ing au­toscal­ing or ephemeral in­fra­struc­ture, a ded­i­cated server of­ten de­liv­ers bet­ter per­for­mance at a frac­tion of the cost.

All Python scripts used in this mi­gra­tion are open-sourced and avail­able on GitHub:

* do_list_­do­main­s_ttl.py — List all DigitalOcean do­mains with their A records, IPs, and TTLs

* do_­to_het­zn­er_bulk_dns_record­s_im­port.py — Migrate all DNS zones from DigitalOcean to Hetzner DNS

* do_­cu­tover_­to_new_ip.py — Flip all A records from old server IP to new server IP

* mysql_­com­pare.py — Compare row counts across all ta­bles on two MySQL servers

* fi­nal_git­lab_web­hook_up­date.py — Update all GitLab pro­ject web­hooks to the new server IP

All scripts sup­port a DRY_RUN = True mode so you can safely pre­view changes be­fore ap­ply­ing them.

Anonymous re­quest-to­ken com­par­isons from the com­mu­nity, show­ing how Opus 4.6 and Opus 4.7 dif­fer on real in­puts

In 2025, the Kdenlive team con­tin­ued grind­ing to push the pro­ject for­ward through steady de­vel­op­ment, col­lab­o­ra­tion, and com­mu­nity sup­port. Over the past year we’ve found a nice bal­ance be­tween adding new fea­tures, bug fix­ing, pol­ish­ing the user in­ter­face, and im­prov­ing per­for­mance and work­flow, with sta­bil­ity tak­ing pri­or­ity over fea­ture creep. We re­launched the web­site with a new con­tent man­age­ment sys­tem, re­freshed some con­tent and the de­sign, and re­stored his­toric con­tent dat­ing back to 2002. We also strength­ened up­stream col­lab­o­ra­tion with the MLT de­vel­op­ers and con­tributed sev­eral im­prove­ments to OpenTimelineIO.Here’s a look at what we’ve been up to and what is ahead.As part of KDE Apps, we fol­low the KDE Gear re­lease cy­cle, with three ma­jor re­leases each year—in April, August, and December—each fol­lowed by three point main­te­nance re­leases.This re­lease added a pow­er­ful au­to­matic mask­ing tool and brought the last batch of fea­tures from our last fundraiser.The new Object Segmentation plu­gin based on the [SAM2][4] model al­lows to re­move any se­lected ob­ject from the back­ground.We rewrote our OpenTimelineIO im­port and ex­port func­tion us­ing the C++ li­brary. Now you can ex­change pro­jects with other edit­ing ap­pli­ca­tions that sup­port this open source file for­mat.Au­dio wave­form gen­er­a­tion got a 300% per­for­mance boost, along with a refac­tored sam­pling method that ac­cu­rately ren­ders the au­dio sig­nal and higher-res­o­lu­tion wave­forms for greater pre­ci­sion.This re­lease fo­cused heav­ily on sta­bi­liza­tion, bring­ing over 300 com­mits and fix­ing more than 15 crashes. Instead of ma­jor new fea­tures, the ef­fort went into pol­ish­ing and bug fix­ing.We re­designed the au­dio mixer bring­ing lev­els with clearer vi­su­als and thresh­olds. We also did some code refac­tor­ing and cleanup. This change fixes is­sues with HiDPI dis­plays with frac­tional scal­ing.Guides and Markers got a ma­jor over­haul this re­lease to im­prove the pro­ject or­ga­ni­za­tion.This re­lease the ti­tler re­ceived some much needed love like im­proved SVG and im­age sup­port with abil­ity to move and re­size items, added cen­ter re­size with Shift + Drag, and re­named the Pattern tab to Templates and moved the tem­plates drop­down to it­The fo­cus of this re­lease cy­cle was on im­prov­ing the user ex­pe­ri­ence and pol­ish­ing the user in­ter­face.We added a new first-run launch screen for first time users as well as added a Welcome Screen al­low­ing to eas­ily launch re­cent pro­jects.We added a new, more flex­i­ble dock­ing sys­tem that lets you group wid­gets, show or hide them on de­mand, and save lay­outs as sep­a­rate files that can be shared or stored within pro­jects.The au­dio wave­form in the Project Monitor got a re­vamped in­ter­face with an added min­imap.This next re­lease is just around the cor­ner and brings a nice batch of nifty new fea­tures like mon­i­tor mir­ror­ing and an­i­mated tran­si­tion pre­views, mak­ing it much eas­ier to vi­su­al­ize how they will look be­fore ap­ply­ing them. Additionally, drop­ping a tran­si­tion onto the time­line can now au­to­mat­i­cally ad­just its du­ra­tion to match the clips above and be­low, sav­ing time and re­duc­ing man­ual tweak­ing.This fea­ture al­lows you to mir­ror any mon­i­tor while work­ing in fullscreen mode. It’s es­pe­cially use­ful when work­ing with mul­ti­ple dis­plays or col­lab­o­rat­ing with oth­ers in the edit­ing room.Change the play­back speed of mul­ti­ple clips at on­ceIm­port a clip di­rectly from the time­line con­text menu and in­sert it at the click po­si­tionOp­tion to al­ways zoom to­ward the mouse po­si­tion in­stead of the time­line play­head­Our roadmap is con­stantly be­ing re­viewed and up­dated, and some of the up­com­ing high­lights in­clude im­ple­ment­ing the new fea­tures in MLT, the mul­ti­me­dia frame­work which pow­ers Kdenlive. Some ex­cit­ing up­com­ing fea­tures in­clude 10/12 bit color sup­port, play­back op­ti­miza­tions (decoding), and OpenFX sup­port. (Shoutout to a Kdenlive com­mu­nity mem­ber for lead­ing this ef­fort). Also ex­pected is a refac­tor­ing of the sub­ti­tle sys­tem as well as con­tin­u­ing to de­velop the Advanced Trimming Tools.We are cur­rently work­ing on refac­tor­ing the keyfram­ing sys­tem and im­ple­ment­ing a Dopesheet, ba­si­cally it is a ded­i­cated time­line for man­ag­ing and view­ing keyframes from mul­ti­ple ef­fects si­mul­ta­ne­ously. This work will also in­tro­duce per-pa­ra­me­ter keyfram­ing (currently, once you add a keyframe to an ef­fect, it is ap­plied to all pa­ra­me­ters by de­fault). More info can be found in the last sta­tus re­port. This work is made pos­si­ble through an NGI Zero Commons grant via NLnet.We have been work­ing on en­abling and fix­ing mul­ti­ple mod­ules in MLT to com­pile with MSVC al­low­ing us to ship Kdenlive in the Microsoft Store soon. Another ad­van­tage is that it will al­low to run unit tests on our CI for Windows.Currently, the Kdenlive core team is made up of 8 ac­tive mem­bers, in­clud­ing 2 de­vel­op­ers.In 2025, 38 peo­ple con­tributed code to Kdenlive (including the core dev team and other KDE devs), a truly im­pres­sive num­ber! Even more ex­cit­ing, about half of them were first-time con­trib­u­tors, which is al­ways great. We hope to see many of them con­tinue con­tribut­ing in the fu­ture. On be­half of the Kdenlive team, we salute you all!Note that these num­bers re­fer specif­i­cally to con­tri­bu­tions to the Kdenlive ap­pli­ca­tion. Other pro­jects such as the test suite and web­site are hosted in sep­a­rate repos­i­to­ries and are not in­cluded in these fig­ures.In February, part of the Kdenlive core team met in Amsterdam for a short sprint, high­lighted by a visit to the Blender Foundation, where we met with Francesco Siddi and he shared valu­able in­sights into Blender’s his­tory and of­fered ad­vice on prod­uct man­age­ment for Kdenlive. We also at­tended their weekly open ses­sion, where artists and de­vel­op­ers pre­sent progress on on­go­ing pro­jects. During the sprint, we dis­cussed and ad­vanced sev­eral tech­ni­cal top­ics, some high­lights in­clude:Fin­ish­ing an MLT Framework patch to en­able ren­der­ing with­out a dis­play server (needed for Flatpak test­ing)The Berlin sprint was one of our most pro­duc­tive gath­er­ings to date. Most of the team was there in per­son, and we also con­nected on­line with those who could­n’t make it. We dis­cussed just about every as­pect of the pro­ject, from roadmap plan­ning to up­com­ing fea­tures and work­flow im­prove­ments. Some of the high­lights in­clude:Eval­u­ated the cur­rent state of the Titler and dis­cussed pos­si­ble in­te­gra­tion with GlaxnimateDeveloped a proof of con­cept for us­ing KDDockWidgetsRedesigned and started de­vel­op­ment of the au­dio clip view in the Clip MonitorThanks to the nice folks at c-base who kindly hosted us.Akademy is al­ways a great op­por­tu­nity to ex­change ideas with the broader KDE and Qt com­mu­ni­ties. One of the high­lights was meet­ing the main­tainer of Glaxnimate, where we dis­cussed com­mon goals and ways to col­lab­o­rate. This year, Akademy will be in Graz on the 19-24 of September, and we hope to see you there.We’re very happy to see more YouTube chan­nels talk­ing about Kdenlive. Here are some ex­am­ples of what the com­mu­nity has been cre­at­ing.We’d love to see what you’ve been work­ing on in the past year. Share your videos pro­duc­tions in the com­ments!Help us grow the com­mu­nity by or­ga­niz­ing mee­tups, talks, or work­shops in your lo­cal area. Don’t hes­i­tate to con­tact us if you need guid­ance, ma­te­ri­als, or sup­port to get started.Be­low are pho­tos from a work­shop with in­dige­nous com­mu­ni­ties in Paraguay.Kdenlive was down­loaded 11,500,714 times from our down­load page in 2025. Do note that many ad­di­tional in­stalls hap­pen through Linux dis­tri­b­u­tion pack­age man­agers, the Snap Store, Flathub, and other third-party servers, where sta­tis­tics are not al­ways avail­able or re­li­ably mea­sur­able.The Flatpak pack­age from Flathub gets 41,499 down­loads per month.25.04.2 got the most num­ber of down­loads.Files With Most Code ChangesTo the 5 of you in Antarctica, let us know what you are edit­ing. ;)Ever since our last, and very suc­cess­ful, fundraiser in 2022, we haven’t ac­tively asked for do­na­tions, yet the com­mu­nity has con­tin­ued to sup­port us. We are very grate­ful for that.In 2025, we re­ceived a to­tal of €9,344.80 from do­na­tions (down from €11,526.61 in 2024). Around 30% of the amount was given by donors who kindly set up a re­cur­ring plan. The av­er­age do­na­tion was about €25, with the low­est amount be­ing €10 and the high­est €500.We al­lo­cate 20% of our bud­get to KDE e.V. to sup­port in­fra­struc­ture costs (servers and re­lated ex­penses), as well as ad­min­is­tra­tion, le­gal sup­port, and travel. As in pre­vi­ous years, your con­tri­bu­tions en­able us to con­tinue sup­port­ing Jean-Baptiste (Kdenlive’s main­tainer), al­low­ing him to ded­i­cate sev­eral days each month to Kdenlive in ad­di­tion to his vol­un­teer work.WE NEED YOUR SUPPORTKdenlive needs your sup­port to keep grow­ing and im­prov­ing. If just a quar­ter of the peo­ple who down­loaded Kdenlive in 2025 con­tributed €5, our main­tain­ers would be able to ded­i­cate more time to the pro­ject, and it would even al­low us to hire more de­velpers to speed up de­vel­op­ment and im­prove sta­bil­ity. Small amounts can make a big dif­fer­ence, please con­sider mak­ing a do­na­tion.You may also con­tribute by get­ting in­volved and help­ing in:

Japan is the land of the train. 28 per­cent of pas­sen­ger kilo­me­ters in Japan are trav­elled by rail, more than any­where else in the de­vel­oped world. France achieves 10 per­cent, Germany 6.4 per­cent, and the United States just 0.25 per­cent. Travel in Japan is over a hun­dred times more likely to be by rail than travel in the United States.

Japan’s vast rail­way net­work is di­vided be­tween dozens of com­pa­nies, nearly all of them pri­vate. The largest of these, JR East, car­ries more pas­sen­gers than the en­tire rail­way sys­tem of every coun­try other than China and India. Each year, JR East car­ries four times as many pas­sen­gers as the whole British rail­way sys­tem, even though it has fewer kilo­me­ters of track, serves about ten mil­lion fewer peo­ple, and com­petes with eight other com­pa­nies. Japan’s rail­way sys­tem turns a large op­er­at­ing profit and re­ceives far less pub­lic sub­sidy than European and American rail­ways.

Subscribe for $100 to re­ceive six beau­ti­ful is­sues per year.

In most de­vel­oped coun­tries, the rail­ways have strug­gled since the rise of the au­to­mo­bile in the 1950s. From this point on, North America saw the near-to­tal re­place­ment of pas­sen­ger trains with cars and planes. In Europe, it meant vast gov­ern­ment fi­nan­cial sup­port to keep the lines open.

Japan’s dif­fer­ent tra­jec­tory is of­ten at­trib­uted to cul­ture: the Japanese are con­formists who are con­tent to take pub­lic trans­port, un­like free­dom-lov­ing Americans who pre­fer to drive every­where. Europeans are some­where in be­tween. Culture is also used to ex­plain the in­cred­i­ble punc­tu­al­ity of Japanese rail­ways.

These cul­tural ex­pla­na­tions are wrong. The Japanese love cars, but they take trains be­cause they have the best rail­way sys­tem in the world. And their sys­tem ex­cels be­cause of good pub­lic pol­icy: busi­ness struc­ture, land use rules, dri­ving rules, su­pe­rior mod­els for pri­va­ti­za­tion, and sound reg­u­la­tion have given Japan its out­stand­ing rail­ways.

This is good news for friends of rail. Culture is built over cen­turies, and repli­cat­ing it is hard. But suc­cess­ful pub­lic poli­cies can be em­u­lated by one good gov­ern­ment. Much about Japan’s rail­way sys­tem could be replic­a­ble around the world.

Today, the most strik­ing in­sti­tu­tional fea­ture of Japanese rail is that it is pri­vately owned by a throng of com­pet­ing com­pa­nies.

The rail­way ar­rived in Japan in 1872, dur­ing the Meiji Restoration, which opened the coun­try up to for­eign trade, ideas, and tech­nolo­gies. Like most Western coun­tries, Japan na­tion­al­ized its rail­ways in the early twen­ti­eth cen­tury, cre­at­ing what be­came known as Japanese National Railways (JNR). But it did not na­tion­al­ize all of the lines, fo­cus­ing only on main­line rail­ways of na­tional im­por­tance, and new pri­vate rail­ways were still per­mit­ted.

Between 1907 and World War II, Japan saw a boom in new pri­vate elec­tric rail­ways, co­in­cid­ing with rapid ur­ban­iza­tion. Technologically, most of these pri­vate rail­ways were sim­i­lar to the fa­mous in­terur­bans in the United States: they were ba­si­cally elec­tric trams, but run­ning be­tween cities as well as within them. The American net­work even­tu­ally with­ered, and al­most noth­ing of it sur­vives to­day. In Japan, how­ever, the net­work con­sol­i­dated, and the light tram­lines grad­u­ally evolved into heavy-rail in­ter­city con­nec­tions.

These com­pa­nies are to­day known as ‘legacy pri­vate rail­ways’ on ac­count of their hav­ing been pri­vate since their in­cep­tion. There are eight legacy pri­vate rail­ways in the Tokyo met­ro­pol­i­tan area, five in the Osaka–Kobe–Kyoto mega­lopo­lis, two in Nagoya, and one in the fourth city of Fukuoka. There are also dozens of smaller ones else­where. In the three largest ur­ban ar­eas, these op­er­a­tors ac­count for nearly half of rail­way track and sta­tions, as well as a plu­ral­ity of rid­er­ship. The largest, Kintetsu, not only op­er­ates ur­ban ser­vices, but a whole in­ter­city net­work stretch­ing from Osaka to Nagoya.

These com­pa­nies of­ten com­pete head-to-head. At its most ex­treme, three sep­a­rate com­muter lines com­pete for the traf­fic be­tween Osaka and the port city of Kobe, run­ning in par­al­lel, some­times fewer than 500 meters apart.

Meanwhile, the na­tion­al­ized rail­ways were man­aged by JNR. In the post­war era, JNR was re­spon­si­ble for build­ing the fa­mous Shinkansen sys­tem, as well as run­ning com­muter and long-dis­tance lines through­out Japan. But in 1988, it was largely pri­va­tized, bro­ken into six re­gional mo­nop­o­lies for pas­sen­ger ser­vices to­gether with a sin­gle na­tional freight op­er­a­tor. These are col­lec­tively known as the Japan Railways Group (JR).

This means that Japan has ended up with six rail­way com­pa­nies that trace their de­scent to the na­tion­al­ized rail­ways, the six­teen big legacy com­pa­nies that have al­ways been pri­vate, and a host of mi­nor legacy rail­ways, as well as nu­mer­ous un­der­ground met­ros (some pri­vate, some mu­nic­i­pally owned), mono­rails, and tram sys­tems. This in­sti­tu­tional di­ver­sity is strik­ing enough. But equally strik­ing is the con­sis­tent busi­ness model that has evolved amidst this plu­ral­ism: the rail­way that builds a city.

If I take a train to go for a soli­tary walk in the coun­try­side, the rail­way com­pany can cap­ture some of the value it cre­ates by charg­ing me for the jour­ney, just as other com­pa­nies cap­ture the value of the goods and ser­vices they pro­vide by charg­ing for them. However, if I take a train to visit fam­ily, clients, a the­ater, or a shop, an im­por­tant dif­fer­ence ap­pears. The rail­way can cap­ture the value it cre­ates for me by charg­ing me a fare, but it can­not cap­ture the value it cre­ates for those at my des­ti­na­tion. As trans­port in­fra­struc­ture cre­ates ben­e­fits that pro­duce no rev­enue for providers, free mar­kets rarely build enough of it.

Japan has partly solved this prob­lem by en­abling rail­way com­pa­nies to do a great deal be­side run­ning rail­ways. Take the ex­am­ple of the Tokyu cor­po­ra­tion, one of the legacy pri­vate rail­ways in south­ern Tokyo. You can not only travel on its trains, but also ride a Tokyu bus, live in a Tokyu-built house, work in a Tokyu of­fice com­plex, see a doc­tor in a Tokyu hos­pi­tal, buy gro­ceries in a Tokyu su­per­mar­ket, spend an af­ter­noon at a Tokyu mu­seum-the­ater-cin­ema com­plex, take your chil­dren to their amuse­ment park, and even die in a Tokyu re­tire­ment home. The pos­i­tive spillover ef­fects of the rail­way on these things are cap­tured by Tokyu be­cause it owns them. The pres­i­dent of Tokyu has said:

I think that though we are a rail­way com­pany, we con­sider our­selves a city-shap­ing com­pany. In Europe for in­stance, rail­way com­pa­nies sim­ply con­nect cities through their ter­mi­nals. That is a pretty nor­mal way of op­er­at­ing in this in­dus­try, whereas what we do is com­pletely dif­fer­ent: we cre­ate cities and then, as a util­ity fa­cil­ity, we add the sta­tions and the rail­ways to con­nect them one with an­other.

This model was pi­o­neered in the 1950s by what be­came Hankyu Railways. Hankyu’s net­work con­nects cen­tral Osaka to its north­ern sub­urbs, as well as Kyoto and Kobe. Its in­no­v­a­tive founder Kobayashi Ichizo first built sub­ur­ban hous­ing, then a de­part­ment store at the ter­mi­nal sta­tion; he then cre­ated a hot spring re­sort, a zoo, and his own dis­tinc­tive brand of all-women mu­si­cal the­ater, the Takarazuka Revue. He also be­gan to run bus ser­vices to and from his sta­tions. Other com­pa­nies em­u­lated Hankyu’s ex­am­ple: Tokyo Disneyland is a col­lab­o­ra­tion be­tween Disney and the Keisei Railway, while Hanshin in Osaka owns the Hanshin Tigers base­ball team.

Core rail op­er­a­tions are prof­itable for every Japanese pri­vate rail­way com­pany, but they usu­ally only ac­count for a plu­ral­ity or a small ma­jor­ity of rev­enue. The rest is con­tributed by their port­fo­lio of side busi­nesses. There is a nat­ural fi­nan­cial syn­ergy be­tween the re­li­able but un­re­mark­able cash flow of train fares and the prof­itable but riskier real es­tate and com­mer­cial side of the busi­ness. Railway com­pa­nies’ side busi­nesses also at­tract peo­ple to live and work on their rail cor­ri­dor, re­in­forc­ing the cus­tomer base for the rail­way ser­vices them­selves.

This vir­tu­ous cir­cle is en­abled by tran­sit-ori­ented de­vel­op­ment. Japan’s lib­eral land use reg­u­la­tion makes it straight­for­ward to build new neigh­bor­hoods next to rail­way lines, giv­ing com­muters easy ac­cess to city cen­ters. It also en­ables the den­si­fi­ca­tion of these cen­ters, which means that com­muters have more places they want to go.

Railways cost a lot to build, but once they are built, they can move enor­mous num­bers of peo­ple, far more than a road of sim­i­lar size. This means that they work best in cities with a high den­sity of peo­ple, jobs, and other ac­tiv­i­ties. In 2019, New York City was the only American city where rail had a higher modal share than cars, in part be­cause Manhattan has 2.5 mil­lion jobs, two mil­lion res­i­dents, and 50 mil­lion tourist vis­its crammed into 59 square kilo­me­ters.

This does not mean that rail-ori­ented cities must be struc­tured like Chinese cities: is­lands of high-rise apart­ments con­nected by met­ros and sep­a­rated by mo­tor­ways. Japanese cities have the low­est res­i­den­tial den­sity in Asia, and a plu­ral­ity of the Japanese live in houses, usu­ally de­tached ones. The ur­ban area of Tokyo, the dens­est Japanese city, has a weighted pop­u­la­tion den­sity less than that of many European cities, in­clud­ing Paris, Madrid, or Athens. Japanese cities have vast low-rise, pre­dom­i­nantly res­i­den­tial sub­urbs, built at den­si­ties that might be higher than what is typ­i­cal in the United States, but that would be quite nor­mal in Northern Europe.

What makes Japan’s cities par­tic­u­larly suited to rail is thus not their res­i­den­tial dis­tricts, but their huge and hy­per­dense cen­ters. These re­ally are spe­cial: the cores of Tokyo or Osaka are un­like any­thing that ex­ists in Europe or North America. Many of their fea­tures are fa­mous world­wide: the ver­ti­cal street za­kkyo build­ings, un­der­ground streets, shop­ping streets un­der rail tracks, cov­ered ar­cades, el­e­vated sta­tion squares, and ver­ti­cal cities. Getting mil­lions of com­muters and shop­pers into these down­towns is where rail ex­cels be­cause its ex­treme spa­tial ef­fi­ciency means that in­fra­struc­ture with a rel­a­tively mod­est foot­print can trans­port vast num­bers of peo­ple into a small area.

None of this emerged from a co­her­ent mas­ter­plan of tran­sit-ori­ented de­vel­op­ment like Copenhagen’s Finger Plan or Curitiba’s Trinary System. Postwar Japanese opin­ion was com­mit­ted to de­cen­tral­iza­tion both to rural pe­riph­eries and to the sub­urbs through green­belts, mo­tor­ways, and new towns.

Instead, this va­ri­ety and adapt­abil­ity around rail­ways is pos­si­ble be­cause of the way Japanese ur­ban plan­ning works. Since 1919, Japan has had a stan­dard­ized na­tional zon­ing sys­tem, but it is much more lib­eral than de­vel­op­ment con­trol sys­tems in Western coun­tries. The Japanese au­thor­i­ties did not in­tend or even de­sire dense ur­ban cen­ters, but they did not pre­vent them, rather like nine­teenth-cen­tury gov­ern­ments in the West.

This lib­eral zon­ing sys­tem is re­in­forced by pri­vate ac­cess to city plan­ning pow­ers. Thirty per­cent of Japan’s ur­ban land has been sub­ject to land read­just­ment, where agree­ment among two thirds of res­i­dents and landown­ers in an area is enough to al­low its re­plan­ning, in­clud­ing com­pul­so­rily tak­ing and de­mol­ish­ing land for ameni­ties and in­fra­struc­ture. Initially land read­just­ment was used only to as­sem­ble rural land for ur­ban­iza­tion, but over time it was in­creas­ingly used to re­de­velop al­ready ur­ban­ized ar­eas, and new vari­ants were cre­ated to build the sky­scrap­ers that sur­round the ma­jor sta­tions of cen­tral Tokyo.

The his­tory of the pri­vate rail­way com­pa­nies could be writ­ten as a story of land read­just­ment pro­jects: the ini­tial build­ing of the lines in the in­ter­war years pro­ceeded through one land read­just­ment pro­ject af­ter an­other. Postwar im­prove­ments such as dou­ble-track­ing, plat­form length­en­ing, and con­stant re­de­vel­op­ment of sta­tions and their im­me­di­ate thresh­olds were only pos­si­ble be­cause the rail­ways could se­cure land tak­ings co­op­er­a­tively with lo­cal busi­nesses and landown­ers.

Perhaps the great­est ex­am­ple of this phe­nom­e­non in­volved Tokyu. In 1953 the com­pany de­cided to build the Den’en Toshi Line, or Garden City Line, to serve a rural area south­west of Tokyo. This would be en­abled by a se­ries of land read­just­ment pro­jects col­lec­tively among the largest in Japanese his­tory.

Over 30 years, 3,100 hectares were cov­ered, of which only 36 per­cent was de­voted to res­i­den­tial and com­mer­cial de­vel­op­ment, with 20 per­cent for for­est and parks, 17 per­cent for roads, and much of the rest for wa­ter­courses. The pop­u­la­tion of the land read­just­ment zone would rise from 42,000 in 1954 to over 500,000 in 2003.

By con­nect­ing the af­flu­ent south­west­ern sub­urbs to Tokyu’s main real es­tate hub next to Shibuya sta­tion, now the sec­ond busiest in the world, the Den’en Toshi Line al­lowed Tokyu to be­come the largest pri­vate rail­way by rev­enue and rid­er­ship. The Japanese gov­ern­ment and aca­d­e­mics gen­er­ally con­sider the Den’en Toshi Line to be the best cor­ri­dor of tran­sit–ori­ented de­vel­op­ment in Japan.

But the rail­way-as-city-builder model is not the only rea­son Japanese rail­ways have been able to thrive. European coun­tries usu­ally pro­hib­ited rail­ways from run­ning real es­tate side busi­nesses, but in the United States and Canada the prac­tice was ex­tremely wide­spread in the nine­teenth and early twen­ti­eth cen­turies, and many fa­mous rail­way sub­urbs were de­vel­oped this way. Despite this, pas­sen­ger rail in these coun­tries col­lapsed in the mid-twen­ti­eth cen­tury. Part of the dif­fer­ence was that Japan did not ex­tend the same im­plicit sub­si­dies to cars as Western gov­ern­ments did.

The land of Toyota, Nissan, and Honda is not an anti-car nir­vana. In fact, Japan has ex­cel­lent mo­tor­ways, and across the coun­try as a whole a small ma­jor­ity of jour­neys are made by car. But Japan is a place where cars and car-ori­ented lifestyles com­pete on a level play­ing field.

Japan is one of the only coun­tries to have pri­va­tized park­ing. In Europe and North America, vast quan­ti­ties of park­ing space is so­cial­ized: mu­nic­i­pal­i­ties own the streets and al­low peo­ple to park on them at low or zero cost. Initially with the in­ten­tion of en­cour­ag­ing the pro­vi­sion of more park­ing spaces, Japan made it il­le­gal to park on pub­lic roads or pave­ments with­out spe­cial per­mis­sion. Before some­one buys a car, they must prove that they have a re­served night-time space on pri­vate land, ei­ther owned or leased.

Since park­ing on pub­lic land is banned, mu­nic­i­pal­i­ties are not wor­ried about over­spill park­ing from de­vel­op­ments with in­ad­e­quate pri­vate park­ing. They there­fore have no rea­son to im­pose park­ing min­i­mums on de­vel­op­ments: the mar­ket is left to de­cide whether park­ing is the most valu­able use of pri­vate land. Where land is abun­dant, as in rural ar­eas, sub­urbs, or small towns, pri­vate park­ing is plen­ti­ful. But in city cen­ters, it is out­com­peted by other land uses. According to the late Donald Shoup, cen­tral Tokyo has 23 park­ing spaces per hectare and 0.04 park­ing spaces per job, com­pared with 263 and 0.52 for Los Angeles. Even Manhattan, the dens­est ur­ban area in North America with the low­est lev­els of car own­er­ship, has about 60 park­ing spaces per hectare.

Japanese roads are ex­pected to be self-fi­nanc­ing. Motorways are run by self-con­tained pub­lic co­op­er­a­tives, very sim­i­lar to the statu­tory au­thor­i­ties that ran English roads and canals be­tween 1660 and the late 1800s, and funded by tolls on their users. Vehicle reg­is­tra­tion taxes, which are al­lo­cated to lo­cal­i­ties for road con­struc­tion and main­te­nance, are worth three per­cent of the Japanese gov­ern­ment bud­get.

These mea­sures, adopted in the 1950s, were not in­tended to sup­press car use — the point was to fund a mas­sive road ex­pan­sion — but they have forced pri­vate ve­hi­cles to in­ter­nal­ize many of their hid­den costs. In the Tokyo ur­ban area, the av­er­age house­hold spends 71,000 yen ($450) each year on pub­lic trans­port fares and 210,000 yen ($1,350) on car pur­chase and main­te­nance costs.

But the pri­vate car was not the only com­peti­tor faced by the pri­vate rail­ways. For eight decades in the twen­ti­eth cen­tury, they also had to face the jug­ger­naut of Japanese National Railways. Its pri­va­ti­za­tion in 1988 re­moved the fi­nal ob­sta­cle to cre­at­ing the world’s best rail­way sys­tem.

Railway pri­va­ti­za­tion in Britain, New Zealand, Argentina, and Sweden has had a mixed re­cep­tion, and all of those coun­tries, apart from Sweden, have taken steps to re­verse it. In Japan, it has been so suc­cess­ful that the gov­ern­ment sub­se­quently pri­va­tized the metro sys­tems in Tokyo and Osaka.

In the post­war pe­riod, JNR en­joyed real suc­cesses. It built the rev­o­lu­tion­ary Shinkansen, the first high-speed rail­way in the world. It also ag­gres­sively elec­tri­fied and dou­ble-tracked ma­jor trunk lines, quadru­ple-tracked lines into and out of ma­jor cities, and added city-cen­ter loops and freight by­passes. But these achieve­ments were over­shad­owed by two prob­lems.

The first was pol­i­tics. Many coun­tries adapted to the rise of the car by clos­ing the least prof­itable parts of their pas­sen­ger rail net­work, like the con­sol­i­da­tion of American freight rail into the Class I op­er­a­tors or the Beeching Axe in Britain. In Japan, how­ever, the rul­ing Liberal Democratic Party drew its sup­port from rural con­stituen­cies, whose sup­port it re­tained with pork–bar­rel pol­i­tics. Its ‘rail tribe’ group, led by rural MPs, pre­vented JNR from adapt­ing it­self to mass mo­tor­iza­tion.

JNR there­fore did not am­pu­tate gan­grenous rural and freight ser­vices that im­posed heavy costs with few ben­e­fits. Worse, it con­tin­ued to build new loss-mak­ing rural rail­way lines, known in Japanese as Gaden-intetsu, or rail­ways pulled into the rice field.

The sec­ond prob­lem was or­ga­nized la­bor. In gen­eral, Japanese trade unions are known for their mod­er­a­tion and re­spon­si­bil­ity, a gen­er­al­i­sa­tion that also held true for the unions at the legacy pri­vate rail­ways. The JNR unions, how­ever, be­came highly mil­i­tant, se­cure in the knowl­edge that their na­tion­al­ized em­ploy­ers could never go bank­rupt. Their largest se­ries of strikes in 1973 pro­voked ri­ots from com­muters.

The rail­way unions im­posed over­staffing on rev­enue-gen­er­at­ing ur­ban ser­vices, at a time when both in­ter­na­tional and pri­vate do­mes­tic op­er­a­tors were re­duc­ing staffing re­quire­ments against a back­drop of higher wages and the grow­ing au­toma­tion of sig­nal­ing and tick­et­ing. As a re­sult, 78 per­cent of JNR’s costs were re­lated to la­bor, com­pared to 40 per­cent for other Japanese rail­ways. The av­er­age worker at a pri­vate rail­way was 121 per­cent more pro­duc­tive than their JNR coun­ter­part.

By the early 1980s, only seven out of 200 JNR lines made a profit. Successive gov­ern­ments de­ferred se­ri­ous re­form, run­ning up debt, cut­ting down in­vest­ments in new ur­ban lines, rais­ing ticket prices to twice those of com­pa­ra­ble pri­vate rail­ways, and in­creas­ing sub­si­dies — which rose un­til an­nual sub­si­dies equaled the to­tal cost of the Shinkansen.

In 1982, Prime Minister Yasuhiro Nakasone started to pri­va­tize the rail­ways. Unlike other coun­tries, Japan sim­ply re­turned to the tra­di­tional pri­vate rail­way model of the nine­teenth and early twen­ti­eth cen­turies: tracks, trains, sta­tions, and yards were owned by ver­ti­cally in­te­grated re­gional con­glom­er­ates.

There are sub­stan­tial ad­van­tages to ver­ti­cal in­te­gra­tion. Railways are a closed sys­tem that has to be planned as a sin­gle unit. Changing the timetable at sta­tion A can af­fect the timetable at sta­tion Z; buy­ing new trains that can travel faster might re­quire changes to the in­fra­struc­ture so they can reach their top speed, which in turn re­quires rewrit­ing the timeta­bles. This be­comes es­pe­cially com­pli­cated if dif­fer­ent ser­vices share tracks. To pre­vent de­lays from prop­a­gat­ing from one ser­vice to an­other, the timetable needs to be care­fully de­signed to make best use of the avail­able in­fra­struc­ture.

The stark­est ef­fect of pri­va­ti­za­tion was a mas­sive and im­me­di­ate in­crease in la­bor pro­duc­tiv­ity and prof­itabil­ity rel­a­tive to the legacy pri­vate rail­ways. In fact, this be­gan be­fore pri­va­ti­za­tion: its mere threat strength­ened the gov­ern­men­t’s hand when bar­gain­ing with the unions and forced JNR to be­gin clos­ing rural lines.

Privatization saw a gen­eral trend of pro­duc­tiv­ity im­prove­ments, fol­low­ing a big one-time im­prove­ment be­tween 1982 and 1990, when the work­force was cut by more than half, 83 loss-mak­ing lines were re­moved, and JNR’s debts were trans­ferred to a hold­ing com­pany.

The sec­ond great ad­van­tage of pri­va­ti­za­tion was to al­low the JR com­pa­nies to em­u­late the rail­way-as-city-builder model of the legacy pri­vate rail­ways: for in­stance, JR East owns two shop­ping cen­ter brands, a ski re­sort, a cof­fee chain, and even a vend­ing ma­chine drink com­pany. The JR com­pa­nies have not ig­nored their rail busi­ness: they have con­tin­ued to build new high-speed lines and ur­ban tun­nels, up­grade sta­tions, and im­ple­ment a host of other im­prove­ments such as the in­tro­duc­tion in the 1990s of smart cards that al­low pas­sen­gers to pay their fare with a tap.

This does not mean that the Japanese rail­way in­dus­try is a pure crea­ture of free en­ter­prise. No rail­way sys­tem ever has been. The Japanese sys­tem has found an equi­lib­rium that makes rail pol­icy ex­plicit and lim­ited. Leaving aside rail­way safety and busi­ness reg­u­la­tion, there are two main pol­icy levers: fare max­i­mums and cap­i­tal ex­pan­sion sub­si­dies.

Price con­trols are of­ten cited as a clas­sic ex­am­ple of mis­guided gov­ern­ment in­ter­ven­tion, whether through rent con­trols, caps on the price of gaso­line, wage freezes, or min­i­mum agri­cul­tural prices. Tokyo’s in­fa­mously crammed trains are a symp­tom of un­der­priced rush hour traf­fic.

Railways have mar­ket power be­cause the sub­sti­tutes for rail­way trips – coaches, cars and planes – are quite a dif­fer­ent prod­uct. This mo­nop­o­lis­tic po­si­tion has his­tor­i­cally meant trou­ble: mo­nop­oly sys­tems, whether pri­vate or pub­lic, have a ten­dency to abuse their po­si­tion to charge higher prices and run bad ser­vices. For this rea­son, the pri­vate mo­nop­o­lies that were com­mon in the Western world be­fore World War I of­ten had price con­trols im­posed on them. For ex­am­ple, most of the American street­car net­works were op­er­ated as long-term, price-con­trolled fran­chises granted by the city.

Price max­i­mums, if set too low, could have ru­ined Japan’s rail­ways. This is ex­actly what hap­pened to many Western tran­sit ser­vices af­ter the First World War. But the post­war Japanese prac­tice has capped fares gen­er­ously. The sys­tem is ex­plic­itly de­signed to main­tain prof­itabil­ity per rider, which in turn in­cen­tivizes the com­pa­nies to max­i­mize rid­er­ship. That buys po­lit­i­cal le­git­i­macy for the pri­va­tized sys­tem, which is nec­es­sary for the con­tin­ued pro­vi­sion of cap­i­tal ex­pan­sion sub­si­dies. Indeed, dur­ing the long de­fla­tion era be­tween 1992 and 2022, it was com­mon for op­er­a­tors to charge be­low the max­i­mum, and the real value of rail­way fares con­tin­ued to rise. Fare max­i­mums are set on the ba­sis of the av­er­age cost struc­tures of all rail­way op­er­a­tors in a re­gion, so com­pa­nies with be­low-av­er­age costs like Tokyu would of­ten charge be­low the cap to main­tain a com­pet­i­tive edge, pre­vent pub­lic back­lash, and max­i­mize traf­fic to their side-busi­nesses.

Other than the fare max­i­mums, the rail­ways are free to make their own de­ci­sions about timeta­bles, ser­vice pat­terns and day-to-day op­er­a­tions, a highly spe­cial­ized and tech­ni­cal task which re­quires deep ex­per­tise. This con­trasts with the gov­ern­ment med­dling with, say, Amtrak’s routes.

Carefully de­signed pub­lic sub­si­dies also play a use­ful role. Although Japanese rail­ways do not re­ceive sub­si­dies for day-to-day op­er­a­tions, they do re­ceive gov­ern­ment loans and grants for cap­i­tal in­vest­ments. These are typ­i­cally tied to pub­lic pri­or­i­ties, such as dis­abil­ity ac­cess or earth­quake-proof­ing, or to pro­jects that have large spillovers that the rail­way com­pany would be un­able to in­ter­nal­ize, like re­mov­ing level cross­ings, or el­e­vat­ing at-grade rail­ways or trams in or­der to re­duce road con­ges­tion and ac­ci­dent risk. Generally, the lo­cal pre­fec­tural gov­ern­ment will match the con­tri­bu­tion of the na­tional gov­ern­ment. Larger new build pro­jects are sub­ject to lease back or debt-pay­ment con­di­tions that fare rev­enue is ex­pected to pay back.

Railway com­pa­nies in­vested heav­ily in real es­tate busi­nesses, of­ten fund­ing lines through sell­ing land for hous­ing around new sta­tions. Liberal spa­tial pol­icy meant that such de­vel­op­ment hap­pened eas­ily, even as it en­abled dense de­vel­op­ment in ur­ban cores where ra­dial rail lines con­verged. Rail com­pa­nies were gen­er­ally ver­ti­cally in­te­grated re­gional mo­nop­o­lies, own­ing the land, track, and rolling stock, set­ting their own timeta­bles, and em­ploy­ing their staff. The state im­posed con­trols to stop them ex­ploit­ing their mo­nop­oly po­si­tion, but it did so cau­tiously, al­low­ing them to make suf­fi­cient profit that in­cen­tives to in­vest were pre­served. Capital sub­si­dies were tar­geted at pro­vid­ing spe­cific pub­lic goods that nor­mal com­mer­cial op­er­a­tions over­looked.

The above para­graph could be writ­ten by a his­to­rian of the fu­ture about con­tem­po­rary Japan. But every word in it could also be writ­ten by a his­to­rian to­day about the United States in the nine­teenth cen­tury — usu­ally seen as the epit­ome of cap­i­tal­ist in­di­vid­u­al­ism. This strik­ing fact con­tra­dicts the idea that America’s sup­posed in­di­vid­u­al­ism fore­or­dains it to be the land of the car, or that Japan’s sup­posed com­mu­ni­tar­i­an­ism fore­or­dained it to be the land of rail.

It also puts pres­sure on the idea that the demise of rail is the in­evitable con­se­quence of cars. All coun­tries saw some shift to cars in the twen­ti­eth cen­tury, and all rail in­dus­tries had to re­spond to that. But pub­lic pol­icy had an enor­mous ef­fect on how suc­cess­fully they did so. The rise of zon­ing re­stric­tions on den­sity, ex­ces­sive price con­trols, na­tion­al­iza­tion, and ver­ti­cally dis­in­te­grated pri­va­ti­za­tion have ham­pered Western rail in re­main­ing com­pet­i­tive against cars since the 1920s. By main­tain­ing and restor­ing the in­sti­tu­tions that built the first rail­way sys­tems in the nine­teenth cen­tury, the Japanese have cre­ated the might­i­est rail­way sys­tem of the twenty-first.

When they re­ceived the call to re­spond to an Israeli airstrike in the city of Mayfadoun, in south­ern Lebanon, most of the para­medics held back, hav­ing pre­vi­ously seen col­leagues killed by dou­ble-tap at­tacks tar­get­ing res­cuers. But the medics from the Islamic Health Association (IHA) rushed to the scene.

By the time the other emer­gency work­ers ar­rived at the site, they found the IHA medics had in­deed been caught in a sec­ond strike. They started evac­u­at­ing their wounded col­leagues, only for their am­bu­lances to be hit in two fur­ther at­tacks.

One of the para­medics cov­ered his ears and screamed, con­vuls­ing in pain as shrap­nel shat­tered the back win­dow of the am­bu­lance.

The res­cue mis­sion on Wednesday af­ter­noon had turned into a night­mare as Israel car­ried out three con­sec­u­tive strikes on three sets of am­bu­lances and med­ical work­ers.

In to­tal, the at­tacks killed four medics and wounded six more, from three dif­fer­ent am­bu­lance corps, ac­cord­ing to med­ical sources. Three of the medics were from the Hezbollah-affiliated IHA and Amal-affiliated med­ical corps, while one was from the Nabatieh emer­gency ser­vices or­gan­i­sa­tion. Under in­ter­na­tional law, all medics are pro­tected and are con­sid­ered non-com­bat­ants, re­gard­less of po­lit­i­cal af­fil­i­a­tion.

Rescuers in Lebanon have long been wary of the dou­ble-tap at­tack, when Israeli forces tar­get a lo­ca­tion, wait un­til peo­ple gather to help sur­vivors, and then strike again. Wednesday’s three-wave at­tack af­ter the ini­tial one prompted the coin­ing of a fear­some new term: the quadru­ple tap.

In a video taken by one of the para­medics at the site, res­cuers are seen load­ing two wounded peo­ple into their am­bu­lances when a bomb lands next to their ve­hi­cle. Paramedics rush to ex­tract the dri­ver, who is mo­tion­less and limp as they pull him from the am­bu­lance, which is splashed with blood. “Oh God, oh God,” the man film­ing can be heard say­ing. They carry two more blood-cov­ered medics out of their ve­hi­cle and on to stretch­ers.

Among the para­medics killed was Fadel Sarhan, 43, who is sur­vived by his eight-year-old daugh­ter.

“Fadel was a very loved per­son. He had a bold per­son­al­ity, but at the same time, he was emo­tional. He was well liked and re­spon­si­ble,” said Ali Nasr al-Deen, the head of the Mayfadoun civil de­fence cen­tre who grew up with Sarhan.

“He used to feed the cats and dogs. He would bring pet food from Beirut so they would­n’t go hun­gry. He was that kind of per­son, car­ing and at­ten­tive. It’s a huge loss for us,” said Nasr al-Deen.

Medics mourned their col­leagues on Thursday at fu­ner­als in Nabatieh, a city near Mayfadoun. Such events have be­come in­creas­ingly com­mon, with health­care work­ers killed by Israeli bomb­ings on a near daily ba­sis.

Mohammed Suleiman, whose 16-year-old son, Joud, was killed while on duty as a para­medic by an Israeli strike weeks ear­lier, joined his peers in bury­ing an­other of his friends on Thursday. A few hours af­ter the fu­ner­als, Israel car­ried out an­other wave of airstrikes on Nabatieh.

Israel has so far killed 91 health­care work­ers and wounded 214 more in Lebanon since the Israel-Hezbollah war started on 2 March. It has given lit­tle jus­ti­fi­ca­tion for its re­peated at­tacks on med­ical in­fra­struc­ture and work­ers, apart from ac­cus­ing Hezbollah of us­ing am­bu­lances and hos­pi­tals to trans­port fight­ers and weapons, with­out pro­vid­ing ev­i­dence for the claim.

The Lebanese min­istry of health ac­cused Israel of de­lib­er­ately tar­get­ing am­bu­lance crews. “Paramedics have be­come di­rect tar­gets, pur­sued re­lent­lessly in a bla­tant vi­o­la­tion that con­firms a to­tal dis­re­gard for all norms and prin­ci­ples es­tab­lished by in­ter­na­tional hu­man­i­tar­ian law,” the min­istry said in a state­ment.

The Israeli mil­i­tary did not im­me­di­ately re­spond to a re­quest for com­ment.

In the video taken of the quadru­ple tap on Wednesday, the frame was frozen on the in­te­rior of the am­bu­lances, as the Nabatieh emer­gency ser­vices high­lighted that the ve­hi­cle clearly con­tained no weapons.

A few hours af­ter Israel hit the am­bu­lances out­side Nabatieh, it bombed the vicin­ity of the gov­ern­men­tal hos­pi­tal in Tebnine, south Lebanon. It was the sec­ond time in two days that Israeli bomb­ings dam­aged the health­care fa­cil­ity, which is the only re­main­ing pub­lic hos­pi­tal in the area. The strikes in­jured 11 hos­pi­tal work­ers and dam­ag­ing the emer­gency de­part­ment, ac­cord­ing to the World Health Organization (WHO).

A video of Tebnine hos­pi­tal from 14 April showed work­ers try­ing to clear shat­tered con­crete and de­bris from the emer­gency de­part­ment af­ter a strike blew in the win­dows.

Commenting on the strike in Tebnine, the head of the WHO, Tedros Adhanom Ghebreyesus, said: “I re­it­er­ate the call for the im­me­di­ate pro­tec­tion of health­care fa­cil­i­ties, health work­ers, am­bu­lances and pa­tients. There must be safe, sus­tained and un­hin­dered hu­man­i­tar­ian ac­cess across Lebanon.”

An am­bu­lance in Tebnine was also struck on Thursday, lead­ing to the crit­i­cal in­jury of two medics, ac­cord­ing to the Lebanese min­istry of health. As health­care work­ers watched their col­leagues and friends be­ing killed by Israel, the men­tal toll was be­com­ing al­most too much to bear.

“We have to go to places to res­cue peo­ple, but then we get dou­ble tapped,” said Abbas Atwi, the head of the IHA’s emer­gency de­part­ment in Nabatieh, shortly af­ter a med­ical cen­tre was tar­geted in March, killing his friends and col­leagues. “But we will stay and keep go­ing, we will not leave.”

Sixteen bets made $100,000 each ac­cu­rately pre­dict­ing the tim­ing of the US airstrikes against Iran on 27 February. Later, a sin­gle user would make over $550,000 af­ter bet­ting that Ayatollah Ali Khamenei would top­ple, just mo­ments be­fore his as­sas­si­na­tion by Israeli forces. On 7 April, right be­fore Donald Trump an­nounced a tem­po­rary cease­fire with Iran, traders bet $950m that oil prices would come down. They did.

These bets and other well-timed wa­gers ac­cu­rately pre­dicted the pre­cise tim­ing of ma­jor de­vel­op­ments in the US-Israel war with Iran, cre­at­ing huge wind­falls and rais­ing con­cerns among law­mak­ers and ex­perts over po­ten­tial in­sider trad­ing.

Betting — once largely siloed to sport­ing events — has now spread to in­clude con­tracts on news events where in­sider in­for­ma­tion could give some traders an ad­van­tage.

The pro­lif­er­a­tion of on­line bet­ting mar­kets like Polymarket and Kalshi has al­lowed bets on vir­tu­ally any news event. It’s also eas­ier than ever to buy com­mod­ity de­riv­a­tives like oil fu­tures, where traders gam­ble on what the price of oil will be in the fu­ture.

Leaders of some US fed­eral agen­cies and some mem­bers of Congress said they want to crack down on sus­pi­cious trad­ing tak­ing place across dif­fer­ent mar­ket­places, but it’s un­clear how much head­way reg­u­la­tors will make.

“Is the prob­lem that we don’t have leg­is­la­tion or that we don’t have en­force­ment ca­pa­bil­i­ties?” said Joshua Mitts, a law pro­fes­sor at Columbia University. “To have a law that can’t re­ally be en­forced ef­fec­tively given the tech­no­log­i­cal lim­i­ta­tions, it’s sort of putting the cart be­fore the horse.”

On the night of 27 February, the day be­fore the US and Israel would carry out strikes on Iran, an un­usual in­flux of about 150 ac­counts on Polymarket placed bets that the US would strike Iran the next day. A New York Times analy­sis found the bets to­taled $855,000, with 16 ac­counts pock­et­ing more than $100,000 each.

Soon af­ter, a sin­gle anony­mous Polymarket user, un­der an ac­count named “Magamyman”, made over $553,000 af­ter bet­ting that Khamenei would be “removed” from power just mo­ments be­fore he was killed by an Israeli airstrike, ac­cord­ing to a com­plaint filed to the Commodity Futures Trading Commission (CFTC), the fed­eral agency that reg­u­lates fu­tures mar­kets, by Public Citizen, a con­sumer ad­vo­cacy group. The com­plaint also cites a crypto-an­a­lyt­ics firm that iden­ti­fied six “suspected in­sid­ers” who made a to­tal of $1.2m on Polymarket af­ter Khamenei was killed.

The well-timed surge of wa­gers were seen again on 7 April, when at least 50 Polymarket ac­counts placed bets that the US and Iran would reach a cease­fire hours be­fore Trump would an­nounce it in a Truth Social post. Earlier, the pres­i­dent had said “a whole civ­i­liza­tion will die tonight” if Iran did not open the strait of Hormuz.

But traders weren’t just ac­tive on Polymarket: there were sim­i­lar surges of oil fu­tures trad­ing ac­tiv­ity just hours be­fore Trump an­nounced up­dates to the con­flict that would lower oil prices.

On 23 March, traders placed $580m in bets on the oil fu­tures mar­ket just 15 min­utes be­fore Trump said on so­cial me­dia that the US was hav­ing “productive” talks with Iran, ac­cord­ing to the Financial Times. The traders made a wind­fall af­ter Trump’s com­ments trig­gered a sell-off in the oil mar­kets that made oil prices plum­met.

The same thing hap­pened again on 7 April, this time when traders spent $950m on oil fu­tures, bet­ting that the price of oil would fall just hours be­fore the cease­fire with Iran was an­nounced.

“We can’t say from the out­set whether any of these trades were il­le­gal. Any one of them could be lucky, and any one of them could be based on law­ful in­for­ma­tion,” said Andrew Verstein, a law pro­fes­sor at the University of California at Los Angeles. “But many of them bear the hall­marks of sus­pi­cious trades that would nat­u­rally war­rant in­ves­ti­ga­tion.”

For those who closely fol­low trad­ing pat­terns, the rush of ac­tiv­ity that hap­pened be­fore these events seem too big to sim­ply be bets hedg­ing on luck.

“Not only the tim­ing, but the amount of these bets makes it look very likely that some­one had in­sider knowl­edge … and placed very, very sub­stan­tial bets on it,” said Craig Holman, a gov­ern­ment af­fairs lob­by­ist for Public Citizen who filed the group’s com­plaint to the CFTC.

Holman said he is skep­ti­cal about how bold the CFTC will be in its in­ves­ti­ga­tions given its cur­rent struc­ture un­der the Trump ad­min­is­tra­tion. The com­mis­sion typ­i­cally has five bi­par­ti­san mem­bers that are ap­pointed by the pres­i­dent. Now, the CFTC has only one com­mis­sioner: Michael Selig, whom Trump ap­pointed at the end of 2025 and who has po­si­tioned him­self as friendly to­ward pre­dic­tion mar­kets.

Over the last few months, the CFTC has been roiled in fights with state leg­is­la­tures who ar­gue that reg­u­la­tion of these on­line bet­ting mar­ket­places be­longs to the states.

Kalshi, Polymarket’s com­peti­tor, was tem­porar­ily banned in Nevada af­ter the state sued the com­pany for of­fer­ing con­tacts in the state with­out a gam­bling li­cense. Arizona mean­while filed crim­i­nal charges against the com­pany for al­low­ing peo­ple to place bets on elec­tions. In both cases, Kalshi de­nied any wrong­do­ing and has ar­gued that the CFTC has ex­clu­sive ju­ris­dic­tion over on­line pre­dic­tion mar­kets.

“It’s a wild west phase, when we’re talk­ing about the pre­dic­tion mar­ket in­dus­try, and now it’s spilled over into the stock mar­ket as well,” Holman said.

Anonymous sources told Reuters and Bloomberg that the CFTC launched an in­ves­ti­ga­tion into the oil fu­tures trades that were placed on 27 March and 7 April, though the agency has not pub­licly an­nounced it is con­duct­ing an in­ves­ti­ga­tion.

Speaking to Congress this week, Selig said that the agency is pre­pared to go af­ter those who are sus­pected of in­sider trad­ing, warn­ing “we will find you and you will face the full force of the law”, but said that the com­mis­sion would not is­sue any new reg­u­la­tions un­til it had five seated com­mis­sion­ers.

Polymarket did not re­spond to re­quest for com­ment. In a state­ment, White House spokesper­son Davis Ingle said “federal em­ploy­ees are sub­ject to gov­ern­ment ethics guide­lines that pro­hibit the use of non­pub­lic in­for­ma­tion for fi­nan­cial ben­e­fit”.

“Any im­pli­ca­tion that ad­min­is­tra­tion of­fi­cials are en­gaged in such ac­tiv­ity with­out ev­i­dence is base­less and ir­re­spon­si­ble re­port­ing,” Ingle said. “The CFTC will al­ways up­hold its duty to mon­i­tor fraud, ma­nip­u­la­tion and il­licit ac­tiv­ity daily.”

Federal law pro­hibits gov­ern­ment em­ploy­ees, in­clud­ing those work­ing for Congress or the White House, from us­ing non-pub­lic in­for­ma­tion for per­sonal profit.

In late March, a bi­par­ti­san group of rep­re­sen­ta­tives in­tro­duced a bill that would ban mem­bers of Congress and se­nior staff within the fed­eral gov­ern­ment from par­tic­i­pat­ing in pre­dic­tion mar­ket con­tracts re­lated to po­lit­i­cal events or pol­icy de­ci­sions.

But ex­perts warn that in­sider trad­ing law is com­pli­cated, and the new tech­nol­ogy that makes it eas­ier to place bets on­line leaves a com­pli­cated pa­per trail that can be hard to fol­low.

Historically, in­sider trad­ing takes place when a per­son uses ex­clu­sive in­for­ma­tion about a com­pany to buy or sell stocks right be­fore in­for­ma­tion be­comes pub­lic. These types of il­le­gal trades are reg­u­lated by the Securities and Exchange Commission (SEC), which reg­u­lates the stock ex­changes.

Insider fu­tures trad­ing could be seen as a sub­set of this typ­i­cal in­sider trad­ing, but the ter­ri­tory is new.

“The trick is that there are es­sen­tially no clean cases of peo­ple get­ting in trou­ble for com­mod­ity fu­tures in­sider trad­ing,” Verstein said. “The law there is just not well-de­vel­oped.”

In a pa­per pub­lished last month, Mitts, the Columbia law pro­fes­sor, and other re­searchers screened more than 200,000 “suspicious wal­let-mar­ket pairs” from February 2024 to February 2026 and found that traders in this group achieved a nearly 70% win rate, mak­ing $143m in well-timed bets tied to every­thing from the cap­ture of for­mer Venezuelan leader Nicolás Maduro to Taylor Swift’s en­gage­ment to Travis Kelce. The pa­per notes that in­formed traders face fewer le­gal con­straints by trad­ing on plat­forms like Polymarket or Kalshi be­cause these mar­kets still op­er­ate in a le­gal gray area.

“The chal­lenge here is that this trad­ing is oc­cur­ring through the blockchain or other anonymized means, so it is go­ing to be quite dif­fi­cult for a reg­u­la­tor en­force­ment au­thor­ity or pros­e­cu­tor to de­ter­mine the iden­tity of the trader,” Mitts said. “They would also have to prove the trader traded on the ba­sis of in­for­ma­tion that had been wrongly mis­ap­pro­pri­ated.”

But the stakes are high. Insider trad­ing in­volv­ing clas­si­fied mil­i­tary in­for­ma­tion can lead to dis­trust of both mar­kets and gov­ern­ments.

“Unlike cor­po­rate in­sider trad­ing, there’s a lot of ways for the gov­ern­ment to make it­self be cor­rect. You can just make the war that would oc­cur, and that’s con­cern­ing be­cause then the real econ­omy is be­ing dis­torted,” Verstein said. “Real de­ci­sions, in­clud­ing per­haps fi­nan­cial de­ci­sions, are be­ing dis­torted by fi­nan­cial bets.”

I tried Claude Design yes­ter­day and I have a the­ory for how this whole thing shakes out.

As prod­uct teams scaled and de­sign needed to jus­tify it­self in­side en­gi­neer­ing orgs, it was pushed to­ward sys­tem­ati­za­tion — and Figma in­vented its own prim­i­tives to make that work: com­po­nents, styles, vari­ables, props, and so on. Some con­cepts are bor­rowed from pro­gram­ming, some aren’t, and the whole thing does­n’t neatly map onto any­thing. Guidance evolves, mi­gra­tions pile up, and if you want to au­to­mate any of it you’re stuck with a hand­ful of shoddy plu­g­ins. The beast is hairy enough that en­tire de­sign roles now spe­cial­ize in wran­gling the sys­tem it­self.

There’s al­ways been a tense push-pull be­tween Figma and code over what the source of truth should be. Figma won over Sketch par­tially by stak­ing its claim there — their tool­ing would be canon­i­cal.

That vic­tory had a hid­den cost. By na­ture of hav­ing a locked-down, largely-un­doc­u­mented for­mat that’s painful to work with pro­gram­mat­i­cally, Figma ac­ci­den­tally ex­cluded them­selves from the train­ing data that would have made them rel­e­vant in the agen­tic era. LLMs were trained on code, not Figma prim­i­tives, so mod­els never learned them. As code be­comes eas­ier for de­sign­ers to write and agents keep im­prov­ing, the source of truth will nat­u­rally mi­grate back to code. And all the baroque in­fra­struc­ture Figma had to in­tro­duce over the past decade will look nuts by com­par­i­son. Why fuss around in a lossy ap­prox­i­ma­tion of the thing when you can work di­rectly in the medium where it will ac­tu­ally live? If we want to make pot­tery, why are we paint­ing wa­ter­col­ors of the pot in­stead of just throw­ing the clay?

At work, we’ve spent quite a bit of time back-port­ing de­sign changes made di­rectly in code back to Figma and it is not fun. I can’t share that file, but for a fair com­par­i­son, this is Figma’s own de­sign sys­tem file for their prod­uct. I have to as­sume it was built by the most com­pe­tent de­sign sys­tem team you can find. And yet…

These are Figma’s own files. Built by their own team. This is the gold stan­dard.

Imagine de­bug­ging a color that looks wrong. You check the com­po­nent. The com­po­nent uses a vari­able. The vari­able is aliased to an­other vari­able. That vari­able ref­er­ences a mode. The mode is over­rid­den at the in­stance level. The in­stance lives in­side a nested com­po­nent with a li­brary swap ap­plied. At this point, you’re ei­ther con­sid­er­ing pick­ing up code or mov­ing to the coun­try­side and be­com­ing a sheep farmer be­cause one more minute of this will make you lose your god­damn mind.

So as the source of truth shifts back to code, Figma is left in an odd spot: hold­ing a largely man­ual, pre-agen­tic sys­tem that no­body in their right mind would de­sign from scratch to­day.

I think de­sign tool­ing forks into two dis­tinct shapes from here — and there’s al­most a clock re­set­ting be­tween Figma and every other tool com­pet­ing to an­swer the same ques­tion they an­swered in 2016: who can help me, a de­signer, get my ideas out fastest?

Spoiler: it’s not Figma Make. Figma Make feels like it pri­mar­ily ben­e­fits peo­ple who have al­ready drunk the Kool-Aid — it reads from Figma styles, com­po­nent li­braries, and pro­pri­etary props (or, as I like to call them, Prop Props), and it’s the only tool in this new land­scape still pre­tend­ing the de­sign file is canon­i­cal. It’s the tool for peo­ple who want to (or have no choice but to) stay in­side the sys­tem.

Claude Design is the first of those two tools, and takes the op­po­site bet. There’s an Arts and Crafts prin­ci­ple called “truth to ma­te­ri­als” — the idea that a thing should be hon­est about what it is and how it’s made, rather than mas­querad­ing as some­thing else. Figma ended up be­ing the op­po­site of this: a set of ex­tremely rigid schemas with a free-form “just vibes, man” cos­tume over the top. Like a Type-A per­son­al­ity phys­i­cally in­ca­pable of re­lax­ing, forced to per­form chill while in­ter­nally scream­ing that your frames aren’t nested and your to­kens are de­tached and noth­ing is on the grid. Claude Design, for all its rough­ness, is at least hon­est about what it is: HTML and JS all the way down.

And it has a mas­sive struc­tural ad­van­tage: its sib­ling is Claude Code. Eventually, I can see Claude Design just dump­ing things di­rectly into Claude Code and vice versa. Claude Design’s on­board­ing al­ready lets you im­port your re­pos. The feed­back loop be­tween de­sign and im­ple­men­ta­tion — which has been a source of fric­tion since the be­gin­ning of time — be­comes a sin­gle con­ver­sa­tion.

The other tool that emerges from this mo­ment will have no ex­pec­ta­tion of code at all. It’ll be a pure ex­plo­ration en­vi­ron­ment — some­where to drop rec­tan­gles, stack layer styles, fuss with blend modes and gra­di­ents, and go com­pletely nuts, un­con­strained by sys­tems or prompt­ing con­ven­tions. Maybe it’s an iPad app with Pencil sup­port where you just quickly sketch a bunch of rec­tan­gles. 37signals could do some­thing re­ally funny right now. Or maybe it goes in the op­po­site di­rec­tion — some­thing more like Photoshop that goes all-in on high-fi­delity com­posit­ing and lets our imag­i­na­tions run wild, now that we’re no longer be­holden to the ceil­ing of what you can do with CSS ef­fects. Doesn’t it seem kinda weird how for 90% of its life, Figma’s only layer ef­fect was a drop shadow or a blur?

Figma’s Sketch mo­ment is rapidly ap­proach­ing. And if you said that sen­tence to a Victorian child, they would prob­a­bly have a stroke.

The fol­low­ing are mes­sages meant only for the teams be­hind Sketch and Figma. If nei­ther ap­ply to you, you can skedad­dle.

To Figma: I can see a world where this post does num­bers in the Figma in­ter­nal Slack. If that’s the case and you’re read­ing this from Figma: this would­n’t have hap­pened if you hired me last year when I was in­ter­view­ing. Your loss, big dawg.

To Sketch: GET YOUR HEADS OUTTA YOUR ASSES AND GIVE EM HELL. ADD PARTICLE EFFECTS. ADD DEBOSSING EFFECTS. MESH TRANSFORMS. FUCK IT, ADD METAL SHADERS. GO NUTS. STOP COASTING OFF OF BEING MAC NATIVE. QUIT DRINKING COCOA AND GET THIRSTY FOR BLOOD.

To mom: Sorry for curs­ing.

@jonnyburch on Twitter shared a link to their blog post with sim­i­lar thoughts, it’s quite good if you wanna go deeper.

Given a set of ob­jects, there can be nu­mer­ous cri­te­ria, based on which to or­der them (depending on the ob­jects them­selves) — size, weight, age, al­pha­bet­i­cal or­der etc.

However, cur­rently we are not in­ter­ested in the cri­te­ria that we can use to or­der ob­jects, but in the na­ture of the re­la­tion­ships that de­fine or­der. Of which there can be sev­eral types as well.

Mathematically, the or­der as a con­struct is rep­re­sented (much like a monoid) by two com­po­nents.

An or­der is a set of el­e­ments, to­gether with a bi­nary re­la­tion be­tween the el­e­ments of the set, which obeys cer­tain laws.

We de­note the el­e­ments of our set, as usual, like this.

And the bi­nary re­la­tion is a re­la­tion be­tween two el­e­ments, which is of­ten de­noted with an ar­row.

As for the laws, they are dif­fer­ent de­pend­ing on the type of or­der.

Let’s start with an ex­am­ple — the most straight­for­ward type of or­der that you think of is lin­ear or­der i.e. one in which every ob­ject has its place de­pend­ing on every other ob­ject. In this case the or­der­ing cri­te­ria is com­pletely de­ter­min­is­tic and leaves no room for am­bi­gu­ity in terms of which el­e­ment comes be­fore which. For ex­am­ple, or­der of col­ors, sorted by the length of their light-waves (or by how they ap­pear in the rain­bow).

Using set the­ory, we can rep­re­sent this or­der, as well as any other or­der, as a sets of pairs of the or­der’s un­der­ly­ing set with it­self (a sub­set of the prod­uct set).

And in pro­gram­ming, or­ders are de­fined by pro­vid­ing a func­tion which, given two ob­jects, tells us which one of them is “bigger” (comes first) and which one is “smaller”. It is­n’t hard to see that this func­tion de­fines a set of pairs (we are given a pair and we have to say whether or not it be­longs to the set).

However (this is where it gets in­ter­est­ing) not all such func­tions (and not all sets of pairs) de­fine or­ders. For such func­tion to re­ally de­fine an or­der i.e. to have the same out­put every time, in­de­pen­dent of how the ob­jects were shuf­fled ini­tially, it has to obey sev­eral rules.

Incidentally, (or rather not in­ci­den­tally at all), these rules are nearly equiv­a­lent to the math­e­mat­i­cal laws that de­fine the cri­te­ria of the or­der re­la­tion­ship i.e. those are the rules that de­fine which el­e­ment can point to which.

A lin­ear or­der is a set of el­e­ments, to­gether with a bi­nary re­la­tion be­tween the el­e­ments of the set, which obeys the laws of re­flex­iv­ity, tran­si­tiv­ity, an­ti­sym­me­try, to­tal­ity.

Let’s check what they are.

Let’s get the bor­ing law out of the way — each ob­ject has to be big­ger or equal to it­self, or $a ≤ a$ for all $a$ (the re­la­tion­ship be­tween el­e­ments in an or­der is com­monly de­noted as $≤$ in for­mu­las, but it can also be rep­re­sented with an ar­row from first ob­ject to the sec­ond.)

This law only ex­ist to cover the “base case”: we can for­mu­late it the op­po­site way too and say that each ob­ject should not have the re­la­tion­ship to it­self, in which case we would have a re­la­tion than re­sem­bles big­ger than, as op­posed to big­ger or equal to and a slightly dif­fer­ent type of or­der, some­times called a strict or­der.

The sec­ond law is maybe the least ob­vi­ous, (but prob­a­bly the most es­sen­tial) — it states that if ob­ject $a$ is big­ger than ob­ject $b$, it is au­to­mat­i­cally big­ger than all ob­jects that are smaller than $b$ or $a ≤ b \land b ≤ c \to a ≤ c$.

This is the law that to a large ex­tend de­fines what an or­der is: if I am bet­ter at play­ing soc­cer than my grand­mother, then I would also be bet­ter at it than my grand­moth­er’s friend, whom she beats, oth­er­wise I would­n’t re­ally be bet­ter than her.

The third law is called an­ti­sym­me­try. It states that the func­tion that de­fines the or­der should not give con­tra­dic­tory re­sults (or in other words you have $x ≤ y$ and $y ≤ x$ only if $x = y$).

It also means that no ties are per­mit­ted — ei­ther I am bet­ter than my grand­mother at soc­cer or she is bet­ter at it than me.

The last law is called to­tal­ity (or con­nex­ity) and it man­dates that all el­e­ments that be­long to the or­der should be com­pa­ra­ble ($a ≤ b \lor b ≤ a$). That is, for any two el­e­ments, one would al­ways be “bigger” than the other.

By the way, the law of to­tal­ity makes the re­flex­iv­ity law re­dun­dant, as re­flex­iv­ity is just a spe­cial case of to­tal­ity when $a$ and $b$ are one and the same ob­ject, but I still want to pre­sent it for rea­sons that will be­come ap­par­ent soon.

Actually, here are the rea­sons: the law of to­tal­ity can be re­moved. Orders, that don’t fol­low the to­tal­ity law are called par­tial or­ders, (and lin­ear or­ders are also called to­tal or­ders.)

Task 1: Previously, we cov­ered a re­la­tion that is pretty sim­i­lar to this. Do you re­mem­ber it? What is the dif­fer­ence?

Task 2: Think about some or­ders that you know about and fig­ure out whether they are par­tial or to­tal.

Partial or­ders are ac­tu­ally much more in­ter­est­ing than lin­ear/​to­tal or­ders. But be­fore we dive into them, let’s say a few things about num­bers.

Natural num­bers form a lin­ear or­der un­der the op­er­a­tion big­ger or equal to (the sym­bol of which we have been us­ing in our for­mu­las.)

In many ways, nat­ural num­bers are the quin­tes­sen­tial or­der — every fi­nite or­der of ob­jects is iso­mor­phic to a sub­set of the or­der of num­bers, as we can map the first el­e­ment of any or­der to the num­ber $1$, the sec­ond one to the num­ber $2$ etc (and we can do the op­po­site op­er­a­tion as well).

If we think about it, this iso­mor­phism is ac­tu­ally closer to the every­day no­tion of a lin­ear or­der, than the one de­fined by the laws — when most peo­ple think of or­der, they aren’t think­ing of a tran­si­tive, an­ti­sym­met­ric and to­tal re­la­tion, but are rather think­ing about cri­te­ria based on which they can de­cide which ob­ject comes first, which comes sec­ond etc. So it’s im­por­tant to no­tice that the two no­tions are equiv­a­lent.

From the fact that any fi­nite or­der of ob­jects is iso­mor­phic to the nat­ural num­bers, it also fol­lows that all lin­ear or­ders of the same mag­ni­tude are iso­mor­phic to one an­other.

So, the lin­ear or­der is sim­ple, but it is also (and I think that this iso­mor­phism proves it) the most bor­ing or­der ever, es­pe­cially when looked from a cat­e­gory-the­o­retic view­point — all fi­nite lin­ear or­ders (and most in­fi­nite ones) are just iso­mor­phic to the nat­ural num­bers and so all of their di­a­grams look the same way.

However, this is not the case with par­tial or­ders that we will look into next.

Law of to­tal­ity does not look so “set in stone” as the rest of the laws i.e. we can prob­a­bly think of some sit­u­a­tions in which it does not ap­ply. For ex­am­ple, if we aim to or­der all peo­ple based on soc­cer skills there are many ways in which we can rank a per­son com­pared to their friends their friend’s friends etc. but there is­n’t a way to or­der groups of peo­ple who never played with one an­other.

Remove the law of to­tal­ity from the laws of lin­ear or­ders and we get a par­tial or­der (also a par­tially-or­dered set, or poset).

An par­tial or­der is a set of el­e­ments, to­gether with a bi­nary re­la­tion be­tween the el­e­ments of the set, which obeys the laws of re­flex­iv­ity, tran­si­tiv­ity and an­ti­sym­me­try.

Every lin­ear or­der is also a par­tial or­der (just as a group is still a monoid), but not the other way around.

We can even cre­ate an or­der of or­ders, based on which is more gen­eral.

Partial or­ders are also re­lated to the con­cept of an equiv­a­lence re­la­tions that we cov­ered in chap­ter 1, ex­cept that sym­me­try law is re­placed with an­ti­sym­me­try.

If we re­visit the ex­am­ple of the soc­cer play­ers rank list, we can see that the first ver­sion that in­cludes just my­self, my grand­mother and her friend is a lin­ear or­der.

However, in­clud­ing this other per­son whom none of us played yet, makes the hi­er­ar­chy non-lin­ear i.e. a par­tial or­der.

This is the main dif­fer­ence be­tween par­tial and to­tal or­ders — par­tial or­ders can­not pro­vide us with a def­i­nite an­swer of the ques­tion who is bet­ter than who. But some­times this is what we need — in sports, as well as in other do­mains, there is­n’t al­ways an ap­pro­pri­ate way to rate el­e­ments lin­early.

Before, we said that all lin­ear or­ders can be rep­re­sented by the same chain-like di­a­gram, we can re­verse this state­ment and say that all di­a­grams that look some­thing dif­fer­ent than the said di­a­gram rep­re­sent par­tial or­ders.

An ex­am­ple of this is a par­tial or­der that con­tains a bunch of lin­early-or­dered sub­sets, e.g. in our soc­cer ex­am­ple we can have sep­a­rate groups of friends who play to­gether and are ranked with each other, but not with any­one from other groups.

The dif­fer­ent lin­ear or­ders that make up the par­tial or­der are called chains. There are two chains in this di­a­gram $m \to g \to f$ and $d \to o$.

The chains in an or­der don’t have to be com­pletely dis­con­nected from each other in or­der for it to be par­tial. They can be con­nected as long as the con­nec­tions are not all one-to-one i.e. ones when the last el­e­ment from one chain is con­nected to the first el­e­ment of the other one (this would ef­fec­tively unite them into one chain.)

The above set is not lin­early-or­dered — al­though we know that $d ≤ g$ and that $f ≤ g$, the re­la­tion­ship be­tween $d$ and $f$ is not known — any el­e­ment can be big­ger than the other one.

Although par­tial or­ders don’t give us a de­fin­i­tive an­swer to “Who is bet­ter than who?”, some of them still can give us an an­swer to the more im­por­tant ques­tion (in sports, as well as in other do­mains), namely “Who is num­ber one?” i.e. who is the cham­pion, the player who is bet­ter than any­one else. Or, more gen­er­ally, the el­e­ment that is big­ger than all other el­e­ments.

The great­est el­e­ment of an or­der is an el­e­ment $a$, such that we have we have $x ≤ a$ for any other el­e­ment $x$, Some (not all) par­tial or­ders do have such el­e­ment — in our last di­a­gram $m$ is the great­est el­e­ment, in this di­a­gram, the green el­e­ment is the biggest one.

Sometimes we have more than one el­e­ments that are big­ger than all other el­e­ments, in this case none of them is the great­est.

In ad­di­tion to the great­est el­e­ment, a par­tial or­der may also have a least (smallest) el­e­ment, which is de­fined in the same way.

The least up­per bound of two el­e­ments that are con­nected as part of an or­der is called the join of these el­e­ments, e.g. the green el­e­ment is a join of the other two.

The join of $a$ and $b$ is the small­est el­e­ment $c$ that is big­ger than then, for­mally:

The join of ob­jects $A$ and $B$ is an ob­ject $G$, such that:

It is big­ger than both of these ob­jects, so $A ≤ G$ and $B ≤ G$.

It is smaller than any other ob­ject that is big­ger than them, so for any other ob­ject $P$ such that $P ≤ A$ and $P ≤ B$ then we should also have $G ≤ P$.

Given any two el­e­ments in which one is big­ger than the other (e.g. $a ≤ b$), the join is this big­ger el­e­ment (in this case $b$)

e.g. in a lin­ear or­ders, the join of any two el­e­ments is just the big­ger el­e­ment.

Like with the great­est el­e­ment, if two el­e­ments have sev­eral up­per bounds that are equally big, then none of them is a join (a join must be unique).

If, how­ever, one of those el­e­ments is es­tab­lished as smaller than the rest of them, it im­me­di­ately qual­i­fies.

Task 3: Which con­cept in cat­e­gory the­ory re­minds you of joins?

Given two el­e­ments, the biggest el­e­ment that is smaller than both of them is called the meet of these el­e­ments.

The same rules as for the joins ap­ply, but in re­verse.

The di­a­grams that we use in this sec­tion are called “Hasse di­a­grams” and they work much like our usual di­a­grams, how­ever they have an ad­di­tional rule that is fol­lowed — “bigger” el­e­ments are al­ways po­si­tioned above smaller ones.

In terms of ar­rows, the rule means that if you add an ar­row to a point, the point to which the ar­row points must al­ways be above the one from which it points.

This arrange­ment al­lows us to com­pare any two points by just see­ing which one is above the other e.g. we can de­ter­mine the join of two el­e­ments, by just iden­ti­fy­ing the el­e­ments that they con­nect to and see which one is low­est.

We all know many ex­am­ples of to­tal or­ders (any form of chart or rank­ing is a to­tal or­der), but there are prob­a­bly not so many ob­vi­ous ex­am­ples of par­tial or­ders that we can think of off the top of our head. So let’s see some. This will gives us some con­text, and will help us un­der­stand what joins are.

To stay true to our form, let’s re­visit our color-mix­ing monoid and cre­ate a color-mix­ing par­tial or­der in which all col­ors point to col­ors that con­tain them.

If you go through it, you will no­tice that the join of any two col­ors is the color that they make up when mixed. Nice, right?

The par­tial or­der of num­bers by di­vi­sion

We saw that when we or­der num­bers by “bigger or equal to”, they form a lin­ear or­der. But num­bers can also form a par­tial or­der, for ex­am­ple they form a par­tial or­der if we or­der them by which di­vides which, i.e. if $a$ di­vides $b$, then $a$ is be­fore $b$ e.g. be­cause $2 \times 5 = 10$, $2$ and $5$ come be­fore $10$ (but $3$, for ex­am­ple, does not come be­fore $10$.)

And it so hap­pens (actually for very good rea­son) that the join op­er­a­tion again cor­re­sponds to an op­er­a­tion that is rel­e­vant in the con­text of the ob­jects — the join of two num­bers in this par­tial or­der is their least com­mon mul­ti­ple.

And the meet (the op­po­site of join) of two num­bers is their great­est com­mon di­vi­sor.

Given a col­lec­tion of sets con­tain­ing a com­bi­na­tion of a given set of el­e­ments…

…we can de­fine what is called the in­clu­sion or­der of those sets.

The in­clu­sion or­der of sets is a bi­nary re­la­tion that we can use to or­der a col­lec­tion of sets (usually sets that con­tain some com­mon el­e­ments) in which $a$ comes be­fore $b$ if $a$ in­cludes $b$, or in other words if $b$ is a sub­set of $a$.

In this case the join op­er­a­tion of two sets is their union, and the meet op­er­a­tion is their set in­ter­sec­tion.

This di­a­gram might re­mind you of some­thing — if we take the col­ors that are con­tained in each set and mix them into one color, we get the color-blend­ing par­tial or­der that we saw ear­lier.

The or­der ex­am­ple with the num­ber di­viders is also iso­mor­phic to an in­clu­sion or­der, namely the in­clu­sion or­der of all pos­si­ble sets of prime num­bers, in­clud­ing re­peat­ing ones (or al­ter­na­tively the set of all prime pow­ers). This is con­firmed by the fun­da­men­tal the­ory of arith­metic, which states that every num­ber can be writ­ten as a prod­uct of primes in ex­actly one way.

So far, we saw two dif­fer­ent par­tial or­ders, one based on color mix­ing, and one based on num­ber di­vi­sion, that can be rep­re­sented by the in­clu­sion or­ders of all pos­si­ble com­bi­na­tions of sets of some ba­sic el­e­ments (the pri­mary col­ors in the first case, and the prime num­bers (or prime pow­ers) in the sec­ond one.) Many other par­tial or­ders can be de­fined in this way. Which ones ex­actly, is a ques­tion that is an­swered by an amaz­ing re­sult called Birkhoff’s rep­re­sen­ta­tion the­o­rem. They are the fi­nite par­tial or­ders that meet the fol­low­ing two cri­te­ria:

All el­e­ments have joins and meets.

Those meet and join op­er­a­tions dis­trib­ute over one an­other, that is if we de­note joins as meets as $∨$ or $∧$, then $x ∨ (y ∧ z) = (x ∨ y) ∧ (x ∨ z)$.

The par­tial or­ders that meet the first cri­te­ria are called lat­tices. The ones that meet the sec­ond one are called dis­trib­u­tive lat­tices. Let’s write that down:

Partial or­ders in which all el­e­ments have joins and meets is called a lat­tice. A lat­tice whose meet and join op­er­a­tions dis­trib­ute over one an­other is called a dis­trib­u­tive lat­tice.

And the “prime” el­e­ments which we use to con­struct the in­clu­sion or­der are the el­e­ments that are not the join of any other el­e­ments. They are also called join-ir­re­ducible el­e­ments.

So we may phrase the the­o­rem like this:

Each dis­trib­u­tive lat­tice is iso­mor­phic to an in­clu­sion or­der of its join-ir­re­ducible el­e­ments.

By the way, the par­tial or­ders that are not dis­trib­u­tive lat­tices are also iso­mor­phic to in­clu­sion or­ders, it is just that they are iso­mor­phic to in­clu­sion or­ders that do not con­tain all pos­si­ble com­bi­na­tions of el­e­ments.

We will now talk more about lat­tices (the or­ders for which Birkhoff’s the­o­rem ap­plies). Lattices are par­tial or­ders, in which every two el­e­ments have a join and a meet. So every lat­tice is also par­tial or­der, but not every par­tial or­der is a lat­tice (we will see even more mem­bers of this hi­er­ar­chy).

Most par­tial or­ders that are cre­ated based on some sort of rule are dis­trib­u­tive lat­tices, like for ex­am­ple the par­tial or­ders from the pre­vi­ous sec­tion are also dis­trib­u­tive lat­tices when they are drawn in full, for ex­am­ple the color-mix­ing or­der.

Notice that we added the black ball at the top and the white one at the bot­tom. We did that be­cause oth­er­wise the top three el­e­ments would­n’t have a join el­e­ment, and the bot­tom three would­n’t have a meet.

Our color-mix­ing lat­tice, has a great­est el­e­ment (the black ball) and a least el­e­ment (the white one). Lattices that have a least and great­est el­e­ments are called bounded lat­tices. It is­n’t hard to see that all fi­nite lat­tices are also bounded.

Task 4: Prove that all fi­nite lat­tices are bounded.

We men­tioned or­der iso­mor­phisms sev­eral times al­ready so this is about time to elab­o­rate on what they are.

Given two sets (we will use par­tial or­der of num­bers by di­vi­sion and the prime in­clu­sion or­der as an ex­am­ple) an iso­mor­phism be­tween them is com­prised of the fol­low­ing two func­tions:

One func­tion from the prime in­clu­sion or­der, to the num­ber or­der (which in this case is just the mul­ti­pli­ca­tion of all the el­e­ments in the set)

One func­tion from the num­ber or­der to the prime in­clu­sion or­der (which is an op­er­a­tion called prime fac­tor­iza­tion of a num­ber, con­sist­ing of find­ing the set of prime num­bers that re­sult in that num­ber when mul­ti­plied with one an­other).

An or­der iso­mor­phism is es­sen­tially an iso­mor­phism be­tween the or­ders’ un­der­ly­ing sets (invertible func­tion). However, be­sides their un­der­ly­ing sets, or­ders also have the ar­rows that con­nect them, so there is one more con­di­tion: in or­der for an in­vert­ible func­tion to con­sti­tute an or­der iso­mor­phism, it has to re­spect those ar­rows.

An iso­mor­phism be­tween two or­ders is an in­vert­ible func­tion be­tween their un­der­ly­ing sets, such that ap­ply­ing this func­tion (let’s call it $F$) to any two el­e­ments that have a cer­tain or­der in one set (let’s call them $a$ and $b$) should re­sult in two el­e­ments that have a cor­re­spond­ing or­der in the other set (i.e. $a ≤ b$ if and only if $F(a) ≤ F(b)$).

In the pre­vi­ous sec­tion, we saw how re­mov­ing the law of to­tal­ity from the laws of (linear) or­der pro­duces a dif­fer­ent (and some­what more in­ter­est­ing) struc­ture, called par­tial or­der. Now let’s see what will hap­pen if we re­move an­other one of the laws, namely the an­ti­sym­me­try law.

The an­ti­sym­me­try law man­dated that you can­not have an ob­ject that is at the same time smaller and big­ger than an­other one. (or that $a ≤ b ⟺ b ≰ a$).

Computer chips that cram bil­lions of elec­tronic de­vices into a few square inches have pow­ered the dig­i­tal econ­omy and trans­formed the world. Scientists may be on the cusp of launch­ing a sim­i­lar tech­no­log­i­cal rev­o­lu­tion — this time us­ing light.

In a sig­nif­i­cant ad­vance to­ward that goal, National Institute of Standards and Technology (NIST) sci­en­tists and col­lab­o­ra­tors have pi­o­neered a way to make in­te­grated cir­cuits for light by de­posit­ing com­plex pat­terns of spe­cial­ized ma­te­ri­als onto sil­i­con wafers. These so-called pho­ton­ics chips use op­ti­cal de­vices such as lasers, wave­guides, fil­ters and switches to shut­tle light around and process in­for­ma­tion. The new ad­vance could pro­vide a big boost for emerg­ing tech­nolo­gies such as ar­ti­fi­cial in­tel­li­gence, quan­tum com­put­ers and op­ti­cal atomic clocks.

Making cir­cuitry for light as pow­er­ful and ubiq­ui­tous as cir­cuitry for elec­trons is one of to­day’s tech­no­log­i­cal fron­tiers, says Scott Papp, a NIST physi­cist whose group led the re­search, pub­lished this week in Nature. “We’re learn­ing to make com­plex cir­cuits with many func­tions, cut­ting across many ap­pli­ca­tion ar­eas.”

When it comes to in­for­ma­tion trans­fer and pro­cess­ing, light can do things that elec­tric­ity can’t. Photons — par­ti­cles of light — are far zip­pier than elec­trons at work­ing their way through cir­cuits.

Laser light is also es­sen­tial for con­trol­ling pow­er­ful, emerg­ing quan­tum tech­nolo­gies such as op­ti­cal atomic clocks and quan­tum com­put­ers.

But sev­eral hur­dles re­main be­fore in­te­grated pho­ton­ics can truly hit its stride. One in­volves lasers. High-quality, com­pact and ef­fi­cient lasers ex­ist in only a few wave­lengths, or col­ors, of light. For ex­am­ple, semi­con­duc­tor lasers are very good at gen­er­at­ing in­frared light with a wave­length of 980 nanome­ters, or bil­lionths of a me­ter — a color just out­side the range of hu­man vi­sion.

Emerging tech­nolo­gies such as op­ti­cal atomic clocks and quan­tum com­put­ers need laser light in many other col­ors as well. The lasers that pro­duce those col­ors are big, costly and power-hun­gry, ef­fec­tively con­fin­ing these quan­tum tech­nolo­gies to a hand­ful of spe­cial-pur­pose labs.

By in­te­grat­ing lasers into cir­cuits on chips, sci­en­tists hope to help quan­tum tech­nolo­gies be­come cheaper and more portable, so they can start to ful­fill their vast promise.

The new NIST pho­ton­ics chip is a bit like a layer cake. NIST physi­cists Papp and Grant Brodnik, along with col­leagues, started with a stan­dard wafer of sil­i­con coated with sil­i­con diox­ide (glass) and lithium nio­bate, a so-called non­lin­ear ma­te­r­ial that can change the color of light com­ing into it.

The re­searchers then added pieces of metal to elec­tri­cally con­trol how the cir­cuits con­vert one color of light to oth­ers. The sci­en­tists also cre­ated other metal-lithium nio­bate in­ter­faces that al­lowed them to rapidly turn light on and off within the cir­cuits — a cru­cial abil­ity for data pro­cess­ing and high-speed rout­ing.

The ic­ing on the cake, so to speak, was a sec­ond non­lin­ear ma­te­r­ial called tan­ta­lum pen­tox­ide, or tan­tala. Tantala can trans­form light in ways that feel like magic, tak­ing in a sin­gle laser color and putting out the full rain­bow of vis­i­ble light col­ors plus a wide range of in­frared wave­lengths. Papp and col­leagues have spent years de­vel­op­ing tech­niques to fab­ri­cate cir­cuits out of tan­tala with­out heat­ing it up, al­low­ing the ma­te­r­ial to be de­posited onto other ma­te­ri­als with­out dam­ag­ing them.

By pat­tern­ing the dif­fer­ent ma­te­ri­als on top of each other in a three-di­men­sional stack, the re­searchers pro­duced a sin­gle chip that ef­fi­ciently routes light be­tween lay­ers. That al­lowed them to merge the light-ma­nip­u­lat­ing wiz­ardry of tan­tala with the con­trol­la­bil­ity of lithium nio­bate. The new tech­nique “allows seam­less in­te­gra­tion,” says Brodnik. “The real power is that tan­tala can be added to ex­ist­ing cir­cuitry.”

Ultimately, the re­searchers were able to fit roughly 50 fin­ger­nail-sized chips con­tain­ing 10,000 pho­tonic cir­cuits, each out­putting a unique color, onto a wafer roughly the size of a beer coaster. “We can cre­ate all these dif­fer­ent col­ors, just by de­sign­ing cir­cuits,” says Papp.

Quantum tech­nolo­gies such as clocks and com­put­ers could be among the biggest ben­e­fi­cia­ries of in­te­grated pho­ton­ics. These de­vices of­ten use ar­rays of atoms to store and process in­for­ma­tion. For each type of atom, physi­cists need lasers tai­lored to the atom’s in­ter­nal quan­tum en­ergy lev­els. For ex­am­ple, ru­bid­ium atoms, com­monly used in quan­tum com­put­ers and clocks, re­spond to red light with a wave­length of 780 nanome­ters. Strontium atoms, an­other pop­u­lar choice, “see” blue light at 461 nanome­ters. Shine other col­ors on the atoms and noth­ing hap­pens.

The bulky, costly and com­pli­cated lasers needed to pro­duce these be­spoke col­ors have been a ma­jor hin­drance to get­ting quan­tum com­put­ers and op­ti­cal clocks out of the lab and into the field, where they could have big im­pacts. Cheap, low-power, portable op­ti­cal clocks, for ex­am­ple, could help pre­dict vol­canic erup­tions and earth­quakes, of­fer an al­ter­na­tive to GPS for po­si­tion­ing and nav­i­ga­tion, and help sci­en­tists in­ves­ti­gate sci­en­tific mys­ter­ies such as the na­ture of dark mat­ter. Quantum com­put­ers could of­fer new ways to study the physics and chem­istry of drugs and ma­te­ri­als.

Integrated pho­tonic cir­cuits aren’t just for quan­tum. Papp be­lieves NIST’s pho­ton­ics chips could help ef­fi­ciently shut­tle sig­nals be­tween the spe­cial­ized chips used by tech firms, po­ten­tially mak­ing AI-based tools more pow­er­ful and ef­fi­cient. Tech com­pa­nies are also in­ter­ested in us­ing pho­ton­ics to im­prove vir­tual re­al­ity dis­plays.

While NIST’s chips aren’t yet ready for mass pro­duc­tion, the tech­nique used to cre­ate them pro­vides a path for­ward, Papp and Brodnik say. The NIST sci­en­tists col­lab­o­rated with ex­perts at Octave Photonics, a Louisville, Colorado-based startup com­pany founded by for­mer NIST re­searchers that’s now work­ing to scale up the tech­nol­ogy.

“When you see the chip glow­ing in the lab, tak­ing in in­vis­i­ble light and mak­ing all this vis­i­ble light in one in­te­grated chip — it’s ob­vi­ous how many po­ten­tial ap­pli­ca­tions there could be,” says Papp.