Why pro­cras­ti­na­tion is about man­ag­ing emo­tions, not timeAd­dress the real rea­sons you pro­cras­ti­nate and you’re more likely to start achiev­ing your goals. Like many writ­ers, I’m a supreme ex­pert at pro­cras­ti­na­tion. When I ought to be work­ing on an as­sign­ment, with the clock tick­ing to­wards my dead­line, I’ll sit there watch­ing point­less po­lit­i­cal in­ter­views or box­ing high­lights on YouTube (cat videos aren’t my thing). At its worst I can al­most be­gin to feel a lit­tle crazy – you need to be work­ing, I say to my­self, so what on Earth are you do­ing?

According to tra­di­tional think­ing — still es­poused by uni­ver­sity coun­selling cen­tres around the world, such as the University of Manchester in the UK and the University of Rochester in the US — I, along with my fel­low pro­cras­ti­na­tors, have a time man­age­ment prob­lem. By this view, I haven’t fully ap­pre­ci­ated how long my as­sign­ment is go­ing to take and I’m not pay­ing enough at­ten­tion to how much time I’m cur­rently wast­ing on ‘cyberloafing’. With bet­ter sched­ul­ing and a bet­ter grip on time, so the logic goes, I will stop pro­cras­ti­nat­ing and get on with my work.

Increasingly, how­ever, psy­chol­o­gists are re­al­is­ing this is wrong. Experts like Tim Pychyl at Carleton University in Canada and his col­lab­o­ra­tor Fuschia Sirois at the University of Sheffield in the UK have pro­posed that pro­cras­ti­na­tion is an is­sue with man­ag­ing our emo­tions, not our time. The task we’re putting off is mak­ing us feel bad – perhaps it’s bor­ing, too dif­fi­cult or we’re wor­ried about fail­ing – and to make our­selves feel bet­ter in the mo­ment, we start do­ing some­thing else, like watch­ing videos.Chronic pro­cras­ti­na­tion is linked with men­tal and phys­i­cal health costs, from de­pres­sion and anx­i­ety to car­dio­vas­cu­lar dis­ease (Credit: Alamy)This fresh per­spec­tive on pro­cras­ti­na­tion is be­gin­ning to open up ex­cit­ing new ap­proaches to re­duc­ing the habit; it could even help you im­prove your own ap­proach to work. “Self-change of any of sort is not a sim­ple thing, and it typ­i­cally fol­lows the old adage of two steps for­ward and one step back,” says Pychyl. “All of this said, I am con­fi­dent that any­one can learn to stop pro­cras­ti­nat­ing.”

One of the first in­ves­ti­ga­tions to in­spire the emo­tional view of pro­cras­ti­na­tion was pub­lished in the early 2000s by re­searchers at Case Western Reserve University in Ohio. They first prompted peo­ple to feel bad (by ask­ing them to read sad sto­ries) and showed that this in­creased their in­cli­na­tion to pro­cras­ti­nate by do­ing puz­zles or play­ing video games in­stead of prepar­ing for the in­tel­li­gence test they knew was com­ing. Subsequent stud­ies by the same team showed low mood only in­creases pro­cras­ti­na­tion if en­joy­able ac­tiv­i­ties are avail­able as a dis­trac­tion, and only if peo­ple be­lieve they can change their moods. One study used ‘mood-freezing can­dles’ to trick some vol­un­teers into think­ing their low mood was frozen and, in this case, they did­n’t bother pro­cras­ti­nat­ing.

The emo­tional reg­u­la­tion the­ory of pro­cras­ti­na­tion makes in­tu­itive sense. In my case, it’s not that I don’t re­alise how long my as­sign­ment will take (I know I need to be work­ing on it right now) or that I haven’t sched­uled enough time for my YouTube view­ing – in fact, I don’t re­ally even want to watch those videos, I’m just drawn to them as a way of avoid­ing the dis­com­fort of knuck­ling down to work. In the psy­chol­o­gists’ jar­gon, I’m pro­cras­ti­nat­ing to achieve a short-term pos­i­tive ‘hedonic shift’, at the cost of my longer-term goals.Pro­cras­ti­na­tion — while ef­fec­tively dis­tract­ing in the short-term — can lead to guilt, which ul­ti­mately com­pounds the ini­tial stress­The emo­tional reg­u­la­tion view of pro­cras­ti­na­tion also helps ex­plain some strange mod­ern phe­nom­ena, like the fad for watch­ing on­line cat videos which have at­tracted bil­lions of views on YouTube. A sur­vey of thou­sands of peo­ple by Jessica Myrick at the Media School at Indiana University con­firmed pro­cras­ti­na­tion as a com­mon mo­tive for view­ing the cat videos and that watch­ing them led to a boost in pos­i­tive mood. It’s not that peo­ple had­n’t ad­e­quately sched­uled time for watch­ing the videos; of­ten they were only watch­ing the clips to make them­selves feel bet­ter when they should be do­ing some­thing else less fun.

Myrick’s re­search also high­lighted an­other emo­tional as­pect to pro­cras­ti­na­tion. Many of those sur­veyed felt guilty af­ter watch­ing the cat videos. This speaks to how pro­cras­ti­na­tion is a mis­guided emo­tional reg­u­la­tion strat­egy. While it might bring short-term re­lief, it only stores up prob­lems for later. In my own case, by de­lay­ing my work I just end up feel­ing even more stressed, not to men­tion the gath­er­ing clouds of guilt and frus­tra­tion.

It’s per­haps lit­tle won­der that re­search by Fuschia Sirois has shown chronic pro­cras­ti­na­tion — that is, be­ing in­clined to pro­cras­ti­nate on a reg­u­lar, long-term ba­sis — is as­so­ci­ated with a host of ad­verse men­tal and phys­i­cal health con­se­quences, in­clud­ing anx­i­ety and de­pres­sion, poor health such as colds and flu, and even more se­ri­ous con­di­tions like car­dio­vas­cu­lar dis­ease.Re­searchers say pro­cras­ti­nat­ing helps us feel bet­ter when cer­tain tasks fill us with neg­a­tive emo­tions — if they are too dif­fi­cult or bor­ing, say (Credit: Getty Images)Sirois be­lieves pro­cras­ti­na­tion has these ad­verse con­se­quences through two routes – first, it’s stress­ful to keep putting off im­por­tant tasks and fail­ing to ful­fil your goals, and sec­ond, the pro­cras­ti­na­tion of­ten in­volves de­lay­ing im­por­tant health be­hav­iours, such as tak­ing up ex­er­cise or vis­it­ing the doc­tor. “Over time high stress and poor health be­hav­iours are well known to have a syn­er­gis­tic and cu­mu­la­tive ef­fect on health that can in­crease risk for a num­ber of se­ri­ous and chronic health con­di­tions such as heart dis­ease, di­a­betes, arthri­tis, and even can­cer,” she says.

All of this means that over­com­ing pro­cras­ti­na­tion could have a ma­jor pos­i­tive im­pact on your life. Sirois says her re­search sug­gests that “decreasing a ten­dency to chron­i­cally pro­cras­ti­nate by one point [on a five-point pro­cras­ti­na­tion scale] would also po­ten­tially mean that your risk for hav­ing poor heart health would re­duce by 63%”.

On a pos­i­tive note, if pro­cras­ti­na­tion is an emo­tional reg­u­la­tion is­sue, this of­fers im­por­tant clues for how to ad­dress it most ef­fec­tively. An ap­proach based on Acceptance and Commitment Therapy or ‘ACT’, an off-shoot of Cognitive Behavioural Therapy, seems es­pe­cially apt. ACT teaches the ben­e­fits of ‘psychological flex­i­bil­i­ty’ – that is, be­ing able to tol­er­ate un­com­fort­able thoughts and feel­ings, stay­ing in the pre­sent mo­ment in spite of them, and pri­ori­tis­ing choices and ac­tions that help you get closer to what you most value in life.

Relevant here is cut­ting edge re­search that’s shown stu­dents who pro­cras­ti­nate more tend to score higher on psy­cho­log­i­cal in­flex­i­bil­ity. That is, they’re dom­i­nated by their psy­cho­log­i­cal re­ac­tions, like frus­tra­tion and worry, at the ex­pense of their life val­ues; high scor­ers agree with state­ments like ‘I’m afraid of my feel­ings’ and ‘My painful ex­pe­ri­ences and mem­o­ries make it dif­fi­cult for me to live a life that I would val­ue’. Those who pro­cras­ti­nate more also score lower on ‘committed ac­tion’, which de­scribes how much a per­son per­sists with ac­tions and be­hav­iours in pur­suit of their goals. Low scor­ers tend to agree with state­ments like ‘If I feel dis­tressed or dis­cour­aged, I let my com­mit­ments slide’.Re­search shows that once the first step is made to­wards a task, fol­low­ing through be­comes eas­ier­ACT trains peo­ple both to in­crease their psy­cho­log­i­cal flex­i­bil­ity (for ex­am­ple, through mind­ful­ness) and their com­mit­ted ac­tion (for ex­am­ple, by find­ing cre­ative ways to pur­sue goals that serve their val­ues — what mat­ters most to them in life), and pre­lim­i­nary re­search  in­volv­ing stu­dents has been promis­ing, with ACT prov­ing more ef­fec­tive than CBT in one trial over the longer-term.

Of course, most of us prob­a­bly won’t have the op­tion of sign­ing up to an ACT course any time soon — and in any case we’re bound to keep putting off look­ing for one — so how can we go about ap­ply­ing these prin­ci­ples to­day? “When some­one fi­nally recog­nises that pro­cras­ti­na­tion is­n’t a time man­age­ment prob­lem but is in­stead an emo­tion reg­u­la­tion prob­lem, then they are ready to em­brace my favourite tip,” says Pychyl.

The next time you’re tempted to pro­cras­ti­nate, “make your fo­cus as sim­ple as ‘What’s the next ac­tion – a sim­ple next step – I would take on this task if I were to get started on it now?’”. Doing this, he says, takes your mind off your feel­ings and onto eas­ily achiev­able ac­tion. “Our re­search and lived ex­pe­ri­ence show very clearly that once we get started, we’re typ­i­cally able to keep go­ing. Getting started is every­thing.”

Dr Christian Jarrettis a se­nior ed­i­tor at Aeon mag­a­zine. His next book, about per­son­al­ity change, will be pub­lished in 2021.

After ~12.5 years at Google and ~10 years work­ing on Go (#golang), it’s time for me to do some­thing new. Tomorrow is my last day at Google.

Working at Google and on Go has been a high­light of my ca­reer. Go re­ally made pro­gram­ming fun for me again, and I’ve had fun help­ing make it. I want to thank Rob Pike for let­ting me work on Go full time (instead of just as a dis­trac­tion on painfully long gBus rides) as well as Russ Cox and Ian Lance Taylor and Robert Griesemer and oth­ers for all the pa­tience while I learned my way around. I’ve loved hack­ing on var­i­ous pack­ages and sys­tems with the team and mem­bers of the com­mu­nity, giv­ing a bunch of talks, hang­ing out in Denver, Sydney, MTV, NYC, at FOSDEM and other meet-ups, etc. While I’ve learned a bunch while work­ing on Go, more ex­cit­ingly I dis­cov­ered many things that I did­n’t know I did­n’t know, and it was a joy watch­ing the whole team and com­mu­nity work their (to me) magic.

I’ll still be around the Go com­mu­nity, but less, and dif­fer­ently. My @golang.org email will con­tinue to work and please con­tinue to mail me or copy me on GitHub (@bradfitz), es­pe­cially for some­thing bro­ken that might be my fault.

In some­what chrono­log­i­cal or­der, but not en­tirely:

Worked on the Social Graph API, con­tin­u­ally ex­tract­ing and FOAF and other such

se­man­tic so­cial links be­tween pages, in­dex­ing it all, and ex­port­ing it over a pub­lic API. It’s since been shut down,

but I learned a lot about Google in­dex­ing, pro­duc­tion (Borg, BigTable, MapReduce), got up to speed on Google-used

lan­guages & style (C++, Java, Python, Sawzall).

Web-ified the Google open source CLA process; it pre­vi­ously in­volved fax­ing and stuff, which no­body wanted to do.

Worked on Gmail’s back­end for a bit, specif­i­cally its ad­dress book back­end.

Integrated per­sonal ad­dress book search into Google’s main search. (So search­ing google.com for some­body in your con­tacts

would show your ad­dress book en­try for them.) It dark launched as an ex­per­i­ment, but never went live be­cause at the time

we did­n’t have enough SSDs in enough data cen­ters to meet la­tency bud­gets. (I re­mem­ber meet­ings with var­i­ous teams

draw­ing Gannt charts a few mil­lisec­onds wide, show­ing RPCs and la­tency bud­gets… blew my mind at the time.)

Rewrote mem­cached (which I wrote pre-Google) in Google C++

and us­ing Google’s RPC sys­tem and then added mem­cache sup­port

to App Engine. The Google mem­cache server con­tin­ues to be

used by many teams.

Fixed a bunch of per­for­mance bugs in Android when it first came out, got in­vited to join the team.

Worked on the Android frame­work team do­ing per­for­mance analy­sis and tool­ing and fixes.

(some of the com­mits)

Did a bunch of logs pro­cess­ing

on nightly up­loaded per­for­mance sam­ples from Android team & Googler dog­food phones and iden­ti­fied prob­lem­atic code & stack traces & RPCs

caus­ing UI jank.

Worked on a dis­trib­uted build sys­tem for Android that let you do make -j9999 of ar­bi­trary code out over

a clus­ter of ma­chine build­ing in a cus­tom FUSE filesys­tem that watched what got writ­ten and com­mu­ni­cated back to the co­or­di­na­tor

server. It was in­ter­nal code, but its spirit lives on in han­wen/​ter­mite, a Go im­ple­men­ta­tion writ­ten

later.

Started writ­ing in Go for var­i­ous Android analy­sis tasks.

Started writ­ing in Go for per­sonal pro­jects (Perkeep, then named Camlistore)

Started send­ing changes to Go to fix/​add to the Go stan­dard li­brary. (starting May 5, 2010 with some os. Chtimes

and

http stuff)

Invited to join the Go team.

Worked on most of the Go stan­dard li­brary. Primary au­thor of net/​http, data­base/​sql, os/​exec, Go’s build/​test CI sys­tem, etc.

20% pro­ject: co-au­thor of PubSubHubBub, which be­came WebSub.

Rewrote the Google down­load server from C++ to Go. This was one

of (or the?) first pro­duc­tion Go ser­vice at Google, and in­volved plenty of de­pen­dency work to make it hap­pen. It also showed

var­i­ous short­com­ings in the Go run­time which are long since fixed.

Gave many Go talks at var­i­ous con­fer­ences.

Wrote Go’s HTTP/2 im­ple­men­ta­tion, client and server, in Go 1.6.

8 man­agers, IIRC. (not at once, for­tu­nately; “Eight bosses?” “Eight.” “Eight, Bob”)

15 Go re­leases (and sev­eral be­fore Go 1)

See go/​brad­fitz for the in­ter­nal ver­sion of this doc­u­ment. (It’s ap­prox­i­mately the same but with a bit more stuff I can’t or don’t want to share pub­licly.)

Why are you leav­ing?

Little bored. Not learn­ing as much as I used to. I’ve been do­ing the same thing too long and need a change. It’d be nice to pri­mar­ily work in Go rather than work on Go.

When I first joined Google it was a chaotic first cou­ple years while I learned Google’s in­ter­nal code­base, build sys­tem, a bunch of new lan­guages, Borg, Bigtable, etc. Then I joined Android it was fun/​learn­ing chaos again. Go was the same when I joined and it was a new, fast-mov­ing ex­per­i­ment. Now Go is very pop­u­lar, sta­ble and, while there’s a lot to do, things–of­ten nec­es­sar­ily–move pretty slowly. Moving slowly is fine, and hy­per-spe­cial­iz­ing in small cor­ners of Go makes sense at scale (few per­cent im­prove­ments add up!), but I want to build some­thing new again.

I don’t want to get stuck in a com­fort­able rut. (And Google cer­tainly is com­fort­able, ex­cept for open floor plans.)

TBA. But build­ing some­thing new.

Skip to con­tent

earth or some­where close to it

You can add lo­ca­tion in­for­ma­tion to your Tweets, such as your city or pre­cise lo­ca­tion, from the web and via third-party ap­pli­ca­tions. You al­ways have the op­tion to delete your Tweet lo­ca­tion his­tory. Learn more

Public · Anyone can fol­low this list

Private · Only you can ac­cess this list

Here’s the URL for this Tweet. Copy it to eas­ily share with friends.

Add this Tweet to your web­site by copy­ing the code be­low. Learn more

Add this video to your web­site by copy­ing the code be­low. Learn more

Hmm, there was a prob­lem reach­ing the server.

By em­bed­ding Twitter con­tent in your web­site or app, you are agree­ing to the Twitter Developer Agreement and Developer Policy.

Why you’re see­ing this ad

Not on Twitter? Sign up, tune into the things you care about, and get up­dates as they hap­pen.

» See SMS short codes for other coun­tries

This time­line is where you’ll spend most of your time, get­ting in­stant up­dates about what mat­ters to you.

Hover over the pro­file pic and click the Following but­ton to un­fol­low any ac­count.

When you see a Tweet you love, tap the heart — it lets the per­son who wrote it know you shared the love.

Add your thoughts about any Tweet with a Reply. Find a topic you’re pas­sion­ate about, and jump right in.

Get in­stant in­sight into what peo­ple are talk­ing about now.

Get more of what you love

Follow more ac­counts to get in­stant up­dates about top­ics you care about.

See the lat­est con­ver­sa­tions about any topic in­stantly.

Catch up in­stantly on the best sto­ries hap­pen­ing as they un­fold.

This is why i use ad block­ers and a pi-hole server…….. re­quest map of ny­times.pic.twit­ter.com/​yf9Sv4­daQH

Thanks. Twitter will use this to make your time­line bet­ter.

Loading seems to be tak­ing a while.

Twitter may be over ca­pac­ity or ex­pe­ri­enc­ing a mo­men­tary hic­cup. Try again or visit Twitter Status for more in­for­ma­tion.

An an­tivirus pro­gram used by hun­dreds of mil­lions of peo­ple around the world is sell­ing highly sen­si­tive web brows­ing data to many of the world’s biggest com­pa­nies, a joint in­ves­ti­ga­tion by Motherboard and PCMag has found. Our re­port re­lies on leaked user data, con­tracts, and other com­pany doc­u­ments that show the sale of this data is both highly sen­si­tive and is in many cases sup­posed to re­main con­fi­den­tial be­tween the com­pany sell­ing the data and the clients pur­chas­ing it.

The doc­u­ments, from a sub­sidiary of the an­tivirus gi­ant Avast called Jumpshot, shine new light on the se­cre­tive sale and sup­ply chain of peo­ples’ in­ter­net brows­ing his­to­ries. They show that the Avast an­tivirus pro­gram in­stalled on a per­son’s com­puter col­lects data, and that Jumpshot repack­ages it into var­i­ous dif­fer­ent prod­ucts that are then sold to many of the largest com­pa­nies in the world. Some past, pre­sent, and po­ten­tial clients in­clude Google, Yelp, Microsoft, McKinsey, Pepsi, Home Depot, Condé Nast, Intuit, and many oth­ers. Some clients paid mil­lions of dol­lars for prod­ucts that in­clude a so-called “All Clicks Feed,” which can track user be­hav­ior, clicks, and move­ment across web­sites in highly pre­cise de­tail.

Avast claims to have more than 435 mil­lion ac­tive users per month, and Jumpshot says it has data from 100 mil­lion de­vices. Avast col­lects data from users that opt-in and then pro­vides that to Jumpshot, but mul­ti­ple Avast users told Motherboard they were not aware Avast sold brows­ing data, rais­ing ques­tions about how in­formed that con­sent is.

The data ob­tained by Motherboard and PCMag in­cludes Google searches, lookups of lo­ca­tions and GPS co­or­di­nates on Google Maps, peo­ple vis­it­ing com­pa­nies’ LinkedIn pages, par­tic­u­lar YouTube videos, and peo­ple vis­it­ing porn web­sites. It is pos­si­ble to de­ter­mine from the col­lected data what date and time the anonymized user vis­ited YouPorn and PornHub, and in some cases what search term they en­tered into the porn site and which spe­cific video they watched.

Do you know about any other com­pa­nies sell­ing data? We’d love to hear from you. Using a non-work phone or com­puter, you can con­tact Joseph Cox se­curely on Signal on +44 20 8133 5190, Wickr on joseph­cox, OTR chat on jf­cox@jab­ber.ccc.de, or email joseph.cox@vice.com.

Although the data does not in­clude per­sonal in­for­ma­tion such as users’ names, it still con­tains a wealth of spe­cific brows­ing data, and ex­perts say it could be pos­si­ble to deanonymize cer­tain users.

In a press re­lease from July, Jumpshot claims to be “the only com­pany that un­locks walled gar­den data” and seeks to “provide mar­keters with deeper vis­i­bil­ity into the en­tire on­line cus­tomer jour­ney.” Jumpshot has pre­vi­ously dis­cussed some of its clients pub­licly. But other com­pa­nies men­tioned in Jumpshot doc­u­ments in­clude Expedia, IBM, Intuit, which makes TurboTax, Loreal, and Home Depot. Employees are in­structed not to talk pub­licly about Jumpshot’s re­la­tion­ships with these com­pa­nies.

“It’s very gran­u­lar, and it’s great data for these com­pa­nies, be­cause it’s down to the de­vice level with a time­stamp,” the source said, re­fer­ring to the speci­ficity and sen­si­tiv­ity of the data be­ing sold. Motherboard granted the source anonymity to speak more can­didly about Jumpshot’s processes.

Until re­cently, Avast was col­lect­ing the brows­ing data of its cus­tomers who had in­stalled the com­pa­ny’s browser plu­gin, which is de­signed to warn users of sus­pi­cious web­sites. Security re­searcher and AdBlock Plus cre­ator Wladimir Palant pub­lished a blog post in October show­ing that Avast har­vest user data with that plu­gin. Shortly af­ter, browser mak­ers Mozilla, Opera, and Google re­moved Avast’s and sub­sidiary AVG’s ex­ten­sions from their re­spec­tive browser ex­ten­sion stores. Avast had pre­vi­ously ex­plained this data col­lec­tion and shar­ing in a blog and fo­rum post in 2015. Avast has since stopped send­ing brows­ing data col­lected by these ex­ten­sions to Jumpshot, Avast said in a state­ment to Motherboard and PCMag.

However, the data col­lec­tion is on­go­ing, the source and doc­u­ments in­di­cate. Instead of har­vest­ing in­for­ma­tion through soft­ware at­tached to the browser, Avast is do­ing it through the anti-virus soft­ware it­self. Last week, months af­ter it was spot­ted us­ing its browser ex­ten­sions to send data to Jumpshot, Avast be­gan ask­ing its ex­ist­ing free an­tivirus con­sumers to opt-in to data col­lec­tion, ac­cord­ing to an in­ter­nal doc­u­ment.

“If they opt-in, that de­vice be­comes part of the Jumpshot Panel and all browser-based in­ter­net ac­tiv­ity will be re­ported to Jumpshot,” an in­ter­nal prod­uct hand­book reads. “What URLs did these de­vices visit, in what or­der and when?” it adds, sum­maris­ing what ques­tions the prod­uct may be able to an­swer.

Senator Ron Wyden, who in December asked Avast why it was sell­ing users’ brows­ing data, said in a state­ment, “It is en­cour­ag­ing that Avast has ended some of its most trou­bling prac­tices af­ter en­gag­ing con­struc­tively with my of­fice. However I’m con­cerned that Avast has not yet com­mit­ted to delet­ing user data that was col­lected and shared with­out the opt-in con­sent of its users, or to end the sale of sen­si­tive in­ter­net brows­ing data. The only re­spon­si­ble course of ac­tion is to be fully trans­par­ent with cus­tomers go­ing for­ward, and to purge data that was col­lected un­der sus­pect con­di­tions in the past.”

Despite Avast cur­rently ask­ing users to opt back into the data col­lec­tion via a pop-up in the an­tivirus soft­ware, mul­ti­ple Avast users said they did not know that Avast was sell­ing brows­ing data.

“I was not aware of this,” Keith, a user of the free Avast an­tivirus prod­uct who only pro­vided their first name, told Motherboard. “That sounds scary. I usu­ally say no to data track­ing,” they said, adding that they haven’t yet seen the new opt-in pop-up from Avast.

“Did not know that they did that :(,” an­other free Avast an­tivirus user said in a Twitter di­rect mes­sage.

Motherboard and PCMag con­tacted over two dozen com­pa­nies men­tioned in in­ter­nal doc­u­ments. Only a hand­ful re­sponded to ques­tions ask­ing what they do with data based on the brows­ing his­tory of Avast users.

“We some­times use in­for­ma­tion from third-party providers to help im­prove our busi­ness, prod­ucts and ser­vices. We re­quire these providers to have the ap­pro­pri­ate rights to share this in­for­ma­tion with us. In this case, we re­ceive anonymized au­di­ence data, which can­not be used to iden­tify in­di­vid­ual cus­tomers,” a Home Depot spokesper­son wrote in an emailed state­ment.

Microsoft de­clined to com­ment on the specifics of why it pur­chased prod­ucts from Jumpshot, but said that it does­n’t have a cur­rent re­la­tion­ship with the com­pany. A Yelp spokesper­son wrote in an email, “In 2018, as part of a re­quest for in­for­ma­tion by an­titrust au­thor­i­ties, Yelp’s pol­icy team was asked to es­ti­mate the im­pact of Google’s an­ti­com­pet­i­tive be­hav­ior on the lo­cal search mar­ket­place. Jumpshot was en­gaged on a one-time ba­sis to gen­er­ate a re­port of anonymized, high-level trend data which val­i­dated other es­ti­mates of Google’s si­phon­ing of traf­fic from the web. No PII was re­quested or ac­cessed.”

“Every search. Every click. Every buy. On every site.”

Southwest Airlines said it had dis­cus­sions with Jumpshot but did­n’t reach an agree­ment with the com­pany. IBM said it did not have a record of be­ing a client, and Altria said it is not work­ing with Jumpshot, al­though did­n’t spec­ify if it did so pre­vi­ously. Sephora said it has not worked with Jumpshot. Google did not re­spond to a re­quest for com­ment.

On its web­site and in press re­leases, Jumpshot names Pepsi, and con­sult­ing gi­ants Bain & Company and McKinsey as clients.

As well as Expedia, Intuit, and Loreal, other com­pa­nies which are not al­ready men­tioned in pub­lic Jumpshot an­nounce­ments in­clude cof­fee com­pany Keurig, YouTube pro­mo­tion ser­vice vidIQ, and con­sumer in­sights firm Hitwise. None of those com­pa­nies re­sponded to a re­quest for com­ment.

On its web­site, Jumpshot lists some pre­vi­ous case stud­ies for us­ing its brows­ing data. Magazine and dig­i­tal me­dia gi­ant Condé Nast, for ex­am­ple, used Jumpshot’s prod­ucts to see whether the me­dia com­pa­ny’s ad­ver­tise­ments re­sulted in more pur­chases on Amazon and else­where. Condé Nast did not re­spond to a re­quest for com­ment.

Jumpshot sells a va­ri­ety of dif­fer­ent prod­ucts based on data col­lected by Avast’s an­tivirus soft­ware in­stalled on users’ com­put­ers. Clients in the in­sti­tu­tional fi­nance sec­tor of­ten buy a feed of the top 10,000 do­mains that Avast users are vis­it­ing to try and spot trends, the prod­uct hand­book reads.

Another Jumpshot prod­uct is the com­pa­ny’s so-called “All Click Feed.” It al­lows a client to buy in­for­ma­tion on all of the clicks Jumpshot has seen on a par­tic­u­lar do­main, like Amazon.com, Walmart.com, Target.com, BestBuy.com, or Ebay.com.

In a tweet sent last month in­tended to en­tice new clients, Jumpshot noted that it col­lects “Every search. Every click. Every buy. On every site” [emphasis Jumpshot’s.]

Jumpshot’s data could show how some­one with Avast an­tivirus in­stalled on their com­puter searched for a prod­uct on Google, clicked on a link that went to Amazon, and then maybe added an item to their cart on a dif­fer­ent web­site, be­fore fi­nally buy­ing a prod­uct, the source who pro­vided the doc­u­ments ex­plained.

One com­pany that pur­chased the All Clicks Feed is New York-based mar­ket­ing firm Omnicom Media Group, ac­cord­ing to a copy of its con­tract with Jumpshot. Omnicom paid Jumpshot $2,075,000 for ac­cess to data in 2019, the con­tract shows. It also in­cluded an­other prod­uct called “Insight Feed” for 20 dif­fer­ent do­mains. The fee for data in 2020 and then 2021 is listed as $2,225,000 and $2,275,000 re­spec­tively, the doc­u­ment adds.

Jumpshot gave Omnicom ac­cess to all click feeds from 14 dif­fer­ent coun­tries around the world, in­clud­ing the U. S., England, Canada, Australia, and New Zealand. The prod­uct also in­cludes the in­ferred gen­der of users “based on brows­ing be­hav­ior,” their in­ferred age, and “the en­tire URL string” but with per­son­ally iden­ti­fi­able in­for­ma­tion (PII) re­moved, the con­tract adds.

Omnicom did not re­spond to mul­ti­ple re­quests for com­ment.

According to the Omnicom con­tract, the “device ID” of each user is hashed, mean­ing the com­pany buy­ing the data should not be able to iden­tify who ex­actly is be­hind each piece of brows­ing ac­tiv­ity. Instead, Jumpshot’s prod­ucts are sup­posed to give in­sights to com­pa­nies who may want to see what prod­ucts are par­tic­u­larly pop­u­lar, or how ef­fec­tive an ad cam­paign is work­ing.

“What we don’t do is re­port on the Jumpshot Device ID that ex­e­cuted the clicks to pro­tect against the tri­an­gu­la­tion of PII,” one in­ter­nal Jumpshot doc­u­ment reads.

But Jumpshot’s data may not be to­tally anony­mous. The in­ter­nal prod­uct hand­book says that de­vice IDs do not change for each user, “unless a user com­pletely unin­stalls and re­in­stalls the se­cu­rity soft­ware.” Numerous ar­ti­cles and aca­d­e­mic stud­ies have shown how it is pos­si­ble to un­mask peo­ple us­ing so-called anonymized data. In 2006, New York Times re­porters were able to iden­tify a spe­cific per­son from a cache of sup­pos­edly anony­mous search data that AOL pub­licly re­leased. Although the tested data was more fo­cused on so­cial me­dia links, which Jumpshot redacts some­what, a 2017 study from Stanford University found it was pos­si­ble to iden­tify peo­ple from anony­mous web brows­ing data.

“De-identification has shown to be a very fail­ure-prone process. There are so many ways it can go wrong,” Günes Acar, who stud­ies large-scale in­ter­net track­ing at the Computer Security and Industrial Cryptography re­search group at the Department of Electrical Engineering of the Katholieke Universiteit Leuven, said.

De-anonymization be­comes a greater con­cern when con­sid­er­ing how the even­tual end-users of Jumpshot’s data could com­bine it with their own data.

“Most of the threats posed by de-anonymiza­tion—where you are iden­ti­fy­ing peo­ple—comes from the abil­ity to merge the in­for­ma­tion with other data,” Acar said. A set of Jumpshot data ob­tained by Motherboard and PCMag shows how each vis­ited URL comes with a pre­cise time­stamp down to the mil­lisec­ond, which could al­low a com­pany with its own bank of cus­tomer data to see one user vis­it­ing their own site, and then fol­low them across other sites in the Jumpshot data.

“It’s al­most im­pos­si­ble to de-iden­tify data,” Eric Goldman, a pro­fes­sor at the Santa Clara University School of Law, said. “When they promise to de-iden­tify the data, I don’t be­lieve it.”

Motherboard and PCMag asked Avast a se­ries of de­tailed ques­tions about how it pro­tects user anonymity as well as de­tails on some of the com­pa­ny’s con­tracts. Avast did not an­swer most of the ques­tions but wrote in a state­ment, “Because of our ap­proach, we en­sure that Jumpshot does not ac­quire per­sonal iden­ti­fi­ca­tion in­for­ma­tion, in­clud­ing name, email ad­dress or con­tact de­tails, from peo­ple us­ing our pop­u­lar free an­tivirus soft­ware.”

“Users have al­ways had the abil­ity to opt out of shar­ing data with Jumpshot. As of July 2019, we had al­ready be­gun im­ple­ment­ing an ex­plicit opt-in choice for all new down­loads of our AV, and we are now also prompt­ing our ex­ist­ing free users to make an ex­plicit choice, a process which will be com­pleted in February 2020,” it said, adding that the com­pany com­plies with the California Consumer Privacy Act (CCPA) and Europe’s General Data Protection Regulation (GDPR) across its en­tire global user base.

“We have a long track record of pro­tect­ing users’ de­vices and data against mal­ware, and we un­der­stand and take se­ri­ously the re­spon­si­bil­ity to bal­ance user pri­vacy with the nec­es­sary use of data,” the state­ment added.

“It’s al­most im­pos­si­ble to de-iden­tify data.”

When PCMag in­stalled Avast’s an­tivirus prod­uct for the first time this month, the soft­ware did ask if they wanted to opt-in to data col­lec­tion.

“If you al­low it, we’ll pro­vide our sub­sidiary Jumpshot Inc. with a stripped and de-iden­ti­fied data set de­rived from your brows­ing his­tory for the pur­pose of en­abling Jumpshot to an­a­lyze mar­kets and busi­ness trends and gather other valu­able in­sights,” the opt-in mes­sage read. The pop-up did not go into de­tail on how Jumpshot then uses this brows­ing data, how­ever.

“The data is fully de-iden­ti­fied and ag­gre­gated and can­not be used to per­son­ally iden­tify or tar­get you. Jumpshot may share ag­gre­gated in­sights with its cus­tomers,” the pop-up added.

Just a few days ago, the Twitter ac­count for Avast sub­sidiary AVG tweeted, “Do you re­mem­ber the last time you cleaned your #browser his­tory? Storing your brows­ing his­tory for a long time can take up mem­ory on your de­vice and can put your pri­vate info at risk.”

Update: This piece has been up­dated to in­clude a re­sponse from Sephora.

What would you like to do?

Embed this gist in your web­site.

Clone with Git or check­out with SVN us­ing the repos­i­to­ry’s web ad­dress.

Here are some terms to mute on Twitter to clean your time­line up a bit.

Sign up for free

to join this con­ver­sa­tion on GitHub.

Already have an ac­count?

Sign in to com­ment

You can’t per­form that ac­tion at this time.

You signed in with an­other tab or win­dow. Reload to re­fresh your ses­sion.

You signed out in an­other tab or win­dow. Reload to re­fresh your ses­sion.

Monty Python stars have led the trib­utes to their co-star Terry Jones, who has died at the age of 77.

The Welsh ac­tor and writer played a va­ri­ety of char­ac­ters in the iconic com­edy group’s Flying Circus TV se­ries, and di­rected sev­eral of their films.

He died on Tuesday, four years af­ter con­tract­ing a rare form of de­men­tia known as FTD.

David Walliams and Simon Pegg were among other co­me­di­ans who re­mem­bered him.

Fellow Python star Sir Michael Palin de­scribed Jones as “one of the fun­ni­est writer-per­form­ers of his gen­er­a­tion”.

In a tweet, John Cleese said he was “a man of so many tal­ents and such end­less en­thu­si­asm”.

Eric Idle, an­other mem­ber of the highly in­flu­en­tial com­edy troupe, re­called the “many laughs [and] mo­ments of to­tal hi­lar­ity” they shared.

“It’s too sad if you knew him, but if you did­n’t you will al­ways smile at the many won­der­fully funny mo­ments he gave us,” he went on.

Terry Gilliam, with whom Jones di­rected the group’s film The Holy Grail in 1975, de­scribed his fel­low Python as a “brilliant, con­stantly ques­tion­ing, icon­o­clas­tic, right­eously ar­gu­men­ta­tive and an­gry but out­ra­geously funny and gen­er­ous and kind hu­man be­ing”.

“One could never hope for a bet­ter friend,” he said.

Palin added: “Terry was one of my clos­est, most val­ued friends. He was kind, gen­er­ous, sup­port­ive and pas­sion­ate about liv­ing life to the full.

“He was far more than one of the fun­ni­est writer-per­form­ers of his gen­er­a­tion, he was the com­plete Renaissance co­me­dian - writer, di­rec­tor, pre­sen­ter, his­to­rian, bril­liant chil­dren’s au­thor, and the warmest, most won­der­ful com­pany you could wish to have.”

Screenwriter Charlie Brooker posted: “RIP the ac­tual ge­nius Terry Jones. Far too many bril­liant mo­ments to choose from.”

David Walliams thanked his com­edy hero “for a life­time of laugh­ter”.

Simon Pegg - who acted in Jones’ fi­nal film as di­rec­tor, 2015′s Absolutely Anything - said: “Terry was a sweet, gen­tle, funny man who was a joy to work with and im­pos­si­ble not to love.”

And co­me­dian Eddie Izzard told BBC News: “It’s a tragedy - the good go too early. Monty Python changed the face of world com­edy. It will live for­ever. It’s a ter­ri­ble loss.”

Shane Allen, BBC con­troller of com­edy com­mis­sion­ing, wrote that it was a “sad day to lose an ab­solute ti­tan of British com­edy” and “one of the found­ing fa­thers of the most in­flu­en­tial and pi­o­neer­ing com­edy en­sem­bles of all time”.

Jones was born in Colwyn Bay and went on to study at Oxford University, where he met his fu­ture Python pal Palin in the Oxford Revue - a stu­dent com­edy group.

Alongside Palin, Idle and the likes of David Jason, he ap­peared in the BBC chil­dren’s satir­i­cal sketch show Do Not Adjust Your Set, which would set the tem­plate for their work to come with Python.

He wrote and starred in Monty Python’s Flying Circus TV show and the com­edy col­lec­tive’s films, as a range of much-loved char­ac­ters. These in­cluded Arthur “Two Sheds” Jackson, Cardinal Biggles of the Spanish Inquisition and Mr Creosote.

In ad­di­tion to di­rect­ing The Holy Grail with Gilliam, Jones took sole di­rec­to­r­ial charge of 1979′s Life of Brian and The Meaning of Life in 1983.

Cleese said: “Of his many achieve­ments, for me the great­est gift he gave us all was his di­rec­tion of Life of Brian. Perfection.”

Beyond Monty Python, he wrote the screen­play for the 1986 film Labyrinth, star­ring David Bowie.

Monty Python’s Flying Circus, the ground­break­ing com­edy se­ries that made Jones and his fel­low cast mem­bers in­ter­na­tional stars, first aired on BBC One in October 1969.

Surreal, an­ar­chic and bawdily ir­rev­er­ent, the show’s blend of live-ac­tion sketches and an­i­mated in­ter­ludes mocked both broad­cast­ing con­ven­tions and so­ci­etal norms.

Jones and Palin had met at Oxford, while Cleese, Graham Chapman and Eric Idle stud­ied at Cambridge. After uni­ver­sity, they took part in var­i­ous com­edy shows be­fore form­ing Monty Python with US-born an­i­ma­tor Terry Gilliam.

After four se­ries, the troupe moved to the big screen to make Arthurian spoof Monty Python and the Holy Grail and Monty Python’s Life of Brian, a con­tro­ver­sial par­ody of Biblical epics.

Monty Python’s The Meaning of Life, their fi­nal film as a col­lec­tive, re­turned to the orig­i­nal se­ries’ sketch-based for­mat.

The sur­viv­ing mem­bers re­united pe­ri­od­i­cally af­ter Chapman’s death in 1989, most no­tably for a run of live shows at the O2 in London in 2014.

* “Now, you lis­ten here! He’s not the Messiah. He’s a very naughty boy!” - as Brian’s mother in Monty Python’s Life of Brian

* “I’m alive, I’m alive!” - as the naked her­mit who gives away the lo­ca­tion of a hid­ing Brian in Life of Brian

* “I shall use my largest scales” - as Sir Belvedere, who over­sees a witch trial in Monty Python and the Holy Grail

* “What, the cur­tains?” - as Prince Herbert, who is told “One day, lad, all this will be yours” in Holy Grail

* “Spam, spam, spam, spam, spam, spam, spam” - as the greasy spoon wait­ress in a Monty Python sketch

The state­ment from Jones’ fam­ily noted his “uncompromising in­di­vid­u­al­ity, re­lent­less in­tel­lect and ex­tra­or­di­nary hu­mour [that] has given plea­sure to count­less mil­lions across six decades”.

“Over the past few days his wife, chil­dren, ex­tended fam­ily and many close friends have been con­stantly with Terry as he gen­tly slipped away at his home in north London.

“His work with Monty Python, his books, films, tele­vi­sion pro­grammes, po­ems and other work will live on for­ever, a fit­ting legacy to a true poly­math.”

The fam­ily thanked Jones’ “wonderful med­ical pro­fes­sion­als and car­ers for mak­ing the past few years not only bear­able but of­ten joy­ful”.

They said: “We hope that this dis­ease will one day be erad­i­cated en­tirely. We ask that our pri­vacy be re­spected at this sen­si­tive time and give thanks that we lived in the pres­ence of an ex­tra­or­di­nar­ily tal­ented, play­ful and happy man liv­ing a truly au­then­tic life, in his words ‘Lovingly frosted with glu­cose.’”

* FTD is an un­com­mon type of de­men­tia that mainly af­fects the front and sides of the brain

* The ini­tial symp­toms are, in most cases, changes in be­hav­iour, men­tal abil­ity and lan­guage, which will get worse over time

* Language prob­lems can in­clude loss of vo­cab­u­lary, rep­e­ti­tion and for­get­ting the mean­ing of com­mon words

* FTD is caused by ab­nor­mal pro­tein clumps form­ing in­side the brain cells, which are thought to dam­age the cells and stop them work­ing. Why this hap­pens is not fully un­der­stood

* Although there are cur­rently no cures or treat­ments to slow FTD down, there are treat­ments that can help con­trol some symp­toms

Follow us on Facebook, or on Twitter @BBCNewsEnts. If you have a story sug­ges­tion email en­ter­tain­ment.news@bbc.co.uk.

Today, on Wikipedia’s 19th birth­day, the Wikimedia Foundation has re­ceived re­ports that ac­cess to Wikipedia in Turkey is ac­tively be­ing re­stored.* This lat­est de­vel­op­ment fol­lows a 26 December 2019 rul­ing by the Constitutional Court of Turkey that the more than two and a half year block im­posed by the Turkish gov­ern­ment was un­con­sti­tu­tional. Earlier to­day, the Turkish Constitutional Court made the full text of that rul­ing avail­able to the pub­lic, and shortly af­ter, we re­ceived re­ports that ac­cess was re­stored to Wikipedia.

We are thrilled that the peo­ple of Turkey will once again be able to par­tic­i­pate in the largest global con­ver­sa­tion about the cul­ture and his­tory of Turkey on­line and con­tinue to make Wikipedia a vi­brant source of in­for­ma­tion about Turkey and the world.

“We are thrilled to be re­united with the peo­ple of Turkey,” said Katherine Maher, Executive Director of the Wikimedia Foundation. “At Wikimedia we are com­mit­ted to pro­tect­ing every­one’s fun­da­men­tal right to ac­cess in­for­ma­tion. We are ex­cited to share this im­por­tant mo­ment with our Turkish con­trib­u­tor com­mu­nity on be­half of knowl­edge-seek­ers every­where.”

We are ac­tively re­view­ing the full text of the rul­ing by the Constitutional Court of Turkey. In the mean­time, our case be­fore the European Court of Human Rights is still be­ing con­sid­ered by the Court. We filed a pe­ti­tion in the European Court of Human Rights in spring of last year, and in July, the Court granted our case pri­or­ity sta­tus. We will con­tinue to ad­vo­cate for strong pro­tec­tions for free ex­pres­sion on­line in Turkey and around the world.

Wikipedia is a global free knowl­edge re­source writ­ten and edited by peo­ple around the world. Because of this open edit­ing model, Wikipedia is also a re­source every­one can be a part of ac­tively shap­ing  — adding knowl­edge about their cul­ture, coun­try, in­ter­ests, stud­ies, and more through Wikipedia’s ar­ti­cles. Volunteers work to­gether to write ar­ti­cles about many dif­fer­ent top­ics rang­ing from his­tory, pop cul­ture, sci­ence, sports, and more us­ing re­li­able sources to ver­ify the facts. It is through this col­lec­tive process of writ­ing, dis­cus­sion, and de­bate that Wikipedia be­comes more neu­tral, more com­pre­hen­sive, and more rep­re­sen­ta­tive of the world’s knowl­edge.

More than 85 per­cent of the ar­ti­cles on Wikipedia are in lan­guages other than English, which in­cludes the Turkish Wikipedia’s more than 335,000 ar­ti­cles, writ­ten by Turkish-speaking vol­un­teers for Turkish-speaking peo­ple.

In the time that the block was in ef­fect, we heard from stu­dents, teach­ers, pro­fes­sion­als and more in Turkey about how the block had im­pacted their daily lives. For many stu­dents, the block had oc­curred just days be­fore their fi­nal ex­ams. On so­cial me­dia, mem­bers of the in­ter­na­tional vol­un­teer Wikipedia ed­i­tor com­mu­nity and count­less in­di­vid­u­als shared mes­sages of sup­port with #WeMissTurkey and their de­sire to once again col­lab­o­rate with the peo­ple of Turkey on Wikipedia.

With the de­ci­sion to­day, our ed­i­tors in Turkey will once again be able to fully par­tic­i­pate in shar­ing and con­tribut­ing to free knowl­edge on­line.

* We have re­ceived re­ports that sev­eral in­ter­net ser­vice providers in Turkey, de­pend­ing on the lo­ca­tion, have re­stored ac­cess to Wikipedia in Turkey, with some still in the process of restor­ing ac­cess. We will keep this state­ment up­dated as fur­ther ac­cess is re­stored.

Click on a cell be­low to tog­gle bit val­ues. Use this to build in­tu­ition for the IEEE float­ing-point for­mat. See Wikipedia for de­tails on the half-pre­ci­sion, sin­gle-pre­ci­sion and dou­ble-pre­ci­sion float­ing-point for­mats.

7. Searching in code

What do I want

11. Future and my holy grail of search

Having in­for­ma­tion in the dig­i­tal form, col­lect­ing and writ­ing notes is in­cred­i­bly valu­able.

Our brains are good at as­so­ci­a­tions, pat­tern match­ing and cre­ative think­ing, not stor­ing ar­rays of struc­tured data, and ex­ter­nal mem­ory is one of the main think­ing hacks com­put­ers aid us with.

However this in­for­ma­tion is not so use­ful if you can’t ac­cess and search it quickly. Instant search changes the way you think. Ever got sense of flow while work­ing through some prob­lem, and try­ing dif­fer­ent things from Stackoverflow or doc­u­men­ta­tion?

These days, if you have de­cent con­nec­tion, you are sec­onds away from find­ing al­most any pub­lic knowl­edge in the in­ter­net. However, there is an­other as­pect of in­for­ma­tion: per­sonal and spe­cific to your needs, work and hob­bies. It’s your todo list, your pri­vate notes, books you are read­ing. Of course, it’s not that well in­te­grated with the out­side world, hence the tool­ing and ex­pe­ri­ence of in­ter­act­ing with it is very dif­fer­ent.

To find some­thing from my Messenger his­tory with a friend, I need to be on­line, open Facebook, nav­i­gate to search and use the in­ter­face Facebook’s em­ploy­ees thought con­ve­nient (spoiler: it sucks)

It’s my in­for­ma­tion, some­thing that came out from my brain. Why can’t I have it avail­able any­where, any­time, pre­sented the way I pre­fer?

To find some­thing in my Kobo ebook, I need to reach my de­vice phys­i­cally and type the query us­ing the vir­tual key­board (yep, e-ink lag!). Not a very pleas­ant ex­pe­ri­ence.

It’s some­thing I own and have read. Why does it have to be so hard?

Such things are pretty frus­trat­ing to me, so I’ve been work­ing on mak­ing them eas­ier. Search has to be in­cre­men­tal, fast and as con­ve­nient to use as pos­si­ble. I’ll be shar­ing some of work­flows, tricks and thoughts in this post.

The post is geared to­wards us­ing Emacs and Org-mode, but hope­fully you’ll find some use­ful tricks for your cur­rent tools and work­flow even if you don’t. There is (almost) noth­ing in­her­ently spe­cial about Emacs, I’m sure you can achieve sim­i­lar work­flows in other mod­ern text ed­i­tors given they are flex­i­ble enough.

Note: through­out the post I will link to my emacs con­fig snip­pets. To pre­vent code ref­er­ences from stal­ing, I use perma­links, but check mas­ter branch as well in case of patches or more com­ments in code.

* my per­sonal notes, tasks and knowl­edge repos­i­tory (this blog in­cluded)

By per­sonal in­for­ma­tion I re­fer to things like todo list, per­sonal wiki or what­ever you use to store in­for­ma­tion rel­e­vant to your life.

For the most part, I keep things in Org mode, and I use Emacs to work with it. Apart from reg­u­lar means of plain­text search (I’ll write about it later), for me it’s im­por­tant to search over tags:

org-tags-view is avail­able by de­fault and an easy way to run sim­ple tag searches

It’s a rel­a­tively new pack­age, with new query syn­tax as the main fea­ture, which is much eas­ier to use and re­mem­ber than builtin Org query syn­tax: com­par­i­son.

I mainly use these com­mands:

helm-org-ql for in­cre­men­tal search in the cur­rent buffer

org-ql-search in­ter­ac­tively prompts you for the search tar­get, sort and group­ing

Another no­table men­tion is org-ri­fle, which is an en­try based search, pre­sent­ing head­ings along with the matched con­tent in Helm buffer. However as the au­thor men­tioned it might be ob­so­leted by org-ql soon.

Here are some typ­i­cal work­flows with my org-mode:

I see an in­ter­est­ing ar­ti­cle or think of some­thing which would be good to share with a friend, but at the mo­ment it’s not quite a good time to send it. I can just cap­ture it and at­tach a tag (e.g. or ). That way next time we chat I can just look up things un­der their tag and send them.

It works the other way around as well: imag­ine they sent me a link or asked me to do some­thing, but I can’t do it im­me­di­ately. I have a spe­cial script that con­verts chat mes­sages into todo items and au­to­mat­i­cally at­taches the cor­re­spond­ing tag. I write more about it here.

Unfortunately, I can’t just sit and write com­pre­hen­si­ble texts with­out prepa­ra­tion. Typically I have thoughts on the topic now and then, which I just note down and mark with the tag. When I feel it’s time to pre­pare the post, I can just search by the tag (e.g. tags:search for this post), and re­file the items into the file with the post draft.

As I men­tioned, I find hav­ing to switch to the browser, wait till the web­site loads and cope with crappy search im­ple­men­ta­tions very dis­tract­ing and frus­trat­ing.

What is more, of­ten you don’t even re­mem­ber whether ex­actly you were dis­cussing some­thing: on Telegram or Facebook or Reddit? So hav­ing a sin­gle point of en­try to your in­for­ma­tion and uni­fied search over all of your stuff is ex­tremely help­ful.

For in­stant mes­sag­ing, I’m us­ing plain­text mir­rors, so chat his­tory is al­ways avail­able in plain­text on my com­put­ers:

Telegram mes­sages

Didn’t bother with org-mode be­cause files would be too huge and there is­n’t much struc­ture any­way.

Vkontakte mes­sages

Sadly ex­port tool stopped work­ing be­cause of API re­stric­tions, but I’m not us­ing VK much any­more ei­ther. At least I got his­toric mes­sages.

Most ser­vices where I can com­ment, write or leave an­no­ta­tion, I’m mir­ror­ing as org-mode. I write about it in de­tail here: part I, part II.

That gives me source data for a search en­gine over any­thing I’ve ever:

All these files are ei­ther non-Org or some­what heavy for struc­tured Org-mode search. In ad­di­tion, I have many old files from my pre-orgmode era when I was us­ing Gitit or Zim.

To search over them, I’m us­ing Emacs and Ripgrep (you can read why later):

my/​search

runs rip­grep against my/​search-tar­gets vari­able con­tains paths to notes, chat logs, Orger out­puts etc.

The in­ter­est­ing bit about my/​search is –my/one-off-helm-follow-mode call. It’s a some­what hor­ri­ble hack that au­to­mat­i­cally en­ables helm-fol­low mode so you don’t have to press C-c C-f every time you in­voke helm.

Finally, to make sure I can in­voke search in an in­stance, I’m us­ing a global key­bind­ing.

Here’s a demo gif (5Mb) of us­ing this to search ‘greg egan’ in my knowl­edge repos­i­tory. You can see that as a re­sult, I’m get­ting my Kobo high­lights (), my read­ing list () and even some video ()!

Recoll is an in­dexer that runs as dae­mon (or a reg­u­lar cron job) and a full text search tool.

It sup­ports many for­mats and other fea­tures, so I sug­gest check­ing them out for your­self.

Even though I in­dex all my doc­u­ments, I find it quicker to run grep I de­scribed above to search in plain­text. So for me, Recoll is mostly for search­ing and quickly jump­ing to re­sults in PDFs and EPUBs (see screen­shot).

There is helm-rec­oll Emacs mod­ule, but I found it a bit awk­ward to use, and Recoll GUI feels sig­nif­i­cantly su­pe­rior. Basically only thing helm-rec­oll does is pre­sent­ing you list of file­names that match your query. It feels that it should be straight­for­ward to mod­ify the mod­ule and in­te­grate ab­stract, snip­pets and other things you can query Recoll for.

Considering I don’t need use Recoll it too of­ten, I just gave up on helm-rec­oll and us­ing GUI.

I’m also run­ning a Web UI on my VPS, so I can use it from my phone, or po­ten­tially from other com­put­ers.

Recoll’s dis­tin­guish­ing fea­tures are proper search query lan­guage

and re­al­time, in­o­tify based in­dex­ing. I don’t have that much data yet to ben­e­fit mas­sively from proper search queries, but I can see that it could be po­ten­tially use­ful in fu­ture as amount of per­sonal data grows.

Most of my notes and knowl­edge repos­i­tory are plain­text, so it is eas­ily and con­tin­u­ously shared on my phone via Dropbox/Syncthing.

Since us­ing Emacs on Android is hardly a mean­ing­ful ex­pe­ri­ence, I’m work­ing around it by us­ing other apps.

I can’t rec­om­mend it enough, it’s got many things done right, very fast and the code is ex­tremely read­able and well tested so it’s easy to con­tribute.

It has its own small query lan­guage (at the time org-ql did­n’t ex­ist).

You can save search queries, which ends up be­ing pretty sim­i­lar to cus­tom Org-mode agen­das. Searches can be dis­played as per­sis­tent wid­gets, e.g. I find con­ve­nient to have a phone screen ded­i­cated to ‘Buy’ search (t.buy query) or ‘Do at work’ search (t.@work query).

As I de­scribed above, I keep few saved search queries for some friends so I can re­call what I wanted to dis­cuss with them.

Docsearch is a not very well-known tool (e.g. zero search re­sults on Reddit or Twitter), but I don’t know any al­ter­na­tives for it.

It’s a full­text in­dex­ing and search app for plain­text files, but ap­par­ently it even sup­ports EPUBs and PDFs.

Here’s how matches list looks. Screenshots on Google Play give a pretty good idea what the app does.

I find it con­ve­nient for quick search over things that are not im­ported in Orgzly, e.g. chat logs (Telegram, VK) and huge org-mode files I de­scribed above.

It’s a bit back­wards in terms of UI (even though I like that it’s com­pact and func­tional), but main down­side is it’s not open­source. I’d be ex­tremely happy to re­place this with some open source ap­pli­ca­tion, so please let me know if you know one!

On the rare oc­ca­sions when I need to search in pdfs or books (which I don’t sync on my phone) , I just use Recoll Web UI that I’m self­host­ing.

If you’re read­ing this at all, chances you’re quite good at us­ing web search al­ready. Gwern got a good writeup on the sub­ject.

Knowing how to com­pose a search query is one thing, but nav­i­gat­ing to the ser­vice, wait­ing till it loads, mov­ing to search­box takes pre­cious time. Many peo­ple for­get about cus­tom search en­gines. Here are ones I’m us­ing:

Some of these ob­vi­ous, some de­serve sep­a­rate men­tion:

red­dit con­tains vast amounts of (somewhat cu­rated) hu­man knowl­edge

Google search of­ten gives du­bi­ous and not very mean­ing­ful re­sults on cer­tain top­ics (e.g. prod­uct re­views, ex­er­cise, di­et­ing). On red­dit, you’d at least find real peo­ple shar­ing their hon­est and real opin­ions. Chances are that if a link is good, you would find it on on red­dit any­way.

twit­ter is sim­i­lar: there is cer­tainly more spam there, but some­times it’s in­ter­est­ing to type a link or blog post ti­tle in twit­ter search to see how real peo­ple re­acted.

That has lim­ited util­ity, e.g. does­n’t work with politi­cized con­tent, but if the topic of in­ter­est is rare, could be very use­ful.

* pin­board is an awe­some source of cu­rated con­tent as well

Next, I find it very con­ve­nient to have some code doc­u­men­ta­tion avail­able lo­cally. First, it helps when you’re on wonky in­ter­net or just of­fline for what­ever rea­son. Second, it’s feels re­ally fast, even if you’re on fiber.

Here’s what I’m us­ing for that:

Sadly, the ex­ten­sion men­tioned above does­n’t work with schema for some rea­son, so to add it in Firefox, you can use the method de­scribed here, it’s as easy as adding a book­mark.

Recently I ran into de­v­docs.io, it’s us­ing your browser’s of­fline stor­age to cache the doc­u­men­ta­tion. I’m still get­ting used to it, but it’s amaz­ing how faster it is than jump­ing to doc­u­men­ta­tion on­line. You can use it with mul­ti­ple lan­guages, you just type the search en­gine pre­fix first, and then lan­guage pre­fix (e.g. dd cpp em­place_back).

Finally, it may be con­ve­nient to set up en­gine-mode in Emacs, or search-en­gine layer in Spacemacs. It lets you in­voke a browser search di­rectly from Emacs (e.g. SPC s G to do google search). I find it con­ve­nient when I need to search many things in bulk.

I find it con­ve­nient to en­able ‘highlight all’ for search within a page.

When I used Chrome, one thing that an­noyed me was that it pop­u­lates search en­gines au­to­mat­i­cally, and there is not way to dis­able it.

There is a nice open source ex­ten­sion that pre­vents Chrome from do­ing it.

TLDR: I tried dif­fer­ent ex­ist­ing code search tools, was dis­ap­pointed and ended up us­ing Emacs + Ripgrep.

I’ve got lots of per­sonal pro­jects, ex­per­i­ments, data pro­cess­ing and backup scripts on my com­puter. I also tend to cre­ate a git repos­i­tory at a slight­est op­por­tu­nity pri­mar­ily as a means of code backup/​roll­back and progress track­ing, but of­ten it re­sults in ac­tual pro­jects, so I would need a repos­i­tory any­way. Naturally, these repos­i­to­ries end up scat­tered across the whole filesys­tem, mak­ing it tricky to re­mem­ber where I’ve put the code or that it even ex­isted in the first place.

It’s very con­ve­nient to have some sort of code search en­gine if you’re in a sim­i­lar sit­u­a­tion to mine for mul­ti­ple rea­sons:

For in­stance, I want to re­move some un­used func­tion or refac­tor some­thing in my pack­age, which is a Python li­brary to ac­cess my per­sonal data. It’s used in lots of scripts or dash­boards that run in Cron every day.

I could just go for it, re­move the func­tion and hope noth­ing fails, but if it does then I’d have to deal with fix­ing it again. It’s frus­trat­ing and I’d rather search for func­tion us­ages in all of my code and make sure it’s ac­tu­ally safe to re­move.

When you’re get­ting fa­mil­iar to some new li­brary or frame­work, you of­ten end up googling how to solve prob­lems twice. Sometimes you re­mem­ber solv­ing the prob­lem you’ve al­ready had, but don’t quite re­call where.

For in­stance for me, such li­brary is Sqlalchemy. It’s very con­ve­nient for han­dling data­bases, but I only need it in­fre­quently, so can never re­mem­ber how to work with it. Reading doc­u­men­ta­tion all over again is not very help­ful be­cause I’ve got very few use­cases and queries that are spe­cific to my pur­poses.

If I can search for sqlalchemy in my code, it shows every repos­i­tory where I used it so I can quickly copy bit of code I’m in­ter­ested at.

It hap­pens that I re­mem­ber writ­ing code for some pur­pose, but don’t quite re­call where I put it. Even if you keep all your re­pos in the same lo­ca­tion, you might for­get how you named it.

Full text search, how­ever, al­lows to find it if you re­mem­ber some com­ments or class/​func­tion names.

However good is li­brary’s doc­u­men­ta­tion, some­times it just is­n’t cov­er­ing your typ­i­cal needs. If you’re a power user, docs are al­most never enough and you end up read­ing the code to bend the li­brary into do­ing what you want.

For me such li­braries are Org mode or Hakyll, so I of­ten had to search in their code on Github. Searching on Github how­ever is quite awk­ward. It’s slow, it’s not in­cre­men­tal and lacks nav­i­ga­tion.

If I have a lo­cal clone of the repos­i­tory on my disk, I can search over it in an in­stant (without hav­ing it opened in the first place) and use fa­mil­iar tools (e.g. IDE) for nav­i­ga­tion.

At the time, I was just us­ing re­cur­sive grep and then open­ing some of re­sults in vim to re­fine. That’s a pretty pa­thetic work­flow.

run against code on my filesys­tem

Just any source files, so it would­n’t have to fetch repos­i­to­ries from Github and keep them some­where sep­a­rately.

Ideally, in­o­tify-based, but any means of re­fresh­ing search in­dex with­out hav­ing to com­mit first would be nice.

se­man­tic search in de­f­i­n­i­tions/​vari­ables etc with fall­back to sim­ple search if the lan­guage is­n’t sup­ported

So, I wanted some code search and in­dex­ing tool that could watch over all the source file on my filesys­tem and let me search through them.

It sounds as a fairly straight­for­ward wish, but to my sur­prise, none of ex­ist­ing pro­jects I found and tried do the job:

Lets you in­dex Github/Bitbucket/Gitlab re­pos etc, but the process for adding lo­cal repos­i­to­ries is ex­tremely te­dious. Also ap­par­ently, it clones repos­i­to­ries first so it’s not ex­actly re­al­time in­dex­ing.