- Hacker News Frontpage

- Ask Hacker News

Hacker News Frontpage

- UK sees privatisation 'opportunities' in Ukraine war

- My new POWER Indigo 2

- Americans see their savings vanish in Synapse fintech crisis

- Quake 3 Source Code Review: Network Model (2012)

- MaXX Interactive Desktop -- the little brother of the great SGI Desktop on IRIX

- The Neuroscience Link Between Neuroticism and Social Media Addiction

- RGFW: Single-header C99 window abstraction library

- Translating My Grandfather's Biograpy

- Understanding Google's Quantum Error Correction Breakthrough

- Phased Array Microphone (2023)

- Rendering "modern" Winamp skins in the browser

- Amazon to invest another $4B in Anthropic

- The Deceptively Asymmetric Unit Sphere

- Salmon return to lay eggs in historic habitat after dam removal project

- Runtime-Extensible SQL Parsers Using Peg

- From string to AST: parsing (2019)

- Prince Nico Mbarga’s biggest hit outsold any of The Beatles’ (2017)

- Show HN: A Marble Madness-inspired WebGL game we built for Netlify

- FaSTer: Atari ST Digital Magazine

Amid the devastating war in Ukraine, British economic aid to the country is focused on promoting pro-private sector reforms and on pressing the government to open up its economy to foreign investors.

Recently-published Foreign Office documents on its flagship aid project in Ukraine, which supports privatisation, note that the war provides “opportunities” for Ukraine delivering on “some hugely important reforms”.

The government in Kyiv has in recent months been responding positively to these calls. Last month, president Volodymyr Zelensky signed a new law expanding the privatisation of state-owned banks in the country.

It follows the Ukrainian government’s announcement in July of its ‘Large-Scale Privatisation 2024’ programme that is intended to drive foreign investment into the country and raise money for Ukraine’s struggling national budget, not least to fight Russia.

Large assets slated for privatisation currently include the country’s biggest producer of titanium ore, a leading producer of concrete products and a mining and processing plant.

Ukraine envisaged privatising the country’s roughly 3,500 state-owned enterprises in a law of 2018, which said foreign citizens and companies could become owners.

The process stalled as a result of coronavirus and then Russia’s invasion in February 2022. But hundreds of smaller-scale enterprises are now being privatised, bringing in revenues of UAH 9.6bn (£181m) in the past two years.

“The resumption of privatisation amid the full-scale war is an important step, which is already yielding results,” Ukraine’s economy minister Yulia Svyrydenko said last month.

Another law enacted in June 2023 allows large-scale assets to be sold to foreigners or Ukrainians during the current martial law regime.

RELATED

‘Good governance’

Britain’s main economic aid project in Ukraine runs from 2022-25 and is called the Good Governance Fund. One of its aims is to ensure that “Ukraine adopts and implements economic reforms that create a more inclusive economy, enhancing trade opportunities with the UK”.

A recent project update from the Foreign Office is explicit about the goals. It states these should see “the invasion not only as a crisis, but also as an opportunity”.

It notes: “Ironically, despite the horrific circumstances in which interventions are being delivered, the operating context has provided a unique opportunity to really demonstrate to the GoU [government of Ukraine] and the Ukrainian population the importance and effectiveness of targeted technical assistance interventions designed to deliver reforms that generate tangible benefits”.

These reforms are also variously described in the UK project documents as “better integration with Euro-Atlantic markets” and “aligning it [Ukraine] more closely with Western markets”.

A Foreign Office spokesperson told Declassified: “The UK is committed to supporting Ukraine to not only win the war but to win the peace, by emerging as a strong, prosperous, and free nation.

“The Good Governance Fund’s work in Ukraine has strengthened Ukraine’s economic resilience and future growth, and its performance has been commended by the Independent Commission on Aid Impact who scrutinise aid spending.

“We will continue to support a Ukraine-led effort to emerge from the war with a modernised economy resilient to Russian threats.”

Advancing privatisation

One key strand of the Good Governance Fund project is direct support to privatisation in Ukraine.

This involves a seven-year sub-programme called SOERA (State-owned enterprises reform activity in Ukraine), which is funded by USAID with the UK Foreign Office as a junior partner.

SOERA works to “advance privatization of selected SOEs [state-owned enterprises], and develop a strategic management model for SOEs remaining in state ownership.”

UK documents note the programme has already “prepared the groundwork” for privatisation, a key plank of which is to change Ukraine’s legislation.

“SOERA worked hand-in-hand with GoU and proposed 25 pieces of legislation of which 13 were adopted and implemented”, the most recent documents note.

SOERA has also worked with the deputy economy minister to begin mapping “thousands of SOEs, which will be a launchpad for further state property transfer”.

The aid project has run alongside the Ukraine Recovery Conference process, in which the British government hosted an international summit in June 2023.

This sought to “help Ukraine to unlock the investment and expertise it needs to rebuild as a resilient democracy with a green and modernised economy and the ability to deter and withstand future Russian aggression”.

Mainly, the conference aimed to encourage private sector investment in Ukraine and “to cement Ukrainian commitment to advancing the reform agenda”, as the Foreign Office puts it.

Declassified made a freedom of information request asking the Foreign Office to provide the briefing notes for then foreign secretary James Cleverly for the conference. It replied saying the request was “too broad”.

“The UK is hoping to reap benefits for UK firms from Ukraine’s reconstruction”, observes a report on British aid to Ukraine earlier this year by the aid watchdog, ICAI.

RELATED

Conditionality

Britain’s privatisation agenda in Ukraine is part of a wider push by the World Bank and the International Monetary Fund (IMF), which routinely promote privatisation in low income countries, often as a condition of providing aid.

Zelensky’s recent announcement on state-owned banks is based on World Bank recommendations and gives international donors a role in selecting financial advisers for the sales.

Earlier this month the World Bank announced it was allocating $593m to support Ukraine’s private sector, focused on “improving the regulatory environment”. The initiative will “strengthen our deregulation efforts”, economy minister Svyrydenko said.

One of the conditions imposed by the IMF in last year’s $15.6bn loan to Ukraine is for the government in Kyiv to produce a strategy on privatisation.

Similarly, the European’s Union’s recently-agreed Ukraine Plan – which will provide 50bn euros to Ukraine in grants and loans during 2024-27 – is also conditional on, among other things, the “entry into force of the legislation on corporate governance of state-owned enterprises”.

Rustem Umyerov, the head of the State Property Fund, which presides over Ukraine’s privatisation strategy, said in July that “international partners support the start of large-scale privatization and are ready to facilitate pitches to the business communities in their countries.”

“The search for strategic investors… is an opportunity for their development and a path to leadership in the world market”, he added.

Foreign investment in rebuilding Ukraine’s economy is being coordinated by the world’s largest asset manager, Blackrock.

‘Strategic communications’

Ukraine’s economy shrank by around a third in the first year of the Russian invasion, leaving the government in Kyiv searching for new sources of finance.

The country’s state-owned companies have often been inefficient and a notorious source of corruption, and most analysts regard private sector reform as critical.

Kyiv is not proposing to privatise all state-owned companies. The Ukrainian government notes that privatisation can benefit the country by reducing subsidies, provide income to the state budget, and “increase public benefits through market-oriented products and services”.

But privatisation can also do the opposite, as British citizens need little reminding – it can create private monopolies, reduce accountability to government and overcharge the public.

The key goal for Western states supposedly ‘aiding’ Ukraine’s privatisation process is to find access to new markets, and to bring Ukraine into their commercial orbit, fully detaching it from their rival, Russia.

A sign that the Ukrainian public needs persuading about this Western-backed privatisation is that the US/UK’s SOERA project includes a public relations dimension. One of its goals is to “assist the government in strategic communications to enhance reforms”.

My new POWER Indigo 2

Posted on 2024-11-22

On Mastodon, I recently came across a post by Kestral, with a link to their website:

I find myself with significantly less time than I’d like to work on restoring all of these devices, so my hope is that folk in the community will take some off my hands, lovingly restore, and help to keep computer history alive. Get in touch with me in the Fediverse if anything here sparks joy.

Oooh, me me me!

After exchanging a few messages, we arranged to meet at the Centre for Computing History, which happened to be hosting the annual Retro Computer Festival. I covered the cost of entry to the museum, and in exchange I received two things. One of which we're going to talk about here, and the other I'll talk about later.

I am incredibly grateful to Kestral for the opportunity to enjoy these extraordinary pieces of technology. Thank you!

But you know what this one is, because you read the title.

(Click any image for a large version)

This is a Silicon Graphics workstation from 1995. Specifically, it is an 'Teal' Indigo 2 (as opposed to a 'Purple' Indigo 2, which came later). Ordinarily that's rare enough - these things were about £30,000 brand new. A close look at the case badge though, marks this out as a 'Teal' POWER Indigo 2 - where instead of the usual MIPS R4600 or R4400SC CPU modules, we have the rare, unusual, expensive and short-lived MIPS R8000 module.

But first, let's open it.

Flaps down to view the drives. We haven't got any. That pained squeak you can hear is coming from my wallet.

This panel comes off.

Pop the catches and up comes the lid.

Yeah, definitely no drives. Just some dust.

At this point, I have no idea if the POWER Indigo 2 works, and if it doesn't, what it might take to make it work. They say there's nothing in life as expensive as a cheap Jaguar. Well, a cheap Silicon Graphics workstation is right up there.

It's a very well engineered case design. I mean, it should be for the price. I can see three spaces for slotting in what I assume at this point at SCSI drives (they are) fitted to some kind of disk sled I don't have (and which turn out to be quite expensive). There's a flat flex cable joining the various drive bays, and it looks incredibly easy to damage. Over on the left is some kind of expansion bay. I cannot see the CPU or the RAM at this point.

Ah, there are the RAM slots - in the centre at the back. There is no RAM, which is a problem. We'll come back to that.

#

The graphics card is enormous. This is the Silicon Graphics 'Express' series 'XZ' graphics card, which is mid-range option for a 'Teal' Indigo 2. The lower-end card was the 'XL24', which took one slot. The high-end card was the 'Extreme' which took three slots. This XZ just takes two slots, as we'll see in a moment.

Close ups of the chips on the VB2 - the uppermost board in an XZ.

Just to confuse things, there were actually two graphics cards from SGI called 'XZ' - one with two Geometry Engines and one with four Geometry Engines. Why SGI decided to upgrade the card without renaming it I don't know. This is the later four-GE version, which is also known as 'Elan' after the high-end card that shipped in the earlier "Indigo". But this isn't an Elan, it's a four-GE XZ, because it's from an Indigo 2. This guide to 'Express' graphics explains it all.

The GR5 board, with its four Geometry Engines, is underneath. They are connected with a large (and dusty) inter-board connector. They connect to the expansion board riser using SGI's proprietary GIO64 bus, the connectors for which sit in parallel with EISA bus slots for slower cards.

This is the back of the GR5 board.

Some better shots of the expansion riser.

The inside of a drive bay has this weird almost-but-not-quite SCA connector. It carries 10 MB/sec Fast SCSI-2, which was cooking for 1993 when this chassis was first introduced. You need a 'sled' go in here, which holds a standard 50-pin Fast SCSI drive and adapts the drives connectors to something that fits here.

This is the back of the sled.

Now here's the CPU module, with a curious warning screwed on.

It reads:

WHEN INSTALLING R8000 MODULE IT IS CRITICLE (sic) THAT ALL MFG SCREWS ARE INSTALLED AS THEY PROVIDE PWR AND GND!

MEMORY SIMMS THAT EXCEED 1.26 INCHES IN HEIGHT MAY NOT BE INSTALLED UNDER R8000 MODULE!

Let's unscrew the very important power delivery screws and flip the module up.

The R8000 is not a CPU in the traditional sense. It is a processor, but that processor is comprised of many individual chips, some of which you can see and some of which are hidden under the heatsink.

The MIPS R8000 was apparently an attempt to wrestle back the Floating-Point crown from rivals. Some accounts report that at 75 MHz, it has around ten times the double-precision floating point throughput of an equivalent Pentium. However, code had to be specially optimised to take best advantage of it and most code wasn't. It lasted on the market for around 18 months, before bring replaced by the MIPS R10K in the 'Purple' Indigo 2.

Based on some Swiss price lists I found, a 64MB POWER Indigo 2 with XZ Graphics and a 2GB SCSI drive would run you around £58,000. I hope the first owner felt they got their money's worth. Some 30 years later, I know I did.

Now the CPU card is out of the way, we can see the chipset underneath and those final two RAM slots. Definitely empty.

The ribbon looks delicate. Let's not flip this incredibly rare CPU module up out of the way very often.

Does it boot though? Well to find out I needed to order some RAM and a 13W3-to-VGA adapter. The lovely people at SGI Depot hooked me up, and I only had to wait a day or so for delivery. I am very impressed with the service, and Ian was kind enough to chat over e-mail about exactly what I had here, and what I might need to get it running.

I found this Personal Computer World archive online. Referring to the June 1995 issue and page 286, we can see Power Mark selling "MEMORY for SILICON GRAPHICS Workstations". A 64MB RAM kit for an Indigo (which like my Indigo 2 took standard 72-pin parity SIMMs) came in at £1,999. That amount of money was enough to buy a Dan Technologies Pentium 100 machine with 16MB RAM, 1GB HDD and a 14" monitor.

But does it boot?

First I pulled the PSU plugs from the mainboard and tested it in isolation. It wasn't regulating 12V terribly well, but the 5V was about right, and it was at least not putting out 12V (or 230V) on the 5V rail, so I figured it was safe to plugin.

I livestreamed, this process, which you can see here:

You can see the picture below already - of course it boots. These things were build like tanks. At 10:03 we do the first power-on and you can hear the lovely chime sound. Very elegant.

This is my Dell 1908FP 19" TFT, which has a native resolution of 1280x1024. This is fortuitous, because the SGI puts out a native resolution of 1280x1024. I think you can change the NVRAM settings to make it run at a lower resolution, but I think my NVRAM battery is flat and this is the default. I didn't know at the time that the 1908FP accepted Sync-On-Green video, so I'm running it though my orange McBazel OSSC. Shortly after these pictures were taken, the OSSC output started flashing up black. The unit then expired entirely and refused to boot or show anything on its LCD.

The options on the boot screen are:

- Start System

- Install System Software

- Run Diagnostics

- Recover System

- Enter Command Monitor

- Select Keyboard Layout

You thought your UEFI BIOS was fancy with its bitmap display and mouse input? Silicon Graphics were doing this in the early 1990s.

In this next image we're looking at the installer for IRIX 6.2, which I've booted off my BlueSCSI.

To install IRIX 6.2 onto a blank hard drive you have to:

1. Boot into the SASH shell, with boot -f dksc(1, 5, 8)sash64

2. Boot from SASH into the stand-alone partition tool with boot -f dksc(1, 5, 7)/stand/fx64

3. Put the partition tool into 'extended' mode, and then auto-initialise the drive. This will run a complete bad-block sweep and lay down an SGI disk label - their equivalent of an MS-DOS partition table.

Here is the IRIX 6.2 "Magic Desktop". I'm setting up SoftWindows Version 2.0, which was shipped with the OS. It's automatically installing Windows 3.1 for me, which seems to be bundled with SoftWindows, although you do need to find a licence key to make it work.

I made some art, using "IRIS Showcase". Quite a nice vector graphics package, it reminds of of Acorn's !Draw. But better. And with 3D shapes. And 'Gizmos' (which are non-modal dialogs you can leave open, like Lotus SmartSuite tried to claim was so novel several years later).

That's it for now! I bought a sled, so now IRIX is installed on a real 4GB SCSI Quantum Fireball HDD ... whilst it lasts, anyway. The machine hasn't crashed once so far, and Magic Desktop is a real joy to use. I've even been FTP'ing things over from my main machine, although IRIX 6.2 can't mount loop-back images so actually most software is installed by dropping an ISO onto the BlueSCSI's SD Card and then mounting that inside IRIX, just like I did with the OS install images.

I like this machine a lot.

Further Reading:

About Me · Blog · RSS · My CV

Built with Zola.

Maintained with ♥ for the web.

Oscar Wong | Moment | Getty Images

For 15 years, former Texas schoolteacher Kayla Morris put every dollar she could save into a home for her growing family.

When she and her husband sold the house last year, they stowed away the proceeds, $282,153.87, in what they thought of as a safe place — an account at the savings startup Yotta held at a real bank.

Morris, like thousands of other customers, was snared in the collapse of a behind-the-scenes fintech firm called Synapse and has been locked out of her account for six months as of November. She held out hope that her money was still secure. Then she learned how much Evolve Bank & Trust, the lender where her funds were supposed to be held, was prepared to return to her.

"We were informed last Monday that Evolve was only going to pay us $500 out of that $280,000," Morris said during a court hearing last week, her voice wavering. "It's just devastating."

The crisis started in May when a dispute between Synapse and Evolve Bank over customer balances boiled over and the fintech middleman turned off access to a key system used to process transactions. Synapse helped fintech startups like Yotta and Juno, which are not banks, offer checking accounts and debit cards by hooking them up with small lenders like Evolve.

In the immediate aftermath of Synapse's bankruptcy, which happened after an exodus of its fintech clients, a court-appointed trustee found that up to $96 million of customer funds was missing.

The mystery of where those funds are hasn't been solved, despite six months of court-mediated efforts between the four banks involved. That's mostly because the estate of Andreessen Horowitz-backed Synapse doesn't have the money to hire an outside firm to perform a full reconciliation of its ledgers, according to Jelena McWilliams, the bankruptcy trustee.

But what is now clear is that regular Americans like Morris are bearing the brunt of that shortfall and will receive little or nothing from savings accounts that they believed were backed by the full faith and credit of the U.S. government.

The losses demonstrate the risks of a system where customers didn't have direct relationships with banks, instead relying on startups to keep track of their funds, who offloaded that responsibility onto middlemen like Synapse.

Zach Jacobs, 37, of Tampa, Florida helped form a group called Fight For Our Funds after losing more than $94,000 that he had in a fintech savings account called Yotta.

Courtesy: Zach Jacobs

'Reverse bank robbery'

There are thousands of others like Morris. While there's not yet a full tally of those left shortchanged, at Yotta alone, 13,725 customers say they are being offered a combined $11.8 million despite putting in $64.9 million in deposits, according to figures shared by Yotta co-founder and CEO Adam Moelis.

CNBC spoke to a dozen customers caught in this predicament, people who are owed sums ranging from $7,000 to well over $200,000.

From FedEx drivers to small business owners, teachers to dentists, they described the loss of years of savings after turning to fintechs like Yotta for the higher interest rates on offer, for innovative features or because they were turned away from traditional banks.

One Yotta customer, Zach Jacobs, logged onto Evolve's website on Nov. 4 to find he was getting back just $128.68 of the $94,468.92 he had deposited — and he decided to act.

Zach Jacobs decided to act after logging onto Evolve’s website on Nov. 4 to find he was getting just $128.68 of his $94,468.92 in deposits.

Courtesy: Zach Jacobs

The 37-year-old Tampa, Florida-based business owner began organizing with other victims online, creating a board of volunteers for a group called Fight For Our Funds. It's his hope that they gain attention from press and politicians.

So far, 3,454 people have signed on, saying they've lost a combined $30.4 million.

"When you tell people about this, it's like, 'There's no way this can happen,'" Jacobs said. "A bank just robbed us. This is the first reverse bank robbery in the history of America."

Andrew Meloan, a chemical engineer from Chicago, said he had hoped to see the return of $200,000 he'd deposited with Yotta. Early this month, he received an unexpected PayPal remittance from Evolve for $5.

"When I signed up, they gave me an Evolve routing and account number," Meloan said. "Now they're saying they only have $5 of my money, and the rest is someplace else. I feel like I've been conned."

A bank just robbed us. This is the first reverse bank robbery in the history of America.”

Zach Jacobs

Yotta customer

Cracks in the system

Unlike meme stocks or crypto bets, in which the user naturally assumes some risk, most customers viewed funds held in Federal Deposit Insurance Corp.-backed accounts as the safest place to keep their money. People relied on accounts powered by Synapse for everyday expenses like buying groceries and paying rent, or for saving for major life events like home purchases or surgeries.

Several people CNBC interviewed said signing up seemed like a good bet since Yotta and other fintechs advertised that deposits were FDIC-insured through Evolve.

"We were assured that this was just a savings account," Morris said during last week's hearing. "We are not risk-takers, we're not gamblers."

A Synapse contract that customers received after signing up for checking accounts stated that user money was insured by the FDIC for up to $250,000, according to a version seen by CNBC.

"According to the FDIC, no depositor has ever lost a penny of FDIC-insured funds," the 26 page contract states.

'We are responsible'

Abandoned by U.S. regulators who have so far declined to act, they are left with few clear options to recoup their money.

In June, the FDIC made it clear that its insurance fund doesn't cover the failure of nonbanks like Synapse, and that in the event of such a firm's failure, recovering funds through the courts wasn't guaranteed.

The next month, the Federal Reserve said that as Evolve's primary federal regulator it would monitor the bank's progress "in returning all customer funds" to users.

"We are responsible for working to ensure that the bank operates in a safe and sound manner and complies with applicable laws, including laws protecting consumers," Fed general counsel Mark E. Van Der Weide said in a letter.

In September, the FDIC proposed a new rule that would force banks to keep detailed records for customers of fintech apps, improving the chances that they qualify for coverage in a future calamity and cutting the risk that funds would go missing.

McWilliams, herself a former FDIC chair during the first Trump presidency, told the California judge handling the Synapse bankruptcy case last week she was "disheartened" that every financial regulator has decided not to help.

The FDIC and Fed declined to comment for this story, and McWilliams didn't respond to emails.

Jelena McWilliams, chairman of the Federal Deposit Insurance Corporation, testifies during a House Financial Services Committee hearing in Rayburn Building titled "Oversight of Prudential Regulators: Ensuring the Safety, Soundness and Accountability of Megabanks and Other Depository Institutions," on Thursday, May 16, 2019.

Tom Williams | CQ-Roll Call, Inc. | Getty Images

Winners and losers

Things hadn't always seemed so dire. Early in the proceedings, McWilliams suggested to Judge Martin Barash that customers be given a partial payment, essentially spreading the pain among everyone.

But that would've required more coordination between Evolve and the other lenders that held customer funds than what ultimately happened.

As the hearings dragged on, the three other institutions, AMG National Trust, Lineage Bank and American Bank, began disbursing the funds they had, while Evolve took months to perform what it initially said would be a comprehensive reconciliation.

Around the time Evolve completed its efforts in October, it said it could only figure out the user funds it held, not the location of the missing funds. That's at least partly because of "very large bulk transfers" of funds without identification of who owned the money, a lawyer for Evolve testified last week.

As a result, the bankruptcy process has minted relative winners and losers.

Some end users recently received all their funds back, while others, like Indiana FedEx driver Natasha Craft, received none, she told CNBC.

Natasha Craft, a 25-year-old FedEx driver from Mishawaka, Indiana. She has been locked out of her Yotta banking account since May 11.

Courtesy: Natasha Craft

As of Nov. 12, the four banks released $193 million to customers, or more than 85% of what they held earlier in the year.

The Nov. 13 hearing has provided the only public venue for victims to register their distress; dozens of victims queued up in the hopes they could testify about receiving a tiny fraction of what they're owed. The event went longer than three hours.

"You can't imagine the panic when it said I was getting 81 cents," said Andreatte Caliguire, who said she is owed $22,000. "I have no money, I have no path forward, I have nothing."

'Nothing optimistic'

Evolve says that "the vast majority" of funds held for Yotta and other customers were moved to other banks in October and November of 2023 on directions from Synapse, according to an Evolve spokesman.

"Where those end user funds went after that is an important question, but unfortunately not one Evolve can answer with the data it currently has," the spokesman said.

Yotta says that Evolve has given fintech firms and the trustee no information about how it determined payouts, "despite acknowledging in court that a shortfall existed at Evolve prior to October 2023," according to a spokesman for the startup, who noted that several executives have recently left the bank. "We hope regulators take notice and act."

In statements released ahead of this month's hearing, Evolve said that other banks refused to participate in its efforts to create a master ledger, while AMG and Lineage said that Evolve's implication that they had the missing funds was "irresponsible and disingenuous."

As the banks and other parties hurl accusations at each other and lawsuits pile up, including pending class-action efforts, the window for cooperation is rapidly closing, Barash said last week.

"As time goes by, my impression is that unless the banks that are involved can sort this out voluntarily, it may not get sorted out," Barash said. "There's nothing optimistic about what I'm telling you."

Don’t miss these insights from CNBC PRO

June 30, 2012

Quake 3 Source Code Review: Network Model (Part 3 of 5) >>

The network model of Quake3 is with no doubt the most elegant part of the engine. At the lower level Quake III still abstract communications with the NetChannel module that first appeared in Quake World. The most important thing to understand is:

In a fast paced environment any information that is not received on first transmission is not worth sending again because it will be too old anyway.

As a result the engine relies essentially on UDP/IP: There is no trace of TCP/IP anywhere since the "Reliable transmission" aspect introduced intolerable latency. The network stack has been augmented with two mutually exclusive layers:

- Encryption using preshared key.

- Compression with pre-computed huffman key.

But where the design really shine is on the server side where an elegant system minimize the size of each UDP datagram while compensating for the unreliablity of UDP: An history of snapshots generate deltas packets via memory introspection.

Architecture

The Client side of the network model is fairly simple: Client sends commands to the Server each frame and receive update for the gamestate. The Server side is as bit more complex since it has to propagate the Master gamestate to each Client while accounting for lost UDP packets. This mechanism features three key elements:

- A Master Gamestate that is the universal true state of things. Clients send theirs commands on the Netchannel. They are transformed in event_t which will modifiy the state of the game when they arrive on the Server.

- For each Client the server keeps the 32 last gamestate sent over the network in a cycling array: They are called snapshots. The array cycle with the famous binary mask trick I mentioned in Quake World Network (Some elegant things).

- The server also features a "dummy" gamestate with every single field set to zero. This is used to delta snapshots when there is no "previous state" available.

When the server decides to send an update to a client it uses each three elements in order to generate a message that is then carried over the NetChannel.

Trivia : To keep so many gamestate for each players consumes a lot of memory: 8 MB for 4 players according to my measurements.

Snapshot systems

In order to understand the snapshop system, here is an example with the following conditions:

- The server is sending update to a Client1.

- The server is attempting to propagate the state of Client2 which has four 4 fields (3 ints position[X], position[Y], position[Z] and one int health).

- Communication are done over UDP/IP: Those messages gets lost quite often on the internet.

Server Frame 1:

The Server has received a few updates from every client. They have impacted the Master gamestate (in green). It is now time to propagate the state to Client1:

In order to generate a message the network module will ALWAYS do the following :

1. Copy the Master gamestate in the next Client history slot.

2. Compare it with an other snapshot.

This is what we can see in the next drawing:

1. Master gamestate is copied at index 0 in Client1 history: It is now called"Snapshot1".

2. Since this is the first udpate, there are no valid snapshot in Client1 history so the engine is going to use the "Dummy snapshot" where all fields are always ZEROed. This results in a FULL update since every single field is sent to the NetChannel.

The key point to understand here is that if no valid snapshots are available in the client history the engine will pick "dummy snapshot" to generate a delta message. This will result in a full udpate sent to the Client using 132 bits (each field is preceeded by a bit marker): [1 A_on32bits 1 B_on32bits 1 B_on32bits 1 C_on32bits].

Server Frame 2:

Now let's move forward in time: this is the Server second frame. As we can see each client have sent commands and they have impacted the Master gamestate: Client2 has moved on the Y axis so pos[1] is now equal to E (in blue). Client1 has also sent commands but more important it has also acknowledged receiving the previous udpate so Snapshot1 has been marked as "ACK":

The process is the same:

1. Copy the Master gamestate in the next Client history slot: (index 1): This is Snapshot2

2. This time we have a valid snapshot in the client history (snapshot1). Compare those two snapshots

As result only a partial update ( pos[1] = E ) is sent over the network. This is the beauty of the design: The process is always the same.

Note : Since each field is preceeded by a bit marker (1=changed, 0=not changed) the partial update above would uses 36 bits: [0 1 32bitsNewValue 0 0].

Server Frame 3:

Let's move forward one more step in order to see how the system deals with lost packets. This is now Frame 3. Clients have kept on sending commands to the server. Client2 has lost life and health is now equal to H. But Client1 has not acknowledged the last update. Maybe the Server's UDP got lost, maybe the ACK from the Client got lost but bottom line is that it cannot be used.

Regardless the process remains the same:

1. Copy the Master gamestate in the next Client history slot: (index 2): This is Snapshot3

2. Compare with the last valid acknowledged snapshot (snapshot1).

As a result the message sent it partial and contains a combination of old changes and new changes: (pos[1]=E and health=H). Note that snapshot1 could have been too old to be used. In this case the engine would have used the "dummy snapshot" again, resulting in a full update.

The beauty and elegance of the system resides in its simplicity. The same algorithm automatically:

- Generate partial or full update.

- Resend OLD information that were not received and NEW information in a single message.

Memory introspection with C

You may wonder how Quake3 is comparing snapshots with introspection...since C does not have introspection.

The answer is that each field locations for a netField_t is preconstructed via an array and some clever preprocessing directives:

typedef struct {

char    *name;

int     offset;

int     bits;

} netField_t;

// using the stringizing operator to save typing...

#define NETF(x) #x,(int)&((entityState_t*)0)->x

netField_t entityStateFields[] =

{

{ NETF(pos.trTime), 32 },

{ NETF(pos.trBase[0]), 0 },

{ NETF(pos.trBase[1]), 0 },

...

}

The full code of this part can be found in snapshot.c's MSG_WriteDeltaEntity. Quake3 does not even know what it is comparing: It just blindly follow entityStateFields's index,offset and size...and sends the difference over the network.

Pre-fragmentation

Digging into the code we see that the NetChannel module slices messages in chunks of 1400 bytes ( Netchan_Transmit), even though the maximum size of an UDP datagram is 65507 bytes. Doing so the engine avoid having its packets fragmented by routers while traveling over the internet since most network MTU are 1500 bytes. Avoiding router fragmentation is very important since:

- Upon entering the network the router must block the packet while it is fragmenting it.

- Upon leaving the network problems are even worse since every part of the datagram have to be waited on and then time-costly re-assembled.

Reliable and Unreliable messages

If the snapshot system compensate for UDP datagrams lost over the network, some messages and commands must be GUARANTEED to be delivered (when a player quits or when the Server needs the Client to load a new level).

This guarantee is abstracted by the NetChannel: I wrote about it a few years ago (wow my drawings have come a long way !!).

Recommended readings

Brian Hooks a member of the developing team wrote a little bit about the Network Model.

The author of Unlagged: Neil "haste" Toronto also described it.

Next part

The Virtual Machine system

@

Welcome to MaXX Interactive Desktop

New Release Available Today

November  21st  9PM EST

MaXX Interactive Desktop is the little brother of the great SGI Desktop on IRIX

MaXXdesktop v2.2.0 (alpha) running in Modern Look & Feel with Indigo Magic SGI Scheme (4K @120Hz)

Introduction

The MaXX Interactive Desktop a.k.a. MaXXdesktop is the continuation of the 5Dwm project released many years back. So don't be mistaken, there is only one real  re-implementation of the IRIX Interactive Desktop found on SGI systems, and  it's MaXXdesktop!

MaXXdesktop is a true re-implementation of the "SGI Desktop" with the added benefits of using a modern software stack of highly tuned loosely-coupled components delivering maximum performance and stability while using as little resources as possible. We believe that high performance  computing and energy friendly are not mutually exclusive. What if you could run applications in a smarter, greener and sustainable way? MaXXdesktop aims at improving just that, do more with less.

While respecting the original retro CLASSIC SGI look and feel, which is very important for us to get it right,  we created a newer and more modern user experience called the MODERN look that feels like a natural evolution of the original SGI look, as if SGI did it themselves throughout the years perfecting an already pretty awesome recipe. The MODERN look still supports SGI Colour Schemes but introduce Unicode and UTF-8 support, anti-aliased font rendering, more hardware acceleration and a new virtual-desktop manager. The user can switch between looks with one mouse-click, now that's cool.

Goals and Features List

We believe in the need of  a highly tuned workstation environment where performance, robustness, low resources consumption are at the core of everything.

Here are the goals we want to achieve and features that makes a great modern desktop:

- Lean, very fast and robust. Basically MaXXdesktop gets out of your way.

- Smart and efficient multi-cores workload management with CPU and Core partitioning which help reduce process bouncing of  CPU cores and computation resources allocation/partitioning translate in better throughput (MaXXscope).

- Built on solid and time proven foundations.

- Desktop Applications are distributed as self contained AppImages.

- Provides a robust modern and high performance asynchronous multi-threaded messaging sub-system (with Shared Memory support and Zero copy principle for local communications) for fast and efficient inter-processes communications (MaXXlinks).

- Based on a modular micro-services like architecture that allow decentralization of core desktop services as we rapidly moving toward containerization for better security.

- Modern Configuration Management sub-system with both a command line interface (CLI) or C/C++/Java API for easy application integration (MaXXsettings).

- Highly focused on facilitating quick and easy access to your content with fast content previewers right from the file-manager.

- Leverage hardware acceleration and optimization for CPU  and GPU.

- Centralized system monitoring sub-system (MaXXmonitor).

- HiDPI supports for X/Motif applications and easy presets control for applications.

- Much needed MotifTM face-lift and modernization of the ViewKit framework with new modern components that are fully integrated with all MaXX Desktop sub-systems.

- Developer friendly software development environment with integration to CLion, Intelli-J and PyCharm IDEs from Jetbrains which translate into building better and faster applications.

- To support multiple CPU architectures (x86, Arm64 and RISC-V)

- To run on multiple OS: Linux, FreeBSD and Windows11 WSL2.

- To provide a source code compatibility for IRIX visual applications.

>> More details on what MaXXdesktop is made of

Our Mission

More for your creativity.

Our mission is to bring back this great user experience which focused on performance, stability and productivity while sporting a smooth-clean-minimalism look and feel with low-resources consumption. A smart and green Desktop that puts the user's application in the forefront.

We believe in a High Performance Desktop Environment that provides the right set of tools to maximize creativity and productivity without sacrificing  your system resources to some eye candy nonsense.  Again, less is more... And it keeps you focused.

From the Ground Up

Our design philosophy is simple, do more with less...

The MaXXdesktop is designed from the ground up for speed,  fast/responsive, lightweight/simplicity over eye-candidness, but more importantly, to get the heck out of your way... The name MaXX Interactive doesn't mean maximum visual interaction, which are distractions or so what we call, UI noises. It's means maximum creativity with interactive assistance from the Desktop. In many ways, it's made for you and your brain so that it can relax, focus on let the creative juice flowing with far fewer distractions. We see desktop notifications in a very different way, but this is for another discussion. In short, the MaXXdesktop let you focus on the creative  tasks ahead without interference or visual distractions.

Experience Matters

Our team is sharing the same vision of making use of the right set of technologies with industry proven best practices and guidelines to build the right software, the right way.  We aim at providing a consistent and pleasant user experience built on top of modern and stable foundations.  Every good and useful piece of tech deserves to be future-proof and this is where several decades of experience in building battle-proven mission critical systems for high performance Enterprise class applications comes in. If it's architect-ed properly,  it can evolve without breaking apart!

Is MaXX Desktop for you?

MaXXdesktop's typical users are old IRIX users/sysadmins, Computer Graphics Artists, Motion Pictures and Special Effects Studios, Software/Game Developers, Visualization/Simulation, Virtual Reality power-users or Oil and Gas research to name a few. MaXXdesktop is also for anyone who wants get a break from all the surrounding noises and create stuff while sporting a very unique/cool daily driver. If it is the case, then MaXX is for you.

Navigation

This site is powered by BookStack (a type of Wiki engine) and you can navigate it by using the upper-right links 'Shelves and Books'.  You may use the search bar on top or the convenient links below.

The MaXXdesktop team

The Neuroscience link between Neuroticism and Social Media Addiction

Introduction

Personality is defined as the combination of characteristics or qualities that form an individual’s distinctive character. Everyone is considered to have a different and unique personality, based on personal experiences and different motivations and emotions. In the neuroscience world, personality has often been defined by a series of tests and examinations to boil down factors of personality. One of these tests is the Big-Five Personality test. This test is a valid test and confirmed through hundreds of papers as reliable. This personality test was coined by Meyers-Briggs in the early 1900s and identified five major factors of personality: openness, extroversion, conscientiousness, neuroticism and agreeableness (Shiota, 2018, chapter 13). All five of these factors have contributions to personality to help explain why one individual is similar or different from another. Throughout this paper, the link between these personality traits, particularly neuroticism, will be linked to a new phenomenon: social media addiction and its relation to amygdala activation. This paper will attempt to answer to the question, do neurotic individuals show higher amygdala activity and more addictive tendencies toward social media use?

To build an argument to answer this question, personality traits will first be linked to addictive tendencies. This will then build on the link between neuroticism and social media addiction to infer the amygdala activation in social media addicted users.

In this paper, the primary personality trait that will be explored is neuroticism. Neuroticism is a trait defined by the tendency towards higher levels of anxiety, depression, self-doubt and overall negative feelings around one’s self and the environment surrounding them. In general, those who are identified to have higher levels of neuroticism are  more susceptible to depression, anxiety and other debilitating mental illnesses and disabilities (Kanthamani, 1973). Those with higher levels of neuroticism are seen to be “unbalanced,” “unstable,” and in general a little more insecure and less confident in their abilities. With common interactions including self-blame for situations they are not responsible for, and insecure thoughts regarding their decisions, those with neuroticism are generally linked with feelings of “not feeling good enough,” “doubt in self or others around them,” or “insecurity in friendship and stability.” Neuroticism is determined based on a Likert scale and can range in severity and intensity (Shiota, 2018, chapter 13). The cause of neurotic tendencies is unknown and highly disputed between being caused by genetic or environmental factors.

Behavioral Basis of Neuroticism

As stated above, the cause of neurotic tendencies is highly debated. Some researchers argue that neuroticism is purely environmentally influenced, as in those with higher levels of neuroticism are linked to unhealthy family lives or discomfort in their current situation. However, other researchers believe that evidence suggests that neurotic tendencies could be caused by both environmental factors and genetic factors. Therefore, it can be assumed that neuroticism is a factor of personality that is fluid and can be influenced by both the actual genetic makeup of an organism in addition to the surrounding environment and influences. In accordance with this, researchers from the University of Glasgow found that neuroticism is linked with both genetic factors and social factors. The researchers at the University of Glasgow specifically linked neuroticism to 9 different genes within a large population of 100,00 participants from a variety of different databases. This research was originally intended to be utilized to provide treatment methods for those suffering with mental health and mental illnesses. The researchers of this study specifically noted, “neuroticism is an important risk factor for depression… hence, insights into the biological mechanisms underlying neuroticism may eventually be informative for the development of drugs to treat depression,” (Smith, 2016). By linking these genomes to different personality traits, personality was able to be studied from a genetic perspective to help identify patterns for treatment of different mental health disorders and discontinuities.

Several different studies have been completed with the intention of linking back different genomes to personality traits. In a study by Nagel et al., participants allowed their DNA and genome data to be analyzed and linked back to neurological processes including specific activation in the brain amongst different kinds of neurons and genome loci. These studies found several overlapping genes and pathways linked to different aspects of personality, including those related to neuroticism as a personality trait. This study found enrichment in different brain regions and different cell types to be contributing factors that link back to neuroticism. Nagel et al. located the different loci of information to be targeted for mental health treatment (Nagel et al., 2018). The researchers found these overlapping loci to link these symptoms of negative feelings back to a genetic influence, implying that neurotic characteristics are largely genomic in nature.

Although neurotic neuroticism is liked to genetic factors, , it can also be linked back to environmental factors and influences. A study by Van, Park and Jones (2001) aimed to see how the environment influences neurotic tendencies. Previous research had indicated that early stressful life events could influence neurotic tendencies. Results found that higher occurrence of traumatic early life events - defined as events before age 16 - including deaths, accidents, injuries, divorce and other crises, resulted in higher neurotic tendencies at adult ages. In participants in this study, the highest levels of neuroticism resulting from these environmental factors was observed in adults between the ages of 36 and 43. Van et al. thus concluded that early stressful life events could largely impact the prevalence of neurotic tendencies later in adulthood. Researchers also noted some genetic factors, stating that individuals with certain genetic patterns were more susceptible to being  effected by stressful events early in life (Van et al., 2001). These results indicate that neurotic individuals are influenced by more than just genetics and are largely influenced by environmental conditions as well. This emphasizes the point of nature and nurture working together to build personality and characteristics. The question is now, how neuroticism presents itself neurologically in more neurotic individuals.

Changes in Amygdala Activation Amongst Neurotic Individuals

The amygdala is known to be the emotional center of the brain, modulating mood disorders, emotions and instincts. It is part of the limbic system, a system in the brain known for the complex behaviors stated previously. In addition, the nucleus accumbens, which is adjacent to the amygdala, is also known to be part of the limbic system and responsible for emotional regulation and behavior. In the amygdala, the release of dopamine is commonly linked with building fear memories and responses. Higher amygdala activation is largely due to strong emotional drive or emotional experiences. The nucleus accumbens - an important player in the limbic system - works directly with the amygdala and other structures to regulate emotions. In particular, the nucleus accumbens is generally related to the release of dopamine and the feeling of pleasure that comes with dopamine release (Bienvenu, 2003). This paper will focus on these two areas in the brain and the roles they play in neurotic tendencies.

Neuroticism is not only a factor of personality; it is also commonly linked with different levels of amygdala activation in the brain. Bienvenu et al. notes that amygdala and cingulate cortex activation are associated with anxiety and mood disorders; those with higher levels of neuroticism typically are found to have higher amygdala activation. For example, there is increased amygdala activation in those with anxiety or depression, or other disorders that are commonly linked with neurotic personalities. This suggests that those that are highly neurotic may be more emotionally conflicted and have more arousal in comparison to those who display less neuroticism. The other proposal is that highly neurotic individuals are proposed to be more vigilant when processing conflict, especially in cases of emotion conflict and disagreement. This would suggest that amygdala activation in neurotics could largely be due to internal conflicts, which result in abnormal amygdala activation. By spending more time on the processing of emotions, neurotics have greater amygdala activation and conflict within the brain, potentially leading to higher amygdala activation (Haas, 2007).

Other studies have indicated that high levels of neuroticism are linked with relatively less inhibitory control over negative facial expression analysis and control. A study completed by Cremers et al. (2010) found that individuals who scored higher on the neurotic personality exam displayed enhanced connectivity between the right amygdala and right dorsomedial prefrontal cortex (dmPFC) during the processing of angry and sad faces. This allowed researchers to conclude that those with higher levels of neuroticism have less inhibitory control over negative facial expressions. This means that individuals that are highly neurotic tend to get more hung up over negative facial emotions and expressions, facilitating greater linkage between these two areas of the brain (Cremers, 2010).

Neurotic individuals are known to focus on internal states, and previous research has only found a link between increased focus and negative internal states. However, some recent studies have found that neurotic individuals tend to have higher activation in the amygdala in response to not only negative experiences, but to positive stimuli and events as well. Britton et al. (2007) note that individuals with higher neuroticism scores were directly associated with higher levels of activation in the dmPFC, an area known for experiencing positive emotions. In this case, neurotic individuals also tend to focus and ruminate on positive emotional states as well. This is an interesting study as it also highlights the overall self-rumination that neurotic individuals tend to experience, pulling from a more positive emotional perspective. This perspective is extremely important in removing the negative stigma behind neurotic individuals, and it can help us to highlight the positive sides of neurotic individuals (Britton et al., 2007). Overall, it can be concluded that neurotic individuals have a larger response regarding all emotions, due to the tendency to focus on internal emotional states, both good and bad.

A study by Aghanji (2014) also notes similar characteristics amongst neurotic individuals. The results of this study did not suggest any association between neurotic individuals and amygdala activation in the hypothesized regions of the ventral affective system—the subgenual anterior cingulate cortical, dorsomedial prefrontal cortex, dorsolateral prefrontal cortex and the orbitofrontal cortex- a stark contrast to other studies. However, it did find that there were significant differences amongst amygdala activation in neurotic individuals in different regions of the amygdala, including left amygdala resting state functional connectivity (RSFC) with the precuneus. This area of the brain plays a role in self-referential informational processing, such as perception and the processing of individual personality traits and self-descriptions. These results therefore note increased amygdala activity in the part of the brain that is generally linked with feelings of self-consciousness, remorse, self-confidence and responsibility. In addition, areas of the brain related to social interaction and understanding of other’s feelings showed  decreased amygdala activation (Aghanji, 2014). This emphasizes that the areas of the brain associated with self-reflective tendencies saw greater activation in comparison to that of social areas of the brain. It can therefore be inferred that this is due to neurotic personality traits and characteristics, as more neurotic individuals are more likely to suffer from anxiety or depression or rumination in their own thoughts, and less likely to build relationships with others around them due to the anxiety and depression they feel as well as the fear of rejection from others.

Addictive Tendencies

The  link between high levels of neuroticism and social media addiction can be determined through studying addiction tendencies. . Addiction is commonly thought of   in relation to drugs. However, from a broader sense of the word, addiction is defined as a “treatable, chronic, medical disease involving complex interactions among brain circuits, genetics, the environment and an individual’s life experiences. People with addiction use substances or engage in behaviors that become compulsive and often continue despite harmful consequences,” (American Society of Addictive Medicine, 2019). By this definition, addiction can be classified as a compulsive engagement in different behaviors. Therefore, social media addiction would depend on compulsive and repeated use of social media.

A metanalysis completed by Aghaii et al. (2012) explored the factors that influence addictive tendencies. This study found that environmental factors were linked directly with addiction in 61% of studies and individual personalities/genetics contributed to addictive tendencies in approximately 45% of participants in 16 different studies. Thus, by this metanalysis, environmental factors effect addictive tendencies more prevalently than individual and genetic factors. However, this analysis does not discount the importance of behavioral and genetic conditions that could also influence addictive tendencies and patterns (Aghaii et al., 2012). Ultimately, addiction is dependent on a variety of cofactors; therefore, it is important to understand how different personalities have differing susceptibilities to addiction.

Franken (2006) implemented Gray’s model of personality and addiction to attempt to link factors of personality to addictive tendencies. Gray’s model narrows down to two basic brain systems that control behavior and emotions: the behavioral inhibition system (BIS) and the behavioral approach system (BAS). Gray’s study concluded that the BAS is activated by stimuli attached to a reward and the BIS is activated by stimuli attached to a punishment or the removal of a reward (Gray, 1993). This study linked the release of dopamine to addictive tendencies in an individual; thus, if a behavior released more dopamine, an individual was found to have more addictive tendencies towards that behavior.

Franken’s present study found that the BAS system is heightened in drug addicts but notes that this seems to be on an individualized basis based on the substance that one is addicted to. Franken found that the heightened BAS system was found mostly in those more apt to be thrill seeking and reckless (Franken, 2006). This is important in understanding the topic at hand as neurotic individuals are generally the opposite of this: they tend to show hesitation in their actions, resulting in behaviors and tendencies that are generally less risky.

But then, why is there a link between internet addiction and neuroticism? Understanding the prevalence of internet addiction amongst neurotics will hopefully help answer this question in order to link social media addiction, neuroticism and brain region activation.

Social Media Addiction and Neuroticism

What is typically referred to as social media addiction is a growing problem in society, influenced by recent technological advances, including the creation of the smart phone, several popular social media sites (i.e. Facebook, Twitter, Snapchat, Instagram, etc.), and greater accessibility to the internet. Statistics from 2019 found that on average, adults spend over three hours of time on their phone a day. In 2018, over two billion people were present on social media, with adults spending about 153 minutes per day on social media on average (Broadband, 2019). This relationship between social media and the individual is even stronger in those with more neurotic tendencies. One researcher’s 2017 study specifically looked at two of the personality traits, neuroticism and extraversion, as predictors for social media use in relationship to the fear of missing out (FOMO) with attachment style as a moderator. This paper will highlight what this study found in relation to neuroticism in particular. To start, the study defined social media addiction as “the inability to control one’s social media use,” (Blackwell, 2017). Then the study talked about different types of attachment styles and provided predictions as to why anxiously attached individuals in particular, who often display neurotic tendencies, might become addicted to social media. Anxiously attached individuals fear rejection; thus, it is predicted they may use social media to connect with those around them in a way that reduces possibilities for rejection. They also noted that anxiously attached individuals are more likely to suffer from FOMO, the fear that others around them are having fun without them. This was a correlation study using anonymous surveys to analyze student’s social media use, personality characteristics and attachment styles. The results found suggest that neuroticism is a predictor of social media use and internet addiction; the researchers concluded that more neurotic individuals tend to feel more anxious about relationships, and so they tend to rely more heavily on social media to maintain those relationships. In addition, FOMO was a moderator that predicted social media use more than personality style, an interesting fact that could potentially relate back to the key characteristics of addiction (Blackwell, 2017).

Similar findings were found amongst other researchers exploring neuroticism and social media overuse. Seidman (2013) also used neurotic individuals as a prediction of social media overuse. The results of this study found that amongst the individuals studied, the ones who were found to be more neurotic used Facebook more on average. Seidman predicted that this could be due to two different possibilities: 1) neurotic individuals seek Facebook as a way for them to experience feelings of belonging they may not feel in person to person contact, or 2) neurotic individuals may use Facebook as a safe place for their self-perseveration and identity (Seidman, 2013). These findings are consistent with previous findings, finding that neurotic individuals could potentially rely heavily on social media to fill a void in their life of social interaction caused by their anxious or negative feelings about themselves.

However, it is important to note that, in stark contrast to other studies, a study conducted by Wilson et al. (2009) did not find any correlation between social media use and neurotic tendencies. They suggest that neurotic tendencies are likely a predictor, but not a significantly associated factor leading to internet and social media use. The researchers in this study concluded that the results they obtained can be explained by the fact that neurotic young adults are more anxious and insecure in nature, and so they tend to dislike the idea of posting photos or information about themselves online where they could be criticized. They predict that rather, the high internet time seen in neurotic individuals is not from social media use, but from other outlets online. In addition, they looked at another factor – self-esteem – and its relation to social media addiction, and concluded that self-esteem could either increase or decrease social networking use, depending on the tone of feedback between both people on either side of the online conversation. With more positively received interaction, lower self-esteem could be linked to increased social media use, but with poorly received interaction, lower self-esteem would likely be linked to decreased social media use (Wilson et al., 2009). Overall, the findings of this study are an interesting paradox to previous studies that could largely be due to differences in culture. The individuals in this study were all Australian, whereas the previous studies pulled participants from American institutions and universities. Perhaps different cultural perspectives contribute to different personality traits.

Tying it all Together

Finally, research into how an individual’s sex affects their tendency to become addicted to social media was also explored. Different sexes tend to have different personality characteristics, with women being generally regarded as more “sensitive”, and men seen stereotypically to try to “shield their emotions.” Therefore, it would be interesting to see if the self-conscious characteristics associated with neuroticism are more present in one sex in comparison to the other.

A study completed in 2018 specifically aimed to look at social-media addiction and neuroticism between sexes. This study recruited Israeli College students and asked them to fill out self-reporting data. Researchers predicted that sex, neuroticism and well-being would moderate Facebook usage. The results found suggest that between men and women, more neurotic personality traits presented with lower well-being scores, once again emphasizing the negative effects of neurotic tendencies in individuals. In addition, researchers found differences in how neuroticism affects social media use between men and women. In this study, women high in neuroticism were more likely to have negative social networking site consequences in comparison to men (Turel et al., 2018). This is interesting as it indicates further that perhaps these social media struggles are moderated by a variety of characteristics: amygdala activation, sex differences, age or personality traits. This study is important in understanding how many cofactors can contribute to overall differences in social media addiction.

Although important, the results presented so far do not yet answer the question of why neurotic individuals show a trend to social media addiction, but don’t have the characteristics typical with drug addiction. Those who are more susceptible to addictive tendencies, in this case substance abuse, are generally more reckless and known to take more risks with their activities they are involved in. Meanwhile, neurotic individuals are typically described as reserved, more self-conscious and anxious - characteristics that directly confound with the typical drug or substance abuse. One inference that can be made to explain this disparity could be made in relation to the riskiness of behaviors associated with drug abuse and social media addiction. Addiction largely plays upon the reward system of the brain, commonly known to release the neurotransmitter dopamine for a feeling of pleasure right after the addictive behavior is completed. After drug use or alcohol use, a surge of dopamine is released in the nucleus accumbens. For example, cocaine, a commonly abused substance, leads to more dopamine in the synaptic cleft by block dopamine reuptake transporters; thus, allowing for a surge of dopamine to be present in the synaptic cleft (Harvard Health Publishing, 2011). This release of dopamine then causes a pleasurable feeling, creating a craving and drive to experience the behavior again for the rewarding feelings.

Similarly, social media use is also believed to result in a release of neurotransmitters. Habitual social media use can lead to addiction because it has an unpredictable reward schedule. Using social media does not guarantee a good feeling or a bad feeling, but leaves users wondering what could happen next. This causes a variable reinforcement schedule, resulting in users becoming addicted to wondering when their next “good” or “pleasurable” feeling could come (Griffiths, 2018). It is debated whether dopamine is released in this variable reward schedule; however, Griffith notes that “the use of ‘like’ buttons [has] hijack[ed] the social reward systems of a user’s brain,” (Griffiths, 2018). In addition, Griffiths notes that Sean Parker, the founding president of Facebook, even acknowledges that the company aims for a like or comment to give “you a little dopamine hit,” (Parkin, 2018). This suggests reason to believe that receiving ‘likes’ could lead to a release of dopamine and cause a pleasurable feeling in a social media user’s mind, which could cause addiction in a similar way that drugs cause addiction. This is a hypothesis that has not been adequately confirmed or denied and which is still a research work in progress but completing further research into this topic seems promising. Overall, despite the fact that it has not been studied fully, for now it seems evident that dopamine could be a factor at play in addiction to social media and overuse of social media; further research will help to clarify whether or not it is a factor.

Exploring whether social media use causes dopamine to be released is also important as it has been assumed that architects of social media sites exploit a “vulnerability in human psychology,” (Parkin, 2018). In other words, social media creators play up on human psychology to ultimately try and build that addiction-like dopamine release. They also play up on the variable reward schedule to keep users on their toes and wanting more, building anticipation and a desire for the next great thing on social media. Thus, it has recently been coined that social media is a “persuasive technology,” meant to play off behavior and reward systems like the dopamine reward system (Parkin, 2018). Therefore, although the mechanism of social media addiction has not been fully pinned down, it could be largely due to several different emotional regulation areas in the brain that should be further researched.

Conclusion

To conclude, it can be inferred that the social media addiction epidemic linked to neurotic individuals versus other addictive tendencies could be largely due to the riskiness of the behavior. Those who are prone to drug abuse or substance abuse, generally are risk-takers, searching for pleasure. However, neurotic individuals could be using social media as a tool to facilitate the same feelings in a less risky way. Neurotic individuals might be hung up on the unknown factor of social media, causing anxiety and excitement, directly playing into their stereotypical characteristics.

Moreover, amygdala activation between drug addiction, neurotics and social media abusers could vary slightly as the same pleasurable feeling of dopamine release could be modulated in different brain regions, particularly the nucleus accumbens or amygdala. In this case, social media addiction may be modulated by the amygdala, like neuroticism, whereas, drug addiction and dependency could be modulated by the nucleus accumbens, perhaps similar to other personalities of the Big Five.

Overall, this is a new topic as social media presence has heightened in the past 10-15 years with the introduction of Myspace, Facebook, the smart phone, etc. This limits the analysis at hand because there is simply not enough deep diving information on social media addiction yet. A lot is unknown and future studies offer much, such as whether dopamine presence is linked to social media use. This is an expanding topic and has much to come in the research world.

References

Aghaii, S. S. H., Kamaly, A., & Esfahani, M. (2012). Meta-Analysis of Individual and Environmental Factors that Influence People’s Addiction Tendencies. International journal of high risk behaviors & addiction, 1(3), 92.

Aghajani, M., Veer, I. M., Van Tol, M. J., Aleman, A., Van Buchem, M. A., Veltman, D. J., … & van der Wee, N. J. (2014). Neuroticism and extraversion are associated with amygdala resting-state functional connectivity. Cognitive, Affective, & Behavioral Neuroscience, 14(2), 836-848.

Blackwell, D., Leaman, C., Tramposch, R., Osborne, C., & Liss, M. (2017). Extraversion, neuroticism, attachment style and fear of missing out as predictors of social media use and addiction. Personality and Individual Differences, 116, 69-72.

Britton, J. C., Ho, S. H., Taylor, S. F., & Liberzon, I. (2007). Neuroticism associated with neural activation patterns to positive stimuli. Psychiatry Research: Neuroimaging, 156(3), 263-267.

Cremers, H. R., Demenescu, L. R., Aleman, A., Renken, R., van Tol, M. J., van der Wee, N. J., … & Roelofs, K. (2010). Neuroticism modulates amygdala—prefrontal connectivity in response to negative emotional facial expressions. Neuroimage, 49(1), 963-970.

Franken, I. H., Muris, P., & Georgieva, I. (2006). Gray’s model of personality and addiction. Addictive behaviors, 31(3), 399-403.

Gray, J. A. (1993). Framework for a taxonomy of psychiatric disorder. In S. van Gozen, N. van de Poll, & J. A. Sergeant (Eds.), Emotions: Essays on emotion theory (pp. 29 – 59). New Jersey7 Lawrence Erlbaum Associates Inc

Haas, B. W., Omura, K., Constable, R. T., & Canli, T. (2007). Emotional conflict and neuroticism: personality-dependent activation in the amygdala and subgenual anterior cingulate. Behavioral neuroscience, 121(2), 249.

Kanthamani, B. K., & Rao, K. R. (1973). Personality characteristics of ESP subjects: IV. Neuroticism and ESP. The Journal of Parapsychology, 37(1), 37.

Nagel, M., Watanabe, K., Stringer, S., Posthuma, D., & Van Der Sluis, S. (2018). Item-level analyses reveal genetic heterogeneity in neuroticism. Nature communications, 9(1), 905.

Seidman, G. (2013). Self-presentation and belonging on Facebook: How personality influences social media use and motivations. Personality and individual differences, 54(3), 402-407.

Shiota, M. N., & Kalat, J. W. (2018). Emotion. New York, NY: Oxford University Press.

Smith, D. J. (2016) Genome-wide analysis in UK Biobank identifies four loci associated with mood instability and genetic correlation with MDD, anxiety disorder and schizophrenia.

Turel, O., & Gil-Or, O. (2018). Neuroticism magnifies the detrimental association between social media addiction symptoms and wellbeing in women, but not in men: a three-way moderation model. Psychiatric Quarterly, 89(3), 605-619.

Van Os, J., Park, S. B. G., & Jones, P. B. (2001). Neuroticism, life events and mental health: evidence for person-environment correlation. The British Journal of Psychiatry, 178(S40), s72-s77.

Wilson, K., Fornasier, S., & White, K. M. (2010). Psychological predictors of young adults’ use of social networking sites. Cyberpsychology, behavior, and social networking, 13(2), 173-177.

Griffiths, M. D. (2018). Adolescent social networking: how do social media operators facilitate habitual use?. Education and Health, 36(3), 66-69.

Parkin, S. (2018). Has dopamine got us hooked on tech? The Guardian, March 4. Available at: https://www.theguardian.com/technology/2018/mar/04/hasdopamine-got-us-hooked-on-tech-facebook-apps-addiction#img-1 (Accessed 30th July 2018).

Bienvenu, O. J., & Stein, M. B. (2003). Personality and anxiety disorders: a review. Journal of Personality disorders, 17(2: Special issue), 139-151.

Broadband. (2019). Average Time Spent Daily on Social Media (with 2019 Data). Retrieved from https://www.broadbandsearch.net/blog/average-daily-time-on-social-media

Harvard Health Publishing. (2011, July). How addiction hijacks the brain. Retrieved December 12, 2019, from https://www.health.harvard.edu/newsletter_article/how-addiction-hijacks-the-brain.

American Society of Addiction Medicine. (2019). Definition of Addiction. Retrieved December 12, 2019, from https://www.asam.org/resources/definition-of-addiction.

Riley's Graphics library FrameWork

A cross-platform lightweight single-header very simple-to-use window abstraction library for creating graphics Libraries or simple graphical programs. Written in pure C99.

RGFW is a free multi-platform single-header very simple-to-use window abstraction framework for creating graphics Libraries or simple graphical programs. it is meant to be used as a very small and flexible alternative library to GLFW.

The window backend supports XLib (UNIX), Cocoas (MacOS), webASM (emscripten) and WinAPI (tested on windows XP, 10 and 11, and reactOS)

Windows 95 & 98 have also been tested with RGFW, although results are iffy

Wayland: to compile wayland add (RGFW_WAYLAND=1). Wayland support is very experimental and broken.

The graphics backend supports OpenGL (EGL, software, OSMesa, GLES), Vulkan, DirectX, Metal and software rendering buffers.

RGFW was designed as a backend for RSGL, but it can be used standalone or for other libraries, such as Raylib which uses it as an optional alternative backend.

RGFW is multi-paradigm,

By default RGFW uses a flexible event system, similar to that of SDL, however you can use callbacks if you prefer that method.

This library

1. is single header and portable (written in C99 in mind)

2. is very small compared to other libraries

3. only depends on system API libraries, Winapi, X11, Cocoa

4. lets you create a window with a graphics context (OpenGL, Vulkan or DirectX) and manage the window and its events only with a few function calls

This library does not

1. Handle any rendering for you (other than creating your graphics context)

2. do anything above the bare minimum in terms of functionality

Officially tested Platforms

- Linux

- Raspberry PI OS

- Windows, (XP, Windows 10, 11, ReactOS)

- MacOS, (10.13, 10.14, 14.5) (x86_64)

- HTML5 (webasm / Emscripten)

A list of GUI libraries that can be used with RGFW can be found on the RGFW wiki here

The examples can also run in your browser with emscripten

The examples can be compiled by using make debug, which compiles them in debug mode and then runs them

or make which simply compiles them.

The dx11 example has its own Makefile functions because it is Windows only, those include make DX11 and make debugDX11

You can do CC=compiler to specify a specific compiler

Tested and supported compilers include, gcc, clang, [x86_64 / i686-w64]-w64-mingw32-gcc, cl (linux AND windows)

tcc has also been tested but work on linux only

A basic example can be found in examples/basic, it includes a basic OpenGL example of just about all of RGFW's functionalities.

The event example can be found in examples/events, it shows all the events and the data they send.

The callback example can be found in examples/callbacks, it shows all the events and the data they send, but processed with callbacks instead.

examples/dx11 is a minimalistic example of the use of DirectX with RGFW

examples/gl33 is a minimalistic example of the use of OpenGL 3.3 with RGFW, this example was made by AICDG

examples/gles2 is a minimalistic example of the use of OpenGL ES 2 with RGFW

examples/vk10 is a minimalistic example of the use of Vulkan with RGFW, this example was made by AICDG

It also includes examples/vk10/RGFW_vulkan.h which can be used to create a basic vulkan context for RGFW.

A basic example can be found in examples/basic, it includes a basic OpenGL example of just about all of RGFW's functionalities.

examples/buffer is an example that shows how you can use software rendering with RGFW using RGFW_BUFFER mode which allows you to render directly to the window's draw buffer.

examples/PortableGL is an example that shows how you'd use RGFW with portablegl.h.

examples/first-person-camera is an example that shows how you'd make a game with a first person camera with RGFW

#define RGFW_IMPLEMENTATION

#include "RGFW.h"

u8 icon[4 * 3 * 3] = {0xFF, 0x00, 0x00, 0xFF,    0xFF, 0x00, 0x00, 0xFF,     0xFF, 0x00, 0x00, 0xFF,   0xFF, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0x00, 0xFF, 0xFF, 0xFF, 0x00, 0xFF,     0xFF, 0x00, 0x00, 0xFF, 0xFF, 0x00, 0x00, 0xFF, 0xFF, 0x00, 0x00, 0xFF};

void keyfunc(RGFW_window* win, u32 keycode, char keyName[16], u8 lockState, u8 pressed) {

printf("this is probably early\n");

}

int main() {

RGFW_window* win = RGFW_createWindow("name", RGFW_RECT(500, 500, 500, 500), (u64)RGFW_CENTER);

RGFW_window_setIcon(win, icon, RGFW_AREA(3, 3), 4);

RGFW_setKeyCallback(keyfunc); // you can use callbacks like this if you want

i32 running = 1;

while (running) {

while (RGFW_window_checkEvent(win)) { // or RGFW_window_checkEvents(); if you only want callbacks

if (win->event.type == RGFW_quit || RGFW_isPressed(win, RGFW_Escape)) {

running = 0;

break;

}

if (win->event.type == RGFW_keyPressed) // this is the 'normal' way of handling an event

printf("This is probably late\n");

}

glClearColor(0xFF / 255.0f, 0XFF / 255.0f, 0xFF / 255.0f, 0xFF / 255.0f);

glClear(GL_COLOR_BUFFER_BIT);

RGFW_window_swapBuffers(win);

}

RGFW_window_close(win);

}

linux : gcc main.c -lX11 -lXcursor -lGL

windows : gcc main.c -lopengl32 -lshell32 -lgdi32

macos : gcc main.c -framework Foundation -framework AppKit -framework OpenGL -framework CoreVideo

There is a lot of in-header-documentation, but more documentation can be found at https://colleagueriley.github.io/RGFW/docs/index.html If you wish to build the documentation yourself, there is also a Doxygen file attached.

A list of bindings can be found on the RGFW wiki here

A list of projects that use RGFW can be found on the RGFW wiki here

There is a RGFW wiki page about things you can do if you want to support the development of RGFW here.

A comparison of RGFW and GLFW can be found at on the wiki

RGFW uses the Zlib/libPNG license, this means you can use RGFW freely as long as you do not claim you wrote this software, mark altered versions as such and keep the license included with the header.

Permission is granted to anyone to use this software for any purpose,

including commercial applications, and to alter it and redistribute it

freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not

claim that you wrote the original software. If you use this software

in a product, an acknowledgment in the product documentation would be

appreciated but is not required.

2. Altered source versions must be plainly marked as such, and must not be

misrepresented as being the original software.

3. This notice may not be removed or altered from any source distribution.

Translating my Grandfather’s biograpy - Korny’s Blog

On this page

My grandfather, Dr Kornelis Sietsma was a Dutch Reformed Church minister in wartime Amsterdam. He preached in ways that offended the Nazi occupiers, and they deported him to the Dachau concentration camp, where he died.

This was a fascinating family history to me growing up - I was named after my grandfather, and his heroic attitude and tragic death informed my view of the world.

Now I have kids of my own, and I started thinking about how I could explain their great-grandfather to them. I knew quite a few broad facts, but hadn’t dug into the specific details. Also information about him is scattered all over the place, and is mostly in dutch - there’s a dutch Wikipedia article for example, but no English one. I thought it’d be good to start building something I could put on the sietsma.com website.

One important source I knew of was a short biography, written by a colleague of Dr Sietsma in 1946. It was titled “Een Waarlijk Vrie” which translates as “a Truly Free person” - and I already had translation made by my Uncle Arie in the 1990s. I thought of just putting that online, however, looking at it, it was quite wordy and hard to read. I could have cleaned it up and simplified it, but I didn’t feel confident just altering his translation, I thought it’d be good to cross-check it against the original dutch text.

It also occurred to me that this could be a good use of a modern Large Language Model based AI. I’m rather skeptical about LLMs in many cases, but this seemed one of the areas they might really add value - there’s a lot of data to build on for translations, and in this case I could check the results - both against my Uncle’s translation, but also I have Dutch speaking friends who could double check specific parts.

So I hunted online and found I could buy the book from a second-hand dealer in the Netherlands - a few days later it arrived:

I now needed to turn it into text - a bit of digging and I found that Mac OSX has built in text recognition (OCR) tooling - you can just load a picture into Preview or the Photos app, select some text, and copy it into a document.

So I scanned it in - you can read the Dutch version of the book here though as I don’t read Dutch it may have mistakes! I did have to do a lot of manual work as well - mostly things like handling unusual letter pairs like ij, and long words split at line endings with hyphens. But the Mac OCR tooling did most of the hard work.

And then I went looking for good translation options. The main tools I used were DeepL, a set of translation tools using an LLM based engine, and Claude a more general purpose LLM AI. (There are many others to choose from - I used these as I had friends who recommended them and they both had good reputations).

I then spent some time looking at the versions side-by-side, using Obsidian as it works not just as a note taker, but also is a quite good markdown editor. And slowly I built up my own version from the 3 different sources:

The results were surprisingly good. Not perfect - in some cases Arie’s translation was clearer, in some cases there was disagreement and I had to check with friends who read Dutch about which idiom was correct. In some cases the LLMs just returned literal word-for-word translations which didn’t make much sense.

I also found the LLMs quite inconsistent with place names - in some places a church is “Amsterdam South”, in others “Amsterdam-Zuid” - something a human would probably notice and catch. (I haven’t cleaned all these up - it probably needs another human pass!)

On average Claude did somewhat better than DeepL. I think that’s largely because I could give it a specific prompt, to tell it that the text was 1940s Dutch, which helped it do a better job at translating dated terms and idioms. For reference this was the prompt I used:

I have a biography of my grandfather written in 1946. I need your help to translate it accurately so that his original meaning and intention is preserved. Please take note of the approximate time period as Dutch is likely to have been spoken in an older style than today. I would like it translated into a modern British english style.

Claude also inserted explanations about some key points, which was sometimes useful where there was ambiguity or specific references - I used these explanations to add a lot of footnotes to the final translation.

The main place both LLMs struggled was with idioms and archaic expressions. One example from grandfather’s childhood: (key differences in bold):

Dus naar de borderij om alles te leeren: ploegen, zaaien, wieden, oogsten, melken, mennen — en leeren beteekent hier doen, hard vermoeiend werken. Maar Kees gaf geen kamp. Hij leerde het boerenwerk en gaf zich ook daarin; deed het niet met een hangend hoofd doch als zin plicht.

My uncle translated this as:

Thus it was back to the farm to learn everything like plowing, sowing, weeding, harvesting, milking, winnowing  and learning on a farm means doing hard, tiring work. But Kees gave no quarter. He learned the farm work and gave himself totally, not with a desultory air but as his duty.

but DeepL was a bit different:

So back to the farm to learn everything: ploughing, sowing, weeding,

harvesting, milking, driving - and learning here means doing, hard

tiring work. But Kees did not give a camp. He learned farm work and gave

himself in it; did not do it with a drooping head but as a sense of

duty.

Claude was different again, but with some extra notes:

So [I went] to the farm to learn everything: plowing, sowing, weeding, harvesting, milking, driving [horses or cattle] — and learning here means doing, hard exhausting work. But Kees didn’t give up. He learned the farm work and devoted himself to it; he didn’t do it with a hanging head but as his duty.

1. “Mennen” refers to driving or controlling horses or cattle, a common farm task at that time.

2. “Gaf geen kamp” is an idiomatic expression meaning “didn’t give up” or “persevered.”

3. “Zin plicht” is likely a misspelling or older form of “zijn plicht,” meaning “his duty.”

- (I checked the original and indeed it was an OCR failure, the original book had “zijn plicht”)

It was interesting here which to choose. My uncle in this case had gone for more modern idioms (though it seems ‘winnowing’ was probably a mistake). “Gaf geen kamp” was translated literally as “gave a camp” by DeepL, converted to “didn’t give up” by Claude, and “gave no quarter” by my Uncle. I checked this phrase with a Dutch friend and he confirmed that Claude was probably best here, and it’s an old idiom not in common use these days.

As an Aussie I ended up using “droving” for the first word - it is apparently an English word originally but quite common in Australian history and folk songs.

My final version of this section was:

So it was back to the farm he went, to learn everything: ploughing, sowing, weeding, harvesting, milking, droving - and learning here means doing, hard tiring work. But Kees didn’t give up. He learned farm work and committed himself to it; not grudgingly but with a sense of duty.

As I worked my way through the book I found quite a few more translation differences - but they were mostly small. A few things each of the AIs failed to translate, or translated literally like “give a camp” above, but generally I was impressed how well they did - nothing was outrageously wrong or hallucinated.

I’ll share one more example, which was quite significant given it was a passage from my Grandfather’s final sermon, and was the particular topic the Nazis used to justify his deportation:

De prediker bracht den Christus in het licht in zijn heiligen, principieelen strijd met den Satan; deed ook uitkomen, dat de Satan nog de machtsverzoeking wel toepaste en dat hij zocht te verleiden met de koninkrijken der aarde.

The tenses here are interesting.  My uncle translated this as:

The preacher brought Christ to light in his holy, principial battle with Satan. He made it clear that Satan applied the temptation of power and sought to seduce Him with all the kingdoms on earth.

But Claude (and DeepL) seemed a bit different:

The preacher brought Christ to light in his holy, principled struggle with Satan; also made it clear that Satan still applied the temptation of power and that he sought to seduce with the kingdoms of the earth.

That “still” seems to suggest that this is an ongoing temptation by Satan; rather than something that just happened in biblical times.  And so “he sought to seduce” applies to modern people, not just to Jesus (as implied by the “Him” in the first translation).

I asked Claude to elaborate, it replied:

In the original Dutch text “dat de Satan nog de machtsverzoeking wel toepaste,” the word “nog” (still) combined with the past tense “toepaste” (applied) creates an interesting temporal meaning.

From the context, this appears to be describing Satan’s actions during the time period that the preacher was discussing (likely referring to the Biblical temptation of Christ) and extending to the time when the sermon was being given (1942). The use of “nog” (still) suggests a continuation of this behaviour from the Biblical past up through the time of writing.

Again I checked this with a Dutch speaker, and they replied:

… I wanted to confirm that Claude is correct in their explanation. The original text suggests current and ongoing temptation.

During Dr Sietsma’s interrogation by the Nazis, they asked whether the “temptation of power” could also be found in National Socialism (Nazism) - he said that it could. This was the key point they used to send him to Dachau concentration camp.

I made a few other changes to the text - trying to retain the facts and the message of the original. The text referred to my grandfather occasionally as “Kees” (which is what people called him) but more often “Sietsma” or “Dr Sietsma” - I converted most of these to “Kees” as I felt it made the language more personal and direct for a modern audience.

I also broke up long paragraphs, and added more headings to make navigating the text easier. The book has some very long passages about his sermons and writings, and I suspect many people would like to jump straight to the wartime section! Though the rest provides some great background and context.

I also added maps and images, and quite a few explanatory footnotes, some based on Claude’s notes and some where I thought a phrase might not be obvious to a younger audience, not brought up on terms like “National Socialism”.

My finished translation is at https://sietsma.com/dr_k_sietsma/een_waarlijk_vrie/2024_translation/ - there’s an overview page with the Dutch text and other comments at https://sietsma.com/dr_k_sietsma/een_waarlijk_vrie/.

This was a fascinating exercise - both to learn more about my Grandfather, and also uses of modern AI tools. It reinforces that they can be extremely useful in cases where you can verify the results, and adjust any errors.

This is similar to what I’ve found using Copilot and other AI based software tools - I wouldn’t trust them to write code unsupervised; even when they are ‘correct’ they often produce poor quality code that could easily become a maintenance nightmare. But when you can check the results and adjust by hand, they can be a very helpful tool.

Imagine trying to balance thousands of spinning tops at the same time—each top representing a qubit, the fundamental building block of a quantum computer. Now imagine these tops are so sensitive that even a slight breeze, a tiny vibration, or a quick peek to see if they’re still spinning could make them wobble or fall. That’s the challenge of quantum computing: Qubits are incredibly fragile, and even the process of controlling or measuring them introduces errors.

This is where Quantum Error Correction (QEC) comes in. By combining multiple fragile physical qubits into a more robust logical qubit, QEC allows us to correct errors faster than they accumulate. The goal is to operate below a critical threshold—the point where adding more qubits reduces, rather than increases, errors. That’s precisely what Google Quantum AI has achieved with their recent breakthrough [1].

Google’s Breakthrough Achievement

To grasp the significance of Google’s result, let’s first understand what success in error correction looks like. In classical computers, error-resistant memory is achieved by duplicating bits to detect and correct errors. A method called majority voting is often used, where multiple copies of a bit are compared, and the majority value is taken as the correct bit. In quantum systems, physical qubits are combined to create logical qubits, where errors are corrected by monitoring correlations among qubits instead of directly observing the qubits themselves. It involves redundancy like majority voting, but does not rely on observation but rather entanglement. This indirect approach is crucial because directly measuring a qubit’s state would disrupt its quantum properties. Effective quantum error correction maintains the integrity of logical qubits, even when some physical qubits experience errors, making it essential for scalable quantum computing.

However, this only works if the physical error rate is below a critical threshold. In fact, intuition says that increasing the number of physical qubits that make a logical qubit should allow for better error correction. In truth if each physical qubit is very error-prone, adding qubits makes errors accumulate faster than we can detect and correct them. In other words, quantum error correction works only if each qubit can operate below an error threshold even before any error correction. Having more physical qubits allows to increase the QEC code distance, which is a measure of a quantum code’s ability to detect and correct errors.

By showing logical error decreased by a factor of 2.14 when increasing code distance from five to seven, Google has now demonstrated below-threshold operation using surface codes—a specific type of quantum error correction code.  This reduction in errors (which is exponential with increasing code distance) is the smoking gun proving that their QEC strategy works. With this, Google could show that their logical qubit lasted more than twice as long as their best physical qubit, as shown in Figure 1, demonstrating that logical qubits didn’t just survive—they outperformed physical ones.

A distance-7 surface code on 101 qubits effectively doubled the logical qubit’s lifetime (blue line in Figure 1c) compared to uncorrected physical qubits (green line in Figure 1c). This accomplishment demonstrates that error-corrected qubits can preserve coherence for longer periods, which is crucial for running extended quantum algorithms and computations.

A Control Engineering Perspective: How Google Made It Work.

The experiment wasn’t just a test of surface codes—it was a carefully orchestrated feat of engineering and control. The control system had to deliver flawless precision on multiple fronts—synchronization, frequency control, measurement fidelity, real-time decoding, and stability—over many hours of operation. Let’s stop for a second to talk about some of these interesting challenges.

At the heart of the system was real-time synchronization. Every correction cycle had to complete within 1.1 µs—a narrow window in which the qubits were measured. The precision of this synchronization was critical to preventing errors from accumulating and destabilizing the computation. Achieving this required precise coordination of control signals across the qubit array, ensuring that every gate operation, measurement, was perfectly aligned.

One of the most important components was real-time decoding. Decoding refers to the process of analyzing measurement data to determine where and how errors have occurred. To use logical qubits to perform universal quantum computation, certain gates called non-Clifford gates must be applied. Applying these gates, required correcting errors in real-time based on the real-time decoding. In Google’s system, the real-time decoder maintained a constant latency of about 63 µs while operating over one million correction cycles. Namely, the real-time error correction pipeline could process the measurements fast enough to avoid congestion. This rapid decoding process was essential, as any delay could allow errors to propagate and accumulate, potentially destabilizing the logical qubits.

The experiment also demanded high-fidelity gate operations. Errors in qubit gates could easily propagate through the system, jeopardizing the stability of the logical qubit. Google achieved single-qubit gate errors below 0.1% and two-qubit CZ gate errors around 0.3%—thresholds essential to keeping logical qubits stable over time. For this goal, high performance of the control electronics is paramount, as fidelity can directly be impaired by errors of control pulses. These fidelities are especially critical when scaling surface codes, where even minor gate errors could degrade the effectiveness of error correction.

As quantum computers scale to more qubits and longer computations, these and more control requirements will only grow more demanding, making the development of advanced control hardware essential for the future of fault-tolerant quantum computing.

Out of the requirements above, real-time decoding, in particular, is fundamental for any scalable quantum computing system, as it provides the rapid response required to keep quantum information stable.

A deeper dive into real-time decoding

Google’s work highlights that the feasibility of the decoding depends on the decoder latency and throughput, as one of the most important pieces for running QEC below threshold.

Decoding is a classical compute task, and it can be done effectively on various classical architectures, such as FPGAs or GPUs. However, there is usually a trade-off between computational resources. FPGAs for example, are limited in computing power, but operate deterministically and in strict timing, making them suitable to manage the qubit control and measurement tasks as well as perform dedicated classical computations with low latency. On the other hand, CPUs or GPUs might have increased latency but enable far more advanced and larger computation. At Quantum Machines, we partnered with NVIDIA to deliver a unique platform, called DGX Quantum, that provides a unique combination of ultra-low controller-decoder latency, high-performance computational power, and flexible SW programmability. Our platform, which includes a less than 4 µs communication between our controller, OPX1000 and the CPU/GPU, allows to easily program and execute QEC workflows, including real-time decoding such as Google’s decoding. The SW programmability allows iterating over the decoding algorithm and scheme very quickly. A feature we believe is key for faster progress towards scalable and effective QEC. The truth is that a lot more experimentation and benchmarking is needed to learn what decoders to use, which classical resources optimize performance and meet requirements and how to design systems that can eventually run QEC on a much larger scale. What we know so far is that the latency of decoders should be less than 10 µs for QEC schemes to converge. Watch our CEO Itamar Sivan explaining this further with the example of Shor’s algorithm for factorizing the number 21.

DGX-quantum is already live, showcasing less than 4 µs round-trip latency between controller and GPU. To learn more, watch the IEEE QCE 2024 tutorial below, on DGX-quantum, co-authored by QM and NVIDIA.

So, what’s next?

Google’s demonstration of below-threshold quantum error correction marks a milestone towards fault-tolerant quantum computing. By demonstrating that logical qubits can outperform physical qubits and showing that errors can be corrected faster than they accumulate, they’ve paved the way for scalable quantum processors.

However, this is just the beginning. In the future, to perform universal quantum computation with error corrected logical qubits, the full feedback loop must be closed, meaning that the control system needs to make decisions in real-time based on the decoder computation. Future developments will require faster decoders, better error mitigation strategies, automated calibrations embedded within quantum programs to stabilize parameters, and control hardware that tightly integrates and manages classical and quantum workflows.

Google’s achievement signifies a substantial step toward fault-tolerant quantum computing. By demonstrating that logical error rates can be exponentially suppressed through the use of surface codes, the work provides a scalable and practical pathway to reliable quantum computing. As code distance increases, errors decrease at a rapid rate, setting the stage for quantum processors capable of handling complex operations with higher fidelity. Furthermore, this implementation of fast decoding represents a fundamental advancement in QEC. This technique allows for correction of errors faster than their propagation, minimizing the chance for errors to propagate through the quantum system.

Quantum Error Correction and the Vision for Fault Tolerance

Real-time, low-latency feedback loops are going to be an essential element of future fault tolerant quantum devices, to ensure that errors are corrected faster than they accumulate. This principle resonates across the broader quantum computing community, where rapid and robust control mechanisms are viewed as the key to achieving large-scale, reliable quantum operations.

By focusing on low-latency, high-fidelity feedback and decoding, the broader quantum technology field is advancing toward the shared goal of fault-tolerant quantum computing, just as Google’s milestone achievement shows. The evolution of quantum control systems that support agile error correction and real-time adaptability will continue to play a central role in the pursuit of stable, scalable quantum computing systems that can be deployed in practical applications. And with DGX-quantum, we are just starting this exciting journey, so stay tuned for what’s to come!

Reference

[1] Acharya, Rajeev, et al. “Quantum error correction below the surface code threshold.” arXiv preprint arXiv:2408.13687 (2024).

A 192-channel phased array microphone, with FPGA data acquisition and beamforming/visualization on the GPU. Phased arrays allow for applications not possible with traditional directional microphones, as the directionality can be changed instantly, after the recording is made, or even be focused at hundreds of thousands of points simultaneously in real time.

All designs are open source:

Block diagram

Glamor shot

Hardware

To create a phased array microphone, a large number of microphones needs to be placed in an arrangement with a wide distribution of spacing. For a linear array, exponential spacing between microphones is found to be optimal for broadband signals. To create a 2d array, these symmetrical linear arrays (“arms”) are be placed radially, which allows the central (“hub”) board to be compact. The total cost for the array is approximately $700.

Arms

The length of each arm is dictated by the limits of PCB manufacturing and assembly. These boards were made at JLCPCB, where the maximum length for manufacturing and assembly of 4 layer PCBs was 570mm.

The microphones chosen were the cheapest digital output MEMS microphone (because there are a lot of them!), which were about $0.5. At this bottom of the barrel price range, there is little differentiation in the performance characteristics between different microphones. Most have decent performance up to 10khz and unspecified matching of phase delay and volume.

These microphones output data using pulse density modulation (PDM), which provides a one bit output at a frequency significantly higher than the audible range (up to 4 MHz), with the high sampling rate compensating for quantization noise. These microphones also support latching the data either on the rising or falling edge of the clock (DDR), which allows two microphones to be multiplexed on a single wire, reducing the amount of connections required.

Each arm contains 8 microphones sharing 4 output lines, as well as an output buffer on the clock input line. This ensures the rise times are reasonable, even with hundreds of microphones sharing the same clock signal.

For some reason (likely the low rigidity of the panel and some suboptimal solder paste stencil patterns combined with the LGA microphone footprints), the yields on the arm PCBs are not very good, with only 50% of them working out of the box. The most common fault was the clock line being shorted to either 3V3 or ground, which unfortunately requires trial and error of removing microphones from the PCB until the short is resolved. Next time some series resistors on the clock line would speed this process up a lot, and improving the panelization and paste stencil would likely improve yields so extensive rework isn’t required.

Even with rework, there are still some microphones which produce bogus data. These are just masked out in the code, as there are enough working ones to make up for it (and it’s too much work to remove a bunch of arms to do more rework…)

Hub

An FPGA is used to collect all the data, due to the large number of low latency IOs available combined with the ability to communicate using high speed interfaces (e.g. Gigabit Ethernet). Specifically, the Colorlight i5 card is used, as it has enough IOs, is cheap and readily available, and has two integrated ethernet PHYs (only one is used for this project). The card is originally designed as an ethernet interface for LED panels, but has been fully reverse engineered. About 100 GPIOs are broken out over the DDR2 connector, which is much easier to fan out than the BGA of the original FPGA.

Other than the FPGA, the hub contains some simple power management circuitry, and connectors for the arm boards as well as an Ethernet connector with integrated magnetics.

Mechanical Design

The arms are attached with M3 screws to the hub using PCB mounted standoffs/nuts , which conveniently can be assembled with SMD processes. The connections from each arm to the hub is made with 8 pin, 2mm pitch connectors.

The original mechanical design consists of slots on the arm PCBs which interlock with circumferential structural PCBs, however the low torsional rigidity of the arms means the whole structure deformed too easily.

The final mechanical design consists of pieces of laser cut 1/4th inch MDF around the outer edge of the array, with each arm attached to the MDF with some zip ties.

As the microphone array is mounted on the wall (which is very susceptible to reflections), a layer of acoustic foam is used to attenuate the reflections to make calibration easier.

Gateware

The main objective for the gateware is to reliably transmit the raw acquired data losslessly to the computer for further processing, while keeping it as simple as possible. Performing decimation and filtering on the FPGA would reduce the data rate, but sending the raw PDM data is achievable with Gigabit Ethernet. This reduces the complexity of the FPGA code and allowing faster iteration. Compiling is much faster than place and route, and it’s much easier to use a debugger in code than in gateware!

There are three major components to the gateware, a module for interfacing with the PDM interfaces, a module for creating fixed size packets from those readings, and a UDP streamer to write the packets to the Ethernet interface.

PDM Interface

The PDM input module is a relatively simple piece of logic, which divides the 50 MHz system clock by a factor of 16 to output a 3.125MHz PDM clock, latches all 96 of the input pins after each clock edge, and then shifts out 32 bits of the data on each clock cycle. Each chunk of 192 bits is has a header added which is a 32 bit incrementing integer.

The PDM interface receives data at a rate of 3.125Mhz * 96 (input pins) * 2 (DDR), which is 600Mbps. With the header, the data rate output from this module is 700Mbps, or approximately 40% utilization of the 32 bit output data path.

Packetizer

The packetizer is essentially a FIFO buffer with a special interface on the input. A standard FIFO marks the output as available whenever there is at least one item in the queue, but this would lead to smaller packets than requested as the ethernet interface operates faster than the PDM output (leading to buffer underruns). Thus, the packetizer waits until there is at least a full packet worth of data in the queue before starting a packet, which ensures constant sized packets.

48 PDM output blocks at 224 bits (192 bits of data with a 32 bit header) are placed into each packet, which totals 1344 bytes of data per packet, plus a 20 byte IPv4 header and an 8 byte UDP header, at a rate of approximately 65k pps.

This leads to a wire rate of 715 Mbps, or about 70% utilization of Gigabit Ethernet.

UDP Streamer

The LiteEth project made this module very easy, as it abstracts out the lower level complexities of UDP and IP encapsulation, ARP tables and the like, and provides a convenient interface for simply hooking up a FIFO to a UDP stream. Occasionally there is some latency, but there is enough slack in the bus and buffer in the packetizer FIFO to absorb any hiccups.

Utilization and Timing

The FPGA on the Colorlight i5 is a LFE5U-25F-6BG381C, which has 25k LUTs. The design is placed and routed with the open source Project Trellis toolchain. By keeping the gateware very simple, the utilization on the device is quite low, and there is lots of room for additional functionality.

Info: Device utilisation:

Info:                 DP16KD:    16/   56    28%

Info:                EHXPLLL:     1/    2    50%

Info:             TRELLIS_FF:  1950/24288     8%

Info:           TRELLIS_COMB:  3701/24288    15%

Info:           TRELLIS_RAMW:    49/ 3036     1%

Info: Max frequency for clock                   '$glbnet$crg_clkout': 73.17 MHz (PASS at 50.00 MHz)

Warning: Max frequency for clock '$glbnet$eth_clocks1_rx$TRELLIS_IO_IN': 124.07 MHz (FAIL at 125.00 MHz)

(Timing violations on eth rx clock is due to false positive from gray counter in liteeth)

Software

CIC Filter

Each microphone produces a 1 bit signal at 3.125Mhz, and needs to be reduced to a more reasonable sample rate and bit depth for further processing. This is done very efficiently with a CIC filter, which only requires a few arithmetic operations to process each sample. For understanding more about CIC filters, this series of blog posts from Tom Verbeure provides an excellent introduction. Following the nice graphs from there, I decided on a 4 stage, 16x decimation CIC filter which reduced the sample rate to a much reasonable 195kHz, at 32 bits.

To ingest the data at 3.125Mhz, the filter must be able to process each set of samples in 320ns. A naive implementation in Rust wasn’t fast enough on a single core, but an implementation with some less abstraction (and a hence some more autovectorization) got there, and is what was used at the end. I also experimented with a version using SIMD intrinsics which was much faster, but ended up running into alignment issues when using it in together with other code.

Even with close to a billion bits per second of data to process, a single CPU core can do quite a few operations on each individual bit!

test cic::bench_cic       ... bench: 574 ns/iter (+/- 79) = 41 MB/s

test cic::bench_fast_cic  ... bench: 181 ns/iter (+/- 24) = 132 MB/s

test cic::bench_simd_cic  ... bench:  36 ns/iter (+/- 0)  = 666 MB/s

Calibration

To perform array calibration, a speaker playing white noise is moved around the room in front of the array. An FFT based cross correlation is performed between all pairs of microphones to compute relative delays.

A cross correlation can be performed by computing the FFT of both signals (cached and computed once for each signal), and then computing the inverse FFT of the complex multiplication of the two. This is quite compute intensive, as there are over 18 thousand pairs! For the window sizes used of 16-64k, the FFTs are memory bound, and thus the IFFT and peak finding is fused to avoid writing the results to memory, which results in a 15x speedup. On a 7950X, this process runs in realtime.

Then the positions of the source at each timestep and the positions of each microphone is optimized using gradient descent (when you know PyTorch, all optimization problems look like gradient descent…). The loss function tries to minimize the difference between the measured correlations and the ideal correlations, while trying to minimize the deviation of the microphone positions from the initial positions as well as the jerk of the source trajectory.

As part of the calibration, the speed of sound is also a parameter which is optimized to obtain the best model of the system, which allows this whole procedure to act as a ridiculously overengineered thermometer.

After a few hundred iterations, it converges to a reasonable solution for both the source positions and the microphone positions, as well as constants such as the speed of sound. Fortunately this problem vectorizes well for GPU, and converges in a few seconds.

The final mean position error is on the order of 1mm, and is able to correct for large scale systematic distortions such as concavity from the lack of structural rigidity. The largest position error between the calibrated positions and the designed positions is on the order of 5mm, which is enough to introduce significant phase errors to high frequency sound if uncorrected, although perhaps not strictly necessary (10khz sound has a wavelength of ~3.4cm).

Beamforming

Beamforming is how the raw microphone inputs are processed to produce directional responses. The simplest method of beamforming is delay-and-sum (DAS), where each signal is delayed according to its distance from the source. This is the type of beamforming implemented for this process, with the beamforming happening in the frequency domain.

In the frequency domain, a delay can be implemented by the complex multiplication of the signal with a linear phase term proportional to the delay required, which also nicely handles delays which are not integer multiples of the sampling period.

Multiple nested subarrays of the original array are used for different frequency ranges. This reduces the processing required for beamforming, as each frequency does not need to be beamformed with all the microphone. This also ensures that the beamforming gains of all the frequencies are matched.

Two different types of beamforming visualizations are implemented, a 3d near field beamformer and a 2d far field beamformer. When the audio source is far away, the wavefront is essentially a flat plane, and how far away the source is does not meaningfully change the signals at the array. On the other hand, if the source is close to the array, the wavefront will have a significant curvature which allows the 3d location of the source to be determined.

The beamformer is implemented as a Triton kernel, a Python DSL which compiles to run on Nvidia GPUs. When beamforming to hundreds of thousands of points, the massive parallelism provided by GPUs allows for results to be produced in real time. Some current limitations with the Triton language around support to indexing with shared memory arrays lead to slightly suboptimal performance, but writing CUDA C++ doesn’t seem very fun…

Near Field 3D Beamforming

Near field 3D beamforming is performed a 5cm voxel grid with size 64x64x64. An update rate of 12hz is achieved on a RTX 4090 with higher update rates limited by overhead of suboptimal CPU-GPU synchronization with the smaller work units. The voxel grid is then visualized using VisPy, a high performance data visualization library which uses OpenGL. Modern games have millions of polygons, so rendering a quarter million semi-transparent voxels at interactive framerates is no issue.

A quick demo of the voxel visualization below, note the reflection off the roof!

Far Field 2D Beamforming

Far field beamforming works similarly, but can be performed in higher resolution as there is no depth dimension required. A 512x512 pixel grid is used, and the same 12hz update rate achieved. (The far field beamforming uses an approximation of just putting the points far away instead of actually assuming planar wavefront due to laziness…)

A demo of the 2d visualization here, but it’s not very exciting due to the poor acoustic environment of my room around the array, with lots of reflections and multipath.

Directional Audio

The previous two beamforming implementations compute the energy of sound from each location, but never materializes the beamformed audio in memory. A time domain delay and sum beamformer is implemented to allow for directional audio recording. It takes a 3D coordinate relative from array center and outputs audio samples. An interesting aspect about this beamformer is that it is differentiable with regard to the location from the output. This means the location of the audio sources can be optimized based on some differentiable loss function (like neural network), which might allow for some interesting applications such as using a forced alignment model of a multi-party transcript to determine the physical location of each speaker.

A speaker playing some audio is placed in front of the array, with another speaker placed approximately 45 degrees away at the same distance from array center, playing white noise. The effectiveness of the beamforming can be demonstrated by comparing the raw audio from a single microphone with the output from the beamforming.

Raw audio from a single microphone:

Beamformed audio:

Recording

As the data from the microphone array is just UDP packets, it can be recorded with tools like tcpdump, and the packet capture file can be read to reinject the packets back into the listener. All the programs in the previous sections are designed to work at real time, but can also work on recorded data using this process.

The tradeoff with this recording implementation is that the output data rate is quite high (due to faithfully recording everything, even the quantization noise). At 87.5 MBps, a 1-hour recording would be 315 GB! A more optimized implementation would do some compression, and do the recording after the CIC filter at a lower sample rate.

Next Steps

I consider this project essentially complete, and don’t plan to work on it any further for the foreseeable future, but there are still lots of possible cool extensions if you’d like to build one!

- Using more advanced beamforming algorithms (DAMAS etc.)

- Better GUI to combine all existing functions (e.g. See where sound is coming from, and record audio from there)

- Combine differentiable beamforming and neural models (e.g. forced alignment example mentioned above)

Rendering “modern” Winamp skins in the browser

Nov 19, 2024

Discussed on Hacker News

TL;DR several years ago I got a proof of concept working where I was able to render highly interactive “modern” Winamp skins in the browser by reverse engineering Maki byte code and implementing an interpreter for it in JavaScript. You can try the proof of concept in your browser.

One of the most rewarding projects I’ve worked on was Webamp. Seeing classic Winamp skins come to life in my browser via code I wrote was intoxicating and eventually inspired me to create the Winamp Skin Museum. But Webamp “just” implements “classic” Winamp skins, which were basically a glorified set of sprite sheets. They could change the appearance of the player but not the layout, and they could not add any custom interactions.

However, after skipping Winamp 4, Winamp 5 introduced a new, dramatically more powerful, skinning engine. The new engine was powered by XML files describing the UI, which was made interactive via skinner defined scripts written in a bespoke language called MAKI (”Make a killer interface”). Together, XML and Maki worked much like HTML and JavaScript. They enabled “skinners” to create highly dynamic UIs. This included interfaces with custom animations, interactive elements, and more.

After tackling classic Winamp skins it was only natural that I should be curious about modern skins, and I was! Could I get modern skins to run in the browser?

After reading up on how these skins were implemented, I learned that the modern skins were distributed as .zip files with the extension .wal that consisted of .xml and .maki files along with images. The .maki scripting files contained a custom compiled bytecode. Some skins also included the source .m files, but not all. The skin authors had been required to compile their skins before uploading them. If I was going to render these skins in the browser, I was going to need to understand the bytecode.

Reverse engineering Maki

At this point I was new to the concepts of byte code, interpreters and reverse engineering, so I needed some help. Luckily, I found an ancient Maki dissembler written by Ralf Engels. This Perl script would take a Maki byte code file and try to construct a source file from it. The intended audience was skinners who wanted to learn from an existing skin’s script which was not distributed with its source code. Aside: The tool’s page has an interesting meditation on the ethics of enabling people to see code that skinners had potentially intentionally tried to hide from others.

Since the Perl code had to understand the semantics of the byte code in order to produce the equivalent source code, I was able to read the Perl code and (slowly!) build my own JavaScript parser capable of converting Maki byte code files into a structured representation. By leveraging a large collection of real .maki files extracted from downloaded skins, I was able to fuzz my implementation and rattle out many bugs.

As a resource to any fellow traveler who decides to go down this same path, I attempted to document all my findings in this file: maki-bytecode.md

Crafting an interpreter

With a structured version of the byte code in hand, I was able to start work on an interpreter. As a newbie to this type of programming I leaned into a “learn by doing” approach. To any reader interested in reading this type of work, I highly recommend Bob Nystrom’s Crafting Interpreters.

Partially because the language is a bit quirky, and mostly because I had no idea what I was doing, I spent a fair bit of time getting hung up on things like:

- How return pointers worked (do they go on the stack? Is there some other return stack?)

- Some mysterious byte codes which the decompiler implied had to do with “stack protection”

- How to model both scalars and complex objects on the stack

Each of these was an interesting puzzle to solve! At one point I even tried disassembling Winamp itself using Ghidra, and while I was able to locate the main interpreter loop, my C++/disassembly chops were not sufficient for this to provide much insight.

But, with enough iteration, and enough test cases (again mined from my collection of real skins) I was able to get it basically working!

You can see the current version of the interpreter here on GitHub.

The standard library and the DOM (equivalent)

Having an interpreter was actually just the beginning. Just like having a JavaScript engine is not sufficient to build a browser, I needed to figure out how to parse the accompanying XML files, bind them to the scripts in the .maki files and also implement all the “standard library” of Maki. This included things from basic utility functions, all the way up to the various classes that modeled all the different types of UI objects. On the order of 65 classes with many methods each (you can find a full list here).

Basically each of these classes needed to be implemented and define some mapping/binding from its properties and methods to an equivilent DOM representation. I took a pragmatic approach. I picked the simplest skin I could find and started implementing the classes and methods needed just to render that one skin. Slowly but surely I was able to get the one skin rendering! After that first skin, I pick another small skin and over time I had a small handful working and then dozens!

CornerAmp_Redux.wal, the first skin I got working in Webamp Modern

But this is eventually where I lost steam. The API surface was just to large for me to complete with my available time, and even figuring out what the expected behavior of any class/method/property was required hours of manual trial and error in Winamp. But more importantly, I never found a satisfactory way to connect these nested objects to the DOM that was scalable to implement reliably, performant, didn’t leak, and preserved the subtle difference to how the DOM and Maki worked. I suspect a way exists, I just wasn’t able to find it.

A hero comes along

Despite the project basically sitting on ice, x2nie appeared in our Discord one day and wanted to drive the project forward. However, his style was dramatically different than my own. Sprawling ambitious PRs focusing on getting things to “work” instead of carefully considered incremental improvements, focusing on detailed parity with Winamp and elegant architecture on the JavaScript side.

I was stuck in an awkward position of holding up PRs because they were time consuming to review and I often didn’t find the approaches taken to be satisfactory. At the same time, I didn’t have the time or brain space to come up with satisfactory solutions to help unblock him. After all, it was my inability to come up with satisfactory solutions to these hard problems that had lead me to stall out on the project in the first place!

In the end, I had to admit that my approach was stalled out and his approach, while different than my own, had forward momentum. I opted to just try to get out of his way and “let him cook”.

I revived the progress dashboard I had made which introspected the implementation to derive an always-up-to-date progress report, and just stamped his diffs as they poured in.

In the end, he made considerable progress, getting many additional features working, but the project still struggled to feel robust or complete. Eventually his attention moved on to other things, but I’m still grateful for the energy he brought to the project!

Conclusion, for now?

I still don’t have a clear idea how to structure the JS code to make it scalable to fill in all the blanks needed while being largely “correct”. And, while more features work now than when I last actively worked on it, the code is likely harder for me to get into the shape I envision. Mostly because it’s no-longer code I wrote.

That said, several things have changed in the intervening years. LLMs have made highly repetitive/derivative programming tasks easier to scale, and the source code for Winamp has been released as “open”, so in theory I could go look at the actual source code and get more authoritative answers to what the expected Winamp behavior should be, and maybe have a higher likelihood of being able to cover all the ground needed to get a fully working version.

Unfortunately, the license of the released Winamp code is not actually permissive and they’ve actually pulled the code from GitHub. At this point, it actually feels legally more risky to build derivative works than it did before the source was “opened up”.

I still hold out hope that I’ll be motivated at some point in the future to come back to the project and have some epiphany. But in the mean time, I’m very happy to have seen the project come this far!

Give it a try! https://webamp.org/modern/

Gallery

I’ll end with a collection of screenshots showing some of the interesting skins that Webamp Modern is capable of rendering. I’d encourage you to click into the skins to try them in your browser and interact with them to see the animations and explore all the little drawers and tabs.

https://webamp.org/modern/

https://webamp.org/modern/?skin=https://r2.webampskins.org/skins/c2273648295a986350f0e2007b705e85.wal

https://webamp.org/modern/?skin=https://r2.webampskins.org/skins/84be4029fa8dd4305b3eee70c648749b.wal

https://webamp.org/modern/?skin=https://r2.webampskins.org/skins/00bf47f38660c04f89c3abe06eacd5af.wal

https://webamp.org/modern/?skin=https://r2.webampskins.org/skins/97a759e2f0261eb0b7c65452d70318d0.wal

https://webamp.org/modern/?skin=https://r2.webampskins.org/skins/026d840ca4bebf678704f460f740790b.wal

https://webamp.org/modern/?skin=https://r2.webampskins.org/skins/2f2d4a3b9aff93ed9d1a240597c298c6.wal

Amazon to invest another $4 billion in Anthropic, OpenAI's biggest rival

Amazon on Friday announced it would invest an additional $4 billion in Anthropic, the artificial intelligence startup founded by ex-OpenAI research executives.

The new funding brings the tech giant's total investment to $8 billion, though Amazon will retain its position as a minority investor, according to Anthropic, the San Francisco-based company behind the Claude chatbot and AI model.

Amazon Web Services will also become Anthropic's "primary cloud and training partner," according to a blog post. From now on, Anthropic will use AWS Trainium and Inferentia chips to train and deploy its largest AI models.

Anthropic is the company behind Claude — one of the chatbots that, like OpenAI's ChatGPT and Google's Gemini, has exploded in popularity. Startups like Anthropic and OpenAI, alongside tech giants such as Google, Amazon, Microsoft and Meta, are all part of a generative AI arms race to ensure they don't fall behind in a market predicted to top $1 trillion in revenue within a decade.

Some, like Microsoft and Amazon, are backing generative AI startups with hefty investments as well as working on in-house generative AI.

The partnership announced Friday will also allow AWS customers "early access" to an Anthropic feature: the ability for an AWS customer to do fine-tuning with their own data on Anthropic's Claude. It's a unique benefit for AWS customers, according to a company blog post.

In March, Amazon's $2.75 billion investment in Anthropic was the company's largest outside investment in its three-decade history. The companies announced an initial $1.25 billion investment in September 2023.

Amazon does not have a seat on Anthropic's board.

News of Amazon's additional investment comes one month after Anthropic announced a significant milestone for the company: AI agents that can use a computer to complete complex tasks like a human would.

Anthropic's new Computer Use capability, part of its two newest AI models, allows its tech to interpret what's on a computer screen, select buttons, enter text, navigate websites, and execute tasks through any software and real-time internet browsing.

The tool can "use computers in basically the same way that we do," Jared Kaplan, Anthropic's chief science officer, told CNBC in an interview last month, adding it can do tasks with "tens or even hundreds of steps."

Amazon had early access to the tool, Anthropic told CNBC at the time, and early customers and beta testers included Asana, Canva and Notion. The company had been working on the tool since early this year, according to Kaplan.

In September, Anthropic rolled out Claude Enterprise, its biggest new product since its chatbot's debut, designed for businesses looking to integrate Anthropic's AI. In June, the company debuted its more powerful AI model, Claude 3.5 Sonnet, and in May, it rolled out its "Team" plan for smaller businesses.

Last year, Google committed to invest $2 billion in Anthropic, after previously confirming it had taken a 10% stake in the startup alongside a large cloud contract between the two companies.

The Deceptively Asymmetric Unit Sphere

Previously, we talked about [Pinhole Obsession](https://www.tangramvision.com/blog/camera-modeling-pinhole-obsession) and the downsides of assuming a default pinhole model. We presented an alternative formulation where rays are modeled on the unit sphere \\(\mathcal{S}^2\\) instead of the normalized image plane \\(\mathbb{T}^2\\).

We’ve also previously written posts about how many problems in robotics can be formulated as [an optimization](https://www.tangramvision.com/blog/introduction-to-optimization-theory). At a high level, many continuous optimization algorithms perform the following steps

- Compute a quantity to minimize — error, misalignment, risk, loss, etc.

- Compute a direction along which the quantity is locally reduced

- Move the parameters in the quantity-reducing direction

- Repeat until the problem converges

Many continuous optimization problems are modeled on [vector spaces](https://en.wikipedia.org/wiki/Vector_space) and “moving” the parameters is simply vector addition. However, our prescription of using the unit sphere to model rays is obviously **not** a linear space! Therefore, we must start our discussion by describing how to travel on the sphere.

## Traveling on the Sphere

Generally, when we *travel* we want to do so in the most efficient manner. Undoubtedly, you’ve heard the expression “get from point a to b,” perhaps also with the implication “as fast as possible.” Therefore, we’ll want to minimize the distance that we travel or equivalently travel along the shortest path. In Differential Geometry, we call these “shortest paths” *geodesics.*

An alternate way to view a *geodesic* is by starting at a point \\(p\\) on the manifold and traveling in the direction of some vector \\(v\\), carefully minimizing the distance at each increment. However, minimizing the distance at each step may mean we have to *change direction* along our shortest path.

“Changing direction” and “traveling in the shortest path” may seem to counter one another. After all, we all know that the shortest path between two points is a line. However, imagine we are flying in a plane starting at the equator and a heading of 45°. If our goal is to travel *as far as possible* from our starting point, how should we chart our course?

> As usual, ignore air resistance, the jet stream and assume a spherical earth. This is a thought exercise, not pilot’s school.

A simple suggestion would be to maintain a heading of 45° during our travel, like we would locally: travel in a constant direction. However, if we maintain a fixed heading indefinitely, we will end up close to the north pole! The line traced by a path of constant heading is known as a [Rhumb Line](https://en.wikipedia.org/wiki/Rhumb_line) and, as pictured, is certainly not the shortest distance between two points on a sphere.

![Loxodrome.png](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f6319ebc7cd01b693200a_Loxodrome.png)

*PC: [https://en.wikipedia.org/wiki/Rhumb_line#/media/File:Loxodrome.png](https://en.wikipedia.org/wiki/Rhumb_line#/media/File:Loxodrome.png)*

Alternatively, to truly travel the *farthest distance,* we travel along the [great circle](https://en.wikipedia.org/wiki/Great-circle_distance) until the aircraft runs out of fuel. Here it’s more obvious that the geodesic’s heading is continuously changing. One implication of this is that if we start at one point along the geodesic with a vector \\(v\\), we may not end up in the same location as if we start at a different point along the geodesic with the same vector \\(v\\).

![Illustration_of_great-circle_distance.svg](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f8b9f2d9bfdd0a7e52425_Illustration_of_great-circle_distance.png)

*PC: [https://en.wikipedia.org/wiki/Great-circle_distance#/media/File:Illustration_of_great-circle_distance.svg](https://en.wikipedia.org/wiki/Great-circle_distance#/media/File:Illustration_of_great-circle_distance.svg)*

In Differential Geometry, the operator that traces the geodesic is known as “the exponential map.” For the reasons listed above, it’s defined as a local operator originating at a point \\(p\\) and traveling in a direction \\(v\\) i.e.

$$

\text{Exp}_p(v)

$$

Similarly, the operator that computes the direction and distance between two points on the manifold (the manifold’s logarithm) is defined as a local operator \\(\text{Log}_p(q)\\).

So to perform optimization on the unit sphere, we

- Compute the quantity to minimize — error, misalignment, risk, loss, etc.

- Compute a direction \\(v\\) along which the quantity can be locally reduced

- Move the parameters along the great circle using the exponential map \\(\text{Exp}_p(v)\\)

- Repeat until the problem converges

This begs the question: how do we compute this direction \\(v\\)? What does it really mean?

---

## The Briefest Introduction to Differential Geometry

> ⚠️ Many maths below! But they’re cool maths, we promise.

To explain what a direction \\(v\\) means on the sphere and to explain why working with \\(\mathcal{S}^2\\) is difficult, we have to briefly touch on one of my favorite subjects: [Differential Geometry](https://en.wikipedia.org/wiki/Differential_geometry). This post will dive deep into the motivations behind why we use differential geometry in computer vision, and what advantages it brings. Much of this may seem like a tangent (no pun intended), but don’t worry: we’ll bring it all back to our Pinhole Obsession and optimization at the end!

> 💡 Obviously, we cannot cover the entirety of Differential Geometry in one blog post. There are many great resources to learn Differential Geometry. Some of our favorites are:

> - [Introduction to Smooth Manifolds](https://link.springer.com/book/10.1007/978-1-4419-9982-5)

> - [A Micro Lie Theory for State Estimation in Robotics](https://arxiv.org/pdf/1812.01537)

> - [Differential Geometry and Lie Groups](https://link.springer.com/book/10.1007/978-3-030-46040-2)

Historically, differential geometry arose from the study of “curves and surfaces.” As traditionally taught in multi-variable calculus, a curve is a mapping \\(\gamma: \mathbb{R} \to \mathbb{R}^3\\) and likewise, a surface is a mapping \\(\sigma: \mathbb{R}^2 \to \mathbb{R}^3\\).

Differential geometry generalizes these “curves and surfaces” to arbitrary dimensions. These generalized “curves and surfaces” are known as *smooth manifolds*. In this post, we’ll skip the rigorous definition of a smooth manifold and simply define it as

> *Smooth Manifold:* An N-dimensional space without corners, edges or self-intersections.

This smoothness (called continuity) allows us to perform optimization “on the manifold” using our standard calculus techniques. We can compute errors and also compute the directions along which we can reduce those errors.

## Intrinsic vs Extrinsic Manifolds

To assist intuition, we’ve leaned on prior knowledge of multi-variable calculus of curves and surfaces. At the undergraduate level, curves and surfaces are presented as [embedded entities](https://en.wikipedia.org/wiki/Nash_embedding_theorems) living in a higher-dimensional ambient space e.g. 3D Euclidean Space. However, this ambient space isn’t *strictly* needed for the development of differential geometry. In their search for a minimal set of axioms, differential geometers have taken great care to remove this dependence on an ambient space; they describe *smooth manifolds* as objects existing in their own right. Thus, there are two ways of thinking about differential geometry:

- *Extrinsic*: manifolds as embedded objects in space

- *Intrinsic*: manifolds are entities in their own right

In this manner, the error-minimizing direction we are searching for can either be viewed as *extrinsic* (embedded in ambient space) or it can be viewed as *intrinsic* (living alongside the manifold).

As engineers, the decision to use the intrinsic view of a manifold vs. the extrinsic view mostly impacts representation and computation. Thus, for a specific manifold, we choose the representation that’s computationally easiest to work with. Although the remainder of this post only briefly touches on the representation of manifolds, it’s **always** necessary to determine what representation is being used. The decision of an intrinsic or extrinsic representation will change how computation is performed on the manifold. Regardless of the choice of *intrinsic* or *extrinsic* representation, we often “bundle” the representation of the manifold with the representation directions on that manifold.

> ⚠️ The concepts of *intrinsic* and *extrinsic* used in differential geometry are unrelated to intrinsics (interior orientation) and extrinsics (exterior orientation) of cameras. This is simply an unfortunate naming collision at the intersection of two fields.

## A Brief Tangent for Tangents

…which brings us to **the** key concept in differential geometry: the [Tangent Bundle](https://en.wikipedia.org/wiki/Tangent_bundle). The Tangent Bundle is the underlying manifold stitched together with the vector spaces that are tangent to the manifold at each point.

To illustrate this, consider the unit circle \\(\mathcal{S}^1\\) below. The blue circle represents the underlying manifold and the red lines represent the one dimensional tangent space at each point on the circle. Together, the points and tangent spaces form the tangent bundle. It is important to note that that vectors in one tangent space should be regarded as distinct and separate from vectors in another tangent space.

![Tangent_bundle.svg](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f63501b983d8e9953910b_image.png)

*PC: [https://commons.wikimedia.org/wiki/File:Tangent_bundle.svg](https://commons.wikimedia.org/wiki/File:Tangent_bundle.svg)*

Now that we have the correct picture in our heads, let’s visualize tangent vectors and the tangent bundle in more detail.

Consider an arbitrary manifold \\(M\\), and furthermore consider a curve on that manifold \\(\gamma: \mathbb{R} \to M\\). This curve \\(\gamma(t)\\) can be thought of “the location on the manifold at time \\(t\\).” Furthermore, let \\(\gamma(0)\\) be some point of interest \\(p \in M\\) on the manifold.

For visualization, consider this wavy circle manifold:

![GenericManifold_ManimCE_v0.18.1.png](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f636093b3a75b96cec282_GenericManifold_ManimCE_v0.18.1.png)

Now, consider the curve \\(\gamma(t)\\) living on this manifold.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GenericManifoldWithCurve.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GenericManifoldWithCurve.mp4">download the video to view it</a></video>

If we take the derivative of this curve, we obtain a pair \\((\gamma(t), \gamma'(t))\\), where the first entry describes the point along the curve and the second entry describes how the curve is changing in time. This can be visualized as a vector “riding along” the curve.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GenericManifoldWithCurveAndTangent.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GenericManifoldWithCurveAndTangent.mp4">download the video to view it</a></video>

The magnitude and sign of this vector can be changed by re-parametrizing the curve as

$$\gamma_{\alpha}(t) = \gamma(\alpha t)$$

Specifically, note that

$$\gamma'_\alpha(0) = \alpha \gamma'(0)$$

In this manner, we can “scale” curves in the same way that we would scale vectors. Similarly, we can derive addition, inverse and zero curves and form a “vector space of curves.”

Thus, we define the tangent space \\(T_pM\\) as the set of all curves that pass through a point of interest \\(p \in M\\). The tangent bundle of a manifold \\(TM\\) is the set of all tangent spaces combined under a *disjoint* union. In this context, a disjoint union simply means we cannot combine vectors in distinct tangent spaces. This is perhaps the most important takeaway: tangent vectors living in distinct tangent spaces cannot be added, combined or related without using some additional manifold structure.

> ⚠️ Here, we have *de facto* assumed that we can take the derivative of a curve on the manifold. This is made more rigorous in most differential geometry books by either using charts on the manifold or an extrinsic view of a manifold facilitated by an embedding. Furthermore, we didn’t really show how to “add curves.” We haven’t done anything *improper* with this explanation, but it certainly is lacking detail! You can refer to the previously mentioned resources to learn more about the mathematically precise definitions of the Tangent Bundle.

If this is your first time studying Differential Geometry, you may be very uncomfortable with the notion of a vector being *attached* in any way to a specific point. After all, vectors are commonly taught as being geometric entities that can be moved anywhere in space, e.g. vector addition is taught using the “head to tail” method. It’s unclear when presenting these geometric explanations if the vectors are “sitting on top of” the underlying manifold or if they are in their own “vector space.”

To see why this “head to tail” method breaks down on manifolds, imagine traveling 1000 miles north and then 1000 miles west. This will take you to a very different location than if you had traveled 1000 miles west and then 1000 miles north. In other words, vectors may point in different directions at different points on the manifold, so we can only really discuss what a vector means *locally*.

The representation we have presented here of \\((\gamma(0), \gamma'(0))\\) , is useful in describing the construction tangent vectors on a manifold and also useful in derivations of certain operators, but it is typically not used in computation. In “the real world,” we typically choose some canonical coordinates for the manifold and a basis for the associated tangent space and think of tangent vectors as \\((p, v) \in M \times \mathbb{R}^n\\).

With this understanding of tangent spaces, the exponential map can be understood more formally as an operator mapping vectors in a tangent space onto manifold.

$$

\text{Exp}_p(v): T_pM \to M

$$

Likewise, the logarithmic map can be understood as an operator mapping points on the manifold back into a tangent space

$$

\text{Log}_p(q): M \times M \to T_pM

$$

> 🚧 Ok, *technically*, more structure is needed for the existence of an exponential map. Since the exponential map minimizes distance along a path, we must have some notion of *length.* The subfield of Differential Geometry with *lengths* and *angles* is called [Riemannian Geometry](https://en.wikipedia.org/wiki/Riemannian_geometry) and is a fascinating subject in its own right. The definition of *lengths* and *angles* for a specific manifold is called the Riemannian Metric. We don’t have the space in this post to develop these ideas fully but we encourage the motivated reader to learn more!

With this understanding, we have the following prescription for optimization on a generic manifold \\(M\\)

- Compute the quantity to minimize — perhaps using our logarithmic map \\(\text{Log}_p(q)\\)

- Compute a direction \\(v\\) in \\(T_pM\\)  along which the quantity is locally reduced

- Move the parameters using the exponential map \\(\text{Exp}_p(v)\\)

- Repeat until the problem converges

At this point, you are probably saying

> “Why isn’t multi-variable optimization taught like this?”

> “I haven’t had to worry about *tangent spaces* and *exponential maps* before.”

> “Adding vectors has always *just worked* for me in the past.”

Yes, BUT. The secret here is that you’ve been exploiting a very special property of euclidean space: *continuous symmetry* of its tangent bundle.

## Continuous Symmetry

Some smooth manifolds exhibit a natural symmetry. By symmetry, we mean a concept that implies more than reflections or rotations of polygons taught in grade school. For instance, reflections and rotations of polygons are examples of *discrete symmetry.* However, in the study of smooth manifolds, we are interested in *continuous* *symmetry* of the Tangent Bundle. This is what makes optimization so simple in euclidean space.

To illustrate, consider the smooth manifold of real numbers of dimension \\(n\\): \\(\mathbb{R}^n\\) or Euclidean Space. We will visualize the plane \\(\mathbb{R}^2\\), but realize that these properties apply in higher dimensions, too.

Let’s place a uniform vector field on \\(\mathbb{R}^n\\), where “the same” vector \\(v\\) is attached to each point in the space \\(p\\). Imagine those points *flowing* along the vectors, as if the vectors describe the current of a river and the points are rafts in that river.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftPoints.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftPoints.mp4">download the video to view it</a></video>

You’ll notice that the points before and after the flow are effectively the same! Since this flow doesn’t *fundamentally* change the total set of points, it’s called an *invariant flow.*

Now let’s flip it: instead of moving each point along the flow of individual vectors, let’s recenter the points on a new origin.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftOrigin.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftOrigin.mp4">download the video to view it</a></video>

Notice that this recentering effectively produces the same transformation as the vector field above. This is one important property of continuous symmetry of the manifold: the equivalence of local transformations under *constant* vector fields and global transformations of the entire space.

We can also recenter both the vector field and its associated points. Doing so, we find that the points and vector field are fundamentally *unchanged.*

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftOriginPointsVectors.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftOriginPointsVectors.mp4">download the video to view it</a></video>

Furthermore, the points and vector field don’t have to be recentered in the same direction as the vector field. Under any arbitrary translation, the points and the *constant* vector field remain unchanged*.* In this way, both the manifold and the Tangent Bundle exhibit continuous symmetry.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftOriginPointsVectors2.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridShiftOriginPointsVectors2.mp4">download the video to view it</a></video>

So to recap, when we say *continuous symmetry* of the tangent bundle of \\(\mathbb{R}^n\\) we mean three things:

- Local flows along an invariant vector field are the same as a global translation of all points

- Arbitrary translations of points yields the same set of points

- Arbitrary translations of *constant* vector fields yields the same *constant* vector field

One consequence of this continuous symmetry is that we can effectively treat *any* point in \\(\mathbb{R}^n\\) as the origin. To illustrate this, we will “subtract” two points \\(p\\) and \\(q\\)  and get a vector pointing from \\(p\\) to \\(q\\),  and then flow \\(p\\) along the vector to recover \\(q\\).

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridSubtractPoints.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridSubtractPoints.mp4">download the video to view it</a></video>

If we follow this animation *carefully,* we have to remember that this red vector lives in \\(T_p\mathbb{R}^n\\) and then *flow* the point starting at \\(p\\) until it reaches \\(q\\). Now, let’s first recenter the origin at \\(p\\) and then perform the subtraction.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridSubtractPointsAtOrigin.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/GridSubtractPointsAtOrigin.mp4">download the video to view it</a></video>

We see that this produces the *same* result, and this time the vector was “attached to the origin.” In effect, we can do the subtraction *as if* the space was centered at \\(p\\) and then only worry about vectors living in \\(T_0 \mathbb{R}^n\\).

You probably didn’t realize it, but we’ve set ourselves up for easy computation! If we use the standard basis of \\(\mathbb{R}^n\\), we simply subtract the coordinates of each point element-wise to get the vector. If we want to translate a point by a vector, we again can simply perform element-wise addition of the coordinates. The annoying book-keeping that comes with this vector distribution is effectively non-existent, because we just treat all vectors *as if* they lived in the tangent space at the origin.

In fact, since \\(T_0\mathbb{R}^n\\) is essentially just a copy of \\(\mathbb{R}^n\\), we can confuse points and vectors and likely *get away with it*. The distinction between a point and the vector that points from the origin to a point is so subtle that in practice it’s often not worth mentioning.

Using the continuous symmetry of \\(\mathbb{R}^n\\) we formulate our optimization steps as

- Compute the quantity to minimize *as if* our parameters represent the origin

- Compute a direction along which the quantity is locally reduced

- Add the minimizing direction to our parameters *as if* our parameters represent the origin

- Repeat until the problem converges

With continuous symmetry in euclidean space on our side, we can use the \\(\mathbb{R}^n\\) hammer on *everything*. Recalling our [previous post](https://www.tangramvision.com/blog/camera-modeling-pinhole-obsession): consider \\(\mathbb{T}^2\\). You’ll notice now that it is essentially \\(\mathbb{R}^2\\) with a couple of “extra points at infinity.” This makes computer vision with camera rays and the pinhole camera model feel easy! We just treat \\(\mathbb{T}^2\\)  as \\(\mathbb{R}^2\\) and compute reprojection error by subtracting points and minimizing that error.

## Lie Groups: *Continuing* the Symmetry Party

But, I hear you mutter to yourself,

> “Are there manifolds that have continuous symmetry of their Tangent Bundle, but are not flat like \\(\mathbb{R}^n\\)?”

Great question! The answer is yes! An example of manifolds that fit these criteria are [Lie Groups](https://en.wikipedia.org/wiki/Lie_group). A Lie Group (we’ll call it \\(\mathcal{G})\\) is a [mathematical group](https://en.wikipedia.org/wiki/Group_(mathematics)) that is also a smooth manifold.

Many commonly-used manifolds in robotics are Lie Groups:

- Special Orthogonal Group \\(SO(3)\\) - used for describing rigid-body rotations

- Special Euclidean Group \\(SE(3)\\) - used for describing rigid-body motion

- Special Linear Group \\(SL(3)\\) - used for describing continuous-time homographies

- Similarity Group \\(Sim(3)\\) - used for describing similarity transforms (rigid-body + scale)

Given their wide usage, it’s worthwhile to explore what makes them so useful.

A Lie Group is a mathematical group; it’s in the name. This means it has a way to compose elements, typically denoted as \\(g \circ h\\) for group elements \\(g,h \in \mathcal{G}\\). Additionally, the group has an identity element (typically called \\(e\\)). In some fashion, this identity can be considered the “origin”. Since a Lie group is *also* a smooth manifold, it has a tangent space at this “origin.” The tangent space at the identity element is called the [Lie Algebra](https://en.wikipedia.org/wiki/Lie_algebra) and is denoted by \\(T_e \mathcal{G} = \mathfrak{g}.\\)

To help us develop some intuition of Lie Groups, we will consider the simplest non-euclidean Lie Group, the unit circle \\(\mathcal{S}^1\\) and its associated Lie Algebra. Note that unlike our visualization of \\(\mathbb{R}^n,\\) we have chosen to explicitly display a tangent space of unit circle, since it takes a different shape than the underlying manifold.

![CircleLieAlgebra_ManimCE_v0.18.1.png](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f8903a9b45c2d4aa6779b_CircleLieAlgebra_ManimCE_v0.18.1.png)

Now, we proceed to choose some vector in this tangent space at the identity.

![CircleVector_ManimCE_v0.18.1.png](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f89037c2ce14a94eb854b_CircleVector_ManimCE_v0.18.1.png)

Now, let’s try to move a point along our chosen vector.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleVectorFlowWrong.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleVectorFlowWrong.mp4">download the video to view it</a></video>

If we simply move the point along the vector, the point leaves the manifold! This is not a valid flow, because the points must *by definition* live on the underlying manifold. To fix this, we will define the *flow* as the curve \\(\varphi(t)\\) such that  \\(\varphi'(t)\\) matches the tangent vector. For a vector field \\(X\\) defined on the entire manifold, we denote this flow as \\(\varphi_X(t)\\).

However, this definition makes solving for \\(\varphi_X(t)\\) a bit tough since we have to solve an ordinary differential equation on a manifold. If we instead think of Euler integration, we can *approximate* a solution by “moving along” the manifold in the direction of the vector field.

In \\(\mathbb{R}^n\\), for a given vector field \\(X: \mathbb{R}^n \to T\mathbb{R}^n\\) euler integration gives us the recurrence of

$$

p_k = p_{k-1} + X(p_{k-1}) \Delta t

$$

On the manifold we can use the exponential map to similar effect. For a given vector field \\(X: M \to TM\\) “euler integration” takes the form of

<p>$$p_k = \text{Exp}_{p_{k-1}}(X(p_{k-1})\Delta t)$$</p><!-- manual html to prevent showdown changing underscores into emphasis tags before mathjax gets to it -->

We can imagine this exponential map as a “pushed down” vector on the manifold.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleVectorFlowRight.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleVectorFlowRight.mp4">download the video to view it</a></video>

> ⚠️ In actuality, there is no such thing as a “pushed down vector,” however it’s a useful concept for visualization. If we use the definition of the exponential map, we can imagine this “pushed down vector” as the line drawn by the equation \\(\text{Exp}(\alpha v)\\) for a fixed vector \\(v\\) and a scalar \\(\alpha \in [0,1]\\).

Now following our example with \\(\mathbb{R}^2\\), let’s try to create a *constant* vector field by copying a vector into all points on the manifold. If we naively copy this vector in the same way that we did in the euclidean case (remember that big grid of arrows?), we get the following:

![CircleTangentSectionWrong_ManimCE_v0.18.1.png](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f89065b7a81cc793276c6_CircleTangentSectionWrong_ManimCE_v0.18.1.png)

This cannot be correct; our vectors wouldn’t live in their own respective tangent spaces. These “tangent vectors” have some component pointing normal to the manifold, which implies they are not tangent vectors at all! How do we address this mathematically?

> ⚠️ Here we are claiming that this visualization is incorrect because we are considering the unit circle and its tangent bundle as entities *embedded* in ambient euclidean space \\(\mathbb{R}^2\\). In reality, we only care about the geometric and algebraic structure of the unit circle and our visualization is *somewhat independent* of that structure. It it not uncommon to visualize a manifold and an associated fibre bundle (e.g. the tangent bundle) as living in orthogonal spaces but connected at a single point. This explains why you sometimes may see the tangent bundle visualized as a manifold with non-overlapping tangent spaces.

> ![Tangent_bundle.png](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f8903a9b1afcd2cb9b285_Tangent_bundle.png)

*PC: [https://commons.wikimedia.org/wiki/File:Tangent_bundle.svg](https://commons.wikimedia.org/wiki/File:Tangent_bundle.svg)*

> In this post, we’ll continue with the embedded and overlapping view to aid intuition, but don’t be surprised if you see the tangent bundle drawn differently in other resources.

Instead of naively copying the vector at identity to every point on the lie group, let’s consider an arbitrary point \\(g \in \mathcal{G}\\). We can consider this point on the manifold as simply a point or, using the group structure, we can consider it as an operator since \\(g \circ e = g\\). In other words, \\(g\\) can be regarded as an operator that takes the identity into \\(g\\). This “operatorness” of \\(g\\) is typically denoted as the map

$$

\begin{aligned}

L_g: \mathcal{G} &\to \mathcal{G} \\\\

h &\mapsto g \circ h

\end{aligned}

$$

where the \\(L\\) denotes that we are composing with \\(g\\) on the left. You can form a similar map by composing on the right; we’ll call it \\(R_g\\).

We can use this operator concept to “push” vectors around on the manifold. Instead of \\(L_g\\) operating on a fixed value \\(h\\), we can imagine it operating on a curve \\(\gamma(t)\\) with \\(\gamma(0) = h\\).

Now we have a new *shifted* curve

$$

L_g(\gamma(t)) = g \circ \gamma(t)

$$

We can now consider the derivative of this new *shifted vector* as a vector living in the tangent space \\(T_{g \circ h} \mathcal{G}\\):

$$

\frac{d}{dt}L_g(\gamma(t))\vert_{t=0}

$$

…just like we did in our original vector field in \\(\mathbb{R}^n\\). For convenience of notation, we’ll drop the explicit parametrization of the curves and denote this vector shift as \\(dL_g(h,v)\\) for some \\(v \in T_h \mathcal{G}.\\)

<blockquote>

💡

The argument made here to derive this <em>vector shift</em> function actually extends to arbitrary manifolds. If we have an map between manifolds

$$

\begin{aligned}

f: M \to N

\end{aligned}

$$

that obeys the continuity properties of the manifold, we can create an induced map on the tangent bundle

$$

df: TM \to TN

$$

This induced map is often referred to as the differential or “push forward” map because it “pushes” vectors in tangent spaces of one manifold onto tangent spaces of another manifold.

</blockquote>

Now, we will state a few facts without proof about \\(dL_g(h,v)\\)

1. \\(dL_g(h,v)\\)  is linear in \\(v\\)

2. \\(dL_g(h, v)\\) actually does not depend on \\(h\\), so we can write it as \\(dL_g(v)\\)

3. \\(dL_g(dL_h(v)) = dL_{g \circ h}(v)\\) or in other words group composition is compatible with vector shifting.

Got all that? Good! Now that we have this structure in place, let’s return to our tangent vector at the origin.

![CircleVector_ManimCE_v0.18.1.png](https://cdn.prod.website-files.com/5fff85e7f613e35edb5806ed/673f89037c2ce14a94eb854b_CircleVector_ManimCE_v0.18.1.png)

This time we will copy the vector to the other points on the manifold by *pushing* the vector with \\(dL_g\\) to each \\(g\\) on the manifold.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleTangentSectionRight.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleTangentSectionRight.mp4">download the video to view it</a></video>

If we flow all the points on the manifold along these vectors we’ll notice that all the points change in a consistent way! In the case for \\(\mathcal{S}^1\\) they are all rotating around the circle.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleTangentSectionFlow.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleTangentSectionFlow.mp4">download the video to view it</a></video>

Here, we’ve found another invariant flow field. Let’s determine if the vector field is *unchanged* under the combined transformation

$$

(h, v) \mapsto (L_g(h), dL_g(v))

$$

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleShiftVectors.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleShiftVectors.mp4">download the video to view it</a></video>

Just like the case for \\(\mathbb{R}^n\\), the points and vector field remain *unchanged*. Furthermore, like \\(\mathbb{R}^n\\) the transformation does not have to be *in the same direction* as the vector field.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleShiftVectors2.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleShiftVectors2.mp4">download the video to view it</a></video>

Since these vectors don’t all point in the same direction (at least from an extrinsic point of view), we cannot call this a *constant* vector field. However, the vector field is invariant to group transformations, so we will call it an *invariant* vector field.

With these observations, we can state that all Lie Groups have a *continuously symmetric* Tangent Bundle*.* The difference from the euclidean case is that points and (certain) vector fields are invariant under group *transformations* instead of *translations*.

> 💡Although this difference may seem significant at first, it turns that \\(\mathbb{R}^n\\) is actually Lie Group with translation as the group operator! Would you look at that.

As an added bonus, for these *invariant* vector fields, the Euler integration using the exponential map is **exact.** Specifically, for an *invariant* vector field the following closed-form integration holds:

$$

\varphi_X(t) = \text{Exp}_{p}(X(p)t)

$$

> ⚠️ Here in this section, we’ve implied a parallel between *geodesics* on manifolds with a Riemannian Metric and the exponential map on Lie Groups. Technically, we have not defined a notion of a Riemannian Metric for Lie Groups. In this manner, it is possible to define a Riemannian Metric on Lie Groups such that the *geodesic exponential* and the *group exponential* are distinct. Regardless, broadly speaking, in both cases the exponential map helps us move along the manifold in a coherent manner.

As with Euclidean space, we can “recenter” the manifold by using the group operator to move \\(g\\) to the identity \\(e\\) while maintaining the distance to all other points. To do this, we simply apply the operator \\(L_{g^{-1}}\\) to all points on the manifold.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleRecenter.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleRecenter.mp4">download the video to view it</a></video>

Likewise, we can “subtract” any two points \\(h\\) and \\(g\\) on the manifold by shifting the point we are subtracting to the origin and then using our aforementioned logarithm \\(\text{Log}_e(g^{-1} \circ h)\\).

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleRecenterSubtract.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/CircleRecenterSubtract.mp4">download the video to view it</a></video>

Leveraging this *continuous symmetry* of Lie Groups, our optimization steps become

- Compute the quantity to minimize *as if* our parameters represent the origin i.e. using \\(\text{Log}_e(g^{-1} \circ h)\\)

- Compute a direction along which the quantity is locally reduced

- Add the minimizing direction to our parameters *as if* our parameters represent the origin i.e. using \\(g \circ \text{Exp}_e(v)\\)

- Repeat until the problem converges

Which makes for a pretty simple algorithm! We really only have to worry about two extra operators \\(\text{Log}_e\\) and \\(\text{Exp}_e\\). Maybe optimizing “on the manifold” isn’t so difficult after all? Let’s apply this *continuous symmetry* concept to the unit sphere to finally cure our Pinhole Obsession.

## Symmetry of the Sphere

Back to our unit sphere! The whole reason we did this in the first place.

Ideally, we’d want the unit sphere to exhibit *continuous symmetry* of the Tangent Bundle so we can take advantage of its benefits:

- Local flow along *invariant* vector fields is the same as a global transformation

- Some arbitrary concept of “origin” or “identity”

- Invariance of points and certain vector fields under global transformations

At first glance, the sphere *looks* fairly symmetric, so constructing an *invariant* vector field should be straightforward. Let’s try to find such a vector field. To start, let’s choose a vector field where all the vectors point at the north pole.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/longitude_sphere.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/longitude_sphere.mp4">download the video to view it</a></video>

Here we notice that the points on the manifold are generally not all moving in the “same direction.” The points close to the north pole are converging and the points close to the south pole are diverging. This is not an invariant flow field.

Let’s again try to find an invariant flow field but this time choose the vector field where all the vectors are pointing east.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/east_sphere.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/east_sphere.mp4">download the video to view it</a></video>

Following the flow, the total set of points remains unchanged*.* This flow field yields a rotation of the sphere. So, the manifold itself exhibits some *continuous symmetry* because rotation leaves the set of points on the manifold unchanged.

> ⭐ If you are wondering why these points flowing east are not moving along great circles, then you are a very astute reader! Remember, that in general using the exponential map \\(\text{Exp}_p(v)\\) to solve the flow \\(\varphi_X(t)\\) of a vector field \\(X\\) is an *approximation.* The true definition is that derivative of the flow \\(\varphi_X'(t)\\) must match the vector field \\(X\\) at every point along the curve.

Does this symmetry extend to the tangent bundle \\(T\mathcal{S}^2\\)? Well, if we swap the north pole and south pole, the vector field is now pointing west! So, the vector field that produces rotations is *not* *invariant* under rotations.

<video width="100%" preload="metadata" loop muted controls><source src="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/flip_sphere.mp4" type="video/mp4"> If this video doesn't work in browser, <a href="https://tangram-vision-blog-resources.s3.us-west-1.amazonaws.com/video/flip_sphere.mp4">download the video to view it</a></video>

These visualizations, and the aircraft example from beginning of this post point to \\(\mathcal{S}^2\\) not having *continuous symmetry* of its Tangent Bundle. This lack of Tangent Bundle symmetry can be proven rigorously by using the [Hairy Ball theorem](https://en.wikipedia.org/wiki/Hairy_ball_theorem). Unfortunately, this means that we cannot take advantage of **all** the benefits of symmetry when optimizing over \\(\mathcal{S}^2\\). Specifically:

- There is no invariant vector field under a set of global transformations

- We can’t *recenter* a point of interest to do tangent-space computations in a more convenient location

- Each tangent space must be regarded as distinct and not simply *shifted copies* of an origin tangent space

Ultimately, optimizing over a generic asymmetric manifold is a large increase in complexity as opposed optimizing over Euclidean Space or Lie Groups. For each computation, we must perform careful book-keeping to ensure the coherence between:

- Points on the manifold: \\(p \in \mathcal{S}^2\\)

- Vectors in the tangent spaces at each point: \\(v \in T\mathcal{S}^2\\)

- Exponential maps to “move along the manifold”: \\(\text{Exp}_p(v)\\)

- Logarithmic maps to compute the difference between points: \\(\text{Log}_p(q)\\)

If this coherence is not maintained and properly modeled, the optimization is likely to fail. After really digging into the full complexity of optimization over a generic asymmetric manifold, Lie Groups and Euclidean Space seem even more pleasant to work with.

## “Grouping” it All Up

As we can see, optimization over Lie Groups (including Euclidean space) is actually a special case of optimization over a manifold. With *continuous symmetry,* we can actually ignore the effects of the starting point on the exponential operator*. Continuous symmetry* allows us to recenter the manifold on our starting point \\(g\\) (the Lie algebra) and *pretend* that we started at the origin. This is *incredibly handy* and in robotics we use this property a lot!

You’ll notice that in the optimization steps For Lie Groups, we only used the exponential and logarithmic maps at one point: the identity \\(e\\). From a mathematical perspective, this is **the** defining characteristic of Lie Groups. This defining characteristic can be described as an isomorphism between

- The group itself \\(\mathcal{G}\\)

- The Lie Algebra \\(\mathfrak{g} = T_e\mathcal{G}\\)

- The invariant vector fields on \\(\mathcal{G}\\)

There is a number of deep mathematical connections that can be made here, and we encourage the interested reader to learn more. However, for our use-case, it means we can rewrite our exponential map as

$$

\text{Exp}_g(v) = g \circ \text{Exp}_e(v)

$$

If you’ve read the previously mentioned “Micro Lie Theory” paper, you’ll recognize this as the “circle plus” operator i.e. \\(g \oplus v\\). With Lie Groups, there **really** is only one exponential map of concern — the one defined at the identity. This does not hold true for general manifolds. This is why when talking about Lie Groups **the** exponential \\(\text{Exp}\\) map will be described instead of mentioning exponential maps at each point.

So, when using Lie Groups for optimization, we can ignore the specifics about “which tangent space” we’re in. We operate either on the group \\(\mathcal{G}\\), or in the [Lie Algebra](https://en.wikipedia.org/wiki/Lie_algebra) \\(\mathfrak{g}\\). To recall our tools analogy, In the land of Lie Groups, we have two tools to master: a hammer \\(\mathcal{G}\\)  and… screwdriver \\(\mathfrak{g}\\).

---

Did you have fun? We sure did. Thanks for going on this deep dive into Differential Geometry and its implications for optimization problems in robotics. Everyone loves Lie groups, no doubt about it.

Of course, if you’re experiencing these concerns in your day-to-day development, it’s probably time you considered working with the Tangram team. We’ve solved these problems so many different ways, on twice as many platforms. It’s our specialty! All this cool math in an easy-to-use package that can shorten your time to market; what’s not to love?

‍

Salmon return to lay eggs in historic habitat after largest dam removal project in US history - OPB

A giant female Chinook salmon flips on her side in the shallow water and wriggles wildly, using her tail to carve out a nest in the riverbed as her body glistens in the sunlight. In another moment, males butt into each other as they jockey for a good position to fertilize eggs.

These are scenes local tribes have dreamed of seeing for decades as they fought to bring down four hydroelectric dams blocking passage for struggling salmon along more than 400 miles (644 kilometers) of the Klamath River and its tributaries along the Oregon-California border.

Now, less than a month after those dams came down in the largest dam removal project in U.S. history, salmon are once more returning to spawn in cool creeks that have been cut off to them for generations. Video shot by the Yurok Tribe show that hundreds of salmon have made it to tributaries between the former Iron Gate and Copco dams, a hopeful sign for the newly freed waterway.

“Seeing salmon spawning above the former dams fills my heart,” said Joseph L. James, chairman of the Yurok Tribe. “Our salmon are coming home. Klamath Basin tribes fought for decades to make this day a reality because our future generations deserve to inherit a healthier river from the headwaters to the sea.”

FILE - Excess water spills over the top of a dam on the Lower Klamath River known as Copco 1 near Hornbrook, Calif.

Gillian Flaccus / AP

The Klamath River flows from its headwaters in southern Oregon and across the mountainous forests of northern California before it reaches the Pacific Ocean.

The completion of the hydroelectric dam removal project on Oct. 2 marked a major victory for local tribes. Through protests, testimony and lawsuits, the tribes showcased the environmental devastation caused by the dams, especially to salmon, which were cut off from their historic habitat and dying in alarming numbers because of poor water-quality.

There have been lower concentrations of harmful algae blooms since the dam removal, Toz Soto, fisheries program manager with the Karuk Tribe, said during a press conference after the dams came down. In October, the water temperature during the day was an average of 8 degrees Celsius (14 degrees Fahrenheit) cooler compared to the same month over the last nine years, according to the Klamath River Renewal Corporation, the nonprofit entity created to oversee the project.

“All in all, the fish that came up this year were really healthy,” Soto said. “I didn’t see fish with bacterial infections and things like that, so water temperature’s already having an impact on the fishes’ health.”

The number of salmon that have quickly made it into previously inaccessible tributaries has also been encouraging. Experts have counted 42 redds, or salmon egg nests, and have tallied as many as 115 Chinook salmon in one day in Spencer Creek, which is above the former J.C. Boyle dam, the furthest upstream of the four removed dams, said Mark Hereford with the Oregon Department of Fish and Wildlife.

“They’re showing us where the good habitat is; they’re showing us where there’s a lack of habitat,” said Barry McCovey Jr, director of the Yurok Tribal Fisheries department. “So we can use these fish to inform us as river managers, as scientists, where restoration needs to take place.”

FILE - A view shows the Copco 1 Dam in Hornbrook, Calif., Sunday, Sept. 17, 2023.

Haven Daley / AP

Power company PacifiCorp built the dams to generate electricity between 1918 and 1962. But the structures halted the natural flow of the waterway that was once known as the third-largest salmon-producing river on the West Coast. They disrupted the lifecycle of the region’s salmon, which spend most of their life in the Pacific Ocean but return to the chilly mountain streams to lay eggs.

At the same time, the dams only produced a fraction of PacifiCorp’s energy at full capacity, enough to power about 70,000 homes. They also didn’t provide irrigation, drinking water or flood control, according to Klamath River Renewal Corporation.

McCovey said the return of so many salmon happened faster than he had expected and makes him hopeful for the future of the river.

“Out of all the milestones that we’ve had, this one to me is the most significant,” he said. “It feels like catharsis. It feels like the right path.”

___

Associated Press reporter Sophie Austin contributed to this report.

Runtime-Extensible SQL Parsers Using PEG

Hannes Mühleisen and Mark Raasveldt

Published on 2024-11-22

TL;DR: Despite their central role in processing queries, parsers have not received any noticeable attention in the data systems space. State-of-the art systems are content with ancient old parser generators. These generators create monolithic, inflexible and unforgiving parsers that hinder innovation in query languages and frustrate users. Instead, parsers should be rewritten using modern abstractions like Parser Expression Grammars (PEG), which allow dynamic changes to the accepted query syntax and better error recovery. In this post, we discuss how parsers could be re-designed using PEG, and validate our recommendations using experiments for both effectiveness and efficiency.

This post is a shortened version of our peer-reviewed research paper "Runtime-Extensible Parsers" that was accepted for publication and presentation at the 2025 Conference on Innovative Data Systems Research (CIDR) that is going to be held in Amsterdam between January 19 and 22, 2025. You can read the full paper if you prefer.

The parser is the DBMS component that is responsible for turning a query in string format into an internal representation which is usually tree-shaped. The parser defines which queries are going to be accepted at all. Every single SQL query starts its journey in a parser. Despite its prominent position in the stack, very little research has been published on parsing queries for data management systems. There seems to have been very little movement on the topic in the past decades and their implementations are largely stuck in sixty-year-old abstractions and technologies.

The constant growth of the SQL specification with niche features (e.g., support for graph queries in SQL/PGQ or XML support) as well as the desire to support alternative query notations like dplyr, piped SQL, PRQL or SaneQL makes monolithic parsers less and less practical: in their traditional design, parser construction is a compile-time activity where enormous grammar files are translated into state machine transition lookup tables which are then baked in a system binary. Having those always be present in the parser might be wasteful especially for size-conscious binary distributions like WebAssembly (Wasm).

Many if not most SQL systems use a static parser created using a YACC-style parser toolkit: we are able to easily confirm this for open-source systems like PostgreSQL and MySQL/MariaDB. From analyzing their binaries' symbol names, we also found indications that Oracle, SQL Server and IBM Db2 use YACC. Internally, YACC and its slightly more recent variant GNU Bison as well as the "Lemon" parser generator used by SQLite all use a "single look-ahead left-to-right rightmost derivation" LALR(1) parser generator. This generator translates a formal context-free set of grammar rules in Extended Backus-Naur Form (EBNF) to a parser state machine. LALR parsers are a more space-efficient specialization of LR(k) parsers as first described by Knuth. But in effect, the most advanced SQL systems of 2024 use parser technology from the 1960s. Given that the rest of data management systems have been greatly overhauled since this should raise the question of why the parser did not receive any serious engineering attention.

Database systems are moving towards becoming ecosystems instead of pre-built monoliths. Much of the innovation in the PostgreSQL, SQLite, and DuckDB communities now comes from extensions, which are shared libraries that are loaded into the database system at run-time to extend the database system with features like vector similarity search, geospatial support, file systems, or graph processing. Bundling all those features upfront would be difficult due to additional binary size, external dependencies. In addition, they are often maintained independently by their communities. Thus far, at least in part due to the ubiquity of YACC-style parsers, those community extensions have been restricted from extending syntax. While this is also true in other ecosystems like Python, the design of SQL with its heavy focus on syntax and not function calls makes the extensions second-class citizens that have to somehow work around the restrictions by the original parser, e.g., by embedding custom expressions in strings.

We propose to re-think data management system parser design to create modern, extensible parsers, which allow a dynamic configuration of the accepted syntax at run-time, for example to allow syntax extensions, new statements, or to add entirely new query languages. This would allow to break up the monolithic grammars currently in use and enable more creativity and flexibility in what syntax a data management system can accept, both for industrial and research use. Extensible parsers allow for new grammar features to be easily integrated and tested, and can also help bridge the gap between different SQL dialects by adding support for the dialect of one system to the parser of another. Conversely, it might also be desirable in some use cases to restrict the acceptable grammar, e.g., to restrict the complexity of queries, or to enforce strict compliance with the SQL standard.

Modernizing parser infrastructure also has additional benefits: one of the most-reported support issues with data management systems are unhelpful syntax errors. Some systems go to great lengths to try to provide a meaningful error message, e.g., this column does not exist, did you mean ..., but this is typically limited to resolving identifiers following the actual parsing. YACC-style parsers exhibit "all-or-nothing" behavior, the entire query or set of queries either is accepted entirely or not at all. This is why queries with actual syntactical errors (e.g., SELEXT instead of SELECT are usually harshly rejected by a DBMS. MySQL for example is notorious for its unhelpful error messages:

You have an error in your SQL syntax; check the manual that corresponds

to your MySQL server version for the right syntax to use near 'SELEXT'

at line 1.

Parsing Expression Grammar (PEG) parsers represent a more modern approach to parsing. PEG parsers are top-down parsers that effectively generate a recursive-descent style parser from a grammar. Through the "packrat" memoization technique PEG parsers exhibit linear time complexity in parsing at the expense of a grammar-dependent amount of extra memory. The biggest difference from a grammar author perspective is the choice operator where multiple syntax options can be matched. In LALR parsers options with similar syntax can create ambiguity and reduce conflicts. In PEG parsers the first matching option is always selected. Because of this, PEG parsers cannot be ambiguous by design.

As their name suggests, parsing expression grammar consists of a set of parsing expressions. Expressions can contain references to other rules, or literal token references, both as actual strings or character classes similar to regular expressions. Expressions can be combined through sequences, quantifiers, optionals, groupings and both positive and negative look-ahead. Each expression can either match or not, but it is required to consume a part of the input if it matches. Expressions are able to look ahead and consider the remaining input but are not required to consume it. Lexical analysis is typically part of the PEG parser itself, which removes the need for a separate step.

One big advantage is that PEG parsers do not require a compilation step where the grammar is converted to for example a finite state automaton based on lookup tables. PEG can be executed directly on the input with minimal grammar transformation, making it feasible to re-create a parser at runtime. PEG parsers are gaining popularity, for example, the Python programming language has recently switched to a PEG parser.

Another big advantage of PEG parsers is error handling: the paper "Syntax Error Recovery in Parsing Expression Grammars" describes a practical technique where parser rules are annotated with "recovery" actions, which can (1) show more than a single error and (2) annotate errors with a more meaningful error message.

A possible disadvantage of memoized packrat parsing is the memory required for memoization: the amount required is proportional to the input size, not the stack size. Of course, memory limitations have relaxed significantly since the invention of LALR parsers sixty years ago and queries typically are not "Big Data"` themselves.

To perform experiments on parser extensibility, we have implemented an – admittedly simplistic – experimental prototype PEG parser for enough of SQL to parse all the TPC-H and TPC-DS queries. This grammar is compatible with the cpp-peglib single-header C++17 PEG execution engine.

cpp-peglib uses a slightly different grammar syntax, where / is used to denote choices. The symbol ? shows an optional element, and * defines arbitrary repetition. The special rules Parens() and List() are grammar macros that simplify the grammar for common elements. The special %whitespace rule is used to describe tokenization.

Below is an abridged version of our experimental SQL grammar, with the Expression and Identifier syntax parsing rules omitted for brevity:

Statements <- SingleStmt (';' SingleStmt )* ';'*

SingleStmt <- SelectStmt

SelectStmt <- SimpleSelect (SetopClause SimpleSelect)*

SetopClause <-

('UNION' / 'EXCEPT' / 'INTERSECT') 'ALL'?

SimpleSelect <- WithClause? SelectClause FromClause?

WhereClause? GroupByClause? HavingClause?

OrderByClause? LimitClause?

WithStatement <- Identifier 'AS' SubqueryReference

WithClause <- 'WITH' List(WithStatement)

SelectClause <- 'SELECT' ('*' / List(AliasExpression))

ColumnsAlias <- Parens(List(Identifier))

TableReference <-

(SubqueryReference 'AS'? Identifier ColumnsAlias?) /

(Identifier ('AS'? Identifier)?)

ExplicitJoin <- ('LEFT' / 'FULL')? 'OUTER'?

'JOIN' TableReference 'ON' Expression

FromClause <- 'FROM' TableReference

((',' TableReference) / ExplicitJoin)*

WhereClause <- 'WHERE' Expression

GroupByClause <- 'GROUP' 'BY' List(Expression)

HavingClause <- 'HAVING' Expression

SubqueryReference <- Parens(SelectStmt)

OrderByExpression <- Expression ('DESC' / 'ASC')?

('NULLS' 'FIRST' / 'LAST')?

OrderByClause <- 'ORDER' 'BY' List(OrderByExpression)

LimitClause <- 'LIMIT' NumberLiteral

AliasExpression <- Expression ('AS'? Identifier)?

%whitespace <- [ \t\n\r]*

List(D) <- D (',' D)*

Parens(D) <- '(' D ')'

All experiments were run on a 2021 MacBook Pro with the M1 Max CPU and 64 GB of RAM. The experimental grammar and the code for experiments are available on GitHub.

Loading the base grammar from its text representation into the cpp-peglib grammar dictionary with symbolic rule representations takes 3 ms. In case that delay should become an issue, the library also allows to define rules programmatically instead of as strings. It would be straightforward to pre-compile the grammar file into source code for compilation, YACC-style. While somewhat counter-intuitive, it would reduce the time required to initialize the initial, unmodified parser. This difference matters for some applications of e.g., DuckDB where the database instance only lives for a few short milliseconds.

For the actual parsing, YACC parses TPC-H Query 1 in ca. 0.03 ms, where cpp-peglib takes ca. 0.3 ms, a ca. 10 times increase. To further stress parsing performance, we repeated all TPC-H and TPC-DS queries six times to create a 36,840 line SQL script weighing in at ca. 1 MB. Note that a recent study has found that the 99-percentile of read queries in the Amazon Redshift cloud data warehouse are smaller than 16.5 kB.

Postgres takes on average 24 ms to parse this file using YACC. Note that this time includes the execution of grammar actions that create Postgres' parse tree. cpp-peglib takes on average 266 ms to parse the test file. However, our experimental parser does not have grammar actions defined yet. When simulating actions by generating default AST actions for every rule, parsing time increases to 339 ms. Note that the AST generation is more expensive than required, because a node is created for each matching rule, even if there is no semantic meaning in the grammar at hand.

Overall, we can observe a ca. 10 times slowdown in parsing performance when using the cpp-peglib parser. However, it should be noted that the absolute duration of those two processes is still tiny; at least for analytical queries, sub-millisecond parsing time is more than acceptable as parsing still only accounts for a tiny fraction of overall query processing time. Furthermore, there are still ample optimization opportunities in the experimental parsers we created using an off-the-shelf PEG library. For example, the library makes heavy use of recursive function calls, which can be optimized e.g., by using a loop abstraction.

In the following, we present some experiments in extending the prototype parser with support for new statements, entirely new syntax and with improvements in error messages.

It is already possible to replace DuckDB's parser by providing an alternative parser. Several community extensions such as duckpgq, prql and psql use this approach. When trying to parse a query string, DuckDB first attempts to use the default parser. If this fails, it switches to the extension parsers as failover. Therefore, these extensions cannot simply extend the parser with a few extra rules – instead, they implement the complete grammar of their target language.

Let's assume we would want to add a new top-level UNPIVOT statement to turn columns into rows to a SQL dialect. UNPIVOT should work on the same level as e.g., SELECT, for example to unpivot a table t1 on a specific list of columns or all columns (*), we would like to be able to write:

UNPIVOT t1 ON (c1, c2, c3);

UNPIVOT t1 ON (*);

It is clear that we would have to somehow modify the parser to allow this new syntax. However, when using a YACC parser, this would require modifying the grammar, re-running the parser generator, hoping for the absence of shift-reduce conflicts, and then recompiling the actual database system. However, this is not practical at run-time which is when extensions are loaded, ideally within milliseconds.

In order to add UNPIVOT, we have to define a grammar rule and then modify SingleStmt to allow the statement in a global sequence of SQL statements. This is shown below. We define the new UnpivotStatement grammar rule by adding it to the dictionary, and we then modify the SingleStmt rule entry in the dictionary to also allow the new statement.

UnpivotStatement <- 'UNPIVOT' Identifier

'ON' Parens(List(Identifier) / '*')

SingleStmt <- SelectStatement / UnpivotStatement

Note that we re-use other machinery from the grammar like the Identifier rule as well as the Parens() and List() macros to define the ON clause. The rest of the grammar dictionary remains unchanged. After modification, the parser can be re-initialized in another 3 ms. Parser execution time was unaffected.

Let's now assume we would want to modify the SELECT syntax to add support for SQL/PGQ graph matching patterns. Below is an example query in SQL/PGQ that finds the university name and year for all students called Bob:

SELECT study.classYear, study.name

FROM GRAPH_TABLE (pg,

MATCH

(a:Person WHERE a.firstName = 'Bob')-[s:studyAt]->(u:University)

COLUMNS (s.classYear, u.name)

) study;

We can see that this new syntax adds the GRAPH_TABLE clause and the pattern matching domain-specific language (DSL) within. To add support for this syntax to a SQL parser at runtime, we need to modify the grammar for the SELECT statement itself. This is fairly straightforward when using a PEG. We replace the rule that describes the FROM clause to also accept a sub-grammar starting at the GRAPH_TABLE keyword following by parentheses. Because the parser does not need to generate a state machine, we are immediately able to accept the new syntax.

Below we show a small set of grammar rules that are sufficient to extend our experimental parser with support for the SQL/PGQ GRAPH_TABLE clause and the containing property graph patterns. With this addition, the parser can parse the query above. Parser construction and parser execution timings were unaffected.

Name <- (Identifier? ':' Identifier) / Identifier

Edge <- ('-' / '<-') '[' Name ']' ('->' / '-')

Pattern <- Parens(Name WhereClause?) Edge

Parens(Name WhereClause?)

PropertyGraphReference <- 'GRAPH_TABLE'i '('

Identifier ','

'MATCH'i List(Pattern)

'COLUMNS'i Parens(List(ColumnReference))

')' Identifier?

TableReference <-

PropertyGraphReference / ...

dplyr, the "Grammar of Data Manipulation", is the de facto standard data transformation language in the R Environment for Statistical Computing. The language uses function calls and a special chaining operator (%>%) to combine operators. Below is an example dplyr query:

df %>%

group_by(species) %>%

summarise(

n = n(),

mass = mean(mass, na.rm = TRUE)

) %>%

filter(n > 1, mass > 50)

For those unfamiliar with dplyr, the query is equivalent to this SQL query:

SELECT * FROM (

SELECT count(*) AS n, AVG(mass) AS mass

FROM df

GROUP BY species)

WHERE n > 1 AND mass > 50;

With an extensible parser, it is feasible to add support for completely new query languages like dplyr to a SQL parser. Below is a simplified grammar snippet that enables our SQL parser to accept the dplyr example from above.

DplyrStatement <- Identifier Pipe Verb (Pipe Verb)*

Verb <- VerbName Parens(List(Argument))

VerbName <- 'group_by' / 'summarise' / 'filter'

Argument <- Expression / (Identifier '=' Expression)

Pipe <- '%>%'

SingleStmt <- SelectStatement /

UnpivotStatement / DplyrStatement

It is important to note that the rest of the experimental SQL parser still works, i.e., the dplyr syntax now also works. Parser construction and parser execution timings were again unaffected.

As mentioned above, PEG parsers are able to generate better error messages elegantly. A common novice SQL user mistake is to mix up the order of keywords in a query, for example, the ORDER BY must come after the GROUP BY. Assume an inexperienced user types the following query:

SELECT customer, SUM(sales)

FROM revenue

ORDER BY customer

GROUP BY customer;

By default, both the YACC and the PEG parsers will report a similar error message about an unexpected 'GROUP' keyword with a byte position. However, with a PEG parser we can define a "recovery" syntax rule that will create a useful error message. We modify the OrderByClause from our experimental grammar like so:

OrderByClause <- 'ORDER'i 'BY'i List(OrderByExpression)

%recover(WrongGroupBy)?

WrongGroupBy <- GroupByClause

{ error_message "GROUP BY must precede ORDER BY" }

Here, we use the %recover construct to match a misplaced GROUP BY clause, re-using the original definition, and then trigger a custom error message that advises the user on how to fix their query. And indeed, when we parse the wrong SQL example, the parser will output the custom message.

In this post, we have proposed to modernize the ancient art of SQL parsing using more modern parser generators like PEG. We have shown how by using PEG, a parser can be extended at run-time at minimal cost without re-compilation. In our experiments we have demonstrated how minor grammar adjustments can fundamentally extend and change the accepted syntax.

An obvious next step is to address the observed performance drawback observed in our prototype. Using more efficient implementation techniques, it should be possible to narrow the gap in parsing performance between YACC-based LALR parsers and a dynamic PEG parser. Another next step is to address some detail questions for implementation: for example, parser extension load order should ideally not influence the final grammar. Furthermore, while parser actions can in principle execute arbitrary code, they may have to be restrictions on return types and input handling.

We plan to switch DuckDB's parser, which started as a fork of the Postgres YACC parser, to a PEG parser in the near future. As an initial step, we have performed an experiment where we found that it is possible to interpret the current Postgres YACC grammar with PEG. This should greatly simplify the transitioning process, since it ensures that the same grammar will be accepted in both parsing frameworks.

We would like to thank Torsten Grust, Gábor Szárnyas and Daniël ten Wolde for their valuable suggestions. We would also like to thank Carlo Piovesan for his translation of the Postgres YACC grammar to PEG.

Whether you have to do with data in form of CSV, JSON or a full-blooded programming language like C, JavaScript, Scala, or maybe a query language like SQL, you always transform some sequence of characters (or binary values) into a structured representation. Whatever you’ll do with that representation depends on your domain and business goals, and is quite often the core value of whatever you are doing. With a plethora of tools doing the parsing for us (including the error-handling), we might easily overlook how complex and interesting process it is.

Formal grammars

First of all, most input formats that we handle follow some formal definition, telling e.g. how key-values are organized (JSON), how do you separate column names/values (CSV), how do you express projections and conditions (SQL). These rules are defined in an unambiguous way so that the input could be interpreted in a very deterministic way. It directly opposed language we use to communicate with other people, which often is ambiguous and put into the context. Wanna grab some burger? might be a nice suggestion if you are talking to a colleague that have to skip lunch and likes burgers, but might be offensive if told in a sarcastic tone to someone who doesn’t like meat. Then, words can have different meaning depending on the culture you are currently in, in which times you live or what are you and your conversationalist social position (vide, e.g. Japanese and how your position and suffixes you add at the end of the name change the tone of the whole conversation). Languages we use when communicating with a computer must be free of such uncertainties. The meaning should depend only on the input we explicitly entered and interpreted deterministically. (Just in case: by deterministically, I mean, deterministically interpreted, which doesn’t mean that it would always produce the same result. If I write currentTimeMillis(), the function will always return a different result, but the meaning will be always the same - compiler/interpreter will understand that I want to call currentTimeMillis() function, and it won’t suddenly decide that I want to e.g. change the compiler flag. Of course, the meaning of the function can change in time - for instance, if I edit the source code in between runs - and surely the value returned by it, which is bound to time).

Initially, it wasn’t known, how to parse languages. The reason, that we had to start with punching cards, sometime later moved on to assembly, and later on invent Fortran and Lisp, go through whole spaghetti code with Basic, get The case against goto statement by Dijkstra, until we could - slowly - started developing more sophisticated compilers we have today, was that there were no formal foundations to it.

Linguists know, that we can distinguish some parts of speech like: noun (specific thing, e.g. cat, Alice, Bob), pronoun (generic replacement for a specific thing, e.g. I, you, he, she), verb (action), adjective (description or trait of something, e.g. red, smart), etc. However, the also know that the function of part of speech changes depending on how we construct a sentence - that’s why we also have the parts of the sentence: subject (who performs the action: e.g. Alice in Alice eats dinner), object (who is the target of the action, e.g. dinner in Alice eats dinner), modifiers and compliments, etc. We can only tell which part of the speech and sentence the word is in the context of a whole sentence:

- An alarm is set to 12 o’clock - here, set is a verb,

- This function returns an infinite set - here, set is a noun and an object,

- The set has the cardinality of 2 - here, set is a noun and a subject,

- All is set and done - here, set is an adverb and a modifier.

As we can see the same work might be a completely different thing depending on the context. This might be a problem when we try to process the sentence bottom-up, just like we (supposedly) do when we analyze them in English lessons. This is a noun. That is a verb. This noun is subject, this verb is an object. This is how subsentences relate to one another. Now we can analyze the nice tree of relations between words and understand the meaning. As humans, we can understand the relationship between the words on the fly, the whole exercise is only about formalizing our intuition.

But machines have no intuition. They can only follow the rules, we establish for them. And when dealing with computers we quite often establish them using the divide-and-conquer strategy: split the big problem into smaller ones, and then combine the solutions. With natural languages the context makes it quite challenging, which is why no simple solution appeared even though we were regularly trying. Current progress was made mostly using machine learning, which tackles the whole problem at once, trying to fit whole parts of the sentence as patterns, without analyzing what is what. However, when it comes to communication with a computer, ambiguities can be avoided, simply by designing a language in a way that doesn’t allow them. But how to design a language?

One of the first researchers, that made the progress possible was Noam Chomsky. Interestingly, he is not considered a computer scientist - he is (among others) linguists, who is credited with cognitive revolution. Chomsky believes, that how we structure languages is rooted in how our brains process speech, reading, etc. Therefore similarities between languages’ structures (parts of speech, parts of sentences, structuring ideas into sentences in the first place, grammar cases) are a result of how processes inside our brain. While he wasn’t the first one who tried to formalize a language into a formal grammar (we know of e.g. Pāṇini), Chomsky was the first to formalize generative grammars, that is grammars where you define a set of rules, and create a language by combining the rules.

How can we define these rules? Well, we want to be able to express each text in such grammar as a tree - at leaves, we’ll have words or punctuation marks of sorts. Then, there will be nodes aggregating words/punctuation marks by their function (part of a sentence). At the top of the tree we’ll have a root, which might be (depending on grammar) a sentence/a statement/an expression, or maybe a sequence of sentences (a program). The definitions will work this way: take a node (starting with root) and add some children to it: the rules will say how the specific node (or nodes) can have children appended (and what kind of children). The grammar definitions will rarely be expressed with specific values (e.g. you won’t write down all possible names), but rather using symbols:

Sentence→Subject verb Object. Subject→name surname∣nickname Object→item∣animal

Here, Sentence could be a start symbol. We would build a sentence by unrolling notes according to rules. Here there is only one rule going from Sentence - one that allows adding Subject, verb, Object and the dot sign (.) children (order matters!). verb is written with a small capital because it is (or eventually will be) a leaf - since unrolling ends (terminates) at leaves, we would call symbols allowed to be leaves as terminal symbols. As you might guess, nodes become nonterminal symbols. Terminal symbols will eventually be replaced with an actual word, unless they are keywords (have you noticed, how if, else, function, class, … get special treatment in many languages?) or special symbols (;, (, ), ,, …).

Having, Subject verb Object., we can continue unrolling. Our second rule lets us turn Subject into name surname or nickname (the vertical line ∣ is a shortcut - A→B∣C should be understood as A→B or A→C). And our third rule allows us to turn Object to item or animal. We can go both times with the first option and obtain name surname verb item. (e.g. John Smith eats cereals.). We might go both times with the second option - nickname verb animal. (Johnny likes cats.). And so on.

Notice that in the end, we’ll always end up with a sequence of terminals. If we couldn’t, there would be something wrong with a language. This definition that takes a sequence of symbols and returns another sequence of symbols is called production rules. We can describe each formal language as a quadruple G=(N,Σ,P,S), where T is a finite set of nonterminal symbols, Σ a finite set of terminal symbols, P is a set of production rules and S∈Σ is a start symbol.

Besides formalization of generative grammars, Chomsky did something else. He was responsible for the organization of formal languages in a hierarchy called after him the Chomsky hierarchy.

The Chomsky hierarchy

On the top of the hierarchy are type-0 languages or unrestricted languages. There is no restriction placed upon how we define such language. A production rule might be any sequence of terminals and nonterminals into any sequence of terminals and nonterminals (in the earlier example there was always nonterminal symbol on the left side - that is not a rule in general!). These languages are hard to deal with, so we try to define data format and programming languages in term of a bit more restrained grammars, that are easier to analyze.

First restriction appears with type-1 languages or context-sensitive grammars (CSG). They require, that all production rules would be in form of:

αAβ→αγβ

where A∈N, α,β∈(N∪Σ)∗, γ∈(N∪Σ)+ (where ∗ and + are Kleene star and Kleene plus). In other words, we can perform A→γ only, if immediately before A is α and immediately after is β (rule is applied in context). But, even these grammars appears to be difficult to deal with. That is why we apply even more restrictions to grammars we use in our everyday life.

More specifically, we might want our grammars to be independent of context. Type-2 languages or context-free grammars (CFG), are CSGs where context is always empty, or in other words, where each production rule is in form of:

A→γ

where A∈N and γ∈(N∪Σ)+. These grammars, are quite well researched, so we have plenty of tools helping us analyze them, check if something belongs to language, how to generate parser etc. That’s why the majority (all?) of programming languages and data inputs are defined using CFGs.

To be precise, when it comes to programming languages, we quite often deal with context-sensitive grammars, but it is easier to deal with them as if they were context-free - call that syntactical analysis (what meaning we can attribute to a words basing on their position in a sentence) - and then take the generated tree, called abstract syntax tree, and check if makes semantic sense (is the name a function, a variable or a type? Does it makes sense to use it in the context it was placed?). If we expressed it as a context-sensitive grammar we could do much (all?) of semantic analysis in the same time we check syntax, but the grammar could get too complex for us for understand it (or at least to handle it efficiently).

To illustrate the difference between syntax and semantics we can get back to your earlier example.

nickname verb item.

It is a correct semantics in the language. Let’s substitute terminals with some specific values.

Johnny eat integral.

What we got is correct according to the rules based on words’ positions in the sentence (syntax), but as a whole - when you analyze the function of each word (semantics) - it makes no sense. Theoretically, we could define our language in an elaborate way, that would make sure that there would always be e.g. eats after the third person in a sentence and something edible after some form of to eat verb, but you can easily imagine, that the number of production rules would explode.

Finally, there is the most restricted kind of grammar in the Chomsky hierarchy. Type-3 grammar or regular grammar is a language, where you basically either prepend or append terminals. That is each production rule must be in the form of one of:

- A→a - where A∈N, a∈Σ,

- A→ϵ, where ϵ in an empty string,

- A→aB - where B∈N.

(We call it right regular grammar - if we instead required that the third rule would be in the form A→Ba it would be left regular grammar). While regular grammars are too restricted to define many programming languages on its own, is we’ll find out later on, that - when combined with CFG - they allow us to build modern parsers.

Regular languages

Let’s start with the most limited grammars, that is regular grammars. No matter how we define production rules, we will end up with a tree of form:

Of course, it doesn’t mean, that each such tree would be the same. For instance we could define our grammar like this:

- A0→aA1

- A1→aA2

- A2→aA3

- A3→ϵ

If we started from A0, the only possible sentence in such language would be aaaϵ. And - since ϵ represents an empty string, it would be actually just aaa. Another example could be grammar like this:

- S→aB

- B→bB∣bC

- C→c

If our starting symbol would be S, the sentences we could accept as belonging to grammar would be abc, abbc, abbbc, … If you’ve been programming for a while and you ever had to find some pattern in a text, you should have a feeling that looks familiar. Indeed, the regular language is formalism used to describe the regular expressions.

The first example you be described just as aaa, while the second as a(b+)c (or ab(b∗)c). Here, ∗ and + corresponds directly with Kleene star and Kleene plus. Now, that we know we are talking about regexpes, we can provide another definition of what could be a regular language, that would be equivalent to production-rule-based, but easier to work with.

A regular expression is anything build using the following rules:

- ϵ is a regular expression accepting the empty word as belonging to the language,

- a is a regular expression accepting 'a' belonging to some alphabet A (nonterminals) as a word belonging to the language,

- when you concatenate two regular expressions, e.g. AB, you accept words made by concatenating all valid words in A with all valid words in B (e.g. if a accepts only "a" and b accepts only "b", then ab accepts "ab"),

- you can sum up regular languages A∣B, to accept all words valid either in A or in B (e.g. a∣b would accept "a" or "b", but not "ab"),

- you can use Kleene star A∗ and Kleene plus A+, to define (respectively) any number of occurrences or at least one occurrence of a pattern A as words accepted by regular language (e.g. a(b+)c).

That is enough to define all regular expressions, though usually, we would have some utilities provided by regexp engines, e.g. [a−z], which is a shortcut for a∣b∣...∣y∣z or A? which is a shortcut for A∣ϵ. Just in case, I’ll also mention, that some implementations of regexp allows so-called backreferences - expression build using there are no longer regular languages, which has some practical implications. What are these implications?

Well, we haven’t discussed it so far, but there are some very close relationships between types of formal grammars and computation models. It just happens, that if we wanted to define a function checking whether a word/sentence/etc belongs to a regular grammar/a regular expression - which is equivalent to defining the language - is done by defining a finite-state automaton (FSA), that accepts this language. And vice-versa, each FSA defines a regular language. That correspondence dictates, how we implement regexp patterns - basically, each time we compile a regexp pattern, we are building a FSA, that would accept all words of grammar and only them.

In case you’ve never met FSA, let us remind what they are. Finite-state automaton or finite-state machine (FSM) is a 5-tuple (Q,Σ,δ,q0,F), where:

- Q is a finite set of states,

- Σ is a finite set of input symbols (an alphabet - equal to set of terminals without ϵ),

- a transition function δ:Q×Σ→Q, which would take the current state and next input symbol to return the next state,

- an initial state q0∈Q,

- a set of accepting states F⊆Q.

On a side note: an automaton - singular, meaning a machine, automata - plural, meaning machines. Other nerdy words which works like that: a criterion vs criteria.

For instance: our alphabet contains 3 possible characters Σ={a,b,c}. We want to build a finite state machine accepting regular expression a(b∗)c. Such machine:

- would have to start with a state indicating that nothing was yet matched, but also that nothing is wrong yet. Let’s mark it as q0,

- if first incoming input symbol is a, everything is OK, and we can move on to matching (b)∗. However, if b or c arrives, we can already tell, that the result is wrong. Let’s mark error as e. On e no matter, what comes next, we’ll just stay at e state,

- in this particular case we can safely assume, that if things started to go wrong, there is no way to recover, but it is not a general rule (if there was e.g. an alternative, then failing to match one expression, wouldn’t mean that we will fail to match the other expression),

- to indicate that we matched a, let’s create a new state q1. We have to do it, because FSM (which shares its acronym with its noodle excellency Flying Spaghetti Monster) can only remember the current state it is in, so we want to change the behavior (and when we move to match b∗ we will), we can only achieve that, by creating a different state for each behavior (and step of the algorithm) and using state transition to indicate progress of computation,

- OK, we arrived at state q1, so a was matched, and we want to match b∗. b∗ means that there could be 0 or more occurrences of b. In case there is 0, we have to match whatever is immediately after b∗, that is c. Accidentally, that would be the case when we are accepting input. Let’s mark that case as q2 and let’s put it into the set accepting states,

- In case we are in q1 and b arrives, we can… simply keep the current state. Until anything from Σ−{b} arrives, we can reuse current state,

- if we are in q1 and a arrives, input is wrong and we’ll go to e,

- at this point, we are at q2 (or e which means that things mismatched and we cannot recover). q2 is accepting state, so, if there is nothing else, we matched the input. However, if there is anything else incoming, this means we have e.g. abbca which shouldn’t be matched. So no matter what will come, we are moving to e.

What we defined right, now could be described like this:

Σ={a,b,c} Q={q0,q1,q2,e} δ={(q0,a)→q1,(q0,b)→e,(q0,c)→e,(q1,a)→e,(q1,b)→q1,(q1,c)→q2,(q2,a)→e,(q2,b)→e,(q2,c)→e,(e,a)→e,(e,b)→e,(e,c)→e} F={q2}

We could also make it more visual (bold border for accepting state):

As we can see, each state has to have defined transition for every possible letter of the alphabet (even if that transition is returning the current state as the next state). So, the size of machine definition (all possible transitions) is ∣Q∣×∣Σ∣.

Additionally, constructing the machine required some effort. We would like to automate the generation of FSM from regular expressions, and creating it in the final version might be troublesome. What we created is actually called deterministic finite state machine / deterministic finite automaton (DFA). It guarantees, that every single time we will deterministically get accepted a state for accepted input and non-accepted state for non-accepted input.

In practice, it is usually easier to define a non-deterministic finite automaton (NFA). The difference is that NFA can have several possible state moves for each state-input pair and picks one at random. So, it cannot match the right input always. However, we can say that it accepts input if there exists path within a graph, that accepts the whole input, or it accepts input if there is a non-zero probability of ending up in accepting state.

Let’s say we want to parse a∗∣aba. It’s an alternative of two regular expressions. The first could be expressed as:

and the second as:

Now, if we wanted to simply merge these two DFAs, we would have a problem: they both start with accepting a, so we would have to know beforehand which one to choose in order to accept a valid input. With NFA we can make some (even all!) of transitions non-deterministic, because we are checking if a path exists, and we don’t require that we will always walk it on valid input. So let’s say we have 2 valid choices from an initial state - with an empty string ϵ go into the first machine or the second machine (yes, we can use the empty string as well!):

(q0 and es were relabelled to distinct which one came from which automaton).

What would we have to do to make it deterministic? In this particular case, we can notice, that:

- correct input is either empty or starts with a,

- if it starts with a what comes next is either a sequence of more as or ba.

Let’s modify our NFA for that observation:

Let us think for a moment what happened here. We now have a deterministic version of a∗∣aba expression! In order to remove non-determinism, we had to look ahead to determine which branch we should go with. That was achieved by introducing and using extra states - which could be mapped into a corresponding state in either of branches - until we received the first piece of information, that would make it clear, which branch we need to go from now on because the other is no longer an option:

- if we just started we could assume that if nothing arrives we are OK,

- however if we got a, there is uncertainty - should we expect following a∗ or ba, so we carry on the information that a arrived and we will delay our decision,

- if nothing else arrives we are at valid input so we accept the state,

- if a or b arrives we finally resolved ambiguity - from now on, we can simply go into a branch directly copy-pasted from the original DFA that created.

Of course, this could be optimized a bit - states e, e1 and e2 could be merged into one, while states q0 and q1 are inaccessible so we can remove them. This way we would arrive at the final DFA.

The process, that we showed here is called determination of NFA. In practice, this tracing of things until we have enough data to finally decide, requires us to create a node for each combination of “it can go here” and “it can go there”, so we effectively end up building a powerset. This means that in the worst case we would have to turn our n-state NFA into 2n DFA before we even run optimizer! Once we’ve done that, testing if a string is accepted by the language is relatively cheap and requires going through the whole input once. This is an important observation, because using regular expressions is so simple nowadays, that we can forget that building a matcher is itself a costly operation (very costly).

That explains, why in older generations of compilers the proffered flow was to generate source code with already build DFA which could be compiled into a native code, that didn’t require any building in the runtime - you paid the cost of building DFA once before you even started the compilation of a program.

However, it is not the most comfortable flow, especially, since now we have a bit faster computers and a bit higher requirements about the speed of delivery and software maintenance. For that reason, we have 2 alternatives: one based on a lazy evaluation - you build the required pieces of DFA lazily as you go through the parsed input, or with the usage of backtracking. The former is done by simulating NFA internally and building DFA states on demand. The later is probably the easiest way to implement regular expression, though the resulting implementation is no longer Θ(n) but O(2n). (Confusingly, the algorithm is called NFA, though the implementation is not a finite-state automaton at all).

Regular expressions in practice

The format(s) used to describe regular expressions are directly taken from, how regular languages are defined: each symbol normally represents itself (so regexp a would match a), ∗/+ after an expression represent Kleenie star/plus (0 or more, 1 or more repetitions of an input - a* would match empty string, a, aa, …), concatenation of expressions represents concatenated language (aa would match aa, a+b would match ab, aab, …). Or ∣ or a sum of regular languages is represented by | (a|b matches a, b). Parenthesis can be used to clarify in which order regular expression are concatenated (ab+ means (ab)+, so if we wanted a(b+) the parenthesis helps us achieve what we want). There are also some utilities like [abc] which translates to (a|b|c) and allows us to use ranges instead of listing all characters manually (e.g. [a-z] represents (a|b|c|...|z)), ? which means zero or one occurrence (a? is the same as(|a)) or predefined sets of symbols like \s (whitespace character), \S (non-whitespace character) and so on. For details, you can always consult the manual for the particular implementation that you are using.

If you are interested about the process of implementing regular expressions and building finite state machines out of the regexp format I recommend getting a book like Compilers: Principles, Techniques, and Tools by Aho, Lam, Sethi, and Ullman. There are too many details about implementations which aren’t interesting to the majority of the readers to justify rewriting and shortening them just so they would fit into this short article.

Since we got familiar with RE, we can try out a bit more powerful category of languages.

Context-Free Grammars and Push-Down Automata

Any finite state machine can store a constant amount of information - namely the current state, which is a single element of a set of values defined upfront. It doesn’t let us dynamically store some additional data for the future and then retrieve data stored somewhere in the past.

An example of a problem, that could be solved, if we had this ability is checking if a word is a palindrome, that is you read it the same way left-to-right and right-to-left. Anna, exe, yay would be palindromes (assuming case doesn’t matter). Anne, axe, ay-ay would not be. If we wanted to check for some specific palindrome, we could use a finite state machine. But if we wanted to check for any? A. Aba. Ab(5-million b’s)c(5-million b’s)ba. No matter what kind of FSA we came up with is easy to find a word that it would not match, but which is a valid palindrome.

But let’s say, we are a bit more flexible than finite state automaton. What kind of information would be helpful in deciding if we are on the right track? We could, for instance, write each letter on a piece of paper, e.g. sticky notes. We met a, we write down a and stick it to someplace. Then, we see b, we write it down and stick it on top of a previous sticky note. Now, let’s go non-deterministic. It some point if we see the same letter arriving as we see on the top of the sticky notes stack, we don’t add a new one, but take the top one instead - we are guessing, that we are in the middle of a palindrome. Then each time top note patches with an incoming letter you take it off. If you had an even-length palindrome you should end up with an empty stack. Well, we would have to think a bit more to handle the odd-length case as well, but hey! We are on the right track as the length of the word is no longer an issue!

What helped us get there? We had a state-machine of sorts with 2 states: insert-card-mode (push) and take-matching-card-mode (pop) (for odd-length palindrome we could use a third state for skipping over one letter - the middle one - without pushing and popping anything). Then we had a stack that we can push things on top, take a look at the top element, and take an element from the top. Actually, this data structure (which could be also thought of as a last-in-first-out queue) is really named stack. In combination with finite state automaton, it creates push-down automaton (PDA).

As a matter of the fact, what we defined for our palindrome problem is an example of non-deterministic push-down automaton. We could define deterministic PDAs (DPDA) as a 7-tuple (Q,Σ,Γ,δ,q0,Z,F), where:

- Q is a finite set of states,

- Σ is a finite set of input symbols or an input alphabet,

- Γ is a finite set of stack symbols or a stack alphabet (because we can use different sets for input and stack, e.g. the later could be a superset of the former),

- a transition function δ:Q×Σ×Γ→Q×Γ∗, which would take the current state, the top (popped) stack symbol and next input symbol (possibly empty) to return the next state and what to push to stack (which can be represented as a word made of a stack alphabet) ,

- an initial state q0∈Q,

- an initial stack symbol Z∈Γ,

- a set of accepting states F⊆Q.

A non-deterministic version (NDPDA) would allow ϵ as a valid symbol in Σ, and return several possible values in transition function δ instead of one.

The palindrome example showed us that there are problems that PDA can solve that FSA cannot. However, PDA can solve all problems that FSA - all you need to do is basically ignore the stack in your transition function, and you get the FSA. Therefore, push-down automata are a strict superset of finite-state automata.

But we were supposed to talk about formal languages. Just like finite-state machines are related to regular languages, pushdown automata are related to context-free grammars. Reminder: it’s a formal language where all production rules are in the form of:

A→γ

where A∈N (non-terminals) and γ∈(N∪Σ)+ (non-terminals and terminals). Another reminder: with CFG we are parsing structures, that are basically trees (in case of programming languages, where this tree represent the language’s syntax it’s called abstract syntax tree), and terminals are these parts of the syntax which are leaves of the tree, while non-terminals are nodes. The names come from the fact, that when you expand the tree according to production rules, you have to end up with terminals in all leaves. They are the ends of the tree.

Thing is, when we are parsing, we are actually given a sequence of terminals, and we must combine them into non-terminals until we get to the root of the project. Kind of opposite to what we are given in language description. How could that look like? Let’s do some motivating example.

Normally when we describe the order of arithmetic operations like +, −, ×, ÷ we are inserting them in-between numbers. Because operations have priorities (×/÷ before +/−) if we want to change the default order we have to use parenthesis (2+2×2 vs (2+2)×2). This is called infix notation as the operator is between operands. But, you could use alternative notations: one where operator is before operands (prefix notation aka Polish notation) or after operands (postfix notation aka Reverse Polish notation/RPN). Both of them doesn’t require usage of parenthesis, as the order of operation is unambiguous due to their position. The later is quite useful when you are working with compilers.

(1+2)×(3+4)

becomes

1 2 + 3 4 + ×

When it comes to calculating the value of such expression, we can use stack:

- we start with an empty stack,

- when we see the number, we push it to the stack,

- when we see + we take the top 2 elements on the stack, we add them and we push the result to the stack,

- same with ×, take 2 top elements from the stack, multiply them and push the result to the stack,

- at the end the result of our calculation would be on top of a stack.

Let’s check for 1 2 + 3 4 + ×:

- we start with an empty stack,

- 1 arrives, we push it to the stack,

- stack is: 1,

- 2 arrives, we push it to the stack,

- stack is 1 2,

- + arrives, we take 2 top elements from the stack (1 2), add them (3) and push the result to the stack,

- stack is: 3,

- 3 arrives, we push it to the stack,

- stack is: 3 3,

- 4 arrives, we push it to the stack,

- stack is: 3 3 4,

- + arrives, we take 2 top elements from the stack (3 4), add them (7) and push the result to the stack,

- stack is: 3 7,

- × arrives, we take 2 top elements from the stack (3 7), multiply them (21) and push the result to the stack,

- stack is: 21,

- input ends, so our result is the only number on stack (21).

If you ever wrote (or will write) a compiler, that outputs assembler or bytecode, or something similar low-level - that’s basically how you write down expressions. If there is an expression in an infix form, you translate it into postfix, as it pretty much aligns with how mnemonics works in many architectures.

To be precise, quite a lot of them would require you to have the added/multiplied/etc values in registers instead of stack, however to implement a whole expression you probably use stack and copy data from stack to registers and vice-versa, but is an implementation detail irrelevant to what we want to show here.

Of course, the example above is not a valid grammar. We cannot have a potentially infinite number of non-terminals (numbers) and production rules (basically all results of addition/multiplication/etc). But we can describe the general idea of postfix arithmetics:

BinaryOperator→+∣−∣×∣÷ Expression→Number∣Expression Expression BinaryOperator

We have terminals Σ={Number,+,−,×,÷} and non-terminals N={BinaryOperator,Expression}. We could create an ADT:

sealed trait Terminal

final case class Number(value: java.lang.Number)

extends Terminal

sealed trait BinaryOperator

case object Plus extends BinaryOperator with Terminal

case object Minus extends BinaryOperator with Terminal

case object Times extends BinaryOperator with Terminal

case object Div extends BinaryOperator with Terminal

sealed trait Expression

final case class FromNumber(number: Number) extends Expression

final case class FromBinary(operand1: Expression,

operand2: Expression,

bin: BinaryOperator)

extends Expression

and now it should be possible to somehow translate List[Terminal] into Expression. (Assuming the input is a correct example of this grammar - if it isn’t we should fail). In this very simple example, it could actually be done in a similar way we evaluated the expression:

- if Terminal is Number, wrap it with FromNumber push it to the stack,

- if Terminal is BinaryOperation, we take 2 Expressions from the stack, put it as operand1 and operand2, and together with BinaryOperator put it into FromBinary and push to stack,

- if the input is correct, we should end up with a stack with a single element,

- if the input is incorrect, we should end up with a stack with more than one element, or during one of the operations we will miss some Expressions while popping on a stack.

It is almost enough to represent our language as PDA. To create a binary operation we look at the two elements on top of the stack, while it is legal to only know one. But we could, represent that as a state. Initial stack symbol could be a single EmptyStack. Actually, we could also make sure that we end up with an empty stack at the end - if there are elements on stack, it’s an error (becasue no operator consumed some elements). If at some point we are missing some elements it’s also an error. We could end up with something like:

This PDA doesn’t calculate the value of RPN. It only checks if it is valid. We are pushing Numbers on a stack, and on a binary operation, we consume 2 Numbers from the stack. At any point we can start “checking” - if we are at the end of input, a stack is empty (meaning that EmptyStack is the top element) we can assume that the input was correct so we move to OK through CheckingMode. However, if we start checking and there is some input left or there are elements on the stack - we are erring.

To make sure we understand what happened here we should remember that this is non-deterministic PDA - so for each valid input there should exist a valid path (and each path ending in an accepted state should describe a valid input), but we don’t have to necessarily walk it each time. The other thing is that on each step of PDA we have to pop from stack - if we don’t want to change stack we have to pop the same element back, if we want to add something we can pop 2 elements or more and if we want to get rid of top elements, then we simply don’t pop it back.

Parsers in practice

Actually, there are 2 approaches to parsing context-free grammars:

- top-down approach: We start from the root of the AST tree and take a look at the possible transitions. We try to make a prediction - if we get the next alphabet element, do we know, which transition to go? If that is not enough you could try to look at transitions going from these transitions and check if any prediction is possible to do know, etc. We don’t necessarily look 1 symbol ahead to determine our path - we could set some k and assume that we can look up to k symbols ahead before making a decision (which would be potentially reflected in the number of states). If our language contains recursion it might affect how we can and what is the minimal number of lookahead to decide. We are parsing input left-to-right, and the top-down strategy with lookahead will make us choose branch basing on leftmost non-terminal. That is why this approach is called LL (left-to-right, leftmost derivation). The LL parser with k tokens lookahead is called LL(k),

- bottom-up approach: We start with terminals and look at the production rules in reverse - we try to combine incoming terminals into terminals and then terminals and non-terminals until we get to the root. (This is what we have done in the PDA example above). Just like with LL we might need to make some predictions so we can look ahead of k elements. Just like with LL we read left-to-right. However, contrary to LL we can make a decision when we get the last element of a production rule, rightmost non-terminal. This is why this approach is called LR and if our parser requires k tokens lookahead it is an example of LR(k) parser. For k=1 we can use some specific, simple implementation like Simple LR (SLR). There is also general implementation of LR(k) called look-ahead LR(k) (LALR(k)) which, for k=1 are called simply LALR.

Both approaches are usually used to build a parsing table, though they differ in how you arrive at the final table.

With LL(k) you can pretend that you can look ahead k chars while simply applying production rules - that k-symbol lookahead is simulated by adding additional states. When we simulate seeing kth symbol ahead, we are actually already at this symbol, but with state transitions arranged, so that we end up in a state that we should end up if we really were k symbols ago and made the decision based on a prediction. Notice, that for k=1 this basically means that we are always following first matching production rule and never going back, which results in a quite simple parser.

LR(k), on the other hand, uses things called shift and reduce. Shift advances parsing by one symbol (shifts it by one symbol) (which doesn’t apply any production rule), while reduce combines (reduces) several non-terminals and/or terminals into a single terminal (goes into the reverse direction of production rule). When an algorithm generates such a table for an input we passed it, we might see a complaint about shift-reduction conflict - since well-defined LR grammar should for each PDA assign either a shift operation or a reduce operation, it shows that there is an ambiguity in the grammar, that the parser generator managed to resolve (and produce a working code), but which will bite us by parsing some inputs not the way we wanted.

For defining context-free grammars parser generators quite often use syntax heavily influenced by (extended) Backus-Naur form ((E)BNF). In EBNF, the previous example:

BinaryOperator→+∣−∣×∣÷ Expression→Number∣Expression Expression BinaryOperator

could look like this:

binary operator = "+" | "-" | "*" | "/" ;

digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;

number = number, digit | digit ;

expression = number, | expression, expression binary operator ;

Notice, that here terminal symbols are defined as digits. It might be quite inconvenient, which is why a lot of parser generators would rather:

- assume that terminals are results of regular expression matching - the input would be matched against a set of regular expressions, each of which would be related to a terminal symbol. We would require them to accept whole input as a sequence of words matched by any of regular expressions. This way we would turn a sequence of input symbols into a sequence of terminal symbols. The part of a program responsible for this tokenization is called lexer. Such approach is seen e.g. with parser generators based on lex (lexer) and yacc (Yet Another Compiler-Compiler) and their GNU reimplementations flex (free lex) and bison (an allusion to gnu as a lot of GNU tooling is based on bison). (It should also explain why certain languages have weird rules regarding class/method/function/variable names - since tokenization takes place in the very beginning, it has to reliable classify each piece of code unambiguously as a terminal symbol),

- alternatively allow you to use regular expressions directly in a parser-defining syntax. As this approach is much more readable it was also used in parser combinators.

Right, we haven’t mentioned parser combinators. What are they, and why they became more popular recently?

Parser combinators

When computer resources were really scarce, we didn’t have the comfort of building parsers in the most convenient way - the idea behind parsing generators was generating fast, ready to use PDA which would parse input with linear time and memory (that is, directly proportional to the input). Overhead had to be limited to the minimum, so the best way was to do all the calculations (both lexing and parsing) during code generation, so when we would run the program, it would be able to parse as soon as the code was loaded from the disk to the memory. All in all generating imperative code was the way to go.

But nowadays the situation is different. We have much faster computers with a lot more memory. And the requirements we have regarding programs are much higher, so the process of validating the parsed input became much more complex - so small overhead for parsing is not as painful. Additionally, we made much more progress when it comes to functional programming.

This opened the gate to an alternative approach called parser combinators (which is not that new considering, that it was described in Recursive programming Techniques by Burge from 1975 as parsing functions). What we do is basically, a function composition.

Let’s try by example. This time we’ll try to implement infix syntax. At first we’ll do something about lexing terminal symbols (and using spaces for separation):

def number(input: String) = """\s*([0-9]+)(\s*)"""

.r

.findPrefixMatchOf(input)

.map { n =>

val terminal = Number(n.group(1).toInt)

val unmatched = input.substring(n.group(0).length)

terminal -> unmatched

}

def plus(input: String) = """\s*(\+)(\s*)"""

.r

.findPrefixMatchOf(input)

.map { n =>

val terminal = Plus

val unmatched = input.substring(n.group(0).length)

terminal -> unmatched

}

def minus(input: String) = """\s*(-)(\s*)"""

.r

.findPrefixMatchOf(input)

.map { n =>

val terminal = Minus

val unmatched = input.substring(n.group(0).length)

terminal -> unmatched

}

def times(input: String) = """\s*(\*)(\s*)"""

.r

.findPrefixMatchOf(input)

.map { n =>

val terminal = Times

val unmatched = input.substring(n.group(0).length)

terminal -> unmatched

}

def div(input: String) = """\s*(\/)(\s*)"""

.r

.findPrefixMatchOf(input)

.map { n =>

val terminal = Div

val unmatched = input.substring(n.group(0).length)

terminal -> unmatched

}

It’s quite repetitive so we can introduce a helper utility:

type Parser[+A] = String => Option[(A, String)]

object Parser{

def apply[A](re: String)(f: String => A): Parser[A] =

input => s"""\\s*($re)(\\s*)""".r

.findPrefixMatchOf(input)

.map { n =>

val terminal = f(n.group(1))

val unmatched = input.substring(n.group(0).length)

terminal -> unmatched

}

}

and simplify definitions a bit:

val number = Parser[Number]("""[0-9]+""")(n => Number(n.toInt))

val plus = Parser[Plus.type]("""\+""")(_ => Plus)

val minus = Parser[Minus.type]("""-""")(_ => Minus)

val times = Parser[Times.type]("""\*""")(_ => Times)

val div = Parser[Div.type]("""\/""")(_ => Div)

then we could start combining them:

val binaryOperator: Parser[BinaryOperator] = in => {

if (in.isEmpty) None

else plus(in) orElse minus(in) orElse times(in) orElse div(in)

}

The curious reader might notice that this is a good candidate for a ReaderT/Kleisli composition, but we’ll try to keep this example as simple as possible. That is why we’ll create some specific utility for this case:

implicit class ParserOps[A](parser: Parser[A]) {

// making another by-name param helps to prevent

// stack overflow in some recursive definitions

def |[B >: A](another: => Parser[B]): Parser[B] =

input => if (input.isEmpty) None

else parser(input) orElse another(input)

}

and rewrite binaryOperator as:

val binaryOperator: Parser[BinaryOperator] =

plus | minus | times | div

Now we are missing the concatenation - or moving input forward as we matched something already:

def expression: Parser[Expression] = {

val fromNumber: Parser[FromNumber] = in => {

number(in).map { case (n, in2) => FromNumber(n) -> in2 }

}

def fromBinary: Parser[FromBinary] = in => for {

(ex1, in2) <- (fromNumber | inParenthesis)(in)

(bin, in3) <- binaryOperator(in2)

(ex2, in4) <- (fromNumber | inParenthesis)(in3)

} yield FromBinary(ex1, ex2, bin) -> in4

fromBinary | fromNumber

}

def inParenthesis: Parser[Expression] = in => for {

(_, in2) <- Parser[Unit]("""\(""")(_ => ())(in)

(ex, in3) <- expression(in2)

(_, in4) <- Parser[Unit]("""\)""")(_ => ())(in3)

} yield ex -> in4

If we tested that code (which now looks like a candidate for a state monad) we would find that it parses one step of the way (so it doesn’t run recursion infinitely):

expression(""" 12  + 23 """)

res1: Option[(Expression, String)] =

Some((FromBinary(FromNumber(Number(12)), FromNumber(Number(23)), Plus), ""))

We can prettify the code a bit:

implicit class ParserOps[A](parser: Parser[A]) {

def |[B >: A](another: => Parser[B]): Parser[B] =

input => if (input.isEmpty) None

else parser(input) orElse another(input)

def &[B](another: => Parser[B]): Parser[(A, B)] =

input => if (input.isEmpty) None

else for {

(a, in2) <- parser(input)

(b, in3) <- another(in2)

} yield (a, b) -> in3

def map[B](f: A => B): Parser[B] =

input => parser(input).map { case (a, in2) => f(a) -> in2 }

}

def expression: Parser[Expression] = {

def fromNumber =

number.map(FromNumber(_))

def fromBinary =

((fromNumber | inParenthesis) &

binaryOperator &

(fromNumber | inParenthesis)).map {

case ((ex1, bin), ex2) => FromBinary(ex1, ex2, bin)

}

fromBinary | fromNumber

}

def inParenthesis: Parser[Expression] =

(Parser[Unit]("""\(""")(_ => ()) &

expression &

Parser[Unit]("""\)""")(_ => ())).map {

case ((_, ex), _) => ex

}

expression(""" 12 + 23 """).map(_._1).foreach(println)

// FromBinary(FromNumber(Number(12)),FromNumber(Number(23)),Plus)

(Complete example you can see on gist).

Not bad! It already shows us the potential of creating small parsers and composing them as higher order functions. It should also explain to us why such a concept was named parser combinators.

But, can we have parser combinators out-of-the-box? We would need an implementation which:

- is statically typed,

- gives us concatenation &, alternative |, and mapping of parsers,

- let us gives suggestions if certain matching should be greedy (match whatever it can, potentially indefinitely) or lazy (finish ASAP),

- is probably more complex than a simple function from input into output with the unmatched part. It could e.g. make use of lookahead,

- give us a lot of utilities like e.g. regular expression support.

Luckily for us, such implementation already exists, so we can just use it. FastParse is a parser combinator library written by Li Haoyi (the same guy who created Ammonite and Mill). While it provides us a nice, functional interface, it uses Scala macros to generate fast code with little overhead (which gives us hardly any reason for considering parser generators, at least for Scala).

Our parser can be rewritten into fastparse this way:

// import $ivy.`com.lihaoyi::fastparse:2.1.0`

import fastparse._

import ScalaWhitespace._ // gives us Scala commens

// and whitespaces out-of-the-box

object Parsers {

// terminals

def number[_ : P] =

P( CharIn("0-9").rep(1).!).map(n => Number(n.toInt) )

// ! makes parser catch input as String

def plus[_ : P] = P("+").map(_ => Plus)

def minus[_ : P] = P("-").map(_ => Minus)

def times[_ : P] = P("*").map(_ => Times)

def div[_ : P] = P("/").map(_ => Div)

// non-terminals

def binaryOperator[_ : P] = P(plus | minus | times | div)

def fromNumber[_ : P]: P[FromNumber] =

P(number.map(FromNumber(_)))

def fromBinary[_ : P]: P[FromBinary] =

P(((fromNumber | inParenthesis) ~

binaryOperator ~

(fromNumber | inParenthesis)).map {

case (ex1, op, ex2) => FromBinary(ex1, ex2, op)

})

def expression[_ : P] =

P(fromBinary | fromNumber)

def inParenthesis[_ : P] =

P("(" ~ expression ~ ")")

def program[_ : P] = P( (expression | inParenthesis ) ~ End)

}

parse("12 + 23", Parsers.program(_))

Before we jump the hype train - parser combinators are not equal to LL parsers and/or LR parsers. As we saw, we could define a parser accepting reverse Polish notation. However, if we tried to write a parser combinator that would accept it, then we would find, that recursive definition of expression would translate into a recursive function call without a terminating condition (parser combinators are just higher-order functions after all). LL or LR parser would push a symbol on the stack and take an input symbol from the input sequence, so at some point, they would have to stop (at least when the input finished). A parser combinator would need some hint e.g. closing block (which means that usually, it is not a problem), but we can see that parser combinators are not covering all context-free grammars.

Actually, LL parsers are not equal to LR parser either. Seeing how they work, one might argue that LL parsers correspond to Polish notation (because they make a decision at leftmost symbol - a prefix) while LR corresponds to reverse Polish notation (because they take a decision at rightmost symbol - a postfix). (See a nice post about it: LL and LR parsing demysitfied). Both can be treated as special cases of PDA, while it a set of all PDAs that corresponds with a whole CFG set.

Turing machines, linear-bounded automata, unrestrained and context-sensitive grammars

For the sake of completion, we can mention remaining computational models and grammar types, though this post is supposed to talk about parsing, so I’ll try to keep it short.

Turning machines and unrestrained grammars

A finite state machine at any given time remembers only in which one of a finite number of states it is. We read each symbol in the input once.

A push-down automaton remembers a current state and the last thing it put on a stack - it can “remember” things from a stack in reverse order in which it stored them there for later. You cannot remember something from the middle of the stack without forgetting everything that was stacked before it. In a way, you can think that you can read each input element twice - once in incoming order, once in reverse order, and the only nuance is how you entangle these two modes.

A Turing machine (defined by Alan Turing, the same guy who designed cryptologic bombe against German Navy’s improved Enigma, the cryptologic bombe against original Enigma was designed by Polish Cipher Bureau) improved upon, that by using infinite tape, where the automaton could read and store symbol in one cell of that tape, and then move forward or backward. This allows us to “remember” something as many times as we need it.

Because of that ability to read the thing as many times as we want, it is possible that your machine will get into an infinite loop and never end. The question whether we can guess if a specific machine will ever return for a given input is called the halting problem (HP) and is proven to be in impossible to solve for a general case. The proof assumes, that you have a program that could use the halting problem solver on itself and loop if solvers says it should return and returns if solvers says it should loop - so it shows by contradiction that such thing cannot be constructed. A halting problem is used in a lot of proofs, that certain problem is impossible to solve - a reduction from the halting problem makes you use that problem to solve HP - since it is impossible to solve HP the problem is also unsolvable.

Turing machines are equal to unrestrained grammars, that is formal grammars that have no restriction about how you define a production rule. They are also equivalent to lambda calculus, register machine, and several other models. Usually, if we want to have a universal programming language, we make it Turing-complete (equal in power to TM, allowing you to simulate TM on it).

Linear-Bounded Automata and Context-Sensitive Grammars

Between push-down automata and Turing machines lies linear-bounded automata (LBA). I decided to describe them after TMs because they are basically restricted form of TM. It puts some limits on both sides of the infinite tape, that your automaton cannot cross.

It was proven that LBAs are equal to context-sensitive grammars, that is grammars in the form of:

αAβ→αBβ

meaning that you can turn A into B only if it appears in the context of α and β.

Back to parsing

Majority of programming languages are Turing-complete. However, the first part of interpretation or compilation doesn’t require that we have this much power.

Some very simple interpreters can be build when you lexing (tokenization) and parsing and on reduction you immediately evaluate the computation inside parser. However, it is quite messy to maintain in the long run.

After all, parsers and context-free grammars can only take care of syntax analysis. So, you could preserve the results of syntax analysis into a data structure - abstract syntax tree - and then perform semantic analysis. Was variable with this name already defined? Is this identifier describing a class, object, constant? Actually, when you take into consideration how complex some of these things are, you might not be surprised, that certain compilers could decide to introduce several steps of a whole compilation process - just for verifying, that the AST is correct. scalac has over 20 phases in total:

$ scalac -Xshow-phases

phase name  id  description

----------  --  -----------

parser   1  parse source into ASTs, perform simple desugaring

namer   2  resolve names, attach symbols to named trees

packageobjects   3  load package objects

typer   4  the meat and potatoes: type the trees

patmat   5  translate match expressions

superaccessors   6  add super accessors in traits and nested classes

extmethods   7  add extension methods for inline classes

pickler   8  serialize symbol tables

refchecks   9  reference/override checking, translate nested objects

uncurry  10  uncurry, translate function values to anonymous classes

fields  11  synthesize accessors and fields, add bitmaps for lazy vals

tailcalls  12  replace tail calls by jumps

specialize  13  @specialized-driven class and method specialization

explicitouter  14  this refs to outer pointers

erasure  15  erase types, add interfaces for traits

posterasure  16  clean up erased inline classes

lambdalift  17  move nested functions to top level

constructors  18  move field definitions into constructors

flatten  19  eliminate inner classes

mixin  20  mixin composition

cleanup  21  platform-specific cleanups, generate reflective calls

delambdafy  22  remove lambdas

jvm  23  generate JVM bytecode

terminal  24  the last phase during a compilation run

By the way, this is a good moment to mention what compilation actually is. From the point of view of formal languages theory, a compilation is just translation work from one formal grammar into another. Scala into JVM byte code, C++ into binary code, Elm into JavaScript, TypeScript into JavaScript, ECMAScript 6 into ECMAScript 5… There is no need to introduce something like transpiler to describe compilation from one language to another. If we would use this word, then only to specify a compiler that translates into another high-level language, not because a compiler doesn’t cover that case.

Interpreter would be something, that instead of translating into another formal grammar, translates directly into a computation. However, if we assume that we want to be pure, we would return something, that could be turned into a computation - e.g. free algebra. That explains, why Typed Tagless Final Interpreter has interpreter in its name, even though it doesn’t necessarily run computations immediately.

Separation of phases serves two purposes. One is maintainability. The other is that we can separate front-end of a compiler (parsing and validating AST) and back-end (using AST to generate output). For instance, in case of Scala, we can have one front-end and several back-ends: JVM Scala, Scala.js and Native Scala (though, truth to be told Scala.js and Native Scala need to expand a language a bit).

If we go fully functional with all the phases (so each phase is a function working on AST element), then we have option to compose functions (phase fusion) - if our language of choice allows us to optimize combined functions, then we can obtain a compiler which is both maintainable and performant.

Of course, the parser doesn’t have to be a part of a compiler. The resulting tree might be our goal after all. XML, JSON or YML parsers exist in order to take some text representation and turn it into a tree of objects that is easier to work on. Notice, that grammars of languages like XML or HTML are too complex to be handled by something like regular expression, so if you want to use it, you’d better grab a parser.

Error handling

If you want to discover and return to a user all errors, possibly with some meaningful description of what could go wrong and how they could fix it, it is problematic.

As you noticed our naive parser combinator simply returned unmatched part of the input - hardly helpful. fastparse is slightly better - it can also tell you around which character things broke and what terminal/non-terminal it was (especially if you use cuts, to tell the parser where it should not perform a backtracking).

With parsers created by generators, the situation is similar - out of the box you usually only get the information that parsing failed. So, how come all these compilers and document parsers can give you meaningful messages? More meaningful than things broke at n-th character?

When it comes to parser generators like Bison, you have a special terminal symbol - error. When you match it, you can extract the position in text and create an element of AST which could mock the element, that should be there but put the error element instead. Whether you do it by having each element of AST being a coproduct of valid and invalid version, or if you will make each step of the way something like (A,B,C)→Either[Error,Element] is up to you. If that sound, as if you had to predict all possible ways a user can break the code and putting error there - that is exactly what you have to do there.

With parser combinators like fastparse, things are similar - you can tell the parser to consume some input after your current match, so you could e.g. try to match all right cases and - if they fail - consume the part of the input, that would fail and turn it into invalid AST element version.

Now, you should understand why this is not something, that you get for every language and why only some of them have user-friendly error handling. It increases the effort related to parser maintenance tremendously.

Summary

In this post, we had a rather high-level overview of parsing (text) into AST. We briefly talked about the Chomsky hierarchy and relations between regular languages and different models of computation. We talked a little more about regular languages and context-free grammars, though without an in-depth description of algorithms used to create those.

How regular languages, computational models and compilers work in greater detail you can learn from books like Compilers: Principles, Techniques, and Tools by Aho, Lam, Sethi, and Ullman or Structure and Interpretation of Computer Programs by Abelson, Sussman, and Sussman.

I hope, that it will help you appreciate how many thoughts and effort went into letting us build REPLs, a plethora of languages - and all of that with much better syntax, than those of the first programming languages, which very unwelcome. This unblocked us from thinking about how to design the language to be friendly and readable. And, while we are still trying to figure out better ways of designing our tools, we should remember that all of that was possible thanks to pioneers in the formal languages theory.

His Biggest Hit Sold More Copies Than Any of the Beatles’. So Why Haven’t You Heard of Him?

Photos by Sami Kent | Edited by Brendan Spiegel

Twenty years ago the man who recorded one of the most successful songs of all time was thrown off a motorbike by a car in Calabar, Nigeria. He hit his head on the road and was rushed to the hospital, where he lay for two weeks, in and out of consciousness, but deteriorating all the time. On June 24, 1997, Prince Nico Mbarga was pronounced dead. “Sweet Mother,” his 1976 one-hit wonder, had sold at least thirteen million copies across the African continent – more than The Beatles’ bestseller “I Want to Hold Your Hand.” But no global media outlet thought to cover the life and death of the artist behind Africa’s most popular song. https://www.youtube.com/watch?v=3mecNrIaWOA   Today, the only internet accounts of his life reach around four paragraphs and bookend Mbarga’s career with two big political events of the time: the Biafran War in 1967 that saw him, at 17, flee across the border to Cameroon, where he mastered the guitar; and the expuls…

Netlify Celebrates 5 Million Developers | 3D Game

One of the things in my big Atari ST haul from a couple months ago was an issue of FaSTer disk magazine. I had thought that it was a sealed copy, but although it was still in its shrink-wrap, it was not actually sealed.

FaSTer was a Canadian magazine and what made it unique is that it was distributed on floppy disk1!

I had never heard of this magazine and was not all that familiar with this type of magazine distribution, but I think it is pretty neat. Being a digital magazine, it’s a lot like Goto 10!

Each disk was single-sided and worked in both color and monochrome. Producing a disk magazine was certainly much less expensive than a printed one, yet the magazine itself did cost more than most printed magazines of the time. Although the price was in line with magazines that came with a disk, such as Compute!’s ST Disk and Magazine or STart. At $9.95 an issue, that is about $28 in 2024.

The magazine itself consisted of just a disk nested between two pieces of cardboard. The front “cover” highlights what is on the disk.

I have Volume #2, Issue #2. I’m not exactly sure how many issues were actually published, but ST News has a collection of some other issues, including:

The blurb from ST News says this:

F.A.S.T.E.R.: The magazine that started everything with regard to a neat user interface - one of the very earliest ST disk magazines, having started somewhere in the autumn of 1986. It was Canadian of origin, and started out bilingually with a set of English articles and their copies in French. Later issues were English only. They were the first disk magazine with a user interface, however they survived only a little over a year - probably because they were commercial, which tends to make things more complicated than they need be. The "F.A.S.T.E.R." user group lived on longer than the magazine.

Status: Commercial.

User interface: Yes. A custom one (the first one).

Latest known issue: Volume 2 Issue 5.

Language: Used to be French and English. Later issues were only in English.

Looking at FaSTer

Happily the disk I have actually worked and it even runs on my Mega STE! I was able to archive it and made it available to Richard at ST News. You can also download it from here as an MSA (Magic Shadow Archive):

FASTERv2n5

Here is a walkthrough what the magazine actually looks like when viewing it on the ST in medium resolution color.

The File menu offers an option to print the magazine. This seems crazy to me today, but back then I do remember printing lots of things, especially program listings. So having a printout of the magazine to read outside on the porch would have been cool. I wonder how long that would have taken to print on a 9-pin dot matrix printer, though? Bzz… Brappp, Bzzz…

The about screen has a cute little devil logo. Not really sure what that has to do with the magazine.

The menu bar at the top is hidden until you move the mouse up to it. As you can see below, there are quite a few sections to the magazine. Overall, the app for interacting with the magazine is nicely done.

This is the main editorial, which is presented as slightly formatted text. I did not notice any embedded graphics in the articles. To read an article you have to scroll down using the scrollbar on the right. I sure do miss not having a scroll wheel on my ST!

This is the “centerfold” that is mentioned on the packaging. I guess it is supposed to be David Bowie?

The Personal Option column is about public domain software.

There is an interview with a company called Arrakis, a nice Dune reference.

The Pascal tutorial is about how to make it like C. Not sure why you’d want to do that…

There is a column covering new products and there would have been plenty in 1986, which was the ST’s heyday.

There is a review of the shoot-em-up, GoldRunner.

I’ve been meaning to play Sundog so I’m looking forward to more thoroughly reading this introduction to it.

There is also a column highlighting an Atari user group and the one for this issue is the original Atari Computer Enthusiast (ACE) group that I wrote about earlier this year.

These were the subscription prices for FaSTer. Don’t forget to factor in inflation, which makes the $50/year price be about $144 in 2024. Compared to that, Goto 10 is a bargain!

The cribbage game did not launch from the disk menu, but I was able to run it separately from the desktop. It has a title screen and large cards.

It’s pretty colorful and easy to play. The score is counted automatically. I did end up losing to the computer, although in my defense I haven’t played cribbage in a while.

You can try the FaSTer disk magazine using your favorite emulator just by mounting the disk images linked above. Enjoy!

Ask Hacker News

- Recommend any clipboard app for Mac

- Free Qwen 2.5, Llama Nemotron, OpenBioLLM, Gemini Exp 1121, and LearnLM 1.5 Pro

- Ask HN: How else to live more minimally?

- Ask HN: Discords or chat groups for builders/hackers?

- Ask HN: Typed "AirPods" in notes on iPhone, 2 hours later AirPods ad on YouTube?

- Ask HN: Best non-fiction book you read in 2024?

- AIMagicStudio – Turn Keywords or URLs into Videos in Minutes

- Ask HN: What gives Elon Musk's companies their edge?

- Ask HN: A monitoring app for new comments on your favorite HN posts

- Where Can I Find San Francisco Bay Area Angel Investors for My Podcast?

- Is there demand for free product manager advisory services in startup community?

- Ask HN: Is distributed LLM training in browsers (WebRTC and WebGPU) possible?

- Ask HN: To those with successful browser extension(s), how did you grow it?

- Ask HN: Will more people leave cities due to satellite internet in rural areas?

- Ask HN: How do you manage ideas, tasks, notes and other stuff?

- Our SaaS hit 50 paying subscribers today. Here's how we did it with $0:

- How to build something people will pay you for, in 5 steps

- Ask HN: If every company is using AI now, how to set your startup apart?

- Ask HN: How do I get off marketing email lists from IT vendors?

- Ask HN: Do you want more software developer in your team?

Recommend any clipboard app for Mac

news.ycombinator.com/item?id=42219580

1 point by bqc 17 minutes ago | hide | past | favorite | discuss

Free Qwen 2.5, Llama Nemotron, OpenBioLLM, Gemini Exp 1121, and LearnLM 1.5 Pro

news.ycombinator.com/item?id=42219384

1 point by gizai 1 hour ago | hide | past | favorite | discuss

Giz.ai has announced free access to some of the most advanced AI models available today. This update includes Alibaba's Qwen 2.5 (72B) and Qwen 2.5 Coder (32B), Nvidia's Llama 3.1 Nemotron (70B), and the Llama 3.1 OpenBioLLM (70B) specialized in medical applications. Additionally, Google's experimental Gemini models—Exp 1121, Exp 1114—and the LearnLM 1.5 Pro, designed for educational applications, are now available.

Explore the models at https://www.giz.ai

Ask HN: How else to live more minimally?

news.ycombinator.com/item?id=42219232

2 points by purple-leafy 2 hours ago | hide | past | favorite | 6 comments

So far

No social media. None. Unless you count HN. Using job search sites for jobs, not LinkedIn.

⃠ Facebook

⃠ LinkedIn

⃠ Reddit

⃠ TikTok

No news.

⃠ Global news

⃠ Local news

Browsing

Non-Google search engine

Private email service w/ aliases

Password manager

Ad-blocker

URL black-listing

Devices

No smartphones. They are modern slot-machines.

Separate devices by function.

Flip phone / Brick phone

Basic MP3 player

Pocket notebook to write in.

solardev 1 hour ago | next [–]

IMHO: I don't think living minimally is solely (or maybe even primarily) about what you cut out, but how you spend your time and value your life.

If you want to live locally, I think it's more about building connections and community – local friends, shops, artists, musicians, workout studios, businesspeople, industry, ranchers, farmers, students, museums, city councils, etc. And that can look very different depending on where you live. "Local" in the Bay Area is going to be mostly techies. "Local" in rural middle-America is going to be ranchers and farmers. "Local" in some small outdoorsy town will be a lot of hikers, skiers, bikers, etc.

The thing is... people can have all the things you mentioned above, but still live "minimally" because they don't focus on those things. They don't give more than a passing thought to those things, whether they have them or not. Their lives are too full of the things and people they love the most.

It's not about what you cut out, but what you embrace. Chase (or find) your passions and the rest just naturally fall aside.

GianFabien 1 hour ago | prev | next [–]

Tried living with a Nokia 2720 flip-fone. It was Ok, but I missed Google Maps and don't particularly want to carry a street directory. So using a basic Android smartphone with no social media, banking, etc on it. Probably use it no more than 15-20 minutes a day.

From the title I thought minimally meant like in a log cabin deep in some woods or on a mountain top. To answer your question, don't load apps that you don't really need and use a bit of discipline to curtail unproductive time on the browser, etc.

__warlord__ 1 hour ago | prev | next [–]

What’s your goal in living minimally?

purple-leafy 1 hour ago | parent | next [–]

Happiness, less emotional/politic manipulation, removal of negative media.

Shift to local-only thinking and living.

Independence.

Present in the day-day.

Breaking of bad habits.

Living deliberately

talldayo 2 hours ago | prev [–]

I don't feel like you need much to "be happy" in the tech sense. For the past 5 years I've been using Linux, reading HN and RSS for news and using an Android smartphone without debilitating social side effects. I don't scroll feeds, I wait to read texts until respectful times and I have a Gmail that has a few thousand marketing emails I never read. Some nights I watch a couple hours of YouTube (gasp!).

A lot of solutions to the harm tech causes can be remediated by figuring out how these things affect you. Obsessing over the news makes me upset - when I realize this, I divert my attention to other things. If you can't foster the willpower to resist these ills on your own, you're not prepared to use devices with a healthy mindset in the first place.

purple-leafy 1 hour ago | parent [–]

I don't think one can use a smart phone without being affected by advertising, the media, or social media, or other agents of manipulation/negativity/control.

Our devices are perfectly engineered slot-machines. I've tried many times to overcome their effects through willpower alone. Doesn't work long term.

Look around you next time you're outdoors. People are glued to their phones.

I think its crazy that people don't realise this or work to mitigate access to such devices.

And no news is good news, no point staying informed about all the nasty shit in the world. What good is knowing?

Ask HN: Discords or chat groups for builders/hackers?

news.ycombinator.com/item?id=42219218

1 point by drdeafenshmirtz 2 hours ago | hide | past | favorite | discuss

Anyone know any active discord groups for developers/hackers who like to share projects they're working on, talk through problems, beta test each others' projects, etc?

Ask HN: Typed "AirPods" in notes on iPhone, 2 hours later AirPods ad on YouTube?

news.ycombinator.com/item?id=42219150

4 points by andrewstuart 3 hours ago | hide | past | favorite | 6 comments

I asked my son what he wanted for Xmas. I made a note in IiPhone with heading "Presents" and a few other lines including "AirPods".

Two hours later YouTube showed me an ad for Airpods.

I asked if anyone had searched for AirPods and they said no.

Connection or coincidence?

runjake 2 hours ago | next [–]

Coincidence. You have an iPhone. YouTube knows that and a bunch more about you. Also everyone gets AirPods ads.

legitster 2 hours ago | prev | next [–]

Coincidence.

I work in marketing technology and no such connection exists.

People are, however, predictable. It is Christmas season and you are both iPhone users. Both of you are saturated in ads for Airpods before and after it was on the list.

derwiki 2 hours ago | prev | next [–]

What other ads did you see in the same time period?

andrewstuart 2 hours ago | parent | next [–]

Pretty much none, maybe one or two.

Turned on YouTube two hours later, there was the ad.

chrsw 2 hours ago | prev | next [–]

You might not even have had to make the note

iab 2 hours ago | prev [–]

Someone typed “AirPods” into google is my guess

Ask HN: Best non-fiction book you read in 2024?

news.ycombinator.com/item?id=42218828

44 points by vintageclothldn 4 hours ago | hide | past | favorite | 32 comments

Tyler Cowen just published his best non-fiction books of 2024 list on Marginal Revolution: https://marginalrevolution.com/marginalrevolution/2024/11/best-non-fiction-of-2024.html

The best non-fiction books I have read this year are:

"Six Pillars of Self-Esteem" by Nathaniel Branden - Interesting insights into the concept of self-esteem and how it is influenced by the wider world

"The Code Book" by Simon Singh - An oldie but saw it recommended last year on HN and found it to be a fascinating look at the history of cryptography, written in an engaging and approachable way

"Poor Charlie's Almanack: The Essential Wit and Wisdom of Charles T. Munger" by Charles T. Munger - After Charlie's unfortunate passing, I read this again. So much life wisdom on every page

What's the best non-fiction book you read in 2024?

Meleagris 4 hours ago | next [–]

I really enjoyed reading "The Blue Machine: How The Ocean Works".

It turns out the Ocean is fascinating, and I learned something crazy:

"Between 1950 and 1973, world fish harvest trippled, but the amount of fish directly consumed by humans stayed the same. The rest went into fishmeal, as a supplemental food for livestock, and this became an essential ingredient for modern industrial farming".

I didn't realize that fishmeal was a primary input to modern animal agriculture. Global fish stocks collapsed not only because people ate fish, but also because of animal agriculture in general. It's fascinating how it's all connected.

Also, sea turtles cry 8 litres of tears an hour.

Needless to say, this book ended up as a permanent fixture on my bookshelf.

tuna-piano 4 hours ago | prev | next [–]

Without a doubt "Endurance: Shackleton's Incredible Voyage" by Alfred Lansing.

Read it, read it now.

It's the kind of book that you once you read, you'll never forget. 4.8 stars with 23k reviews on Amazon. Similar survival story vibes as books like "Into Thin Air" or "Into the Wild" but just on another level. It follows the story of a journey to Antarctica in 1914 that goes wrong and ends up with the ship trapped in ice for many months, and follows the crews absolutely insane attempts at survival.

I am waiting for time to forget enough of it so that I can read it again.

ozcap 4 hours ago | parent | next [–]

Definitely adding this to my list. I was stationed in Antarctica for three months earlier this year and it really gave me a newfound respect for these early explorers. I can’t imagine navigating such an environment with only relatively basic equipment and the stars.

pieix 4 hours ago | parent | prev | next [–]

> I am waiting for time to forget enough of it so that I can read it again.

Incredible book and story, I read it this year too and your statement resonates.

ARob109 4 hours ago | parent | prev | next [–]

Yes. I could not put this book down.

If you liked this one as an against the odds shipwreck survival story, David Grann’s The Wager is also very good.

dilawar 4 hours ago | prev | next [–]

Culture of Flowers by Jack Goodie. An anthropology book that compares the culture of Flowers across civilizations. "No flowers in Africa" to China being most gifted place in terms of flowers. Variability across Europe and Muslim civilizations etc etc.

As an Indian, I found how civilizations treat luxury (flowers) and necessity fascinating. Apparently Europe had a culture of garlands before the banquet culture. India is still predominantly garlands expect for a few highly westernised places. Muslim treatment or rather rejection of flowers in mosque but giving the world spectacle of flower gardens was very fascinating.

The book is not going to win any prize for prose or even coherence structure.

rashidae 4 hours ago | prev | next [–]

I’ve been getting into self-optimization lately, and sleep has been a game-changer. Two books that really opened my eyes are Why We Sleep by Matthew Walker and Outlive by Peter Attia. Walker breaks down why sleep is so critical, while Attia ties it into the bigger picture of health and longevity. Also been exploring Bryan Johnson’s Blueprint Protocol.

magnio 4 hours ago | prev | next [–]

"Heavenly Numbers: Astronomy and Authority in Early Imperial China" by Christopher Cullen

I read it for a course assignment. The book is structured as a continuous narrative of the development of astronomy throughout the Qin and Han dynasty in early China. It balances general information with mathematical details very nicely, so I feel it will appeal to both readers interested in the history or the astronomy. A pretty fun read for me, despite my purpose.

Brajeshwar 3 hours ago | prev | next [–]

This year, I decided to read more fiction, and I like it.[1] Here are some of the interesting non-fiction books I liked;

- Chip War: The Fight for the World’s Most Critical Technology by Chris Miller

- How We Got to Now: Six Innovations That Made the Modern World by Steven Johnson

- The Accidental Masterpiece: On the Art of Life and Vice Versa by Michael Kimmelman

- The Autobiography of Benjamin Franklin

- The TATAS: How a Family Built a Business and a Nation by Girish Kuber

1. https://brajeshwar.com/2024/books/

ilyaivanov 1 hour ago | prev | next [–]

I like reading Sophie's World by Jostein Gaarder. It is an introduction to philosophy in an interesting way.

kaycebasques 2 hours ago | prev | next [–]

Fundamentals: Ten Keys To Reality by Frank Wilczek (winner of 2004 Nobel in Physics)

rugyoga 3 hours ago | prev | next [–]

Material World: The Six Raw Materials That Shape Modern Civilization

I guessed about four but i was completely blown away by some of the details. Absolutely fascinating.

EasyMark 3 hours ago | parent | next [–]

This sounds right yp my alley. I'm not much of history or "human interest" reader like most of the other suggestions but this one sounds good.

wingworks 3 hours ago | prev | next [–]

The Simple Path to Wealth by JL Collins (the guy that started the FIRE movement.)

Great read, and not dry like many finance/investing books can be. Worth a read if you want to retire early, or even just retire normally.

If you happen to live in NZ, then Rich Enough by Mary Holm is a great read. (again about best way to manage your money, but focused on NZ)

stocknoob 2 hours ago | parent | next [–]

I think Your Money or Your Life (Vicki Robbins, 1992) predates it and helped inspire Mr Money Moustache, but Collins’ book is great too.

For everyone reading this, pursuing FIRE will impact your life more than any other book. Set the ball in motion and wait. Soon you’ll have time to enjoy all these other great books.

KRAKRISMOTT 4 hours ago | prev | next [–]

Extracting book titles from comments used to be an excellent exercise in named entity recognition, but with ChatGPT and transformer, it's now trivial.

kristianp 3 hours ago | prev | next [–]

The Ray Tracer Challenge by Jamis Buck. It uses Cucumber tests to specify the behaviour of the code, without giving you the code. Not a single weekend project. I've been doing it in C++ to learn about the modern language, but any language can be used.

tptacek 4 hours ago | prev | next [–]

I'll put a word in for _High And Rising_:

https://www.amazon.com/High-Rising-k-Soul/dp/0358494885

yaskour 3 hours ago | prev | next [–]

Code: The Hidden Language of Computer Hardware and Software

this book is a good read if you wanna know computer history , how it started and how it evolved

habosa 4 hours ago | prev | next [–]

I don’t read much non-fiction but for this year my pick is “Everyone Who is Gone is Here” by Jonathan Blitzer.

It’s a well-researched and original account of immigration at the US Southern border. It tells the story of many individuals and also whole Central/South American countries. The news media will tell you about caravans and people sneaking through Mexico but it’s hard to get a spin-free version of who is actually coming here and why.

It was enlightening and infuriating. Both political parties have lied to us for their benefit, and it made me realize I understood almost nothing about immigration before reading this. Highly recommend. Those of us who were born in the US are very lucky to not have to fight our way in.

ibash 4 hours ago | prev | next [–]

Oh no! Is it that time of year again?

khazhoux 4 hours ago | parent | next [–]

Yes. Cue the “I only read 453 books this year, but here’s my top 50” comments

etra0 4 hours ago | root | parent | next [–]

I am a terrible reader and I'm really ashamed of it. I don't know how people can read that many books, I think in my lifetime I've finished at most 4 books? and two of them were technical books.

I think I suffer from aphantasia which may be related to the impossibility of getting hook to anything by reading. I've tried mangas as well and it's a bit easier, but still find hard to really enjoy.

Of course, most of the books I tried are the ones that I had to read in school, but I also tried a few recommendations from friends and the result was the same.

It makes me a bit sad.

Brajeshwar 3 hours ago | root | parent | next [–]

To build up the habit or hopefully get passionate, try to read with/for your or your relative’s kid(s). Not necessarily together with the same book, just sit beside them on the couch, park, or wherever and see what conversation follows.

Need time to read? I find the best ones are while waiting to board your flight, Uber, and other transport.

nicbou 3 hours ago | root | parent | prev | next [–]

Try picking books just for fun, and reading in a pleasant setting. It makes a big difference.

hehehheh 4 hours ago | root | parent | prev | next [–]

I read 453 books if you run cat | wc on all the HN submissions I read.

vintageclothldn 4 hours ago | root | parent | prev | next [–]

It's the most wonderful time on the year..

Brajeshwar 3 hours ago | root | parent | next [–]

And we haven’t even started putting up the Christmas lights.

adastra22 4 hours ago | prev | next [–]

"The Alchemy of Air" by Thomas Hager. This covers the discovery of fertilizer, its strategic implications (it was the oil of its day), the creation in Germany of the Haber-Bosch process for producing fertilizer and gunpowder precursors from air, and how it fueled two world wars. An incredible story that is not widely known these days.

"The Making of the Atomic Bomb" by Richard Rhodes. This is an old one, but still a classic. The first half is like a scientific detective story about the discovery and development of quantum theory itself, in Germany and Denmark, and can be read by itself as an engaging history of physics. Then Hitler rises to power, we learn how most of the characters we've met so far escape to England and the USA, and the others except Bohr become part of the German nuclear project. It then becomes an engineering management history of the Manhattan project, and a fascinating look at the challenges they overcame. The final chapter is a sobering play by play description of what happened to the people of Hiroshima, and was hardest but necessary to read.

pinewurst 3 hours ago | prev | next [–]

Brezhnev: The Making of a Statesman by Susanne Schattenberg

andsoitis 4 hours ago | prev | next [–]

Sapiens

lovestory 1 hour ago | prev [–]

I only have time to listen to audiobooks so I hope this list is not too basic.

On the Edge: The Art of Risking Everything by Nate Silver

The End of Reality: How Four Billionaires Are Selling a Fantasy Future of the Metaverse, Mars, and Crypto by Jonathan Taplin

The Value of Others by Orion Taraban

Rinsed: From Cartels to Crypto: How the Tech Industry Washes Money for the World's Deadliest Crooks by Geoff White

Why We Die: The New Science of Aging and the Quest for Immortality by Venki Ramakrishnan

Extremely Hardcore: Inside Elon Musk's Twitter by Zoe Schiffer

Bottle of Lies: The Inside Story of the Generic Drug Boom by Katherine Eban

AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference by Arvind Narayanan and Sayash Kapoor

Hijacking Bitcoin: The Hidden History of BTC by Roger Ver

Go Woke, Go Broke: The Inside Story of the Radicalization of Corporate America by Charles Gasparino

The Catalyst: RNA and the Quest to Unlock Life's Deepest Secrets by Thomas R Cech

and my book of the year

Good Energy: The Surprising Connection Between Metabolism and Limitless Health by Casey Means M D

AIMagicStudio – Turn Keywords or URLs into Videos in Minutes

news.ycombinator.com/item?id=42218715

1 point by nagudharan 5 hours ago | hide | past | favorite | discuss

Hi HN,

Over the past year, I've been building AIMagicStudio, a tool designed to simplify video creation and save countless hours for creators, marketers, and entrepreneurs. It's a side project that grew from my frustration with creating professional videos, which is a complex and time-consuming process.

Today, I'm conducting a soft launch to gather feedback and start a discussion about the challenges of video creation and automation.

How AIMagicStudio Works: 1. Provide a keyword, URL, or Amazon product link, and AIMagicStudio generates a full video for you. This includes scripts, voiceovers, AI images, and captions.

2. Choose from over 15 subtitle styles and 10+ layouts to match your branding.

3. Add personalized AI-generated images for consistent visual storytelling.

4. Schedule videos to be created automatically based on a list of keywords or URLs.

5. Automate uploading to platforms like YouTube, Instagram, and TikTok with optimized tags and descriptions.

Perfect for creators who want to remain behind the scenes while producing engaging content.

Why AIMagicStudio? I built AIMagicStudio to address the struggles many face with video marketing: the technical barriers, the time sink, and the cost of hiring professionals. Whether you're creating Amazon product reviews, instructional videos, or short-form content, AIMagicStudio automates the most tedious parts so you can focus on growing your audience or business.

Exclusive for HackerNews: To celebrate the launch and gather early feedback, I’m offering a 7-day free trial of the Premium Plan, no credit card required.

Get your 7-day Premium Plan free trial: https://app.aimagicstudio.com/buy?package=Premium%20Plan

What’s Next? I’m looking for your thoughts on:

- The usability of the platform. - Features you'd love to see. - Ideas to make this more valuable for creators like yourself.

This is still a soft launch, so I'm prioritizing feedback and improving the experience. Feel free to poke around and let me know what you think.

Looking forward to your feedback and insights, HN!

Cheers,

Ask HN: What gives Elon Musk's companies their edge?

news.ycombinator.com/item?id=42218573

5 points by atleastoptimal 6 hours ago | hide | past | favorite | 9 comments

Something I learned over time is to judge people by the results they bring, not my personal feelings and whether I like them. As a consequence I've had to concede the fact that Elon Musk possesses a "key" to managing tech companies that routinely puts him far above the norm despite his many detractors.

His track record in terms of reliability, range and magnitude of success is unprecedented and signals that despite all his apparent personal faults he possesses a skill, sensitivity or decision making capability that puts him ahead of thousands of other very smart, hardworking people.

Does anyone have any insight, based on what they've read, seen or know about working at his companies that provides an explanation for this?

WheelsAtLarge 5 hours ago | next [–]

In the past, I've met people who are driven by results but don't always consider the feelings of others. They expect tasks to be completed without thinking about how the process affects those involved. I believe this mindset has contributed to Musk's success; he expects both himself and those around him to fulfill their assigned roles. If someone doesn't thrive in that competitive environment, they'll likely leave. Musk seems to attract individuals who are ready for this level of intensity. After all, he doesn't do all the work himself; thousands of people working under him carry out the tasks. He's simply an effective leader. Many may dislike his style, but evidently, those who work for him do not.

Additionally, his control over social media has significantly contributed to his ability to attract substantial capital. With this capital, he can pursue various initiatives. He has a knack for crafting a compelling story that encourages people to invest in his company in both time and money.

GianFabien 4 hours ago | prev | next [–]

Based entirely on observations from public sources, I think Elon is first and foremost a superb salesman. Beginning with his move from South Africa, to Canada and thence USA is the beginning of his process of incrementally hustling to the next best thing . It is how he parlays his wins from one investment into the next.

Far less attention is given to the fact that he must be hiring lots of extremely talented people and motivating them to deliver above average outcomes. As a stereotypical manager of managers he bestows upon himself all the credit in public. Thus further enhancing his reputation .

His private life is further proof of his complete disregard for norms and conventions. Elon shows a cavalier disregard of public institutions, e.g. SEC, yet manages to avail his companies to billions in government subsidies.

k310 4 hours ago | prev | next [–]

Long story deleted. He runs things like a Bond villain.

If you want some granularity in that: risk taking, going against the grain, breaking rules, drive, grandiosity, good physics skills and poor people skills (lack of empathy). I read somewhere that the laws of physics constrain his engineering companies, whereas absolutely nothing constrains the "social" business, though the section 230 free ride may be grinding to a halt.

https://www.thebignewsletter.com/p/judges-rule-big-techs-fre...

talldayo 6 hours ago | prev | next [–]

It's really easy to promise things you have no intent of doing, raising capital, and then liquidating the shares you own at inflated value to sponsor even more insane moonshot ideas. If you're smart you can even shoehorn in a few government subsidies predicated on a product category you abandoned behind closed doors.

Between FSD, the $25,000 EV and uh... X, I'm getting pretty convinced that publicly recognized success isn't the goal anymore.

atleastoptimal 5 hours ago | parent | next [–]

If it were so easy then why doesn't every billionaire/CEO do it? Elon's companies are vastly outcompeting in their specific domains (Tesla for electric cars, SpaceX for space, satellite internet), despite the many failures on the way.

austin-cheney 5 hours ago | root | parent | next [–]

Are they? I don’t think there is any evidence of that.

https://companiesmarketcap.com/automakers/largest-automakers...

When looking at revenue there is no magic there for any of these companies. It’s just about fundraising.

bdangubic 5 hours ago | root | parent | prev | next [–]

it is also not easy to become POTUS but someone like Donald pulled it off twice :)

talldayo 4 hours ago | root | parent | prev | next [–]

It can't be that hard, Theranos and Enron figured it out too.

DemocracyFTW2 4 hours ago | prev [–]

Lies, Lies, Lies, and then some more Lies

Ask HN: A monitoring app for new comments on your favorite HN posts

news.ycombinator.com/item?id=42218521

2 points by timz 6 hours ago | hide | past | favorite | 2 comments

Thinking of the following weekend project, which would allow users to get notified over email whenever a HN post in their favorites, gets new comments or updates. Could be your post or someone else.

It could work as follows:

- Go to HN and find a post you like, visit the post page like https://news.ycombinator.com/item?id=1 - Add it to your favorites list by clicking [favorite] right at the top

The following signup needs to be done only once:

- visit https://todo-hn-notifier.com - sign in with email - provide your HN username

Whenever a new comment appears in one of the favorite posts, an email is being sent. (With an easy unsubscribe link)

WDYT? Would you find such a web service useful? Any extra features would be cool to add?

Thank you.

walterbell 5 hours ago [–]

Aren't new comments blocked after N days?

timz 4 hours ago | parent [–]

yes, indeed, they are. but within those 2 weeks notifications could still be useful, i think.

Where Can I Find San Francisco Bay Area Angel Investors for My Podcast?

news.ycombinator.com/item?id=42217868

1 point by ttsemih 8 hours ago | hide | past | favorite | 2 comments

Hi all,I have a startup podcast that focuses on introducing the startup ecosystem to the younger generation. I have successfully completed two episodes featuring investors who have backed unicorn startups as angel investors. I want to create some face-to-face episodes and invite more angel investors. Do you know where I can find a list of angel investors living around the San Francisco Bay Area? Or do you have any recommendations for individuals to invite?

If you want to check out my podcast, you can visit https://www.tiktok.com/@founderapp.com.

jschveibinz 8 hours ago [–]

I suggest you start here:

https://angelcapitalassociation.org/

ttsemih 6 hours ago | parent [–]

Thank you! I start now

Is there demand for free product manager advisory services in startup community?

news.ycombinator.com/item?id=42217837

1 point by ProductMngrUK 8 hours ago | hide | past | favorite | discuss

Hi everyone,

I’m a UK-based B2B product manager with experience working with both small and large software businesses, owning entire suites of large and complex products. Selling into the upper market, including government.

I'm looking to offer free advisory services to help small software businesses or startups with product strategy, roadmap planning, and general product management challenges.

Why? Well, I'm hoping to network, build some connections and friendships, and maybe get a few testimonials.

However, I'm curious to know if there is actually a demand for it. Are any of you founders of funded or revenue-generating businesses that would appreciate some extra support and advice each week? Happy to even get a little hands-on.

If you're interested, feel free to respond to this post. If your contact details are public on your Hacker News profile, I'm happy to reach out directly to arrange an initial chat.

Looking forward to connecting with some of you!

Ask HN: Is distributed LLM training in browsers (WebRTC and WebGPU) possible?

news.ycombinator.com/item?id=42217823

3 points by trekhleb 8 hours ago | hide | past | favorite | 5 comments

xyzsparetimexyz 43 minutes ago | next [–]

Distributed training in general isn't practical.

trekhleb 8 hours ago | prev | next [–]

What I mean is training something like GPT-3 in a distributed manner using a large number of regular browsers or laptops with average WebGPU support/power and WebRTC for communication.

Does it even make sense to ask this? Is it reasonable or feasible?

I understand there are many nuances, such as the size and source of the training data, the size of the model (which would be too large for any browser to handle), network overhead, and the challenge of merging all the pieces together, among others. However, speculative calculations suggest that GPT-3 required around 3x10^22 FLOPs, which might (very speculatively) be equivalent to about 3,000 regular GPUs, each with an average performance of 6 TFLOPs, training it for ~30 days (which also sounds silly, I understand).

Of course, these are naive and highly speculative calculations that don’t account for whether it’s even possible to split the dataset, model, and training process into manageable pieces across such a setup.

But if this direction is not totally nonsensical, does it mean that even with a tremendous network overhead there is a huge potential for scaling (there are potentially a lot of laptops connected to the internet that potentially and voluntary could be used for training)?

astlouis44 8 hours ago | prev | next [–]

Client-side inference will be a bigger opportunity for AI in the browser than client-side training, in my opion.

soheil 8 hours ago | prev | next [–]

This is the promise of WebGPU. Once all major browsers fully adopt it it'll be a game change for any ML training. Apple Silicon is the giant no one is talking about training is almost exclusively done on NVDA chips. Cost per TFlop is the lowest on Apple chips both operational and acquisition (mac mini).

gtsop 8 hours ago | prev [–]

Possible? Yes. Practical? No

It is possible because training is already being done using distributed methods in the datacenter. It is not practical because of the sheer volume of network delay piled on top of the computation

Ask HN: To those with successful browser extension(s), how did you grow it?

news.ycombinator.com/item?id=42217504

52 points by dandrew5 9 hours ago | hide | past | favorite | 38 comments

I maintain a handful of browser extensions and one of them is gaining traction. I'm now looking for marketing avenues and curious to hear from anyone that may have gone down this path before.

Since it appears the Firefox and Chrome stores themselves don't offer any kind of visibility-boosting features, are there any other tools/services you've found useful to increase the number of active weekly users?

kevmo314 8 hours ago | next [–]

I used to have a pretty successful browser extension that I shut down because the site it worked on shut down.

I grew it by adding features that people would ask for on the site's forums, for example a user would say "hey can we have x feature?" and I would respond saying "great idea, I added x feature to my extension y!" This was very effective and over time others would start responding saying my extension had whatever feature they were asking, capitalizing on how relatively slow companies are to implement features.

This does rely on the extension having a site it operates on and having a forum for users though. If I were to do it today I'd focus on finding places where my extension's users concentrate, Discord, a community Slack, or otherwise, and doing the same thing.

throwup238 8 hours ago | parent | next [–]

I have a similar type of product that depends on a third party and I use LLMs to automate most of the work. I feed it batches of Discord/Discourse messages from the third party’s community along with a prompt containing my feature set and it flags messages that might be relevant so that I can reply to them (manually, although the LLM generates some starting points based on examples replies).

The false positive rate at first was over 50% but with some prompt tweaks and back testing, it’s approaching 10-20%

kevmo314 7 hours ago | root | parent | next [–]

That tool itself might be worth publishing. I've noticed some cofounders (Supabase's kiwicopple comes to mind) that are very on top of HN comments and I am not sure if they're literally reading all the HN comments or doing that in some automated, "notify me when someone talks about Supabase" kind of way. Could be very valuable to notify when even a related discussion happens!

rovr138 2 hours ago | root | parent | next [–]

‘Brand monitoring’ is definitely a thing and there are platforms for it.

You could even start very basic with Google alerts for example.

Still up for new things. Looks like a fun project.

dandrew5 8 hours ago | parent | prev | next [–]

Great ideas, thanks. I've had similar success searching for common problems using Google, finding the highest-positioned Reddit result, and leaving a response with a link.

mfrisbie 7 hours ago | prev | next [–]

I run Track & Trace Tools[1], an extension directly targeted at the legal cannabis industry's main compliance platform Metrc.

Since it's targeted at businesses and not consumers, word of mouth has been powerful for me. I've rapidly grown in the state of Michigan thanks to a few very enthusiastic operators.

I also went on a slate of podcasts last year to talk about (read: promote) my product. Podcast hosts are always looking for guests, and most of them didn't cost me a thing.

It's also been helpful to be a Google Developer Expert for web extensions, the Chrome team interviewed me and that gave me a platform to talk about what I was working on[2].

[1] https://trackandtrace.tools

[2] https://youtu.be/8P-Sc8ZaViY?si=tLhx3LMIlNqHBvfB

pimlottc 7 hours ago | parent | next [–]

> Podcast hosts are always looking for guests, and most of them didn't cost me a thing.

That implies that for some podcasts you can pay to be a guest?

mfrisbie 7 hours ago | root | parent | next [–]

Some charge a nominal fee, and one I had to travel to a studio to record. Made for a nice weekend in LA with my wife.

purple-leafy 3 hours ago | prev | next [–]

I have made 4 extensions in the last 1 year, with over 1000 users.

- 1 over 100,000

- 1 over 10,000

- 1 over 5,000

- 1 over 1000

Only marketing was a single post on Reddit for each.

Getting a featured badge is a must for organic growth on the chrome store. Without it, you're screwed.

I have other extensions hovering around ~100 users that don't grow as they don't have a featured badge.

dandrew5 3 hours ago | parent | next [–]

For your Reddit posts, did you target a subreddit that might have interest in the extension or did you use one meant specifically for extension development/promotion? If the latter, which one?

purple-leafy 1 hour ago | root | parent | next [–]

I used mainly r/webdev showoff saturdays

r/chromeextensions is too small to get any traction, and only has other extension developers

switz 9 hours ago | prev | next [–]

I have no experience to share on the creator side, but as a user I am ever fearful of extensions getting rugpulled. Many popular but fading extensions sell out to nefarious companies just looking for users to adspam. Please don't do that.

The simplest way to grow it is to share the link to it in this thread – you have eyes on you, take advantage of them.

dandrew5 8 hours ago | parent | next [–]

Thanks for the idea of sharing the extension but I'd like to focus any discussion around growth itself.

As for being concerned about a rugpull, I definitely get it. My approach is to keep as much of the product open source as possible and be very transparent around any ownership changes.

ATechGuy 8 hours ago | root | parent | next [–]

I've built a tool for this. How can I DM you?

dandrew5 8 hours ago | root | parent | next [–]

I've added my contact email to my profile, thank you.

ATechGuy 8 hours ago | root | parent | next [–]

You will hear from me shortly. Thanks!

Alupis 7 hours ago | root | parent | next [–]

You all have to know this guy is most likely trying to scam you?

Has a secret tool that will grant you success, can only DM you, won't share in public. Scamola if I've ever smelled one...

datasert 8 hours ago | root | parent | prev | next [–]

I would like to know as well. Can you please DM me?

ATechGuy 8 hours ago | root | parent | next [–]

Sure, but there's no contact info in your profile.

tconfrey 8 hours ago | prev | next [–]

I think it's like marketing any other type of software or service online - SEO, find user communities, influencers, ads/paid promotions etc etc. In my case I got lucky and impressed a tech writer who gave BrainTool a write up on ZDNet and gave me a jumpstart on the first 2K+ downloads. I wrote about the experience here:

https://braintool.org/2021/01/14/First-Two-Thousand-BrainToo...

acorn221 4 hours ago | prev | next [–]

Hey, I am the creator and owner of LighterFuel For Tinder, an extension which shows when your matches made their accounts amongst other features designed to help identify fake accounts. LighterFuel grew by its self over 3 years thankfully, without any external promotion.

I've recently been trying to promote my other extensions though and I'm having a lot of trouble getting any traction with them too, and I managed to successfully run a small media campaign where I made myself the most liked man on Tinder for a day by making my users swipe on my account for April fools. Although I did meet my target of 1 tabloid news article, there was no noticeable uptick in installs from that article, however it did get the attention of some YouTubers recently and I'm just working with them now to promote LighterFuel and my other extensions.

It's difficult to know the best way to promote your extensions but I'd personally start with focusing on the Chrome web store, as it's the biggest extension store, getting your listings to look great so you can then get your featured badge which looks also good to users, then linking a website to your extension to get the "verified" badge which means "Created by the owner of the listed website. The publisher has a good record with no history of violations. " which makes it look more official.

From there, I would recommend getting in key words if possible (in my case Tinder helped) to just get better SEO and discovery.

Then finally, I would always recommend setting up your own analytics (I use Google Analytics) as I've found the CWS store analytics to be unreliable.

Get your featured badge here: https://support.google.com/chrome_webstore/contact/one_stop_...

My news article: https://www.thesun.co.uk/news/27142401/tinder-hack-swiping-r... The LighterFuel listing: https://chromewebstore.google.com/detail/lighterfuel-for-tin... My LinkedIn article about the campaign: https://www.linkedin.com/pulse/how-i-became-most-liked-man-t...

jborden13 8 hours ago | prev | next [–]

Do you have any creative ways to source traffic?

I run:

https://www.getspence.ai/

I've sourced data important to our users. I am leveraging it by making it useful for our users and to drive traffic and visibility to our services. My data will be discoverable on broad platforms.

wordpad25 7 hours ago | parent | next [–]

looks like you format resumes, what kinda data are you talking about?

jborden13 3 hours ago | root | parent | next [–]

It's related to the service and a means to source traffic. Unfortunately it's not launched yet but relevant to the services utility.

smallerfish 8 hours ago | prev | next [–]

Since this is the right crowd, for those who make money with your extension, how do you do it? I'm contemplating building an API for license checking and a simple webapp for registration / subscription management, but if anybody's using the off-the-shelf options I'd be interested to hear about your experience.

tconfrey 8 hours ago | parent | next [–]

I've heard good things about ExtensionPay[1]. It didn't have what I needed at the time so I hand rolled my own on Firebase using a Stripe integration component[2]. As you probably know the issue with extensions is that the whole codebase is exposed so you need some kind of back end if you're handling payments and associated secret stores.

[1] https://extensionpay.com/ [2] https://extensions.dev/extensions/invertase/firestore-stripe...

dandrew5 8 hours ago | parent | prev | next [–]

I'm in the process of adding payments as well and I'm starting things off with Stripe. Specifically, their payment links and subscriptions features. Longer term, I may switch, and am building things in a way that migration will be easy.

tconfrey 7 hours ago | prev | next [–]

While there aren't visibility-boosting features, don't downplay SEO on the app stores. That is, if your's is the kind of extension that people will be searching for when looking the solve their problem. The Chrome store also shows a bunch of 'related' extensions at the bottom of every listing. It can't hurt to show up there.

Also, since you don't mention it, list on the Edge Add-ons store. My user count over there is ~40% of the Chrome store number.

dandrew5 7 hours ago | parent | next [–]

Great call on expanding to other add-ons stores. I'll need to look into compatibility and find a Windows machine to test...

tconfrey 7 hours ago | root | parent | next [–]

The Edge store was stricter in some ways but I find I can submit pretty much the same bundle to both. Note also that Edge runs fine on MacOS.

Guillaume86 7 hours ago | root | parent | prev | next [–]

Edge has Linux and MacOS builds

OlegWock 8 hours ago | prev | next [–]

I was lucky enough that video about my extension blew up on tiktok (it wasn't even my video lol), which bring around 34k users. After that extension started slowly getting more traction from here and there (tech sites, personal newsletters, etc), but that didn't bring any substantial audience, to this day (1.5 years passed) tiktok was the biggest success and provided most of current audience, partially because I stopped actively working on the extension about year or so ago

serial_dev 8 hours ago | parent | next [–]

It would be great to have a link to the TikTok video and your extension.

OlegWock 33 minutes ago | root | parent | next [–]

Unfortunately, original poster deleted his account/got banned on tiktok

julianeon 8 hours ago | prev | next [–]

What are some useful extensions for the HN crowd?

hehehheh 4 hours ago | parent | next [–]

YARC - Yet Another Rest Client https://chromewebstore.google.com/detail/yet-another-rest-cl...

dmitrygr 8 hours ago | prev [–]

> to increase the number of active weekly users?

what is the motivation to seek this goal? Is your extension paid?

dandrew5 8 hours ago | parent [–]

Yes, I have a monetization plan in mind.

Ask HN: Will more people leave cities due to satellite internet in rural areas?

news.ycombinator.com/item?id=42217428

4 points by amichail 10 hours ago | hide | past | favorite | 11 comments

Would you consider doing that?

solardev 7 hours ago | next [–]

FWIW, in between, there are also a lot of small towns that are only "semi-rural": short distance from nature and the outdoors, reasonable access to essential healthcare, entertainment, groceries, etc., but without the crazy traffic, costs, and crime of the big cities. You can easily get internet access (cable, satellite, fiber sometimes), things aren't quite as expensive as the metro areas (though not cheap by rural standards either), and there's still a fair amount of diversity (of culture/thought/politics/food/etc.).

Problem is, these towns tend to have a limited lifespan, often being undiscovered gems for a few decades, and then either they turn into ghost towns or they get too popular and rapidly gentrified and become unaffordable (like Boulder, CO).

It would be a fun GIS problem to identify the next up-and-coming town like this and move there a few years before it really explodes. Then do it again for the next one, etc. You'd have a great quality of life and also probably make a fortune flipping houses a decade later. Assuming you can stay remote, of course... but the same man making flying internet hates it when his minions aren't in their cages.

marssaxman 9 hours ago | prev | next [–]

My family has already tried that, given up on it, and moved back to central Seattle. The rural life is lonely and boring, and driving everywhere all the time got really old.

blackeyeblitzar 8 hours ago | parent | next [–]

Getting around anywhere in Seattle, even though the distances are short, is still hard. Traffic is far, far worse than it was ten years ago. It takes more time to the point that I can’t say the access is truly greater than a rural area unless you choose to live in a neighborhood with high walkability and your chosen destinations aren’t outside of that.

marssaxman 8 hours ago | root | parent | next [–]

Yes, that's exactly what we've done - I walk every day now, almost everywhere I go. It's nice.

chozang 3 hours ago | prev | next [–]

I would consider doing that, but we have a house now, plus my wife likes being near the city.

dave4420 9 hours ago | prev | next [–]

I can get satellite internet in the city. Why would I move to the sticks to get satellite internet?

solardev 7 hours ago | parent | next [–]

Less particulate pollution in the air reflecting the signal means higher bandwidth!

linguae 8 hours ago | prev | next [–]

Three years ago I had a hybrid job based on Silicon Valley; it was fully remote during COVID but switched to in office once per week when restrictions were lifted (this switch did not come as a surprise; we knew throughout the pandemic that full remote was a temporary situation). I wanted to buy a home in Santa Cruz County, but I kept getting outbid beyond my (admittedly small) budget of about $525,000. I then considered rural areas; I even drove to an open house near Lake San Antonio about 30 minutes northwest of Paso Robles to see a brand new house that sat on a one-acre lot. The home was around $450,000, and it was located in Starlink’s service zone.

After some consideration, however, I ruled out living so far away from town. One significant factor was access to health care. Specialty doctors could be hard to find, and if I’m in an emergency, it’s a long trip to either King City or Paso Robles. Another major concern, though this is not a universal problem, is the exploding cost of home and fire insurance, especially in fire-prone zones like many of California’s less-populated areas near hills and mountains.

I like the peacefulness and beauty of less-populated areas, but I rely on city services, and so I prefer to live in or near a city.

pestatije 7 hours ago | prev | next [–]

no, people are not stupid...if Elon has one of his happy ideas you might end up with horrendously expensive internet or, who knows, no internet at all.

WarOnPrivacy 9 hours ago | prev | next [–]

> Will more people leave cities due to satellite internet in rural areas?

Finding a suitable property can be super difficult. Internet providers can be an invisible factor in that kind of equation. Folks tend to get where they're going, realize there isn't a worthwhile wireline ISP and sign up for 4G/5G wireless (less expensive than satellite).

Where 4G/5G isn't available, other services are also lacking (shopping, medical, schooling). That is a stronger decider.

> Would you consider doing that?

Within state, no. My semi-rural county was trenched for fiber last year. I have symmetrical 2Gb @ $80/mo.

To leave the state, I probably would. I'd give up a lot to escape Florida.

blackeyeblitzar 8 hours ago | prev [–]

I am not sure how much it will help actually rural areas, but it could definitely help with smaller cities where the internet infrastructure is poor. I think though it is less about satellite internet and more about the higher tolerance for remote work.

Ask HN: How do you manage ideas, tasks, notes and other stuff?

news.ycombinator.com/item?id=42217029

2 points by vednig 11 hours ago | hide | past | favorite | 2 comments

Here's link to a previus thread https://news.ycombinator.com/item?id=8282569 I encountered.

But in the last 10 years a lot has changed, new apps, new tools, new research and new ideologies have surfaced. I'd like to know how you actually manage alot of things.

Personally being a 20 year founder I find this very hard and miss a lot of things because of unorganized management.

ale_jacques 9 hours ago [–]

I've been using NotePlan ( https://noteplan.co ) with the Projects + Reviews plugin. It's been a game changer for me. The (almost) perfect combination of tasks + notes.

I also manage my personal stuff with it.

It's a paid macOS app but, IMO, worth every penny.

vednig 7 hours ago | parent [–]

Had a look, at their website, it does have everything I would ask for in Management App. Does it also have Raycast support?

Our SaaS hit 50 paying subscribers today. Here's how we did it with $0:

news.ycombinator.com/item?id=42216617

7 points by davidheikka 11 hours ago | hide | past | favorite | 3 comments

I will get to the marketing part, but first we needed an actual good product. Else marketing won’t work.

Here’s the 4 step approach we followed for building something people wanted:

1. We started by validating our idea before building anything at all.

2. Once we got a green check on that, we built an MVP fast.

3. Then we launched the MVP, got our initial users through content marketing which validated the project further.

4. At this point we built the full product.

Going to market is a lot easier when you know you have a validated product.

Our first marketing move was to launch on Product Hunt.

We had heard some skepticism around Product Hunt launches so we didn’t expect much but here’s what happened:

- Before launching we had 0 paying subscribers.

- On the evening of our launch day we got our first paying subscriber.

- One week later we had 20.

Definitely a success.

What we did to maximize our launch:

We were very active on X on the launch day, posting a lot of content and keeping people updated on how the launch was going.

That led to a lot of momentum actually. People became invested in our launch.

We also posted on Reddit, but much less so.

After the launch we experienced a dip, which was to be expected. But we worked our way through it.

The next steps were clear for us:

1. Talk to users and improve our product

2. Keep up the momentum we built from content marketing

3. Find a better long term marketing source (we’re focusing on SEO now)

We have now hit 50 paying subscribers and growth is looking really good.

That’s how we took our project from 0 to our first paying subscribers.

I hope I was able to provide some insight on how that journey might play out. Happy to provide more input in the comments.

cryptozeus 11 hours ago | next [–]

Very nice intro and product ? Did you use your own product to launch your own product:) ?

davidheikka 11 hours ago | prev [–]

The project: https://buildpad.io

bootstrpppin 7 hours ago | parent [–]

Feature request: bake in Google traffic volume data.

Google traffic represents a much wider sample of the world/audience compared to Reddit which represents a very narrow audience in comparison.

How to build something people will pay you for, in 5 steps

news.ycombinator.com/item?id=42216533

3 points by UltraIngo 12 hours ago | hide | past | favorite | 2 comments

1. Look at all the problems you experience yourself, pick a painful one.

2. Talk to people who experience the same problem through DMs or surveys.

3. Build MVP that solves problem.

4. Share MVP to the same people you talked to before.

5. Improve MVP based on their feedback.

Now you have a product worth paying for.

This is how I did it for my project Buildpad and it's now at 1800+ users + 50+ paying customers, so don't say it doesn't work.

jschveibinz 7 hours ago | next [–]

Congratulations on your success.

I would suggest a slightly different set of steps that are more generic (for startups other than just software):

1. Look at all the problems you experience yourself, pick a painful one.

2. Talk to people who experience the same problem through DMs or surveys. Note those who might be excited early adopters.

3. Write out your requirements and analyze. Note the really important stuff, the high value stuff, the hard stuff, the safety stuff, and the regulation or standardization stuff. These are "must do" requirements.

4. Design and build one or more prototypes that solve(s) the problem or an important aspect of the problem. If it's a service business, prototype your processes. Concentrate on the most important requirements first.

5. Share or demonstrate your prototype(s) with the excited people you talked to before. Get their feedback, and revise your requirements where necessary. Do your final design, as required--drawings, BOM, suppliers, manufacturers, assemblers, etc.

6. Now build your MVP based on the updated requirements and design--build multiple copies, as required--and go back for more demos and conversations.

The next focus is on getting support for initial orders and/or marketing assistance ...then begin pre-marketing in earnest in advance of launch to build interest and some pre-orders.

UltraIngo 12 hours ago | prev [–]

Here's a link for the curious: https://buildpad.io

Ask HN: If every company is using AI now, how to set your startup apart?

news.ycombinator.com/item?id=42216367

3 points by ATechGuy 12 hours ago | hide | past | favorite | 4 comments

codingdave 12 hours ago | next [–]

Same as before AI - by solving a problem. AI is a tool, not a goal. If it can solve a problem, great... build a solution with it. If not, build a solution without it. But focus on what you are solving, not what flavor of hammer you are using to pound that nail.

ATechGuy 12 hours ago | parent | next [–]

I understand what you mean. However, since GenAI brings efficiency to the table, a number of startups are now trying to compete with existing giants (solving real problems) and claiming to make processes more efficient. The problem arises when existing companies adopt GenAI as well and make the same claims.

handfuloflight 12 hours ago | prev [–]

For starters, don't shout at the market that AI is your differentiator. Lead by value, not by adoption of technology.

ATechGuy 12 hours ago | parent [–]

See my comment about regarding adding value.

Ask HN: How do I get off marketing email lists from IT vendors?

news.ycombinator.com/item?id=42216261

2 points by therealfiona 12 hours ago | hide | past | favorite | 2 comments

I get a bunch of emails from folks trying to sell my employer stuff. I am simply not the person to be selling stuff to...

I have even received phone calls on my personal cell phone about it many times. One guy actually told me which list they use and I was able to remove myself from it. This cut down a lot of spam, but not all.

Does anyone have a collection of Opt-Out links for this kind of spam?

The one I know of is for Apollo.

https://www.apollo.io/privacy-policy/remove

pwg 12 hours ago | next [–]

> have even received phone calls on my personal cell phone

For this, if you are in the US, first, if you have not already done so, add your personal cell phone # to the do not call list.

Then, for any of these calls, first collect the business names/info and names of those calling (which they will likely give, being salespeople). After you get that info written down, then inform them that they are in violation of the do not call law and that you are reporting them to the FTC for violating the do not call law.

And, of course, follow through on reporting them to the FTC:

https://www.donotcall.gov/faq.html

Doing so just might scare some of them into either removing your number, or adding it to a block list so they don't call it anymore.

toomuchtodo 12 hours ago | prev [–]

1. Unsubscribe link

2. Block at mail server (first individual email, domain if mails persist)

(our corp policy is to ask for these unwanted solicitations to be forwarded to a mail alias to be dropped at the email security gateway, if emails persist)

Ask HN: Do you want more software developer in your team?

news.ycombinator.com/item?id=42216247

1 point by Khemmapich 12 hours ago | hide | past | favorite | discuss

Hi I have idea to build Software Developers AI agent that can help you complete software development tasks faster by just assign it without your intervention after that with context from your Github repository And it can also reflect themselves for its work and it can edit until it think that is the best work to submit to you

Example: - Design architecture - Write code with ability to Research on the internet by itself, Read docs that you provide to them - Help debug code - Write test and Run test - Help review your code on Github - Open Pull Request - It can talk with you like your colleagues but focus on software development and your project as a context

Table of Contents

HN With Actual Comments

Hacker News Frontpage

UK sees privatisation 'opportunities' in Ukraine war

My new POWER Indigo 2

Americans see their savings vanish in Synapse fintech crisis

Quake 3 Source Code Review: Network Model (2012)

MaXX Interactive Desktop -- the little brother of the great SGI Desktop on IRIX

The Neuroscience Link Between Neuroticism and Social Media Addiction

RGFW: Single-header C99 window abstraction library

Translating My Grandfather's Biograpy

Understanding Google's Quantum Error Correction Breakthrough

Phased Array Microphone (2023)

Rendering "modern" Winamp skins in the browser

Amazon to invest another $4B in Anthropic

The Deceptively Asymmetric Unit Sphere

Salmon return to lay eggs in historic habitat after dam removal project

Runtime-Extensible SQL Parsers Using Peg

From string to AST: parsing (2019)

Prince Nico Mbarga’s biggest hit outsold any of The Beatles’ (2017)

Show HN: A Marble Madness-inspired WebGL game we built for Netlify

FaSTer: Atari ST Digital Magazine

Ask Hacker News

Recommend any clipboard app for Mac

Free Qwen 2.5, Llama Nemotron, OpenBioLLM, Gemini Exp 1121, and LearnLM 1.5 Pro

Ask HN: How else to live more minimally?

Ask HN: Discords or chat groups for builders/hackers?

Ask HN: Typed "AirPods" in notes on iPhone, 2 hours later AirPods ad on YouTube?

Ask HN: Best non-fiction book you read in 2024?

AIMagicStudio – Turn Keywords or URLs into Videos in Minutes

Ask HN: What gives Elon Musk's companies their edge?

Ask HN: A monitoring app for new comments on your favorite HN posts

Where Can I Find San Francisco Bay Area Angel Investors for My Podcast?

Is there demand for free product manager advisory services in startup community?

Ask HN: Is distributed LLM training in browsers (WebRTC and WebGPU) possible?

Ask HN: To those with successful browser extension(s), how did you grow it?

Ask HN: Will more people leave cities due to satellite internet in rural areas?

Ask HN: How do you manage ideas, tasks, notes and other stuff?

Our SaaS hit 50 paying subscribers today. Here's how we did it with $0:

How to build something people will pay you for, in 5 steps

Ask HN: If every company is using AI now, how to set your startup apart?

Ask HN: How do I get off marketing email lists from IT vendors?

Ask HN: Do you want more software developer in your team?